My working approach using Tekton kaniko to push image to AWS ECR

Published in

FAUN — Developer Community 🐾

4 min readJan 12, 2024

Background

In continuous integration, the tasks like git pull from repo, build, test then push the image to docker repo is a typical chain of actions.

Tekton provide a starter guide on doing that, which target a docker hub repo.

In most other enterprise, we are not pushing to docker hub, like mine, we push to AWS, problem of AWS secret is its password live a short life of 12 hour, so we cannot prepare a static password docker file as secret/config map and use it every time.

Luckily kaniko have the AWS credential helper built in, but the starter guide approach won’t work.

Why the starter guide approach not work?

The stater guide provide an example of Tekton pipeline that use a task reference to a prepared kaniko task, they even have a section that tell you how to prepare your credential.

First I take a look at the Tekton kaniko task documentation, it does not quite explain how the secret is being pass to kaniko.

Then go to kaniko github page, they mentioned the usage of AWS credential helper, and having a section to explain that.

As I prefer the secret approach, I prepare the secret as the document said:

kubectl create secret generic aws-secret --from-file=<path to .aws/credentials>

But I do not understand the pod spec (which hours of Googling copy and paste the same information in many articles I read) in the article:

The problem is the starter guide use a Task reference, which the kaniko task is already written, and so the AWS serect is not mounted to the correct location.

The solution

The solution is rather straight forward after understanding that, I have to modify the kaniko task Tekton provided (download it from the link)

And prepare a new workspace that mount the secret at the pipeline run and pass it all the way to the task

Making this new change to the kaniko task definition downloaded

Then prepare the dockerconfig as config map with content as follow (this is following AWS credential helper instruction):

{
 "credsStore": "ecr-login"
}

And it should work OK.

Alternate approach that work too is using env variable (instead of mounting a secret), which still need to modify the Tekton kaniko task:

Other gotcha

use the default profile of AWS credential

Because I have multiple profiles in my aws credential, when I try to prepare my secret by cutting out the preferred profile, my profile is like:

[profileX]
AWS_ACCESS_KEY=XXX
AWS_SECRET_ACCESS_KEY=YYY

This would not work until I rename it to default profile:

[default]
AWS_ACCESS_KEY=XXX
AWS_SECRET_ACCESS_KEY=YYY

According to the documentation of AWS credential helper, one can also prepare environment variable AWS_PROFILE:

Some debug step might help

Using the kaniko debug image section, it mentioned to run into the shell of the container image and do the checking

Conclusion

This issue worth a good 8 hours of my life, while all the Googling and talking to ChatGPT does not really help to reach the solution, I hope this article serve some space on the internet to help some lost soul.