Debugging minikube secrets on windows

Lately I was working on implementing liveness probe in one application. During that task I was debugging minikube secrets on windows and checking why it does not work as expected. Process of setting up minikube was a little bit more complicated due to mixed environments in my system (other users were installing minikube and docker on their accounts), but finally I managed to make it work. However the most interesting part was debugging authentication process which was needed to download docker image from external registry.

Setting up minikube on windows

I was setting minikube using windows + hyper-v. One takeaway for users who want to get images from external registry, is that you need to set minikube with --insecure-registry flag. In my case command looked like that:

minikube start --vm-driver hyperv --hyperv-virtual-switch "Primary Virtual Switch"  --insecure-registry="harbor.example.local"

Deployment file looked similarly to this one. I deleted some not significant parts.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-web-deploy
  labels:
    app: app
  namespace: stage
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app
  template:
    metadata:
      labels:
        app: app
        component: app-web
    spec:
      containers:
      - name: app-web
        env:
        - name: ASPNETCORE_ENVIRONMENT
          value: Stage
        - name: APPWEB_ServiceWebApiUrl
          value: http://example.local/app
        image: harbor.example.local/default/app/app.web:some_commit_id
        imagePullPolicy: Always
        ports:
        - containerPort: 80
      imagePullSecrets:
      - name: gitlab-registry

As you can see the source of the image is external registry – harbor.

So, let’s set up a deployment (keep on reading and wait for suprise 😉 ) !

Open your console (mine favourite is Cmder).

Create namespace for staging projects and check whether namespace is created:

kubectl create namespace stage
kubectl get namespaces

Everything is fine, namespace is created.

Create secret, we will need one to authorize to harbor (k8s docs about secrets):

kubectl -n stage create secret docker-registry regcred --docker-server=harbor.example.local --docker-username=myUserName --docker-password=mypa$$^word --docker-email=my.email@email.com

Creating deployment

kubectl create -f stage.yml

Let’s see whether pod was created:

kubectl get pods -n stage

No, it wasn’t. I got ImagePullBackOff status. Let’s see what we get when we describe pod:

kubectl describe pod app-web-deploy-6d4bbf9544-dmgvt -n stage

Error response from daemon: pull access denied for myUserName/app-web, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

Hmm that was strange, I tried few other combinations of email and domain (these were changed lately) but nothing was working. There was LOT OF googling around and trying to understand the issue until I stopped thinking and looked whether secret have proper credentials. I rechecked multiple times and I was 100% sure that I typed it in correctly.

Decoding the secret

So, how we can see the contents of secret? It turns out that it’s pretty easy, just type:

kubectl get secret regcred -n stage --output="jsonpath={.data.\.dockerconfigjson}" | base64 --decode

It’s worth to know that somebody can decode your secrets after saving in K8S isn’t it ? 🙂 That’s why I am convincing everyone to use pass managers (suprising how few people are actually are using them). But back to the subject, we supposed to speak here about debugging minikube secrets on windows, right ? 🙂

What I saw after decoding the password, was literally jaw-dropping:

{"auths":{"harbor.example.local":{"username":"myUserName","password":"mypaSSword","email":"my.email@email.com","auth":"ddfkljs3365fjkskfcn32356"}}}

Can you spot the problem? Look few paragraphs up, and recheck credentials used to create the secret.

Yes, you are right! I typed the password: mypa$$^word and secret contained mypaSSword – how is that even possible?! Let’s retype (repaste) command – same result.

Actual debugging

Hmm for sure I’m not doing it right… let’s “Understand the system” and read the docs.

(Fake as you will see soon) EUREKA !

Special characters such as $, \, *, and ! will be interpreted by your shell and require escaping. In most common shells, the easiest way to escape the password is to surround it with single quotes (').

Kubernetes docs ( https://kubernetes.io/docs/concepts/configuration/secret/ )

Caret (^) was not there but I was hoping that I almost found the solution. I created secret once again with single quotes around the password this time.

Let’s decode the secret now:

{"auths":{"harbor.example.local":{"username":"myUserName","password":"'mypaSSword'","email":"my.email@email.com","auth":"ddfkljs3365fjkskfcn32356"}}}

As you can see single quotes are around password, but caret is still not there. How is that possible? I reread the docs but I found nothing more about my case. Next session of googling – nothing. I took short break and reminded that one case that I worked on some time ago. I worked with docker containers on windows and I had a problem calling some command on one kind of terminal which didn’t appear in other one.

If you’re reader with eye-to-details you remember that in the beginning of that post I’ve wrote “Open your console (mine favourite is Cmder)“.

Let’s give a try to GitBash terminal and create secret (without single quotes around password), than decode secret once again.

You should see my face when I saw proper mypaSS^word after decoding secret! 🙂

I would never assume that terminal can cut (we will see that right word here is interpret) anything from the command which I am runnig.

BTW That’s the echo of Cmder with all special signs from number’s row in keyboard (notice missing caret). Same situation happens in CMD. PowerShell and GitBash are not cutting the caret.

λ echo one!@#$%^&*()two
one!@#$%&*()two

I googled a little bit around and it seems that:

• caret is reserved character in command line (introduced with Windows XP)
• batch files and cmd interpreting caret as continuation of new line

So if you need to type command containing caret in windows, use powershell/gitbash or type double carets and you will see proper output:

λ echo one^^two<br>
one^two

Takeaways:

Key takeaways when you are debugging minikube secrets on windows :

• CMD/Cmder will interpret caret as new line continuation sign
• if you need to type command with caret in windows, use powershell/gitbash or type double carets
• always try to see the source of the problem (rule 3). Checking logs from POD was helpful, but seeing stored secret with wrong password was crucial when solving the problem
• read the docs (rule 1). In the end of the day it will save you time 🙂