'Can't create Secret in Kubernetes: illegal base64 data at input

I want to create a secret for my kubernetes cluster. So I composed following dummy-secret.yaml file:

apiVersion: v1
kind: Secret
metadata:
  name: dummy-secret
type: Opaque
data:
  API_KEY: bWVnYV9zZWNyZXRfa2V5
  API_SECRET: cmVhbGx5X3NlY3JldF92YWx1ZTE=

When I run kubectl create -f dummy-secret.yaml I receive back following message:

Error from server (BadRequest): error when creating "dummy-secret.yaml": Secret in version "v1" cannot be handled as a Secret: v1.Secret: Data: decode base64: illegal base64 data at input byte 8, error found in #10 byte of ...|Q89_Hj1Aq","API_SECR|..., bigger context ...|sion":"v1","data":{"API_KEY":"af76fsdK_cQ89_Hj1Aq","API_SECRET":"bsdfmkwegwegwe"},"kind":"Secret","m|...

Not sure why it happens.

As I understood, I need to encode all values under the data key in the yaml file. So I did base64 encoding, but kubernetes still doesn't handle the yaml secret file as I expect.

UPDATE:

I used this command to encode data values on my mac:

echo -n 'mega_secret_key' | openssl base64

Solution 1:^[1]

I got the decoded values "mega_secret_key" and "really_secret_value1" from from your encoded data. Seems they are not encoded in right way. So, encode your data in right way:

$ echo "mega_secret_key" | base64
bWVnYV9zZWNyZXRfa2V5Cg==

$ echo "really_secret_value1" | base64
cmVhbGx5X3NlY3JldF92YWx1ZTEK

Then check whether they are encoded properly:

$ echo "bWVnYV9zZWNyZXRfa2V5Cg==" | base64 -d
mega_secret_key

$ echo "cmVhbGx5X3NlY3JldF92YWx1ZTEK" | base64 -d
really_secret_value1

So they are ok. Now use them in your dummy-secret.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: dummy-secret
type: Opaque
data:
  API_KEY: bWVnYV9zZWNyZXRfa2V5Cg==
  API_SECRET: cmVhbGx5X3NlY3JldF92YWx1ZTEK

And run $ kubectl create -f dummy-secret.yaml.

UPDATE on 11-02-2022:

The newer versions of Kubernetes support the optional stringData property where one can provide the value against any key without decoding.

All key-value pairs in the stringData field are internally merged into the data field. If a key appears in both the data and the stringData field, the value specified in the stringData field takes precedence.

apiVersion: v1
kind: Secret
metadata:
  name: dummy-secret
type: Opaque
stringData:
  API_KEY: mega_secret_key
  API_SECRET: really_secret_value1

UPDATE:

If you use -n flag while running $ echo "some_text", it will trim the trailing \n (newline) from the string you are printing.

$ echo "some_text"
some_text
$ echo -n "some_text"
some_text?

Just try it,

# first encode
$ echo -n "mega_secret_key" | base64
bWVnYV9zZWNyZXRfa2V5
$ echo -n "really_secret_value1" | base64
cmVhbGx5X3NlY3JldF92YWx1ZTE=
# then decode and check whether newline is stripped
$ echo "bWVnYV9zZWNyZXRfa2V5" | base64 -d
mega_secret_key?
$ echo "cmVhbGx5X3NlY3JldF92YWx1ZTE=" | base64 -d
really_secret_value1?

You can use these newly (without newline) decoded data in your secret instead. That also should fine.

$ cat - <<-EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: dummy-secret
type: Opaque
data:
  API_KEY: bWVnYV9zZWNyZXRfa2V5
  API_SECRET: cmVhbGx5X3NlY3JldF92YWx1ZTE=
EOF
secret/dummy-secret created

At the time of update, my kubernetes version is,

Minor:"17", GitVersion:"v1.17.3",
GitCommit:"06ad960bfd03b39c8310aaf92d1e7c1 2ce618213",
GitTreeState:"clean", BuildDate:"2020-02-11T18:14:22Z",
GoVersion:"go1.13.6", Compiler:"gc", Platform:"l inux/amd64"} Server
Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3",
GitCommit:"06ad960bfd03b39c8310aaf92d1e7c1 2ce618213",
GitTreeState:"clean", BuildDate:"2020-02-11T18:07:13Z",
GoVersion:"go1.13.6", Compiler:"gc", Platform:"l inux/amd64"} ```

Solution 2:^[2]

This was already answered but for future reference, there is no need to encode the strings by using stringData instead of data field as shown below:

#secrets.yaml
apiVersion: v1
kind: Secret
metadata:
  name: mysecret
type: Opaque
stringData:
  API_KEY: "STRING_IN_CLEAR_TEXT"
  API_SECRET: "STRING_IN_CLEAR_TEXT"

Solution 3:^[3]

Looks like your error message happens with a different dummy-secret.yaml.

apiVersion: v1
kind: Secret
metadata:
  name: dummy-secret
type: Opaque
data:
  API_KEY: af76fsdK_cQ89_Hj1Aq
  API_SECRET: bsdfmkwegwegwe

Then:

$ kubectl create -f s.yaml
Error from server (BadRequest): error when creating "dummy-secret.yaml": Secret in version "v1" cannot be handled as a Secret: v1.Secret.Data: decode base64: illegal base64 data at input byte 8, error found in #10 byte of ...|Q89_Hj1Aq","API_SECR|..., bigger context ...|sion":"v1","data":{"API_KEY":"af76fsdK_cQ89_Hj1Aq","API_SECRET":"bsdfmkwegwegwe"},"kind":"Secret","m|...

If I use your original it works fine:

apiVersion: v1
kind: Secret
metadata:
  name: dummy-secret
type: Opaque
data:
  API_KEY: bWVnYV9zZWNyZXRfa2V5
  API_SECRET: cmVhbGx5X3NlY3JldF92YWx1ZTE=

Then:

$ kubectl create -f dummy-secret.yaml
secret/dummy-secret created

I'm using the following version:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"17c77c7898218073f14c8d573582e8d2313dc740", GitTreeState:"clean", BuildDate:"2018-10-30T21:39:38Z", GoVersion:"go1.11.1", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:36:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}

Solution 4:^[4]

This may also happen when trying to remove the new line characters in a wrong way (the correct way is to remove the suffix "Cg==").

I used base64 from cli although there are workarounds to avoid the NL, like

https://superuser.com/questions/1225134/why-does-the-base64-of-a-string-contain-n/1225334

they don't work in MacOS, I found it simpler to use python like this

import base64

data = "abc123!?$*&()'-=@~"

# Standard Base64 Encoding
encodedBytes = base64.b64encode(data.encode("utf-8"))
encodedStr = str(encodedBytes, "utf-8")

Or still using a pure bash-based solution:

echo 'secret_to_encode' | tr -d \\n | base64

Solution 5:^[5]

In my case, I forgot to convert a value for a key. It can be a thing like this :). Check your values

Solution 6:^[6]

make backup

    sudo mkdir backup 
    sudo cp -R /etc/kubernetes backup/
    sudo tar -cvzf backup/pki_backup_`hostname`-`date +%Y%m%d`.tar.gz backup/kubernetes/

look to dir /etc/kubernetes/

ls -l

total 80
-rw------- 1 root root 5440 Mar  3 13:21 admin.conf
drwxr-xr-x 2 root root 4096 Aug 17  2020 audit-policy
-rw-r--r-- 1 root root  368 Mar  4  2020 calico-config.yml
-rw-r--r-- 1 root root  270 Mar  4  2020 calico-crb.yml
-rw-r--r-- 1 root root  341 Mar  4  2020 calico-cr.yml
-rw-r--r-- 1 root root  147 Mar  4  2020 calico-node-sa.yml
-rw-r--r-- 1 root root 6363 Mar  4  2020 calico-node.yml
-rw------- 1 root root 5472 Mar  3 13:21 controller-manager.conf
-rw-r--r-- 1 root root 3041 Aug 14  2020 kubeadm-config.v1alpha3.yaml
-rw------- 1 root root 5548 Mar  3 13:21 kubelet.conf
-rw-r--r-- 1 root root 1751 Mar  4  2020 kubelet.env
drwxr-xr-x 2 kube root 4096 Aug 14  2020 manifests
lrwxrwxrwx 1 root root   28 Mar  4  2020 node-kubeconfig.yaml -> /etc/kubernetes/kubelet.conf
-rw------- 1 root root 5420 Mar  3 13:21 scheduler.conf
drwxr-xr-x 3 kube root 4096 Mar  3 10:20 ssl

try to find k8s cluster config, in my case this is kubeadm-config.v1alpha3.yaml if same file absent you can generate

kubectl get cm kubeadm-config -n kube-system -o yaml > /etc/kubernetes/kubeadm-config.yaml

also in my case absent folder /etc/kubernetes/pki and exist ~/ssl I create symlink /etc/kubernetes/pki from /etc/kubernetes/ssl

ln -s /etc/kubernetes/ssl /etc/kubernetes/pki

let regenerate certs

kubeadm alpha phase certs apiserver  --config /etc/kubernetes/kubeadm-config.v1alpha3.yaml

[certificates] Using the existing apiserver certificate and key.

kubeadm alpha phase certs apiserver-kubelet-client

I0303 13:12:24.543254   40613 version.go:236] remote version is much newer: v1.20.4; falling back to: stable-1.12
[certificates] Using the existing apiserver-kubelet-client certificate and key.

kubeadm alpha phase certs front-proxy-client

I0303 13:12:35.660672   40989 version.go:236] remote version is much newer: v1.20.4; falling back to: stable-1.12
[certificates] Using the existing front-proxy-client certificate and key.

kubeadm alpha phase certs  etcd-server --config /etc/kubernetes/kubeadm-config.v1alpha3.yaml

[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [prod-uct1-mvp-k8s-0001 localhost] and IPs [127.0.0.1 ::1]

kubeadm alpha phase certs  etcd-server --config /etc/kubernetes/kubeadm-config.v1alpha3.yaml

[certificates] Using the existing etcd/server certificate and key.

kubeadm alpha phase certs  etcd-healthcheck-client --config /etc/kubernetes/kubeadm-config.v1alpha3.yaml

[certificates] Generated etcd/healthcheck-client certificate and key.

kubeadm alpha phase certs  etcd-peer --config /etc/kubernetes/kubeadm-config.v1alpha3.yaml

[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [product1-mvp-k8s-0001 localhost] and IPs [192.168.4.201 127.0.0.1 ::1]

check actuality certs

find /etc/kubernetes/pki/ -name '*.crt' -exec openssl x509 -text -noout -in {} \; | grep -A2 Validity

        Validity
            Not Before: Mar  4 10:29:44 2020 GMT
            Not After : Mar  2 10:29:44 2030 GMT
--
        Validity
            Not Before: Mar  4 10:29:44 2020 GMT
            Not After : Mar  3 10:07:29 2022 GMT
--
        Validity
            Not Before: Mar  4 10:29:44 2020 GMT
            Not After : Mar  3 10:07:52 2022 GMT
--
        Validity
            Not Before: Mar  4 10:29:44 2020 GMT
            Not After : Mar  3 10:06:48 2022 GMT
--
        Validity
            Not Before: Mar  4 10:29:44 2020 GMT
            Not After : Mar  2 10:29:44 2030 GMT
--
        Validity
            Not Before: Mar  4 10:29:44 2020 GMT
            Not After : Mar  2 19:39:56 2022 GMT
--
        Validity
            Not Before: Mar  4 10:29:43 2020 GMT
            Not After : Mar  2 10:29:43 2030 GMT
--
        Validity
            Not Before: Mar  4 10:29:43 2020 GMT
            Not After : Mar  2 19:40:13 2022 GMT
--
        Validity
            Not Before: Mar  4 10:29:44 2020 GMT
            Not After : Mar  2 19:36:38 2022 GMT

next step to generate new config files: admin.conf, controller-manager.conf, kubelet.conf, scheduler.conf, in first step move old files to /tmp

cd /etc/kubernetes/
mv {admin.conf,controller-manager.conf,kubelet.conf,scheduler.conf} /tmp/

kubeadm alpha phase kubeconfig all  --config /etc/kubernetes/kubeadm-config.v1alpha3.yaml 

[kubeconfig] Using existing up-to-date KubeConfig file: "/etc/kubernetes/admin.conf"
[kubeconfig] Using existing up-to-date KubeConfig file: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Using existing up-to-date KubeConfig file: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Using existing up-to-date KubeConfig file: "/etc/kubernetes/scheduler.conf"

after restart master kubelet and check status after restart

sudo systemctl stop kubelet; sudo docker stop $(docker ps -aq); sudo docker rm $(docker ps -aq); sudo systemctl start kubelet

systemctl status kubelet -l

? kubelet.service - Kubernetes Kubelet Server
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2021-03-03 14:00:22 MSK; 10s ago
     Docs: https://github.com/GoogleCloudPlatform/kubernetes
  Process: 52998 ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volume-plugins (code=exited, status=0/SUCCESS)
 Main PID: 53001 (kubelet)
   Memory: 51.2M
   CGroup: /system.slice/kubelet.service

check access master node

kubectl get nodes

kubectl get ns

NAME                  STATUS   AGE
default               Active   464d
product1-mvp          Active   318d
infra-logging         Active   315d
infra-nginx-ingress   Active   386d
kube-public           Active   464d
kube-system           Active   464d
pg                    Active   318d

check cert

notAfter=Mar  3 07:40:43 2022 GMT

repeat procedure for all master nodes.

Now we ready to renew certs for work nodes. In first step we must delete or move kubelet.conf

cd /etc/kubernetes/

mv kubelet.conf kubelet.conf_old

after change bootstrap-kubelet.conf

**apiVersion: v1 clusters:

cluster: certificate-authority-data: | LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJd01ETX server: https://192.168.4.201:6443 name: product1 contexts:
context: cluster: product1 user: tls-bootstrap-token-user name: tls-bootstrap-token-user@product1 current-context: tls-bootstrap-token-user@product1 kind: Config preferences: {} users:
name: tls-bootstrap-token-user user: token: fgz9qz.lujw0bwsdfhdsfjhgds**

where: certificate-authority-data – root certificate PKI CA master, may take from /etc/kubernetes/kubelet.conf on master nodes

server: https://192.168.4.201:6443 - ip api server master node virtual balance ip

token: fgz9qz.lujw0bwsdfhdsfjhgds - token, generate on master node

kubeadm token create

after restart and check kubelet work node and work node come back to cluster:

systemctl restart kubelet

systemctl status kubelet -l

? kubelet.service - Kubernetes Kubelet Server
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2021-03-03 14:06:33 MSK; 11s ago
     Docs: https://github.com/GoogleCloudPlatform/kubernetes
  Process: 54615 ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volume-plugins (code=exited, status=0/SUCCESS)
 Main PID: 54621 (kubelet)
   Memory: 52.1M
   CGroup: /system.slice/kubelet.service

check update certs may be ls in dir

ls -las /var/lib/kubelet/pki/

total 24
4 -rw-------. 1 root root 1135 Mar  3 14:06 kubelet-client-2021-03-03-14-06-34.pem
0 lrwxrwxrwx. 1 root root   59 Mar  3 14:06 kubelet-client-current.pem -> /var/lib/kubelet/pki/kubelet-client-2021-03-03-14-06-34.pem
4 -rw-r--r--. 1 root root 2267 Mar  2 10:40 kubelet.crt
4 -rw-------. 1 root root 1679 Mar  2 10:40 kubelet.key

after repeat this procedure by all work nodes.

Solution 7:^[7]

I encountered the same issue.
In my case the encoding was fine - I just copied the output without the last character of "=".

Additional scenarios that might lead to the same error:

Misplaced whitespaces or broken lines in the base64 copied output.
Not specifying the type field in the secret.

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution	Source
Solution 1
Solution 2	Tomas Fornara
Solution 3	Rico
Solution 4
Solution 5	Radu Gabriel
Solution 6	Nikolay Baranenko
Solution 7	RtmY

'Can't create Secret in Kubernetes: illegal base64 data at input

Solution 1:[1]

Solution 2:[2]

Solution 3:[3]

Solution 4:[4]

Solution 5:[5]

Solution 6:[6]

Solution 7:[7]