Kubernetes Secrets and Service Accounts: What Are They?

Brian McClain & Tiffany Jernigan

In software, there’s often data that you want to keep separate from your build process. These could be simple configuration properties, such as URLs or IP addresses, or more sensitive data, such as usernames and passwords, OAuth tokens or TLS certificates. In Kubernetes, these are referred to as Secrets.

Secrets

It’s worth noting that while the name “secret” may imply “secure”, there are some qualifiers. By default, all secrets are stored unencrypted in etcd. As of Kubernetes 1.13 though, operators are given the option of encrypting data at rest in etcd. Additionally, you can integrate with an external Key Management Service, such as Google Cloud KMS or HashiCorp Vault. This guide doesn’t cover these topics, but the above links are a great start to learn more.

All examples used in this guide can be found on GitHub.

Before you can get started using secrets, you first need to create a secret. As you may expect, this can be done by defining an object of kind Secret:

apiVersion: v1
kind: Secret
metadata:
  name: mysecret
type: Opaque
data:
  username: bXl1c2VybmFtZQo= #Base64 encoded value of "myusername"
  password: bXlwYXNzd29yZAo= #Base64 encoded value of "mypassword"

Secrets in Kubernetes are, at their most basic form, a collection of keys and values. The above example creates a secret named mysecret with two keys: username and password. There’s one very important thing to note though, which is that the values of these key/value pairs are encoded as base64. Remember that base64 is an encoding algorithm, not an encryption algorithm. This is done to help facilitate data that may not be entirely alpha-numeric, and instead could include binary data, non-ASCII data, etc. You apply can this YAML as you would if you were creating any other Kubernetes object:

kubectl apply -f https://raw.githubusercontent.com/BrianMMcClain/k8s-secrets-and-sa/main/secret-base64.yaml

Once applied, you can see that while you can get the secret with kubectl, it avoids printing the values of each key by default:

$ kubectl describe secret mysecret

Name:         mysecret
Namespace:    default
Labels:       <none>
Annotations:  
Type:         Opaque

Data
====
password:  11 bytes
username:  11 bytes

Of course, if you want to see the base64-encoded contents of the secret, you can still fetch them with a slightly different command:

$ kubectl get secret mysecret -o yaml

apiVersion: v1
data:
  password: bXlwYXNzd29yZAo=
  username: bXl1c2VybmFtZQo=
kind: Secret
...

Great! With your secret created, it’s time to start creating pods to use it! You’re faced with another decision, however, since Kubernetes provides a couple of methods for presenting secrets to a pod. The first example you’ll look at is mounting them as files in a volume:

apiVersion: v1
kind: Pod
metadata:
  name: secret-as-file
spec:
  containers:
  - name: secret-as-file
    image: nginx
    volumeMounts:
    - name: mysecretvol
      mountPath: "/etc/mysecret"
      readOnly: true
  volumes:
  - name: mysecretvol
    secret:
      secretName: mysecret

Here, a new pod named secret-as-file is created from the NGINX Docker image.

NOTE: The nginx container image is used here simply because it’s an easily accessible long-running process, this would look the same for your own container image.

There are two sections to point out, the first being the volumes section, which defines a new volume. Kubernetes has many different types of volumes to choose from, but for this case you’re specifically interested in creating a volume of type secret. These volumes are backed by tmpfs, a RAM-based file system, rather than written to a persistent disk. Secret volumes require you to define the secret to mount (in the secretName field), and for each key in your secret, it creates a file that contains the key’s value. You can see this in action by applying this YAML and then listing the files at the mountPath:

kubectl apply -f https://raw.githubusercontent.com/BrianMMcClain/k8s-secrets-and-sa/main/pod-secret-as-file.yaml
$ kubectl exec secret-as-file -- ls /etc/mysecret

password
username

$ kubectl exec secret-as-file -- cat /etc/mysecret/username

myusername

As you can see, there are two files in the volume that was created: password and username. If you print out the contents of the username file, you can see the secret’s value of myusername.

$ kubectl exec secret-as-file -- cat /etc/mysecret/username

myusername

Alternatively, secrets can also be presented to your container as environment variables. Consider the following YAML:

apiVersion: v1
kind: Pod
metadata:
  name: secret-as-env
spec:
  containers:
  - name: secret-as-env
    image: nginx
    env:
    - name: SECRET_USERNAME
      valueFrom:
        secretKeyRef:
          name: mysecret
          key: username
    - name: SECRET_PASSWORD
      valueFrom:
        secretKeyRef:
          name: mysecret
          key: password

Here, instead of defining volumes that reference your secret, two environment variables are defined and reference the secret name and key name. Applying this YAML allows you to retrieve these environment variables from a shell in the pod:

kubectl apply -f https://raw.githubusercontent.com/BrianMMcClain/k8s-secrets-and-sa/main/pod-secret-as-env.yaml
$ kubectl exec secret-as-env -- sh -c "echo \$SECRET_USERNAME"

myusername

As you can see, the value of your secret is stored in the $SECRET_USERNAME environment variable as defined!

Service Accounts

When you interact directly with Kubernetes, using kubectl for example, you’re using a user account. When processes in pods need to interact with Kubernetes though, they use a service account, which describes the set of permissions they have within Kubernetes. The good news is that out of the box, all pods are given the default service account. Unless your Kubernetes administrator has changed the default service account though, the permissions are limited. If you run kubectl in a container on Kubernetes, it will automatically know where to find the cluster that it’s running on. You can verify this by standing up a pod and running kubectl version, which will show information about the server it’s connected to:

kubectl run -it kubectl --restart=Never --rm --image=brianmmcclain/kubectl-alpine -- /bin/bash
$ kubectl version

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.4", GitCommit:"c96aede7b5205121079932896c4ad89bb93260af", GitTreeState:"clean", BuildDate:"2020-06-17T11:41:22Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-30T20:19:45Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

Notice that you haven’t provided any credentials or configuration file. This information is provided by Kubernetes and the default service account. However, almost any attempt at interacting with the Kubernetes API will be greeted with denial:

$ kubectl get pods

Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:default:default" cannot list resource "pods" in API group "" in the namespace "default"

$ exit

Note: If you need to check if you have permission to run a command before actually running it, you can use the kubectl auth can-i command:

$ kubectl auth can-i get pods
no

To address this, you can create a new service account with a wider set of permissions. This is demonstrated in the following YAML:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: pod-read-role
rules:
- apiGroups: [""] # "" indicates the core API group
  resources: ["pods"]
  verbs: ["get", "watch", "list"]

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: pod-read-sa

---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: pod-read-rolebinding
  namespace: default
subjects:
- kind: ServiceAccount
  name: pod-read-sa
  apiGroup: ""
roleRef:
  kind: Role
  name: pod-read-role
  apiGroup: ""

Here, three things are created: a Role named “pod-read-role”, a ServiceAccount, and a RoleBinding to tie them together. Specifically, the Role gives access to the “get”, “watch” and “list” actions on the resource “pods”. Described more simply, this role allows you to read information about pods, but not write or delete information about pods. This will allow you to do things like kubectl get pods, but not kubectl delete pod. You can see this in action by applying this YAML, creating a pod with this service account and running the commands yourself:

kubectl apply -f https://raw.githubusercontent.com/BrianMMcClain/k8s-secrets-and-sa/main/role-sa-pod-read.yaml
kubectl run -it --restart=Never --rm kubectl-with-sa --image=brianmmcclain/kubectl-alpine --serviceaccount=pod-read-sa -- /bin/bash
$ kubectl get pods

NAME              READY   STATUS    RESTARTS   AGE
kubectl           1/1     Running   1          22s
kubectl-with-sa   1/1     Running   0          6s
secret-as-env     1/1     Running   0          3h40m
secret-as-file    1/1     Running   0          4h10m

$ kubectl delete pod secret-as-file

Error from server (Forbidden): pods "secrets-as-file" is forbidden: User "system:serviceaccount:default:pod-read-sa" cannot delete resource "pods" in API group "" in the namespace "default"

$ exit

Finally, the combination of secrets and service accounts can be leveraged to pull container images from private registries by using the imagePullSecrets configuration property. You can create a secret from the command line with the following command:

kubectl create secret docker-registry myregistrykey --docker-server=DUMMY_SERVER \
        --docker-username=DUMMY_USERNAME --docker-password=DUMMY_DOCKER_PASSWORD \
        --docker-email=DUMMY_DOCKER_EMAIL

Note: The secret used here isn’t exposed to the pod in the same way that you’ve seen earlier. The processes inside the containers of the pod don’t have access to this information. Instead, Kubernetes knows that it needs to use these credentials to pull the container images.

Once you create the secret by filling in your registry’s server, username, password, and email, you can create a service account, or edit an existing one, to use this secret when pulling container images. For example, you can add this to the default service account. Make note, however, that this will overwrite any imagePullSecret previously set:

kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "myregistrykey"}]}'

Since this adds the imagePullSecrets property to the default service account, any pod that you create without specifying a different service account will have these permissions. However, it’s worth noting that you can also specify imagePullSecrets on an individual pod if it fits your deployment model better.

Cleanup

To remove the resources that you’ve created, you can use kubectl delete -f command and provide the file names used when applying them:

kubectl delete -f <insert file>

Learn More

As with all things Kubernetes, the best place to go to keep learning is the official documentation, which covers secrets and service accounts in even greater detail. You can also see where these are used in other guides, such as Getting Started with kpack and Getting Started with Tekton.