Getting Started with External Secrets Operator

Why External Secrets Operator

Whenever we talk about secrets in Kubernetes, we typically mean sensitive data that should be shared with applications. These are things like API keys, database credentials, certificates, tokens, etc.

Kubernetes has a native Secret Object that can expose secrets to pods, however, there are some drawbacks with the built-in approach:

• Kubernetes Secrets are, by default, stored unencrypted in the API server’s underlying data store (etcd). Anyone with API access can retrieve or modify a Secret, and so can anyone with access to etcd (although RBAC and encryption-at-rest can help reduce exposure).
• The secrets are confined to the Kubernetes cluster where they are created. Secrets are difficult to access outside the cluster, and are lost if the cluster is destroyed.
• There are currently no features to sync or update secrets from an external data source. You will need to update the secrets manually if they change.

To solve these challenges, some brilliant minds that were already working on similar tools merged their efforts to create the open-source project External Secrets Operator (ESO).

This article goes through an example of using ESO with AWS Secrets Manager – but don’t worry if you’re not running on AWS, ESO supports a wide range of secrets sources, referred to as “providers”.

Using the External Secrets Operator with AWS Secrets Manager

I’m going to skip over the installation steps, since the official docs do a good job of that. Instead, I will walk through the process of creating a necessary resources to sync a secret from AWS Secrets Manager and expose the secret to a pod as an environment variable.

But before we get started, it’s important to explain the two main types of objects in ESO:

• SecretStore – a resource that specifies how to access the data stored in an external source. The syntax for configuring authorization to the secrets backend depends on the specific provider used.
• ExternalSecret – describes what data should be fetched and how the data should be transformed. This resource will create a Kubernetes secret.

But wait, you say: If ESO creates a regular Kubernetes Secrets, in the end, aren’t those secrets also stored in etcd?

The secrets are stored in etcd, but they only contain references such as paths and names of secrets stored in an external secret management system. The secret data are dynamically fetched and returned as needed.

With that out of the way, let’s get started.

The Secret Data

First, we’ll create a secret called MyTestSecret in AWS Secrets Manager using the AWS cli:

aws secretsmanager create-secret \
    --name MyTestSecret \
    --description "My test secret created with the CLI." \
    --secret-string "my-secret-data"

Now that we’ve created our secret, we can configure the required Objects for the External Secrets Operator (ESO).

As mentioned, we will first need a SecretStore Object to give ESO the ability to fetch the secret MyTestSecret from AWS Secrets Manager.

To do this, we will use the EKS Service Account credentials authentication method, which requires these steps:

• Create an IAM role
• Add the necessary policies to the role
• Create a Kubernetes Serviceaccount
• Associate the IAM Role to the Serviceaccount using IAM roles for service accounts

This feature will allow us to use short-lived service account tokens to authenticate with AWS. You must have Service Account Volume Projection enabled – it is by default on EKS.

This authentication method is arguably better than using Access Key and Secret Access Keys because you don’t need to generate and manage the keys, which themselves are confidential data. Also, AWS access keys are long-lived and could result in a serious security breach if they are leaked.

We will use terraform to create the IAM role and required policies.

IAM Role and Policies

If you are copying this code, be sure to replace the values as required.

// create the IAM role
resource "aws_iam_role" "secret_store" {
  name        = "eso-secretstore-role"
  description = "Role for external secrets operator"

  assume_role_policy = data.aws_iam_policy_document.eso_serviceaccount.json
}

// assume role policy for the IAM role
data "aws_iam_policy_document" "eso_serviceaccount" {

  statement {

    principals {

      type        = "Federated"
      identifiers = [
        "arn:aws:iam::<aws-account-id>:oidc-provider/<oidc-issuer-url>"
      ]

    }

    actions = [
      "sts:AssumeRoleWithWebIdentity"
    ]

    condition {
      test     = "StringLike"
      variable = "<oidc-issuer-url>:sub"
      values   = [
        "system:serviceaccount:<namespace>:<serviceaccount-name>",
      ]

    }
  }
}

// IAM policy resource
resource "aws_iam_policy" "eso_secretstore" {
  name        = "eso-secretstore-policy"
  description = "Policy for external secrets operator"
  policy      = data.aws_iam_policy_document.eso_secretstore.json
}

// IAM policy document
data "aws_iam_policy_document" "eso_secretstore" {

  statement {

    actions = [
      "secretsmanager:GetRandomPassword",
      "secretsmanager:ListSecrets"
    ]

    resources = [   "arn:aws:ssm:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:*"
    ]

  }

  statement {

    actions = [
      "secretsmanager:GetResourcePolicy",
      "secretsmanager:GetSecretValue",
      "secretsmanager:DescribeSecret",
      "secretsmanager:ListSecretVersionIds"
    ]

    // note - the wildcard below will allow access to all secrets.  Restrict this as needed.

    resources = [
"arn:aws:secretsmanager:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:secret:*"
    ]

  }
}

// attach policy document to policy
resource "aws_iam_policy_attachment" "eso" {
  name       = "eso-secretstore-policy-attachment"
  roles      = [aws_iam_role.secret_store.name]
  policy_arn = aws_iam_policy.eso_secretstore.arn
}

// output the IAM role arn, which we will need later
output "role_arn" {
  value = aws_iam_role.secret_store.arn
}

The next step is to create a serviceaccount, associating the IAM role we just created.

Kubernetes Serviceaccount

kubectl create -f - <<EOF

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<aws-account-id>:role/eso-secretstore-role
  name: external-secrets
EOF

Note that the exact annotation key eks.amazonaws.com/role-arn must used.

SecretStore Object

Now we can (finally) create the SecretStore, using the SecretsManager provider syntax:

kubectl create -f - <<EOF

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: my-secretstore
spec:
  provider:
    aws:
      service: SecretsManager
      region: <your-aws-region>
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets
EOF

Let’s check it …

❯ kubectl get secretstore
NAME             AGE   STATUS   CAPABILITIES   READY
my-secretstore   3s    Valid    ReadWrite      True

Now we are ready to create the ExternalSecret.

ExternalSecret

The ExternalSecret needs to reference the SecretStore created earlier, as well as information about the secret we want to retrieve.

kubectl create -f - <<EOF

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: my-first-external-secret
spec:
  secretStoreRef:
    name: my-secretstore
    kind: SecretStore
  target:
    name: my-app-secret
    creationPolicy: Owner
  data:
  - secretKey: my-secret
    remoteRef:
      key: MyTestSecret

EOF

❯ kubectl get externalsecret
NAME                       STORE            REFRESH INTERVAL   STATUS         READY
my-first-external-secret   my-secretstore   1h                 SecretSynced   True

Note that the “refresh interval” has a default of 1 hour, which is configurable. You’ll need to think about how often your secret changes. Setting a lower value will result in increased api calls to AWS, while a value that is too high can cause your secret not to be refreshed in a timely manner.

At this point, ESO should have created the Kubernetes Secret too. Let’s check.

❯ kubectl get secret
NAME            TYPE     DATA   AGE
my-app-secret   Opaque   1      3m31s

❯ kubectl get secret my-app-secret -ojsonpath='{ @.data.my-secret }'|base64 -D

my-secret-data

Finally, we will create a pod that uses the secret:

kubectl create -f - <<EOF

apiVersion: v1
kind: Pod
metadata:
  name: busybox
spec:
  containers:
  - image: busybox
    command:
      - sleep
      - "3600"
    name: busybox
    env:
      - name: MY_SECRET
        valueFrom:
          secretKeyRef:
            name: my-app-secret
            key: my-secret
  restartPolicy: Always

EOF

The secretKeyRef.key is “my-secret” because we configured the ExternalSecret to set the secret value under this key:

❯ kubectl get secret -o yaml
apiVersion: v1
items:
- apiVersion: v1
  data:
    my-secret: bXktc2VjcmV0LWRhdGE=
  immutable: false
  kind: Secret
  metadata:
  ...

Also, there is nothing “special” about the Kubernetes Secret created with ESO – it is used just like any other Kubernetes Secret

❯ kubectl get pod
NAME      READY   STATUS    RESTARTS   AGE
busybox   1/1     Running   0          18s

Let’s have a look inside our pod to verify the secret is set as the MY_SECRET env variable.

❯ kubectl exec -it pod/busybox -- /bin/sh
/ #
/ # echo $MY_SECRET
my-secret-data

Summary

This was a quick walk-though of the External Secrets Operator (ESO). I encourage you to read the excellent official documentation. There are many other features such as templating, cluster-wide resources, and lifecycle options – just to name a few.

If you work in a large enterprise where you need to support multi-tenacy Kubernetes clusters with different levels of access, you may need to consider restricting secrets on a namespace level, and restricting policies that allow SecretStore to access the secret data, to ensure the principle of least privilege.

Thanks for reading. Please give a “like” or a comment if you found this useful.