Secrets Store CSI Driver vs. External Secrets Operator

Managing secrets is critical for running secure and professional Kubernetes clusters. Sensitive data like API keys, database passwords, and TLS certificates need to be handled with care. While Kubernetes offers its own Secret object, it is often better to delegate the actual storage and management of secrets to external, specialized systems like AWS Secrets Manager, HashiCorp Vault, or Azure Key Vault.

This post will explore two popular solutions for integrating external secret management systems with Kubernetes: the Secrets Store CSI Driver and the External Secrets Operator (ESO). We’ll explore their architecture, usage, key differences, and how they handle secret changes.

Why Externalize Secrets?

Before diving deeper, it’s important to understand why you’d want to use an external secrets manager. Kubernetes (native) Secret objects are merely base64 encoded, not encrypted. You would never want to save a Secret yaml manifest in git, for example, because base64 can easily be printed as plain text using the base64 command line utility or other program. Also, depending on your Kubernetes control plane configuration, the values saved in etcd (Kubernetes’ database) may not be encrypted-at-rest.

Besides secret encyption, an external secret managers offers other benefits such as:

• Centralized Management: Manage secrets for multiple clusters and applications from a single, secure location.
• Enhanced Security Features: Features like fine-grained access control, audit logging, and automated secret rotation.  
• Compliance: Meet specific industry or organizational compliance standards.

With that out of the way, let’s discuss Secrets Store CSI Driver and External Secrets Operator. These are two different, separate solutions for bridging the gap between secret management tools and your Kubernetes workloads.

Secrets Store CSI Driver: Mounting Secrets as Volumes

The Secrets Store CSI Driver is a Kubernetes volume plugin, the same mechanism used for storage volumes. Its primary function is to retrieve secrets from external secret management systems and mount them as files directly into your Kubernetes pods. Kubernetes native “Secrets” objects are not used with this strategy. With the Secrets Store CSI Driver, applications access secrets as if they were regular files in the pod’s filesystem.

Key characteristics:

• CSI Standard: Uses the Container Storage Interface, a standard for exposing storage systems to Kubernetes.
• Provider-Specific Plugins: You’ll need a specific provider plugin for the external secret manager you intend to use (e.g., AWS Secrets Manager provider, Azure Key Vault provider, HashiCorp Vault provider).
• SecretProviderClass CRD: You define how secrets are fetched using a Custom Resource Definition (CRD) called SecretProviderClass. This CRD specifies the provider, the secrets to retrieve, and any necessary parameters.
• Pod Volume Mounts: Secrets are made available to pods by mounting a volume that uses the secrets-store.csi.k8s.io driver and references a SecretProviderClass
• Optional Sync to Kubernetes Secrets: The driver can also (optionally) sync the fetched secrets into native Kubernetes Secret objects. However, there are caveats: these synced secrets are deleted if all pods consuming them are deleted, and they only sync once a pod mounting the secret starts

How it works:

1. You install the Secrets Store CSI Driver and the relevant provider plugin onto your cluster (the driver runs as a Daemonset).
2. You create a SecretProviderClass resource defining which external secrets to fetch.
3. In your Pod specification, you define a volume using the CSI driver and link it to the SecretProviderClass.
4. When the pod starts, the Kubelet (via the CSI driver) communicates with the external secret store.
5. The secrets are fetched and mounted as files into the specified mountPath within the pod.

Secrets Store CSI Example (mounting as volume)

# Pod definition
apiVersion: v1
kind: Pod
metadata:
  name: my-app-pod
spec:
  containers:
    - name: my-app-container
      image: my-app
      volumeMounts:
      - name: secrets-volume
        mountPath: "/mnt/secrets-store" # Secrets will be available as files here
        readOnly: true
  volumes:
    - name: secrets-volume
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "my-azure-keyvault-secrets" # References your SecretProviderClass

In this example, secrets defined in the my-azure-keyvault-secrets SecretProviderClass would be mounted into the /mnt/secrets-store directory of the my-app-container.

External Secrets Operator (ESO): Syncing Secrets to Kubernetes Objects

The External Secrets Operator (ESO) takes a different approach. Instead of volume mounts, ESO synchronizes secrets from external APIs into native Kubernetes Secret objects. Your applications then consume these Kubernetes Secret objects as they normally would (e.g., via environment variables or volume mounts of the Kubernetes Secret).

Key characteristics:

• Operator Pattern: ESO runs as an operator in your cluster, watching for its own Custom Resource Definitions (CRDs).
• CRDs for Configuration:
  • SecretStore or ClusterSecretStore: These CRDs define how to connect to an external secret provider (e.g., AWS Secrets Manager, HashiCorp Vault). SecretStore is namespaced, while ClusterSecretStore is cluster-wide.
  • ExternalSecret: This CRD specifies which secrets to fetch from a SecretStore or ClusterSecretStore and how to transform/name them in the resulting Kubernetes Secret object.
• Automatic Secret Generation: Once an ExternalSecret resource is created, ESO automatically fetches the specified data from the external provider and creates or updates a corresponding native Kubernetes Secret object.

How it Works:

1. You install the External Secrets Operator in your cluster.
2. You create a SecretStore (or ClusterSecretStore) CRD to configure access to your external secret manager.
3. You create an ExternalSecret CRD, referencing the SecretStore and specifying the external secret(s) to retrieve and the desired name for the Kubernetes Secret that will be created (can also be a ClusterExternalSecret).
4. ESO detects the ExternalSecret resource.
5. ESO fetches the secret data from the external provider.
6. ESO creates or updates a native Kubernetes Secret object with the fetched data.
7. Your applications can then use this Kubernetes Secret in the standard ways (env vars, volume mounts).

Note that both the Secret Store and the External Secret work together to create the final Kubernetes Secret. The Secret Store focuses on how the secret is retrieved from the secret management system, while the External Secret configures how the Secret is created. The official API documentation has detailed information about the configuration options for each type.

ESO External Secrets Example Definition

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: my-app-db-credentials
spec:
  refreshInterval: "1h" # How often to check for updates
  secretStoreRef:
    name: my-aws-secretstore # References your SecretStore
    kind: SecretStore
  target:
    name: db-credentials # Name of the Kubernetes Secret to be created
    creationPolicy: Owner
  data:
    - secretKey: username # Key in the Kubernetes Secret
      remoteRef:
        key: my-app/db/username # Key in the external secret store
    - secretKey: password
      remoteRef:
        key: my-app/db/password

In this example, ESO would create a Kubernetes Secret named db-credentials containing username and password keys, with values fetched from the specified paths in the external secret store defined in my-aws-secretstore.

Comparison of Secret Store CSI Driver vs ESO

Feature	Secrets Store CSI Driver	External Secrets Operator (ESO)
Primary Mechanism	Mounts secrets as files into pod volumes. Can optionally sync to K8s Secrets.	Syncs secrets into native Kubernetes Secret objects.
Backward Compatibility	May require application YAML changes if migrating from existing K8s secrets to use the CSI volume mount directly.	Generally backward compatible, as applications continue to use standard Kubernetes Secret objects.
imagePullSecrets	Can be problematic as secrets might only sync when a pod mounting them starts, which is too late for imagePullSecrets.	Works well with imagePullSecrets as it creates standard Kubernetes Secret objects.
Secret Accessibility	Secrets mounted as volumes are directly accessible by the pod. Synced K8s Secrets are deleted if no pod consumes them. Secrets might not be readable if not mounted to a pod.	Creates regular Kubernetes Secret objects that can be inspected and used independently of specific pods.
Security Context	The driver daemonset runs as root in a privileged pod. Provider plugins also typically require root.	Operates by managing CRDs and Kubernetes API objects.
Use Cases Outside K8s	Primarily focused on Kubernetes pod integration.	Can potentially be used to manage credentials for systems outside Kubernetes that can leverage K8s secrets.
Secret Update Propagation	Periodically updates pod’s mount content and synced K8s Secrets (if configured). Frequency not typically configurable.	Configurable refreshInterval for checking external provider for updates.

Access Policies to Secret Backends

Both External Secrets Operator and Secrets Store CSI Driver need access to the secrets management system (HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, or GCP Secret Manager, etc) in order to fetch the secrets and expose them to pods. For AWS, this means utilizing IAM Roles for Service Accounts (IRSA), which maps an AWS IAM role to a Kubernetes Service Account, allowing pods to assume the role and inherit its permissions without the need for explicit AWS access keys. When configuring an IAM role or token, be sure to follow the principle of least privilege to only allow access to secrets the Kubernetes workload requires. For instance, you may want to create a policy that allows read-only access to specific secrets or groups of secrets.

Secret Change Management

Knowing how each solution updates secrets is important so you can ensure applications have the most recent versions, for example after secrets are rotated.

• Secrets Store CSI Driver: It periodically updates the pod’s mounted content and any synced Kubernetes Secrets. However, the update frequency isn’t typically configurable. If secrets are used as environment variables (from a synced K8s secret), pods generally need a restart to pick up changes. If mounted as files, applications need to be able to detect and reload these file changes.
• External Secrets Operator: ESO uses a configurable refreshInterval to check the external provider for secret updates. When an update is detected, ESO updates the corresponding Kubernetes Secret object. Similar to the CSI driver, if these Kubernetes secrets are consumed as environment variables, pods will need a restart (potentially managed by a tool like Reloader). If mounted as volumes (from the K8s secret), the volume will be updated by Kubernetes, and applications need to handle the content change.

Which Solution Should You Use?

From personal experience, I much prefer the External Secrets Operator (ESO). It creates standard Kubernetes Secrets objects, which is easier to debug and more importantly, the secrets are not dependent on the lifecycle of the pod. As mentioned earlier, mounted (CSI) secrets cannot be used for imagePullSecrets because the image pull secrets are needed to pull the container images before secrets are mounted in the pod.

Furthermore, I find the ESO documentation very extensive, with many examples. I also like the “separation of concerns” with the ESO architecture: Secret Store specifies how secrets are retrieved from the external system, while the External Secret deals with the creation of the Secret and the specific data the secret should contain.

Lastly, the Kubernetes Secrets Store CSI Driver is a DaemonSet, meaning it needs to run on every Kubernetes node! It also requires “privileged mode”. If you are running a production Kubernetes cluster, you no doubt are familiar with the pains of running numerous utility Daemonsets that compete for resources with application pods running with business logic. On the other hand, ESO consists of one or more controller pods that run as part of a Deployment. You can run ESO controllers on dedicated nodes separate from business services if desired.

Where I work we use AWS Secrets Manager in a central account, and use cross-account IAM roles to create secrets in different clusters in different AWS accounts. Check out my article about centralized secrets with ESO and AWS Secrets Manager.

There may be certain use-cases for Secret Store CSI, however I use ESO extensively and am pretty happy with it.