Skip to content

External Secrets Operator

This document explains the configuration and usage of the External Secrets Operator for synchronizing secrets from Google Cloud Secret Manager to native Kubernetes secrets, which can then be utilized by applications. The document is divided into two parts: a guide for Kubernetes cluster administrators and one for application developers.

Useful References

Guide for Cluster Administrators

The next steps don't require any actions, but they explain the functionality of the current Terraform code:

  1. Workload Identity Configuration: Workload identity is utilized to grant Kubernetes Service Accounts the necessary permissions for accessing resources within Google Cloud Secret Manager. That's configured in stages/00_iam/iam.tf. Note: it's possible to create multiple service accounts with distinct access to GCP Secrets.

  2. External Secrets Operator Helm Chart Installation: Terraform code. Note: The Helm chart also creates a Kubernetes namespace for the Operator.

  3. Service Account Creation for Applications: Within each Kubernetes namespace associated with an application, a Kubernetes Service Account must be created. These service accounts are designated for pulling secrets from Google Cloud Platform (GCP). Annotation within the service account's specifications determines the corresponding Google Service Account (created in step 1) to be employed for secret retrieval. Terraform code

  4. SecretStore Configuration for Applications: In each application's Kubernetes namespace, a SecretStore must be created to define the origin of the secrets and specify the appropriate Kubernetes Service Account for accessing them. By default, the configuration uses the GKE location and project ID to facilitate the storage and access the original GCP Secrets. Terraform code

Note: sensitive information (passwords and other secrets), created within terraform runtime can be saved right in Kubernetes secrets without the need to use External Secrets Operator.

Guide for Application Developers

GCP Secret Creation

Initially it's required to create the origin secret in Google Cloud Secret Manager (How to). It's recommended to use the next naming convention for the secrets names:

$CLUSTER_$NS_$SECRET

Were $SECRET may also contain a short application name if the namespace contains several applications.

Note: maximum length of a secret ID is 255 characters

Synchronization into Kubernetes

Synchronization of particular secrets from GCP into a native Kubernetes Secrets can be established by a corresponding ExternalSecret kubernetes resource. Example: 

# external_secret_demo.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: external-secret-demo
  # In this namespace the k8s Secret will be created and it must contain pre-configured SecretStore.
  # In our example it's dev namespace, but most probably it will be a namespace with the application
  namespace: dev
spec:
  refreshInterval: 10m            # rate SecretManager pulls GCPSM
  secretStoreRef:
    # SecretStore is usully provisioned with the namespace by the cluster admins
    # It contains info about origin secrets storage and how to access it
    kind: SecretStore
    name: gcp-secrets-manager
  target:
    name: db-password          # Name of the k8s Secret to be created
    creationPolicy: Owner
  data:
    - secretKey: password      # Key name in the created k8s Secret
      remoteRef:
        key: "k8s-dev_dev_db-password"     # Secret name in GCP
        #version: latest
        #property: TEST_CREDENTIAL1 # JSON property

Once such manifest is deployed, the External Secrets Operator will automatically create a native Kubernetes Secret within the same namespace and with the defined in the manifest name and data.

Notes:

  1. It's expected to deploy ExternalSecret resources along with other application yaml manifests.
  2. It's expected SecretStore and Service Accounts to be pre-created by terraform code along with the namespace creation during the Kubernetes cluster configuration.

How to Mount a Secret into a Kubernetes Pod

Here is an example of application deployment manifest that has a "db-password" secret as an environment variable:

# application-deloyment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment-example  # Name of the deployment
spec:
  replicas: 1  # Number of desired replicas
  selector:
    matchLabels:
      app: deployment-example  # Label selector for pods managed by this deployment
  template:
    metadata:
      labels:
        app: deployment-example # Labels for pods created from this template
    spec:
      containers:
        - name: app-container   # Name of the container
          image: alpine:latest  # Application Docker image
          command: ["/bin/sh"]  # Override the default command
          args: ["-c", "echo Password: $PASSWORD; sleep 3600"]  # Secret is mounter as a env variable in this case
          env:
            - name: PASSWORD    # Environment variable name for the secret
              valueFrom:
                secretKeyRef:
                  # See spec.data of the ExternalSecret
                  name: db-password  # Name of the Kubernetes Secret resource in the same namespace
                  key: password      # Key within the secret
      volumes:
        - name: secret-volume        # Name of the volume
          secret:
            secretName: db-password  # Name of the secret used as a volume

Accessing Multiple Secrets

When you have multiples secrets stored as key-value pairs within a JSON structure in a GCP Secret Manager. It is possible to retrieve it by making use of property field.

#example-secret.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: external-secret-demo
  namespace: dev
spec:
  refreshInterval: 10m            # rate SecretManager pulls GCPSM
  secretStoreRef:
    kind: SecretStore
    name: gcp-secrets-manager
  target:
    name: db-password          # Name of the k8s Secret to be created
    creationPolicy: Owner
  data:
    - secretKey: password      # Key name in the created k8s Secret
      remoteRef:
        key: "k8s-dev_credentials"     # Secret name in GCP
        property: KEY1
    - secretKey: username      # Key name in the created k8s Secret
      remoteRef:
        key: "k8s-dev_credentials"     # Secret name in GCP
        property: KEY2
Here, KEY1 and KEY2 refer to the key in the GCP Secret Manager.

Troubleshooting

Get status of a deployed ExternalSecret:

$ kubectl get ExternalSecret -n dev
NAME             STORE                 REFRESH INTERVAL   STATUS         READY
dev-poc-secret   gcp-secrets-manager   3m                 SecretSynced   True

If the sync is failed, more detailed log can be found in external-secrets pod logs in external-secrets namespace. Example:

kubectl get pods -n external-secrets
NAME                                                READY   STATUS    RESTARTS   AGE
external-secrets-57ddc5b469-92r2r                   1/1     Running   0          25h
external-secrets-cert-controller-6dc74f5db7-9gfh4   1/1     Running   0          25h
external-secrets-webhook-57ffc57cb8-nfz64           1/1     Running   0          25h

$ kubectl logs external-secrets-57ddc5b469-92r2r -n external-secrets