ArgoCD + AWS Secrets Manager

GitOps in trends these days, it has pros and cons, but no doubt it gives you clear understanding what is deployed right now and makes CD process much smoother.

Once we started implementation, right away we have faced with trivial question “How to store secrets?”. GitOps approach drives you to have all stuff stored in git, but what to do with a sensitive data?

We use Kubernetes Secrets for storing secrets and somehow ArgoCD should create update them. (Keep aside that secrets are in the file system and not consumed by application in real time, this is another topic…)

Luckily Argo developers cared about it and provided flexible API for creating various cool plugins. One of them is Argo Vault plugin (kudos to the community). Despite of the plugin name, it could also work with AWS Secrets Manager. (we decided not couple ourselves with Hashicorp Vault, cause nowadays we a small startup and additional maintenance unneeded).

So here is the flow:

• Secrets stored in AWS Secrets manager
• ArgoCD deploys helm template for Kubernetes Secret which has annotation with name of secret and placeholders with keys of secrets parsed by Argo Vault Plugin

kind: Secret
apiVersion: v1
metadata:
  name: secret
  annotations:
    avp.kubernetes.io/path: /some/test/secret
type: Opaque
stringData:
  password: <foo>

• In case you change secret in ASM it’s instantly changing in you Kubernetes without any redeploy.

Now more details…

1. We have to create secret in AWS secrets manager, some basic key:value, for example here was created secret some/test/secret with foo:bar

2. Kubernetes nodes should have policy for getting secrets, something like this (modify based your security best practices, resource could be only node/nodes where argocd-server deployed):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1644847676679",
      "Action": [
        "secretsmanager:DescribeSecret",
        "secretsmanager:GetResourcePolicy",
        "secretsmanager:GetSecretValue",
        "secretsmanager:ListSecretVersionIds",
        "secretsmanager:ListSecrets"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}

3. Patch your argocd server deployment with following command, it will add init container which download plugin + adding variables AWS_REGION and AVP_TYPE for server container.

kubectl patch deployment argocd-repo-server -n argocd --patch-file patch.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-repo-server
spec:
  template:
    spec:
      containers:
      - name: argocd-repo-server
        env:
          - name: AWS_REGION
            value: us-east-1
          - name: AVP_TYPE
            value: awssecretsmanager
        volumeMounts:
        - name: custom-tools
          mountPath: /usr/local/bin/argocd-vault-plugin
          subPath: argocd-vault-plugin

      volumes:
      - name: custom-tools
        emptyDir: {}
      initContainers:
      - name: download-tools
        image: alpine:3.8
        command: [sh, -c]

        # Don't forget to update this to whatever the stable release version is
        # Note the lack of the `v` prefix unlike the git tag
        env:
          - name: AVP_VERSION
            value: "1.8.0"
        args:
          - >-
            wget -O argocd-vault-plugin
            https://github.com/argoproj-labs/argocd-vault-plugin/releases/download/v${AVP_VERSION}/argocd-vault-plugin_${AVP_VERSION}_linux_amd64 &&
            chmod +x argocd-vault-plugin &&
            mv argocd-vault-plugin /custom-tools/
        volumeMounts:
          - mountPath: /custom-tools
            name: custom-tools

      # Not strictly necessary, but required for passing AVP configuration from a secret and for using Kubernetes auth to Hashicorp Vault
      automountServiceAccountToken: true

4. Patch ArgoCD configmap and register new plugin (we are using option with helm)

kubectl patch configmap argocd-cm -n argocd — patch-file patch.yaml

data:
  configManagementPlugins: |
    - name: argocd-vault-plugin-helm
      init:
        command: [sh, -c]
        args: ["helm dependency build"]
      generate:
        command: ["sh", "-c"]
        args: ["helm template $ARGOCD_APP_NAME ${helm_args} . | argocd-vault-plugin generate -"]

As you see it will parse helm template and output will be also parsed with help of argocd-vault-plugin

5. Finally lets create some ArgoCD application and bind it to plugin

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: example-app
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
  project: default
  source:
    repoURL: git@github.com:somerepo.git
    targetRevision: HEAD
    path: /app
    plugin:
      name: argocd-vault-plugin-helm
      env:
        - name: helm_args
          value: -f values.yaml
  destination:
    server: https://kubernetes.default.svc
    namespace: default

Hope that aggregated information in this article will save your time for research. Cheers.