The Problem

Back in the good old days you used to be able to run your project locally, connect straight to the production DB and blow it up in the matter of seconds. Nowadays there's a bunch of pipeline steps and similar hurdles between you and a valuable lesson.

One of these hurdles is that all of the sensitive information needed for the application to run are often stored in in HashiCorp Vault and made available to your app through runtime-speicific methods like sidecar containers in Kubernetes. If you have access to Vault, you can of course copy, paste and format all of the secrets needed locally, but if you have multiple environments for your application, this gets tedious really quickly.

Prerequisites

• Access to the Vault instance.
• Vault CLI on your local machine.

The Solution - Vault Agent

Surprisingly there aren't a lot of solutions to this problem online, or perhaps they are hard to google for. The only blogpost I found does essentially the same thing as we're going to do, but it relies on Python to format the secrets. Our approach, in contrast, relies only on the Vault Agent, which is included with every vault installation.

We're going to need three files in the root of your project's directory. I suggest committing the first two to your VCS, so your colleagues can also use this secret injection setup: - vault_config - The configuration for for the vault agent. - secrets_template.ctmpl - The consul template used to render the secrets in the desired format. - vault_token - Your personal auth token.

Replace the UPPERCASE_PLACEHOLDERS with your desired values.

The vault_config looks like this:

vault {
 retry {
   num_retries = -1
 }
}

exit_after_auth = true

auto_auth {
  method {
    type = "token_file"

    config = {
      token_file_path = "./vault_token"
    }
  }
}

template_config {
  exit_on_retry_failure = true
}

template {
  source      = "./secrets_template.ctmpl"
  destination = "DESTINATION_PATH"
  error_on_missing_key = true
}

The secrets_template.ctmpl is essentially a go template with some extensions. I've set it up so it outputs all secrets as KEY=VALUE lines in the generated file. This format is known as an .env or .properties file:

{{- $stage := mustEnv "STAGE" -}}
{{- $url := printf "SECRET_API_PATH/%s" $stage -}}
{{- with secret $url -}}
{{- range $key, $value := .Data.data -}}
{{ $key }}={{$value}}
{{ end }}
{{- end -}}

To create the vault_token file, click on the profile icon in your vault instance and click on Copy token. Paste it in this file and make sure it doesn't end up in the repository with all the other files.

Finally to render the secret, execute the following in the command line:

STAGE=development VAULT_ADDR=YOUR_VAULT_HOSTNAME vault agent -config vault_config 

After all that, you should have a file DESTINATION_PATH with the rendered secrets. Just note that if the secret or some of the path parameters change and you want to re-build the output, first delete the old file. There might be a way around this but I was too lazy to find it. All you need now is to read the output file in your application.

Bonus: How to Read the Secret Values in Spring Boot

If your app is already running in Kubernetes, chances are that your it already has a way to read secrets, but if they are not rendered into a file, adding this line to your Spring properties files will make the application read them on startup:

spring.config.import=optional:file:DESTINATION_PATH

If you're having trouble with this, add

logging.level.org.springframework.boot.context.config=trace