Caution
Buildbot no longer supports Python 2.7 on the Buildbot master.
2.4. Secret Management¶
2.4.1. Requirements¶
Buildbot steps might need secrets to execute their actions. Secrets are used to execute commands or to create authenticated network connections. Secrets may be a SSH key, a password, or a file content like a wgetrc file or a public SSH key. To preserve confidentiality, the secrets values must not be printed or logged in the twisted or steps logs. Secrets must not be stored in the Buildbot configuration (master.cfg), as the source code is usually shared in SCM like git.
2.4.2. How to use Buildbot Secret Management¶
2.4.2.1. Secrets and providers¶
Buildbot implements several providers for secrets retrieval:
- File system based: secrets are written in a file. This is a simple solution for example when secrets are managed by config management system like Ansible Vault.
- Third party backend based: secrets are stored by a specialized software. These solution are usually more secured.
Secrets providers are configured if needed in the master configuration. Multiple providers can be configured at once. The secret manager is a Buildbot service. The secret manager returns the specific provider results related to the providers registered in the configuration.
2.4.2.2. How to use secrets in Buildbot¶
Secret can be used in Buildbot via the IRenderable mechanism.
Two IRenderable actually implement secrets.
Interpolate can be used if you need to mix secrets and other interpolation in the same argument.
Interpolate can be used if your secret is directly used as a component argument.
As argument to steps¶
The following example shows a basic usage of secrets in Buildbot.
from buildbot.plugins import secrets, util
# First we declare that the secrets are stored in a directory of the filesystem
# each file contain one secret identified by the filename
c['secretsProviders'] = [secrets.SecretInAFile(dirname="/path/toSecretsFiles")]
# then in a buildfactory:
# use a secret on a shell command via Interpolate
f1.addStep(ShellCommand(util.Interpolate("wget -u user -p '%(secret:userpassword)s' '%(prop:urltofetch)s'")))
# .. or non shell form:
f1.addStep(ShellCommand(["wget", "-u", "user", "-p", util.Secret("userpassword"), util.Interpolate("%(prop:urltofetch)s")]))
Secrets are also interpolated in the build like properties are, and will be used in a command line for example.
As argument to services¶
You can use secrets to configure services. All services arguments are not compatible with secrets. See their individual documentation for details.
# First we declare that the secrets are stored in a directory of the filesystem
# each file contain one secret identified by the filename
c['secretsProviders'] = [secrets.SecretInAFile(dirname="/path/toSecretsFiles")]
# then for a reporter:
c['services'] = [GitHubStatusPush(token=util.Secret("githubToken"))]
2.4.2.3. Secrets storages¶
SecretInAFile¶
c['secretsProviders'] = [secrets.SecretInAFile(dirname="/path/toSecretsFiles")]
In the passed directory, every file contains a secret identified by the filename.
e.g: a file user contains the text pa$$w0rd.
Arguments:
dirname- (required) Absolute path to directory containing the files with a secret.
strip- (optional) if
True(the default), trailing newlines are removed from the file contents.
SecretInVault¶
c['secretsProviders'] = [secrets.SecretInVault(
vaultToken=open('VAULT_TOKEN').read(),
vaultServer="http://localhost:8200",
apiVersion=2
)]
Vault secures, stores, and tightly controls access to secrets. Vault presents a unified API to access multiple backends. At the moment buildbot supports KV v1 and v2 backends. To be authenticated in Vault, Buildbot needs to send a token to the vault server. The token is generated when the Vault instance is initialized for the first time.
In the master configuration, the Vault provider is instantiated through the Buildbot service manager as a secret provider with the Vault server address and the Vault token. The provider SecretInVault allows Buildbot to read secrets in Vault. For more information about Vault please visit: Vault: https://www.vaultproject.io/
2.4.2.4. How to populate secrets in a build¶
To populate secrets in files during a build, 2 steps are used to create and delete the files on the worker. The files will be automatically deleted at the end of the build.
f = BuildFactory()
with f.withSecrets(secrets_list):
f.addStep(step_definition)
or
f = BuildFactory()
f.addSteps([list_of_step_definitions], withSecrets=[secrets_list])
In both cases the secrets_list is a list of tuple (secret path, secret value).
secrets_list = [('/first/path', Interpolate('write something and %(secret:somethingmore)s')),
('/second/path', Interpolate('%(secret:othersecret)s')]
The Interpolate class is used to render the value during the build execution.
2.4.2.5. How to configure a Vault instance¶
Vault being a very generic system, it can be complex to install for the first time. Here is a simple tutorial to install the minimal Vault to use with Buildbot.
Use Docker to install Vault¶
A Docker image is available to help users installing Vault. Without any arguments, the command launches a Docker Vault developer instance, easy to use and test the functions. The developer version is already initialized and unsealed. To launch a Vault server please refer to the VaultDocker documentation:
In a shell:
docker run vault
Starting the vault instance¶
Once the Docker image is created, launch a shell terminal on the Docker image:
docker exec -i -t ``docker_vault_image_name`` /bin/sh
Then, export the environment variable VAULT_ADDR needed to init Vault.
export VAULT_ADDR='vault.server.adress'
Writing secrets¶
By default Vault is initialized with a mount named secret. To add a new secret:
vault kv put secret/new_secret_key value=new_secret_value
