AWS EC2 is a web service that allows you to start virtual machines in an Amazon data center. Please see their website for details, incuding costs. Using the AWS EC2 latent buildslaves involves getting an EC2 account with AWS and setting up payment; customizing one or more EC2 machine images ("AMIs") on your desired operating system(s) and publishing them (privately if needed); and configuring the buildbot master to know how to start your customized images for "substantiating" your latent slaves.
To start off, to use the AWS EC2 latent buildslave, you need to get an AWS developer account and sign up for EC2. These instructions may help you get started:
Now you need to create an AMI and configure the master. You may need to run through this cycle a few times to get it working, but these instructions should get you started.
Creating an AMI is out of the scope of this document. The EC2 Getting Started Guide is a good resource for this task. Here are a few additional hints.
Now let's assume you have an AMI that should work with the EC2LatentBuildSlave. It's now time to set up your buildbot master configuration.
You will need some information from your AWS account: the "Access Key Id" and the "Secret Access Key". If you've built the AMI yourself, you probably already are familiar with these values. If you have not, and someone has given you access to an AMI, these hints may help you find the necessary values:
When creating an EC2LatentBuildSlave in the buildbot master configuration, the first three arguments are required. The name and password are the first two arguments, and work the same as with normal buildslaves. The next argument specifies the type of the EC2 virtual machine (available options as of this writing include "m1.small", "m1.large", 'm1.xlarge", "c1.medium", and "c1.xlarge"; see the EC2 documentation for descriptions of these machines).
Here is the simplest example of configuring an EC2 latent buildslave. It specifies all necessary remaining values explicitly in the instantiation.
from buildbot.ec2buildslave import EC2LatentBuildSlave
c['slaves'] = [EC2LatentBuildSlave('bot1', 'sekrit', 'm1.large',
ami='ami-12345',
identifier='publickey',
secret_identifier='privatekey'
)]
The "ami" argument specifies the AMI that the master should start. The "identifier" argument specifies the AWS "Access Key Id," and the "secret_identifier" specifies the AWS "Secret Access Key." Both the AMI and the account information can be specified in alternate ways.
Note that whoever has your identifier and secret_identifier values can request AWS work charged to your account, so these values need to be carefully protected. Another way to specify these access keys is to put them in a separate file. You can then make the access privileges stricter for this separate file, and potentially let more people read your main configuration file.
By default, you can make an .ec2 directory in the home folder of the user running the buildbot master. In that directory, create a file called aws_id. The first line of that file should be your access key id; the second line should be your secret access key id. Then you can instantiate the build slave as follows.
from buildbot.ec2buildslave import EC2LatentBuildSlave
c['slaves'] = [EC2LatentBuildSlave('bot1', 'sekrit', 'm1.large',
ami='ami-12345')]
If you want to put the key information in another file, use the "aws_id_file_path" initialization argument.
Previous examples used a particular AMI. If the Buildbot master will be deployed in a process-controlled environment, it may be convenient to specify the AMI more flexibly. Rather than specifying an individual AMI, specify one or two AMI filters.
In all cases, the AMI that sorts last by its location (the S3 bucket and manifest name) will be preferred.
One available filter is to specify the acceptable AMI owners, by AWS account number (the 12 digit number, usually rendered in AWS with hyphens like "1234-5678-9012", should be entered as in integer).
from buildbot.ec2buildslave import EC2LatentBuildSlave
bot1 = EC2LatentBuildSlave('bot1', 'sekrit', 'm1.large',
valid_ami_owners=[11111111111,
22222222222],
identifier='publickey',
secret_identifier='privatekey'
)
The other available filter is to provide a regular expression string that will be matched against each AMI's location (the S3 bucket and manifest name).
from buildbot.ec2buildslave import EC2LatentBuildSlave
bot1 = EC2LatentBuildSlave(
'bot1', 'sekrit', 'm1.large',
valid_ami_location_regex=r'buildbot\-.*/image.manifest.xml',
identifier='publickey', secret_identifier='privatekey')
The regular expression can specify a group, which will be preferred for the sorting. Only the first group is used; subsequent groups are ignored.
from buildbot.ec2buildslave import EC2LatentBuildSlave
bot1 = EC2LatentBuildSlave(
'bot1', 'sekrit', 'm1.large',
valid_ami_location_regex=r'buildbot\-.*\-(.*)/image.manifest.xml',
identifier='publickey', secret_identifier='privatekey')
If the group can be cast to an integer, it will be. This allows 10 to sort after 1, for instance.
from buildbot.ec2buildslave import EC2LatentBuildSlave
bot1 = EC2LatentBuildSlave(
'bot1', 'sekrit', 'm1.large',
valid_ami_location_regex=r'buildbot\-.*\-(\d+)/image.manifest.xml',
identifier='publickey', secret_identifier='privatekey')
In addition to using the password as a handshake between the master and the slave, you may want to use a firewall to assert that only machines from a specific IP can connect as slaves. This is possible with AWS EC2 by using the Elastic IP feature. To configure, generate a Elastic IP in AWS, and then specify it in your configuration using the "elastic_ip" argument.
from buildbot.ec2buildslave import EC2LatentBuildSlave
c['slaves'] = [EC2LatentBuildSlave('bot1', 'sekrit', 'm1.large',
'ami-12345',
identifier='publickey',
secret_identifier='privatekey',
elastic_ip='208.77.188.166'
)]
The EC2LatentBuildSlave supports all other configuration from the standard BuildSlave. The "missing_timeout" and "notify_on_missing" specify how long to wait for an EC2 instance to attach before considering the attempt to have failed, and email addresses to alert, respectively. "missing_timeout" defaults to 20 minutes.
The "build_wait_timeout" allows you to specify how long an EC2LatentBuildSlave should wait after a build for another build before it shuts down the EC2 instance. It defaults to 10 minutes.
"keypair_name" and "security_name" allow you to specify different names for these AWS EC2 values. They both default to "latent_buildbot_slave".