2.13. Deployment

This page aims at describing the common pitfalls and best practices when deploying buildbot.

2.13.1. Using A Database Server

Buildbot uses the sqlite3 database backend by default. If you plan to host a lot of data, you may consider using a more suitable database server.

If you want to use a database server (e.g., MySQL or Postgres) as the database backend for your Buildbot, use option buildbot create-master –db to specify the connection string for the database, and make sure that the same URL appears in the db_url of the db parameter in your configuration file.

2.13.1.1. Additional Requirements

Depending on the selected database, further Python packages will be required. Consult the SQLAlchemy dialect list for a full description. The most common choice for MySQL is mysqlclient. Any reasonably recent version should suffice.

The most common choice for Postgres is Psycopg Any reasonably recent version should suffice.

2.13.2. Maintenance

The buildmaster can be configured to send out email notifications when a worker has been offline for a while. Be sure to configure the buildmaster with a contact email address for each worker so these notifications are sent to someone who can bring it back online.

If you find you can no longer provide a worker to the project, please let the project admins know, so they can put out a call for a replacement.

The Buildbot records status and logs output continually, each time a build is performed. The status tends to be small, but the build logs can become quite large. Each build and log are recorded in a separate file, arranged hierarchically under the buildmaster’s base directory. To prevent these files from growing without bound, you should periodically delete old build logs. A simple cron job to delete anything older than, say, two weeks should do the job. The only trick is to leave the buildbot.tac and other support files alone, for which find’s -mindepth argument helps skip everything in the top directory. You can use something like the following (assuming builds are stored in ./builds/ directory):

@weekly cd BASEDIR && find . -mindepth 2 i-path './builds/*' \
    -prune -o -type f -mtime +14 -exec rm {} \;
@weekly cd BASEDIR && find twistd.log* -mtime +14 -exec rm {} \;

Alternatively, you can configure a maximum number of old logs to be kept using the --log-count command line option when running buildbot-worker create-worker or buildbot create-master.

2.13.3. Troubleshooting

Here are a few hints on diagnosing common problems.

2.13.3.1. Starting the worker

Cron jobs are typically run with a minimal shell (/bin/sh, not /bin/bash), and tilde expansion is not always performed in such commands. You may want to use explicit paths, because the PATH is usually quite short and doesn’t include anything set by your shell’s startup scripts (.profile, .bashrc, etc). If you’ve installed buildbot (or other Python libraries) to an unusual location, you may need to add a PYTHONPATH specification (note that Python will do tilde-expansion on PYTHONPATH elements by itself). Sometimes it is safer to fully-specify everything:

@reboot PYTHONPATH=~/lib/python /usr/local/bin/buildbot \
    start /usr/home/buildbot/basedir

Take the time to get the @reboot job set up. Otherwise, things will work fine for a while, but the first power outage or system reboot you have will stop the worker with nothing but the cries of sorrowful developers to remind you that it has gone away.

2.13.3.2. Connecting to the buildmaster

If the worker cannot connect to the buildmaster, the reason should be described in the twistd.log logfile. Some common problems are an incorrect master hostname or port number, or a mistyped bot name or password. If the worker loses the connection to the master, it is supposed to attempt to reconnect with an exponentially-increasing backoff. Each attempt (and the time of the next attempt) will be logged. If you get impatient, just manually stop and re-start the worker.

When the buildmaster is restarted, all workers will be disconnected, and will attempt to reconnect as usual. The reconnect time will depend upon how long the buildmaster is offline (i.e. how far up the exponential backoff curve the workers have travelled). Again, buildbot-worker restart BASEDIR will speed up the process.

2.13.3.3. Contrib Scripts

While some features of Buildbot are included in the distribution, others are only available in master/contrib/ in the buildbot-contrib source directory. The latest versions of such scripts are available at master/contrib.