2.12.5. Upgrading to Buildbot 0.9.0

Upgrading a Buildbot instance from 0.8.x to 0.9.x may require a number of changes to the master configuration. Those changes are summarized here. If you are starting fresh with 0.9.0 or later, you can safely skip this section.

First important note is that Buildbot does not support an upgrade of a 0.8.x instance to 0.9.x. Notably the build data and logs will not be accessible anymore if you upgraded, thus the database migration scripts have been dropped.

You should not pip upgrade -U buildbot, but rather start from a clean virtualenv aside from your old master. You can keep your old master instance to serve the old build status.

Buildbot is now composed of several Python packages and Javascript UI, and the easiest way to install it is to run the following command within a virtualenv:

pip install 'buildbot[bundle]'

2.12.5.1. Config File Syntax

In preparation for compatibility with Python 3, Buildbot configuration files no longer allow the print statement:

print "foo"

To fix, simply enclose the print arguments in parentheses:

print("foo")

2.12.5.2. Plugins

Although plugin support was available in 0.8.12, its use is now highly recommended. Instead of importing modules directly in master.cfg, import the plugin kind from buildbot.plugins:

from buildbot.plugins import steps

Then access the plugin itself as an attribute:

steps.SetProperty(..)

See Plugin Infrastructure in Buildbot for more information.

2.12.5.3. Web Status

The most prominent change is that the existing WebStatus class is now gone, replaced by the new www functionality.

Thus an html.WebStatus entry in c['status'] should be removed and replaced with configuration in c['www']. For example, replace:

from buildbot.status import html
c['status'].append(html.WebStatus(http_port=8010, allowForce=True)

with:

c['www'] = {
    "port": 8010,
    "plugins": {
        "waterfall_view": {},
        "console_view": {}
    }
}

See www for more information.

2.12.5.4. Status Classes

Where in 0.8.x most of the data about a build was available synchronously, it must now be fetched dynamically using the Data API. All classes under the Python package buildbot.status should be considered deprecated. Many have already been removed, and the remainder have limited functionality. Any custom code which refers to these classes must be rewritten to use the Data API. Avoid the temptation to reach into the Buildbot source code to find other useful-looking methods!

Common uses of the status API are:

  • getBuild in a custom renderable

  • MailNotifier message formatters (see below for upgrade hints)

  • doStepIf functions on steps

Import paths for several classes under the buildbot.status package but which remain useful have changed. Most of these are now available as plugins (see above), but for the remainder, consult the source code.

2.12.5.5. BuildRequest Merging

Buildbot 0.9.x has replaced the old concept of request merging (mergeRequests) with a more flexible request-collapsing mechanism. See collapseRequests for more information.

2.12.5.6. Status Reporters

In fact, the whole c['status'] configuration parameter is gone.

Many of the status listeners used in the status hierarchy in 0.8.x have been replaced with “reporters” that are available as buildbot plugins. However, note that not all status listeners have yet been ported. See the release notes for details.

Including the "status" key in the configuration object will cause a configuration error. All reporters should be included in c['services'] as described in Reporters.

The available reporters as of 0.9.0 are

See the reporter index for the full, current list.

A few notes on changes to the configuration of these reporters:

  • MailNotifier argument messageFormatter should now be a buildbot.reporters.message.MessageFormatter, due to the removal of the status classes (see above), such formatters must be re-implemented using the Data API.

  • MailNotifier argument previousBuildGetter is not supported anymore

  • MailNotifier no longer forces SSL 3.0 when useTls is true.

  • GerritStatusPush callbacks slightly changed signature, and include a master reference instead of a status reference.

  • GitHubStatusPush now accepts a context parameter to be passed to the GitHub Status API.

  • buildbot.status.builder.Results and the constants buildbot.status.results.SUCCESS should be imported from the buildbot.process.results module instead.

2.12.5.7. Steps

Buildbot-0.8.9 introduced “new-style steps”, with an asynchronous run method. In the remaining 0.8.x releases, use of new-style and old-style steps were supported side-by-side. In 0.9.x, old-style steps are emulated using a collection of hacks to allow asynchronous calls to be called from synchronous code. This emulation is imperfect, and you are strongly encouraged to rewrite any custom steps as New-Style Build Steps in Buildbot 0.9.0.

Note that new-style steps now “push” their status when it changes, so the describe method no longer exists.

2.12.5.8. Identifiers

Many strings in Buildbot must now be identifiers. Identifiers are designed to fit easily and unambiguously into URLs, AMQP routes, and the like. An “identifier” is a nonempty unicode string of limited length, containing only UTF-8 alphanumeric characters along with - (dash) and _ (underscore), and not beginning with a digit

Unfortunately, many existing names do not fit this pattern.

The following fields are identifiers:

  • worker name (50-character)

  • builder name (70-character)

  • step name (50-character)

2.12.5.9. Serving static files

Since version 0.9.0 Buildbot doesn’t use and doesn’t serve master’s public_html directory. You need to use third-party HTTP server for serving static files.

2.12.5.10. Transition to “worker” terminology

Since version 0.9.0 of Buildbot “slave”-based terminology is deprecated in favor of “worker”-based terminology.

All identifiers, messages and documentation were updated to use “worker” instead of “slave”. Old API names are still available in Buildbot versions from 0.9.0 to 1.8.0, but deprecated. The support for old API names has been removed in Buildbot version 2.0.0. To upgrade pre-0.9.0 Buildbot installation a two-stage upgrade is recommended. First, upgrade to Buildbot version 1.8.0, then fix all deprecation warnings and finally upgrade to Buildbot version 2.x.y.

For details about changed API and how to control generated warnings see Transition to “worker” terminology in BuildBot 0.9.0.

2.12.5.11. Other Config Settings

The default master.cfg file contains some new changes, which you should look over:

  • c['protocols'] = {'pb': {'port': 9989}} (the default port used by the workers)

  • Waterfall View: requires installation (pip install buildbot-waterfall-view) and configuration (c['www'] = { ..., 'plugins': {'waterfall_view': {} }).

2.12.5.12. Build History

There is no support for importing build history from 0.8.x (where the history was stored on-disk in pickle files) into 0.9.x (where it is stored in the database).

2.12.5.13. Data LifeTime

Buildbot Nine data being implemented fully in an SQL database, the buildHorizon feature had to be reworked. Instead of being number-of-things based, it is now time based. This makes more sense from a user perspective but makes it harder to predict the database average size. Please be careful to provision enough disk space for your database.

The old c['logHorizon'] way of configuring is not supported anymore. See JanitorConfigurator to learn how to configure. A new __Janitor builder will be created to help keep an eye on the cleanup activities.

2.12.5.14. Upgrading worker

Upgrading worker requires updating the buildbot.tac file to use the new APIs.

The easiest solution is to simply delete the worker directory and re-run buildbot-worker create-worker to get the stock buildbot.tac.

If the loss of the cached worker state is a problem, then the buildbot.tac can be updated manually:

  1. Replace:

    from buildslave.bot import BuildSlave

    with:

    from buildbot_worker.bot import Worker

  2. Replace:

    application = service.Application('buildslave')

    with:

    application = service.Application('buildbot-worker')

  3. Replace:

    s = BuildSlave(buildmaster_host, port, slavename, passwd, basedir,
               keepalive, usepty, umask=umask, maxdelay=maxdelay,
               numcpus=numcpus, allow_shutdown=allow_shutdown)

    with:

    s = Worker(buildmaster_host, port, slavename, passwd, basedir,
           keepalive, umask=umask, maxdelay=maxdelay,
           numcpus=numcpus, allow_shutdown=allow_shutdown)

2.12.5.15. More Information

For minor changes not mentioned here, consult the release notes for the versions over which you are upgrading.

Buildbot-0.9.0 represents several years’ work, and as such we may have missed potential migration issues.