2.12.6. New-Style Build Steps in Buildbot 0.9.0

In Buildbot-0.9.0, many operations performed by BuildStep subclasses return a Deferred. As a result, custom build steps which call these methods will need to be rewritten.

Buildbot-0.8.9 supports old-style steps natively, while new-style steps are emulated. Buildbot-0.9.0 supports new-style steps natively, while old-style steps are emulated. Buildbot-3.0 no longer supports old-style steps at all. All custom steps should be rewritten in the new style as soon as possible.

Buildbot distinguishes new-style from old-style steps by the presence of a run method. If this method is present, then the step is a new-style step.

2.12.6.1. Summary of Changes

  • New-style steps have a run method that is simpler to implement than the old start method.

  • Many methods are now asynchronous (return Deferreds), as they perform operations on the database.

  • Logs are now implemented by a completely different class. This class supports the same log-writing methods (addStderr and so on), although they are now asynchronous. However, it does not support log-reading methods such as getText. It was never advisable to handle logs as enormous strings. New-style steps should, instead, use a LogObserver or (in Buildbot-0.9.0) fetch log lines bit by bit using the data API.

  • buildbot.process.buildstep.LoggingBuildStep is deprecated and cannot be used in new-style steps. Mix in buildbot.process.buildstep.ShellMixin instead.

  • Step strings, derived by parameters like description, descriptionDone, and descriptionSuffix, are no longer treated as lists. For backward compatibility, the parameters may still be given as lists, but will be joined with spaces during execution (using join_list).

2.12.6.2. Backward Compatibility

Some hacks are in place to support old-style steps. These hacks are only activated when an old-style step is detected. Support for old-style steps has been dropped in Buildbot-3.0.

  • The Deferreds from all asynchronous methods invoked during step execution are gathered internally. The step is not considered finished until all such Deferreds have fired, and is marked EXCEPTION if any fail. For logfiles, this is accomplished by means of a synchronous wrapper class.

  • Logfile data is available while the step is still in memory. This means that logs returned from step.getLog have the expected methods getText, readlines and so on.

  • ShellCommand subclasses implicitly gather all stdio output in memory and provide it to the createSummary method.

2.12.6.3. Rewriting start

If your custom buildstep implements the start method, then rename that method to run and set it up to return a Deferred, either explicitly or via inlineCallbacks. The value of the Deferred should be the result of the step (one of the codes in buildbot.process.results), or a Twisted failure instance to complete the step as EXCEPTION. The new run method should not call self.finished or self.failed, instead signalling the same via Deferred.

For example, the following old-style start method :

def start(self):  ## old style
    cmd = remotecommand.RemoteCommand('stat', {'file': self.file })
    d = self.runCommand(cmd)
    d.addCallback(lambda res: self.convertResult(cmd))
    d.addErrback(self.failed)

Becomes :

@defer.inlineCallbacks
def run(self):  ## new style
    cmd = remotecommand.RemoteCommand('stat', {'file': self.file })
    yield self.runCommand(cmd)
    return self.convertResult(cmd)

2.12.6.4. Newly Asynchronous Methods

The following methods now return a Deferred:

Any custom code in a new-style step that calls these methods must handle the resulting Deferred. In some cases, that means that the calling method’s signature will change. For example :

def summarize(self):  ## old-style
    for m in self.MESSAGES:
        if counts[m]:
            self.addCompleteLog(m, "".join(summaries[m]))
        self.setProperty("count-%s" % m, counts[m], "counter")

Is a synchronous function, not returning a Deferred. However, when converted to a new-style test, it must handle Deferreds from the methods it calls, so it must be asynchronous. Syntactically, inlineCallbacks makes the change fairly simple:

@defer.inlineCallbacks
def summarize(self):  ## new-style
    for m in self.MESSAGES:
        if counts[m]:
            yield self.addCompleteLog(m, "".join(summaries[m]))
        self.setProperty("count-%s" % m, counts[m], "counter")

However, this method’s callers must now handle the Deferred that it returns. All methods that can be overridden in custom steps can return a Deferred.

2.12.6.5. Properties

The API for properties is the same synchronous API as was available in old-style steps. Properties are handled synchronously during the build, and persisted to the database at completion of each step.

2.12.6.6. Log Objects

Old steps had two ways of interacting with logfiles, both of which have changed.

The first is writing to logs while a step is executing. When using addCompleteLog or addHTMLLog, this is straightforward, except that in new-style steps these methods return a Deferred.

The second method is via buildbot.process.buildstep.BuildStep.addLog. In new-style steps, the returned object (via Deferred) has the following methods to add log content:

All of these methods now return Deferreds. None of the old log-reading methods are available on this object:

  • hasContents

  • getText

  • readLines

  • getTextWithHeaders

  • getChunks

If your step uses such methods, consider using a LogObserver instead, or using the Data API to get the required data.

The undocumented and unused subscribeConsumer method of logfiles has also been removed.

The subscribe method now takes a callable, rather than an instance, and does not support catchup. This method was primarily used by LogObserver, the implementation of which has been modified accordingly. Any other uses of the subscribe method should be refactored to use a LogObserver.

2.12.6.7. Status Strings

The self.step_status.setText and setText2 methods have been removed. Similarly, the _describe and describe methods are not used in new-style steps. In fact, steps no longer set their status directly.

Instead, steps call buildbot.process.buildstep.BuildStep.updateSummary whenever the status may have changed. This method calls getCurrentSummary or getResultSummary as appropriate and update displays of the step’s status. Steps override the latter two methods to provide appropriate summaries.

2.12.6.8. Statistics

Support for statistics has been moved to the BuildStep and Build objects. Calls to self.step_status.setStatistic should be rewritten as self.setStatistic.