3.4.7. BuildSteps

There are a few parent classes that are used as base classes for real buildsteps. This section describes the base classes. The "leaf" classes are described in Build Steps.

See Writing New BuildSteps for a guide to implementing new steps.

3.4.7.1. BuildStep

class buildbot.process.buildstep.BuildStep(name, description, descriptionDone, descriptionSuffix, locks, haltOnFailure, flunkOnWarnings, flunkOnFailure, warnOnWarnings, warnOnFailure, alwaysRun, progressMetrics, useProgress, doStepIf, hideStepIf)

All constructor arguments must be given as keyword arguments. Each constructor parameter is copied to the corresponding attribute.

name

The name of the step. Note that this value may change when the step is started, if the existing name was not unique.

stepid

The ID of this step in the database. This attribute is not set until the step starts.

description

The description of the step.

descriptionDone

The description of the step after it has finished.

descriptionSuffix

Any extra information to append to the description.

locks

List of locks for this step; see Interlocks.

progressMetrics

List of names of metrics that should be used to track the progress of this build, and build ETA's for users. This is generally set in the

useProgress

If true (the default), then ETAs will be calculated for this step using progress metrics. If the step is known to have unpredictable timing (e.g., an incremental build), then this should be set to false.

doStepIf

A callable or bool to determine whether this step should be executed. See Common Parameters for details.

hideStepIf

A callable or bool to determine whether this step should be shown in the waterfall and build details pages. See Common Parameters for details.

The following attributes affect the behavior of the containing build:

haltOnFailure

If true, the build will halt on a failure of this step, and not execute subsequent tests (except those with alwaysRun).

flunkOnWarnings

If true, the build will be marked as a failure if this step ends with warnings.

flunkOnFailure

If true, the build will be marked as a failure if this step fails.

warnOnWarnings

If true, the build will be marked as warnings, or worse, if this step ends with warnings.

warnOnFailure

If true, the build will be marked as warnings, or worse, if this step fails.

alwaysRun

If true, the step will run even if a previous step halts the build with haltOnFailure.

logEncoding

The log encoding to use for logs produced in this step, or None to ues the global default. See Log Handling.

rendered

At the beginning of the step, the renderable attributes are rendered against the properties. There is a slight delay however when those are not yet rendered, which lead to weird and difficult to reproduce bugs. To address this problem, a rendered attribute is available for methods that could be called early in the buildstep creation.

results

This is the result (a code from buildbot.process.results) of the step. This attribute only exists after the step is finished, and should only be used in getResultSummary.

A few important pieces of information are not available when a step is constructed, and are added later. These are set by the following methods; the order in which these methods are called is not defined.

setBuild(build)
Parameters:build -- the Build instance controlling this step.

This method is called during setup to set the build instance controlling this worker. Subclasses can override this to get access to the build object as soon as it is available. The default implementation sets the build attribute.

build

The build object controlling this step.

setWorker(build)
Parameters:build -- the Worker instance on which this step will run.

Similarly, this method is called with the worker that will run this step. The default implementation sets the worker attribute.

worker

The worker that will run this step.

workdir

Implemented as a property. Workdir where actions of the step are happening. The workdir is by order of priority

  • workdir of the step, if defined via constructor argument

  • workdir of the BuildFactory (itself defaults to 'build').

    BuildFactory workdir can be a function of sourcestamp. See Factory Workdir Functions

setDefaultWorkdir(workdir)
Parameters:workdir -- the default workdir, from the build

Note

This method is deprecated and should not be used anymore, as workdir is calculated automatically via a property

setupProgress()

This method is called during build setup to give the step a chance to set up progress tracking. It is only called if the build has useProgress set. There is rarely any reason to override this method.

Execution of the step itself is governed by the following methods and attributes.

startStep(remote)
Parameters:remote -- a remote reference to the worker-side WorkerForBuilderPb instance
Returns:Deferred

Begin the step. This is the build's interface to step execution. Subclasses should override run to implement custom behaviors.

run()
Returns:result via Deferred

Execute the step. When this method returns (or when the Deferred it returns fires), the step is complete. The method's return value must be an integer, giving the result of the step -- a constant from buildbot.process.results. If the method raises an exception or its Deferred fires with failure, then the step will be completed with an EXCEPTION result. Any other output from the step (logfiles, status strings, URLs, etc.) is the responsibility of the run method.

Subclasses should override this method. Do not call finished or failed from this method.

start()
Returns:None or SKIPPED, optionally via a Deferred.

Begin the step. BuildSteps written before Buildbot-0.9.0 often override this method instead of run, but this approach is deprecated.

When the step is done, it should call finished, with a result -- a constant from buildbot.process.results. The result will be handed off to the Build.

If the step encounters an exception, it should call failed with a Failure object.

If the step decides it does not need to be run, start can return the constant SKIPPED. In this case, it is not necessary to call finished directly.

finished(results)
Parameters:results -- a constant from results

A call to this method indicates that the step is finished and the build should analyze the results and perhaps proceed to the next step. The step should not perform any additional processing after calling this method. This method must only be called from the (deprecated) start method.

failed(failure)
Parameters:failure -- a Failure instance

Similar to finished, this method indicates that the step is finished, but handles exceptions with appropriate logging and diagnostics.

This method handles BuildStepFailed specially, by calling finished(FAILURE). This provides subclasses with a shortcut to stop execution of a step by raising this failure in a context where failed will catch it. This method must only be called from the (deprecated) start method.

interrupt(reason)
Parameters:reason (string or Failure) -- why the build was interrupted

This method is used from various control interfaces to stop a running step. The step should be brought to a halt as quickly as possible, by cancelling a remote command, killing a local process, etc. The step must still finish with either finished or failed.

The reason parameter can be a string or, when a worker is lost during step processing, a ConnectionLost failure.

The parent method handles any pending lock operations, and should be called by implementations in subclasses.

stopped

If false, then the step is running. If true, the step is not running, or has been interrupted.

A step can indicate its up-to-the-moment status using a short summary string. These methods allow step subclasses to produce such summaries.

updateSummary()

Update the summary, calling getCurrentSummary or getResultSummary as appropriate. New-style build steps should call this method any time the summary may have changed. This method is debounced, so even calling it for every log line is acceptable.

getCurrentSummary()
Returns:dictionary, optionally via Deferred

Returns a dictionary containing status information for a running step. The dictionary can have a step key with a unicode value giving a summary for display with the step. This method is only called while the step is running.

New-style build steps should override this method to provide a more interesting summary than the default u"running".

getResultSummary()
Returns:dictionary, optionally via Deferred

Returns a dictionary containing status information for a completed step. The dictionary can have keys step and build, each with unicode values. The step key gives a summary for display with the step, while the build key gives a summary for display with the entire build. The latter should be used sparingly, and include only information that the user would find relevant for the entire build, such as a number of test failures. Either or both keys can be omitted.

This method is only called while the step is finished. The step's result is available in self.results at that time.

New-style build steps should override this method to provide a more interesting summary than the default, or to provide any build summary information.

getBuildResultSummary()
Returns:dictionary, optionally via Deferred

Returns a dictionary containing status information for a completed step. This method calls getResultSummary, and automatically compute a build key from the step key according to the updateBuildSummaryPolicy

describe(done=False)
Parameters:done -- If true, the step is finished.
Returns:list of strings

Describe the step succinctly. The return value should be a sequence of short strings suitable for display in a horizontally constrained space.

Note

Be careful not to assume that the step has been started in this method. In relatively rare circumstances, steps are described before they have started. Ideally, unit tests should be used to ensure that this method is resilient.

Note

This method is not called for new-style steps. Instead, override getCurrentSummary and getResultSummary.

Build steps have statistics, a simple key/value store of data which can later be aggregated over all steps in a build. Note that statistics are not preserved after a build is complete.

hasStatistic(stat)
Parameters:stat (string) -- name of the statistic
Returns:True if the statistic exists on this step
getStatistic(stat, default=None)
Parameters:
  • stat (string) -- name of the statistic
  • default -- default value if the statistic does not exist
Returns:

value of the statistic, or the default value
getStatistics()
Returns:a dictionary of all statistics for this step
setStatistic(stat, value)
Parameters:
  • stat (string) -- name of the statistic
  • value -- value to assign to the statistic
Returns:

value of the statistic

Build steps support progress metrics - values that increase roughly linearly during the execution of the step, and can thus be used to calculate an expected completion time for a running step. A metric may be a count of lines logged, tests executed, or files compiled. The build mechanics will take care of translating this progress information into an ETA for the user.

setProgress(metric, value)
Parameters:
  • metric (string) -- the metric to update
  • value (integer) -- the new value for the metric

Update a progress metric. This should be called by subclasses that can provide useful progress-tracking information.

The specified metric name must be included in progressMetrics.

The following methods are provided as utilities to subclasses. These methods should only be invoked after the step is started.

workerVersion(command, oldversion=None)
Parameters:
  • command (string) -- command to examine
  • oldversion -- return value if the worker does not specify a version
Returns:

string

Fetch the version of the named command, as specified on the worker. In practice, all commands on a worker have the same version, but passing command is still useful to ensure that the command is implemented on the worker. If the command is not implemented on the worker, workerVersion will return None.

Versions take the form x.y where x and y are integers, and are compared as expected for version numbers.

Buildbot versions older than 0.5.0 did not support version queries; in this case, workerVersion will return oldVersion. Since such ancient versions of Buildbot are no longer in use, this functionality is largely vestigial.

workerVersionIsOlderThan(command, minversion)
Parameters:
  • command (string) -- command to examine
  • minversion -- minimum version
Returns:

boolean

This method returns true if command is not implemented on the worker, or if it is older than minversion.

checkWorkerHasCommand(command)
Parameters:command (string) -- command to examine

This method raise WorkerTooOldError if command is not implemented on the worker

getWorkerName()
Returns:string

Get the name of the worker assigned to this step.

Most steps exist to run commands. While the details of exactly how those commands are constructed are left to subclasses, the execution of those commands comes down to this method:

runCommand(command)
Parameters:command -- RemoteCommand instance
Returns:Deferred

This method connects the given command to the step's worker and runs it, returning the Deferred from run.

The BuildStep class provides methods to add log data to the step. Subclasses provide a great deal of user-configurable functionality on top of these methods. These methods can be called while the step is running, but not before.

addLog(name, type="s", logEncoding=None)
Parameters:
  • name -- log name
  • type -- log type; see logchunk
  • logEncoding -- the log encoding, or None to use the step or global default (see Log Handling)
Returns:

Log instance via Deferred

Add a new logfile with the given name to the step, and return the log file instance.

getLog(name)
Parameters:

name -- log name
Raises:
  • KeyError -- if there is no such log
  • KeyError -- if no such log is defined
Returns:

Log instance

Return an existing logfile, previously added with addLog. Note that this return value is synchronous, and only available after addLog's deferred has fired.

addCompleteLog(name, text)
Parameters:
  • name -- log name
  • text -- content of the logfile
Returns:

Deferred

This method adds a new log and sets text as its content. This is often useful to add a short logfile describing activities performed on the master. The logfile is immediately closed, and no further data can be added.

If the logfile's content is a bytestring, it is decoded with the step's log encoding or the global default log encoding. To add a logfile with a different character encoding, perform the decode operation directly and pass the resulting unicode string to this method.

addHTMLLog(name, html)
Parameters:
  • name -- log name
  • html -- content of the logfile
Returns:

Deferred

Similar to addCompleteLog, this adds a logfile containing pre-formatted HTML, allowing more expressiveness than the text format supported by addCompleteLog.

addLogObserver(logname, observer)
Parameters:
  • logname -- log name
  • observer -- log observer instance

Add a log observer for the named log. The named log need not have been added already: the observer will be connected when the log is added.

See Adding LogObservers for more information on log observers.

addLogWithFailure(why, logprefix='')
Parameters:
  • why (Failure) -- the failure to log
  • logprefix -- prefix for the log name
Returns:

Deferred

Add log files displaying the given failure, named <logprefix>err.text and <logprefix>err.html.

addLogWithException(why, logprefix='')
Parameters:
  • why (Exception) -- the exception to log
  • logprefix -- prefix for the log name
Returns:

Deferred

Similar to addLogWithFailure, but for an Exception instead of a Failure.

Along with logs, build steps have an associated set of links that can be used to provide additional information for developers. Those links are added during the build with this method:

addURL(name, url)
Parameters:
  • name -- URL name
  • url -- the URL

Add a link to the given url, with the given name to displays of this step. This allows a step to provide links to data that is not available in the log files.

3.4.7.2. LoggingBuildStep

class buildbot.process.buildstep.LoggingBuildStep(name, locks, haltOnFailure, flunkOnWarnings, flunkOnFailure, warnOnWarnings, warnOnFailure, alwaysRun, progressMetrics, useProgress, doStepIf, hideStepIf)

The remaining arguments are passed to the BuildStep constructor.

Warning

Subclasses of this class are always old-style steps. As such, this class will be removed after Buildbot-0.9.0. Instead, subclass BuildStep and mix in ShellMixin to get similar behavior.

This subclass of BuildStep is designed to help its subclasses run remote commands that produce standard I/O logfiles. It:

  • tracks progress using the length of the stdout logfile
  • provides hooks for summarizing and evaluating the command's result
  • supports lazy logfiles
  • handles the mechanics of starting, interrupting, and finishing remote commands
  • detects lost workers and finishes with a status of RETRY
logfiles

The logfiles to track, as described for ShellCommand. The contents of the class-level logfiles attribute are combined with those passed to the constructor, so subclasses may add log files with a class attribute:

class MyStep(LoggingBuildStep):
    logfiles = dict(debug='debug.log')

Note that lazy logfiles cannot be specified using this method; they must be provided as constructor arguments.

setupLogsRunCommandAndProcessResults(cmd, stdioLog=None, closeLogWhenFinished=True, errorMessages=None, logfiles=None, lazylogfiles=False):
Parameters:
  • command -- the RemoteCommand instance to start
  • stdioLog -- an optional Log object where the stdout of the command will be stored.
  • closeLogWhenFinished -- a boolean
  • logfiles -- optional dictionary see ShellCommand
  • lazylogfiles -- optional boolean see ShellCommand
Returns:

step result from buildbot.process.results

Note

This method permits an optional errorMessages parameter, allowing errors detected early in the command process to be logged. It will be removed, and its use is deprecated.

Handle all of the mechanics of running the given command. This sets up all required logfiles, and calls the utility hooks described below.

Subclasses should use that method if they want to launch multiple commands in a single step. One could use that method, like for example

@defer.inlineCallbacks
def run(self):
    cmd = RemoteCommand(...)
    res = yield self.setupLogRunCommandAndProcessResults(cmd)
    if res == results.SUCCESS:
         cmd = RemoteCommand(...)
         res = yield self.setupLogRunCommandAndProcessResults(cmd)
    defer.returnValue(res)

To refine the status output, override one or more of the following methods. The LoggingBuildStep implementations are stubs, so there is no need to call the parent method.

commandComplete(command)
Parameters:command -- the just-completed remote command

This is a general-purpose hook method for subclasses. It will be called after the remote command has finished, but before any of the other hook functions are called.

evaluateCommand(command)
Parameters:command -- the just-completed remote command
Returns:step result from buildbot.process.results

This hook should decide what result the step should have.

3.4.7.3. CommandMixin

The runCommand method can run a RemoteCommand instance, but it's no help in building that object or interpreting the results afterward. This mixin class adds some useful methods for running commands.

This class can only be used in new-style steps.

class buildbot.process.buildstep.CommandMixin

Some remote commands are simple enough that they can boil down to a method call. Most of these take an abandonOnFailure argument which, if true, will abandon the entire buildstep on command failure. This is accomplished by raising BuildStepFailed.

These methods all write to the stdio log (generally just for errors). They do not close the log when finished.

runRmdir(dir, abandonOnFailure=True)
Parameters:
  • dir -- directory to remove
  • abndonOnFailure -- if true, abandon step on failure
Returns:

Boolean via Deferred

Remove the given directory, using the rmdir command. Returns False on failure.

runMkdir(dir, abandonOnFailure=True)
Parameters:
  • dir -- directory to create
  • abndonOnFailure -- if true, abandon step on failure
Returns:

Boolean via Deferred

Create the given directory and any parent directories, using the mkdir command. Returns False on failure.

pathExists(path)
Parameters:path -- path to test
Returns:Boolean via Deferred

Determine if the given path exists on the worker (in any form - file, directory, or otherwise). This uses the stat command.

runGlob(path)
Parameters:path -- path to test
Returns:list of filenames

Get the list of files matching the given path pattern on the worker. This uses Python's glob module. If the runGlob method fails, it aborts the step.

getFileContentFromWorker(path, abandonOnFailure=False)
Parameters:path -- path of the file to download from worker
Returns:string via deferred (content of the file)

Get the content of a file on the worker.

3.4.7.4. ShellMixin

Most Buildbot steps run shell commands on the worker, and Buildbot has an impressive array of configuration parameters to control that execution. The ShellMixin mixin provides the tools to make running shell commands easy and flexible.

This class can only be used in new-style steps.

class buildbot.process.buildstep.ShellMixin

This mixin manages the following step configuration parameters, the contents of which are documented in the manual. Naturally, all of these are renderable.

command
workdir
env
want_stdout
want_stderr
usePTY
logfiles
lazylogfiles
timeout
maxTime
logEnviron
interruptSignal
sigtermTime
initialStdin
decodeRC
setupShellMixin(constructorArgs, prohibitArgs=[])
Parameters:
  • constructorArgs (dict) -- constructor keyword arguments
  • prohibitArgs (list) -- list of recognized arguments to reject
Returns:

keyword arguments destined for BuildStep

This method is intended to be called from the shell constructor, passed any keyword arguments not otherwise used by the step. Any attributes set on the instance already (e.g., class-level attributes) are used as defaults. Attributes named in prohibitArgs are rejected with a configuration error.

The return value should be passed to the BuildStep constructor.

makeRemoteShellCommand(collectStdout=False, collectStderr=False, **overrides)
Parameters:
  • collectStdout -- if true, the command's stdout wil be available in cmd.stdout on completion
  • collectStderr -- if true, the command's stderr wil be available in cmd.stderr on completion
  • overrides -- overrides arguments that might have been passed to setupShellMixin
Returns:

RemoteShellCommand instance via Deferred

This method constructs a RemoteShellCommand instance based on the instance attributes and any supplied overrides. It must be called while the step is running, as it examines the worker capabilities before creating the command. It takes care of just about everything:

  • Creating log files and associating them with the command
  • Merging environment configuration
  • Selecting the appropriate workdir configuration

All that remains is to run the command with runCommand.

The ShellMixin class implements getResultSummary, returning a summary of the command. If no command was specified or run, it falls back to the default getResultSummary based on descriptionDone. Subclasses can override this method to return a more appropriate status.

3.4.7.5. Exceptions

exception buildbot.process.buildstep.BuildStepFailed

This exception indicates that the buildstep has failed. It is useful as a way to skip all subsequent processing when a step goes wrong. It is handled by BuildStep.failed.