3.10.6. BuildSteps
There are a few parent classes that are used as base classes for real buildsteps. This section describes the base classes. The “leaf” classes are described in Build Steps.
See Writing New BuildSteps for a guide to implementing new steps.
3.10.6.1. BuildStep
- class buildbot.process.buildstep.BuildStep(name, description, descriptionDone, descriptionSuffix, locks, haltOnFailure, flunkOnWarnings, flunkOnFailure, warnOnWarnings, warnOnFailure, alwaysRun, progressMetrics, useProgress, doStepIf, hideStepIf)
All constructor arguments must be given as keyword arguments. Each constructor parameter is copied to the corresponding attribute.
All arguments passed to constructor of the
BuildStep
subclass being constructed are also copied to a separate internal storage. This is used to create new instances in the same way the original instance is created without any interference that the constructors themselves may have. The copying of arguments is done by overriding__new__
.- name
The name of the step. Note that this value may change when the step is started, if the existing name was not unique.
- stepid
The ID of this step in the database. This attribute is not set until the step starts.
- description
The description of the step.
- descriptionDone
The description of the step after it has finished.
- descriptionSuffix
Any extra information to append to the description.
- locks
List of locks for this step; see Interlocks.
- progressMetrics
List of names of metrics that should be used to track the progress of this build and build ETA’s for users.
- useProgress
If true (the default), then ETAs will be calculated for this step using progress metrics. If the step is known to have unpredictable timing (e.g., an incremental build), then this should be set to false.
- doStepIf
A callable or bool to determine whether this step should be executed. See Parameters Common to all Steps for details.
- hideStepIf
A callable or bool to determine whether this step should be shown in the waterfall and build details pages. See Parameters Common to all Steps for details.
The following attributes affect the behavior of the containing build:
- haltOnFailure
If true, the build will halt on a failure of this step, and not execute subsequent steps (except those with
alwaysRun
).
- flunkOnWarnings
If true, the build will be marked as a failure if this step ends with warnings.
- flunkOnFailure
If true, the build will be marked as a failure if this step fails.
- warnOnWarnings
If true, the build will be marked as warnings, or worse, if this step ends with warnings.
- warnOnFailure
If true, the build will be marked as warnings, or worse, if this step fails.
- alwaysRun
If true, the step will run even if a previous step halts the build with
haltOnFailure
.
- logEncoding
The log encoding to use for logs produced in this step, or None to use the global default. See Log Handling.
- rendered
At the beginning of the step, the renderable attributes are rendered against the properties. There is a slight delay however when those are not yet rendered, which leads to weird and difficult to reproduce bugs. To address this problem, a
rendered
attribute is available for methods that could be called early in the buildstep creation.
- results
This is the result (a code from
buildbot.process.results
) of the step. This attribute only exists after the step is finished, and should only be used ingetResultSummary
.
A few important pieces of information are not available when a step is constructed and are added later. These are set by the following methods; the order in which these methods are called is not defined.
- setBuild(build)
- Parameters:
build – the
Build
instance controlling this step.
This method is called during setup to set the build instance controlling this worker. Subclasses can override this to get access to the build object as soon as it is available. The default implementation sets the
build
attribute.
- build
The build object controlling this step.
- setWorker(worker)
- Parameters:
worker – the
Worker
instance on which this step will run.
Similarly, this method is called with the worker that will run this step. The default implementation sets the
worker
attribute.
- worker
The worker that will run this step.
- workdir
Directory where actions of the step will take place. The workdir is set by order of priority:
workdir of the step, if defined via constructor argument
workdir of the BuildFactory (itself defaults to ‘build’)
BuildFactory workdir can also be a function of a sourcestamp (see Factory Workdir Functions).
- setDefaultWorkdir(workdir)
- Parameters:
workdir – the default workdir, from the build
Note
This method is deprecated and should not be used anymore, as workdir is calculated automatically via a property.
- setupProgress()
This method is called during build setup to give the step a chance to set up progress tracking. It is only called if the build has
useProgress
set. There is rarely any reason to override this method.
Execution of the step itself is governed by the following methods and attributes.
- run()
- Returns:
result via Deferred
Execute the step. When this method returns (or when the Deferred it returns fires), the step is complete. The method’s return value must be an integer, giving the result of the step – a constant from
buildbot.process.results
. If the method raises an exception or its Deferred fires with failure, then the step will be completed with an EXCEPTION result. Any other output from the step (logfiles, status strings, URLs, etc.) is the responsibility of therun
method.The function is not called if the step is skipped or otherwise not run.
Subclasses should override this method.
- interrupt(reason)
- Parameters:
reason (string or
Failure
) – why the build was interrupted
This method is used from various control interfaces to stop a running step. The step should be brought to a halt as quickly as possible, by cancelling a remote command, killing a local process, etc.
The
reason
parameter can be a string or, when a worker is lost during step processing, aConnectionLost
failure.The parent method handles any pending lock operations, and should be called by implementations in subclasses.
- stopped
If false, then the step is running. If true, the step is not running, or has been interrupted.
- timed_out
If
True
, then one or more remote commands of the step timed out.
A step can indicate its up-to-the-moment status using a short summary string. These methods allow step subclasses to produce such summaries.
- updateSummary()
Update the summary, calling
getCurrentSummary
orgetResultSummary
as appropriate. Build steps should call this method any time the summary may have changed. This method is debounced, so even calling it for every log line is acceptable.
- getCurrentSummary()
- Returns:
dictionary, optionally via Deferred
Returns a dictionary containing status information for a running step. The dictionary can have a
step
key with a unicode value giving a summary for display with the step. This method is only called while the step is running.Build steps may override this method to provide a more interesting summary than the default
"running"
.
- getResultSummary()
- Returns:
dictionary, optionally via Deferred
Returns a dictionary containing status information for a completed step. The dictionary can have keys
step
andbuild
, each with unicode values. Thestep
key gives a summary for display with the step, while thebuild
key gives a summary for display with the entire build. The latter should be used sparingly, and include only information that the user would find relevant for the entire build, such as a number of test failures. Either or both keys can be omitted.This method is only called when the step is finished. The step’s result is available in
self.results
at that time.Build steps may override this method to provide a more interesting summary than the default, or to provide any build summary information.
- getBuildResultSummary()
- Returns:
dictionary, optionally via Deferred
Returns a dictionary containing status information for a completed step. This method calls
getResultSummary
, and automatically computes abuild
key from thestep
key according to theupdateBuildSummaryPolicy
.
- describe(done=False)
- Parameters:
done – If true, the step is finished.
- Returns:
list of strings
Describe the step succinctly. The return value should be a sequence of short strings suitable for display in a horizontally constrained space.
Note
Be careful not to assume that the step has been started in this method. In relatively rare circumstances, steps are described before they have started. Ideally, unit tests should be used to ensure that this method is resilient.
Note
This method is not called for new-style steps. Instead, override
getCurrentSummary
andgetResultSummary
.
- addTestResultSets()
The steps may override this to add any test result sets for this step via
self.addTestResultSet()
. This function is called just before the step execution is started. The function is not called if the step is skipped or otherwise not run.
- addTestResultSet(description, category, value_unit)
- Parameters:
description – Description of the test result set
category – Category of the test result set
value_unit – The value unit of the test result set
- Returns:
The ID of the created test result set via a Deferred.
Creates a new test result set to which test results can be associated.
There are standard values of the
category
andvalue_unit
parameters, see TODO.
- addTestResult(setid, value, test_name=None, test_code_path=None, line=None, duration_ns=None)
- Parameters:
setid – The ID of a test result set returned by
addTestResultSet
value – The value of the result as a string
test_name – The name of the test
test_code_path – The path to the code file that resulted in this test result
line – The line within
test_code_path
file that resulted in this test resultduration_ns – The duration of the test itself, in nanoseconds
Creates a test result. Either
test_name
ortest_code_path
must be specified. The function queues the test results and will submit them to the database when enough test results are added so that performance impact is minimized.
- finishTestResultSets()
The steps may override this to finish submission of any test results for the step.
Build steps have statistics, a simple key-value store of data which can later be aggregated over all steps in a build. Note that statistics are not preserved after a build is complete.
- setBuildData(self, name, value, source)
- Parameters:
name (unicode) – the name of the data
value (bytestr) – the value of the data as
bytes
source (unicode) – the source of the data
- Returns:
Deferred
Builds can have transient data attached to them which allows steps to communicate to reporters and among themselves. The data is a byte string and its interpretation depends on the particular step or reporter.
- hasStatistic(stat)
- Parameters:
stat (string) – name of the statistic
- Returns:
True if the statistic exists on this step
- getStatistic(stat, default=None)
- Parameters:
stat (string) – name of the statistic
default – default value if the statistic does not exist
- Returns:
value of the statistic, or the default value
- getStatistics()
- Returns:
a dictionary of all statistics for this step
- setStatistic(stat, value)
- Parameters:
stat (string) – name of the statistic
value – value to assign to the statistic
- Returns:
value of the statistic
Build steps support progress metrics - values that increase roughly linearly during the execution of the step, and can thus be used to calculate an expected completion time for a running step. A metric may be a count of lines logged, tests executed, or files compiled. The build mechanics will take care of translating this progress information into an ETA for the user.
- setProgress(metric, value)
- Parameters:
metric (string) – the metric to update
value (integer) – the new value for the metric
Update a progress metric. This should be called by subclasses that can provide useful progress-tracking information.
The specified metric name must be included in
progressMetrics
.
The following methods are provided as utilities to subclasses. These methods should only be invoked after the step has started.
- workerVersion(command, oldversion=None)
- Parameters:
command (string) – command to examine
oldversion – return value if the worker does not specify a version
- Returns:
string
Fetch the version of the named command, as specified on the worker. In practice, all commands on a worker have the same version, but passing
command
is still useful to ensure that the command is implemented on the worker. If the command is not implemented on the worker,workerVersion
will returnNone
.Versions take the form
x.y
wherex
andy
are integers, and are compared as expected for version numbers.Buildbot versions older than 0.5.0 did not support version queries; in this case,
workerVersion
will returnoldVersion
. Since such ancient versions of Buildbot are no longer in use, this functionality is largely vestigial.
- workerVersionIsOlderThan(command, minversion)
- Parameters:
command (string) – command to examine
minversion – minimum version
- Returns:
boolean
This method returns true if
command
is not implemented on the worker, or if it is older thanminversion
.
- checkWorkerHasCommand(command)
- Parameters:
command (string) – command to examine
This method raise
WorkerSetupError
ifcommand
is not implemented on the worker
- getWorkerName()
- Returns:
string
Get the name of the worker assigned to this step.
Most steps exist to run commands. While the details of exactly how those commands are constructed are left to subclasses, the execution of those commands comes down to this method:
- runCommand(command)
- Parameters:
command –
RemoteCommand
instance- Returns:
Deferred
This method connects the given command to the step’s worker and runs it, returning the Deferred from
run
.
The
BuildStep
class provides methods to add log data to the step. Subclasses provide a great deal of user-configurable functionality on top of these methods. These methods can be called while the step is running, but not before.- addLog(name, type='s', logEncoding=None)
- Parameters:
name – log name
type – log type; see
logchunk
logEncoding – the log encoding, or None to use the step or global default (see Log Handling)
- Returns:
Log
instance via Deferred
Add a new logfile with the given name to the step, and return the log file instance.
- getLog(name)
- Parameters:
name – log name
- Raises:
KeyError – if there is no such log
KeyError – if no such log is defined
- Returns:
Log
instance
Return an existing logfile, previously added with
addLog
. Note that this return value is synchronous, and only available afteraddLog
’s deferred has fired.
- addCompleteLog(name, text)
- Parameters:
name – log name
text – content of the logfile
- Returns:
Deferred
This method adds a new log and sets
text
as its content. This is often useful to add a short logfile describing activities performed on the master. The logfile is immediately closed, and no further data can be added.If the logfile’s content is a bytestring, it is decoded with the step’s log encoding or the global default log encoding. To add a logfile with a different character encoding, perform the decode operation directly and pass the resulting unicode string to this method.
- addHTMLLog(name, html)
- Parameters:
name – log name
html – content of the logfile
- Returns:
Deferred
Similar to
addCompleteLog
, this adds a logfile containing pre-formatted HTML, allowing more expressiveness than the text format supported byaddCompleteLog
.
- addLogObserver(logname, observer)
- Parameters:
logname – log name
observer – log observer instance
Add a log observer for the named log. The named log need not have been added already. The observer will be connected when the log is added.
See Adding LogObservers for more information on log observers.
- addLogWithFailure(why, logprefix='')
- Parameters:
why (Failure) – the failure to log
logprefix – prefix for the log name
- Returns:
Deferred
Add log files displaying the given failure, named
<logprefix>err.text
and<logprefix>err.html
.
- addLogWithException(why, logprefix='')
- Parameters:
why (Exception) – the exception to log
logprefix – prefix for the log name
- Returns:
Deferred
Similar to
addLogWithFailure
, but for an Exception instead of a Failure.
Along with logs, build steps have an associated set of links that can be used to provide additional information for developers. Those links are added during the build with this method:
- addURL(name, url)
- Parameters:
name – URL name
url – the URL
Add a link to the given
url
, with the givenname
to displays of this step. This allows a step to provide links to data that is not available in the log files.
3.10.6.2. CommandMixin
The runCommand
method can run a RemoteCommand
instance, but it’s no help in building that object or interpreting the results afterward.
This mixin class adds some useful methods for running commands.
This class can only be used in new-style steps.
- class buildbot.process.buildstep.CommandMixin
Some remote commands are simple enough that they can boil down to a method call. Most of these take an
abandonOnFailure
argument which, if true, will abandon the entire buildstep on command failure. This is accomplished by raisingBuildStepFailed
.These methods all write to the
stdio
log (generally just for errors). They do not close the log when finished.- runRmdir(dir, abandonOnFailure=True)
- Parameters:
dir – directory to remove
abndonOnFailure – if true, abandon step on failure
- Returns:
Boolean via Deferred
Remove the given directory, using the
rmdir
command. Returns False on failure.
- runMkdir(dir, abandonOnFailure=True)
- Parameters:
dir – directory to create
abndonOnFailure – if true, abandon step on failure
- Returns:
Boolean via Deferred
Create the given directory and any parent directories, using the
mkdir
command. Returns False on failure.
- pathExists(path)
- Parameters:
path – path to test
- Returns:
Boolean via Deferred
Determine if the given path exists on the worker (in any form - file, directory, or otherwise). This uses the
stat
command.
- runGlob(path)
- Parameters:
path – path to test
- Returns:
list of filenames
Get the list of files matching the given path pattern on the worker. This uses Python’s
glob
module. If therunGlob
method fails, it aborts the step.
- getFileContentFromWorker(path, abandonOnFailure=False)
- Parameters:
path – path of the file to download from worker
- Returns:
string via deferred (content of the file)
Get the content of a file on the worker.
3.10.6.3. ShellMixin
Most Buildbot steps run shell commands on the worker, and Buildbot has an impressive array of configuration parameters to control that execution.
The ShellMixin
mixin provides the tools to make running shell commands easy and flexible.
This class can only be used in new-style steps.
- class buildbot.process.buildstep.ShellMixin
This mixin manages the following step configuration parameters, the contents of which are documented in the manual. Naturally, all of these are renderable.
- command
- workdir
- env
- want_stdout
- want_stderr
- usePTY
- logfiles
- lazylogfiles
- timeout
- maxTime
- logEnviron
- interruptSignal
- sigtermTime
- initialStdin
- decodeRC
- setupShellMixin(constructorArgs, prohibitArgs=[])
- Parameters:
constructorArgs (dict) – constructor keyword arguments
prohibitArgs (list) – list of recognized arguments to reject
- Returns:
keyword arguments destined for
BuildStep
This method is intended to be called from the shell constructor, and be passed any keyword arguments not otherwise used by the step. Any attributes set on the instance already (e.g., class-level attributes) are used as defaults. Attributes named in
prohibitArgs
are rejected with a configuration error.The return value should be passed to the
BuildStep
constructor.
- makeRemoteShellCommand(collectStdout=False, collectStderr=False, **overrides)
- Parameters:
collectStdout – if true, the command’s stdout will be available in
cmd.stdout
on completioncollectStderr – if true, the command’s stderr will be available in
cmd.stderr
on completionoverrides – overrides arguments that might have been passed to
setupShellMixin
- Returns:
RemoteShellCommand
instance via Deferred
This method constructs a
RemoteShellCommand
instance based on the instance attributes and any supplied overrides. It must be called while the step is running, as it examines the worker capabilities before creating the command. It takes care of just about everything:Creating log files and associating them with the command
Merging environment configuration
Selecting the appropriate workdir configuration
All that remains is to run the command with
runCommand
.
The
ShellMixin
class implementsgetResultSummary
, returning a summary of the command. If no command was specified or run, it falls back to the defaultgetResultSummary
based ondescriptionDone
. Subclasses can override this method to return a more appropriate status.
3.10.6.4. Exceptions
- exception buildbot.process.buildstep.BuildStepFailed
This exception indicates that the buildstep has failed. It is useful as a way to skip all subsequent processing when a step goes wrong.