4.11.4.1 Using ShellCommands

This is a useful base class for just about everything you might want to do during a build (except for the initial source checkout). It runs a single command in a child shell on the buildslave. All stdout/stderr is recorded into a LogFile. The step finishes with a status of FAILURE if the command's exit code is non-zero, otherwise it has a status of SUCCESS.

The preferred way to specify the command is with a list of argv strings, since this allows for spaces in filenames and avoids doing any fragile shell-escaping. You can also specify the command with a single string, in which case the string is given to '/bin/sh -c COMMAND' for parsing.

On Windows, commands are run via cmd.exe /c which works well. However, if you're running a batch file, the error level does not get propagated correctly unless you add 'call' before your batch file's name: cmd=['call', 'myfile.bat', ...].

ShellCommand arguments:

command

a list of strings (preferred) or single string (discouraged) which specifies the command to be run. A list of strings is preferred because it can be used directly as an argv array. Using a single string (with embedded spaces) requires the buildslave to pass the string to /bin/sh for interpretation, which raises all sorts of difficult questions about how to escape or interpret shell metacharacters.

workdir

All ShellCommands are run by default in the “workdir”, which defaults to the “build” subdirectory of the slave builder's base directory. The absolute path of the workdir will thus be the slave's basedir (set as an option to buildbot create-slave, see Creating a buildslave) plus the builder's basedir (set in the builder's c['builddir'] key in master.cfg) plus the workdir itself (a class-level attribute of the BuildFactory, defaults to “build”).

For example:

          f.addStep(ShellCommand(command=["make", "test"],
                                 workdir="build/tests"))

env

a dictionary of environment strings which will be added to the child command's environment. For example, to run tests with a different i18n language setting, you might use

          f.addStep(ShellCommand(command=["make", "test"],
                                 env={'LANG': 'fr_FR'}))

These variable settings will override any existing ones in the buildslave's environment or the environment specified in the Builder. The exception is PYTHONPATH, which is merged with (actually prepended to) any existing $PYTHONPATH setting. The value is treated as a list of directories to prepend, and a single string is treated like a one-item list. For example, to prepend both /usr/local/lib/python2.3 and /home/buildbot/lib/python to any existing $PYTHONPATH setting, you would do something like the following:

          f.addStep(ShellCommand(
                        command=["make", "test"],
                        env={'PYTHONPATH': ["/usr/local/lib/python2.3",
                                             "/home/buildbot/lib/python"] }))

Those variables support expansion so that if you just want to prepend /home/buildbot/bin to the PATH environment variable, you can do it by putting the value ${PATH} at the end of the string like in the example below. Variables that doesn't exists on the slave will be replaced by "".

          f.addStep(ShellCommand(
                        command=["make", "test"],
                        env={'PATH': "/home/buildbot/bin:${PATH}"}))

want_stdout

if False, stdout from the child process is discarded rather than being sent to the buildmaster for inclusion in the step's LogFile.

want_stderr

like want_stdout but for stderr. Note that commands run through a PTY do not have separate stdout/stderr streams: both are merged into stdout.

usePTY

Should this command be run in a pty? The default is to observe the configuration of the client (see Buildslave Options), but specifying True or False here will override the default.

The advantage of using a PTY is that “grandchild” processes are more likely to be cleaned up if the build is interrupted or times out (since it enables the use of a “process group” in which all child processes will be placed). The disadvantages: some forms of Unix have problems with PTYs, some of your unit tests may behave differently when run under a PTY (generally those which check to see if they are being run interactively), and PTYs will merge the stdout and stderr streams into a single output stream (which means the red-vs-black coloring in the logfiles will be lost).

logfiles

Sometimes commands will log interesting data to a local file, rather than emitting everything to stdout or stderr. For example, Twisted's “trial” command (which runs unit tests) only presents summary information to stdout, and puts the rest into a file named _trial_temp/test.log. It is often useful to watch these files as the command runs, rather than using /bin/cat to dump their contents afterwards.

The logfiles= argument allows you to collect data from these secondary logfiles in near-real-time, as the step is running. It accepts a dictionary which maps from a local Log name (which is how the log data is presented in the build results) to either a remote filename (interpreted relative to the build's working directory), or a dictionary of options. Each named file will be polled on a regular basis (every couple of seconds) as the build runs, and any new text will be sent over to the buildmaster.

If you provide a dictionary of options instead of a string, you must specify the filename key. You can optionally provide a follow key which is a boolean controlling whether a logfile is followed or concatenated in its entirety. Following is appropriate for logfiles to which the build step will append, where the pre-existing contents are not interesting. The default value for follow is False, which gives the same behavior as just providing a string filename.

          f.addStep(ShellCommand(
                        command=["make", "test"],
                        logfiles={"triallog": "_trial_temp/test.log"}))

          f.addStep(ShellCommand(
                        command=["make", "test"],
                        logfiles={"triallog": {"filename": "_trial_temp/test.log",
          			       "follow": True,}}))

lazylogfiles

If set to True, logfiles will be tracked lazily, meaning that they will only be added when and if something is written to them. This can be used to suppress the display of empty or missing log files. The default is False.

timeout

if the command fails to produce any output for this many seconds, it is assumed to be locked up and will be killed.

maxTime

if the command takes longer than this many seconds, it will be killed.

description

This will be used to describe the command (on the Waterfall display) while the command is still running. It should be a single imperfect-tense verb, like “compiling” or “testing”. The preferred form is a list of short strings, which allows the HTML displays to create narrower columns by emitting a <br> tag between each word. You may also provide a single string.

descriptionDone

This will be used to describe the command once it has finished. A simple noun like “compile” or “tests” should be used. Like description, this may either be a list of short strings or a single string.

If neither description nor descriptionDone are set, the actual command arguments will be used to construct the description. This may be a bit too wide to fit comfortably on the Waterfall display.

          f.addStep(ShellCommand(command=["make", "test"],
                                 description=["testing"],
                                 descriptionDone=["tests"]))

logEnviron

If this option is true (the default), then the step's logfile will describe the environment variables on the slave. In situations where the environment is not relevant and is long, it may be easier to set logEnviron=False.