2.5.12.42. Transferring Files
- class buildbot.steps.transfer.FileUpload
- class buildbot.steps.transfer.FileDownload
Most of the work involved in a build will take place on the worker.
But occasionally it is useful to do some work on the buildmaster side.
The most basic way to involve the buildmaster is simply to move a file from the worker to the master, or vice versa.
There are a pair of steps named FileUpload
and FileDownload
to provide this functionality.
FileUpload
moves a file up to the master, while FileDownload
moves a file down from the master.
As an example, let’s assume that there is a step which produces an HTML file within the source tree that contains some sort of generated project documentation.
And let’s assume that we run nginx web server on the buildmaster host for serving static files.
We want to move this file to the buildmaster, into a /usr/share/nginx/www/
directory, so it can be visible to developers.
This file will wind up in the worker-side working directory under the name docs/reference.html
.
We want to put it into the master-side /usr/share/nginx/www/ref.html
, and add a link to the HTML status to the uploaded file.
from buildbot.plugins import steps
f.addStep(steps.ShellCommand(command=["make", "docs"]))
f.addStep(steps.FileUpload(workersrc="docs/reference.html",
masterdest="/usr/share/nginx/www/ref.html",
url="http://somesite/~buildbot/ref.html"))
The masterdest=
argument will be passed to os.path.expanduser
, so things like ~
will be expanded properly.
Non-absolute paths will be interpreted relative to the buildmaster’s base directory.
Likewise, the workersrc=
argument will be expanded and interpreted relative to the builder’s working directory.
Note
The copied file will have the same permissions on the master as on the worker, look at the mode=
parameter to set it differently.
To move a file from the master to the worker, use the FileDownload
command.
For example, let’s assume that some step requires a configuration file that, for whatever reason, could not be recorded in the source code repository or generated on the worker side:
from buildbot.plugins import steps
f.addStep(steps.FileDownload(mastersrc="~/todays_build_config.txt",
workerdest="build_config.txt"))
f.addStep(steps.ShellCommand(command=["make", "config"]))
Like FileUpload
, the mastersrc=
argument is interpreted relative to the buildmaster’s base directory, and the workerdest=
argument is relative to the builder’s working directory.
If the worker is running in ~worker
, and the builder’s builddir
is something like tests-i386
, then the workdir is going to be ~worker/tests-i386/build
, and a workerdest=
of foo/bar.html
will get put in ~worker/tests-i386/build/foo/bar.html
.
Both of these commands will create any missing intervening directories.
Other Parameters
The maxsize=
argument lets you set a maximum size for the file to be transferred.
This may help to avoid surprises: transferring a 100MB coredump when you were expecting to move a 10kB status file might take an awfully long time.
The blocksize=
argument controls how the file is sent over the network: larger blocksizes are slightly more efficient but also consume more memory on each end, and there is a hard-coded limit of about 640kB.
The mode=
argument allows you to control the access permissions of the target file, traditionally expressed as an octal integer.
The most common value is probably 0o755
, which sets the x executable bit on the file (useful for shell scripts and the like).
The default value for mode=
is None
, which means the permission bits will default to whatever the umask of the writing process is.
The default umask tends to be fairly restrictive, but at least on the worker you can make it less restrictive with a --umask
command-line option at creation time (Worker Options).
The keepstamp=
argument is a boolean that, when True
, forces the modified and accessed time of the destination file to match the times of the source file.
When False
(the default), the modified and accessed times of the destination file are set to the current time on the buildmaster.
The url=
argument allows you to specify an url that will be displayed in the HTML status.
The title of the url will be the name of the item transferred (directory for DirectoryUpload
or file for FileUpload
).
This allows the user to add a link to the uploaded item if that one is uploaded to an accessible place.
For FileUpload
, the urlText=
argument allows you to specify the url title that will be displayed in the web UI.
Transferring Directories
- class buildbot.steps.transfer.DirectoryUpload
To transfer complete directories from the worker to the master, there is a BuildStep
named DirectoryUpload
.
It works like FileUpload
, just for directories.
However it does not support the maxsize
, blocksize
and mode
arguments.
As an example, let’s assume an generated project documentation, which consists of many files (like the output of doxygen or epydoc).
And let’s assume that we run nginx web server on buildmaster host for serving static files.
We want to move the entire documentation to the buildmaster, into a /usr/share/nginx/www/docs
directory, and add a link to the uploaded documentation on the HTML status page.
On the worker-side the directory can be found under docs
:
from buildbot.plugins import steps
f.addStep(steps.ShellCommand(command=["make", "docs"]))
f.addStep(steps.DirectoryUpload(workersrc="docs",
masterdest="/usr/share/nginx/www/docs",
url="~buildbot/docs"))
The DirectoryUpload
step will create all necessary directories and transfers empty directories, too.
The maxsize
and blocksize
parameters are the same as for FileUpload
, although note that the size of the transferred data is implementation-dependent, and probably much larger than you expect due to the encoding used (currently tar).
The optional compress
argument can be given as 'gz'
or 'bz2'
to compress the datastream.
For DirectoryUpload
the urlText=
argument allows you to specify the url title that will be displayed in the web UI.
Note
The permissions on the copied files will be the same on the master as originally on the worker, see option buildbot-worker create-worker --umask
to change the default one.
Transferring Multiple Files At Once
- class buildbot.steps.transfer.MultipleFileUpload
In addition to the FileUpload
and DirectoryUpload
steps there is the MultipleFileUpload
step for uploading a bunch of files (and directories) in a single BuildStep
.
The step supports all arguments that are supported by FileUpload
and DirectoryUpload
, but instead of a the single workersrc
parameter it takes a (plural) workersrcs
parameter.
This parameter should either be a list, something that can be rendered as a list or a string which will be converted to a list.
Additionally it supports the glob
parameter if this parameter is set to True
all arguments in workersrcs
will be parsed through glob
and the results will be uploaded to masterdest
.:
from buildbot.plugins import steps
f.addStep(steps.ShellCommand(command=["make", "test"]))
f.addStep(steps.ShellCommand(command=["make", "docs"]))
f.addStep(steps.MultipleFileUpload(workersrcs=["docs", "test-results.html"],
masterdest="/usr/share/nginx/www/",
url="~buildbot"))
The url=
parameter, can be used to specify a link to be displayed in the HTML status of the step.
The way URLs are added to the step can be customized by extending the MultipleFileUpload
class.
The allUploadsDone method is called after all files have been uploaded and sets the URL.
The uploadDone method is called once for each uploaded file and can be used to create file-specific links.
import os
from buildbot.plugins import steps
class CustomFileUpload(steps.MultipleFileUpload):
linkTypes = ('.html', '.txt')
def linkFile(self, basename):
name, ext = os.path.splitext(basename)
return ext in self.linkTypes
def uploadDone(self, result, source, masterdest):
if self.url:
basename = os.path.basename(source)
if self.linkFile(basename):
self.addURL(self.url + '/' + basename, basename)
def allUploadsDone(self, result, sources, masterdest):
if self.url:
notLinked = [src for src in sources if not self.linkFile(src)]
numFiles = len(notLinked)
if numFiles:
self.addURL(self.url, '... %d more' % numFiles)
For MultipleFileUpload
the urlText=
argument allows you to specify the url title that will be displayed in the web UI.
2.5.12.43. Transferring Strings
- class buildbot.steps.transfer.StringDownload
- class buildbot.steps.transfer.JSONStringDownload
- class buildbot.steps.transfer.JSONPropertiesDownload
Sometimes it is useful to transfer a calculated value from the master to the worker. Instead of having to create a temporary file and then use FileDownload, you can use one of the string download steps.
from buildbot.plugins import steps, util
f.addStep(steps.StringDownload(util.Interpolate("%(src::branch)s-%(prop:got_revision)s\n"),
workerdest="buildid.txt"))
StringDownload
works just like FileDownload
except it takes a single argument, s
, representing the string to download instead of a mastersrc
argument.
from buildbot.plugins import steps
buildinfo = {
'branch': Property('branch'),
'got_revision': Property('got_revision')
}
f.addStep(steps.JSONStringDownload(buildinfo, workerdest="buildinfo.json"))
JSONStringDownload
is similar, except it takes an o
argument, which must be JSON serializable, and transfers that as a JSON-encoded string to the worker.
from buildbot.plugins import steps
f.addStep(steps.JSONPropertiesDownload(workerdest="build-properties.json"))
JSONPropertiesDownload
transfers a json-encoded string that represents a dictionary where properties maps to a dictionary of build property name
to property value
; and sourcestamp
represents the build’s sourcestamp.