4.11.6 Transferring Files

Most of the work involved in a build will take place on the buildslave. But occasionally it is useful to do some work on the buildmaster side. The most basic way to involve the buildmaster is simply to move a file from the slave to the master, or vice versa. There are a pair of BuildSteps named FileUpload and FileDownload to provide this functionality. FileUpload moves a file up to the master, while FileDownload moves a file down from the master.

As an example, let's assume that there is a step which produces an HTML file within the source tree that contains some sort of generated project documentation. We want to move this file to the buildmaster, into a ~/public_html directory, so it can be visible to developers. This file will wind up in the slave-side working directory under the name docs/reference.html. We want to put it into the master-side ~/public_html/ref.html.

     from buildbot.steps.shell import ShellCommand
     from buildbot.steps.transfer import FileUpload
     
     f.addStep(ShellCommand(command=["make", "docs"]))
     f.addStep(FileUpload(slavesrc="docs/reference.html",
                          masterdest="~/public_html/ref.html"))

The masterdest= argument will be passed to os.path.expanduser, so things like “~” will be expanded properly. Non-absolute paths will be interpreted relative to the buildmaster's base directory. Likewise, the slavesrc= argument will be expanded and interpreted relative to the builder's working directory.

To move a file from the master to the slave, use the FileDownload command. For example, let's assume that some step requires a configuration file that, for whatever reason, could not be recorded in the source code repository or generated on the buildslave side:

     from buildbot.steps.shell import ShellCommand
     from buildbot.steps.transfer import FileDownload
     
     f.addStep(FileDownload(mastersrc="~/todays_build_config.txt",
                            slavedest="build_config.txt"))
     f.addStep(ShellCommand(command=["make", "config"]))

Like FileUpload, the mastersrc= argument is interpreted relative to the buildmaster's base directory, and the slavedest= argument is relative to the builder's working directory. If the buildslave is running in ~buildslave, and the builder's “builddir” is something like tests-i386, then the workdir is going to be ~buildslave/tests-i386/build, and a slavedest= of foo/bar.html will get put in ~buildslave/tests-i386/build/foo/bar.html. Both of these commands will create any missing intervening directories.

Other Parameters

The maxsize= argument lets you set a maximum size for the file to be transferred. This may help to avoid surprises: transferring a 100MB coredump when you were expecting to move a 10kB status file might take an awfully long time. The blocksize= argument controls how the file is sent over the network: larger blocksizes are slightly more efficient but also consume more memory on each end, and there is a hard-coded limit of about 640kB.

The mode= argument allows you to control the access permissions of the target file, traditionally expressed as an octal integer. The most common value is probably 0755, which sets the “x” executable bit on the file (useful for shell scripts and the like). The default value for mode= is None, which means the permission bits will default to whatever the umask of the writing process is. The default umask tends to be fairly restrictive, but at least on the buildslave you can make it less restrictive with a –umask command-line option at creation time (see Buildslave Options).

Transfering Directories

To transfer complete directories from the buildslave to the master, there is a BuildStep named DirectoryUpload. It works like FileUpload, just for directories. As an example, let's assume a step has generated project documentation, which consists of many files (like the output of doxygen or epydoc). We want to move the entire documentation to the buildmaster, into a ~/public_html/docs directory. On the slave-side the directory can be found under docs:

     from buildbot.steps.shell import ShellCommand
     from buildbot.steps.transfer import DirectoryUpload
     
     f.addStep(ShellCommand(command=["make", "docs"]))
     f.addStep(DirectoryUpload(slavesrc="docs",
     				masterdest="~/public_html/docs"))

The DirectoryUpload step will create all necessary directories and transfers empty directories, too.

The maxsize and blocksize parameters are the same as for FileUpload, although note that the size of the transferred data is implementation-dependent, and probably much larger than you expect due to the encoding used (currently tar).

The optional compress argument can be given as 'gz' or 'bz2' to compress the datastream.