SVNPoller - BuildBot Manual

Next: Bzr Poller, Previous: BonsaiPoller, Up: Change Sources

4.6.8 SVNPoller

The buildbot.changes.svnpoller.SVNPoller is a ChangeSource which periodically polls a Subversion repository for new revisions, by running the svn log command in a subshell. It can watch a single branch or multiple branches.

SVNPoller accepts the following arguments:

svnurl

The base URL path to watch, like svn://svn.twistedmatrix.com/svn/Twisted/trunk, or http://divmod.org/svn/Divmod/, or even file:///home/svn/Repository/ProjectA/branches/1.5/. This must include the access scheme, the location of the repository (both the hostname for remote ones, and any additional directory names necessary to get to the repository), and the sub-path within the repository's virtual filesystem for the project and branch of interest.

The SVNPoller will only pay attention to files inside the subdirectory specified by the complete svnurl.

split_file

A function to convert pathnames into (branch, relative_pathname) tuples. Use this to explain your repository's branch-naming policy to SVNPoller. This function must accept a single string and return a two-entry tuple. There are a few utility functions in buildbot.changes.svnpoller that can be used as a split_file function; see below for details.

The default value always returns (None, path), which indicates that all files are on the trunk.

Subclasses of SVNPoller can override the split_file method instead of using the split_file= argument.

project

Set the name of the project to be used for the SVNPoller. This will then be set in any changes generated by the SVNPoller, and can be used in a Change Filter for triggering particular builders.

svnuser

An optional string parameter. If set, the --user argument will be added to all svn commands. Use this if you have to authenticate to the svn server before you can do svn info or svn log commands.

svnpasswd

Like svnuser, this will cause a --password argument to be passed to all svn commands.

pollinterval

How often to poll, in seconds. Defaults to 600 (checking once every 10 minutes). Lower this if you want the buildbot to notice changes faster, raise it if you want to reduce the network and CPU load on your svn server. Please be considerate of public SVN repositories by using a large interval when polling them.

histmax

The maximum number of changes to inspect at a time. Every POLLINTERVAL seconds, the SVNPoller asks for the last HISTMAX changes and looks through them for any ones it does not already know about. If more than HISTMAX revisions have been committed since the last poll, older changes will be silently ignored. Larger values of histmax will cause more time and memory to be consumed on each poll attempt. histmax defaults to 100.

svnbin

This controls the svn executable to use. If subversion is installed in a weird place on your system (outside of the buildmaster's $PATH), use this to tell SVNPoller where to find it. The default value of “svn” will almost always be sufficient.

revlinktmpl

This parameter allows a link to be provided for each revision (for example, to websvn or viewvc). These links appear anywhere changes are shown, such as on build or change pages. The proper form for this parameter is an URL with the portion that will substitute for a revision number replaced by ”%s”. For example, 'http://myserver/websvn/revision.php?rev=%s' could be used to cause revision links to be created to a websvn repository viewer.

cachepath

If specified, buildbot will cache processed revisions between restarts. This means you don't miss changes that were committed if the master is down for any reason.

Branches

Each source file that is tracked by a Subversion repository has a fully-qualified SVN URL in the following form: (REPOURL)(PROJECT-plus-BRANCH)(FILEPATH). When you create the SVNPoller, you give it a svnurl value that includes all of the REPOURL and possibly some portion of the PROJECT-plus-BRANCH string. The SVNPoller is responsible for producing Changes that contain a branch name and a FILEPATH (which is relative to the top of a checked-out tree). The details of how these strings are split up depend upon how your repository names its branches.

PROJECT/BRANCHNAME/FILEPATH repositories

One common layout is to have all the various projects that share a repository get a single top-level directory each. Then under a given project's directory, you get two subdirectories, one named “trunk” and another named “branches”. Under “branches” you have a bunch of other directories, one per branch, with names like “1.5.x” and “testing”. It is also common to see directories like “tags” and “releases” next to “branches” and “trunk”.

For example, the Twisted project has a subversion server on “svn.twistedmatrix.com” that hosts several sub-projects. The repository is available through a SCHEME of “svn:”. The primary sub-project is Twisted, of course, with a repository root of “svn://svn.twistedmatrix.com/svn/Twisted”. Another sub-project is Informant, with a root of “svn://svn.twistedmatrix.com/svn/Informant”, etc. Inside any checked-out Twisted tree, there is a file named bin/trial (which is used to run unit test suites).

The trunk for Twisted is in “svn://svn.twistedmatrix.com/svn/Twisted/trunk”, and the fully-qualified SVN URL for the trunk version of trial would be “svn://svn.twistedmatrix.com/svn/Twisted/trunk/bin/trial”. The same SVNURL for that file on a branch named “1.5.x” would be “svn://svn.twistedmatrix.com/svn/Twisted/branches/1.5.x/bin/trial”.

To set up a SVNPoller that watches the Twisted trunk (and nothing else), we would use the following:

     from buildbot.changes.svnpoller import SVNPoller
     c['change_source'] = SVNPoller("svn://svn.twistedmatrix.com/svn/Twisted/trunk")

In this case, every Change that our SVNPoller produces will have .branch=None, to indicate that the Change is on the trunk. No other sub-projects or branches will be tracked.

If we want our ChangeSource to follow multiple branches, we have to do two things. First we have to change our svnurl= argument to watch more than just “.../Twisted/trunk”. We will set it to “.../Twisted” so that we'll see both the trunk and all the branches. Second, we have to tell SVNPoller how to split the (PROJECT-plus-BRANCH)(FILEPATH) strings it gets from the repository out into (BRANCH) and (FILEPATH) pairs.

We do the latter by providing a “split_file” function. This function is responsible for splitting something like “branches/1.5.x/bin/trial” into branch=”branches/1.5.x” and filepath=”bin/trial”. This function is always given a string that names a file relative to the subdirectory pointed to by the SVNPoller's svnurl= argument. It is expected to return a (BRANCHNAME, FILEPATH) tuple (in which FILEPATH is relative to the branch indicated), or None to indicate that the file is outside any project of interest.

(note that we want to see “branches/1.5.x” rather than just “1.5.x” because when we perform the SVN checkout, we will probably append the branch name to the baseURL, which requires that we keep the “branches” component in there. Other VC schemes use a different approach towards branches and may not require this artifact.)

If your repository uses this same PROJECT/BRANCH/FILEPATH naming scheme, the following function will work:

     def split_file_branches(path):
         pieces = path.split('/')
         if pieces[0] == 'trunk':
             return (None, '/'.join(pieces[1:]))
         elif pieces[0] == 'branches':
             return ('/'.join(pieces[0:2]),
                     '/'.join(pieces[2:]))
         else:
             return None

This function is provided as buildbot.changes.svnpoller.split_file_branches for your convenience. So to have our Twisted-watching SVNPoller follow multiple branches, we would use this:

     from buildbot.changes.svnpoller import SVNPoller, split_file_branches
     c['change_source'] = SVNPoller("svn://svn.twistedmatrix.com/svn/Twisted",
                                    split_file=split_file_branches)

Changes for all sorts of branches (with names like “branches/1.5.x”, and None to indicate the trunk) will be delivered to the Schedulers. Each Scheduler is then free to use or ignore each branch as it sees fit.

BRANCHNAME/PROJECT/FILEPATH repositories

Another common way to organize a Subversion repository is to put the branch name at the top, and the projects underneath. This is especially frequent when there are a number of related sub-projects that all get released in a group.

For example, Divmod.org hosts a project named “Nevow” as well as one named “Quotient”. In a checked-out Nevow tree there is a directory named “formless” that contains a python source file named “webform.py”. This repository is accessible via webdav (and thus uses an “http:” scheme) through the divmod.org hostname. There are many branches in this repository, and they use a (BRANCHNAME)/(PROJECT) naming policy.

The fully-qualified SVN URL for the trunk version of webform.py is http://divmod.org/svn/Divmod/trunk/Nevow/formless/webform.py. You can do an svn co with that URL and get a copy of the latest version. The 1.5.x branch version of this file would have a URL of http://divmod.org/svn/Divmod/branches/1.5.x/Nevow/formless/webform.py. The whole Nevow trunk would be checked out with http://divmod.org/svn/Divmod/trunk/Nevow, while the Quotient trunk would be checked out using http://divmod.org/svn/Divmod/trunk/Quotient.

Now suppose we want to have an SVNPoller that only cares about the Nevow trunk. This case looks just like the PROJECT/BRANCH layout described earlier:

     from buildbot.changes.svnpoller import SVNPoller
     c['change_source'] = SVNPoller("http://divmod.org/svn/Divmod/trunk/Nevow")

But what happens when we want to track multiple Nevow branches? We have to point our svnurl= high enough to see all those branches, but we also don't want to include Quotient changes (since we're only building Nevow). To accomplish this, we must rely upon the split_file function to help us tell the difference between files that belong to Nevow and those that belong to Quotient, as well as figuring out which branch each one is on.

     from buildbot.changes.svnpoller import SVNPoller
     c['change_source'] = SVNPoller("http://divmod.org/svn/Divmod",
                                    split_file=my_file_splitter)

The my_file_splitter function will be called with repository-relative pathnames like:

trunk/Nevow/formless/webform.py: This is a Nevow file, on the trunk. We want the Change that includes this to see a filename of formless/webform.py", and a branch of None
branches/1.5.x/Nevow/formless/webform.py: This is a Nevow file, on a branch. We want to get branch=”branches/1.5.x” and filename=”formless/webform.py”.
trunk/Quotient/setup.py: This is a Quotient file, so we want to ignore it by having my_file_splitter return None.
branches/1.5.x/Quotient/setup.py: This is also a Quotient file, which should be ignored.

The following definition for my_file_splitter will do the job:

     def my_file_splitter(path):
         pieces = path.split('/')
         if pieces[0] == 'trunk':
             branch = None
             pieces.pop(0) # remove 'trunk'
         elif pieces[0] == 'branches':
             pieces.pop(0) # remove 'branches'
             # grab branch name
             branch = 'branches/' + pieces.pop(0)
         else:
             return None # something weird
         projectname = pieces.pop(0)
         if projectname != 'Nevow':
             return None # wrong project
         return (branch, '/'.join(pieces))