Caution

This page documents the latest, unreleased version of Buildbot. For documentation for released versions, see https://docs.buildbot.net/current/.

2.6.5. Customizing SVNPoller

Each source file that is tracked by a Subversion repository has a fully-qualified SVN URL in the following form: (REPOURL)(PROJECT-plus-BRANCH)(FILEPATH). When you create the SVNPoller, you give it a repourl value that includes all of the REPOURL and possibly some portion of the PROJECT-plus-BRANCH string. The SVNPoller is responsible for producing Changes that contain a branch name and a FILEPATH (which is relative to the top of a checked-out tree). The details of how these strings are split up depend upon how your repository names its branches.

2.6.5.1. PROJECT/BRANCHNAME/FILEPATH repositories

One common layout is to have all the various projects that share a repository get a single top-level directory each, with branches, tags, and trunk subdirectories:

amanda/trunk
      /branches/3_2
               /3_3
      /tags/3_2_1
           /3_2_2
           /3_3_0

To set up a SVNPoller that watches the Amanda trunk (and nothing else), we would use the following, using the default split_file:

from buildbot.plugins import changes
c['change_source'] = changes.SVNPoller(
   repourl="https://svn.amanda.sourceforge.net/svnroot/amanda/amanda/trunk")

In this case, every Change that our SVNPoller produces will have its branch attribute set to None, to indicate that the Change is on the trunk. No other sub-projects or branches will be tracked.

If we want our ChangeSource to follow multiple branches, we have to do two things. First we have to change our repourl= argument to watch more than just amanda/trunk. We will set it to amanda so that we’ll see both the trunk and all the branches. Second, we have to tell SVNPoller how to split the (PROJECT-plus-BRANCH)(FILEPATH) strings it gets from the repository out into (BRANCH) and (FILEPATH).

We do the latter by providing a split_file function. This function is responsible for splitting something like branches/3_3/common-src/amanda.h into branch='branches/3_3' and filepath='common-src/amanda.h'. The function is always given a string that names a file relative to the subdirectory pointed to by the SVNPoller's repourl= argument. It is expected to return a dictionary with at least the path key. The splitter may optionally set branch, project and repository. For backwards compatibility it may return a tuple of (branchname, path). It may also return None to indicate that the file is of no interest.

Note

The function should return branches/3_3 rather than just 3_3 because the SVN checkout step, will append the branch name to the baseURL, which requires that we keep the branches component in there. Other VC schemes use a different approach towards branches and may not require this artifact.

If your repository uses this same {PROJECT}/{BRANCH}/{FILEPATH} naming scheme, the following function will work:

def split_file_branches(path):
    pieces = path.split('/')
    if len(pieces) > 1 and pieces[0] == 'trunk':
        return (None, '/'.join(pieces[1:]))
    elif len(pieces) > 2 and pieces[0] == 'branches':
        return ('/'.join(pieces[0:2]),
                '/'.join(pieces[2:]))
    else:
        return None

In fact, this is the definition of the provided split_file_branches function. So to have our Twisted-watching SVNPoller follow multiple branches, we would use this:

from buildbot.plugins import changes, util
c['change_source'] = changes.SVNPoller("svn://svn.twistedmatrix.com/svn/Twisted",
                                       split_file=util.svn.split_file_branches)

Changes for all sorts of branches (with names like "branches/1.5.x", and None to indicate the trunk) will be delivered to the Schedulers. Each Scheduler is then free to use or ignore each branch as it sees fit.

If you have multiple projects in the same repository your split function can attach a project name to the Change to help the Scheduler filter out unwanted changes:

from buildbot.plugins import util
def split_file_projects_branches(path):
    if not "/" in path:
        return None
    project, path = path.split("/", 1)
    f = util.svn.split_file_branches(path)
    if f:
        info = {"project": project, "path": f[1]}
        if f[0]:
            info['branch'] = f[0]
        return info
    return f

Again, this is provided by default. To use it you would do this:

from buildbot.plugins import changes, util
c['change_source'] = changes.SVNPoller(
   repourl="https://svn.amanda.sourceforge.net/svnroot/amanda/",
   split_file=util.svn.split_file_projects_branches)

Note here that we are monitoring at the root of the repository, and that within that repository is a amanda subdirectory which in turn has trunk and branches. It is that amanda subdirectory whose name becomes the project field of the Change.

2.6.5.2. BRANCHNAME/PROJECT/FILEPATH repositories

Another common way to organize a Subversion repository is to put the branch name at the top, and the projects underneath. This is especially frequent when there are a number of related sub-projects that all get released in a group.

For example, Divmod.org hosts a project named Nevow as well as one named Quotient. In a checked-out Nevow tree there is a directory named formless that contains a Python source file named webform.py. This repository is accessible via webdav (and thus uses an http: scheme) through the divmod.org hostname. There are many branches in this repository, and they use a ({BRANCHNAME})/({PROJECT}) naming policy.

The fully-qualified SVN URL for the trunk version of webform.py is http://divmod.org/svn/Divmod/trunk/Nevow/formless/webform.py. The 1.5.x branch version of this file would have a URL of http://divmod.org/svn/Divmod/branches/1.5.x/Nevow/formless/webform.py. The whole Nevow trunk would be checked out with http://divmod.org/svn/Divmod/trunk/Nevow, while the Quotient trunk would be checked out using http://divmod.org/svn/Divmod/trunk/Quotient.

Now suppose we want to have an SVNPoller that only cares about the Nevow trunk. This case looks just like the PROJECT/BRANCH layout described earlier:

from buildbot.plugins import changes
c['change_source'] = changes.SVNPoller("http://divmod.org/svn/Divmod/trunk/Nevow")

But what happens when we want to track multiple Nevow branches? We have to point our repourl= high enough to see all those branches, but we also don’t want to include Quotient changes (since we’re only building Nevow). To accomplish this, we must rely upon the split_file function to help us tell the difference between files that belong to Nevow and those that belong to Quotient, as well as figuring out which branch each one is on.

from buildbot.plugins import changes
c['change_source'] = changes.SVNPoller("http://divmod.org/svn/Divmod",
                                       split_file=my_file_splitter)

The my_file_splitter function will be called with repository-relative pathnames like:

trunk/Nevow/formless/webform.py

This is a Nevow file, on the trunk. We want the Change that includes this to see a filename of formless/webform.py, and a branch of None

branches/1.5.x/Nevow/formless/webform.py

This is a Nevow file, on a branch. We want to get branch='branches/1.5.x' and filename='formless/webform.py'.

trunk/Quotient/setup.py

This is a Quotient file, so we want to ignore it by having my_file_splitter return None.

branches/1.5.x/Quotient/setup.py

This is also a Quotient file, which should be ignored.

The following definition for my_file_splitter will do the job:

def my_file_splitter(path):
    pieces = path.split('/')
    if pieces[0] == 'trunk':
        branch = None
        pieces.pop(0) # remove 'trunk'
    elif pieces[0] == 'branches':
        pieces.pop(0) # remove 'branches'
        # grab branch name
        branch = 'branches/' + pieces.pop(0)
    else:
        return None # something weird
    projectname = pieces.pop(0)
    if projectname != 'Nevow':
        return None # wrong project
    return {"branch": branch, "path": "/".join(pieces)}

If you later decide you want to get changes for Quotient as well you could replace the last 3 lines with simply:

return {"project": projectname, "branch": branch, "path": '/'.join(pieces)}