Caution
This page documents the latest, unreleased version of Buildbot. For documentation for released versions, see http://docs.buildbot.net/current/.
2.5.3. Change Sources and Changes
A change source is the mechanism which is used by Buildbot to get information about new changes in a repository maintained by a Version Control System.
These change sources fall broadly into two categories: pollers which periodically check the repository for updates; and hooks, where the repository is configured to notify Buildbot whenever an update occurs.
A Change
is an abstract way that Buildbot uses to represent changes in any of the Version Control Systems it supports.
It contains just enough information needed to acquire specific version of the tree when needed.
This usually happens as one of the first steps in a Build
.
This concept does not map perfectly to every version control system. For example, for CVS, Buildbot must guess that version updates made to multiple files within a short time represent a single change.
Change
s can be provided by a variety of ChangeSource
types, although any given project will typically have only a single ChangeSource
active.
2.5.3.1. How Different VC Systems Specify Sources
For CVS, the static specifications are repository and module.
In addition to those, each build uses a timestamp (or omits the timestamp to mean the latest) and branch tag (which defaults to HEAD
).
These parameters collectively specify a set of sources from which a build may be performed.
Subversion combines the repository, module, and branch into a single Subversion URL parameter.
Within that scope, source checkouts can be specified by a numeric revision number (a repository-wide monotonically-increasing marker, such that each transaction that changes the repository is indexed by a different revision number), or a revision timestamp.
When branches are used, the repository and module form a static baseURL
, while each build has a revision number and a branch (which defaults to a statically-specified defaultBranch
).
The baseURL
and branch
are simply concatenated together to derive the repourl
to use for the checkout.
Perforce is similar.
The server is specified through a P4PORT
parameter.
Module and branch are specified in a single depot path, and revisions are depot-wide.
When branches are used, the p4base
and defaultBranch
are concatenated together to produce the depot path.
Bzr (which is a descendant of Arch/Bazaar, and is frequently referred to as “Bazaar”) has the same sort of repository-vs-workspace model as Arch, but the repository data can either be stored inside the working directory or kept elsewhere (either on the same machine or on an entirely different machine). For the purposes of Buildbot (which never commits changes), the repository is specified with a URL and a revision number.
The most common way to obtain read-only access to a bzr tree is via HTTP, simply by making the repository visible through a web server like Apache.
Bzr can also use FTP and SFTP servers, if the worker process has sufficient privileges to access them.
Higher performance can be obtained by running a special Bazaar-specific server.
None of these matter to the buildbot: the repository URL just has to match the kind of server being used.
The repoURL
argument provides the location of the repository.
Branches are expressed as subdirectories of the main central repository, which means that if branches are being used, the BZR step is given a baseURL
and defaultBranch
instead of getting the repoURL
argument.
Darcs doesn’t really have the notion of a single master repository.
Nor does it really have branches.
In Darcs, each working directory is also a repository, and there are operations to push and pull patches from one of these repositories
to another.
For the Buildbot’s purposes, all you need to do is specify the URL of a repository that you want to build from.
The worker will then pull the latest patches from that repository and build them.
Multiple branches are implemented by using multiple repositories (possibly living on the same server).
Builders which use Darcs therefore have a static repourl
which specifies the location of the repository.
If branches are being used, the source Step is instead configured with a baseURL
and a defaultBranch
, and the two strings are simply concatenated together to obtain the repository’s URL.
Each build then has a specific branch which replaces defaultBranch
, or just uses the default one.
Instead of a revision number, each build can have a context
, which is a string that records all the patches that are present in a given tree (this is the output of darcs changes --context
, and is considerably less concise than, e.g. Subversion’s revision number, but the patch-reordering flexibility of Darcs makes it impossible to provide a shorter useful specification).
Mercurial follows a decentralized model, and each repository can have several branches and tags.
The source Step is configured with a static repourl
which specifies the location of the repository.
Branches are configured with the defaultBranch
argument.
The revision is the hash identifier returned by hg identify
.
Git also follows a decentralized model, and each repository can have several branches and tags.
The source Step is configured with a static repourl
which specifies the location of the repository.
In addition, an optional branch
parameter can be specified to check out code from a specific branch instead of the default master branch.
The revision is specified as a SHA1 hash as returned by e.g. git rev-parse
.
No attempt is made to ensure that the specified revision is actually a subset of the specified branch.
Monotone is another that follows a decentralized model where each repository can have several branches and tags.
The source Step is configured with static repourl
and branch
parameters, which specifies the location of the repository and the branch to use.
The revision is specified as a SHA1 hash as returned by e.g. mtn automate select w:
.
No attempt is made to ensure that the specified revision is actually a subset of the specified branch.
Comparison
Name |
Change |
Revision |
Branches |
---|---|---|---|
CVS |
patch [1] |
timestamp |
unnamed |
Subversion |
revision |
integer |
directories |
Git |
commit |
sha1 hash |
named refs |
Mercurial |
changeset |
sha1 hash |
different repos or (permanently) named commits |
Darcs |
? |
none [2] |
different repos |
Bazaar |
? |
? |
? |
Perforce |
? |
? |
? |
BitKeeper |
changeset |
? |
different repos |
[1] note that CVS only tracks patches to individual files. Buildbot tries to recognize coordinated changes to multiple files by correlating change times.
[2] Darcs does not have a concise way of representing a particular revision of the source.
Tree Stability
Changes tend to arrive at a buildmaster in bursts. In many cases, these bursts of changes are meant to be taken together. For example, a developer may have pushed multiple commits to a DVCS that comprise the same new feature or bugfix. To avoid trying to build every change, Buildbot supports the notion of tree stability, by waiting for a burst of changes to finish before starting to schedule builds. This is implemented as a timer, with builds not scheduled until no changes have occurred for the duration of the timer.
2.5.3.2. Choosing a Change Source
There are a variety of ChangeSource
classes available, some of which are meant to be used in conjunction with other tools to deliver Change
events from the VC repository to the buildmaster.
As a quick guide, here is a list of VC systems and the ChangeSource
s that might be useful with them.
Note that some of these modules are in Buildbot’s master/contrib directory, meaning that they have been offered by other users in hopes they may be useful, and might require some additional work to make them functional.
CVS
CVSMaildirSource
(watching mail sent by master/contrib/buildbot_cvs_mail.py script)PBChangeSource
(listening for connections frombuildbot sendchange
run in a loginfo script)PBChangeSource
(listening for connections from a long-running master/contrib/viewcvspoll.py polling process which examines the ViewCVS database directly)Change Hooks
in WebStatus
SVN
PBChangeSource
(listening for connections from master/contrib/svn_buildbot.py run in a postcommit script)PBChangeSource
(listening for connections from a long-running master/contrib/svn_watcher.py or master/contrib/svnpoller.py polling processSVNCommitEmailMaildirSource
(watching for email sent bycommit-email.pl
)SVNPoller
(polling the SVN repository)Change Hooks
in WebStatus
Darcs
PBChangeSource
(listening for connections from master/contrib/darcs_buildbot.py in a commit script)Change Hooks
in WebStatus
Mercurial
Change Hooks
in WebStatus (including master/contrib/hgbuildbot.py, configurable in achangegroup
hook)BitBucket change hook (specifically designed for BitBucket notifications, but requiring a publicly-accessible WebStatus)
HgPoller
(polling a remote Mercurial repository)BitbucketPullrequestPoller
(polling Bitbucket for pull requests)Mail-parsing ChangeSources, though there are no ready-to-use recipes
Bzr (the newer Bazaar)
PBChangeSource
(listening for connections from master/contrib/bzr_buildbot.py run in a post-change-branch-tip or commit hook)BzrPoller
(polling the Bzr repository)Change Hooks
in WebStatus
Git
PBChangeSource
(listening for connections from master/contrib/git_buildbot.py run in the post-receive hook)PBChangeSource
(listening for connections from master/contrib/github_buildbot.py, which listens for notifications from GitHub)Change Hooks
in WebStatusGitHub
change hook (specifically designed for GitHub notifications, but requiring a publicly-accessible WebStatus)BitBucket
change hook (specifically designed for BitBucket notifications, but requiring a publicly-accessible WebStatus)GitPoller
(polling a remote Git repository)GitHubPullrequestPoller
(polling GitHub API for pull requests)BitbucketPullrequestPoller
(polling Bitbucket for pull requests)
Repo/Gerrit
GerritChangeSource
connects to Gerrit via SSH and optionally HTTP to get a live stream of changesGerritEventLogPoller
connects to Gerrit via HTTP with the help of the plugin events-log
Monotone
PBChangeSource
(listening for connections frommonotone-buildbot.lua
, which is available with Monotone)
All VC systems can be driven by a PBChangeSource
and the buildbot sendchange
tool run from some form of commit script.
If you write an email parsing function, they can also all be driven by a suitable mail-parsing source.
Additionally, handlers for web-based notification (i.e. from GitHub) can be used with WebStatus’ change_hook module.
The interface is simple, so adding your own handlers (and sharing!) should be a breeze.
See Change Source Index for a full list of change sources.
2.5.3.3. Configuring Change Sources
The change_source
configuration key holds all active change sources for the configuration.
Most configurations have a single ChangeSource
, watching only a single tree, e.g.,
from buildbot.plugins import changes
c['change_source'] = changes.PBChangeSource()
For more advanced configurations, the parameter can be a list of change sources:
source1 = ...
source2 = ...
c['change_source'] = [
source1, source2
]
Repository and Project
ChangeSource
s will, in general, automatically provide the proper repository
attribute for any changes they produce.
For systems which operate on URL-like specifiers, this is a repository URL.
Other ChangeSource
s adapt the concept as necessary.
Many ChangeSource
s allow you to specify a project, as well.
This attribute is useful when building from several distinct codebases in the same buildmaster: the project string can serve to differentiate the different codebases.
Schedulers can filter on project, so you can configure different builders to run for each project.
2.5.3.4. Mail-parsing ChangeSources
Many projects publish information about changes to their source tree by sending an email message out to a mailing list, frequently named PROJECT-commits
or PROJECT-changes
.
Each message usually contains a description of the change (who made the change, which files were affected) and sometimes a copy of the diff.
Humans can subscribe to this list to stay informed about what’s happening to the source tree.
Buildbot can also subscribe to a -commits mailing list, and can trigger builds in response to Changes that it hears about. The buildmaster admin needs to arrange for these email messages to arrive in a place where the buildmaster can find them, and configure the buildmaster to parse the messages correctly. Once that is in place, the email parser will create Change objects and deliver them to the schedulers (see Schedulers) just like any other ChangeSource.
There are two components to setting up an email-based ChangeSource. The first is to route the email messages to the buildmaster, which is done by dropping them into a maildir. The second is to actually parse the messages, which is highly dependent upon the tool that was used to create them. Each VC system has a collection of favorite change-emailing tools with a slightly different format and its own parsing function. Buildbot has a separate ChangeSource variant for each of these parsing functions.
Once you’ve chosen a maildir location and a parsing function, create the change source and put it in change_source
:
from buildbot.plugins import changes
c['change_source'] = changes.CVSMaildirSource("~/maildir-buildbot",
prefix="/trunk/")
Subscribing the Buildmaster
The recommended way to install Buildbot is to create a dedicated account for the buildmaster. If you do this, the account will probably have a distinct email address (perhaps buildmaster@example.org). Then just arrange for this account’s email to be delivered to a suitable maildir (described in the next section).
If Buildbot does not have its own account, extension addresses can be used to distinguish between emails intended for the buildmaster and emails intended for the rest of the account.
In most modern MTAs, the e.g. foo@example.org account has control over every email address at example.org which begins with “foo”, such that emails addressed to account-foo@example.org can be delivered to a different destination than account-bar@example.org.
qmail does this by using separate .qmail
files for the two destinations (.qmail-foo
and .qmail-bar
, with .qmail
controlling the base address and .qmail-default
controlling all other extensions).
Other MTAs have similar mechanisms.
Thus you can assign an extension address like foo-buildmaster@example.org to the buildmaster and retain foo@example.org for your own use.
Using Maildirs
A maildir is a simple directory structure originally developed for qmail that allows safe atomic update without locking.
Create a base directory with three subdirectories: new
, tmp
, and cur
.
When messages arrive, they are put into a uniquely-named file (using pids, timestamps, and random numbers) in tmp
.
When the file is complete, it is atomically renamed into new
. Eventually the buildmaster notices the file in new
, reads and parses the contents, then moves it into cur
. A cronjob can be used to delete files in cur
at leisure.
Maildirs are frequently created with the maildirmake tool, but a simple mkdir -p ~/MAILDIR/cur,new,tmp
is pretty much equivalent.
Many modern MTAs can deliver directly to maildirs.
The usual .forward
or .procmailrc
syntax is to name the base directory with a trailing slash, so something like ~/MAILDIR/
.
qmail and postfix are maildir-capable MTAs, and procmail is a maildir-capable MDA (Mail Delivery Agent).
Here is an example procmail config, located in ~/.procmailrc
:
# .procmailrc
# routes incoming mail to appropriate mailboxes
PATH=/usr/bin:/usr/local/bin
MAILDIR=$HOME/Mail
LOGFILE=.procmail_log
SHELL=/bin/sh
:0
*
new
If procmail is not setup on a system wide basis, then the following one-line .forward
file will invoke it.
!/usr/bin/procmail
For MTAs which cannot put files into maildirs directly, the safecat tool can be executed from a .forward
file to accomplish the same thing.
The Buildmaster uses the linux DNotify facility to receive immediate notification when the maildir’s new
directory has changed.
When this facility is not available, it polls the directory for new messages, every 10 seconds by default.
Parsing Email Change Messages
The second component to setting up an email-based ChangeSource
is to parse the actual notices.
This is highly dependent upon the VC system and commit script in use.
A couple of common tools used to create these change emails, along with the Buildbot tools to parse them, are:
- CVS
- Buildbot CVS MailNotifier
- SVN
- svnmailer
commit-email.pl
- Bzr
- Launchpad
- Mercurial
- NotifyExtension
- Git
The following sections describe the parsers available for each of these tools.
Most of these parsers accept a prefix=
argument, which is used to limit the set of files that the buildmaster pays attention to.
This is most useful for systems like CVS and SVN which put multiple projects in a single repository (or use repository names to indicate branches).
Each filename that appears in the email is tested against the prefix: if the filename does not start with the prefix, the file is ignored.
If the filename does start with the prefix, that prefix is stripped from the filename before any further processing is done.
Thus the prefix usually ends with a slash.
CVSMaildirSource
- class buildbot.changes.mail.CVSMaildirSource
This parser works with the master/contrib/buildbot_cvs_mail.py script.
The script sends an email containing all the files submitted in one directory.
It is invoked by using the CVSROOT/loginfo
facility.
The Buildbot’s CVSMaildirSource
knows how to parse these messages and turn them into Change objects.
It takes the directory name of the maildir root.
For example:
from buildbot.plugins import changes
c['change_source'] = changes.CVSMaildirSource("/home/buildbot/Mail")
Configuration of CVS and buildbot_cvs_mail.py
CVS must be configured to invoke the buildbot_cvs_mail.py script when files are checked in. This is done via the CVS loginfo configuration file.
To update this, first do:
cvs checkout CVSROOT
cd to the CVSROOT directory and edit the file loginfo, adding a line like:
SomeModule /cvsroot/CVSROOT/buildbot_cvs_mail.py --cvsroot :ext:example.com:/cvsroot -e buildbot -P SomeModule %@{sVv@}
Note
For cvs version 1.12.x, the --path %p
option is required.
Version 1.11.x and 1.12.x report the directory path differently.
The above example you put the buildbot_cvs_mail.py script under /cvsroot/CVSROOT.
It can be anywhere.
Run the script with --help
to see all the options.
At the very least, the options -e
(email) and -P
(project) should be specified.
The line must end with %{sVv}
.
This is expanded to the files that were modified.
Additional entries can be added to support more modules.
See buildbot_cvs_mail.py --help for more information on the available options.
SVNCommitEmailMaildirSource
- class buildbot.changes.mail.SVNCommitEmailMaildirSource
SVNCommitEmailMaildirSource
parses message sent out by the commit-email.pl
script, which is included in the Subversion distribution.
It does not currently handle branches: all of the Change objects that it creates will be associated with the default (i.e. trunk) branch.
from buildbot.plugins import changes
c['change_source'] = changes.SVNCommitEmailMaildirSource("~/maildir-buildbot")
BzrLaunchpadEmailMaildirSource
- class buildbot.changes.mail.BzrLaunchpadEmailMaildirSource
BzrLaunchpadEmailMaildirSource
parses the mails that are sent to addresses that subscribe to branch revision notifications for a bzr branch hosted on Launchpad.
The branch name defaults to lp:Launchpad path
.
For example lp:~maria-captains/maria/5.1
.
If only a single branch is used, the default branch name can be changed by setting defaultBranch
.
For multiple branches, pass a dictionary as the value of the branchMap
option to map specific repository paths to specific branch names (see example below).
The leading lp:
prefix of the path is optional.
The prefix
option is not supported (it is silently ignored).
Use the branchMap
and defaultBranch
instead to assign changes to branches (and just do not subscribe the Buildbot to branches that are not of interest).
The revision number is obtained from the email text. The bzr revision id is not available in the mails sent by Launchpad. However, it is possible to set the bzr append_revisions_only option for public shared repositories to avoid new pushes of merges changing the meaning of old revision numbers.
from buildbot.plugins import changes
bm = {
'lp:~maria-captains/maria/5.1': '5.1',
'lp:~maria-captains/maria/6.0': '6.0'
}
c['change_source'] = changes.BzrLaunchpadEmailMaildirSource("~/maildir-buildbot",
branchMap=bm)
2.5.3.5. PBChangeSource
- class buildbot.changes.pb.PBChangeSource
PBChangeSource
actually listens on a TCP port for clients to connect and push change notices into the Buildmaster.
This is used by the built-in buildbot sendchange
notification tool, as well as several version-control hook scripts.
This change is also useful for creating new kinds of change sources that work on a push model instead of some kind of subscription scheme, for example a script which is run out of an email .forward
file.
This ChangeSource always runs on the same TCP port as the workers.
It shares the same protocol, and in fact shares the same space of “usernames”, so you cannot configure a PBChangeSource
with the same name as a worker.
If you have a publicly accessible worker port and are using PBChangeSource
, you must establish a secure username and password for the change source.
If your sendchange credentials are known (e.g., the defaults), then your buildmaster is susceptible to injection of arbitrary changes, which (depending on the build factories) could lead to arbitrary code execution on workers.
The PBChangeSource
is created with the following arguments.
port
Which port to listen on. If
None
(which is the default), it shares the port used for worker connections.user
The user account that the client program must use to connect. Defaults to
change
passwd
The password for the connection - defaults to
changepw
. Can be a Secret. Do not use this default on a publicly exposed port!prefix
The prefix to be found and stripped from filenames delivered over the connection, defaulting to
None
. Any filenames which do not start with this prefix will be removed. If all the filenames in a given Change are removed, then that whole Change will be dropped. This string should probably end with a directory separator.This is useful for changes coming from version control systems that represent branches as parent directories within the repository (like SVN and Perforce). Use a prefix of
trunk/
orproject/branches/foobranch/
to only follow one branch and to get correct tree-relative filenames. Without a prefix, thePBChangeSource
will probably deliver Changes with filenames liketrunk/foo.c
instead of justfoo.c
. Of course this also depends upon the tool sending the Changes in (likebuildbot sendchange
) and what filenames it is delivering: that tool may be filtering and stripping prefixes at the sending end.
For example:
from buildbot.plugins import changes
c['change_source'] = changes.PBChangeSource(port=9999, user='laura', passwd='fpga')
The following hooks are useful for sending changes to a PBChangeSource
:
Bzr Hook
Bzr is also written in Python, and the Bzr hook depends on Twisted to send the changes.
To install, put master/contrib/bzr_buildbot.py in one of your plugins locations a bzr plugins directory (e.g., ~/.bazaar/plugins
).
Then, in one of your bazaar conf files (e.g., ~/.bazaar/locations.conf
), set the location you want to connect with Buildbot with these keys:
buildbot_on
one of ‘commit’, ‘push, or ‘change’. Turns the plugin on to report changes via commit, changes via push, or any changes to the trunk. ‘change’ is recommended.
buildbot_server
(required to send to a Buildbot master) the URL of the Buildbot master to which you will connect (as of this writing, the same server and port to which workers connect).
buildbot_port
(optional, defaults to 9989) the port of the Buildbot master to which you will connect (as of this writing, the same server and port to which workers connect)
buildbot_pqm
(optional, defaults to not pqm) Normally, the user that commits the revision is the user that is responsible for the change. When run in a pqm (Patch Queue Manager, see https://launchpad.net/pqm) environment, the user that commits is the Patch Queue Manager, and the user that committed the parent revision is responsible for the change. To turn on the pqm mode, set this value to any of (case-insensitive) “Yes”, “Y”, “True”, or “T”.
buildbot_dry_run
(optional, defaults to not a dry run) Normally, the post-commit hook will attempt to communicate with the configured Buildbot server and port. If this parameter is included and any of (case-insensitive) “Yes”, “Y”, “True”, or “T”, then the hook will simply print what it would have sent, but not attempt to contact the Buildbot master.
buildbot_send_branch_name
(optional, defaults to not sending the branch name) If your Buildbot’s bzr source build step uses a repourl, do not turn this on. If your buildbot’s bzr build step uses a baseURL, then you may set this value to any of (case-insensitive) “Yes”, “Y”, “True”, or “T” to have the Buildbot master append the branch name to the baseURL.
Note
The bzr smart server (as of version 2.2.2) doesn’t know how to resolve bzr://
urls into absolute paths so any paths in locations.conf
won’t match, hence no change notifications will be sent to Buildbot.
Setting configuration parameters globally or in-branch might still work.
When Buildbot no longer has a hardcoded password, it will be a configuration option here as well.
Here’s a simple example that you might have in your ~/.bazaar/locations.conf
.
[chroot-*:///var/local/myrepo/mybranch]
buildbot_on = change
buildbot_server = localhost
2.5.3.6. P4Source
The P4Source
periodically polls a Perforce depot for changes.
It accepts the following arguments:
p4port
The Perforce server to connect to (as
host:port
).p4user
The Perforce user.
p4passwd
The Perforce password.
p4base
The base depot path to watch, without the trailing ‘/…’.
p4bin
An optional string parameter. Specify the location of the perforce command line binary (p4). You only need to do this if the perforce binary is not in the path of the Buildbot user. Defaults to p4.
split_file
A function that maps a pathname, without the leading
p4base
, to a (branch, filename) tuple. The default just returns(None, branchfile)
, which effectively disables branch support. You should supply a function which understands your repository structure.pollInterval
How often to poll, in seconds. Defaults to 600 (10 minutes).
pollRandomDelayMin
Minimum delay in seconds to wait before each poll, default is 0. This is useful in case you have a lot of pollers and you want to spread the polling load over a period of time. Setting it equal to the maximum delay will effectively delay all polls by a fixed amount of time. Must be less than or equal to the maximum delay.
pollRandomDelayMax
Maximum delay in seconds to wait before each poll, default is 0. This is useful in case you have a lot of pollers and you want to spread the polling load over a period of time. Must be less than the poll interval.
project
Set the name of the project to be used for the
P4Source
. This will then be set in any changes generated by theP4Source
, and can be used in a Change Filter for triggering particular builders.pollAtLaunch
Determines when the first poll occurs. True = immediately on launch, False = wait for one pollInterval (default).
histmax
The maximum number of changes to inspect at a time. If more than this number occur since the last poll, older changes will be silently ignored.
encoding
The character encoding of
p4
's output. This defaults to “utf8”, but if your commit messages are in another encoding, specify that here. For example, if you’re using Perforce on Windows, you may need to use “cp437” as the encoding if “utf8” generates errors in your master log.server_tz
The timezone of the Perforce server, using the usual timezone format (e.g:
"Europe/Stockholm"
) in case it’s not in UTC.use_tickets
Set to
True
to use ticket-based authentication, instead of passwords (but you still need to specifyp4passwd
).ticket_login_interval
How often to get a new ticket, in seconds, when
use_tickets
is enabled. Defaults to 86400 (24 hours).revlink
A function that maps branch and revision to a valid url (e.g. p4web), stored along with the change. This function must be a callable which takes two arguments, the branch and the revision. Defaults to lambda branch, revision: (u’’)
resolvewho
A function that resolves the Perforce ‘user@workspace’ into a more verbose form, stored as the author of the change. Useful when usernames do not match email addresses and external, client-side lookup is required. This function must be a callable which takes one argument. Defaults to lambda who: (who)
Example #1
This configuration uses the P4PORT
, P4USER
, and P4PASSWD
specified in the buildmaster’s environment.
It watches a project in which the branch name is simply the next path component, and the file is all path components after.
from buildbot.plugins import changes
s = changes.P4Source(p4base='//depot/project/',
split_file=lambda branchfile: branchfile.split('/',1))
c['change_source'] = s
Example #2
Similar to the previous example but also resolves the branch and revision into a valid revlink.
from buildbot.plugins import changes
s = changes.P4Source(
p4base='//depot/project/',
split_file=lambda branchfile: branchfile.split('/',1))
revlink=lambda branch, revision: 'http://p4web:8080/@md=d&@/{}?ac=10'.format(revision)
c['change_source'] = s
2.5.3.7. SVNPoller
- class buildbot.changes.svnpoller.SVNPoller
The SVNPoller
is a ChangeSource which periodically polls a Subversion repository for new revisions, by running the svn log
command in a subshell.
It can watch a single branch or multiple branches.
SVNPoller
accepts the following arguments:
repourl
The base URL path to watch, like
svn://svn.twistedmatrix.com/svn/Twisted/trunk
, orhttp://divmod.org/svn/Divmo/
, or evenfile:///home/svn/Repository/ProjectA/branches/1.5/
. This must include the access scheme, the location of the repository (both the hostname for remote ones, and any additional directory names necessary to get to the repository), and the sub-path within the repository’s virtual filesystem for the project and branch of interest.The
SVNPoller
will only pay attention to files inside the subdirectory specified by the complete repourl.split_file
A function to convert pathnames into
(branch, relative_pathname)
tuples. Use this to explain your repository’s branch-naming policy toSVNPoller
. This function must accept a single string (the pathname relative to the repository) and return a two-entry tuple. Directory pathnames always end with a right slash to distinguish them from files, liketrunk/src/
, orsrc/
. There are a few utility functions inbuildbot.changes.svnpoller
that can be used as asplit_file
function; see below for details.For directories, the relative pathname returned by
split_file
should end with a right slash but an empty string is also accepted for the root, like("branches/1.5.x", "")
being converted from"branches/1.5.x/"
.The default value always returns
(None, path)
, which indicates that all files are on the trunk.Subclasses of
SVNPoller
can override thesplit_file
method instead of using thesplit_file=
argument.project
Set the name of the project to be used for the
SVNPoller
. This will then be set in any changes generated by theSVNPoller
, and can be used in a Change Filter for triggering particular builders.svnuser
An optional string parameter. If set, the option –user argument will be added to all svn commands. Use this if you have to authenticate to the svn server before you can do svn info or svn log commands. Can be a Secret.
svnpasswd
Like
svnuser
, this will cause a option –password argument to be passed to all svn commands. Can be a Secret.pollInterval
How often to poll, in seconds. Defaults to 600 (checking once every 10 minutes). Lower this if you want the Buildbot to notice changes faster, raise it if you want to reduce the network and CPU load on your svn server. Please be considerate of public SVN repositories by using a large interval when polling them.
pollRandomDelayMin
Minimum delay in seconds to wait before each poll, default is 0. This is useful in case you have a lot of pollers and you want to spread the polling load over a period of time. Setting it equal to the maximum delay will effectively delay all polls by a fixed amount of time. Must be less than or equal to the maximum delay.
pollRandomDelayMax
Maximum delay in seconds to wait before each poll, default is 0. This is useful in case you have a lot of pollers and you want to spread the polling load over a period of time. Must be less than the poll interval.
pollAtLaunch
Determines when the first poll occurs. True = immediately on launch, False = wait for one pollInterval (default).
histmax
The maximum number of changes to inspect at a time. Every
pollInterval
seconds, theSVNPoller
asks for the lasthistmax
changes and looks through them for any revisions it does not already know about. If more thanhistmax
revisions have been committed since the last poll, older changes will be silently ignored. Larger values ofhistmax
will cause more time and memory to be consumed on each poll attempt.histmax
defaults to 100.svnbin
This controls the svn executable to use. If subversion is installed in a weird place on your system (outside of the buildmaster’s
PATH
), use this to tellSVNPoller
where to find it. The default value of svn will almost always be sufficient.revlinktmpl
This parameter is deprecated in favour of specifying a global revlink option. This parameter allows a link to be provided for each revision (for example, to websvn or viewvc). These links appear anywhere changes are shown, such as on build or change pages. The proper form for this parameter is an URL with the portion that will substitute for a revision number replaced by ‘’%s’’. For example,
'http://myserver/websvn/revision.php?rev=%s'
could be used to cause revision links to be created to a websvn repository viewer.cachepath
If specified, this is a pathname of a cache file that
SVNPoller
will use to store its state between restarts of the master.extra_args
If specified, the extra arguments will be added to the svn command args.
Several split file functions are available for common SVN repository layouts.
For a poller that is only monitoring trunk, the default split file function is available explicitly as split_file_alwaystrunk
:
from buildbot.plugins import changes, util
c['change_source'] = changes.SVNPoller(
repourl="svn://svn.twistedmatrix.com/svn/Twisted/trunk",
split_file=util.svn.split_file_alwaystrunk)
For repositories with the /trunk
and /branches/BRANCH
layout, split_file_branches
will do the job:
from buildbot.plugins import changes, util
c['change_source'] = changes.SVNPoller(
repourl="https://amanda.svn.sourceforge.net/svnroot/amanda/amanda",
split_file=util.svn.split_file_branches)
When using this splitter the poller will set the project
attribute of any changes to the project
attribute of the poller.
For repositories with the PROJECT/trunk
and PROJECT/branches/BRANCH
layout, split_file_projects_branches
will do the job:
from buildbot.plugins import changes, util
c['change_source'] = changes.SVNPoller(
repourl="https://amanda.svn.sourceforge.net/svnroot/amanda/",
split_file=util.svn.split_file_projects_branches)
When using this splitter the poller will set the project
attribute of any changes to the project determined by the splitter.
The SVNPoller
is highly adaptable to various Subversion layouts.
See Customizing SVNPoller for details and some common scenarios.
2.5.3.8. Bzr Poller
If you cannot insert a Bzr hook in the server, you can use the BzrPoller
.
To use it, put master/contrib/bzr_buildbot.py somewhere that your Buildbot configuration can import it.
Even putting it in the same directory as the master.cfg
should work.
Install the poller in the Buildbot configuration as with any other change source.
Minimally, provide a URL that you want to poll (bzr://
, bzr+ssh://
, or lp:
), making sure the Buildbot user has necessary privileges.
# put bzr_buildbot.py file to the same directory as master.cfg
from bzr_buildbot import BzrPoller
c['change_source'] = BzrPoller(
url='bzr://hostname/my_project',
poll_interval=300)
The BzrPoller
parameters are:
url
The URL to poll.
poll_interval
The number of seconds to wait between polls. Defaults to 10 minutes.
branch_name
Any value to be used as the branch name. Defaults to None, or specify a string, or specify the constants from bzr_buildbot.py
SHORT
orFULL
to get the short branch name or full branch address.blame_merge_author
Normally, the user that commits the revision is the user that is responsible for the change. When run in a pqm (Patch Queue Manager, see https://launchpad.net/pqm) environment, the user that commits is the Patch Queue Manager, and the user that committed the merged, parent revision is responsible for the change. Set this value to
True
if this is pointed against a PQM-managed branch.
2.5.3.9. GitPoller
If you cannot take advantage of post-receive hooks as provided by master/contrib/git_buildbot.py for example, then you can use the GitPoller
.
The GitPoller
periodically fetches from a remote Git repository and processes any changes.
It requires its own working directory for operation.
The default should be adequate, but it can be overridden via the workdir
property.
Note
There can only be a single GitPoller pointed at any given repository.
The GitPoller
requires Git-1.7 and later.
It accepts the following arguments:
repourl
The git-url that describes the remote repository, e.g.
git@example.com:foobaz/myrepo.git
(see the git fetch help for more info on git-url formats)branches
One of the following:
a list of the branches to fetch. Non-existing branches are ignored.
True
indicating that all branches should be fetcheda callable which takes a single argument. It should take a remote refspec (such as
'refs/heads/master'
), and return a boolean indicating whether that branch should be fetched.
If not provided,
GitPoller
will useHEAD
to fetch the remote default branch.branch
Accepts a single branch name to fetch. Exists for backwards compatibility with old configurations.
pollInterval
Interval in seconds between polls, default is 10 minutes
pollRandomDelayMin
Minimum delay in seconds to wait before each poll, default is 0. This is useful in case you have a lot of pollers and you want to spread the polling load over a period of time. Setting it equal to the maximum delay will effectively delay all polls by a fixed amount of time. Must be less than or equal to the maximum delay.
pollRandomDelayMax
Maximum delay in seconds to wait before each poll, default is 0. This is useful in case you have a lot of pollers and you want to spread the polling load over a period of time. Must be less than the poll interval.
pollAtLaunch
Determines when the first poll occurs. True = immediately on launch, False = wait for one pollInterval (default).
buildPushesWithNoCommits
Determines if a push on a new branch or update of an already known branch with already known commits should trigger a build. This is useful in case you have build steps depending on the name of the branch and you use topic branches for development. When you merge your topic branch into “master” (for instance), a new build will be triggered. (defaults to False).
gitbin
Path to the Git binary, defaults to just
'git'
category
Set the category to be used for the changes produced by the
GitPoller
. This will then be set in any changes generated by theGitPoller
, and can be used in a Change Filter for triggering particular builders.project
Set the name of the project to be used for the
GitPoller
. This will then be set in any changes generated by theGitPoller
, and can be used in a Change Filter for triggering particular builders.usetimestamps
Parse each revision’s commit timestamp (default is
True
), or ignore it in favor of the current time, so that recently processed commits appear together in the waterfall page.encoding
Set encoding will be used to parse author’s name and commit message. Default encoding is
'utf-8'
. This will not be applied to file names since Git will translate non-ascii file names to unreadable escape sequences.workdir
The directory where the poller should keep its local repository. The default is
gitpoller_work
. If this is a relative path, it will be interpreted relative to the master’s basedir. Multiple Git pollers can share the same directory.only_tags
Determines if the GitPoller should poll for new tags in the git repository.
sshPrivateKey
(optional)Specifies private SSH key for git to use. This may be either a Secret or just a string. This option requires Git-2.3 or later. The master must either have the host in the known hosts file or the host key must be specified via the sshHostKey option.
sshHostKey
(optional)Specifies public host key to match when authenticating with SSH public key authentication. This may be either a Secret or just a string. sshPrivateKey must be specified in order to use this option. The host key must be in the form of <key type> <base64-encoded string>, e.g. ssh-rsa AAAAB3N<…>FAaQ==.
sshKnownHosts
(optional)Specifies the contents of the SSH known_hosts file to match when authenticating with SSH public key authentication. This may be either a Secret or just a string. sshPrivateKey must be specified in order to use this option. sshHostKey must not be specified in order to use this option.
auth_credentials
(optional) An username/password tuple to use when running git for fetch operations. The worker’s git version needs to be at least 1.7.9.
git_credentials
(optional) See GitCredentialOptions. The worker’s git version needs to be at least 1.7.9.
A configuration for the Git poller might look like this:
from buildbot.plugins import changes
c['change_source'] = changes.GitPoller(repourl='git@example.com:foobaz/myrepo.git',
branches=['master', 'great_new_feature'])
2.5.3.10. HgPoller
The HgPoller
periodically pulls a named branch from a remote Mercurial repository and processes any changes.
It requires its own working directory for operation, which must be specified via the workdir
property.
The HgPoller
requires a working hg
executable, and at least a read-only access to the repository it polls (possibly through ssh keys or by tweaking the hgrc
of the system user Buildbot runs as).
The HgPoller
will not transmit any change if there are several heads on the watched named branch.
This is similar (although not identical) to the Mercurial executable behaviour.
This exceptional condition is usually the result of a developer mistake, and usually does not last for long.
It is reported in logs.
If fixed by a later merge, the buildmaster administrator does not have anything to do: that merge will be transmitted, together with the intermediate ones.
The HgPoller
accepts the following arguments:
name
The name of the poller. This must be unique, and defaults to the
repourl
.repourl
The url that describes the remote repository, e.g.
http://hg.example.com/projects/myrepo
. Any url suitable forhg pull
can be specified.bookmarks
A list of the bookmarks to monitor.
branches
A list of the branches to monitor; defaults to
['default']
.branch
The desired branch to pull. Exists for backwards compatibility with old configurations.
workdir
The directory where the poller should keep its local repository. It is mandatory for now, although later releases may provide a meaningful default.
It also serves to identify the poller in the buildmaster internal database. Changing it may result in re-processing all changes so far.
Several
HgPoller
instances may share the sameworkdir
for mutualisation of the common history between two different branches, thus easing on local and remote system resources and bandwidth.If relative, the
workdir
will be interpreted from the master directory.pollInterval
Interval in seconds between polls, default is 10 minutes
pollRandomDelayMin
Minimum delay in seconds to wait before each poll, default is 0. This is useful in case you have a lot of pollers and you want to spread the polling load over a period of time. Setting it equal to the maximum delay will effectively delay all polls by a fixed amount of time. Must be less than or equal to the maximum delay.
pollRandomDelayMax
Maximum delay in seconds to wait before each poll, default is 0. This is useful in case you have a lot of pollers and you want to spread the polling load over a period of time. Must be less than the poll interval.
pollAtLaunch
Determines when the first poll occurs. True = immediately on launch, False = wait for one pollInterval (default).
hgbin
Path to the Mercurial binary, defaults to just
'hg'
.category
Set the category to be used for the changes produced by the
HgPoller
. This will then be set in any changes generated by theHgPoller
, and can be used in a Change Filter for triggering particular builders.project
Set the name of the project to be used for the
HgPoller
. This will then be set in any changes generated by theHgPoller
, and can be used in a Change Filter for triggering particular builders.usetimestamps
Parse each revision’s commit timestamp (default is
True
), or ignore it in favor of the current time, so that recently processed commits appear together in the waterfall page.encoding
Set encoding will be used to parse author’s name and commit message. Default encoding is
'utf-8'
.revlink
A function that maps branch and revision to a valid url (e.g. hgweb), stored along with the change. This function must be a callable which takes two arguments, the branch and the revision. Defaults to lambda branch, revision: (u’’)
A configuration for the Mercurial poller might look like this:
from buildbot.plugins import changes
c['change_source'] = changes.HgPoller(repourl='http://hg.example.org/projects/myrepo',
branch='great_new_feature',
workdir='hg-myrepo')
2.5.3.11. GitHubPullrequestPoller
- class buildbot.changes.github.GitHubPullrequestPoller
This GitHubPullrequestPoller
periodically polls the GitHub API for new or updated pull requests.
The author, revision, revlink, branch and files fields in the recorded changes are populated with information extracted from the pull request.
This allows to filter for certain changes in files and create a blamelist based on the authors in the GitHub pull request.
The GitHubPullrequestPoller
accepts the following arguments:
owner
The owner of the GitHub repository. This argument is required.
repo
The name of the GitHub repository. This argument is required.
branches
List of branches to accept as base branch (e.g. master). Defaults to None and accepts all branches as base.
pollInterval
Poll interval between polls in seconds. Default is 10 minutes.
pollAtLaunch
Whether to poll on startup of the buildbot master. Default is False and first poll will occur pollInterval seconds after the master start.
category
Set the category to be used for the changes produced by the
GitHubPullrequestPoller
. This will then be set in any changes generated by theGitHubPullrequestPoller
, and can be used in a Change Filter for triggering particular builders.project
Set the name of the project to be used for the
GitHubPullrequestPoller
. This will then be set in any changes generated by theGitHubPullrequestPoller
, and can be used in a Change Filter for triggering particular builders. If unset, the default is to use the full name of the project as returned by the GitHub API.baseURL
GitHub API endpoint. Default is
https://api.github.com
.pullrequest_filter
A callable which takes a dict which contains the decoded JSON object of the GitHub pull request as argument. All fields specified by the GitHub API are accessible. If the callable returns False the pull request is ignored. Default is True which does not filter any pull requests.
token
A GitHub API token to execute all requests to the API authenticated. It is strongly recommended to use a API token since it increases GitHub API rate limits significantly.
repository_type
Set which type of repository link will be in the repository property. Possible values
https
,svn
,git
orsvn
. This link can then be used in a Source Step to checkout the source.magic_link
Set to True if the changes should contain
refs/pulls/<PR #>/merge
in the branch property and a link to the base repository in the repository property. These properties can be used by theGitHub
source to pull from the special branch in the base repository. Default is False.github_property_whitelist
A list of
fnmatch
expressions which match against the flattened pull request information JSON prefixed withgithub
. For examplegithub.number
represents the pull request number. Available entries can be looked up in the GitHub API Documentation or by examining the data returned for a pull request by the API.
2.5.3.12. BitbucketPullrequestPoller
- class buildbot.changes.bitbucket.BitbucketPullrequestPoller
This BitbucketPullrequestPoller
periodically polls Bitbucket for new or updated pull requests.
It uses Bitbuckets powerful Pull Request REST API to gather the information needed.
The BitbucketPullrequestPoller
accepts the following arguments:
owner
The owner of the Bitbucket repository. All Bitbucket Urls are of the form
https://bitbucket.org/owner/slug/
.slug
The name of the Bitbucket repository.
auth
Authorization data tuple
(usename, password)
(optional). If set, it will be used as authorization headers at Bitbucket API.branch
A single branch or a list of branches which should be processed. If it is
None
(the default) all pull requests are used.pollInterval
Interval in seconds between polls, default is 10 minutes.
pollAtLaunch
Determines when the first poll occurs.
True
= immediately on launch,False
= wait for onepollInterval
(default).category
Set the category to be used by the
BitbucketPullrequestPoller
. This will then be set in any changes generated by theBitbucketPullrequestPoller
, and can be used in a Change Filter for triggering particular builders.project
Set the name of the project to be used by the
BitbucketPullrequestPoller
. This will then be set in any changes generated by theBitbucketPullrequestPoller
, and can be used in a Change Filter for triggering particular builders.pullrequest_filter
A callable which takes one parameter, the decoded Python object of the pull request JSON. If it returns
False
, the pull request is ignored. It can be used to define custom filters based on the content of the pull request. See the Bitbucket documentation for more information about the format of the response. By default, the filter always returnsTrue
.usetimestamps
Parse each revision’s commit timestamp (default is
True
), or ignore it in favor of the current time, so that recently processed commits appear together in the waterfall page.bitbucket_property_whitelist
A list of
fnmatch
expressions which match against the flattened pull request information JSON prefixed withbitbucket
. For examplebitbucket.id
represents the pull request ID. Available entries can be looked up in the BitBucket API Documentation or by examining the data returned for a pull request by the API.encoding
This parameter is deprecated and has no effects. Author’s name and commit message are always parsed in
'utf-8'
.
A minimal configuration for the Bitbucket pull request poller might look like this:
from buildbot.plugins import changes
c['change_source'] = changes.BitbucketPullrequestPoller(
owner='myname',
slug='myrepo',
)
Here is a more complex configuration using a pullrequest_filter
.
The pull request is only processed if at least 3 people have already approved it:
def approve_filter(pr, threshold):
approves = 0
for participant in pr['participants']:
if participant['approved']:
approves = approves + 1
if approves < threshold:
return False
return True
from buildbot.plugins import changes
c['change_source'] = changes.BitbucketPullrequestPoller(
owner='myname',
slug='myrepo',
branch='mybranch',
project='myproject',
pullrequest_filter=lambda pr : approve_filter(pr,3),
pollInterval=600,
)
Warning
Anyone who can create pull requests for the Bitbucket repository can initiate a change, potentially causing the buildmaster to run arbitrary code.
2.5.3.13. GerritChangeSource
- class buildbot.changes.gerritchangesource.GerritChangeSource
The GerritChangeSource
class connects to a Gerrit server by its SSH interface and uses its event source mechanism, gerrit stream-events.
Optionally it may use the events-log plugin to retrieve any events that occur while Buildbot is not connected. If events-log mechanism is not used any events that occur while buildbot is not connected to Gerrit will be lost.
Note
The GerritChangeSource
requires either the txrequest
or the treq
package for
using the HTTP API.
The GerritChangeSource
accepts the following arguments:
gerritserver
The dns or ip that host the Gerrit ssh server
gerritport
The port of the Gerrit ssh server
username
The username to use to connect to Gerrit
identity_file
Ssh identity file to for authentication (optional). Pay attention to the ssh passphrase
handled_events
Event to be handled (optional). By default processes patchset-created and ref-updated
get_files
Populate the files attribute of emitted changes (default False). Buildbot will run an extra query command for each handled event to determine the changed files.
ssh_server_alive_interval_s
Sets the
ServerAliveInterval
option of the ssh client (default 15). This causes client to emit periodic keepalive messages in case the connection is not otherwise active. If the server does not respond at leastssh_server_alive_count_max
times, a reconnection is forced. This helps to avoid stuck connections in case network link is severed without notification in the TCP layer. SpecifyingNone
will omit the option from the ssh client command line.ssh_server_alive_count_max
Sets the
ServerAliveCountMax
option of the ssh client (default 3). If the server does not respond at leastssh_server_alive_count_max
times, a reconnection is forced. This helps to avoid stuck connections in case network link is severed without notification in the TCP layer. SpecifyingNone
will omit the option from the ssh client command line.http_url
(optional) HTTP URL to use when fetching events from the Gerrit internal database. This is used to fill in events that have occurred when Buildbot was not connected to the SSH API. If the URL of the events-log endpoint for your server is
https://example.com/a/plugins/events-log/events/
then thehttp_url
ishttps://example.com
.http_auth
(optional) authentication credentials for events-log plugin. If Gerrit is configured with
BasicAuth
, then it shall be('login', 'password')
. If Gerrit is configured withDigestAuth
, then it shall berequests.auth.HTTPDigestAuth('login', 'password')
from the requests module. However, note that usage ofrequests.auth.HTTPDigestAuth
is incompatible withtreq
.http_poll_interval
(optional) frequency to poll the HTTP API when events are not being received through the SSH connection. The default is 30 seconds.
debug
Print Gerrit event in the log (default False). This allows to debug event content, but will eventually fill your logs with useless Gerrit event logs.
By default this class adds a change to the Buildbot system for each of the following events:
patchset-created
A change is proposed for review. Automatic checks like
checkpatch.pl
can be automatically triggered. Beware of what kind of automatic task you trigger. At this point, no trusted human has reviewed the code, and a patch could be specially crafted by an attacker to compromise your workers.ref-updated
A change has been merged into the repository. Typically, this kind of event can lead to a complete rebuild of the project, and upload binaries to an incremental build results server.
But you can specify how to handle events:
Any event with change and patchSet will be processed by universal collector by default.
In case you’ve specified processing function for the given kind of events, all events of this kind will be processed only by this function, bypassing universal collector.
An example:
from buildbot.plugins import changes
class MyGerritChangeSource(changes.GerritChangeSource):
"""Custom GerritChangeSource
"""
def eventReceived_patchset_created(self, properties, event):
"""Handler events without properties
"""
properties = {}
self.addChangeFromEvent(properties, event)
This class will populate the property list of the triggered build with the info received from Gerrit server in JSON format.
Warning
If you selected GerritChangeSource
, you must use Gerrit
source step: the branch
property of the change will be target_branch/change_id
and such a ref cannot be resolved, so the Git
source step would fail.
In case of patchset-created
event, these properties will be:
event.change.branch
Branch of the Change
event.change.id
Change’s ID in the Gerrit system (the ChangeId: in commit comments)
event.change.number
Change’s number in Gerrit system
event.change.owner.email
Change’s owner email (owner is first uploader)
event.change.owner.name
Change’s owner name
event.change.project
Project of the Change
event.change.subject
Change’s subject
event.change.url
URL of the Change in the Gerrit’s web interface
event.patchSet.number
Patchset’s version number
event.patchSet.ref
Patchset’s Gerrit “virtual branch”
event.patchSet.revision
Patchset’s Git commit ID
event.patchSet.uploader.email
Patchset uploader’s email (owner is first uploader)
event.patchSet.uploader.name
Patchset uploader’s name (owner is first uploader)
event.type
Event type (
patchset-created
)event.uploader.email
Patchset uploader’s email
event.uploader.name
Patchset uploader’s name
In case of ref-updated
event, these properties will be:
event.refUpdate.newRev
New Git commit ID (after merger)
event.refUpdate.oldRev
Previous Git commit ID (before merger)
event.refUpdate.project
Project that was updated
event.refUpdate.refName
Branch that was updated
event.submitter.email
Submitter’s email (merger responsible)
event.submitter.name
Submitter’s name (merger responsible)
event.type
Event type (
ref-updated
)event.submitter.email
Submitter’s email (merger responsible)
event.submitter.name
Submitter’s name (merger responsible)
A configuration for this source might look like:
from buildbot.plugins import changes
c['change_source'] = changes.GerritChangeSource(
"gerrit.example.com",
"gerrit_user",
handled_events=["patchset-created", "change-merged"])
See master/docs/examples/git_gerrit.cfg
or master/docs/examples/repo_gerrit.cfg
in the Buildbot distribution for a full example setup of Git+Gerrit or Repo+Gerrit of GerritChangeSource
.
2.5.3.14. GerritEventLogPoller
- class buildbot.changes.gerritchangesource.GerritEventLogPoller
The GerritEventLogPoller
class is similar to GerritChangeSource
and
connects to the Gerrit server only by its HTTP interface and uses the events-log plugin.
Note
The GerritEventLogPoller
requires either the txrequest
or the treq
package.
The GerritEventLogPoller
accepts the following arguments:
baseURL
The HTTP url where to find Gerrit. If the URL of the events-log endpoint for your server is
https://example.com/a/plugins/events-log/events/
then thebaseURL
ishttps://example.com/a
. Ensure that/a
is included.auth
A request’s authentication configuration. If Gerrit is configured with
BasicAuth
, then it shall be('login', 'password')
. If Gerrit is configured withDigestAuth
, then it shall berequests.auth.HTTPDigestAuth('login', 'password')
from the requests module. However, note that usage ofrequests.auth.HTTPDigestAuth
is incompatible withtreq
.handled_events
Event to be handled (optional). By default processes patchset-created and ref-updated.
pollInterval
Interval in seconds between polls (default is 30 sec).
pollAtLaunch
Determines when the first poll occurs. True = immediately on launch (default), False = wait for one pollInterval.
gitBaseURL
The git URL where Gerrit is accessible via git+ssh protocol.
get_files
Populate the files attribute of emitted changes (default False). Buildbot will run an extra query command for each handled event to determine the changed files.
debug
Print Gerrit event in the log (default False). This allows to debug event content, but will eventually fill your logs with useless Gerrit event logs.
The same customization can be done as GerritChangeSource
for handling special events.
2.5.3.15. GerritChangeFilter
- class buildbot.changes.gerritchangesource.GerritChangeFilter
GerritChangeFilter
is a ready to use ChangeFilter
you can pass to AnyBranchScheduler
in order to filter changes, to create pre-commit builders or post-commit schedulers.
It has the same api as Change Filter, except it has additional eventtype set of filter (can as well be specified as value, list, regular expression, or callable).
An example is following:
from buildbot.plugins import schedulers, util
# this scheduler will create builds when a patch is uploaded to gerrit
# but only if it is uploaded to the "main" branch
schedulers.AnyBranchScheduler(
name="main-precommit",
change_filter=util.GerritChangeFilter(branch="main", eventtype="patchset-created"),
treeStableTimer=15*60,
builderNames=["main-precommit"])
# this scheduler will create builds when a patch is merged in the "main" branch
# for post-commit tests
schedulers.AnyBranchScheduler(name="main-postcommit",
change_filter=util.GerritChangeFilter("main", "ref-updated"),
treeStableTimer=15*60,
builderNames=["main-postcommit"])
2.5.3.16. Change Hooks (HTTP Notifications)
Buildbot already provides a web frontend, and that frontend can easily be used to receive HTTP push notifications of commits from services like GitHub. See Change Hooks for more information.
2.5.4. Changes
- class buildbot.changes.changes.Change
A Change
is an abstract way Buildbot uses to represent a single change to the source files performed by a developer.
In version control systems that support the notion of atomic check-ins, a change represents a changeset or commit.
Instances of Change
have the following attributes.
2.5.4.1. Who
Each Change
has a who
attribute, which specifies which developer is responsible for the change.
This is a string which comes from a namespace controlled by the VC repository.
Frequently this means it is a username on the host which runs the repository, but not all VC systems require this.
Each StatusNotifier
will map the who
attribute into something appropriate for their particular means of communication: an email address, an IRC handle, etc.
This who
attribute is also parsed and stored into Buildbot’s database (see User Objects).
Currently, only who
attributes in Changes from git
repositories are translated into user objects, but in the future all incoming Changes will have their who
parsed and stored.
2.5.4.2. Files
It also has a list of files
, which are just the tree-relative filenames of any files that were added, deleted, or modified for this Change
.
These filenames are checked by the fileIsImportant
function of a scheduler to decide whether it should trigger a new build or not.
For example, the scheduler could use the following function to only run a build if a C file was checked in:
def has_C_files(change):
for name in change.files:
if name.endswith(".c"):
return True
return False
Certain BuildStep
s can also use the list of changed files to run a more targeted series of tests, e.g. the python_twisted.Trial
step can run just the unit tests that provide coverage for the modified .py files instead of running the full test suite.
2.5.4.4. Project
The project
attribute of a change or source stamp describes the project to which it corresponds, as a short human-readable string.
This is useful in cases where multiple independent projects are built on the same buildmaster.
In such cases, it can be used to control which builds are scheduled for a given commit, and to limit status displays to only one project.
2.5.4.5. Repository
This attribute specifies the repository in which this change occurred. In the case of DVCS’s, this information may be required to check out the committed source code. However, using the repository from a change has security risks: if Buildbot is configured to blindly trust this information, then it may easily be tricked into building arbitrary source code, potentially compromising the workers and the integrity of subsequent builds.
2.5.4.6. Codebase
This attribute specifies the codebase to which this change was made.
As described in source stamps section, multiple repositories may contain the same codebase.
A change’s codebase is usually determined by the codebaseGenerator
configuration.
By default the codebase is ‘’; this value is used automatically for single-codebase configurations.
2.5.4.7. Revision
Each Change can have a revision
attribute, which describes how to get a tree with a specific state: a tree which includes this Change (and all that came before it) but none that come after it.
If this information is unavailable, the revision
attribute will be None
.
These revisions are provided by the ChangeSource
.
Revisions are always strings.
- CVS
revision
is the seconds since the epoch as an integer.- SVN
revision
is the revision number- Darcs
revision
is a large string, the output of darcs changes --context- Mercurial
revision
is a short string (a hash ID), the output of hg identify- P4
revision
is the transaction number- Git
revision
is a short string (a SHA1 hash), the output of e.g. git rev-parse
2.5.4.8. Branches
The Change might also have a branch
attribute.
This indicates that all of the Change’s files are in the same named branch.
The schedulers get to decide whether the branch should be built or not.
For VC systems like CVS, Git, Mercurial and Monotone the branch
name is unrelated to the filename.
(That is, the branch name and the filename inhabit unrelated namespaces.)
For SVN, branches are expressed as subdirectories of the repository, so the file’s repourl
is a combination of some base URL, the branch name, and the filename within the branch.
(In a sense, the branch name and the filename inhabit the same namespace.)
Darcs branches are subdirectories of a base URL just like SVN.
- CVS
branch=’warner-newfeature’, files=[‘src/foo.c’]
- SVN
branch=’branches/warner-newfeature’, files=[‘src/foo.c’]
- Darcs
branch=’warner-newfeature’, files=[‘src/foo.c’]
- Mercurial
branch=’warner-newfeature’, files=[‘src/foo.c’]
- Git
branch=’warner-newfeature’, files=[‘src/foo.c’]
- Monotone
branch=’warner-newfeature’, files=[‘src/foo.c’]
2.5.4.9. Change Properties
A Change may have one or more properties attached to it, usually specified through the Force Build form or sendchange
.
Properties are discussed in detail in the Build Properties section.
2.5.4.3. Comments
The Change also has a
comments
attribute, which is a string containing any checkin comments.