5.2. Release Notes for Buildbot 0.9.0
¶
The following are the release notes for Buildbot 0.9.0
.
This version was released on October 6, 2016.
This is a concatenation of important changes done between 0.8.12 and 0.9.0. This does not contain details of the bug fixes related to the nine beta and rc period. This document was written during the very long period of nine development. It might contain some incoherencies, please help us and report them on irc or trac.
See Upgrading to Nine for a guide to upgrading from 0.8.x to 0.9.x
5.2.1. Master¶
This version represents a refactoring of Buildbot into a consistent, well-defined application composed of loosely coupled components. The components are linked by a common database backend and a messaging system. This allows components to be distributed across multiple build masters. It also allows the rendering of complex web status views to be performed in the browser, rather than on the buildmasters.
The branch looks forward to committing to long-term API compatibility, but does not reach that goal. The Buildbot-0.9.x series of releases will give the new APIs time to “settle in” before we commit to them. Commitment will wait for Buildbot-1.0.0 (as per http://semver.org). Once Buildbot reaches version 1.0.0, upgrades will become much easier for users.
To encourage contributions from a wider field of developers, the web application is designed to look like a normal AngularJS application. Developers familiar with AngularJS, but not with Python, should be able to start hacking on the web application quickly. The web application is “pluggable”, so users who develop their own status displays can package those separately from Buildbot itself.
Other goals:
- An approachable HTTP REST API, with real time event features used by the web application but available for any other purpose.
- A high degree of coverage by reliable, easily-modified tests.
- “Interlocking” tests to guarantee compatibility. For example, the real and fake DB implementations must both pass the same suite of tests. Then no unseen difference between the fake and real implementations can mask errors that will occur in production.
5.2.1.1. Requirements¶
The buildbot
package requires Python 2.7 – Python 2.5 and 2.6 are no longer supported.
The buildbot-slave
package requires Python 2.6 or higher – Python 2.4 and 2.5 are no longer supported.
No additional software or systems, aside from some minor Python packages, are required.
But the devil is in the details:
- If you want to do web development, or build the
buildbot-www
package, you’ll need Node. It’s an Angular app, and that’s how such apps are developed. We’ve taken pains to not make either a requirement for users - you can simply ‘pip install’buildbot-www
and be on your way. This is the case even if you’re hacking on the Python side of Buildbot.- For a single master, nothing else is required.
Note for distro package maintainers: The npm dependency hell¶
In order to build the buildbot-www
package, you’ll need Node.
Node has a very specific package manager named npm, which has the interesting property of allowing different version of package to co-exist in the same environment. The node ecosystem also has the habit of creating packages for a few line of code.
Buildbot UI uses the node ecosystem to build its javascript UI.
The buildsystem that we use is called guanlecoja, which is just an integration of various javascript build tools.
Through npm dependency hell, guanlecoja is depending on 625 npm packages/versions. We do not advise you to try and package all those npm build dependencies. They are not required in order to run buildbot.
We do release pre-built packages in the form of the wheel format on pypi. Those wheels contain the full python source code, and prebuilt javascript source code.
Depending on distro maintainers feedback, we could also release source tarballs with prebuilt javascript, but those would be pypi packages with different names, e.g. buildbot_www_prebuilt.0.9.0.tar.gz
.
Another option would be to package a guanlecoja that would embed all its dependencies inside one package.
Detailed requirements¶
see Requirements
5.2.1.2. Features¶
Buildbot-0.9.0 introduces the Data API, a consistent and scalable method for accessing and updating the state of the Buildbot system. This API replaces the existing, ill-defined Status API, which has been removed. Buildbot-0.9.0 introduces new WWW Server Interface using websocket for realtime updates. Buildbot code that interacted with the Status API (a substantial portion!) has been rewritten to use the Data API. Individual features and improvements to the Data API are not described on this page.
Buildbot now supports plugins. They allow Buildbot to be extended by using components distributed independently from the main code. They also provide for a unified way to access all components. When previously the following construction was used:
from buildbot.kind.other.bits import ComponentClass ... ComponentClass ...
the following construction achieves the same result:
from buildbot.plugins import kind ... kind.ComponentClass ...
Kinds of components that are available this way are described in Plugin Infrastructure in Buildbot.
Note
While the components can be still directly imported as
buildbot.kind.other.bits
, this might not be the case after Buildbot v1.0 is released.Both the P4 source step and P4 change source support ticket-based authentication.
OpenStack latent slaves now support block devices as a bootable volume.
Add new
Cppcheck
step.Add a new Docker latent Workers.
Add a new configuration for creating custom services in out-of-tree CI systems or plugins. See
buildbot.util.service.BuildbotService
Add
try_ssh
configuration file setting and--ssh
command line option for the try tool to specify the command to use for connecting to the build master.GitHub change hook now supports application/json format.
Add support for dynamically adding steps during a build. See Dynamic Build Factories.
GitPoller
now supports detecting new branchesGit
supports an “origin” option to give a name to the remote repo.Mercurial hook was updated and modernized. It is no longer necessary to fork. One can now extend PYTHONPATH via the hook configuration. Among others, it permits to use a buildbot virtualenv instead of installing buildbot in all the system. Added documentation inside the hook. Misc. clean-up and reorganization in order to make the code a bit more readable.
UI templates can now be customizable. You can provide html or jade overrides to the www plugins, to customize the UI
The irc command
hello
now returns ‘Hello’ in a random language if invoked more than once.Triggerable
now accepts areason
parameter.GerritStatusPush
now accepts abuilders
parameter.StatusPush callback now receives build results (success/failure/etc) with the
buildFinished
event.There’s a new renderable type, Transform.
GitPoller
now has abuildPushesWithNoCommits
option to allow the rebuild of already known commits on new branches.Add GitLab authentication plugin for web UI. See
buildbot.www.oauth2.GitLabAuth
.CMake
build step is added. It provides a convenience interface to CMake build system.MySQL InnoDB tables are now supported.
HttpStatusPush
has been ported to reporter API.StashStatusPush
has been ported to reporter API.GithubStatusPush
has been ported to reporter API.summaryCB of
GerritStatusPush
now gets not only pre-processed information but the actual build as well.EC2LatentWorker supports VPCs, instance profiles, and advanced volume mounts.
New steps for Visual Studio 2015 (VS2015, VC14, and MsBuild14).
The
P4
step now obfuscates the password in status logs.Added support for specifying the depth of a shallow clone in
Git
.OpenStackLatentWorker
now uses a single novaclient instance to not require re-authentication when starting or stopping instances.Buildbot UI introduces branch new Authentication, and Authorizations framework.
Please look at their respective guide in WWW Server
buildbot stop
now waits for complete buildmaster stop by default.New
--no-wait
argument forbuildbot stop
which allows not to wait for complete master shutdown.New
LocalWorker
worker to run a worker in the master process, requiresbuildbot-worker
package installed.GerritStatusPush
now includes build properties in thestartCB
andreviewCB
functions.startCB
now must return a dictionary.add tool to send usage data to buildbot.net
buildbotNetUsageData
new
GitHub
which correctly checkout the magic branch likerefs/pull/xx/merge
.Enable parallel builds with Visual Studio and MSBuild.
5.2.1.3. Reporters¶
Status plugins have been moved into the reporters
namespace.
Their API has slightly to changed in order to adapt to the new data API.
See respective documentation for details.
GerritStatusPush
renamed toGerritStatusPush
MailNotifier
renamed toMailNotifier
MailNotifier
argumentmessageFormatter
should now be aMessageFormatter
, due to removal of data api, custom message formatters need to be rewritten.MailNotifier
argumentpreviousBuildGetter
is not supported anymoreGerrit
supports specifying an SSH identity file explicitly.- Added StashStatusPush status hook for Atlassian Stash
MailNotifier
no longer forces SSL 3.0 whenuseTls
is true.GerritStatusPush
callbacks slightly changed signature, and include a master reference instead of a status reference.- new
GitLabStatusPush
to report builds results to GitLab. - new
HipchatStatusPush
to report build results to Hipchat.
5.2.1.4. Fixes¶
Buildbot is now compatible with SQLAlchemy 0.8 and higher, using the newly-released SQLAlchemy-Migrate.
The version check for SQLAlchemy-Migrate was fixed to accept more version string formats.
The
HTTPStep
step’s request parameters are now renderable.With Git(), force the updating submodules to ensure local changes by the build are overwritten. This both ensures more consistent builds and avoids errors when updating submodules.
Buildbot is now compatible with Gerrit v2.6 and higher.
To make this happen, the return result of
reviewCB
andsummaryCB
callback has changed from(message, verified, review)
to
{'message': message, 'labels': {'label-name': value, ... } }
The implications are:
- there are some differences in behaviour: only those labels that were provided will be updated
- Gerrit server must be able to provide a version, if it can’t the
GerritStatusPush
will not work
Note
If you have an old style
reviewCB
and/orsummaryCB
implemented, these will still work, however there could be more labels updated than anticipated.More detailed information is available in
GerritStatusPush
section.P4Source
’sserver_tz
parameter now works correctly.The
revlink
in changes produced by the Bitbucket hook now correctly includes thechanges/
portion of the URL.PBChangeSource
’s git hook master/contrib/git_buildbot.py now supports git tagsA pushed git tag generates a change event with the
branch
property equal to the tag name. To schedule builds based on buildbot tags, one could use something like this:c['schedulers'].append( SingleBranchScheduler(name='tags', change_filter=filter.ChangeFilter( branch_re='v[0-9]+\.[0-9]+\.[0-9]+(?:-pre|rc[0-9]+|p[0-9]+)?') treeStableTimer=None, builderNames=['tag_build']))
Missing “name” and “email” properties received from Gerrit are now handled properly
Fixed bug which made it impossible to specify the project when using the BitBucket dialect.
The
PyLint
step has been updated to understand newer output.Fixed SVN master-side source step: if a SVN operation fails, the repository end up in a situation when a manual intervention is required. Now if SVN reports such a situation during initial check, the checkout will be clobbered.
The build properties are now stored in the database in the
build_properties
table.The list of changes in the build page now displays all the changes since the last successful build.
GitHub change hook now correctly responds to ping events.
GitHub change hook now correctly use the refs/pull/xx/merge branch for testing PRs.
buildbot.steps.http
steps now correctly haveurl
parameter renderableWhen no arguments are used
buildbot checkconfig
now usesbuildbot.tac
to locate the master config file.buildbot.util.flatten now correctly flattens arbitrarily nested lists. buildbot.util.flattened_iterator provides an iterable over the collection which may be more efficient for extremely large lists.
The
PyFlakes
andPyLint
steps no longer parse output in Buildbot log headers (bug #3337).GerritChangeSource
is now less verbose by default, and has adebug
option to enable the logs.P4Source
no longer relies on the perforce server time to poll for new changes.The commit message for a change from
P4Source
now matches what the user typed in.Fix incompatibility with MySQL-5.7 (bug #3421)
Fix incompatibility with postgresql driver psycopg2 (bug #3419, further regressions will be caught by travis)
Made
Interpolate
safe for deepcopy or serialization/deserializationsqlite access is serialized in order to improve stability (bug #3565)
5.2.1.5. Deprecations, Removals, and Non-Compatible Changes¶
Seamless upgrading between buildbot 0.8.12 and buildbot 0.9.0 is not supported. Users should start from a clean install but can reuse their config according to the Upgrading to Nine guide.
BonsaiPoller is removed.
buildbot.ec2buildslave
is removed; usebuildbot.buildslave.ec2
instead.buildbot.libvirtbuildslave
is removed; usebuildbot.buildslave.libvirt
instead.buildbot.util.flatten flattens lists and tuples by default (previously only lists). Additionally, flattening something that isn’t the type to flatten has different behaviour. Previously, it would return the original value. Instead, it now returns an array with the original value as the sole element.
buildbot.tac
does not supportprint
statements anymore. Such files should now useprint
as a function instead (see https://docs.python.org/3.0/whatsnew/3.0.html#print-is-a-function for more details). Note that this applies to both python2.x and python3.x runtimes.Deprecated
workdir
property has been removed,builddir
property should be used instead.To support MySQL InnoDB, the size of six VARCHAR(256) columns
changes.(author, branch, category, name); object_state.name; user.identifier
was reduced to VARCHAR(255).StatusPush
has been removed from buildbot.Please use the much simpler
HttpStatusPush
instead.
Worker changes described in below worker section will probably impact a buildbot developer who uses undocumented ‘slave’ API. Undocumented APIs have been replaced without failover, so any custom code that uses it shall be updated with new undocumented API.
Web server does not provide /png and /redirect anymore (bug #3357). This functionality is used to implement build status images. This should be easy to implement if you need it. One could port the old image generation code, or implement a redirection to http://shields.io/.
Support of worker-side
usePTY
was removed frombuildbot-worker
.usePTY
argument was removed fromWorkerForBuilder
andWorker
classes.html is no longer permitted in ‘label’ attributes of forcescheduler parameters.
public_html
directory is not created anymore inbuildbot create-master
(it’s not used for some time already). Documentation was updated with suggestions to use third party web server for serving static file.usePTY
default value has been changed fromslave-config
toNone
(use ofslave-config
will still work)./json
web status was removed. Data API should be used instead.
WebStatus¶
The old, clunky WebStatus has been removed. You will like the new interface! RIP WebStatus, you were a good friend.
remove it and replace it with www configuration
.
Requirements¶
- Support for python 2.6 was dropped from the master.
- Buildbot’s tests now require at least Mock-0.8.0.
- SQLAlchemy-Migrate-0.6.1 is no longer supported.
- Builder names are now restricted to unicode strings or ASCII bytestrings. Encoded bytestrings are not accepted.
Steps¶
- New-style steps are now the norm, and support for old-style steps is deprecated.
Upgrade your steps to new-style now, as support for old-style steps will be dropped after Buildbot-0.9.0.
See New-Style Build Steps for details.
- Status strings for old-style steps could be supplied through a wide variety of conflicting means (
describe
,description
,descriptionDone
,descriptionSuffix
,getText
, andsetText
, to name just a few). While all attempts have been made to maintain compatibility, you may find that the status strings for old-style steps have changed in this version. To fix steps that callsetText
, try setting thedescriptionDone
attribute directly, instead – or just rewrite the step in the new style.
- Status strings for old-style steps could be supplied through a wide variety of conflicting means (
- Old-style source steps (imported directly from
buildbot.steps.source
) are no longer supported on the master. - The monotone source step got an overhaul and can now better manage its database (initialize and/or migrate it, if needed). In the spirit of monotone, buildbot now always keeps the database around, as it’s an append-only database.
Changes and Removals¶
Buildslave names must now be 50-character identifier. Note that this disallows some common characters in bulidslave names, including spaces,
/
, and.
.Builders now have “tags” instead of a category. Builders can have multiple tags, allowing more flexible builder displays.
ForceScheduler
has the following changes:- The default configuration no longer contains four
AnyPropertyParameter
instances. - Configuring
codebases
is now mandatory, and the deprecatedbranch
,repository
,project
,revision
are not supported anymore inForceScheduler
buildbot.schedulers.forcesched.BaseParameter.updateFromKwargs
now takes acollector
parameter used to collect all validation errors
- The default configuration no longer contains four
Periodic
,Nightly
andNightlyTriggerable
have the following changes:Logs are now stored as Unicode strings, and thus must be decoded properly from the bytestrings provided by shell commands. By default this encoding is assumed to be UTF-8, but the
logEncoding
parameter can be used to select an alternative. Steps and individual logfiles can also override the global default.The PB status service uses classes which have now been removed, and anyway is redundant to the REST API, so it has been removed. It has taken the following with it:
buildbot statuslog
buildbot statusgui
(the GTK client)buildbot debugclient
The
PBListener
status listener is now deprecated and does nothing. Accordingly, there is no external access to status objects via Perspective Broker, aside from some compatibility code for the try scheduler.The
debugPassword
configuration option is no longer needed and is thus deprecated.The undocumented and un-tested
TinderboxMailNotifier
, designed to send emails suitable for the abandoned and insecure Tinderbox tool, has been removed.Buildslave info is no longer available via Interpolate and the
SetSlaveInfo
buildstep has been removed.The undocumented
path
parameter of theMasterShellCommand
buildstep has been renamedworkdir
for better consistency with the other steps.The name and source of a Property have to be unicode or ascii string.
Property values must be serializable in JSON.
IRC
has the following changes:- categories parameter is deprecated and removed. It should be replaced with tags=[cat]
- noticeOnChannel parameter is deprecated and removed.
workdir behavior has been unified:
workdir
attribute of steps is now a property inBuildStep
, and choose the workdir given following priority:- workdir of the step, if defined
- workdir of the builder (itself defaults to ‘build’)
- setDefaultWorkdir() has been deprecated, but is now behaving the same for all the steps: Setting self.workdir if not already set
Trigger
now has agetSchedulersAndProperties
method that can ve overridden to support dynamic triggering.`master.cfg
is now parsed from a thread. Previously it was run in the main thread, and thus slowing down the master in case of big config, or network access done to generate the config.SVNPoller
’s svnurl parameter has been changed to repourl.Providing Latent AWS EC2 credentials by the
.ec2/aws_id
file is deprecated: Use the standard.aws/credentials
file, instead.
5.2.1.6. Changes for Developers¶
Botmaster no longer service parent for workers. Service parent functionality has been transferred to WorkerManager. It should be noted Botmaster no longer has a
slaves
field as it was moved to WorkerManager.The sourcestamp DB connector now returns a
patchid
field.Buildbot no longer polls the database for jobs. The
db_poll_interval
configuration parameter and thedb
key of the same name are deprecated and will be ignored.The interface for adding changes has changed. The new method is
master.data.updates.addChange
(implemented byaddChange
), although the old interface (master.addChange
) will remain in place for a few versions. The new method:- returns a change ID, not a Change instance;
- takes its
when_timestamp
argument as epoch time (UNIX time), not a datetime instance; and - does not accept the deprecated parameters
who
,isdir
,is_dir
, andwhen
. - requires that all strings be unicode, not bytestrings.
Please adjust any custom change sources accordingly.
A new build status, CANCELLED, has been added. It is used when a step or build is deliberately cancelled by a user.
This upgrade will delete all rows from the
buildrequest_claims
table. If you are using this table for analytical purposes outside of Buildbot, please back up its contents before the upgrade, and restore it afterward, translating object IDs to scheduler IDs if necessary. This translation would be very slow and is not required for most users, so it is not done automatically.All of the schedulers DB API methods now accept a schedulerid, rather than an objectid. If you have custom code using these methods, check your code and make the necessary adjustments.
The
addBuildsetForSourceStamp
method has becomeaddBuildsetForSourceStamps
, and its signature has changed. TheaddBuildsetForSourceStampSetDetails
method has becomeaddBuildsetForSourceStampsWithDefaults
, and its signature has changed. TheaddBuildsetForSourceStampDetails
method has been removed. TheaddBuildsetForLatest
method has been removed. It is equivalent toaddBuildsetForSourceStampDetails
withsourcestamps=None
. These methods are not yet documented, and their interface is not stable. Consult the source code for details on the changes.The
preStartConsumingChanges
andstartTimedSchedulerService
hooks have been removed.The triggerable schedulers
trigger
method now requires a list of sourcestamps, rather than a dictionary.The
SourceStamp
class is no longer used. It remains in the codebase to support loading data from pickles on upgrade, but should not be used in running code.The
BuildRequest
class no longer has fullsource
orsources
attributes. Use the data API to get this information (which is associated with the buildset, not the build request) instead.The undocumented
BuilderControl
methodsubmitBuildRequest
has been removed.The debug client no longer supports requesting builds (the
requestBuild
method has been removed). If you have been using this method in production, consider instead creating a new change source, using theForceScheduler
, or using one of the try schedulers.The
buildbot.misc.SerializedInvocation
class has been removed; usebuildbot.util.debounce.method
instead.The
progress
attributes of bothbuildbot.process.buildstep.BuildStep
andbuildbot.process.build.Build
have been removed. Subclasses should only be accessing the progress-tracking mechanics via thebuildbot.process.buildstep.BuildStep.setProgress
method.The
BuilderConfig
nextSlave
keyword argument takes a callable. This callable now receivesBuildRequest
instance in its signature as 3rd parameter. For retro-compatibility, all callable taking only 2 parameters will still work.properties object is now directly present in build, and not in build_status. This should not change much unless you try to access your properties via step.build.build_status. Remember that with PropertiesMixin, you can access properties via getProperties on the steps, and on the builds objects.
5.2.2. Slaves/Workers¶
5.2.2.1. Transition to “worker” terminology¶
Since version 0.9.0 of Buildbot “slave”-based terminology is deprecated in favor of “worker”-based terminology.
For details about public API changes see Transition to “worker” terminology, and Release Notes for Buildbot 0.9.0b8 release notes.
- The
buildbot-slave
package has been renamed tobuildbot-worker
. - Buildbot now requires import to be sorted using isort.
Please run
make isort
before creating a PR or use any available editor plugin in order to reorder your imports.
5.2.2.2. Requirements¶
buildbot-worker
requires Python 2.6
5.2.2.3. Features¶
- The Buildbot worker now includes the number of CPUs in the information it supplies to the master on connection.
This value is autodetected, but can be overridden with the
--numcpus
argument tobuildslave create-worker
. - The
DockerLatentWorker
image attribute is now renderable (can take properties in account). - The
DockerLatentWorker
sets environment variables describing how to connect to the master. Example dockerfiles can be found in master/contrib/docker. DockerLatentWorker
now has ahostconfig
parameter that can be used to setup host configuration when creating a new container.DockerLatentWorker
now has anetworking_config
parameter that can be used to setup container networks.- The
DockerLatentWorker
volumes
attribute is now renderable.
5.2.2.4. Fixes¶
5.2.2.5. Changes for Developers¶
- EC2 Latent Worker upgraded from
boto2
toboto3
.
5.2.2.6. Deprecations, Removals, and Non-Compatible Changes¶
- buildmaster and worker no longer supports old-style source steps.
- On Windows, if a
ShellCommand
step in whichcommand
was specified as a list is executed, and a list element is a string consisting of a single pipe character, it no longer creates a pipeline. Instead, the pipe character is passed verbatim as an argument to the program, like any other string. This makes command handling consistent between Windows and Unix-like systems. To have a pipeline, specifycommand
as a string. - Support for python 2.6 was dropped from the master.
public_html
directory is not created anymore inbuildbot create-master
(it’s not used for some time already). Documentation was updated with suggestions to use third party web server for serving static file.usePTY
default value has been changed fromslave-config
toNone
(use ofslave-config
will still work).GithubStatusPush
reporter was renamed toGitHubStatusPush
.- Worker commands version bumped to 3.0.
- Master/worker protocol has been changed:
slave_commands
key in worker information was renamed toworker_commands
.getSlaveInfo
remote method was renamed togetWorkerInfo
.slave-config
value ofusePTY
is not supported anymore.slavesrc
command argument was renamed toworkersrc
inuploadFile
anduploadDirectory
commands.slavedest
command argument was renamed toworkerdest
indownloadFile
command.- Previously deprecated
WorkerForBuilder.remote_shutdown()
remote command has been removed.
5.2.3. Details¶
For a more detailed description of the changes made in this version, see the git log itself:
git log v0.8.12..v0.9.0