4.4.13. Release Notes for Buildbot 0.9.0b1

The following are the release notes for Buildbot 0.9.0b1. Buildbot 0.9.0b1 was released on the 25th of June, 2015.

4.4.13.1. Master

This version represents a refactoring of Buildbot into a consistent, well-defined application composed of loosely coupled components. The components are linked by a common database backend and a messaging system. This allows components to be distributed across multiple build masters. It also allows the rendering of complex web status views to be performed in the browser, rather than on the buildmasters.

The branch looks forward to committing to long-term API compatibility, but does not reach that goal. The Buildbot-0.9.x series of releases will give the new APIs time to "settle in" before we commit to them. Commitment will wait for Buildbot-1.0.0 (as per http://semver.org). Once Buildbot reaches version 1.0.0, upgrades will become much easier for users.

To encourage contributions from a wider field of developers, the web application is designed to look like a normal AngularJS application. Developers familiar with AngularJS, but not with Python, should be able to start hacking on the web application quickly. The web application is "pluggable", so users who develop their own status displays can package those separately from Buildbot itself.

Other goals:

  • An approachable HTTP REST API, used by the web application but available for any other purpose.
  • A high degree of coverage by reliable, easily-modified tests.
  • "Interlocking" tests to guarantee compatibility. For example, the real and fake DB implementations must both pass the same suite of tests. Then no unseen difference between the fake and real implementations can mask errors that will occur in production.

Requirements

The buildbot package requires Python 2.6 or higher -- Python 2.5 is no longer supported. The buildbot-slave package requires Python 2.5 or higher -- Python 2.4 is no longer supported.

No additional software or systems, aside from some minor Python packages, are required.

But the devil is in the details:

  • If you want to do web development, or build the buildbot-www package, you'll need Node. It's an Angular app, and that's how such apps are developed. We've taken pains to not make either a requirement for users - you can simply 'pip install' buildbot-www and be on your way. This is the case even if you're hacking on the Python side of Buildbot.
  • For a single master, nothing else is required.

Minor Python Packages

  • Buildbot requires at least Twisted-11.0.0.
  • Buildbot works python-dateutil >= 1.5

Known Limitations of 0.9.0b1

The following feature will be implemented for Buildbot 0.9.1 Milestone.

Features

Buildbot-0.9.0 introduces the Data API, a consistent and scalable method for accessing and updating the state of the Buildbot system. This API replaces the existing, ill-defined Status API, which has been removed. Buildbot-0.9.0 introduces new WWW Server Interface using websocket for realtime updates. Buildbot code that interacted with the Status API (a substantial portion!) has been rewritten to use the Data API. Individual features and improvements to the Data API are not described on this page.

  • Buildbot now supports plugins. They allow Buildbot to be extended by using components distributed independently from the main code. They also provide for a unified way to access all components. When previously the following construction was used:

    from buildbot.kind.other.bits import ComponentClass

... ComponentClass ...

    the following construction achieves the same result:

    from buildbot.plugins import kind

... kind.ComponentClass ...

    Kinds of components that are available this way are described in Plugin Infrastructure in Buildbot.

    Note

    While the components can be still directly imported as buildbot.kind.other.bits, this might not be the case after Buildbot v1.0 is released.

  • Both the P4 source step and P4 change source support ticket-based authentication.

  • OpenStack latent slaves now support block devices as a bootable volume.

  • Add new Cppcheck step.

  • Add a new Docker latent BuildSlave.

  • Add a new configuration for creating custom services in out-of-tree CI systems or plugins. See buildbot.util.service.BuildbotService

  • Add try_ssh configuration file setting and --ssh command line option for the try tool to specify the command to use for connecting to the build master.

  • GitHub change hook now supports application/json format.

  • Add support for dynamically adding steps during a build. See Dynamic Build Factories.

  • GitPoller now supports detecting new branches

  • Git supports an "origin" option to give a name to the remote repo.

Reporters

Status plugins have been moved into the reporters namespace. Their API has slightly to changed in order to adapt to the new data API. See respective documentation for details.

  • GerritStatusPush renamed to GerritStatusPush

  • MailNotifier renamed to MailNotifier

  • MailNotifier argument messageFormatter should now be a MessageFormatter, due to removal of data api, custom message formaters need to be rewritten.

  • MailNotifier argument previousBuildGetter is not supported anymore

  • Gerrit supports specifying an SSH identity file explicitly.

  • Added StashStatusPush status hook for Atlassian Stash

  • MailNotifier no longer forces SSL 3.0 when useTls is true.

  • GerritStatusPush callbacks slightly changed signature, and include a master reference instead of a status reference.

  • GitHubStatus now accepts a context parameter to be passed to the GitHub Status API.

  • Buildbot UI introduces branch new Authentication, and Authorizations framework.

    Please look at their respective guide in WWW Server

Fixes

  • Buildbot is now compatible with SQLAlchemy 0.8 and higher, using the newly-released SQLAlchemy-Migrate.

  • The version check for SQLAlchemy-Migrate was fixed to accept more version string formats.

  • The HTTPStep step's requeset parameters are now renderable.

  • With Git(), force the updating submodules to ensure local changes by the build are overwitten. This both ensures more consistent builds and avoids errors when updating submodules.

  • Buildbot is now compatible with Gerrit v2.6 and higher.

    To make this happen, the return result of reviewCB and summaryCB callback has changed from

    (message, verified, review)

    to

    {'message': message,
 'labels': {'label-name': value,
           ...
           }
}

    The implications are:

    • there are some differences in behaviour: only those labels that were provided will be updated
    • Gerrit server must be able to provide a version, if it can't the GerritStatusPush will not work

    Note

    If you have an old style reviewCB and/or summaryCB implemented, these will still work, however there could be more labels updated than anticipated.

    More detailed information is available in GerritStatusPush section.

  • P4Source's server_tz parameter now works correctly.

  • The revlink in changes broduced by the Bitbucket hook now correctly includes the changes/ portion of the URL.

  • PBChangeSource's git hook contrib/git_buildbot.py now supports git tags

    A pushed git tag generates a change event with the branch property equal to the tag name. To schedule builds based on buildbot tags, one could use something like this:

    c['schedulers'].append(
   SingleBranchScheduler(name='tags',
      change_filter=filter.ChangeFilter(
         branch_re='v[0-9]+\.[0-9]+\.[0-9]+(?:-pre|rc[0-9]+|p[0-9]+)?')
      treeStableTimer=None,
      builderNames=['tag_build']))

  • Missing "name" and "email" properties received from Gerrit are now handled properly

  • Fixed bug which made it impossible to specify the project when using the BitBucket dialect.

  • The PyLint step has been updated to understand newer output.

  • Fixed SVN master-side source step: if a SVN operation fails, the repository end up in a situation when a manual intervention is required. Now if SVN reports such a situation during initial check, the checkout will be clobbered.

  • The build properties are now stored in the database in the build_properties table.

  • The list of changes in the build page now displays all the changes since the last sucessful build.

  • GitHub change hook now correctly responds to ping events.

  • buildbot.steps.http steps now correctly have url parameter renderable

  • When no arguments are used buildbot checkconfig now uses buildbot.tac to locate the master config file.

  • buildbot.util.flatten now correctly flattens arbitrarily nested lists. buildbot.util.flattened_iterator provides an iterable over the collection which may be more efficient for extremely large lists.

Deprecations, Removals, and Non-Compatible Changes

  • BonsaiPoller is removed.
  • buildbot.ec2buildslave is removed; use buildbot.buildslave.ec2 instead.
  • buildbot.libvirtbuildslave is removed; use buildbot.buildslave.libvirt instead.
  • buildbot.util.flatten falttens lists and tuples by default (previously only lists). Additionally, flattening something that isn't the type to flatten has different behaviour. Previously, it would return the original value. Instead, it now returns an array with the original value as the sole element.
  • buildbot.tac does not support print statements anymore. Such files should now use print as a function instead (see https://docs.python.org/3.0/whatsnew/3.0.html#print-is-a-function for more details). Note that this applies to both python2.x and python3.x runtimes.

WebStatus

The old, clunky WebStatus has been removed. You will like the new interface! RIP WebStatus, you were a good friend.

remove it and replace it with www configuration.

Requirements

  • Buildbot's tests now require at least Mock-0.8.0.
  • SQLAlchemy-Migrate-0.6.1 is no longer supported.
  • Builder names are now restricted to unicode strings or ASCII bytestrings. Encoded bytestrings are not accepted.

Steps

  • New-style steps are now the norm, and support for old-style steps is deprecated. Such support will be removed in the next release.
    • Status strings for old-style steps could be supplied through a wide variety of conflicting means (describe, description, descriptionDone, descriptionSuffix, getText, and setText, to name just a few). While all attempts have been made to maintain compatibility, you may find that the status strings for old-style steps have changed in this version. To fix steps that call setText, try setting the descriptionDone attribute directly, instead -- or just rewrite the step in the new style.
  • Old-style source steps (imported directly from buildbot.steps.source) are no longer supported on the master.
  • The monotone source step got an overhaul and can now better manage its database (initialize and/or migrate it, if needed). In the spirit of monotone, buildbot now always keeps the database around, as it's an append-only database.

Changes and Removals

  • Buildslave names must now be 50-character identifier. Note that this disallows some common characters in bulidslave names, including spaces, /, and ..

  • Builders now have "tags" instead of a category. Builders can have multiple tags, allowing more flexible builder displays.

  • ForceScheduler has the following changes:

    • The default configuration no longer contains four AnyPropertyParameter instances.
    • Configuring codebases is now mandatory, and the deprecated branch, repository, project, revision are not supported anymore in ForceScheduler
    • buildbot.schedulers.forcesched.BaseParameter.updateFromKwargs now takes a collector parameter used to collect all validation errors

  • Periodic, Nightly and NightlyTriggerable have the following changes:

    • The Periodic and Nightly schedulers can now consume changes and use onlyIfChanged and createAbsoluteTimestamps.
    • All "timed" schedulers now handle codebases the same way. Configuring codebases is strongly recommended. Using the branch parameter is discouraged.

  • Logs are now stored as Unicode strings, and thus must be decoded properly from the bytestrings provided by shell commands. By default this encoding is assumed to be UTF-8, but the logEncoding parameter can be used to select an alternative. Steps and individual logfiles can also override the global default.

  • The PB status service uses classes which have now been removed, and anyway is redundant to the REST API, so it has been removed. It has taken the following with it:

    • buildbot statuslog
    • buildbot statusgui (the GTK client)
    • buildbot debugclient

    The PBListener status listener is now deprecated and does nothing. Accordingly, there is no external access to status objects via Perspective Broker, aside from some compatibility code for the try scheduler.

    The debugPassword configuration option is no longer needed and is thus deprecated.

  • The undocumented and un-tested TinderboxMailNotifier, designed to send emails suitable for the abandoned and insecure Tinderbox tool, has been removed.

  • Buildslave info is no longer available via Interpolate and the SetSlaveInfo buildstep has been removed.

  • The undocumented path parameter of the MasterShellCommand buildstep has been renamed workdir for better consistency with the other steps.

  • The name and source of a Property have to be unicode or ascii string.

  • Property values must be serializable in JSON.

  • IRC has the following changes:

    • categories parameter is deprecated and removed. It should be replaced with tags=[cat]
    • noticeOnChannel parameter is deprecated and removed.

  • workdir behavior has been unified:

    • workdir attribute of steps is now a property in BuildStep, and choose the workdir given following priority:

      • workdir of the step, if defined
      • workdir of the builder (itself defaults to 'build')
      • setDefaultWorkdir() has been deprecated, but is now behaving the same for all the steps: Setting self.workdir if not already set

  • Trigger now has a getSchedulersAndProperties method that can ve overriden to support dynamic triggering.

  • `master.cfg is now parsed from a thread. Previously it was run in the main thread, and thus slowing down the master in case of big config, or network access done to generate the config.

  • SVNPoller's svnurl parameter has been changed to repourl.

Changes for Developers

  • Botmaster no longer service parent for buildslaves. Service parent functionality has been transferred to BuildslaveManager. It should be noted Botmaster no longer has a slaves field as it was moved to BuildslaveManager.

  • The sourcestamp DB connector now returns a patchid field.

  • Buildbot no longer polls the database for jobs. The db_poll_interval configuration parameter and the db key of the same name are deprecated and will be ignored.

  • The interface for adding changes has changed. The new method is master.data.updates.addChange (implemented by addChange), although the old interface (master.addChange) will remain in place for a few versions. The new method:

    • returns a change ID, not a Change instance;
    • takes its when_timestamp argument as epoch time (UNIX time), not a datetime instance; and
    • does not accept the deprecated parameters who, isdir, is_dir, and when.
    • requires that all strings be unicode, not bytestrings.

    Please adjust any custom change sources accordingly.

  • A new build status, CANCELLED, has been added. It is used when a step or build is deliberately cancelled by a user.

  • This upgrade will delete all rows from the buildrequest_claims table. If you are using this table for analytical purposes outside of Buildbot, please back up its contents before the upgrade, and restore it afterward, translating object IDs to scheduler IDs if necessary. This translation would be very slow and is not required for most users, so it is not done automatically.

  • All of the schedulers DB API methods now accept a schedulerid, rather than an objectid. If you have custom code using these methods, check your code and make the necessary adjustments.

  • The addBuildsetForSourceStamp method has become addBuildsetForSourceStamps, and its signature has changed. The addBuildsetForSourceStampSetDetails method has become addBuildsetForSourceStampsWithDefaults, and its signature has changed. The addBuildsetForSourceStampDetails method has been removed. The addBuildsetForLatest method has been removed. It is equivalent to addBuildsetForSourceStampDetails with sourcestamps=None. These methods are not yet documented, and their interface is not stable. Consult the source code for details on the changes.

  • The preStartConsumingChanges and startTimedSchedulerService hooks have been removed.

  • The triggerable schedulers` trigger method now requires a list of sourcestamps, rather than a dictionary.

  • The SourceStamp class is no longer used. It remains in the codebase to support loading data from pickles on upgrade, but should not be used in running code.

  • The BuildRequest class no longer has full source or sources attributes. Use the data API to get this information (which is associated with the buildset, not the build request) instead.

  • The undocumented BuilderControl method submitBuildRequest has been removed.

  • The debug client no longer supports requesting builds (the requestBuild method has been removed). If you have been using this method in production, consider instead creating a new change source, using the ForceScheduler, or using one of the try schedulers.

  • The buildbot.misc.SerializedInvocation class has been removed; use buildbot.util.debounce.method instead.

  • The progress attributes of both buildbot.process.buildstep.BuildStep` and buildbot.process.build.Build have been removed. Subclasses should only be accessing the progress-tracking mechanics via the buildbot.process.buildstep.BuildStep.setProgress method.

4.4.13.2. Slave

Features

Fixes

Deprecations, Removals, and Non-Compatible Changes

  • buildmaster and buildslave no longer supports old-style source steps.
  • On Windows, if a ShellCommand step in which command was specified as a list is executed, and a list element is a string consisting of a single pipe character, it no longer creates a pipeline. Instead, the pipe character is passed verbatim as an argument to the program, like any other string. This makes command handling consistent between Windows and Unix-like systems. To have a pipeline, specify command as a string.

4.4.13.3. Details

For a more detailed description of the changes made in this version, see the git log itself:

git log v0.8.10..v0.9.0b1