4.4.18. Release Notes for Buildbot 0.8.9

The following are the release notes for Buildbot 0.8.9. Buildbot 0.8.9 was released on 14 June, 2014.

4.4.18.1. Master

Features

  • The following optional parameters have been added to EC2LatentBuildSlave
    • Boolean parameter spot_instance, default False, creates a spot instance.
    • Float parameter max_spot_price defines the maximum bid for a spot instance.
    • List parameter volumes, takes a list of (volume_id, mount_point) tuples.
    • String parameter placement is appended to the region parameter, e.g. region='us-west-2', placement='b' will result in the spot request being placed in us-west-2b.
    • Float parameter price_multiplier specifies the percentage bid above the 24-hour average spot price.
    • Dict parameter tags specifies AWS tags as key/value pairs to be applied to new instances.

    With spot_instance=True, an EC2LatentBuildSlave will attempt to create a spot instance with the provided spot price, placement, and so on.

  • The web hooks now include support for Bitbucket, GitLab and Gitorious.

  • The GitHub webhook has been updated to work with v3 of the GitHub webhook API.

  • The GitHub webhook can now optionally ignore non-distinct commits (bug #1861).

  • The HgPoller and GitPoller now split filenames on newlines, rather than whitespace, so files containing whitespace are handled correctly.

  • Add 'pollAtLaunch' flag for polling change sources. This allows a poller to poll immediately on launch and get changes that occurred while it was down.

  • Added the BitbucketPullrequestPoller changesource.

  • The GitPoller can now be configured to poll all available branches (pull request 1010).

  • The P4Source changesource now supports Perforce servers in a different timezone than the buildbot master (pull request 728).

  • Each Scheduler type can now take a 'reason' argument to customize the reason it uses for triggered builds.

  • A new argument createAbsoluteSourceStamps has been added to SingleBranchScheduler for use with multiple codebases.

  • A new argument createAbsoluteSourceStamps has been added to Nightly for use with multiple codebases.

  • The Periodic scheduler now supports codebases.

  • The ForceScheduler now takes a buttonName argument to specify the name of the button on the force-build form.

  • Master side source checkout steps now support patches (bug #2098). The Git and Mercurial steps use their inbuilt commands to apply patches (bug #2563).

  • Master side source checkout steps now support retry option (bug #2465).

  • Master-side source checkout steps now respond to the "stop build" button (bug #2356).

  • Git source checkout step now supports reference repositories.

  • The Git step now uses the git clean option -f twice, to also remove untracked directories managed by another git repository. See bug #2560.

  • The branch and codebase arguments to the Git step are now renderable.

  • Gerrit integration with Git Source step on master side (bug #2485).

  • P4 source step now supports more advanced options.

  • The master-side SVN step now supports authentication for mode=export, fixing bug #2463.

  • The SVN step will now canonicalize URL's before matching them for better accuracy.

  • The SVN step now obfuscates the password in status logs, fixing bug #2468.

  • SVN source step and ShellCommand now support password obfuscation. (bug #2468 and bug #1748).

  • CVS source step now checks for "sticky dates" from a previous checkout before updating an existing source directory.

  • :Repo now supports a depth flag when initializing the repo. This controls the amount of git history to download.

  • The manifestBranch of the bb:step:Repo step is now renderable

  • New source step Monotone added on master side.

  • New source step Darcs added on master side.

  • A new Robocopy step is available for Windows builders (pull request 728).

  • The attributes description, descriptionDone and descriptionSuffix have been moved from ShellCommand to its superclass BuildStep so that any class that inherits from BuildStep can provide a suitable description of itself.

  • A new FlattenList Renderable has been added which can flatten nested lists.

  • Added new build steps for VC12, VS2013 and MsBuild12.

  • The mode parameter of the VS steps is now renderable (bug #2592).

  • The HTTPStep step can make arbitrary HTTP requests from the master, allowing communication with external APIs. This new feature requires the optional txrequests and requests Python packages.

  • A new MultipleFileUpload step was added to allow uploading several files (or directories) in a single step.

  • Information about the buildslaves (admin, host, etc) is now persisted in the database and available even if the slave is not connected.

  • Buildslave info can now be retrieved via Interpolate and a new SetSlaveInfo buildstep.

  • The GNUAutotools factory now has a reconf option to run autoreconf before ./configure.

  • Builder configurations can now include a description, which will appear in the web UI to help humans figure out what the builder does.

  • The WebStatus builder page can now filter pending/current/finished builds by property parameters of the form ?property.<name>=<value>.

  • The WebStatus StatusResourceBuilder page can now take the maxsearch argument

  • The WebStatus has a new authz "view" action that allows you to require users to logged in to view the WebStatus.

  • The WebStatus now shows revisions (+ codebase) where it used to simply say "multiple rev".

  • The Console view now supports codebases.

  • The web UI for Builders has been updated:
    • shows the build 'reason' and 'interested users'
    • shows sourcestamp information for builders that use multiple codebases (instead of the generic "multiple rev" placeholder that was shown before).

  • The waterfall and atom/rss feeds can be filtered with the project url paramter.

  • The WebStatus Authorization support now includes a view action which can be used to restrict read-only access to the Buildbot instance.

  • The web status now has options to cancel some or all pending builds.

  • The WebStatus now interprets ANSI color codes in stdio output.

  • It is now possible to select categories to show in the waterfall help

  • The web status now automatically scrolls output logs (pull request 1078).

  • The web UI now supports a PNG Status Resource that can be accessed publicly from for example README.md files or wikis or whatever other resource. This view produces an image in PNG format with information about the last build for the given builder name or whatever other build number if is passed as an argument to the view.

  • Revision links for commits on SouceForge (Allura) are now automatically generated.

  • The 'Rebuild' button on the web pages for builds features a dropdown to choose whether to rebuild from exact revisions or from the same sourcestamps (ie, update branch references)

  • Build status can be sent to GitHub. Depends on txgithub package. See GitHubStatusPush and GitHub Commit Status.

  • The IRC bot of IRC will, unless useRevisions is set, shorten long lists of revisions printed when a build starts; it will only show two, and the number of additional revisions included in the build.

  • A new argument summaryCB has been added to GerritStatusPush, to allow sending one review per buildset. Sending a single "summary" review per buildset is now the default if neither summaryCB nor reviewCB are specified.

  • The comments field of changes is no longer limited to 1024 characters on MySQL and Postgres. See bug #2367 and pull request 736.

  • HTML log files are no longer stored in status pickles (pull request 1077)

  • Builds are now retried after a slave is lost (pull request 1049).

  • The buildbot status client can now access a build properties via the getProperties call.

  • The start, restart, and reconfig commands will now wait for longer than 10 seconds as long as the master continues producing log lines indicating that the configuration is progressing.

  • Added new config option protocols which allows to configure multiple protocols on single master.

  • RemoteShellCommands can be killed by SIGTERM with the sigtermTime parameter before resorting to SIGKILL (bug #751). If the slave's version is less than 0.8.9, the slave will kill the process with SIGKILL regardless of whether sigtermTime is supplied.

  • Introduce an alternative way to deploy Buildbot and try the pyflakes tutorial using Docker.

  • Added zsh and bash tab-completions support for 'buildbot' command.

  • An example of a declarative configuration is included in https://github.com/buildbot/buildbot/blob/master/master/contrib/SimpleConfig.py, with copious comments.

  • Systemd unit files for Buildbot are available in the https://github.com/buildbot/buildbot/blob/master/master/contrib/ directory.

  • We've added some extra checking to make sure that you have a valid locale before starting buildbot (#2608).

Forward Compatibility

In preparation for a more asynchronous implementation of build steps in Buildbot 0.9.0, this version introduces support for new-style steps. Existing old-style steps will continue to function correctly in Buildbot 0.8.x releases and in Buildbot 0.9.0, but support will be dropped soon afterward. See New-Style Build Steps, below, for guidance on rewriting existing steps in this new style. To eliminate ambiguity, the documentation for this version only reflects support for new-style steps. Refer to the documentation for previous versions for infrormation on old-style steps.

Fixes

  • Fixes an issue where GitPoller sets the change branch to refs/heads/master - which isn't compatible with Git (pull request 1069).
  • Fixed an issue where the Git and CVS source steps silently changed the workdir to 'build' when the 'copy' method is used.
  • The CVS source step now respects the timeout parameter.
  • The Git step now uses the git submodule update option --init when updating the submodules of an existing repository, so that it will receive any newly added submodules.
  • The web status no longer relies on the current working directory, which is not set correctly by some initscripts, to find the templates/ directory (bug #2586).
  • The Perforce source step uses the correct path separator when the master is on Windows and the build slave is on a POSIX OS (pull request 1114).
  • The source steps now correctly interpolate properties in env.
  • GerritStatusPush now supports setting scores with Gerrit 2.6 and newer
  • The change hook no longer fails when passing unicode to change_hook_auth (pull request 996).
  • The source steps now correctly interpolate properties in env.
  • Whitespace is properly handled for StringParameter, so that appropriate validation errors are raised for required parameters (pull request 1084).
  • Fix a rare case where a buildtep might fail from a GeneratorExit exception (pull request 1063).
  • Fixed an issue where UTF-8 data in logs caused RSS feed exceptions (bug #951).
  • Fix an issue with unescaped author names causing invalid RSS feeds (bug #2596).
  • Fixed an issue with pubDate format in feeds.
  • Fixed an issue where the step text value could cause a TypeError in the build detail page (pull request 1061).
  • Fix failures where git clean fails but could be clobbered (pull request 1058).
  • Build step now correctly fails when the git clone step fails (pull request 1057).
  • Fixed a race condition in slave shutdown (pull request 1019).
  • Now correctly unsubscribes StatusPush from status updates when reconfiguring (pull request 997).
  • Fixes parsing git commit messages that are blank.
  • Git no longer fails when work dir exists but isn't a checkout (bug #2531).
  • The haltOnFailure and flunkOnFailure attricutes of ShellCommand are now renderable. (bug #2486).
  • The rotateLength and maxRotatedFile arguments are no longer treated as strings in buildbot.tac. This fixes log rotation. The upgrade_master command will notify users if they have this problem.
  • Builbot no longer specifies a revision when pulling from a mercurial (bug #438).
  • The WebStatus no longer incorrectly refers to fields that might not be visible.
  • The GerritChangeSource now sets a default author, fixing an exception that occured when Gerrit didn't report an owner name/email.
  • Respects the RETRY status when an interrupt occurs.
  • Fixes an off-by-one error when the tryclient is finding the current git branch.
  • Improve the Mercurial source stamp extraction in the try client.
  • Fixes some edge cases in timezone handling for python < 2.7.4 (bug #2522).
  • The EC2LatentBuildSlave will now only consider available AMI's.
  • Fixes a case where the first build runs on an old slave instead of a new one after reconfig (bug #2507).
  • The e-mail address validation for the MailNotifier status receiver has been improved.
  • The --db parameter of buildbot create-master is now validated.
  • No longer ignores default choice for ForceScheduler list parameters
  • Now correctly handles BuilderConfig(..., mergeRequests=False) (bug #2555).
  • Now excludes changes from sourcestamps when they aren't in the DB (bug #2554).
  • Fixes a compatibility issue with HPCloud in the OpenStack latent slave.
  • Allow _ as a valid character in JSONP callback names.
  • Fix build start time retrieval in the WebStatus grid view.
  • Increase the length of the DB fields changes.comments and buildset_properties.property_value.

Deprecations, Removals, and Non-Compatible Changes

  • The slave-side source steps are deprecated in this version of Buildbot, and master-side support will be removed in a future version. Please convert any use of slave-side steps (imported directly from buildbot.steps.source, rather than from a specific module like buildbot.steps.source.svn) to use master-side steps.
  • Both old-style and new-style steps are supported in this version of Buildbot. Upgrade your steps to new-style now, as support for old-style steps will be dropped after Buildbot-0.9.0. See New-Style Build Steps for details.
    • The LoggingBuildStep class has been deprecated, and support will be removed along with support for old-style steps after the Buildbot-0.9.0 release. Instead, subclass BuildStep and mix in ShellMixin to get similar behavior.
  • slavePortnum option deprecated, please use c['protocols']['pb']['port'] to set up PB port
  • The buildbot.process.mtrlogobserver module have been renamed to buildbot.steps.mtrlogobserver.
  • The buildmaster now requires at least Twisted-11.0.0.
  • The buildmaster now requires at least sqlalchemy-migrate 0.6.1.
  • The hgbuildbot Mercurial hook has been moved to contrib/, and does not work with recent versions of Mercurial and Twisted. The runtimes for these two tools are incompatible, yet hgbuildbot attempts to run both in the same Python interpreter. Mayhem ensues.
  • The try scheduler's --connect=ssh method no longer supports waiting for results (--wait).
  • The former buildbot.process.buildstep.RemoteCommand class and its subclasses are now in buildbot.process.remotecommand, although imports from the previous path will continue to work. Similarly, the former buildbot.process.buildstep.LogObserver class and its subclasses are now in buildbot.process.logobserver, although imports from the previous path will continue to work.
  • The undocumented BuildStep method checkDisconnect is deprecated and now does nothing as the handling of disconnects is now handled in the failed method. Any custom steps adding this method as a callback or errback should no longer do so.
  • The build step MsBuild is now called MsBuild4 as multiple versions are now supported. An alias is provided so existing setups will continue to work, but this will be removed in a future release.

Changes for Developers

  • The CompositeStepMixin now provides a runGlob method to check for files on the slave that match a given shell-style pattern.
  • The BuilderStatus now allows you to pass a filter_fn argument to generateBuilds.

4.4.18.2. Slave

Features

  • Added zsh and bash tab-completions support for 'buildslave' command.
  • RemoteShellCommands accept the new sigtermTime parameter from master. This allows processes to be killed by SIGTERM before resorting to SIGKILL (bug #751)
  • Commands will now throw a ValueError if mandatory args are not present.
  • Added a new remote command GlobPath that can be used to call Python's glob.glob on the slave.

Fixes

  • Fixed an issue when buildstep stop() was raising an exception incorrectly if timeout for buildstep wasn't set or was None (see pull request 753) thus keeping watched logfiles open (this prevented their removal on Windows in subsequent builds).
  • Fixed a bug in P4 source step where the timeout parameter was ignored.
  • Fixed a bug in P4 source step where using a custom view-spec could result in failed syncs due to incorrectly generated command-lines.
  • The logwatcher will use /usr/xpg4/bin/tail on Solaris, it if is available (pull request 1065).

Deprecations, Removals, and Non-Compatible Changes

4.4.18.3. Details

For a more detailed description of the changes made in this version, see the git log itself:

git log v0.8.8..v0.8.9