Release Notes for Buildbot 0.9.0b1 ================================== .. Any change that adds a feature or fixes a bug should have an entry here. Most simply need an additional bulleted list item, but more significant changes can be given a subsection of their own. The following are the release notes for Buildbot 0.9.0b1. Buildbot 0.9.0b1 was released on the 25th of June, 2015. Master ------ This version represents a refactoring of Buildbot into a consistent, well-defined application composed of loosely coupled components. The components are linked by a common database backend and a messaging system. This allows components to be distributed across multiple build masters. It also allows the rendering of complex web status views to be performed in the browser, rather than on the buildmasters. The branch looks forward to committing to long-term API compatibility, but does not reach that goal. The Buildbot-0.9.x series of releases will give the new APIs time to "settle in" before we commit to them. Commitment will wait for Buildbot-1.0.0 (as per http://semver.org). Once Buildbot reaches version 1.0.0, upgrades will become much easier for users. To encourage contributions from a wider field of developers, the web application is designed to look like a normal AngularJS application. Developers familiar with AngularJS, but not with Python, should be able to start hacking on the web application quickly. The web application is "pluggable", so users who develop their own status displays can package those separately from Buildbot itself. Other goals: * An approachable HTTP REST API, used by the web application but available for any other purpose. * A high degree of coverage by reliable, easily-modified tests. * "Interlocking" tests to guarantee compatibility. For example, the real and fake DB implementations must both pass the same suite of tests. Then no unseen difference between the fake and real implementations can mask errors that will occur in production. Requirements ~~~~~~~~~~~~ The ``buildbot`` package requires Python 2.6 or higher -- Python 2.5 is no longer supported. The ``buildbot-slave`` package requires Python 2.5 or higher -- Python 2.4 is no longer supported. No additional software or systems, aside from some minor Python packages, are required. But the devil is in the details: * If you want to do web *development*, or *build* the ``buildbot-www`` package, you'll need Node. It's an Angular app, and that's how such apps are developed. We've taken pains to not make either a requirement for users - you can simply 'pip install' ``buildbot-www`` and be on your way. This is the case even if you're hacking on the Python side of Buildbot. * For a single master, nothing else is required. Minor Python Packages ..................... * Buildbot requires at least Twisted-11.0.0. * Buildbot works python-dateutil >= 1.5 Known Limitations of 0.9.0b1 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The following feature will be implemented for Buildbot 0.9.1 Milestone. * Multimaster is not supported as of Buildbot 0.9.0. * Not all status plugin are converted to the new reporter API. Only email and Gerrit reporters are fully supported. Irc support is limited, and not converted to reporter api Features ~~~~~~~~ Buildbot-0.9.0 introduces the :ref:`Data_API`, a consistent and scalable method for accessing and updating the state of the Buildbot system. This API replaces the existing, ill-defined Status API, which has been removed. Buildbot-0.9.0 introduces new :ref:`WWW` Interface using websocket for realtime updates. Buildbot code that interacted with the Status API (a substantial portion!) has been rewritten to use the Data API. Individual features and improvements to the Data API are not described on this page. * Buildbot now supports plugins. They allow Buildbot to be extended by using components distributed independently from the main code. They also provide for a unified way to access all components. When previously the following construction was used:: from buildbot.kind.other.bits import ComponentClass ... ComponentClass ... the following construction achieves the same result:: from buildbot.plugins import kind ... kind.ComponentClass ... Kinds of components that are available this way are described in :doc:`../manual/plugins`. .. note:: While the components can be still directly imported as ``buildbot.kind.other.bits``, this might not be the case after Buildbot v1.0 is released. * Both the P4 source step and P4 change source support ticket-based authentication. * OpenStack latent slaves now support block devices as a bootable volume. * Add new :bb:step:`Cppcheck` step. * Add a new :doc:`Docker latent BuildSlave `. * Add a new configuration for creating custom services in out-of-tree CI systems or plugins. See :py:class:`buildbot.util.service.BuildbotService` * Add ``try_ssh`` configuration file setting and ``--ssh`` command line option for the try tool to specify the command to use for connecting to the build master. * GitHub change hook now supports application/json format. * Add support for dynamically adding steps during a build. See :ref:`DynamicBuildFactories`. * :bb:chsrc:`GitPoller` now supports detecting new branches * :bb:step:`Git` supports an "origin" option to give a name to the remote repo. Reporters ~~~~~~~~~ Status plugins have been moved into the ``reporters`` namespace. Their API has slightly to changed in order to adapt to the new data API. See respective documentation for details. * :class:`~buildbot.status.status_gerrit.GerritStatusPush` renamed to :class:`~buildbot.reporters.gerrit.GerritStatusPush` * :class:`~buildbot.status.mail.MailNotifier` renamed to :class:`~buildbot.reporters.mail.MailNotifier` * :class:`~buildbot.status.mail.MailNotifier` argument ``messageFormatter`` should now be a :class:`~buildbot.status.message.MessageFormatter`, due to removal of data api, custom message formatters need to be rewritten. * :class:`~buildbot.status.mail.MailNotifier` argument ``previousBuildGetter`` is not supported anymore * :class:`~buildbot.reporters.gerrit.Gerrit` supports specifying an SSH identity file explicitly. * Added StashStatusPush status hook for Atlassian Stash * :bb:reporter:`MailNotifier` no longer forces SSL 3.0 when ``useTls`` is true. * :bb:reporter:`GerritStatusPush` callbacks slightly changed signature, and include a master reference instead of a status reference. * :class:`~buildbot.status.github.GitHubStatus` now accepts a ``context`` parameter to be passed to the GitHub Status API. * Buildbot UI introduces branch new Authentication, and Authorizations framework. Please look at their respective guide in :ref:`WWW` Fixes ~~~~~ * Buildbot is now compatible with SQLAlchemy 0.8 and higher, using the newly-released SQLAlchemy-Migrate. * The version check for SQLAlchemy-Migrate was fixed to accept more version string formats. * The :bb:step:`HTTPStep` step's request parameters are now renderable. * With Git(), force the updating submodules to ensure local changes by the build are overwritten. This both ensures more consistent builds and avoids errors when updating submodules. * Buildbot is now compatible with Gerrit v2.6 and higher. To make this happen, the return result of ``reviewCB`` and ``summaryCB`` callback has changed from .. code-block:: python (message, verified, review) to .. code-block:: python {'message': message, 'labels': {'label-name': value, ... } } The implications are: * there are some differences in behaviour: only those labels that were provided will be updated * Gerrit server must be able to provide a version, if it can't the :bb:reporter:`GerritStatusPush` will not work .. note:: If you have an old style ``reviewCB`` and/or ``summaryCB`` implemented, these will still work, however there could be more labels updated than anticipated. More detailed information is available in :bb:reporter:`GerritStatusPush` section. * :bb:chsrc:`P4Source`'s ``server_tz`` parameter now works correctly. * The ``revlink`` in changes produced by the Bitbucket hook now correctly includes the ``changes/`` portion of the URL. * :bb:chsrc:`PBChangeSource`'s git hook :contrib-src:`master/contrib/git_buildbot.py` now supports git tags A pushed git tag generates a change event with the ``branch`` property equal to the tag name. To schedule builds based on buildbot tags, one could use something like this: .. code-block:: python c['schedulers'].append( SingleBranchScheduler(name='tags', change_filter=filter.ChangeFilter( branch_re='v[0-9]+\.[0-9]+\.[0-9]+(?:-pre|rc[0-9]+|p[0-9]+)?') treeStableTimer=None, builderNames=['tag_build'])) * Missing "name" and "email" properties received from Gerrit are now handled properly * Fixed bug which made it impossible to specify the project when using the BitBucket dialect. * The :bb:step:`PyLint` step has been updated to understand newer output. * Fixed SVN master-side source step: if a SVN operation fails, the repository end up in a situation when a manual intervention is required. Now if SVN reports such a situation during initial check, the checkout will be clobbered. * The build properties are now stored in the database in the ``build_properties`` table. * The list of changes in the build page now displays all the changes since the last successful build. * GitHub change hook now correctly responds to ping events. * ``buildbot.steps.http`` steps now correctly have ``url`` parameter renderable * When no arguments are used ``buildbot checkconfig`` now uses :file:`buildbot.tac` to locate the master config file. * `buildbot.util.flatten` now correctly flattens arbitrarily nested lists. `buildbot.util.flattened_iterator` provides an iterable over the collection which may be more efficient for extremely large lists. Deprecations, Removals, and Non-Compatible Changes ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ * `BonsaiPoller` is removed. * ``buildbot.ec2buildslave`` is removed; use ``buildbot.buildslave.ec2`` instead. * ``buildbot.libvirtbuildslave`` is removed; use ``buildbot.buildslave.libvirt`` instead. .. TODO: 0.9.0 release notes should include a warning similar to that in 0.8.9 about new-style steps * `buildbot.util.flatten` flattens lists and tuples by default (previously only lists). Additionally, flattening something that isn't the type to flatten has different behaviour. Previously, it would return the original value. Instead, it now returns an array with the original value as the sole element. * ``buildbot.tac`` does not support ``print`` statements anymore. Such files should now use ``print`` as a function instead (see https://docs.python.org/3.0/whatsnew/3.0.html#print-is-a-function for more details). Note that this applies to both python2.x and python3.x runtimes. WebStatus ......... The old, clunky WebStatus has been removed. You will like the new interface! RIP WebStatus, you were a good friend. remove it and replace it with :bb:cfg:`www configuration `. Requirements ............ * Buildbot's tests now require at least Mock-0.8.0. * SQLAlchemy-Migrate-0.6.1 is no longer supported. * Builder names are now restricted to unicode strings or ASCII bytestrings. Encoded bytestrings are not accepted. Steps ..... * New-style steps are now the norm, and support for old-style steps is deprecated. Such support will be removed in the next release. * Status strings for old-style steps could be supplied through a wide variety of conflicting means (``describe``, ``description``, ``descriptionDone``, ``descriptionSuffix``, ``getText``, and ``setText``, to name just a few). While all attempts have been made to maintain compatibility, you may find that the status strings for old-style steps have changed in this version. To fix steps that call ``setText``, try setting the ``descriptionDone`` attribute directly, instead -- or just rewrite the step in the new style. * Old-style *source* steps (imported directly from ``buildbot.steps.source``) are no longer supported on the master. * The monotone source step got an overhaul and can now better manage its database (initialize and/or migrate it, if needed). In the spirit of monotone, buildbot now always keeps the database around, as it's an append-only database. Changes and Removals .................... * Buildslave names must now be 50-character :ref:`identifier `. Note that this disallows some common characters in bulidslave names, including spaces, ``/``, and ``.``. * Builders now have "tags" instead of a category. Builders can have multiple tags, allowing more flexible builder displays. * :bb:sched:`ForceScheduler` has the following changes: - The default configuration no longer contains four ``AnyPropertyParameter`` instances. - Configuring ``codebases`` is now mandatory, and the deprecated ``branch``, ``repository``, ``project``, ``revision`` are not supported anymore in :bb:sched:`ForceScheduler` - :py:meth:`buildbot.schedulers.forcesched.BaseParameter.updateFromKwargs` now takes a ``collector`` parameter used to collect all validation errors * :bb:sched:`Periodic`, :bb:sched:`Nightly` and :bb:sched:`NightlyTriggerable` have the following changes: - The :bb:sched:`Periodic` and :bb:sched:`Nightly` schedulers can now consume changes and use ``onlyIfChanged`` and ``createAbsoluteTimestamps``. - All "timed" schedulers now handle ``codebases`` the same way. Configuring ``codebases`` is strongly recommended. Using the ``branch`` parameter is discouraged. * Logs are now stored as Unicode strings, and thus must be decoded properly from the bytestrings provided by shell commands. By default this encoding is assumed to be UTF-8, but the :bb:cfg:`logEncoding` parameter can be used to select an alternative. Steps and individual logfiles can also override the global default. * The PB status service uses classes which have now been removed, and anyway is redundant to the REST API, so it has been removed. It has taken the following with it: * ``buildbot statuslog`` * ``buildbot statusgui`` (the GTK client) * ``buildbot debugclient`` The ``PBListener`` status listener is now deprecated and does nothing. Accordingly, there is no external access to status objects via Perspective Broker, aside from some compatibility code for the try scheduler. The ``debugPassword`` configuration option is no longer needed and is thus deprecated. * The undocumented and un-tested ``TinderboxMailNotifier``, designed to send emails suitable for the abandoned and insecure Tinderbox tool, has been removed. * Buildslave info is no longer available via :ref:`Interpolate` and the ``SetSlaveInfo`` buildstep has been removed. * The undocumented ``path`` parameter of the :bb:step:`MasterShellCommand` buildstep has been renamed ``workdir`` for better consistency with the other steps. * The name and source of a Property have to be unicode or ascii string. * Property values must be serializable in JSON. * :bb:reporter:`IRC` has the following changes: - categories parameter is deprecated and removed. It should be replaced with tags=[cat] - noticeOnChannel parameter is deprecated and removed. * workdir behavior has been unified: - ``workdir`` attribute of steps is now a property in :py:class:`~buildbot.process.buildstep.BuildStep`, and choose the workdir given following priority: * workdir of the step, if defined * workdir of the builder (itself defaults to 'build') - setDefaultWorkdir() has been deprecated, but is now behaving the same for all the steps: Setting self.workdir if not already set * :bb:step:`Trigger` now has a ``getSchedulersAndProperties`` method that can ve overridden to support dynamic triggering. * ```master.cfg`` is now parsed from a thread. Previously it was run in the main thread, and thus slowing down the master in case of big config, or network access done to generate the config. * :bb:chsrc:`SVNPoller`'s svnurl parameter has been changed to repourl. Changes for Developers ~~~~~~~~~~~~~~~~~~~~~~ * Botmaster no longer service parent for buildslaves. Service parent functionality has been transferred to BuildslaveManager. It should be noted Botmaster no longer has a ``slaves`` field as it was moved to BuildslaveManager. * The sourcestamp DB connector now returns a ``patchid`` field. * Buildbot no longer polls the database for jobs. The ``db_poll_interval`` configuration parameter and the :bb:cfg:`db` key of the same name are deprecated and will be ignored. * The interface for adding changes has changed. The new method is ``master.data.updates.addChange`` (implemented by :py:meth:`~buildbot.data.changes.ChangeResourceType.addChange`), although the old interface (``master.addChange``) will remain in place for a few versions. The new method: * returns a change ID, not a Change instance; * takes its ``when_timestamp`` argument as epoch time (UNIX time), not a datetime instance; and * does not accept the deprecated parameters ``who``, ``isdir``, ``is_dir``, and ``when``. * requires that all strings be unicode, not bytestrings. Please adjust any custom change sources accordingly. * A new build status, CANCELLED, has been added. It is used when a step or build is deliberately cancelled by a user. * This upgrade will delete all rows from the ``buildrequest_claims`` table. If you are using this table for analytical purposes outside of Buildbot, please back up its contents before the upgrade, and restore it afterward, translating object IDs to scheduler IDs if necessary. This translation would be very slow and is not required for most users, so it is not done automatically. * All of the schedulers DB API methods now accept a schedulerid, rather than an objectid. If you have custom code using these methods, check your code and make the necessary adjustments. * The ``addBuildsetForSourceStamp`` method has become ``addBuildsetForSourceStamps``, and its signature has changed. The ``addBuildsetForSourceStampSetDetails`` method has become ``addBuildsetForSourceStampsWithDefaults``, and its signature has changed. The ``addBuildsetForSourceStampDetails`` method has been removed. The ``addBuildsetForLatest`` method has been removed. It is equivalent to ``addBuildsetForSourceStampDetails`` with ``sourcestamps=None``. These methods are not yet documented, and their interface is not stable. Consult the source code for details on the changes. * The ``preStartConsumingChanges`` and ``startTimedSchedulerService`` hooks have been removed. * The triggerable schedulers' ``trigger`` method now requires a list of sourcestamps, rather than a dictionary. * The :py:class:`~buildbot.sourcestamp.SourceStamp` class is no longer used. It remains in the codebase to support loading data from pickles on upgrade, but should not be used in running code. * The :py:class:`~buildbot.process.buildrequest.BuildRequest` class no longer has full ``source`` or ``sources`` attributes. Use the data API to get this information (which is associated with the buildset, not the build request) instead. * The undocumented ``BuilderControl`` method ``submitBuildRequest`` has been removed. * The debug client no longer supports requesting builds (the ``requestBuild`` method has been removed). If you have been using this method in production, consider instead creating a new change source, using the :bb:sched:`ForceScheduler`, or using one of the try schedulers. * The ``buildbot.misc.SerializedInvocation`` class has been removed; use :py:func:`buildbot.util.debounce.method` instead. * The ``progress`` attributes of both :py:class:`buildbot.process.buildstep.BuildStep` and :py:class:`buildbot.process.build.Build` have been removed. Subclasses should only be accessing the progress-tracking mechanics via the :py:meth:`buildbot.process.buildstep.BuildStep.setProgress` method. Slave ----- Features ~~~~~~~~ Fixes ~~~~~ Deprecations, Removals, and Non-Compatible Changes ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ * buildmaster and buildslave no longer supports old-style source steps. * On Windows, if a :bb:step:`ShellCommand` step in which ``command`` was specified as a list is executed, and a list element is a string consisting of a single pipe character, it no longer creates a pipeline. Instead, the pipe character is passed verbatim as an argument to the program, like any other string. This makes command handling consistent between Windows and Unix-like systems. To have a pipeline, specify ``command`` as a string. Details ------- For a more detailed description of the changes made in this version, see the git log itself: .. code-block:: bash git log v0.8.10..v0.9.0b1