3.2.5. Configuration¶
Wherever possible, Buildbot components should access configuration information
as needed from the canonical source, master.config
, which is an instance of
MasterConfig
. For example, components should not keep a copy of
the buildbotURL
locally, as this value may change throughout the lifetime
of the master.
Components which need to be notified of changes in the configuration should be
implemented as services, subclassing ReconfigurableServiceMixin
, as
described in Reconfiguration.
-
class
buildbot.config.
MasterConfig
¶ The master object makes much of the configuration available from an object named
master.config
. Configuration is stored as attributes of this object. Where possible, other Buildbot components should access this configuration directly and not cache the configuration values anywhere else. This avoids the need to ensure that update-from-configuration methods are called on a reconfig.Aside from validating the configuration, this class handles any backward-compatibility issues - renamed parameters, type changes, and so on - removing those concerns from other parts of Buildbot.
This class may be instantiated directly, creating an entirely default configuration, or via
FileLoader.loadConfig
, which will load the configuration from a config file.The following attributes are available from this class, representing the current configuration. This includes a number of global parameters:
-
buildbotURL
¶ The URL of this buildmaster, for use in constructing WebStatus URLs; from
buildbotURL
.
-
logCompressionLimit
¶ The current log compression limit, from
logCompressionLimit
.
-
logCompressionMethod
¶ The current log compression method, from
logCompressionMethod
.
-
logMaxSize
¶ The current log maximum size, from
logMaxSize
.
-
logMaxTailSize
¶ The current log maximum size, from
logMaxTailSize
.
-
logEncoding
¶ The encoding to expect when logs are provided as bytestrings, from
logEncoding
.
-
properties
¶ A
Properties
instance containing global properties, fromproperties
.
-
collapseRequests
¶ A callable, or True or False, describing how to collapse requests; from
collapseRequests
.
-
prioritizeBuilders
¶ A callable, or None, used to prioritize builders; from
prioritizeBuilders
.
-
codebaseGenerator
¶ A callable, or None, used to determine the codebase from an incoming
Change
, fromcodebaseGenerator
-
multiMaster
¶ If true, then this master is part of a cluster; based on
multiMaster
.
The remaining attributes contain compound configuration structures, usually dictionaries:
-
validation
¶ Validation regular expressions, a dictionary from
validation
. It is safe to assume that all expected keys are present.
-
db
¶ Database specification, a dictionary with key
db_url
. It is safe to assume that this key is present.
-
caches
¶ The cache configuration, from
caches
as well as the deprecatedbuildCacheSize
andchangeCacheSize
parameters.The keys
Builds
andCaches
are always available; other keys should useconfig.caches.get(cachename, 1)
.
-
schedulers
¶ The dictionary of scheduler instances, by name, from
schedulers
.
-
builders
¶ The list of
BuilderConfig
instances frombuilders
. Builders specified as dictionaries in the configuration file are converted to instances.
-
change_sources
¶ The list of
IChangeSource
providers fromchange_source
.
-
user_managers
¶ The list of user managers providers from
user_managers
.
-
services
¶ The list of additional plugin services
-
classmethod
loadFromDict
(config_dict, filename)¶ Parameters: - config_dict (dict) – The dictionary containing the configuration to load.
- filename (string) – The filename to use when reporting errors.
Returns: new
MasterConfig
instanceLoad the configuration from the given dictionary.
-
Loading of the configuration file is generally triggered by the master, using the following class:
-
class
buildbot.config.
FileLoader
¶ -
__init__
(basedir, filename)¶ Parameters: - basedir (string) – directory to which config is relative
- filename (string) – the configuration file to load
The filename is treated as relative to the basedir, if it is not absolute.
-
loadConfig
(basedir, filename)¶ Returns: new MasterConfig
instanceLoad the configuration in the given file. Aside from syntax errors, this will also detect a number of semantic errors such as multiple schedulers with the same name.
-
-
buildbot.config.
loadConfigDict
(basedir, filename)¶ Parameters: - basedir (string) – directory to which config is relative
- filename (string) – the configuration file to load
Raises: ConfigErrors
if any errors occurReturns dict: The
BuildmasterConfig
dictionary.Load the configuration dictionary in the given file.
The filename is treated as relative to the basedir, if it is not absolute.
3.2.5.1. Builder Configuration¶
-
class
buildbot.config.
BuilderConfig
([keyword args])¶ This class parameterizes configuration of builders; see Builder Configuration for its arguments. The constructor checks for errors and applies defaults, and sets the properties described here. Most are simply copied from the constructor argument of the same name.
Users may subclass this class to add defaults, for example.
-
name
¶ The builder’s name.
-
factory
¶ The builder’s factory.
-
workernames
¶ The builder’s worker names (a list, regardless of whether the names were specified with
workername
orworkernames
).
-
builddir
¶ The builder’s builddir.
-
workerbuilddir
¶ The builder’s worker-side builddir.
-
category
¶ The builder’s category.
-
nextWorker
¶ The builder’s nextWorker callable.
-
nextBuild
¶ The builder’s nextBuild callable.
-
canStartBuild
¶ The builder’s canStartBuild callable.
-
locks
¶ The builder’s locks.
-
env
¶ The builder’s environment variables.
-
properties
¶ The builder’s properties, as a dictionary.
-
collapseRequests
¶ The builder’s collapseRequests callable.
-
description
¶ The builder’s description, displayed in the web status.
-
3.2.5.2. Error Handling¶
If any errors are encountered while loading the configuration buildbot.config.error
should be called. This can occur both in the configuration-loading code,
and in the constructors of any objects that are instantiated in the
configuration - change sources, workers, schedulers, build steps, and so on.
-
buildbot.config.
error
(error)¶ Parameters: error – error to report Raises: ConfigErrors
if called at build-timeThis function reports a configuration error. If a config file is being loaded, then the function merely records the error, and allows the rest of the configuration to be loaded. At any other time, it raises
ConfigErrors
. This is done so all config errors can be reported, rather than just the first.
-
exception
buildbot.config.
ConfigErrors
([errors])¶ Parameters: errors (list) – errors to report This exception represents errors in the configuration. It supports reporting multiple errors to the user simultaneously, e.g., when several consistency checks fail.
-
errors
¶ A list of detected errors, each given as a string.
-
addError
(msg)¶ Parameters: msg (string) – the message to add Add another error message to the (presumably not-yet-raised) exception.
-
3.2.6. Configuration in AngularJS¶
The AngularJS frontend often needs access to the local master configuration. This is accomplished automatically by converting various pieces of the master configuration to a dictionary.
The IConfigured
interface represents a way to convert any object into a JSON-able dictionary.
-
class
buildbot.interfaces.
IConfigured
¶ - Providers of this interface provide a method to get their configuration as a dictionary:
-
getConfigDict
()¶ returns: object Return the configuration of this object. Note that despite the name, the return value may not be a dictionary.
Any object can be “cast” to an
IConfigured
provider. ThegetConfigDict
method for basic Python objects simply returns the value.IConfigured(someObject).getConfigDict()
-
-
class
buildbot.util.
ConfiguredMixin
¶ This class is a basic implementation of
IConfigured
. ItsgetConfigDict
method simply returns the instance’sname
attribute.-
name
¶ Each object configured must have a
name
attribute.
-
getConfigDict
(self)¶ Returns: object Return a config dictionary representing this object.
-
All of this is used by to serve /config.js
to the JavaScript frontend.
3.2.6.1. Reconfiguration¶
When the buildmaster receives a signal to begin a reconfig, it re-reads the
configuration file, generating a new MasterConfig
instance, and
then notifies all of its child services via the reconfig mechanism described
below. The master ensures that at most one reconfiguration is taking place at
any time.
See Master Organization for the structure of the Buildbot service tree.
To simplify initialization, a reconfiguration is performed immediately on
master startup. As a result, services only need to implement their
configuration handling once, and can use startService
for initialization.
See below for instructions on implementing configuration of common types of components in Buildbot.
Note
Because Buildbot uses a pure-Python configuration file, it is not possible to support all forms of reconfiguration. In particular, when the configuration includes custom subclasses or modules, reconfiguration can turn up some surprising behaviors due to the dynamic nature of Python. The reconfig support in Buildbot is intended for “intermediate” uses of the software, where there are fewer surprises.
Reconfigurable Services¶
Instances which need to be notified of a change in configuration should be
implemented as Twisted services, and mix in the
ReconfigurableServiceMixin
class, overriding the
reconfigServiceWithBuildbotConfig
method.
-
class
buildbot.config.
ReconfigurableServiceMixin
¶ -
reconfigServiceWithBuildbotConfig
(new_config)¶ Parameters: new_config ( MasterConfig
) – new master configurationReturns: Deferred This method notifies the service that it should make any changes necessary to adapt to the new configuration values given.
This method will be called automatically after a service is started.
It is generally too late at this point to roll back the reconfiguration, so if possible any errors should be detected in the
MasterConfig
implementation. Errors are handled as best as possible and communicated back to the top level invocation, but such errors may leave the master in an inconsistent state.ConfigErrors
exceptions will be displayed appropriately to the user on startup.Subclasses should always call the parent class’s implementation. For
MultiService
instances, this will call any child services’reconfigService
methods, as appropriate. This will be done sequentially, such that the Deferred from one service must fire before the next service is reconfigured.
-
priority
¶ Child services are reconfigured in order of decreasing priority. The default priority is 128, so a service that must be reconfigured before others should be given a higher priority.
-
Change Sources¶
When reconfiguring, there is no method by which Buildbot can determine that a
new ChangeSource
represents the same source
as an existing ChangeSource
, but with
different configuration parameters. As a result, the change source manager
compares the lists of existing and new change sources using equality, stops any
existing sources that are not in the new list, and starts any new change
sources that do not already exist.
ChangeSource
inherits
ComparableMixin
, so change sources are compared
based on the attributes described in their compare_attrs
.
If a change source does not make reference to any global configuration
parameters, then there is no need to inherit
ReconfigurableServiceMixin
, as a simple comparison and
startService
and stopService
will be sufficient.
If the change source does make reference to global values, e.g., as default
values for its parameters, then it must inherit
ReconfigurableServiceMixin
to support the case where the global
values change.
Schedulers¶
Schedulers have names, so Buildbot can determine whether a scheduler has been
added, removed, or changed during a reconfig. Old schedulers will be stopped,
new schedulers will be started, and both new and existing schedulers will see a
call to reconfigService
, if such a
method exists. For backward compatibility, schedulers which do not support
reconfiguration will be stopped, and the new scheduler started, when their
configuration changes.
If, during a reconfiguration, a new and old scheduler’s fully qualified class
names differ, then the old class will be stopped and the new class started.
This supports the case when a user changes, for example, a Nightly
scheduler to a Periodic
scheduler without changing the name.
Because Buildbot uses BaseScheduler
instances directly in the configuration file, a reconfigured scheduler must
extract its new configuration information from another instance of itself.
Custom Subclasses¶
Custom subclasses are most often defined directly in the configuration file, or
in a Python module that is reloaded with reload
every time the
configuration is loaded. Because of the dynamic nature of Python, this creates
a new object representing the subclass every time the configuration is loaded
– even if the class definition has not changed.
Note that if a scheduler’s class changes in a reconfig, but the scheduler’s name does not, it will still be treated as a reconfiguration of the existing scheduler. This means that implementation changes in custom scheduler subclasses will not be activated with a reconfig. This behavior avoids stopping and starting such schedulers on every reconfig, but can make development difficult.
One workaround for this is to change the name of the scheduler before each reconfig - this will cause the old scheduler to be stopped, and the new scheduler (with the new name and class) to be started.
Workers¶
Similar to schedulers, workers are specified by name, so new and old
configurations are first compared by name, and any workers to be added or
removed are noted. Workers for which the fully-qualified class name has changed
are also added and removed. All workers have their
reconfigService
method called.
This method takes care of the basic worker attributes, including changing the PB
registration if necessary. Any subclasses that add configuration parameters
should override reconfigService
and
update those parameters. As with Schedulers, because the
AbstractWorker
instance is given directly
in the configuration, on reconfig instances must extract the configuration from
a new instance.
User Managers¶
Since user managers are rarely used, and their purpose is unclear, they are always stopped and re-started on every reconfig. This may change in figure versions.
Status Receivers¶
At every reconfig, all status listeners are stopped and new versions started.