3.3.7. Utilities

Several small utilities are available at the top-level buildbot.util package.

buildbot.util.naturalSort(list)

Parameters:: list – list of strings
Returns:: sorted strings

This function sorts strings “naturally”, with embedded numbers sorted numerically. This ordering is good for objects which might have a numeric suffix, e.g., winworker1, winworker2

buildbot.util.formatInterval(interval)

Parameters:: interval – duration in seconds
Returns:: human-readable (English) equivalent

This function will return a human-readable string describing a length of time, given a number of seconds.

class buildbot.util.ComparableMixin

This mixin class adds comparability to a subclass. Use it like this:

class Widget(FactoryProduct, ComparableMixin):
    compare_attrs = ( 'radius', 'thickness' )
    # ...

Any attributes not in compare_attrs will not be considered when comparing objects. This is used to implement Buildbot’s reconfig logic, where a comparison between the new and existing objects is used to determine whether the new object should replace the existing object. If the comparison shows the objects to be equivalent, then the old object is left in place. If they differ, the old object is removed from the buildmaster, and the new object is added.

For use in configuration objects (schedulers, changesources, etc.), include any attributes which are set in the constructor based on the user’s configuration. Be sure to also include the superclass’s list, e.g.:

class MyScheduler(base.BaseScheduler):
    compare_attrs = base.BaseScheduler.compare_attrs + ('arg1', 'arg2')

A point to note is that the compare_attrs list is cumulative; that is, when a subclass also has a compare_attrs and the parent class has a compare_attrs, the subclass’ compare_attrs also includes the parent class’ compare_attrs.

This class also implements the buildbot.interfaces.IConfigured interface. The configuration is automatically generated, being the dict of all compare_attrs.

buildbot.util.safeTranslate(str)

Parameters:: str – input string
Returns:: safe version of the input

This function will filter out some inappropriate characters for filenames; it is suitable for adapting strings from the configuration for use as filenames. It is not suitable for use with strings from untrusted sources.

buildbot.util.epoch2datetime(epoch)

Parameters:: epoch – an epoch time (integer)
Returns:: equivalent datetime object

Convert a UNIX epoch timestamp to a Python datetime object, in the UTC timezone. Note that timestamps specify UTC time (modulo leap seconds and a few other minor details). If the argument is None, returns None.

buildbot.util.datetime2epoch(datetime)

Parameters:: datetime – a datetime object
Returns:: equivalent epoch time (integer)

Convert an arbitrary Python datetime object into a UNIX epoch timestamp. If the argument is None, returns None.

buildbot.util.UTC

A datetime.tzinfo subclass representing UTC time. A similar class has finally been added to Python in version 3.2, but the implementation is simple enough to include here. This is mostly used in tests to create timezone-aware datetime objects in UTC:

dt = datetime.datetime(1978, 6, 15, 12, 31, 15, tzinfo=UTC)

buildbot.util.diffSets(old, new)

Parameters:

old (set or iterable) – old set
new (set or iterable) – new set

Returns:

a (removed, added) tuple

This function compares two sets of objects, returning elements that were added and elements that were removed. This is largely a convenience function for reconfiguring services.

buildbot.util.makeList(input)

Parameters:: input – a thing
Returns:: a list of zero or more things

This function is intended to support the many places in Buildbot where the user can specify either a string or a list of strings, but the implementation wishes to always consider lists. It converts any string to a single-element list, None to an empty list, and any iterable to a list. Input lists are copied, avoiding aliasing issues.

buildbot.util.now()

Returns:: epoch time (integer)

Return the current time, using either reactor.seconds or time.time().

buildbot.util.flatten(list[, types])

Parameters:

list – potentially nested list
types – An optional iterable of the types to flatten. By default, if unspecified, this flattens both lists and tuples

Returns:

flat list

Flatten nested lists into a list containing no other lists. For example:

>>> flatten([ [  1, 2 ], 3, [ [ 4 ], 5 ] ])
[ 1, 2, 3, 4, 5 ]

Both lists and tuples are looked at by default.

buildbot.util.flattened_iterator(list[, types])

Parameters:

list – potentially nested list
types – An optional iterable of the types to flatten. By default, if unspecified, this flattens both lists and tuples.

Returns:

iterator over every element whose type isn’t in types

Returns a generator that doesn’t yield any lists/tuples. For example:

   >>> for x in flattened_iterator([ [  1, 2 ], 3, [ [ 4 ] ] ]):
   >>>     print x
   1
   2
   3
   4

Use this for extremely large lists to keep memory-usage down and improve performance when
you only need to iterate once.

buildbot.util.none_or_str(obj)

Parameters:: obj – input value
Returns:: string or None

If obj is not None, return its string representation.

buildbot.util.bytes2unicode(bytestr, encoding='utf-8', errors='strict')

Parameters:

bytestr – bytes
encoding – unicode encoding to pass to str.encode, default utf-8.
errors – error handler to pass to str.encode, default strict.

Returns:

string as unicode

This function is intended to convert bytes to unicode for user convenience. If given a bytestring, it returns the string decoded using encoding. If given a unicode string, it returns it directly.

buildbot.util.string2boolean(str)

Parameters:: str – string
Raises:: KeyError –
Returns:: boolean

This function converts a string to a boolean. It is intended to be liberal in what it accepts: case-insensitive “true”, “on”, “yes”, “1”, etc. It raises KeyError if the value is not recognized.

buildbot.util.toJson(obj)

Parameters:: obj – object
Returns:: UNIX epoch timestamp

This function is a helper for json.dump, that allows to convert non-json able objects to json. For now it supports converting datetime.datetime objects to unix timestamp.

buildbot.util.NotABranch

This is a sentinel value used to indicate that no branch is specified. It is necessary since schedulers and change sources consider None a valid name for a branch. This is generally used as a default value in a method signature, and then tested against with is:

if branch is NotABranch:
    pass # ...

buildbot.util.in_reactor(fn): This decorator will cause the wrapped function to be run in the Twisted reactor, with the reactor stopped when the function completes. It returns the result of the wrapped function. If the wrapped function fails, its traceback will be printed, the reactor halted, and None returned.

buildbot.util.asyncSleep(secs, reactor=None): Yield a deferred that will fire with no result after secs seconds. This is the asynchronous equivalent to time.sleep, and can be useful in tests. In case a custom reactor is used, the reactor parameter may be set. By default, twisted.internet.reactor is used.

buildbot.util.stripUrlPassword(url)

Parameters:: url – a URL
Returns:: URL with any password component replaced with xxxx

Sanitize a URL; use this before logging or displaying a DB URL.

buildbot.util.join_list(maybe_list)

Parameters:: maybe_list – list, tuple, byte string, or unicode
Returns:: unicode string

If maybe_list is a list or tuple, join it with spaces, casting any strings into unicode using bytes2unicode. This is useful for configuration parameters that may be strings or lists of strings.

class buildbot.util.Notifier

This is a helper for firing multiple deferreds with the same result.

wait(): Return a deferred that will fire when when the notifier is notified.

notify(value): Fire all the outstanding deferreds with the given value.

buildbot.util.giturlparse(url)

Parameters:: url – a git url
Returns:: a GitUrl with results of parsed url

This function is intended to help various components to parse git urls. It helps to find the <owner>/<repo> of a git repository url coming from a change, in order to call urls. owner and repo is a common scheme for identifying a git repository between various git hosting services, like GitHub, GitLab, BitBucket, etc. Each service has their own naming for similar things, but we choose to use the GitHub naming as a de-facto standard. To simplify implementation, the parser is accepting invalid urls, but it should always parse valid urls correctly. The unit tests in test_util_giturlparse.py are the references on what the parser accepts. Please feel free to update the parser and the unit tests.

Example use:

from buildbot.util import giturlparse
repourl = giturlparse(sourcestamp['repository'])
repoOwner = repourl.owner
repoName = repourl.repo

class buildbot.util.GitUrl

proto: The protocol of the url

user: The user of the url (as in user@domain)

domain: The domain part of the url

port: The optional port of the url

owner: The owner of the repository (in case of GitLab might be a nested group, i.e contain /, e.g repo/subrepo/subsubrepo)

repo: The name of the repository (in case of GitLab might be a nested group, i.e contain /)

3.3.7.1. `buildbot.util.lru`

class buildbot.util.lru.LRUCache(miss_fn, max_size=50)

Parameters:

miss_fn – function to call, with key as parameter, for cache misses. The function should return the value associated with the key argument, or None if there is no value associated with the key.
max_size – maximum number of objects in the cache.

This is a simple least-recently-used cache. When the cache grows beyond the maximum size, the least-recently used items will be automatically removed from the cache.

This cache is designed to control memory usage by minimizing duplication of objects, while avoiding unnecessary re-fetching of the same rows from the database.

All values are also stored in a weak valued dictionary, even after they have expired from the cache. This allows values that are used elsewhere in Buildbot to “stick” in the cache in case they are needed by another component. Weak references cannot be used for some types, so these types are not compatible with this class. Note that dictionaries can be weakly referenced if they are an instance of a subclass of dict.

If the result of the miss_fn is None, then the value is not cached; this is intended to avoid caching negative results.

This is based on Raymond Hettinger’s implementation, licensed under the PSF license, which is GPL-compatible.

hits: cache hits so far

refhits: cache misses found in the weak ref dictionary, so far

misses: cache misses leading to re-fetches, so far

max_size: maximum allowed size of the cache

get(key, **miss_fn_kwargs)

Parameters:

key – cache key
miss_fn_kwargs – keyword arguments to the miss_fn

Returns:

value via Deferred

Fetch a value from the cache by key, invoking miss_fn(key, **miss_fn_kwargs) if the key is not in the cache.

Any additional keyword arguments are passed to the miss_fn as keyword arguments; these can supply additional information relating to the key. It is up to the caller to ensure that this information is functionally identical for each key value: if the key is already in the cache, the miss_fn will not be invoked, even if the keyword arguments differ.

put(key, value)

Parameters:

key – key at which to place the value
value – value to place there

Add the given key and value into the cache. The purpose of this method is to insert a new value into the cache without invoking the miss_fn (e.g., to avoid unnecessary overhead).

inv(): Check invariants on the cache. This is intended for debugging purposes.

class buildbot.util.lru.AsyncLRUCache(miss_fn, max_size=50)

Parameters:

miss_fn – This is the same as the miss_fn for class LRUCache, with the difference that this function must return a Deferred.
max_size – maximum number of objects in the cache.

This class has the same functional interface as LRUCache, but asynchronous locking is used to ensure that in the common case of multiple concurrent requests for the same key, only one fetch is performed.

3.3.7.2. `buildbot.util.bbcollections`

This package provides a few useful collection objects.

Note

This module used to be named collections, but without absolute imports (PEP 328), this precluded using the standard library’s collections module.

class buildbot.util.bbcollections.defaultdict: This is a clone of the Python collections.defaultdict for use in Python-2.4. In later versions, this is simply a reference to the built-in defaultdict, so Buildbot code can simply use buildbot.util.collections.defaultdict everywhere.

class buildbot.util.bbcollections.KeyedSets

This is a collection of named sets. In principle, it contains an empty set for every name, and you can add things to sets, discard things from sets, and so on.

>>> ks = KeyedSets()
>>> ks['tim']                   # get a named set
set([])
>>> ks.add('tim', 'friendly')   # add an element to a set
>>> ks.add('tim', 'dexterous')
>>> ks['tim']
set(['friendly', 'dexterous'])
>>> 'tim' in ks                 # membership testing
True
>>> 'ron' in ks
False
>>> ks.discard('tim', 'friendly')# discard set element
>>> ks.pop('tim')               # return set and reset to empty
set(['dexterous'])
>>> ks['tim']
set([])

This class is careful to conserve memory space - empty sets do not occupy any space.

3.3.7.3. `buildbot.util.eventual`

This function provides a simple way to say “please do this later”. For example

from buildbot.util.eventual import eventually
def do_what_I_say(what, where):
    # ...
    return d
eventually(do_what_I_say, "clean up", "your bedroom")

The package defines “later” as “next time the reactor has control”, so this is a good way to avoid long loops that block another activity in the reactor.

buildbot.util.eventual.eventually(cb, *args, **kwargs)

Parameters:

cb – callable to invoke later
args – args to pass to cb
kwargs – kwargs to pass to cb

Invoke the callable cb in a later reactor turn.

Callables given to eventually are guaranteed to be called in the same order as the calls to eventually – writing eventually(a); eventually(b) guarantees that a will be called before b.

Any exceptions that occur in the callable will be logged with log.err(). If you really want to ignore them, provide a callable that catches those exceptions.

This function returns None. If you care to know when the callable was run, be sure to provide a callable that notifies somebody.

buildbot.util.eventual.fireEventually(value=None)

Parameters:: value – value with which the Deferred should fire
Returns:: Deferred

This function returns a Deferred which will fire in a later reactor turn, after the current call stack has been completed, and after all other Deferreds previously scheduled with eventually. The returned Deferred will never fail.

buildbot.util.eventual.flushEventualQueue()

Returns:: Deferred

This returns a Deferred which fires when the eventual-send queue is finally empty. This is useful for tests and other circumstances where it is useful to know that “later” has arrived.

3.3.7.4. `buildbot.util.debounce`

It’s often necessary to perform some action in response to a particular type of event. For example, steps need to update their status after updates arrive from the worker. However, when many events arrive in quick succession, it’s more efficient to only perform the action once, after the last event has occurred.

The debounce.method(wait) decorator is the tool for the job.

buildbot.util.debounce.method(wait, get_reactor)

Parameters:

wait – time to wait before invoking, in seconds
get_reactor – A callable that takes the underlying instance and returns the reactor to use. Defaults to instance.master.reactor.

Returns a decorator that debounces the underlying method. The underlying method must take no arguments (except self).

For each call to the decorated method, the underlying method will be invoked at least once within wait seconds (plus the time the method takes to execute). Calls are “debounced” during that time, meaning that multiple calls to the decorated method will result in a single invocation.

Note

This functionality is similar to Underscore’s debounce, except that the Underscore method resets its timer on every call.

The decorated method is an instance of Debouncer, allowing it to be started and stopped. This is useful when the method is a part of a Buildbot service: call method.start() from startService and method.stop() from stopService, handling its Deferred appropriately.

class buildbot.util.debounce.Debouncer

stop()

Returns:: Deferred

Stop the debouncer. While the debouncer is stopped, calls to the decorated method will be ignored. If a call is pending when stop is called, that call will occur immediately. When the Deferred that stop returns fires, the underlying method is not executing.

start(): Start the debouncer. This reverses the effects of stop. This method can be called on a started debouncer without issues.

3.3.7.5. `buildbot.util.poll`

Many Buildbot services perform some periodic, asynchronous operation. Change sources, for example, contact the repositories they monitor on a regular basis. The tricky bit is, the periodic operation must complete before the service stops.

The @poll.method decorator makes this behavior easy and reliable.

buildbot.util.poll.method()

This decorator replaces the decorated method with a Poller instance configured to call the decorated method periodically. The poller is initially stopped, so periodic calls will not begin until its start method is called. The start polling interval is specified when the poller is started. A random delay may optionally be supplied. This allows to avoid the situation of multiple services with the same interval are executing at exactly the same time.

If the decorated method fails or raises an exception, the Poller logs the error and re-schedules the call for the next interval.

If a previous invocation of the method has not completed when the interval expires, then the next invocation is skipped and the interval timer starts again.

A common idiom is to call start and stop from startService and stopService:

class WatchThings(object):

    @poll.method
    def watch(self):
        d = self.beginCheckingSomething()
        return d

    def startService(self):
        self.watch.start(interval=self.pollingInterval, now=False)

    def stopService(self):
        return self.watch.stop()

class buildbot.util.poll.Poller

start(interval=N, now=False, random_delay_min=0, random_delay_max=0)

Parameters:

interval – time, in seconds, between invocations
now – if true, call the decorated method immediately on startup.
random_delay_min – Minimum random delay to apply to the start time of the decorated method.
random_delay_min – Maximum random delay to apply to the start time of the decorated method.

Start the poller.

stop()

Returns:: Deferred

Stop the poller. The returned Deferred fires when the decorated method is complete.

__call__(): Force a call to the decorated method now. If the decorated method is currently running, another call will begin as soon as it completes unless the poller is currently stopping.

3.3.7.6. `buildbot.util.maildir`

Several Buildbot components make use of maildirs to hand off messages between components. On the receiving end, there’s a need to watch a maildir for incoming messages and trigger some action when one arrives.

class buildbot.util.maildir.MaildirService(basedir)

param basedir:

(optional) base directory of the maildir

A MaildirService instance watches a maildir for new messages. It should be a child service of some MultiService instance. When running, this class uses the linux dirwatcher API (if available) or polls for new files in the ‘new’ maildir subdirectory. When it discovers a new message, it invokes its messageReceived method.

To use this class, subclass it and implement a more interesting messageReceived function.

setBasedir(basedir)

Parameters:: basedir – base directory of the maildir

If no basedir is provided to the constructor, this method must be used to set the basedir before the service starts.

messageReceived(filename)

Parameters:: filename – unqualified filename of the new message

This method is called with the short filename of the new message. The full name of the new file can be obtained with os.path.join(maildir, 'new', filename). The method is un-implemented in the MaildirService class, and must be implemented in subclasses.

moveToCurDir(filename)

Parameters:: filename – unqualified filename of the new message
Returns:: open file object

Call this from messageReceived to start processing the message; this moves the message file to the ‘cur’ directory and returns an open file handle for it.

3.3.7.7. `buildbot.util.misc`

buildbot.util.misc.deferredLocked(lock)

Parameters:: lock – a twisted.internet.defer.DeferredLock instance or a string naming an instance attribute containing one

This is a decorator to wrap an event-driven method (one returning a Deferred) in an acquire/release pair of a designated DeferredLock. For simple functions with a static lock, this is as easy as:

someLock = defer.DeferredLock()

@util.deferredLocked(someLock)
def someLockedFunction():
    # ..
    return d

For class methods which must access a lock that is an instance attribute, the lock can be specified by a string, which will be dynamically resolved to the specific instance at runtime:

def __init__(self):
    self.someLock = defer.DeferredLock()

@util.deferredLocked('someLock')
def someLockedFunction():
    # ..
    return d

buildbot.util.misc.cancelAfter(seconds, deferred)

Parameters:

seconds – timeout in seconds
deferred – deferred to cancel after timeout expires

Returns:

the deferred passed to the function

Cancel the given deferred after the given time has elapsed, if it has not already been fired. When this occurs, the deferred’s errback will be fired with a twisted.internet.defer.CancelledError failure.

3.3.7.8. `buildbot.util.netstrings`

Similar to maildirs, netstrings are used occasionally in Buildbot to encode data for interchange. While Twisted supports a basic netstring receiver protocol, it does not have a simple way to apply that to a non-network situation.

class buildbot.util.netstrings.NetstringParser

This class parses strings piece by piece, either collecting the accumulated strings or invoking a callback for each one.

feed(data)

Parameters:: data – a portion of netstring-formatted data
Raises:: twisted.protocols.basic.NetstringParseError

Add arbitrarily-sized data to the incoming-data buffer. Any complete netstrings will trigger a call to the stringReceived method.

Note that this method (like the Twisted class it is based on) cannot detect a trailing partial netstring at EOF - the data will be silently ignored.

stringReceived(string):

Parameters:: string – the decoded string

This method is called for each decoded string as soon as it is read completely. The default implementation appends the string to the strings attribute, but subclasses can do anything.

strings: The strings decoded so far, if stringReceived is not overridden.

3.3.7.9. `buildbot.util.sautils`

This module contains a few utilities that are not included with SQLAlchemy.

class buildbot.util.sautils.InsertFromSelect(table, select)

Parameters:

table – table into which insert should be performed
select – select query from which data should be drawn

This class is taken directly from SQLAlchemy’s compiler.html, and allows a Pythonic representation of INSERT INTO .. SELECT .. queries.

buildbot.util.sautils.sa_version(): Return a 3-tuple representing the SQLAlchemy version. Note that older versions that did not have a __version__ attribute are represented by (0,0,0).

3.3.7.10. `buildbot.util.pathmatch`

class buildbot.util.pathmatch.Matcher

This class implements the path-matching algorithm used by the data API.

Patterns are tuples of strings, with strings beginning with a colon (:) denoting variables. A character can precede the colon to indicate the variable type:

i specifies an identifier (identifier).
n specifies a number (parseable by int).

A tuple of strings matches a pattern if the lengths are identical, every variable matches and has the correct type, and every non-variable pattern element matches exactly.

A matcher object takes patterns using dictionary-assignment syntax:

ep = ChangeEndpoint()
matcher[('change', 'n:changeid')] = ep

and performs matching using the dictionary-lookup syntax:

changeEndpoint, kwargs = matcher[('change', '13')]
# -> (ep, {'changeid': 13})

where the result is a tuple of the original assigned object (the Change instance in this case) and the values of any variables in the path.

iterPatterns(): Returns an iterator which yields all patterns in the matcher as tuples of (pattern, endpoint).

3.3.7.11. `buildbot.util.topicmatch`

class buildbot.util.topicmatch.TopicMatcher(topics)

Parameters:: topics (list) – topics to match

This class implements the AMQP-defined syntax: routing keys are treated as dot-separated sequences of words and matched against topics. A star (*) in the topic will match any single word, while an octothorpe (#) will match zero or more words.

matches(routingKey)

Parameters:: routingKey (string) – routing key to examine
Returns:: True if the routing key matches a topic

3.3.7.12. `buildbot.util.subscription`

The classes in the buildbot.util.subscription module are used for master-local subscriptions. In the near future, all uses of this module will be replaced with message-queueing implementations that allow subscriptions and subscribers to span multiple masters.

3.3.7.13. `buildbot.util.croniter`

(deprecated)

This module is a copy of https://github.com/taichino/croniter, and provides support for converting cron-like time specifications to actual times.

3.3.7.14. `buildbot.util.state`

The classes in the buildbot.util.state module are used for dealing with object state stored in the database.

class buildbot.util.state.StateMixin

This class provides helper methods for accessing the object state stored in the database.

name: This must be set to the name to be used to identify this object in the database.

master: This must point to the BuildMaster object.

getState(name, default)

Parameters:

name – name of the value to retrieve
default – (optional) value to return if name is not present

Returns:

state value via a Deferred

Raises:

KeyError – if name is not present and no default is given
TypeError – if JSON parsing fails

Get a named state value from the object’s state.

setState(name, value)

Parameters:

name – the name of the value to change
value – the value to set - must be a JSONable object
returns – Deferred

Raises:

TypeError – if JSONification fails

Set a named state value in the object’s persistent state. Note that value must be json-able.

3.3.7.15. `buildbot.util.identifiers`

This module makes it easy to manipulate identifiers.

buildbot.util.identifiers.isIdentifier(maxLength, object)

Parameters:

maxLength – maximum length of the identifier
object – object to test for identifier-ness

Returns:

boolean

Is object a identifier?

buildbot.util.identifiers.forceIdentifier(maxLength, str)

Parameters:

maxLength – maximum length of the identifier
str – string to coerce to an identifier

Returns:

identifier of maximum length maxLength

Coerce a string (assuming UTF-8 for bytestrings) into an identifier. This method will replace any invalid characters with _ and truncate to the given length.

buildbot.util.identifiers.incrementIdentifier(maxLength, str)

Parameters:

maxLength – maximum length of the identifier
str – identifier to increment

Returns:

identifier of maximum length maxLength

Raises:

ValueError if no suitable identifier can be constructed

“Increment” an identifier by adding a numeric suffix, while keeping the total length limited. This is useful when selecting a unique identifier for an object. Maximum-length identifiers like _999999 cannot be incremented and will raise ValueError.

3.3.7.16. `buildbot.util.lineboundaries`

class buildbot.util.lineboundaries.LineBoundaryFinder

This class accepts a sequence of arbitrary strings and computes newline-terminated substrings. Input strings are accepted in append function, and newline-terminated substrings are returned.

The class buffers any partial lines until a subsequent newline is seen. It considers any of \r, \n, and \r\n to be newlines. Because of the ambiguity of an append operation ending in the character \r (it may be a bare \r or half of \r\n), the last line of such an append operation will be buffered until the next append or flush.

append(text)

Parameters:: text – text to append to the boundary finder
Returns:: a newline-terminated substring or None

Add additional text to the boundary finder. If the addition of this text completes at least one line, as many complete lines as possible are selected as a result. If no lines are completed, the result will be None.

flush()

Returns:: a newline-terminated substring or None

Flush any remaining partial line by adding a newline.

3.3.7.17. `buildbot.util.service`

This module implements some useful subclasses of Twisted services.

The first two classes are more robust implementations of two Twisted classes, and should be used universally in Buildbot code.

class buildbot.util.service.AsyncMultiService

This class is similar to twisted.application.service.MultiService, except that it handles Deferreds returned from child services startService and stopService methods.

Twisted’s service implementation does not support asynchronous startService methods. The reasoning is that all services should start at process startup, with no need to coordinate between them. For Buildbot, this is not sufficient. The framework needs to know when startup has completed, so it can begin scheduling builds. This class implements the desired functionality, with a parent service’s startService returning a Deferred which will only fire when all child services startService methods have completed.

This class also fixes a bug with Twisted’s implementation of stopService which ignores failures in the stopService process. With AsyncMultiService, any errors in a child’s stopService will be propagated to the parent’s stopService method.

class buildbot.util.service.AsyncService: This class is similar to twisted.application.service.Service, except that its setServiceParent method will return a Deferred. That Deferred will fire after the startService method has completed, if the service was started because the new parent was already running.

Some services in buildbot must have only one “active” instance at any given time. In a single-master configuration, this requirement is trivial to maintain. In a multiple-master configuration, some arbitration is required to ensure that the service is always active on exactly one master in the cluster.

For example, a particular daily scheduler could be configured on multiple masters, but only one of them should actually trigger the required builds.

class buildbot.util.service.ClusteredService

A base class for a service that must have only one “active” instance in a buildbot configuration.

Each instance of the service is started and stopped via the usual twisted startService and stopService methods. This utility class hooks into those methods in order to run an arbitration strategy to pick the one instance that should actually be “active”.

The arbitration strategy is implemented via a polling loop. When each service instance starts, it immediately offers to take over as the active instance (via _claimService).

If successful, the activate method is called. Once active, the instance remains active until it is explicitly stopped (eg, via stopService) or otherwise fails. When this happens, the deactivate method is invoked and the “active” status is given back to the cluster (via _unclaimService).

If another instance is already active, this offer fails, and the instance will poll periodically to try again. The polling strategy helps guard against active instances that might silently disappear and leave the service without any active instance running.

Subclasses should use these methods to hook into this activation scheme:

activate(): When a particular instance of the service is chosen to be the one “active” instance, this method is invoked. It is the corollary to twisted’s startService.

deactivate(): When the one “active” instance must be deactivated, this method is invoked. It is the corollary to twisted’s stopService.

isActive(): Returns whether this particular instance is the active one.

The arbitration strategy is implemented via the following required methods:

_getServiceId(): The “service id” uniquely represents this service in the cluster. Each instance of this service must have this same id, which will be used in the arbitration to identify candidates for activation. This method may return a Deferred.

_claimService(): An instance is attempting to become the one active instance in the cluster. This method must return True or False (optionally via a Deferred) to represent whether this instance’s offer to be the active one was accepted. If this returns True, the activate method will be called for this instance.

_unclaimService(): Surrender the “active” status back to the cluster and make it available for another instance. This will only be called on an instance that successfully claimed the service and has been activated and after its deactivate has been called. Therefore, in this method it is safe to reassign the “active” status to another instance. This method may return a Deferred.

class buildbot.util.service.SharedService

This class implements a generic Service that needs to be instantiated only once according to its parameters. It is a common use case to need this for accessing remote services. Having a shared service allows to limit the number of simultaneous access to the same remote service. Thus, several completely independent Buildbot services can use that SharedService to access the remote service, and automatically synchronize themselves to not overwhelm it.

__init__(self, *args, **kwargs)

Constructor of the service.

Note that unlike BuildbotService, SharedService is not reconfigurable and uses the classical constructor method.

Reconfigurability would mean to add some kind of reference counting of the users, which will make the design much more complicated to use. This means that the SharedService will not be destroyed when there is no more users, it will be destroyed at the master’s stopService It is important that those SharedService life cycles are properly handled. Twisted will indeed wait for any thread pool to finish at master stop, which will not happen if the thread pools are not properly closed.

The lifecycle of the SharedService is the same as a service, it must implement startService and stopService in order to allocate and free its resources.

getName(cls, *args, **kwargs): Class method. Takes same arguments as the constructor of the service. Get a unique name for that instance of a service. This returned name is the key inside the parent’s service dictionary that is used to decide if the instance has already been created before or if there is a need to create a new object. Default implementation will hash args and kwargs and use <classname>_<hash> as the name.

getService(cls, parentService, *args, **kwargs)

Parameters:: parentService – an AsyncMultiService where to lookup and register the SharedService (usually the root service, the master)
Returns:: instance of the service via Deferred

Class method. Takes same arguments as the constructor of the service (plus the parentService at the beginning of the list). Construct an instance of the service if needed, and place it at the beginning of the parentService service list. Placing it at the beginning will guarantee that the SharedService will be stopped after the other services.

class buildbot.util.service.BuildbotService

This class is the combinations of all Service classes implemented in buildbot. It is Async, MultiService, and Reconfigurable, and designed to be eventually the base class for all buildbot services. This class makes it easy to manage (re)configured services.

The design separates the check of the config and the actual configuration/start. A service sibling is a configured object that has the same name of a previously started service. The sibling configuration will be used to configure the running service.

Service lifecycle is as follow:

Buildbot master start
Buildbot is evaluating the configuration file. BuildbotServices are created, and checkConfig() are called by the generic constructor.
If everything is fine, all services are started. BuildbotServices startService() is called, and call reconfigService() for the first time.
User reconfigures buildbot.
Buildbot is evaluating the configuration file. BuildbotServices siblings are created, and checkConfig() are called by the generic constructor.
BuildbotServiceManager is figuring out added services, removed services, unchanged services
BuildbotServiceManager calls stopService() for services that disappeared from the configuration.
BuildbotServiceManager calls startService() like in buildbot start phase for services that appeared from the configuration.
BuildbotServiceManager calls reconfigService() for the second time for services that have their configuration changed.

__init__(self, *args, **kwargs)

Constructor of the service. The constructor initializes the service, calls checkConfig() and stores the config arguments in private attributes.

This should not be overridden by subclasses, as they should rather override checkConfig.

canReconfigWithSibling(self, sibling)
This method is used to check if we are able to call
:py:func:`reconfigServiceWithSibling` with the given sibling.
If it returns `False`, we stop the old service and start a new one,
instead of attempting a reconfig.

checkConfig(self, *args, **kwargs): Please override this method to check the parameters of your config. Please use buildbot.config.error for error reporting. You can replace them *args, **kwargs by actual constructor like arguments with default args, and it have to match self.reconfigService This method is synchronous, and executed in the context of the master.cfg. Please don’t block, or use deferreds in this method. Remember that the object that runs checkConfig is not always the object that is actually started. The checked configuration can be passed to another sibling service. Any actual resource creation shall be handled in reconfigService() or startService()

reconfigService(self, *args, **kwargs)

This method is called at buildbot startup, and buildbot reconfig. *args and **kwargs are the configuration arguments passed to the constructor in master.cfg. You can replace them *args, **kwargs by actual constructor like arguments with default args, and it have to match self.checkConfig

Returns a deferred that should fire when the service is ready. Builds are not started until all services are configured.

BuildbotServices must be aware that during reconfiguration, their methods can still be called by running builds. So they should atomically switch old configuration and new configuration, so that the service is always available.

If this method raises NotImplementedError, it means the service is legacy, and do not support reconfiguration. The BuildbotServiceManager parent, will detect this, and swap old service with new service. This behaviour allow smooth transition of old code to new reconfigurable service lifecycle but shall not be used for new code.

reconfigServiceWithSibling(self, sibling): Internal method that finds the configuration bits in a sibling, an object with same class that is supposed to replace it from a new configuration. We want to reuse the service started at master startup and just reconfigure it. This method handles necessary steps to detect if the config has changed, and eventually call self.reconfigService()

renderSecrets(self, *args)

Utility method which renders a list of parameters which can be interpolated as a secret. This is meant for services which have their secrets parameter configurable as positional arguments. If there are several argument, the secrets are interpolated in parallel, and a list of result is returned via deferred. If there is one argument, the result is directly returned.

Note

For keyword arguments, a simpler method is to use the secrets class variable, whose items will be automatically interpolated just before reconfiguration.

def reconfigService(self, user, password, ...)
    user, password = yield self.renderSecrets(user, password)

def reconfigService(self, token, ...)
    token = yield self.renderSecrets(token)

secrets = ("user", "password")
def reconfigService(self, user=None, password=None, ...):
    # nothing to do; user and password will be automatically interpolated

Advanced users can derive this class to make their own services that run inside buildbot, and follow the application lifecycle of buildbot master.

Such services are singletons accessible to nearly every object in Buildbot (buildsteps, status, changesources, etc) using self.master.namedServices[‘<nameOfYourService>’].

As such, they can be used to factorize access to external services, available e.g using a REST api. Having a single service will help with caching, and rate-limiting access of those APIs.

Here is an example on how you would integrate and configure a simple service in your master.cfg:

class MyShellCommand(ShellCommand):

    def getResultSummary(self):
        # access the service attribute
        service = self.master.namedServices['myService']
        return dict(step="arg value: %d" % (service.arg1,))

class MyService(BuildbotService):
    name = "myService"

    def checkConfig(self, arg1):
        if not isinstance(arg1, int):
            config.error("arg1 must be an integer while it is %r" % (arg1,))
            return
        if arg1 < 0:
            config.error("arg1 must be positive while it is %d" % (arg1,))

    def reconfigService(self, arg1):
        self.arg1 = arg1
        return defer.succeed(None)

c['schedulers'] = [
    ForceScheduler(
        name="force",
        builderNames=["testy"])]

f = BuildFactory()
f.addStep(MyShellCommand(command='echo hei'))
c['builders'] = [
    BuilderConfig(name="testy",
                  workernames=["local1"],
                  factory=f)]

c['services'] = [
    MyService(arg1=1)
]

3.3.7.18. `buildbot.util.httpclientservice`

class buildbot.util.httpclientservice.HTTPClientService

This class implements a SharedService for doing http client access. The module automatically chooses from txrequests and treq and uses whichever is installed. It provides minimalistic API similar to the one from txrequests and treq. Having a SharedService for this allows to limits the number of simultaneous connection for the same host. While twisted application can managed thousands of connections at the same time, this is often not the case for the services buildbot controls. Both txrequests and treq use keep-alive connection polling. Lots of HTTP REST API will however force a connection close in the end of a transaction.

Note

The API described here is voluntary minimalistic, and reflects what is tested. As most of this module is implemented as a pass-through to the underlying libraries, other options can work but have not been tested to work in both backends. If there is a need for more functionality, please add new tests before using them.

static getService(master, base_url, auth=None, headers=None, debug=None, verify=None)

Parameters:

master – the instance of the master service (available in self.master for all the BuildbotService instances)
base_url – The base http url of the service to access. e.g. http://github.com/
auth – Authentication information. If auth is a tuple then BasicAuth will be used. e.g ('user', 'passwd') It can also be a requests.auth authentication plugin. In this case txrequests will be forced, and treq cannot be used.
headers – The headers to pass to every requests for this url
debug – log every requests and every response.
verify – disable the SSL verification.

Returns:

instance of :HTTPClientService

Get an instance of the SharedService. There is one instance per base_url and auth.

The constructor initialize the service, and store the config arguments in private attributes.

This should not be overridden by subclasses, as they should rather override checkConfig.

get(endpoint, params=None)

Parameters:

endpoint – endpoint. It must either be a full URL (starts with http:// or https://) or relative to the base_url (starts with /)
params – optional dictionary that will be encoded in the query part of the url (e.g. ?param1=foo)

Returns:

implementation of :IHTTPResponse via deferred

Performs a HTTP GET

delete(endpoint, params=None)

Parameters:

endpoint – endpoint. It must either be a full URL (starts with http:// or https://) or relative to the base_url (starts with /)
params – optional dictionary that will be encoded in the query part of the url (e.g. ?param1=foo)

Returns:

implementation of :IHTTPResponse via deferred

Performs a HTTP DELETE

post(endpoint, data=None, json=None, params=None)

Parameters:

endpoint – endpoint. It must either be a full URL (starts with http:// or https://) or relative to the base_url (starts with /)
data – optional dictionary that will be encoded in the body of the http requests as application/x-www-form-urlencoded
json – optional dictionary that will be encoded in the body of the http requests as application/json
params – optional dictionary that will be encoded in the query part of the url (e.g. ?param1=foo)

Returns:

implementation of :IHTTPResponse via deferred

Performs a HTTP POST

Note

json and data cannot be used at the same time.

put(endpoint, data=None, json=None, params=None)

Parameters:

endpoint – endpoint. It must either be a full URL (starts with http:// or https://) or relative to the base_url (starts with /)
data – optional dictionary that will be encoded in the body of the http requests as application/x-www-form-urlencoded
json – optional dictionary that will be encoded in the body of the http requests as application/json
params – optional dictionary that will be encoded in the query part of the url (e.g. ?param1=foo)

Returns:

implementation of :IHTTPResponse via deferred

Performs a HTTP PUT

Note

json and data cannot be used at the same time.

class buildbot.util.httpclientservice.IHTTPResponse

Note

IHTTPResponse is a subset of treq Response API described here. The API it is voluntarily minimalistic and reflects what is tested and reliable to use with the three backends (including fake). The API is a subset of the treq API, which is itself a superset of twisted IResponse API. treq is thus implemented as passthrough.

Notably:

There is no API to automatically decode content, as this is not implemented the same in both backends.
There is no API to stream content as the two libraries have very different way for doing it, and we do not see use-case where buildbot would need to transfer large content to the master.

content()

Returns:: raw (bytes) content of the response via deferred

json()

Returns:: json decoded content of the response via deferred

code

Returns:: http status code of the request’s response (e.g 200)

url

Returns:: request’s url (e.g https://api.github.com/endpoint’)

3.3.7.19. `buildbot.test.fake.httpclientservice`

class buildbot.test.fake.httpclientservice.HTTPClientService

This class implements a fake version of the buildbot.util.httpclientservice.HTTPClientService that needs to be used for testing services which need http client access. It implements the same APIs as buildbot.util.httpclientservice.HTTPClientService, plus one that should be used to register the expectations. It should be registered by the test case before the tested service actually requests an HTTPClientService instance, with the same parameters. It will then replace the original implementation automatically (no need to patch anything). The testing methodology is based on AngularJS ngMock.

getService(cls, master, case, *args, **kwargs)

Parameters:

master – the instance of a fake master service
case – a twisted.python.unittest.TestCase instance

getService returns a fake HTTPClientService, and should be used just like the regular getService.

It will make sure the original HTTPClientService is not called, and assert that all expected http requests have been described in the test case.

expect(self, method, ep, params=None, data=None, json=None, code=200, content=None, content_json=None, processing_delay_s=None)

Parameters:

method – expected HTTP method
ep – expected endpoint
params – optional expected query parameters
data – optional expected non-json data (bytes)
json – optional expected json data (dictionary or list or string)
code – optional http code that will be received
content – optional content that will be received
content_json – optional content encoded in json that will be received
processing_delay_s – optional delay that the handling of the request will take

Records an expectation of HTTP requests that will happen during the test. The order of the requests is important. All the request expectation must be defined in the test.

For example:

from twisted.internet import defer
from twisted.trial import unittest

from buildbot.test.fake import httpclientservice as fakehttpclientservice
from buildbot.util import httpclientservice
from buildbot.util import service


class myTestedService(service.BuildbotService):
    name = 'myTestedService'

    @defer.inlineCallbacks
    def reconfigService(self, baseurl):
        self._http = yield httpclientservice.HTTPClientService.getService(
            self.master, baseurl)

    @defer.inlineCallbacks
    def doGetRoot(self):
        res = yield self._http.get("/")
        # note that at this point, only the http response headers are received
        if res.code != 200:
            raise RuntimeError("%d: server did not succeed" % (res.code))
        res_json = yield res.json()
        # res.json() returns a deferred to account for the time needed to fetch the
        # entire body
        return res_json


class Test(unittest.SynchronousTestCase):

    def setUp(self):
        baseurl = 'http://127.0.0.1:8080'
        self.parent = service.MasterService()
        self._http = self.successResultOf(
            fakehttpclientservice.HTTPClientService.getService(self.parent, self,
                                                               baseurl))
        self.tested = myTestedService(baseurl)

        self.successResultOf(self.tested.setServiceParent(self.parent))
        self.successResultOf(self.parent.startService())

    def test_root(self):
        self._http.expect("get", "/", content_json={'foo': 'bar'})

        response = self.successResultOf(self.tested.doGetRoot())
        self.assertEqual(response, {'foo': 'bar'})

    def test_root_error(self):
        self._http.expect("get", "/", content_json={'foo': 'bar'}, code=404)

        response = self.failureResultOf(self.tested.doGetRoot())
        self.assertEqual(response.getErrorMessage(), '404: server did not succeed')

3.3.7.20. `buildbot.util.ssl`

This module is a copy of twisted.internet.ssl except it won’t crash with ImportError if pyopenssl is not installed. If you need to use twisted.internet.ssl, please instead use buildbot.util.ssl, and call ssl.ensureHasSSL in checkConfig to provide helpful message to the user, only if they enabled SSL for your plugin.

buildbot.util.ssl.ensureHasSSL(plugin_name)

Parameters:: plugin_name – name of the plugin. Usually self.__class__.__name__

Call this function to provide helpful config error to the user in case of OpenSSL not installed.

buildbot.util.ssl.skipUnless(f)

Parameters:: f – decorated test

Test decorator which will skip the test if OpenSSL is not installed.

3.3.7. Utilities

3.3.7.1. buildbot.util.lru

3.3.7.2. buildbot.util.bbcollections

3.3.7.3. buildbot.util.eventual

3.3.7.4. buildbot.util.debounce

3.3.7.5. buildbot.util.poll

3.3.7.6. buildbot.util.maildir

3.3.7.7. buildbot.util.misc

3.3.7.8. buildbot.util.netstrings

3.3.7.9. buildbot.util.sautils

3.3.7.10. buildbot.util.pathmatch

3.3.7.11. buildbot.util.topicmatch

3.3.7.12. buildbot.util.subscription

3.3.7.13. buildbot.util.croniter

3.3.7.14. buildbot.util.state

3.3.7.15. buildbot.util.identifiers

3.3.7.16. buildbot.util.lineboundaries

3.3.7.17. buildbot.util.service

3.3.7.18. buildbot.util.httpclientservice

3.3.7.19. buildbot.test.fake.httpclientservice

3.3.7.20. buildbot.util.ssl

3.3.7.1. `buildbot.util.lru`

3.3.7.2. `buildbot.util.bbcollections`

3.3.7.3. `buildbot.util.eventual`

3.3.7.4. `buildbot.util.debounce`

3.3.7.5. `buildbot.util.poll`

3.3.7.6. `buildbot.util.maildir`

3.3.7.7. `buildbot.util.misc`

3.3.7.8. `buildbot.util.netstrings`

3.3.7.9. `buildbot.util.sautils`

3.3.7.10. `buildbot.util.pathmatch`

3.3.7.11. `buildbot.util.topicmatch`

3.3.7.12. `buildbot.util.subscription`

3.3.7.13. `buildbot.util.croniter`

3.3.7.14. `buildbot.util.state`

3.3.7.15. `buildbot.util.identifiers`

3.3.7.16. `buildbot.util.lineboundaries`

3.3.7.17. `buildbot.util.service`

3.3.7.18. `buildbot.util.httpclientservice`

3.3.7.19. `buildbot.test.fake.httpclientservice`

3.3.7.20. `buildbot.util.ssl`