Utilities

Several small utilities are available at the top-level buildbot.util package.

buildbot.util.naturalSort(list)

Parameters:list -- list of strings Returns:sorted strings

This function sorts strings "naturally", with embedded numbers sorted numerically. This ordering is good for objects which might have a numeric suffix, e.g., winslave1, winslave2

buildbot.util.formatInterval(interval)

Parameters:interval -- duration in seconds Returns:human-readable (English) equivalent

This function will return a human-readable string describing a length of time, given a number of seconds.

class buildbot.util.ComparableMixin

This mixin class adds comparability to a subclass. Use it like this:

class Widget(FactoryProduct, ComparableMixin):
    compare_attrs = [ 'radius', 'thickness' ]
    # ...

Any attributes not in compare_attrs will not be considered when comparing objects. This is particularly useful in implementing buildbot's reconfig logic, where a simple comparison between the new and existing objects can determine whether the new object should replace the existing object.

A point to note is that the compare_attrs list is cumulative; that is, when a subclass also has a compare_attrs and the parent class has a compare_attrs, the subclass' compare_attrs also includes the parent class' compare_attrs.

buildbot.util.safeTranslate(str)

Parameters:str -- input string Returns:safe version of the input

This function will filter out some inappropriate characters for filenames; it is suitable for adapting strings from the configuration for use as filenames. It is not suitable for use with strings from untrusted sources.

buildbot.util.epoch2datetime(epoch)

Parameters:epoch -- an epoch time (integer) Returns:equivalent datetime object

Convert a UNIX epoch timestamp to a Python datetime object, in the UTC timezone. Note that timestamps specify UTC time (modulo leap seconds and a few other minor details).

buildbot.util.datetime2epoch(datetime)

Parameters:datetime -- a datetime object Returns:equivalent epoch time (integer)

Convert an arbitrary Python datetime object into a UNIX epoch timestamp.

buildbot.util.UTC

A datetime.tzinfo subclass representing UTC time. A similar class has finally been added to Python in version 3.2, but the implementation is simple enough to include here. This is mostly used in tests to create timezone-aware datetime objects in UTC:

dt = datetime.datetime(1978, 6, 15, 12, 31, 15, tzinfo=UTC)

buildbot.util.diffSets(old, new)

Parameters:

• old (set or iterable) -- old set
• new (set or iterable) -- new set

Returns:

a tuple, (removed, added)

This function compares two sets of objects, returning elements that were added and elements that were removed. This is largely a convenience function for reconfiguring services.

buildbot.util.makeList(input)

Parameters:input -- a thing Returns:a list of zero or more things

This function is intended to support the many places in Buildbot where the user can specify either a string or a list of strings, but the implementation wishes to always consider lists. It converts any string to a single-element list, None to an empty list, and any iterable to a list. Input lists are copied, avoiding aliasing issues.

buildbot.util.now()

Returns:epoch time (integer)

Return the current time, using either reactor.seconds or time.time().

buildbot.util.flatten(list)

Parameters:list -- potentially nested list Returns:flat list

Flatten nested lists into a list containing no other lists. For example:

>>> flatten([ [  1, 2 ], 3, [ [ 4 ] ] ])
[ 1, 2, 3, 4 ]

Note that this looks strictly for lists -- tuples, for example, are not flattened.

buildbot.util.none_or_str(obj)

Parameters:obj -- input value Returns:string or None

If obj is not None, return its string representation.

buildbot.util.NotABranch

This is a sentinel value used to indicate that no branch is specified. It is necessary since schedulers and change sources consider None a valid name for a branch. This is generally used as a default value in a method signature, and then tested against with is:

if branch is NotABranch:
    pass # ...

buildbot.util.in_reactor(fn)

This decorator will cause the wrapped function to be run in the Twisted reactor, with the reactor stopped when the function completes. It returns the result of the wrapped function. If the wrapped function fails, its traceback will be printed, the reactor halted, and None returned.

buildbot.util.asyncSleep(secs)

Yield a deferred that will fire with no result after secs seconds. This is the asynchronous equivalent to time.sleep, and can be useful in tests.

buildbot.util.lru

LRUCache(miss_fn, max_size=50):

Parameters:

• miss_fn -- function to call, with key as parameter, for cache misses. The function should return the value associated with the key argument, or None if there is no value associated with the key.
• max_size -- maximum number of objects in the cache.

This is a simple least-recently-used cache. When the cache grows beyond the maximum size, the least-recently used items will be automatically removed from the cache.

This cache is designed to control memory usage by minimizing duplication of objects, while avoiding unnecessary re-fetching of the same rows from the database.

All values are also stored in a weak valued dictionary, even after they have expired from the cache. This allows values that are used elsewhere in Buildbot to "stick" in the cache in case they are needed by another component. Weak references cannot be used for some types, so these types are not compatible with this class. Note that dictionaries can be weakly referenced if they are an instance of a subclass of dict.

If the result of the miss_fn is None, then the value is not cached; this is intended to avoid caching negative results.

This is based on Raymond Hettinger's implementation, licensed under the PSF license, which is GPL-compatiblie.

buildbot.util.lru.hits

cache hits so far

buildbot.util.lru.refhits

cache misses found in the weak ref dictionary, so far

buildbot.util.lru.misses

cache misses leading to re-fetches, so far

buildbot.util.lru.max_size

maximum allowed size of the cache

buildbot.util.lru.get(key, **miss_fn_kwargs)

Parameters:

• key -- cache key
• miss_fn_kwargs -- keyword arguments to the miss_fn

Returns:

value via Deferred

Fetch a value from the cache by key, invoking miss_fn(key, **miss_fn_kwargs) if the key is not in the cache.

Any additional keyword arguments are passed to the miss_fn as keyword arguments; these can supply additional information relating to the key. It is up to the caller to ensure that this information is functionally identical for each key value: if the key is already in the cache, the miss_fn will not be invoked, even if the keyword arguments differ.

buildbot.util.lru.put(key, value)

Parameters:

• key -- key at which to place the value
• value -- value to place there

Add the given key and value into the cache. The purpose of this method is to insert a new value into the cache without invoking the miss_fn (e.g., to avoid unnecessary overhead).

buildbot.util.lru.inv()

Check invariants on the cache. This is intended for debugging purposes.

AsyncLRUCache(miss_fn, max_size=50):

Parameters:

• miss_fn -- This is the same as the miss_fn for class LRUCache, with the difference that this function must return a Deferred.
• max_size -- maximum number of objects in the cache.

This class has the same functional interface as LRUCache, but asynchronous locking is used to ensure that in the common case of multiple concurrent requests for the same key, only one fetch is performed.

buildbot.util.bbcollections

This package provides a few useful collection objects.

Note

This module used to be named collections, but without absolute imports (PEP 328), this precluded using the standard library's collections module.

class buildbot.util.bbcollections.defaultdict

This is a clone of the Python collections.defaultdict for use in Python-2.4. In later versions, this is simply a reference to the built-in defaultdict, so buildbot code can simply use buildbot.util.collections.defaultdict everywhere.

class buildbot.util.bbcollections.KeyedSets

This is a collection of named sets. In principal, it contains an empty set for every name, and you can add things to sets, discard things from sets, and so on.

>>> ks = KeyedSets()
>>> ks['tim']                   # get a named set
set([])
>>> ks.add('tim', 'friendly')   # add an element to a set
>>> ks.add('tim', 'dexterous')
>>> ks['tim']
set(['friendly', 'dexterous'])
>>> 'tim' in ks                 # membership testing
True
>>> 'ron' in ks
False
>>> ks.discard('tim', 'friendly')# discard set element
>>> ks.pop('tim')               # return set and reset to empty
set(['dexterous'])
>>> ks['tim']
set([])

This class is careful to conserve memory space - empty sets do not occupy any space.

buildbot.util.eventual

This function provides a simple way to say "please do this later". For example:

from buildbot.util.eventual import eventually
def do_what_I_say(what, where):
    # ...
    return d
eventually(do_what_I_say, "clean up", "your bedroom")

The package defines "later" as "next time the reactor has control", so this is a good way to avoid long loops that block other activity in the reactor.

buildbot.util.eventual.eventually(cb, *args, **kwargs)

Parameters:

• cb -- callable to invoke later
• args -- args to pass to cb
• kwargs -- kwargs to pass to cb

Invoke the callable cb in a later reactor turn.

Callables given to eventually are guaranteed to be called in the same order as the calls to eventually -- writing eventually(a); eventually(b) guarantees that a will be called before b.

Any exceptions that occur in the callable will be logged with log.err(). If you really want to ignore them, provide a callable that catches those exceptions.

This function returns None. If you care to know when the callable was run, be sure to provide a callable that notifies somebody.

buildbot.util.eventual.fireEventually(value=None)

Parameters:value -- value with which the Deferred should fire Returns:Deferred

This function returns a Deferred which will fire in a later reactor turn, after the current call stack has been completed, and after all other Deferreds previously scheduled with eventually. The returned Deferred will never fail.

buildbot.util.eventual.flushEventualQueue()

Returns:Deferred

This returns a Deferred which fires when the eventual-send queue is finally empty. This is useful for tests and other circumstances where it is useful to know that "later" has arrived.

buildbot.util.debounce

Often, a method must be called exactly once at a time, but many events may trigger a call to the method. A simple example is the step method updateSummary.

The debounce.method(wait) decorator is the tool for the job.

buildbot.util.debounce.method(wait)

Parameters:wait -- time to wait before invoking, in seconds

Returns a decorator that debounces the underlying method. The underlying method must take no arguments (except self).

For each call to the decorated method, the underlying method will be invocation at least once within wait seconds (plus the time the method takes to execute). Calls are "debounced" during that time, meaning that multiple calls to the decorated method may result in a single invocation.

The decorated method is an instance of Debouncer, allowing it to be started and stopped. This is useful when the method is a part of a Buidbot service: call method.start() from startService and method.stop() from stopService, handling its Deferred appropriately.

class buildbot.util.debounce.Debouncer

stop()

Returns:Deferred

Stop the debouncer. While the debouncer is stopped, calls to the decorated method will be ignored. When the Deferred that stop returns fires, the underlying method is not executing.

start()

Start the debouncer. This reverses the effects of stop. This method can be called on a started debouncer without issues.

buildbot.util.json

This package is just an import of the best available JSON module. Use it instead of a more complex conditional import of simplejson or json:

from buildbot.util import json

buildbot.util.maildir

Several Buildbot components make use of maildirs to hand off messages between components. On the receiving end, there's a need to watch a maildir for incoming messages and trigger some action when one arrives.

class buildbot.util.maildir.MaildirService(basedir)

param basedir:(optional) base directory of the maildir

A MaildirService instance watches a maildir for new messages. It should be a child service of some MultiService instance. When running, this class uses the linux dirwatcher API (if available) or polls for new files in the 'new' maildir subdirectory. When it discovers a new message, it invokes its messageReceived method.

To use this class, subclass it and implement a more interesting messageReceived function.

setBasedir(basedir)

Parameters:basedir -- base directory of the maildir

If no basedir is provided to the constructor, this method must be used to set the basedir before the service starts.

messageReceived(filename)

Parameters:filename -- unqualified filename of the new message

This method is called with the short filename of the new message. The full name of the new file can be obtained with os.path.join(maildir, 'new', filename). The method is un-implemented in the MaildirService class, and must be implemented in subclasses.

moveToCurDir(filename)

Parameters:filename -- unqualified filename of the new message Returns:open file object

Call this from messageReceived to start processing the message; this moves the message file to the 'cur' directory and returns an open file handle for it.

buildbot.util.misc

buildbot.util.misc.deferredLocked(lock)

Parameters:lock -- a twisted.internet.defer.DeferredLock instance or a string naming an instance attribute containing one

This is a decorator to wrap an event-driven method (one returning a Deferred) in an acquire/release pair of a designated DeferredLock. For simple functions with a static lock, this is as easy as:

someLock = defer.DeferredLock()
@util.deferredLocked(someLock)
def someLockedFunction():
    # ..
    return d

For class methods which must access a lock that is an instance attribute, the lock can be specified by a string, which will be dynamically resolved to the specific instance at runtime:

def __init__(self):
    self.someLock = defer.DeferredLock()

@util.deferredLocked('someLock')
def someLockedFunction():
    # ..
    return d

class buildbot.util.misc.SerializedInvocation(method)

This is a method wrapper that will serialize calls to an asynchronous method. If a second call occurs while the first call is still executing, it will not begin until the first call has finished. If multiple calls queue up, they will be collapsed into a single call. The effect is that the underlying method is guaranteed to be called at least once after every call to the wrapper.

Note that if this class is used as a decorator on a method, it will serialize invocations across all class instances. For synchronization specific to each instance, wrap the method in the constructor:

def __init__(self):
    self.someMethod = SerializedInovcation(self.someMethod)

Tests can monkey-patch the _quiet method of the class to be notified when all planned invocations are complete.

buildbot.util.netstrings

Similar to maildirs, netstrings are used occasionally in Buildbot to encode data for interchange. While Twisted supports a basic netstring receiver protocol, it does not have a simple way to apply that to a non-network situation.

class buildbot.util.netstrings.NetstringParser

This class parses strings piece by piece, either collecting the accumulated strings or invoking a callback for each one.

feed(data)

Parameters:data -- a portion of netstring-formatted data Raises:twisted.protocols.basic.NetstringParseError

Add arbitrarily-sized data to the incoming-data buffer. Any complete netstrings will trigger a call to the stringReceived method.

Note that this method (like the Twisted class it is based on) cannot detect a trailing partial netstring at EOF - the data will be silently ignored.

stringReceived(string):

Parameters:string -- the decoded string

This method is called for each decoded string as soon as it is read completely. The default implementation appends the string to the strings attribute, but subclasses can do anything.

strings

The strings decoded so far, if stringReceived is not overridden.

buildbot.util.sautils

This module contains a few utilities that are not included with SQLAlchemy.

class buildbot.util.sautils.InsertFromSelect(table, select)

Parameters:

• table -- table into which insert should be performed
• select -- select query from which data should be drawn

This class is taken directly from SQLAlchemy's compiler.html, and allows a Pythonic representation of INSERT INTO .. SELECT .. queries.

buildbot.util.sautils.sa_version()

Return a 3-tuple representing the SQLAlchemy version. Note that older versions that did not have a __version__ attribute are represented by (0,0,0).

buildbot.util.subscription

The classes in the buildbot.util.subscription module are used for master-local subscriptions. In the near future, all uses of this module will be replaced with message-queueing implementations that allow subscriptions and subscribers to span multiple masters.

buildbot.util.croniter

This module is a copy of https://github.com/taichino/croniter, and provides support for converting cron-like time specifications into actual times.

buildbot.util.state

The classes in the buildbot.util.subscription module are used for dealing with object state stored in the database.

class buildbot.util.state.StateMixin

This class provides helper methods for accessing the object state stored in the database.

name

This must be set to the name to be used to identify this object in the database.

master

This must point to the BuildMaster object.

getState(name, default)

Parameters:

• name -- name of the value to retrieve
• default -- (optional) value to return if name is not present

Returns:

state value via a Deferred

Raises:

• KeyError -- if name is not present and no default is given
• TypeError -- if JSON parsing fails

Get a named state value from the object's state.

getState(name, value)

Parameters:

• name -- the name of the value to change
• value -- the value to set - must be a JSONable object
• returns -- Deferred

Raises TypeError:  

if JSONification fails

Set a named state value in the object's persistent state. Note that value must be json-able.

buildbot.util.identifiers

This module makes it easy to manipulate identifiers.

buildbot.util.identifiers.isIdentifier(maxLength, object)

Parameters:

• maxLength -- maximum length of the identifier
• object -- object to test for identifier-ness

Returns:

boolean

Is object an identifier?

buildbot.util.identifiers.forceIdentifier(maxLength, str)

Parameters:

• maxLength -- maximum length of the identifier
• str -- string to coerce to an identifier

Returns:

identifer of maximum length maxLength

Coerce a string (assuming ASCII for bytestrings) into an identifier. This method will replace any invalid characters with _ and truncate to the given length.

buildbot.util.identifiers.incrementIdentifier(maxLength, str)

Parameters:

• maxLength -- maximum length of the identifier
• str -- identifier to increment

Returns:

identifer of maximum length maxLength

Raises:

ValueError if no suitable identifier can be constructed

"Increment" an identifier by adding a numeric suffix, while keeping the total length limited. This is useful when selecting a unique identifier for an object. Maximum-length identifiers like _999999 cannot be incremented and will raise ValueError.