Utilities¶

Several small utilities are available at the top-level buildbot.util package.

buildbot.util.naturalSort(list)¶

Parameters:	list -- list of strings
Returns:	sorted strings

This function sorts strings "naturally", with embedded numbers sorted numerically. This ordering is good for objects which might have a numeric suffix, e.g., winslave1, winslave2

buildbot.util.formatInterval(interval)¶

Parameters:	interval -- duration in seconds
Returns:	human-readable (English) equivalent

This function will return a human-readable string describing a length of time, given a number of seconds.

class buildbot.util.ComparableMixin¶

This mixin class adds comparability to a subclass. Use it like this:

class Widget(FactoryProduct, ComparableMixin):
    compare_attrs = [ 'radius', 'thickness' ]
    # ...

Any attributes not in compare_attrs will not be considered when comparing objects. This is particularly useful in implementing buildbot's reconfig logic, where a simple comparison between the new and existing objects can determine whether the new object should replace the existing object.

A point to note is that the compare_attrs list is cumulative; that is, when a subclass also has a compare_attrs and the parent class has a compare_attrs, the subclass' compare_attrs also includes the parent class' compare_attrs.

buildbot.util.safeTranslate(str)¶

Parameters:	str -- input string
Returns:	safe version of the input

This function will filter out some inappropriate characters for filenames; it is suitable for adapting strings from the configuration for use as filenames. It is not suitable for use with strings from untrusted sources.

buildbot.util.epoch2datetime(epoch)¶

Parameters:	epoch -- an epoch time (integer)
Returns:	equivalent datetime object

Convert a UNIX epoch timestamp to a Python datetime object, in the UTC timezone. Note that timestamps specify UTC time (modulo leap seconds and a few other minor details).

buildbot.util.datetime2epoch(datetime)¶

Parameters:	datetime -- a datetime object
Returns:	equivalent epoch time (integer)

Convert an arbitrary Python datetime object into a UNIX epoch timestamp.

buildbot.util.UTC¶

A datetime.tzinfo subclass representing UTC time. A similar class has finally been added to Python in version 3.2, but the implementation is simple enough to include here. This is mostly used in tests to create timezone-aware datetime objects in UTC:

dt = datetime.datetime(1978, 6, 15, 12, 31, 15, tzinfo=UTC)

buildbot.util.diffSets(old, new)¶

Parameters:	old (set or iterable) -- old set new (set or iterable) -- new set
Returns:	a tuple, (removed, added)

This function compares two sets of objects, returning elements that were added and elements that were removed. This is largely a convenience function for reconfiguring services.

buildbot.util.makeList(input)¶

Parameters:	input -- a thing
Returns:	a list of zero or more things

This function is intended to support the many places in Buildbot where the user can specify either a string or a list of strings, but the implementation wishes to always consider lists. It converts any string to a single-element list, None to an empty list, and any iterable to a list. Input lists are copied, avoiding aliasing issues.

buildbot.util.now()¶

Returns:	epoch time (integer)

Return the current time, using either reactor.seconds or time.time().

buildbot.util.flatten(list)¶

Parameters:	list -- potentially nested list
Returns:	flat list

Flatten nested lists into a list containing no other lists. For example:

>>> flatten([ [  1, 2 ], 3, [ [ 4 ] ] ])
[ 1, 2, 3, 4 ]

Note that this looks strictly for lists -- tuples, for example, are not flattened.

buildbot.util.none_or_str(obj)¶

Parameters:	obj -- input value
Returns:	string or `None`

If obj is not None, return its string representation.

buildbot.util.NotABranch¶

This is a sentinel value used to indicate that no branch is specified. It is necessary since schedulers and change sources consider None a valid name for a branch. This is generally used as a default value in a method signature, and then tested against with is:

if branch is NotABranch:
    pass # ...

buildbot.util.in_reactor(fn)¶: This decorator will cause the wrapped function to be run in the Twisted reactor, with the reactor stopped when the function completes. It returns the result of the wrapped function. If the wrapped function fails, its traceback will be printed, the reactor halted, and None returned.

buildbot.util.asyncSleep(secs)¶: Yield a deferred that will fire with no result after secs seconds. This is the asynchronous equivalent to time.sleep, and can be useful in tests.

buildbot.util.lru¶

LRUCache(miss_fn, max_size=50):

Parameters:	miss_fn -- function to call, with key as parameter, for cache misses. The function should return the value associated with the key argument, or None if there is no value associated with the key. max_size -- maximum number of objects in the cache.

This is a simple least-recently-used cache. When the cache grows beyond the maximum size, the least-recently used items will be automatically removed from the cache.

This cache is designed to control memory usage by minimizing duplication of objects, while avoiding unnecessary re-fetching of the same rows from the database.

All values are also stored in a weak valued dictionary, even after they have expired from the cache. This allows values that are used elsewhere in Buildbot to "stick" in the cache in case they are needed by another component. Weak references cannot be used for some types, so these types are not compatible with this class. Note that dictionaries can be weakly referenced if they are an instance of a subclass of dict.

If the result of the miss_fn is None, then the value is not cached; this is intended to avoid caching negative results.

This is based on Raymond Hettinger's implementation, licensed under the PSF license, which is GPL-compatiblie.

buildbot.util.lru.hits¶: cache hits so far

buildbot.util.lru.refhits¶: cache misses found in the weak ref dictionary, so far

buildbot.util.lru.misses¶: cache misses leading to re-fetches, so far

buildbot.util.lru.max_size¶: maximum allowed size of the cache

buildbot.util.lru.get(key, **miss_fn_kwargs)¶

Parameters:	key -- cache key miss_fn_kwargs -- keyword arguments to the `miss_fn`
Returns:	value via Deferred

Fetch a value from the cache by key, invoking miss_fn(key, **miss_fn_kwargs) if the key is not in the cache.

Any additional keyword arguments are passed to the miss_fn as keyword arguments; these can supply additional information relating to the key. It is up to the caller to ensure that this information is functionally identical for each key value: if the key is already in the cache, the miss_fn will not be invoked, even if the keyword arguments differ.

buildbot.util.lru.put(key, value)¶

Parameters:	key -- key at which to place the value value -- value to place there

Add the given key and value into the cache. The purpose of this method is to insert a new value into the cache without invoking the miss_fn (e.g., to avoid unnecessary overhead).

buildbot.util.lru.inv()¶: Check invariants on the cache. This is intended for debugging purposes.

AsyncLRUCache(miss_fn, max_size=50):

Parameters:	miss_fn -- This is the same as the miss_fn for class LRUCache, with the difference that this function must return a Deferred. max_size -- maximum number of objects in the cache.

This class has the same functional interface as LRUCache, but asynchronous locking is used to ensure that in the common case of multiple concurrent requests for the same key, only one fetch is performed.

buildbot.util.bbcollections¶

This package provides a few useful collection objects.

Note

This module used to be named collections, but without absolute imports (PEP 328), this precluded using the standard library's collections module.

class buildbot.util.bbcollections.defaultdict¶: This is a clone of the Python collections.defaultdict for use in Python-2.4. In later versions, this is simply a reference to the built-in defaultdict, so buildbot code can simply use buildbot.util.collections.defaultdict everywhere.

class buildbot.util.bbcollections.KeyedSets¶

This is a collection of named sets. In principal, it contains an empty set for every name, and you can add things to sets, discard things from sets, and so on.

>>> ks = KeyedSets()
>>> ks['tim']                   # get a named set
set([])
>>> ks.add('tim', 'friendly')   # add an element to a set
>>> ks.add('tim', 'dexterous')
>>> ks['tim']
set(['friendly', 'dexterous'])
>>> 'tim' in ks                 # membership testing
True
>>> 'ron' in ks
False
>>> ks.discard('tim', 'friendly')# discard set element
>>> ks.pop('tim')               # return set and reset to empty
set(['dexterous'])
>>> ks['tim']
set([])

This class is careful to conserve memory space - empty sets do not occupy any space.

buildbot.util.eventual¶

This function provides a simple way to say "please do this later". For example:

from buildbot.util.eventual import eventually
def do_what_I_say(what, where):
    # ...
    return d
eventually(do_what_I_say, "clean up", "your bedroom")

The package defines "later" as "next time the reactor has control", so this is a good way to avoid long loops that block other activity in the reactor.

buildbot.util.eventual.eventually(cb, *args, **kwargs)¶

Parameters:	cb -- callable to invoke later args -- args to pass to `cb` kwargs -- kwargs to pass to `cb`

Invoke the callable cb in a later reactor turn.

Callables given to eventually are guaranteed to be called in the same order as the calls to eventually -- writing eventually(a); eventually(b) guarantees that a will be called before b.

Any exceptions that occur in the callable will be logged with log.err(). If you really want to ignore them, provide a callable that catches those exceptions.

This function returns None. If you care to know when the callable was run, be sure to provide a callable that notifies somebody.

buildbot.util.eventual.fireEventually(value=None)¶

Parameters:	value -- value with which the Deferred should fire
Returns:	Deferred

This function returns a Deferred which will fire in a later reactor turn, after the current call stack has been completed, and after all other Deferreds previously scheduled with eventually. The returned Deferred will never fail.

buildbot.util.eventual.flushEventualQueue()¶

Returns:	Deferred

This returns a Deferred which fires when the eventual-send queue is finally empty. This is useful for tests and other circumstances where it is useful to know that "later" has arrived.

buildbot.util.debounce¶

Often, a method must be called exactly once at a time, but many events may trigger a call to the method. A simple example is the step method updateSummary.

The debounce.method(wait) decorator is the tool for the job.

buildbot.util.debounce.method(wait)¶

Parameters:	wait -- time to wait before invoking, in seconds

Returns a decorator that debounces the underlying method. The underlying method must take no arguments (except self).

For each call to the decorated method, the underlying method will be invocation at least once within wait seconds (plus the time the method takes to execute). Calls are "debounced" during that time, meaning that multiple calls to the decorated method may result in a single invocation.

The decorated method is an instance of Debouncer, allowing it to be started and stopped. This is useful when the method is a part of a Buidbot service: call method.start() from startService and method.stop() from stopService, handling its Deferred appropriately.

class buildbot.util.debounce.Debouncer¶

stop()¶

Returns:	Deferred

Stop the debouncer. While the debouncer is stopped, calls to the decorated method will be ignored. When the Deferred that stop returns fires, the underlying method is not executing.

start()¶: Start the debouncer. This reverses the effects of stop. This method can be called on a started debouncer without issues.

buildbot.util.json¶

This package is just an import of the best available JSON module. Use it instead of a more complex conditional import of simplejson or json:

from buildbot.util import json

buildbot.util.maildir¶

Several Buildbot components make use of maildirs to hand off messages between components. On the receiving end, there's a need to watch a maildir for incoming messages and trigger some action when one arrives.

class buildbot.util.maildir.MaildirService(basedir)¶

param basedir: (optional) base directory of the maildir

param basedir:	(optional) base directory of the maildir

A MaildirService instance watches a maildir for new messages. It should be a child service of some MultiService instance. When running, this class uses the linux dirwatcher API (if available) or polls for new files in the 'new' maildir subdirectory. When it discovers a new message, it invokes its messageReceived method.

To use this class, subclass it and implement a more interesting messageReceived function.

setBasedir(basedir)¶

Parameters:	basedir -- base directory of the maildir

If no basedir is provided to the constructor, this method must be used to set the basedir before the service starts.

messageReceived(filename)¶

Parameters:	filename -- unqualified filename of the new message

This method is called with the short filename of the new message. The full name of the new file can be obtained with os.path.join(maildir, 'new', filename). The method is un-implemented in the MaildirService class, and must be implemented in subclasses.

moveToCurDir(filename)¶

Parameters:	filename -- unqualified filename of the new message
Returns:	open file object

Call this from messageReceived to start processing the message; this moves the message file to the 'cur' directory and returns an open file handle for it.

buildbot.util.misc¶

buildbot.util.misc.deferredLocked(lock)¶

Parameters:	lock -- a `twisted.internet.defer.DeferredLock` instance or a string naming an instance attribute containing one

This is a decorator to wrap an event-driven method (one returning a Deferred) in an acquire/release pair of a designated DeferredLock. For simple functions with a static lock, this is as easy as:

someLock = defer.DeferredLock()
@util.deferredLocked(someLock)
def someLockedFunction():
    # ..
    return d

For class methods which must access a lock that is an instance attribute, the lock can be specified by a string, which will be dynamically resolved to the specific instance at runtime:

def __init__(self):
    self.someLock = defer.DeferredLock()

@util.deferredLocked('someLock')
def someLockedFunction():
    # ..
    return d

class buildbot.util.misc.SerializedInvocation(method)¶

This is a method wrapper that will serialize calls to an asynchronous method. If a second call occurs while the first call is still executing, it will not begin until the first call has finished. If multiple calls queue up, they will be collapsed into a single call. The effect is that the underlying method is guaranteed to be called at least once after every call to the wrapper.

Note that if this class is used as a decorator on a method, it will serialize invocations across all class instances. For synchronization specific to each instance, wrap the method in the constructor:

def __init__(self):
    self.someMethod = SerializedInovcation(self.someMethod)

Tests can monkey-patch the _quiet method of the class to be notified when all planned invocations are complete.

buildbot.util.netstrings¶

Similar to maildirs, netstrings are used occasionally in Buildbot to encode data for interchange. While Twisted supports a basic netstring receiver protocol, it does not have a simple way to apply that to a non-network situation.

class buildbot.util.netstrings.NetstringParser¶

This class parses strings piece by piece, either collecting the accumulated strings or invoking a callback for each one.

feed(data)¶

Parameters:	data -- a portion of netstring-formatted data
Raises:	`twisted.protocols.basic.NetstringParseError`

Add arbitrarily-sized data to the incoming-data buffer. Any complete netstrings will trigger a call to the stringReceived method.

Note that this method (like the Twisted class it is based on) cannot detect a trailing partial netstring at EOF - the data will be silently ignored.

stringReceived(string):

Parameters:	string -- the decoded string

This method is called for each decoded string as soon as it is read completely. The default implementation appends the string to the strings attribute, but subclasses can do anything.

strings¶: The strings decoded so far, if stringReceived is not overridden.

buildbot.util.sautils¶

This module contains a few utilities that are not included with SQLAlchemy.

class buildbot.util.sautils.InsertFromSelect(table, select)¶

Parameters:	table -- table into which insert should be performed select -- select query from which data should be drawn

This class is taken directly from SQLAlchemy's compiler.html, and allows a Pythonic representation of INSERT INTO .. SELECT .. queries.

buildbot.util.sautils.sa_version()¶: Return a 3-tuple representing the SQLAlchemy version. Note that older versions that did not have a __version__ attribute are represented by (0,0,0).

buildbot.util.subscription¶

The classes in the buildbot.util.subscription module are used for master-local subscriptions. In the near future, all uses of this module will be replaced with message-queueing implementations that allow subscriptions and subscribers to span multiple masters.

buildbot.util.croniter¶

This module is a copy of https://github.com/taichino/croniter, and provides support for converting cron-like time specifications into actual times.

buildbot.util.state¶

The classes in the buildbot.util.subscription module are used for dealing with object state stored in the database.

class buildbot.util.state.StateMixin¶

This class provides helper methods for accessing the object state stored in the database.

name¶: This must be set to the name to be used to identify this object in the database.

master¶: This must point to the BuildMaster object.

getState(name, default)¶

Parameters:	name -- name of the value to retrieve default -- (optional) value to return if name is not present
Returns:	state value via a Deferred
Raises:	KeyError -- if name is not present and no default is given TypeError -- if JSON parsing fails

Get a named state value from the object's state.

getState(name, value)

Parameters:	name -- the name of the value to change value -- the value to set - must be a JSONable object returns -- Deferred
Raises:	TypeError -- if JSONification fails

Set a named state value in the object's persistent state. Note that value must be json-able.

buildbot.util.identifiers¶

This module makes it easy to manipulate identifiers.

buildbot.util.identifiers.isIdentifier(maxLength, object)¶

Parameters:	maxLength -- maximum length of the identifier object -- object to test for identifier-ness
Returns:	boolean

Is object an identifier?

buildbot.util.identifiers.forceIdentifier(maxLength, str)¶

Parameters:	maxLength -- maximum length of the identifier str -- string to coerce to an identifier
Returns:	identifer of maximum length `maxLength`

Coerce a string (assuming ASCII for bytestrings) into an identifier. This method will replace any invalid characters with _ and truncate to the given length.

buildbot.util.identifiers.incrementIdentifier(maxLength, str)¶

Parameters:	maxLength -- maximum length of the identifier str -- identifier to increment
Returns:	identifer of maximum length `maxLength`
Raises:	ValueError if no suitable identifier can be constructed

"Increment" an identifier by adding a numeric suffix, while keeping the total length limited. This is useful when selecting a unique identifier for an object. Maximum-length identifiers like _999999 cannot be incremented and will raise ValueError.

Utilities¶

buildbot.util.lru¶

buildbot.util.bbcollections¶

buildbot.util.eventual¶

buildbot.util.debounce¶

buildbot.util.json¶

buildbot.util.maildir¶

buildbot.util.misc¶

buildbot.util.netstrings¶

buildbot.util.sautils¶

buildbot.util.subscription¶

buildbot.util.croniter¶

buildbot.util.state¶

buildbot.util.identifiers¶

Table Of Contents

Search