3.2.3. Database¶

As of version 0.8.0, Buildbot has used a database as part of its storage backend. This section describes the database connector classes, which allow other parts of Buildbot to access the database. It also describes how to modify the database schema and the connector classes themselves.

3.2.3.1. Database Overview¶

All access to the Buildbot database is mediated by database connector classes. These classes provide a functional, asynchronous interface to other parts of Buildbot, and encapsulate the database-specific details in a single location in the codebase.

The connector API, defined below, is a stable API in Buildbot, and can be called from any other component. Given a master master, the root of the database connectors is available at master.db, so, for example, the state connector's getState method is master.db.state.getState.

The connectors all use SQLAlchemy Core to achieve (almost) database-independent operation. Note that the SQLAlchemy ORM is not used in Buildbot. Database queries are carried out in threads, and report their results back to the main thread via Twisted Deferreds.

3.2.3.2. Schema¶

The database schema is maintained with SQLAlchemy-Migrate. This package handles the details of upgrading users between different schema versions.

The schema itself is considered an implementation detail, and may change significantly from version to version. Users should rely on the API (below), rather than performing queries against the database itself.

3.2.3.3. API¶

types¶

Identifier¶

An "identifier" is a nonempty unicode string of limited length, containing only ASCII alphanumeric characters along with - (dash) and _ (underscore), and not beginning with a digit Wherever an identifier is used, the documentation will give the maximum length in characters. The function buildbot.util.identifiers.isIdentifier is useful to verify a well-formed identifier.

buildrequests¶

exception buildbot.db.buildrequests.AlreadyClaimedError¶: Raised when a build request is already claimed, usually by another master.

exception buildbot.db.buildrequests.NotClaimedError¶: Raised when a build request is not claimed by this master.

class buildbot.db.buildrequests.BuildRequestsConnectorComponent¶

This class handles the complex process of claiming and unclaiming build requests, based on a polling model: callers poll for unclaimed requests with getBuildRequests, then attempt to claim the requests with claimBuildRequests. The claim can fail if another master has claimed the request in the interim.

An instance of this class is available at master.db.buildrequests.

Build requests are indexed by an ID referred to as a brid. The contents of a request are represented as build request dictionaries (brdicts) with keys

buildrequestid
buildsetid
builderid
buildername
priority
claimed (boolean, true if the request is claimed)
claimed_at (datetime object, time this request was last claimed)
claimed_by_masterid (integer, the id of the master that claimed this buildrequest)
complete (boolean, true if the request is complete)
complete_at (datetime object, time this request was completed)
submitted_at (datetime object, time this request was completed)
results (integer result code)
waited_for (boolean)

getBuildRequest(brid)¶

Parameters:	brid -- build request id to look up
Returns:	brdict or `None`, via Deferred

Get a single BuildRequest, in the format described above. This method returns None if there is no such buildrequest. Note that build requests are not cached, as the values in the database are not fixed.

getBuildRequests(buildername=None, complete=None, claimed=None, bsid=None, branch=None, repository=None, resultSpec=None)¶

Parameters:

buildername (string) -- limit results to buildrequests for this builder
complete -- if true, limit to completed buildrequests; if false, limit to incomplete buildrequests; if None, do not limit based on completion.
claimed -- see below
bsid -- see below
repository -- the repository associated with the sourcestamps originating the requests
branch -- the branch associated with the sourcestamps originating the requests
resultSpec -- resultSpec containing filters sorting and paging request from data/REST API. If possible, the db layer can optimize the SQL query using this information.

Returns:

list of brdicts, via Deferred

Get a list of build requests matching the given characteristics.

Pass all parameters as keyword parameters to allow future expansion.

The claimed parameter can be None (the default) to ignore the claimed status of requests; True to return only claimed builds, False to return only unclaimed builds, or a master ID to return only builds claimed by a particular master instance. A request is considered unclaimed if its claimed_at column is either NULL or 0, and it is not complete. If bsid is specified, then only build requests for that buildset will be returned.

A build is considered completed if its complete column is 1; the complete_at column is not consulted.

claimBuildRequests(brids[, claimed_at=XX])¶

Parameters:	brids (list) -- ids of buildrequests to claim claimed_at (datetime) -- time at which the builds are claimed
Returns:	Deferred
Raises:	`AlreadyClaimedError`

Try to "claim" the indicated build requests for this buildmaster instance. The resulting deferred will fire normally on success, or fail with AlreadyClaimedError if any of the build requests are already claimed by another master instance. In this case, none of the claims will take effect.

If claimed_at is not given, then the current time will be used.

As of 0.8.5, this method can no longer be used to re-claim build requests. All given ID's must be unclaimed. Use reclaimBuildRequests to reclaim.

Note

On database backends that do not enforce referential integrity (e.g., SQLite), this method will not prevent claims for nonexistent build requests. On database backends that do not support transactions (MySQL), this method will not properly roll back any partial claims made before an AlreadyClaimedError is generated.

reclaimBuildRequests(brids)¶

Parameters:	brids (list) -- ids of buildrequests to reclaim
Returns:	Deferred
Raises:	`AlreadyClaimedError`

Re-claim the given build requests, updating the timestamp, but checking that the requests are owned by this master. The resulting deferred will fire normally on success, or fail with AlreadyClaimedError if any of the build requests are already claimed by another master instance, or don't exist. In this case, none of the reclaims will take effect.

unclaimBuildRequests(brids)¶

Parameters:	brids (list) -- ids of buildrequests to unclaim
Returns:	Deferred

Release this master's claim on all of the given build requests. This will not unclaim requests that are claimed by another master, but will not fail in this case. The method does not check whether a request is completed.

completeBuildRequests(brids, results[, complete_at=XX])¶

Parameters:	brids (integer) -- build request IDs to complete results (integer) -- integer result code complete_at (datetime) -- time at which the buildset was completed
Returns:	Deferred
Raises:	`NotClaimedError`

Complete a set of build requests, all of which are owned by this master instance. This will fail with NotClaimedError if the build request is already completed or does not exist. If complete_at is not given, the current time will be used.

unclaimExpiredRequests(old)¶

Parameters:	old (int) -- number of seconds after which a claim is considered old
Returns:	Deferred

Find any incomplete claimed builds which are older than old seconds, and clear their claim information.

This is intended to catch builds that were claimed by a master which has since disappeared. As a side effect, it will log a message if any requests are unclaimed.

builds¶

class buildbot.db.builds.BuildsConnectorComponent¶

This class handles builds. One build record is created for each build performed by a master. This record contains information on the status of the build, as well as links to the resources used in the build: builder, master, worker, etc.

An instance of this class is available at master.db.builds.

Builds are indexed by buildid and their contents represented as builddicts (build dictionaries), with the following keys:

id (the build ID, globally unique)
number (the build number, unique only within the builder)
builderid (the ID of the builder that performed this build)
buildrequestid (the ID of the build request that caused this build)
workerid (the ID of the worker on which this build was performed)
masterid (the ID of the master on which this build was performed)
started_at (datetime at which this build began)
complete_at (datetime at which this build finished, or None if it is ongoing)
state_string (short string describing the build's state)
results (results of this build; see Build Result Codes)

getBuild(buildid)¶

Parameters:	buildid (integer) -- build id
Returns:	Build dictionary as above or `None`, via Deferred

Get a single build, in the format described above. Returns None if there is no such build.

getBuildByNumber(builderid, number)¶

Parameters:	builder (integer) -- builder id number (integer) -- build number within that builder
Returns:	Build dictionary as above or `None`, via Deferred

Get a single build, in the format described above, specified by builder and number, rather than build id. Returns None if there is no such build.

getPrevSuccessfulBuild(builderid, number, ssBuild)¶

Parameters:	builderid (integer) -- builder to get builds for number (integer) -- the current build number. Previous build will be taken from this number ssBuild (list) -- the list of sourcestamps for the current build number
Returns:	None or a build dictionary

Returns the last successful build from the current build number with the same repository/repository/codebase

getBuilds(builderid=None, buildrequestid=None, complete=None, resultSpec=None)¶

Parameters:	builderid (integer) -- builder to get builds for buildrequestid (integer) -- buildrequest to get builds for complete (boolean) -- if not None, filters results based on completeness resultSpec -- resultSpec containing filters sorting and paging request from data/REST API. If possible, the db layer can optimize the SQL query using this information.
Returns:	list of build dictionaries as above, via Deferred

Get a list of builds, in the format described above. Each of the parameters limit the resulting set of builds.

addBuild(builderid, buildrequestid, workerid, masterid, state_string)¶

Parameters:	builderid (integer) -- builder to get builds for buildrequestid (integer) -- build request id workerid (integer) -- worker performing the build masterid (integer) -- master performing the build state_string (unicode) -- initial state of the build
Returns:	tuple of build ID and build number, via Deferred

Add a new build to the db, recorded as having started at the current time. This will invent a new number for the build, unique within the context of the builder.

setBuildStateString(buildid, state_string):

Parameters:	buildid (integer) -- build id state_string (unicode) -- updated state of the build
Returns:	Deferred

Update the state strings for the given build.

finishBuild(buildid, results)¶

Parameters:	buildid (integer) -- build id results (integer) -- build result
Returns:	Deferred

Mark the given build as finished, with complete_at set to the current time.

Note

This update is done unconditionally, even if the build is already finished.

getBuildProperties(buildid)¶

Parameters:	buildid -- build ID
Returns:	dictionary mapping property name to `value, source`, via Deferred

Return the properties for a build, in the same format they were given to addBuild.

Note that this method does not distinguish a non-existent build from a build with no properties, and returns {} in either case.

setBuildProperty(buildid, name, value, source)¶

Parameters:	buildid (integer) -- build ID name (string) -- Name of the property to set value -- Value of the property source (string) -- Source of the Property to set
Returns:	Deferred

Set a build property. If no property with that name existed in that build, a new property will be created.

steps¶

class buildbot.db.steps.StepsConnectorComponent¶

This class handles the steps performed within the context of a build. Within a build, each step has a unique name and a unique, 0-based number.

An instance of this class is available at master.db.steps.

Builds are indexed by stepid and their contents represented as stepdicts (step dictionaries), with the following keys:

id (the step ID, globally unique)
number (the step number, unique only within the build)
name (the step name, an 50-character identifier unique only within the build)
buildid (the ID of the build containing this step)
started_at (datetime at which this step began)
complete_at (datetime at which this step finished, or None if it is ongoing)
state_string (short string describing the step's state)
results (results of this step; see Build Result Codes)
urls (list of URLs produced by this step. Each urls is stored as a dictionary with keys name and url)
hidden (true if the step should be hidden in status displays)

getStep(stepid=None, buildid=None, number=None, name=None)¶

Parameters:	stepid (integer) -- the step id to retrieve buildid (integer) -- the build from which to get the step number (integer) -- the step number name (50-character identifier) -- the step name
Returns:	stepdict via Deferred

Get a single step. The step can be specified by

stepid alone;

buildid and number, the step number within that build; or

buildid and name, the unique step name within that build.

getSteps(buildid)¶

Parameters:	buildid (integer) -- the build from which to get the step
Returns:	list of stepdicts, sorted by number, via Deferred

Get all steps in the given build, in order by number.

addStep(self, buildid, name, state_string)¶

Parameters:	buildid (integer) -- the build to which to add the step name (50-character identifier) -- the step name state_string (unicode) -- the initial state of the step
Returns:	tuple of step ID, step number, and step name, via Deferred

Add a new step to a build. The given name will be used if it is unique; otherwise, a unique numerical suffix will be appended.

setStepStateString(stepid, state_string):

Parameters:	stepid (integer) -- step ID state_string (unicode) -- updated state of the step
Returns:	Deferred

Update the state string for the given step.

finishStep(stepid, results, hidden)¶

Parameters:	stepid (integer) -- step ID results (integer) -- step result hidden (bool) -- true if the step should be hidden
Returns:	Deferred

Mark the given step as finished, with complete_at set to the current time.

Note

This update is done unconditionally, even if the steps are already finished.

addURL(self, stepid, name, url)¶

Parameters:	stepid (integer) -- the stepid to add the url. name (string) -- the url name url (string) -- the actual url
Returns:	None via deferred

Add a new url to a step. The new url is added to the list of urls.

logs¶

class buildbot.db.logs.LogsConnectorComponent¶

This class handles log data. Build steps can have zero or more logs. Logs are uniquely identified by name within a step.

Information about a log, apart from its contents, is represented as a dictionary with the following keys, referred to as a logdict:

id (log ID, globally unique)
stepid (step ID, indicating the containing step)
name free-form name of this log
slug (50-identifier for the log, unique within the step)
complete (true if the log is complete and will not receive more lines)
num_lines (number of lines in the log)
type (log type; see below)

Each log has a type that describes how to interpret its contents. See the logchunk resource type for details.

A log is contains a sequence of newline-separated lines of unicode. Log line numbering is zero-based.

Each line must be less than 64k when encoded in UTF-8. Longer lines will be truncated, and a warning logged.

Lines are stored internally in "chunks", and optionally compressed, but the implementation hides these details from callers.

getLog(logid)¶

Parameters:	logid (integer) -- ID of the requested log
Returns:	logdict via Deferred

Get a log, identified by logid.

getLogBySlug(stepid, slug)¶

Parameters:	stepid (integer) -- ID of the step containing this log slug -- slug of the logfile to retrieve
Returns:	logdict via Deferred

Get a log, identified by name within the given step.

getLogs(stepid)¶

Parameters:	stepid (integer) -- ID of the step containing the desired logs
Returns:	list of logdicts via Deferred

Get all logs within the given step.

getLogLines(logid, first_line, last_line)¶

Parameters:	logid (integer) -- ID of the log first_line -- first line to return last_line -- last line to return
Returns:	see below

Get a subset of lines for a logfile.

The return value, via Deferred, is a concatenation of newline-terminated strings. If the requested last line is beyond the end of the logfile, only existing lines will be included. If the log does not exist, or has no associated lines, this method returns an empty string.

addLog(stepid, name, type)¶

Parameters:	stepid (integer) -- ID of the step containing this log name (string) -- name of the logfile slug (50-character identifier) -- slug (unique identifier) of the logfile type (string) -- log type (see above)
Raises:	KeyError -- if a log with the given slug already exists in the step
Returns:	ID of the new log, via Deferred

Add a new log file to the given step.

appendLog(logid, content)¶

Parameters:	logid (integer) -- ID of the requested log content (string) -- new content to be appended to the log
Returns:	tuple of first and last line numbers in the new chunk, via Deferred

Append content to an existing log. The content must end with a newline. If the given log does not exist, the method will silently do nothing.

It is not safe to call this method more than once simultaneously for the same logid.

finishLog(logid)¶

Parameters:	logid (integer) -- ID of the log to mark complete
Returns:	Deferred

Mark a log as complete.

Note that no checking for completeness is performed when appending to a log. It is up to the caller to avoid further calls to appendLog after finishLog.

compressLog(logid)¶

Parameters:	logid (integer) -- ID of the log to compress
Returns:	Deferred

Compress the given log. This method performs internal optimizations of a log's chunks to reduce the space used and make read operations more efficient. It should only be called for finished logs. This method may take some time to complete.

deleteOldLogChunks(older_than_timestamp)¶

Parameters:	older_than_timestamp (integer) -- the logs whose step's `started_at` is older than `older_than_timestamp` will be deleted.
Returns:	Deferred

Delete old logchunks (helper for the logHorizon policy). Old logs have their logchunks deleted from the database, but they keep their num_lines metadata. They have their types changed to 'd', so that the UI can display something meaningful.

buildsets¶

class buildbot.db.buildsets.BuildsetsConnectorComponent¶

This class handles getting buildsets into and out of the database. Buildsets combine multiple build requests that were triggered together.

An instance of this class is available at master.db.buildsets.

Buildsets are indexed by bsid and their contents represented as bsdicts (buildset dictionaries), with keys

bsid
external_idstring (arbitrary string for mapping builds externally)
reason (string; reason these builds were triggered)
sourcestamps (list of sourcestamps for this buildset, by ID)
submitted_at (datetime object; time this buildset was created)
complete (boolean; true if all of the builds for this buildset are complete)
complete_at (datetime object; time this buildset was completed)
results (aggregate result of this buildset; see Build Result Codes)

addBuildset(sourcestamps, reason, properties, builderids, external_idstring=None, parent_buildid=None, parent_relationship=None)¶

Parameters:

sourcestamps (list) -- sourcestamps for the new buildset; see below
reason (short unicode string) -- reason for this buildset
properties (dictionary, where values are tuples of (value, source)) -- properties for this buildset
builderids (list of int) -- builderids specified by this buildset
external_idstring (unicode string) -- external key to identify this buildset; defaults to None
submitted_at (datetime) -- time this buildset was created; defaults to the current time
parent_buildid (int) -- optional build id that is the parent for this buildset
parent_relationship (unicode) -- relationship identifier for the parent, this is is configured relationship between the parent build, and the childs buildsets

Returns:

buildset ID and buildrequest IDs, via a Deferred

Add a new Buildset to the database, along with BuildRequests for each builder, returning the resulting bsid via a Deferred. Arguments should be specified by keyword.

Each sourcestamp in the list of sourcestamps can be given either as an integer, assumed to be a sourcestamp ID, or a dictionary of keyword arguments to be passed to findSourceStampId.

The return value is a tuple (bsid, brids) where bsid is the inserted buildset ID and brids is a dictionary mapping builderids to build request IDs.

completeBuildset(bsid, results[, complete_at=XX])¶

Parameters:	bsid (integer) -- buildset ID to complete results (integer) -- integer result code complete_at (datetime) -- time the buildset was completed
Returns:	Deferred
Raises:	`KeyError` if the buildset does not exist or is already complete

Complete a buildset, marking it with the given results and setting its completed_at to the current time, if the complete_at argument is omitted.

getBuildset(bsid)¶

Parameters:	bsid -- buildset ID
Returns:	bsdict, or `None`, via Deferred

Get a bsdict representing the given buildset, or None if no such buildset exists.

Note that buildsets are not cached, as the values in the database are not fixed.

getBuildsets(complete=None, resultSpec=None)¶

Parameters:	complete -- if true, return only complete buildsets; if false, return only incomplete buildsets; if `None` or omitted, return all buildsets resultSpec -- resultSpec containing filters sorting and paging request from data/REST API. If possible, the db layer can optimize the SQL query using this information.
Returns:	list of bsdicts, via Deferred

Get a list of bsdicts matching the given criteria.

getRecentBuildsets(count=None, branch=None, repository=None,

complete=None):

Parameters:

count (integer) -- maximum number of buildsets to retrieve (required).
branch (string) -- optional branch name. If specified, only buildsets affecting such branch will be returned.
repository (string) -- optional repository name. If specified, only buildsets affecting such repository will be returned.
complete (Boolean) -- if true, return only complete buildsets; if false, return only incomplete buildsets; if None or omitted, return all buildsets

Returns:

list of bsdicts, via Deferred

Get "recent" buildsets, as defined by their submitted_at times.

getBuildsetProperties(buildsetid)¶

Parameters:	bsid -- buildset ID
Returns:	dictionary mapping property name to `value, source`, via Deferred

Return the properties for a buildset, in the same format they were given to addBuildset.

Note that this method does not distinguish a nonexistent buildset from a buildset with no properties, and returns {} in either case.

workers¶

class buildbot.db.workers.WorkersConnectorComponent¶

This class handles Buildbot's notion of workers. The worker information is returned as a dictionary:

id
name - the name of the worker
workerinfo - worker information as dictionary
connected_to - a list of masters, by ID, to which this worker is currently connected. This list will typically contain only one master, but in unusual circumstances the same worker may appear to be connected to multiple masters simultaneously.
configured_on - a list of master-builder pairs, on which this worker is configured. Each pair is represented by a dictionary with keys buliderid and masterid.

The worker information can be any JSON-able object. See worker for more detail.

findWorkerId(name=name)¶

Parameters:	name (50-character identifier) -- worker name
Returns:	worker ID via Deferred

Get the ID for a worker, adding a new worker to the database if necessary. The worker information for a new worker is initialized to an empty dictionary.

getWorkers(masterid=None, builderid=None)¶

Parameters:	masterid (integer) -- limit to workers configured on this master builderid (integer) -- limit to workers configured on this builder
Returns:	list of worker dictionaries, via Deferred

Get a list of workers. If either or both of the filtering parameters either specified, then the result is limited to workers configured to run on that master or builder. The configured_on results are limited by the filtering parameters as well. The connected_to results are limited by the masterid parameter.

getWorker(workerid=None, name=None, masterid=None, builderid=None)¶

Parameters:	name (string) -- the name of the worker to retrieve workerid (integer) -- the ID of the worker to retrieve masterid (integer) -- limit to workers configured on this master builderid (integer) -- limit to workers configured on this builder
Returns:	info dictionary or None, via Deferred

Looks up the worker with the given name or ID, returning None if no matching worker is found. The masterid and builderid arguments function as they do for getWorkers.

workerConnected(workerid, masterid, workerinfo)¶

Parameters:	workerid (integer) -- the ID of the worker masterid (integer) -- the ID of the master to which it connected workerinfo (dict) -- the new worker information dictionary
Returns:	Deferred

Record the given worker as attached to the given master, and update its cached worker information. The supplied information completely replaces any existing information.

workerDisconnected(workerid, masterid)¶

Parameters:	workerid (integer) -- the ID of the worker masterid (integer) -- the ID of the master to which it connected
Returns:	Deferred

Record the given worker as no longer attached to the given master.

workerConfigured(workerid, masterid, builderids)¶

Parameters:	workerid (integer) -- the ID of the worker masterid (integer) -- the ID of the master to which it configured of integer builderids (list) -- the ID of the builders to which it is configured
Returns:	Deferred

Record the given worker as being configured on the given master and for given builders. This method will also remove any other builder that were configured previously for same (worker, master) combination.

deconfigureAllWorkersForMaster(masterid)¶

Parameters:	masterid (integer) -- the ID of the master to which it configured
Returns:	Deferred

Unregister all the workers configured to a master for given builders. This shall happen when master disabled or before reconfiguration

changes¶

class buildbot.db.changes.ChangesConnectorComponent¶

This class handles changes in the buildbot database, including pulling information from the changes sub-tables.

An instance of this class is available at master.db.changes.

Changes are indexed by changeid, and are represented by a chdict, which has the following keys:

changeid (the ID of this change)
parent_changeids (list of ID; change's parents)
author (unicode; the author of the change)
files (list of unicode; source-code filenames changed)
comments (unicode; user comments)
is_dir (deprecated)
links (list of unicode; links for this change, e.g., to web views, review)
revision (unicode string; revision for this change, or None if unknown)
when_timestamp (datetime instance; time of the change)
branch (unicode string; branch on which the change took place, or None for the "default branch", whatever that might mean)
category (unicode string; user-defined category of this change, or None)
revlink (unicode string; link to a web view of this change)
properties (user-specified properties for this change, represented as a dictionary mapping keys to (value, source))
repository (unicode string; repository where this change occurred)
project (unicode string; user-defined project to which this change corresponds)

getParentChangeIds(branch, repository, project, codebase)¶

Parameters:	branch (unicode string) -- the branch of the change repository (unicode string) -- the repository in which this change took place project (unicode string) -- the project this change is a part of codebase (unicode string) --

return the last changeID which matches the repository/project/codebase

addChange(author=None, files=None, comments=None, is_dir=0, links=None, revision=None, when_timestamp=None, branch=None, category=None, revlink='', properties={}, repository='', project='', uid=None)¶

Parameters:

author (unicode string) -- the author of this change
files -- a list of filenames that were changed
comments -- user comments on the change
is_dir -- deprecated
links (list of unicode strings) -- a list of links related to this change, e.g., to web viewers or review pages
revision (unicode string) -- the revision identifier for this change
when_timestamp (datetime instance or None) -- when this change occurred, or the current time if None
branch (unicode string) -- the branch on which this change took place
category (unicode string) -- category for this change (arbitrary use by Buildbot users)
revlink (unicode string) -- link to a web view of this revision
properties (dictionary) -- properties to set on this change, where values are tuples of (value, source). At the moment, the source must be 'Change', although this may be relaxed in later versions.
repository (unicode string) -- the repository in which this change took place
project (unicode string) -- the project this change is a part of
uid (integer) -- uid generated for the change author

Returns:

new change's ID via Deferred

Add a Change with the given attributes to the database, returning the changeid via a Deferred. All arguments should be given as keyword arguments.

The project and repository arguments must be strings; None is not allowed.

getChange(changeid, no_cache=False)¶

Parameters:	changeid -- the id of the change instance to fetch no_cache (boolean) -- bypass cache and always fetch from database
Returns:	chdict via Deferred

Get a change dictionary for the given changeid, or None if no such change exists.

getChangeUids(changeid)¶

Parameters:	changeid -- the id of the change instance to fetch
Returns:	list of uids via Deferred

Get the userids associated with the given changeid.

getRecentChanges(count)¶

Parameters:	count -- maximum number of instances to return
Returns:	list of dictionaries via Deferred, ordered by changeid

Get a list of the count most recent changes, represented as dictionaries; returns fewer if that many do not exist.

Note

For this function, "recent" is determined by the order of the changeids, not by when_timestamp. This is most apparent in DVCS's, where the timestamp of a change may be significantly earlier than the time at which it is merged into a repository monitored by Buildbot.

getChanges()¶

Returns:	list of dictionaries via Deferred

Get a list of the changes, represented as dictionaries; changes are sorted, and paged using generic data query options

getChangesCount()¶

Returns:	list of dictionaries via Deferred

Get the number changes, that the query option would return if no paging option where set

getLatestChangeid()¶

Returns:	changeid via Deferred

Get the most-recently-assigned changeid, or None if there are no changes at all.

getChangesForBuild(buildid)¶

Parameters:	buildid -- ID of the build
Returns:	list of dictionaries via Deferred

Get the "blame" list of changes for a build.

getChangeFromSSid(sourcestampid)¶

Parameters:	sourcestampid -- ID of the sourcestampid
Returns:	chdict via Deferred

returns the change dictionary related to the sourcestamp ID.

changesources¶

exception buildbot.db.changesources.ChangeSourceAlreadyClaimedError¶: Raised when a changesource request is already claimed by another master.

class buildbot.db.changesources.ChangeSourcesConnectorComponent¶

This class manages the state of the Buildbot changesources.

An instance of this class is available at master.db.changesources.

Changesources are identified by their changesourceid, which can be objtained from findChangeSourceId.

Changesources are represented by dictionaries with the following keys:

id - changesource's ID

name - changesource's name

masterid - ID of the master currently running this changesource, or None if it is inactive

Note that this class is conservative in determining what changesources are inactive: a changesource linked to an inactive master is still considered active. This situation should never occur, however; links to a master should be deleted when it is marked inactive.

findChangeSourceId(name)¶

Parameters:	name -- changesource name
Returns:	changesource ID via Deferred

Return the changesource ID for the changesource with this name. If such a changesource is already in the database, this returns the ID. If not, the changesource is added to the database and its ID returned.

setChangeSourceMaster(changesourceid, masterid)¶

Parameters:	changesourceid -- changesource to set the master for masterid -- new master for this changesource, or None
Returns:	Deferred

Set, or unset if masterid is None, the active master for this changesource. If no master is currently set, or the current master is not active, this method will complete without error. If the current master is active, this method will raise ChangeSourceAlreadyClaimedError.

getChangeSource(changesourceid)¶

Parameters:	changesourceid -- changesource ID
Returns:	changesource dictionary or None, via Deferred

Get the changesource dictionary for the given changesource.

getChangeSources(active=None, masterid=None)¶

Parameters:	active (boolean) -- if specified, filter for active or inactive changesources masterid (integer) -- if specified, only return changesources attached associated with this master
Returns:	list of changesource dictionaries in unspecified order

Get a list of changesources.

If active is given, changesources are filtered according to whether they are active (true) or inactive (false). An active changesource is one that is claimed by an active master.

If masterid is given, the list is restricted to schedulers associated with that master.

schedulers¶

exception buildbot.db.schedulers.SchedulerAlreadyClaimedError¶: Raised when a scheduler request is already claimed by another master.

class buildbot.db.schedulers.SchedulersConnectorComponent¶

This class manages the state of the Buildbot schedulers. This state includes classifications of as-yet un-built changes.

An instance of this class is available at master.db.schedulers.

Schedulers are identified by their schedulerid, which can be objtained from findSchedulerId.

Schedulers are represented by dictionaries with the following keys:

id - scheduler's ID

name - scheduler's name

masterid - ID of the master currently running this scheduler, or None if it is inactive

Note that this class is conservative in determining what schedulers are inactive: a scheduler linked to an inactive master is still considered active. This situation should never occur, however; links to a master should be deleted when it is marked inactive.

classifyChanges(objectid, classifications)¶

Parameters:	schedulerid -- ID of the scheduler classifying the changes classifications (dictionary) -- mapping of changeid to boolean, where the boolean is true if the change is important, and false if it is unimportant
Returns:	Deferred

Record the given classifications. This method allows a scheduler to record which changes were important and which were not immediately, even if the build based on those changes will not occur for some time (e.g., a tree stable timer). Schedulers should be careful to flush classifications once they are no longer needed, using flushChangeClassifications.

flushChangeClassifications(objectid, less_than=None)¶

Parameters:	schedulerid -- ID of the scheduler owning the flushed changes less_than -- (optional) lowest changeid that should not be flushed
Returns:	Deferred

Flush all scheduler_changes for the given scheduler, limiting to those with changeid less than less_than if the parameter is supplied.

getChangeClassifications(objectid[, branch])¶

Parameters:	schedulerid (integer) -- ID of scheduler to look up changes for branch (string or None (for default branch)) -- (optional) limit to changes with this branch
Returns:	dictionary via Deferred

Return the classifications made by this scheduler, in the form of a dictionary mapping changeid to a boolean, just as supplied to classifyChanges.

If branch is specified, then only changes on that branch will be given. Note that specifying branch=None requests changes for the default branch, and is not the same as omitting the branch argument altogether.

findSchedulerId(name)¶

Parameters:	name -- scheduler name
Returns:	scheduler ID via Deferred

Return the scheduler ID for the scheduler with this name. If such a scheduler is already in the database, this returns the ID. If not, the scheduler is added to the database and its ID returned.

setSchedulerMaster(schedulerid, masterid)¶

Parameters:	schedulerid -- scheduler to set the master for masterid -- new master for this scheduler, or None
Returns:	Deferred

Set, or unset if masterid is None, the active master for this scheduler. If no master is currently set, or the current master is not active, this method will complete without error. If the current master is active, this method will raise SchedulerAlreadyClaimedError.

getScheduler(schedulerid)¶

Parameters:	schedulerid -- scheduler ID
Returns:	scheduler dictionary or None via Deferred

Get the scheduler dictionary for the given scheduler.

getSchedulers(active=None, masterid=None)¶

Parameters:	active (boolean) -- if specified, filter for active or inactive schedulers masterid (integer) -- if specified, only return schedulers attached associated with this master
Returns:	list of scheduler dictionaries in unspecified order

Get a list of schedulers.

If active is given, schedulers are filtered according to whether they are active (true) or inactive (false). An active scheduler is one that is claimed by an active master.

If masterid is given, the list is restricted to schedulers associated with that master.

sourcestamps¶

class buildbot.db.sourcestamps.SourceStampsConnectorComponent¶

This class manages source stamps, as stored in the database. A source stamp uniquely identifies a particular version a single codebase. Source stamps are identified by their ID. It is safe to use sourcestamp ID equality as a proxy for source stamp equality. For example, all builds of a particular version of a codebase will share the same sourcestamp ID. This equality does not extend to patches: two sourcestamps generated with exactly the same patch will have different IDs.

Relative source stamps have a revision of None, meaning "whatever the latest is when this sourcestamp is interpreted". While such source stamps may correspond to a wide array of revisions over the lifetime of a buildbot install, they will only ever have one ID.

An instance of this class is available at master.db.sourcestamps.

ssid
branch (branch, or None for default branch)
revision (revision, or None to indicate the latest revision, in which case this is a relative source stamp)
patchid (ID of the patch)
patch_body (body of the patch, or None)
patch_level (directory stripping level of the patch, or None)
patch_subdir (subdirectory in which to apply the patch, or None)
patch_author (author of the patch, or None)
patch_comment (comment for the patch, or None)
repository (repository containing the source; never None)
project (project this source is for; never None)
codebase (codebase this stamp is in; never None)
created_at (timestamp when this stamp was first created)

Note that the patch body is a bytestring, not a unicode string.

findSourceStampId(branch=None, revision=Node,

repository=None, project=None, patch_body=None,

patch_level=None, patch_author=None, patch_comment=None,

patch_subdir=None):

Parameters:

branch (unicode string or None) --
revision (unicode string or None) --
repository (unicode string or None) --
project (unicode string or None) --
codebase (unicode string (required)) --
patch_body (unicode string or None) -- patch body
patch_level (integer or None) -- patch level
patch_author (unicode string or None) -- patch author
patch_comment (unicode string or None) -- patch comment
patch_subdir (unicode string or None) -- patch subdir

Returns:

ssid, via Deferred

Create a new SourceStamp instance with the given attributes, or find an existing one. In either case, return its ssid. The arguments all have the same meaning as in an ssdict.

If a new SourceStamp is created, its created_at is set to the current time.

getSourceStamp(ssid)¶

Parameters:	ssid -- sourcestamp to get no_cache (boolean) -- bypass cache and always fetch from database
Returns:	ssdict, or `None`, via Deferred

Get an ssdict representing the given source stamp, or None if no such source stamp exists.

getSourceStamps()¶

Returns:	list of ssdict, via Deferred

Get all sourcestamps in the database. You probably don't want to do this! This method will be extended to allow appropriate filtering.

getSourceStampsForBuild(buildid)¶

Parameters:	buildid -- build ID
Returns:	list of ssdict, via Deferred

Get sourcestamps related to a build.

state¶

class buildbot.db.state.StateConnectorComponent¶

This class handles maintaining arbitrary key/value state for Buildbot objects. Each object can store arbitrary key/value pairs, where the values are any JSON-encodable value. Each pair can be set and retrieved atomically.

Objects are identified by their (user-visible) name and their class. This allows, for example, a nightly_smoketest object of class NightlyScheduler to maintain its state even if it moves between masters, but avoids cross-contaminating state between different classes of objects with the same name.

Note that "class" is not interpreted literally, and can be any string that will uniquely identify the class for the object; if classes are renamed, they can continue to use the old names.

An instance of this class is available at master.db.state.

Objects are identified by objectid.

getObjectId(name, class_name)¶

Parameters:	name -- name of the object class_name -- object class name
Returns:	the objectid, via a Deferred.

Get the object ID for this combination of a name and a class. This will add a row to the 'objects' table if none exists already.

getState(objectid, name[, default])¶

Parameters:	objectid -- objectid on which the state should be checked name -- name of the value to retrieve default -- (optional) value to return if `name` is not present
Returns:	state value via a Deferred
Raises:	KeyError -- if `name` is not present and no default is given
Raises:	TypeError if JSON parsing fails

Get the state value for key name for the object with id objectid.

setState(objectid, name, value)¶

Parameters:	objectid -- the objectid for which the state should be changed name -- the name of the value to change value (JSON-able value) -- the value to set returns -- value actually written via Deferred
Raises:	TypeError if JSONification fails

Set the state value for name for the object with id objectid, overwriting any existing value. In case of two racing writes, the first (as per db rule) one wins, the seconds returns the value from the first.

atomicCreateState(objectid, name, thd_create_callback)¶

Parameters:	objectid -- the objectid for which the state should be created name -- the name of the value to create thd_create_callback -- the function to call from thread to create the value if non-existent. (returns JSON-able value) returns -- Deferred
Raises:	TypeError if JSONification fails

Atomically creates the state value for name for the object with id objectid, If there is an existing value, returns that instead. This implementation ensures the state is created only once for the whole cluster.

Those 3 methods have their threaded equivalent, thdGetObjectId, thdGetState, thdSetState that are intended to run in synchronous code, (e.g master.cfg environment)

users¶

class buildbot.db.users.UsersConnectorComponent¶

This class handles Buildbot's notion of users. Buildbot tracks the usual information about users -- username and password, plus a display name.

The more complicated task is to recognize each user across multiple interfaces with Buildbot. For example, a user may be identified as 'djmitche' in Subversion, 'dustin@v.igoro.us' in Git, and 'dustin' on IRC. To support this functionality, each user as a set of attributes, keyed by type. The findUserByAttr method uses these attributes to match users, adding a new user if no matching user is found.

Users are identified canonically by uid, and are represented by usdicts (user dictionaries) with keys

uid
identifier (display name for the user)
bb_username (buildbot login username)
bb_password (hashed login password)

All attributes are also included in the dictionary, keyed by type. Types colliding with the keys above are ignored.

findUserByAttr(identifier, attr_type, attr_data)¶

Parameters:	identifier -- identifier to use for a new user attr_type -- attribute type to search for and/or add attr_data -- attribute data to add
Returns:	userid via Deferred

Get an existing user, or add a new one, based on the given attribute.

This method is intended for use by other components of Buildbot to search for a user with the given attributes.

Note that identifier is not used in the search for an existing user. It is only used when creating a new user. The identifier should be based deterministically on the attributes supplied, in some fashion that will seem natural to users.

For future compatibility, always use keyword parameters to call this method.

getUser(uid)¶

Parameters:	uid -- user id to look up no_cache (boolean) -- bypass cache and always fetch from database
Returns:	usdict via Deferred

Get a usdict for the given user, or None if no matching user is found.

getUserByUsername(username)¶

Parameters:	username (string) -- username portion of user credentials
Returns:	usdict or None via deferred

Looks up the user with the bb_username, returning the usdict or None if no matching user is found.

getUsers()¶

Returns:	list of partial usdicts via Deferred

Get the entire list of users. User attributes are not included, so the results are not full userdicts.

updateUser(uid=None, identifier=None, bb_username=None, bb_password=None, attr_type=None, attr_data=None)¶

Parameters:	uid (int) -- the user to change identifier (string) -- (optional) new identifier for this user bb_username (string) -- (optional) new buildbot username bb_password (string) -- (optional) new hashed buildbot password attr_type (string) -- (optional) attribute type to update attr_data (string) -- (optional) value for `attr_type`
Returns:	Deferred

Update information about the given user. Only the specified attributes are updated. If no user with the given uid exists, the method will return silently.

Note that bb_password must be given if bb_username appears; similarly, attr_type requires attr_data.

removeUser(uid)¶

Parameters:	uid (int) -- the user to remove
Returns:	Deferred

Remove the user with the given uid from the database. This will remove the user from any associated tables as well.

identifierToUid(identifier)¶

Parameters:	identifier (string) -- identifier to search for
Returns:	uid or `None`, via Deferred

Fetch a uid for the given identifier, if one exists.

masters¶

class buildbot.db.masters.MastersConnectorComponent¶

This class handles tracking the buildmasters in a multi-master configuration. Masters "check in" periodically. Other masters monitor the last activity times, and mark masters that have not recently checked in as inactive.

Masters are represented by master dictionaries with the following keys:

id -- the ID of this master
name -- the name of the master (generally of the form hostname:basedir)
active -- true if this master is running
last_active -- time that this master last checked in (a datetime object)

findMasterId(name)¶

Parameters:	name (unicode) -- name of this master
Returns:	master id via Deferred

Return the master ID for the master with this master name (generally hostname:basedir). If such a master is already in the database, this returns the ID. If not, the master is added to the database, with active=False, and its ID returned.

setMasterState(masterid, active)¶

Parameters:	masterid (integer) -- the master to check in active (boolean) -- whether to mark this master as active or inactive
Returns:	boolean via Deferred

Mark the given master as active or inactive, returning true if the state actually changed. If active is true, the last_active time is updated to the current time. If active is false, then any links to this master, such as schedulers, will be deleted.

getMaster(masterid)¶

Parameters:	masterid (integer) -- the master to check in
Returns:	Master dict or None via Deferred

Get the indicated master.

getMasters()¶

Returns:	list of Master dicts via Deferred

Get a list of the masters, represented as dictionaries; masters are sorted and paged using generic data query options

setAllMastersActiveLongTimeAgo()¶

Returns:	None via Deferred

This method is intended to be call by upgrade-master, and will effectively force housekeeping on all masters at next startup. This method is not intended to be called outside of housekeeping scripts.

builders¶

class buildbot.db.builders.BuildersConnectorComponent¶

This class handles the relationship between builder names and their IDs, as well as tracking which masters are configured for this builder.

Builders are represented by master dictionaries with the following keys:

id -- the ID of this builder
name -- the builder name, a 20-character identifier
masterids -- the IDs of the masters where this builder is configured (sorted by id)

findBuilderId(name)¶

Parameters:	name (20-character identifier) -- name of this builder
Returns:	builder id via Deferred

Return the builder ID for the builder with this builder name. If such a builder is already in the database, this returns the ID. If not, the builder is added to the database.

addBuilderMaster(builderid=None, masterid=None)¶

Parameters:	builderid (integer) -- the builder masterid (integer) -- the master
Returns:	Deferred

Add the given master to the list of masters on which the builder is configured. This will do nothing if the master and builder are already associated.

removeBuilderMaster(builderid=None, masterid=None)¶

Parameters:	builderid (integer) -- the builder masterid (integer) -- the master
Returns:	Deferred

Remove the given master from the list of masters on which the builder is configured.

getBuilder(builderid)¶

Parameters:	builderid (integer) -- the builder to check in
Returns:	Builder dict or None via Deferred

Get the indicated builder.

getBuilders(masterid=None)¶

Parameters:	masterid (integer) -- ID of the master to which the results should be limited
Returns:	list of Builder dicts via Deferred

Get all builders (in unspecified order). If masterid is given, then only builders configured on that master are returned.

3.2.3.4. Writing Database Connector Methods¶

The information above is intended for developers working on the rest of Buildbot, and treating the database layer as an abstraction. The remainder of this section describes the internals of the database implementation, and is intended for developers modifying the schema or adding new methods to the database layer.

Warning

It's difficult to change the database schema significantly after it has been released, and very disruptive to users to change the database API. Consider very carefully the future-proofing of any changes here!

The DB Connector and Components¶

class buildbot.db.connector.DBConnector¶

The root of the database connectors, master.db, is a DBConnector instance. Its main purpose is to hold reference to each of the connector components, but it also handles timed cleanup tasks.

If you are adding a new connector component, import its module and create an instance of it in this class's constructor.

class buildbot.db.base.DBConnectorComponent¶

This is the base class for connector components.

There should be no need to override the constructor defined by this base class.

db¶: A reference to the DBConnector, so that connector components can use e.g., self.db.pool or self.db.model. In the unusual case that a connector component needs access to the master, the easiest path is self.db.master.

checkLength(col, value)¶: For use by subclasses to check that 'value' will fit in 'col', where 'col' is a table column from the model. Ignore this check for database engines that either provide this error themselves (postgres) or that do not enforce maximum-length restrictions (sqlite)

findSomethingId(self, tbl, whereclause, insert_values, _race_hook=None)¶: Find (using C{whereclause}) or add (using C{insert_values) a row to C{table}, and return the resulting ID.

hashColumns(*args)¶: Hash the given values in a consistent manner: None is represented as xf5, an invalid unicode byte; strings are converted to utf8; and integers are represented by their decimal expansion. The values are then joined by '0' and hashed with sha1.

doBatch(batch, batch_n=500)¶: returns an Iterator that batches stuff in order to not push to many thing in a single request. Especially sqlite has 999 limit on argument it can take in a requests.

Direct Database Access¶

The connectors all use SQLAlchemy Core as a wrapper around database client drivers. Unfortunately, SQLAlchemy is a synchronous library, so some extra work is required to use it in an asynchronous context like Buildbot. This is accomplished by deferring all database operations to threads, and returning a Deferred. The Pool class takes care of the details.

A connector method should look like this:

def myMethod(self, arg1, arg2):
    def thd(conn):
        q = ... # construct a query
        for row in conn.execute(q):
            ... # do something with the results
        return ... # return an interesting value
    return self.db.pool.do(thd)

Picking that apart, the body of the method defines a function named thd taking one argument, a Connection object. It then calls self.db.pool.do, passing the thd function. This function is called in a thread, and can make blocking calls to SQLAlchemy as desired. The do method will return a Deferred that will fire with the return value of thd, or with a failure representing any exceptions raised by thd.

The return value of thd must not be an SQLAlchemy object - in particular, any ResultProxy objects must be parsed into lists or other data structures before they are returned.

Warning

As the name thd indicates, the function runs in a thread. It should not interact with any other part of Buildbot, nor with any of the Twisted components that expect to be accessed from the main thread -- the reactor, Deferreds, etc.

Queries can be constructed using any of the SQLAlchemy core methods, using tables from Model, and executed with the connection object, conn.

Note

SQLAlchemy requires the use of a syntax that is forbidden by pep8. If in where clauses you need to select rows where a value is NULL, you need to write (tbl.c.value == None). This form is forbidden by pep8 which requires the use of is None instead of == None. As sqlalchemy is using operator overloading to implement pythonic SQL statements, and is operator is not overloadable, we need to keep the == operators. In order to solve this issue, buildbot uses buildbot.db.NULL constant, which is None. So instead of writting tbl.c.value == None, please write tbl.c.value == NULL)

class buildbot.db.pool.DBThreadPool¶

do(callable, ...)¶

Returns:	Deferred

Call callable in a thread, with a Connection object as first argument. Returns a deferred that will fire with the results of the callable, or with a failure representing any exception raised during its execution.

Any additional positional or keyword arguments are passed to callable.

do_with_engine(callable, ...)¶

Returns:	Deferred

Similar to do, call callable in a thread, but with an Engine object as first argument.

This method is only used for schema manipulation, and should not be used in a running master.

Database Schema¶

Database connector methods access the database through SQLAlchemy, which requires access to Python objects representing the database tables. That is handled through the model.

class buildbot.db.model.Model¶

This class contains the canonical description of the buildbot schema, It is presented in the form of SQLAlchemy Table instances, as class variables. At runtime, the model is available at master.db.model, so for example the buildrequests table can be referred to as master.db.model.buildrequests, and columns are available in its c attribute.

The source file, master/buildbot/db/model.py, contains comments describing each table; that information is not replicated in this documentation.

Note that the model is not used for new installations or upgrades of the Buildbot database. See Modifying the Database Schema for more information.

metadata¶: The model object also has a metadata attribute containing a MetaData instance. Connector methods should not need to access this object. The metadata is not bound to an engine.

The Model class also defines some migration-related methods:

is_current()¶

Returns:	boolean via Deferred

Returns true if the current database's version is current.

upgrade()¶

Returns:	Deferred

Upgrades the database to the most recent schema version.

Caching¶

Connector component methods that get an object based on an ID are good candidates for caching. The cached decorator makes this automatic:

buildbot.db.base.cached(cachename)¶

Parameters:	cache_name -- name of the cache to use

A decorator for "getter" functions that fetch an object from the database based on a single key. The wrapped method will only be called if the named cache does not contain the key.

The wrapped function must take one argument (the key); the wrapper will take a key plus an optional no_cache argument which, if true, will cause it to invoke the underlying method even if the key is in the cache.

The resulting method will have a cache attribute which can be used to access the underlying cache.

In most cases, getter methods return a well-defined dictionary. Unfortunately, Python does not handle weak references to bare dictionaries, so components must instantiate a subclass of dict. The whole assembly looks something like this:

class ThDict(dict):
    pass

class ThingConnectorComponent(base.DBConnectorComponent):

    @base.cached('thdicts')
    def getThing(self, thid):
        def thd(conn):
            ...
            thdict = ThDict(thid=thid, attr=row.attr, ...)
            return thdict
        return self.db.pool.do(thd)

Tests¶

It goes without saying that any new connector methods must be fully tested!

You will also want to add an in-memory implementation of the methods to the fake classes in master/buildbot/test/fake/fakedb.py. Non-DB Buildbot code is tested using these fake implementations in order to isolate that code from the database code, and to speed-up tests.

The keys and types used in the return value from a connector's get methods are described in master/buildbot/test/util/validation.py, via the dbdict module-level value. This is a dictionary of DictValidator objects, one for each return value.

These values are used within test methods like this:

rv = yield self.db.masters.getMaster(7)
validation.verifyDbDict(self, 'masterdict', rv)

3.2.3.5. Modifying the Database Schema¶

Changes to the schema are accomplished through migration scripts, supported by SQLAlchemy-Migrate. In fact, even new databases are created with the migration scripts -- a new database is a migrated version of an empty database.

The schema is tracked by a version number, stored in the migrate_version table. This number is incremented for each change to the schema, and used to determine whether the database must be upgraded. The master will refuse to run with an out-of-date database.

To make a change to the schema, first consider how to handle any existing data. When adding new columns, this may not be necessary, but table refactorings can be complex and require caution so as not to lose information.

Create a new script in master/buildbot/db/migrate/versions, following the numbering scheme already present. The script should have an update method, which takes an engine as a parameter, and upgrades the database, both changing the schema and performing any required data migrations. The engine passed to this parameter is "enhanced" by SQLAlchemy-Migrate, with methods to handle adding, altering, and dropping columns. See the SQLAlchemy-Migrate documentation for details.

Next, modify master/buildbot/db/model.py to represent the updated schema. Buildbot's automated tests perform a rudimentary comparison of an upgraded database with the model, but it is important to check the details - key length, nullability, and so on can sometimes be missed by the checks. If the schema and the upgrade scripts get out of sync, bizarre behavior can result.

Also, adjust the fake database table definitions in master/buildbot/test/fake/fakedb.py according to your changes.

Your upgrade script should have unit tests. The classes in master/buildbot/test/util/migration.py make this straightforward. Unit test scripts should be named e.g., test_db_migrate_versions_015_remove_bad_master_objectid.py.

The master/buildbot/test/integration/test_upgrade.py also tests upgrades, and will confirm that the resulting database matches the model. If you encounter implicit indexes on MySQL, that do not appear on SQLite or Postgres, add them to implied_indexes in master/buidlbot/db/model.py.

3.2.3.6. Foreign key checking¶

PostgreSQL and SQlite db backends are checking the foreign keys consistency. bug #2248 needs to be fixed so that we can support foreign key checking for MySQL.

To maintain consistency with real db, fakedb can check the foreign key consistency of your test data. For this, just enable it with:

self.db = fakedb.FakeDBConnector(self.master, self)
self.db.checkForeignKeys = True

Note that tests that only use fakedb do not really need foreign key consistency, even if this is a good practice to enable it in new code.

3.2.3.7. Database Compatibility Notes¶

Or: "If you thought any database worked right, think again"

Because Buildbot works over a wide range of databases, it is generally limited to database features present in all supported backends. This section highlights a few things to watch out for.

In general, Buildbot should be functional on all supported database backends. If use of a backend adds minor usage restrictions, or cannot implement some kinds of error checking, that is acceptable if the restrictions are well-documented in the manual.

The metabuildbot tests Buildbot against all supported databases, so most compatibility errors will be caught before a release.

Index Length in MySQL¶

MySQL only supports about 330-character indexes. The actual index length is 1000 bytes, but MySQL uses 3-byte encoding for UTF8 strings. This is a longstanding bug in MySQL - see "Specified key was too long; max key length is 1000 bytes" with utf8. While this makes sense for indexes used for record lookup, it limits the ability to use unique indexes to prevent duplicate rows.

InnoDB only supports indexes up to 255 unicode characters, which is why all indexed columns are limited to 255 characters in Buildbot.

Transactions in MySQL¶

Unfortunately, use of the MyISAM storage engine precludes real transactions in MySQL. transaction.commit() and transaction.rollback() are essentially no-ops: modifications to data in the database are visible to other users immediately, and are not reverted in a rollback.

Referential Integrity in SQLite and MySQL¶

Neither MySQL nor SQLite enforce referential integrity based on foreign keys. Postgres does enforce, however. If possible, test your changes on Postgres before committing, to check that tables are added and removed in the proper order.

Subqueries in MySQL¶

MySQL's query planner is easily confused by subqueries. For example, a DELETE query specifying id's that are IN a subquery will not work. The workaround is to run the subquery directly, and then execute a DELETE query for each returned id.

If this weakness has a significant performance impact, it would be acceptable to conditionalize use of the subquery on the database dialect.

Too Many Variables in SQLite¶

Sqlite has a limitation on the number of variables it can use. This limitation is usually SQLITE_LIMIT_VARIABLE_NUMBER=999. There is currently no way with pysqlite to query the value of this limit. The C-api sqlite_limit is just not bound to the python.

When you hit this problem, you will get error like the following:

sqlalchemy.exc.OperationalError: (OperationalError) too many SQL variables
u'DELETE FROM scheduler_changes WHERE scheduler_changes.changeid IN (?, ?, ?, ......tons of ?? and IDs .... 9363, 9362, 9361)

You can use the method doBatch in order to write batching code in a consistent manner.

3.2.3.8. Testing migrations with real databases¶

By default Buildbot test suite uses SQLite database for testings database migrations. To use other database set BUILDBOT_TEST_DB_URL environment variable to value in SQLAlchemy database URL specification.

For example, to run tests with file-based SQLite database you can start tests in the following way:

BUILDBOT_TEST_DB_URL=sqlite:////tmp/test_db.sqlite trial buildbot.test

Run databases in Docker¶

Docker allows to easily install and configure different databases locally in containers.

To run tests with PostgreSQL:

# Install psycopg.
pip install psycopg2
# Start container with PostgreSQL 9.5.
# It will listen on port 15432 on localhost.
sudo docker run --name bb-test-postgres -e POSTGRES_PASSWORD=password \
    -p 127.0.0.1:15432:5432 -d postgres:9.5
# Start interesting tests
BUILDBOT_TEST_DB_URL=postgresql://postgres:password@localhost:15432/postgres \
    trial buildbot.test

To run tests with MySQL:

# Install mysqlclient
pip install mysqlclient
# Start container with MySQL 5.5.
# It will listen on port 13306 on localhost.
sudo docker run --name bb-test-mysql -e MYSQL_ROOT_PASSWORD=password \
    -p 127.0.0.1:13306:3306 -d mysql:5.5
# Start interesting tests
BUILDBOT_TEST_DB_URL=mysql+mysqldb://root:password@127.0.0.1:13306/mysql \
    trial buildbot.test

3.2.3. Database¶

3.2.3.1. Database Overview¶

3.2.3.2. Schema¶

3.2.3.3. API¶

types¶

Identifier¶

buildrequests¶

builds¶

steps¶

logs¶

buildsets¶

workers¶

changes¶

changesources¶

schedulers¶

sourcestamps¶

state¶

users¶

masters¶

builders¶

3.2.3.4. Writing Database Connector Methods¶

The DB Connector and Components¶

Direct Database Access¶

Database Schema¶

Caching¶

Tests¶

3.2.3.5. Modifying the Database Schema¶

3.2.3.6. Foreign key checking¶

3.2.3.7. Database Compatibility Notes¶

Index Length in MySQL¶

Transactions in MySQL¶

Referential Integrity in SQLite and MySQL¶

Subqueries in MySQL¶

Too Many Variables in SQLite¶

3.2.3.8. Testing migrations with real databases¶

Run databases in Docker¶

Table Of Contents

Previous topic

Next topic

This Page