Package buildbot :: Package steps :: Module python_twisted :: Class Trial
[frames] | no frames]

Class Trial

source code

   process.buildstep.BuildStep --+        
                                 |        
process.buildstep.LoggingBuildStep --+    
                                     |    
                    shell.ShellCommand --+
                                         |
                                        Trial
Known Subclasses:

I run a unit test suite using 'trial', a unittest-like testing framework that comes with Twisted. Trial is used to implement Twisted's own unit tests, and is the unittest-framework of choice for many projects that use Twisted internally.

Projects that use trial typically have all their test cases in a 'test' subdirectory of their top-level library directory. I.e. for my package 'petmail', the tests are in 'petmail/test/test_*.py'. More complicated packages (like Twisted itself) may have multiple test directories, like 'twisted/test/test_*.py' for the core functionality and 'twisted/mail/test/test_*.py' for the email-specific tests.

To run trial tests, you run the 'trial' executable and tell it where the test cases are located. The most common way of doing this is with a module name. For petmail, I would run 'trial petmail.test' and it would locate all the test_*.py files under petmail/test/, running every test case it could find in them. Unlike the unittest.py that comes with Python, you do not run the test_foo.py as a script; you always let trial do the importing and running. The 'tests' parameter controls which tests trial will run: it can be a string or a list of strings.

To find these test cases, you must set a PYTHONPATH that allows something like 'import petmail.test' to work. For packages that don't use a separate top-level 'lib' directory, PYTHONPATH=. will work, and will use the test cases (and the code they are testing) in-place. PYTHONPATH=build/lib or PYTHONPATH=build/lib.$ARCH are also useful when you do a'setup.py build' step first. The 'testpath' attribute of this class controls what PYTHONPATH= is set to.

Trial has the ability (through the --testmodule flag) to run only the set of test cases named by special 'test-case-name' tags in source files. We can get the list of changed source files from our parent Build and provide them to trial, thus running the minimal set of test cases needed to cover the Changes. This is useful for quick builds, especially in trees with a lot of test cases. The 'testChanges' parameter controls this feature: if set, it will override 'tests'.

The trial executable itself is typically just 'trial' (which is usually found on your $PATH as /usr/bin/trial), but it can be overridden with the 'trial' parameter. This is useful for Twisted's own unittests, which want to use the copy of bin/trial that comes with the sources. (when bin/trial discovers that it is living in a subdirectory named 'Twisted', it assumes it is being run from the source tree and adds that parent directory to PYTHONPATH. Therefore the canonical way to run Twisted's own unittest suite is './bin/trial twisted.test' rather than 'PYTHONPATH=. /usr/bin/trial twisted.test', especially handy when /usr/bin/trial has not yet been installed).

To influence the version of python being used for the tests, or to add flags to the command, set the 'python' parameter. This can be a string (like 'python2.2') or a list (like ['python2.3', '-Wall']).

Trial creates and switches into a directory named _trial_temp/ before running the tests, and sends the twisted log (which includes all exceptions) to a file named test.log . This file will be pulled up to the master where it can be seen as part of the status output.

There are some class attributes which may be usefully overridden by subclasses. 'trialMode' and 'trialArgs' can influence the trial command line.

Instance Methods
 
__init__(self, reactor=(), python=None, trial=None, testpath=(), tests=None, testChanges=None, recurse=None, randomly=None, trialMode=None, trialArgs=None, **kwargs) source code
 
setupEnvironment(self, cmd) source code
 
start(self)
Begin the step.
source code
 
commandComplete(self, cmd)
This is a general-purpose hook method for subclasses.
source code
 
rtext(self, fmt='%s') source code
 
addTestResult(self, testname, results, text, tlog) source code
 
createSummary(self, loog)
To create summary logs, do something like this: warnings = grep('^Warning:', log.getText()) self.addCompleteLog('warnings', warnings)
source code
 
evaluateCommand(self, cmd)
Decide whether the command was SUCCESS, WARNINGS, or FAILURE.
source code
 
getText(self, cmd, results) source code
 
getText2(self, cmd, results)
We have decided to add a short note about ourselves to the overall build description, probably because something went wrong.
source code

Inherited from shell.ShellCommand: checkForOldSlaveAndLogfiles, describe, setCommand, setDefaultWorkdir, setStepStatus

Inherited from process.buildstep.LoggingBuildStep: addLogFile, checkDisconnect, interrupt, maybeGetText2, setStatus, setupLogfiles, startCommand

Inherited from process.buildstep.BuildStep: acquireLocks, addCompleteLog, addFactoryArguments, addHTMLLog, addLog, addLogObserver, addURL, failed, finished, getLog, getProperty, getSlaveName, getStepFactory, releaseLocks, runCommand, setBuild, setBuildSlave, setProgress, setProperty, setupProgress, slaveVersion, slaveVersionIsOlderThan, startStep

Class Variables
  name = 'trial'
  progressMetrics = ('output', 'tests', 'test.log')
  logfiles = {'test.log': '_trial_temp/test.log'}
a dict mapping log NAMEs to workdir-relative FILENAMEs of their corresponding logfiles.
  flunkOnFailure = True
  python = None
  trial = 'trial'
  trialMode = ['--reporter=bwverbose']
  trialArgs = []
  testpath = ()
  testChanges = False
  recurse = False
  reactor = None
  randomly = False
  tests = None

Inherited from shell.ShellCommand: description, descriptionDone

Inherited from process.buildstep.LoggingBuildStep: parms

Inherited from process.buildstep.BuildStep: alwaysRun, doStepIf, flunkOnWarnings, haltOnFailure, locks, useProgress, warnOnFailure, warnOnWarnings

Instance Variables

Inherited from shell.ShellCommand: command, lazylogfiles

Inherited from process.buildstep.BuildStep: build, progress, step_status

Method Details

__init__(self, reactor=(), python=None, trial=None, testpath=(), tests=None, testChanges=None, recurse=None, randomly=None, trialMode=None, trialArgs=None, **kwargs)
(Constructor)

source code 
Parameters:
  • testpath (string) - use in PYTHONPATH when running the tests. If None, do not set PYTHONPATH. Setting this to '.' will cause the source files to be used in-place.
  • python (string (without spaces) or list) - which python executable to use. Will form the start of the argv array that will launch trial. If you use this, you should set 'trial' to an explicit path (like /usr/bin/trial or ./bin/trial). Defaults to None, which leaves it out entirely (running 'trial args' instead of 'python ./bin/trial args'). Likely values are 'python', ['python2.2'], ['python', '-Wall'], etc.
  • trial (string) - which 'trial' executable to run. Defaults to 'trial', which will cause $PATH to be searched and probably find /usr/bin/trial . If you set 'python', this should be set to an explicit path (because 'python2.3 trial' will not work).
  • trialMode (list of strings) - a list of arguments to pass to trial, specifically to set the reporting mode. This defaults to ['-to'] which means 'verbose colorless output' to the trial that comes with Twisted-2.0.x and at least -2.1.0 . Newer versions of Twisted may come with a trial that prefers ['--reporter=bwverbose'].
  • trialArgs (list of strings) - a list of arguments to pass to trial, available to turn on any extra flags you like. Defaults to [].
  • tests (list of strings) - a list of test modules to run, like ['twisted.test.test_defer', 'twisted.test.test_process']. If this is a string, it will be converted into a one-item list.
  • testChanges (boolean) - if True, ignore the 'tests' parameter and instead ask the Build for all the files that make up the Changes going into this build. Pass these filenames to trial and ask it to look for test-case-name tags, running just the tests necessary to cover the changes.
  • recurse (boolean) - If True, pass the --recurse option to trial, allowing test cases to be found in deeper subdirectories of the modules listed in 'tests'. This does not appear to be necessary when using testChanges.
  • reactor (string) - which reactor to use, like 'gtk' or 'java'. If not provided, the Twisted's usual platform-dependent default is used.
  • randomly (boolean) - if True, add the --random=0 argument, which instructs trial to run the unit tests in a random order each time. This occasionally catches problems that might be masked when one module always runs before another (like failing to make registerAdapter calls before lookups are done).
  • kwargs (dict) - parameters. The following parameters are inherited from ShellCommand and may be useful to set: workdir, haltOnFailure, flunkOnWarnings, flunkOnFailure, warnOnWarnings, warnOnFailure, want_stdout, want_stderr, timeout.
Overrides: process.buildstep.BuildStep.__init__

setupEnvironment(self, cmd)

source code 
Overrides: shell.ShellCommand.setupEnvironment

start(self)

source code 

Begin the step. Override this method and add code to do local processing, fire off remote commands, etc.

To spawn a command in the buildslave, create a RemoteCommand instance and run it with self.runCommand:

 c = RemoteCommandFoo(args)
 d = self.runCommand(c)
 d.addCallback(self.fooDone).addErrback(self.failed)

As the step runs, it should send status information to the BuildStepStatus:

 self.step_status.setText(['compile', 'failed'])
 self.step_status.setText2(['4', 'warnings'])

To have some code parse stdio (or other log stream) in realtime, add a LogObserver subclass. This observer can use self.step.setProgress() to provide better progress notification to the step.:

 self.addLogObserver('stdio', MyLogObserver())

To add a LogFile, use self.addLog. Make sure it gets closed when it finishes. When giving a Logfile to a RemoteShellCommand, just ask it to close the log when the command completes:

 log = self.addLog('output')
 cmd = RemoteShellCommand(args)
 cmd.useLog(log, closeWhenFinished=True)

You can also create complete Logfiles with generated text in a single step:

 self.addCompleteLog('warnings', text)

When the step is done, it should call self.finished(result). 'result' will be provided to the buildbot.process.base.Build, and should be one of the constants defined above: SUCCESS, WARNINGS, FAILURE, or SKIPPED.

If the step encounters an exception, it should call self.failed(why). 'why' should be a Failure object. This automatically fails the whole build with an exception. It is a good idea to add self.failed as an errback to any Deferreds you might obtain.

If the step decides it does not need to be run, start() can return the constant SKIPPED. This fires the callback immediately: it is not necessary to call .finished yourself. This can also indicate to the status-reporting mechanism that this step should not be displayed.

A step can be configured to only run under certain conditions. To do this, set the step's doStepIf to a boolean value, or to a function that returns a boolean value. If the value or function result is False, then the step will return SKIPPED without doing anything, otherwise the step will be executed normally. If you set doStepIf to a function, that function should accept one parameter, which will be the Step object itself.

Overrides: process.buildstep.BuildStep.start
(inherited documentation)

commandComplete(self, cmd)

source code 

This is a general-purpose hook method for subclasses. It will be called after the remote command has finished, but before any of the other hook functions are called.

Overrides: process.buildstep.LoggingBuildStep.commandComplete
(inherited documentation)

createSummary(self, loog)

source code 

To create summary logs, do something like this: warnings = grep('^Warning:', log.getText()) self.addCompleteLog('warnings', warnings)

Overrides: process.buildstep.LoggingBuildStep.createSummary
(inherited documentation)

evaluateCommand(self, cmd)

source code 

Decide whether the command was SUCCESS, WARNINGS, or FAILURE. Override this to, say, declare WARNINGS if there is any stderr activity, or to say that rc!=0 is not actually an error.

Overrides: process.buildstep.LoggingBuildStep.evaluateCommand
(inherited documentation)

getText(self, cmd, results)

source code 
Overrides: process.buildstep.LoggingBuildStep.getText

getText2(self, cmd, results)

source code 

We have decided to add a short note about ourselves to the overall build description, probably because something went wrong. Return a short list of short strings. If your subclass counts test failures or warnings of some sort, this is a good place to announce the count.

Overrides: process.buildstep.LoggingBuildStep.getText2
(inherited documentation)