Package buildbot :: Package process :: Module mtrlogobserver :: Class MTR
[frames] | no frames]

Class MTR

source code

        buildstep.BuildStep --+                
                              |                
     buildstep.LoggingBuildStep --+            
                                  |            
           steps.shell.ShellCommand --+        
                                      |        
steps.shell.WarningCountingShellCommand --+    
                                          |    
                           steps.shell.Test --+
                                              |
                                             MTR


Build step that runs mysql-test-run.pl, as used in MySQL, Drizzle,
MariaDB, etc.

It uses class MtrLogObserver to parse test results out from the
output of mysql-test-run.pl, providing better completion time
estimates and summarising test failures on the waterfall page.

It also provides access to mysqld server error logs from the test
run to help debugging any problems.

Optionally, it can insert into a database data about the test run,
including details of any test failures.

Parameters:

textLimit
    Maximum number of test failures to show on the waterfall page
    (to not flood the page in case of a large number of test
    failures. Defaults to 5.

testNameLimit
    Maximum length of test names to show unabbreviated in the
    waterfall page, to avoid excessive column width. Defaults to 16.

parallel
    Value of --parallel option used for mysql-test-run.pl (number
    of processes used to run the test suite in parallel). Defaults
    to 4. This is used to determine the number of server error log
    files to download from the slave. Specifying a too high value
    does not hurt (as nonexisting error logs will be ignored),
    however if using --parallel value greater than the default it
    needs to be specified, or some server error logs will be
    missing.

dbpool
    An instance of twisted.enterprise.adbapi.ConnectionPool, or None.
    Defaults to None. If specified, results are inserted into the database
    using the ConnectionPool.

    The class process.mtrlogobserver.EqConnectionPool subclass of
    ConnectionPool can be useful to pass as value for dbpool, to
    avoid having config reloads think the Buildstep is changed
    just because it gets a new ConnectionPool instance (even
    though connection parameters are unchanged).

autoCreateTables
    Boolean, defaults to False. If True (and dbpool is specified), the
    necessary database tables will be created automatically if they do
    not exist already. Alternatively, the tables can be created manually
    from the SQL statements found in the mtrlogobserver.py source file.

test_type
test_info
    Two descriptive strings that will be inserted in the database tables if
    dbpool is specified. The test_type string, if specified, will also
    appear on the waterfall page.

Nested Classes
  MyMtrLogObserver
Instance Methods
 
__init__(self, dbpool=None, test_type=None, test_info='', description=None, descriptionDone=None, autoCreateTables=False, textLimit=5, testNameLimit=16, parallel=4, logfiles={}, lazylogfiles=True, warningPattern='MTR\'s internal check of the test case \'.*\' failed', mtr_subdir='mysql-test', **kwargs) source code
 
start(self)
Begin the step.
source code
 
getText(self, command, results) source code
 
runInteractionWithRetry(self, actionFn, *args, **kw)
Run a database transaction with dbpool.runInteraction, but retry the transaction in case of a temporary error (like connection lost).
source code
 
runQueryWithRetry(self, *args, **kw)
Run a database query, like with dbpool.runQuery, but retry the query in case of a temporary error (like connection lost).
source code
 
registerInDB(self) source code
 
doRegisterInDB(self, txn) source code
 
afterRegisterInDB(self, insert_id) source code
 
reportError(self, err) source code

Inherited from steps.shell.Test: describe, setTestResults

Inherited from steps.shell.WarningCountingShellCommand: addSuppression, createSummary, evaluateCommand, maybeAddWarning, setDefaultWorkdir, uploadDone, warnExtractFromRegexpGroups, warnExtractWholeLine

Inherited from steps.shell.ShellCommand: checkForOldSlaveAndLogfiles, setCommand, setStepStatus, setupEnvironment

Inherited from buildstep.LoggingBuildStep: addLogFile, checkDisconnect, commandComplete, getText2, interrupt, maybeGetText2, setStatus, setupLogfiles, startCommand

Inherited from buildstep.BuildStep: acquireLocks, addCompleteLog, addFactoryArguments, addHTMLLog, addLog, addLogObserver, addURL, failed, finished, getLog, getProperty, getSlaveName, getStepFactory, releaseLocks, runCommand, setBuild, setBuildSlave, setProgress, setProperty, setupProgress, slaveVersion, slaveVersionIsOlderThan, startStep

Class Variables

Inherited from steps.shell.Test: command, description, descriptionDone, name, warnOnFailure

Inherited from steps.shell.WarningCountingShellCommand: commentEmptyLineRe, directoryEnterPattern, directoryLeavePattern, suppressionFile, suppressionLineRe, warnCount, warningPattern

Inherited from steps.shell.ShellCommand: flunkOnFailure

Inherited from buildstep.LoggingBuildStep: cmd, parms, progressMetrics

Inherited from buildstep.BuildStep: alwaysRun, doStepIf, flunkOnWarnings, haltOnFailure, locks, useProgress, warnOnWarnings

Instance Variables

Inherited from steps.shell.ShellCommand: lazylogfiles, logfiles

Inherited from buildstep.BuildStep: build, progress, step_status

Method Details

__init__(self, dbpool=None, test_type=None, test_info='', description=None, descriptionDone=None, autoCreateTables=False, textLimit=5, testNameLimit=16, parallel=4, logfiles={}, lazylogfiles=True, warningPattern='MTR\'s internal check of the test case \'.*\' failed', mtr_subdir='mysql-test', **kwargs)
(Constructor)

source code 
Overrides: buildstep.BuildStep.__init__

start(self)

source code 

Begin the step. Override this method and add code to do local processing, fire off remote commands, etc.

To spawn a command in the buildslave, create a RemoteCommand instance and run it with self.runCommand:

 c = RemoteCommandFoo(args)
 d = self.runCommand(c)
 d.addCallback(self.fooDone).addErrback(self.failed)

As the step runs, it should send status information to the BuildStepStatus:

 self.step_status.setText(['compile', 'failed'])
 self.step_status.setText2(['4', 'warnings'])

To have some code parse stdio (or other log stream) in realtime, add a LogObserver subclass. This observer can use self.step.setProgress() to provide better progress notification to the step.:

 self.addLogObserver('stdio', MyLogObserver())

To add a LogFile, use self.addLog. Make sure it gets closed when it finishes. When giving a Logfile to a RemoteShellCommand, just ask it to close the log when the command completes:

 log = self.addLog('output')
 cmd = RemoteShellCommand(args)
 cmd.useLog(log, closeWhenFinished=True)

You can also create complete Logfiles with generated text in a single step:

 self.addCompleteLog('warnings', text)

When the step is done, it should call self.finished(result). 'result' will be provided to the buildbot.process.base.Build, and should be one of the constants defined above: SUCCESS, WARNINGS, FAILURE, or SKIPPED.

If the step encounters an exception, it should call self.failed(why). 'why' should be a Failure object. This automatically fails the whole build with an exception. It is a good idea to add self.failed as an errback to any Deferreds you might obtain.

If the step decides it does not need to be run, start() can return the constant SKIPPED. This fires the callback immediately: it is not necessary to call .finished yourself. This can also indicate to the status-reporting mechanism that this step should not be displayed.

A step can be configured to only run under certain conditions. To do this, set the step's doStepIf to a boolean value, or to a function that returns a boolean value. If the value or function result is False, then the step will return SKIPPED without doing anything, otherwise the step will be executed normally. If you set doStepIf to a function, that function should accept one parameter, which will be the Step object itself.

Overrides: buildstep.BuildStep.start
(inherited documentation)

getText(self, command, results)

source code 
Overrides: buildstep.LoggingBuildStep.getText

runInteractionWithRetry(self, actionFn, *args, **kw)

source code 

Run a database transaction with dbpool.runInteraction, but retry the transaction in case of a temporary error (like connection lost).

This is needed to be robust against things like database connection idle timeouts.

The passed callable that implements the transaction must be retryable, ie. it must not have any destructive side effects in the case where an exception is thrown and/or rollback occurs that would prevent it from functioning correctly when called again.

runQueryWithRetry(self, *args, **kw)

source code 

Run a database query, like with dbpool.runQuery, but retry the query in case of a temporary error (like connection lost).

This is needed to be robust against things like database connection idle timeouts.