3.5.22. Test result set¶

resource type: test_result_set

Attributes

test_result_setid (integer) – the unique ID of this test result set.
builderid (integer) – id of the builder for this test result set.
buildid (integer) – id of the build for this test result set.
stepid (integer) – id of the step for this test result set.
description (string) – Free-form description of the source of the test data
category (string) –
The category of the test result set. This describes what data the test results contain.

Any value is allowed. The following standard categories are defined:
- pass_fail: The test result set contains results that can indicate success or failure of specific test. The values of test results contain success or failure values.
- pass_only: The test result set contains results that can only indicate success of specific test. This is used in cases when failed tests are not reported.
- fail_only: The test result set contains results that can only indicate failure of specific test. This is used in tests when passed tests are not reported.
- code_issue: The test result set contains issues within the code reported by various tooling. This is effectively a subset of fail_only.
- performance: The test result set contains performance results. The values of test results contain some kind of performance metric such as time per operation or the number of operations completed in a time period.
- binary_size: The test result set contains evaluation of binary size. The values of test results contain a binary size metric.
- memory_use: The test result set contains evaluation of dynamic memory use. The values of test results contain a memory use metric.
value_unit (string) –
Describes the unit of the values stored within the test results.

Any value is allowed. The following standard units are defined:
- ps: Picoseconds
- ns: Nanoseconds
- us: Microseconds
- ms: Milliseconds
- s: Seconds
- boolean: A boolean value (0 or 1)
- B: Bytes
- KB: Kilobytes (1000-based)
- KiB: Kibibytes (1024-based)
- MB: Megabytes (1000-based)
- MiB: Mebibytes (1024-based)
- GB: Gigabytes (1000-based)
- GiB: Gibibytes (1024-based)
- TB: Gigabytes (1000-based)
- TiB: Gibibytes (1024-based)
- message: Arbitrary string message
Note that the value of the test result is always stored as string.
tests_passed? (integer) – The number of passed tests in cases when the pass or fail criteria depends only on how that single test runs. For example, performance tests that track regressions across multiple tests do not have the number of passed tests defined.
tests_failed? (integer) – The number of failed tests in cases when the pass or fail criteria depends only on how that single test runs. For example, performance tests that track regressions across multiple tests do not have the number of failed tests defined.
complete (boolean) – true if all test results associated with test result set have been generated. Once set to true this property will never be set back to false

example

{
    "test_result_setid": 412,
    "builderid": 14,
    "buildid": 31,
    "stepid": 3,
    "description": "Performance test via BenchmarkDotNet",
    "category": "performance",
    "value_unit": "ms",
    "complete": true
}

This resource represents a test result set. A test result set consists of a number of related test results. These test results need to be related in that they represent the same type of data and are produced by a single step. In reasonably tested codebases the number of test results in a test result set will approach several or even tens of thousands.

There may be a long delay between the creation of the test result set and full creation of the corresponding test results. This is tracked by the complete property. If it’s true, then the full set of test results have been committed to the database.

The test_result_unparsed_set object tracks test result sets that have not been parsed yet.

3.5.22.1. Update Methods¶

All update methods are available as attributes of master.data.updates.

class buildbot.data.test_result_sets.TestResultSet¶

addTestResultSet(builderid, buildid, stepid, description, category, value_unit)¶

Parameters

builderid (integer) – The ID of the builder for which the test result set is to be created.
buildid (integer) – The ID of the build for which the test result set is to be created.
stepid (integer) – The ID of the step for which the test result set is to be created.
description – Description of the test result set.
category – The category of the test result set.
value_unit – Defines the unit of the values stored in the test results.

Creates a new test result set. Returns the ID of the new test result set.

completeTestResultSet(test_result_setid, tests_passed=None, tests_failed=None):

Parameters

test_result_setid (integer) – The ID of the test result set to complete.
tests_passed (integer) – The number of passed tests, if known
tests_failed (integer) – The number of failed tests, if known

Marks a test result set as complete. The total number of passed and failed tests may be passed to have this information cached as part of a test result set so that expensive re-computations don’t need to be performed.

3.5.22.2. Endpoints¶

path: /builders/{builderid_or_buildername}/test_result_sets

Path Keys: | identifier builderid_or_buildername (number) – the ID or name of the builder

This selects all test result sets that have been created for a particular builder

GET

returns: collection of test_result_set

path: /builds/{buildid}/steps/{step_number_or_name}/test_result_sets

Path Keys

buildid (number) – the id of the build
| number step_number_or_name (identifier) – the name or number of the step

This selects all test result sets that have been created for a particular step

GET

returns: collection of test_result_set

path: /builds/{buildid}/test_result_sets

Path Keys: buildid (number) – the id of the build

This selects all test result sets that have been created for a particular build

GET

returns: collection of test_result_set

path: /steps/{stepid}/test_result_sets

Path Keys: stepid (number) – the id of the step

This selects all test result sets that have been created for a particular step

GET

returns: collection of test_result_set

path: /test_result_sets/{test_result_setid}

Path Keys: test_result_setid (number) – the id of the test result set

Selects a test result set by id

GET

returns: collection of test_result_set