fbtest Reference

Globals

fbtest.DB_NEW = 'New'

str(object=’‘) -> string

Return a nice string representation of the object. If the argument is a string, the return value is the same object.

fbtest.DB_EXISTING = 'Existing'

str(object=’‘) -> string

Return a nice string representation of the object. If the argument is a string, the return value is the same object.

fbtest.DB_RESTORE = 'Restore'

str(object=’‘) -> string

Return a nice string representation of the object. If the argument is a string, the return value is the same object.

fbtest.DB_ACCESS = [None, 'New', 'Existing', 'Restore']

list() -> new empty list list(iterable) -> new list initialized from iterable’s items

fbtest.CHARACTER_SETS = [None, 'NONE', 'ASCII', 'BIG_5', 'CYRL', 'DOS437', 'DOS737', 'DOS775', 'DOS850', 'DOS852', 'DOS857', 'DOS858', 'DOS860', 'DOS861', 'DOS862', 'DOS863', 'DOS864', 'DOS865', 'DOS866', 'DOS869', 'EUCJ_0208', 'GBK', 'GB_2312', 'ISO8859_1', 'ISO8859_2', 'ISO8859_3', 'ISO8859_4', 'ISO8859_5', 'ISO8859_6', 'ISO8859_7', 'ISO8859_8', 'ISO8859_9', 'ISO8859_13', 'KOI8R', 'KOI8U', 'KSC_5601', 'NEXT', 'OCTETS', 'SJIS_0208', 'TIS620', 'UNICODE_FSS', 'UTF8', 'WIN1250', 'WIN1251', 'WIN1252', 'WIN1253', 'WIN1254', 'WIN1255', 'WIN1256', 'WIN1257', 'WIN1258', 'LATIN2']

list() -> new empty list list(iterable) -> new list initialized from iterable’s items

fbtest.PAGE_SIZES = [None, '1024', '2048', '4096', '8192', '16384']

list() -> new empty list list(iterable) -> new list initialized from iterable’s items

fbtest.TYPE_ISQL = 'ISQL'

str(object=’‘) -> string

Return a nice string representation of the object. If the argument is a string, the return value is the same object.

fbtest.TYPE_PYTHON = 'Python'

str(object=’‘) -> string

Return a nice string representation of the object. If the argument is a string, the return value is the same object.

fbtest.TEST_TYPES = ['ISQL', 'Python']

list() -> new empty list list(iterable) -> new list initialized from iterable’s items

fbtest.PLATFORMS = ['Windows', 'Linux', 'MacOS', 'FreeBSD', 'Solaris', 'HP-UX']

list() -> new empty list list(iterable) -> new list initialized from iterable’s items

fbtest.UNKNOWN = 'Unknown'

str(object=’‘) -> string

Return a nice string representation of the object. If the argument is a string, the return value is the same object.

fbtest.script_runner

ScriptRunner instance.

Functions

fbtest.xml_safe(value)

Replaces invalid XML characters with ‘?’.

fbtest.escape_cdata(cdata)

Escape a string for an XML CDATA section.

fbtest.trim_value(value)

Return string with trailing whitespaces from each line removed.

fbtest.quote(value)

Return properly quoted string according its content.

fbtest.compare_versions(lver, rver)

Compare two version strings, returns: 0 for lver = rver -1 for lver < rver 1 for lver > rver

fbtest.as_unicode(value)

Make sure that string is unicode. Accepts Unicode, strings encoded in UTF-8 or None and returns Unicode string or None.

fbtest.as_utf8(value)

Make sure that string is encoded in UTF-8. Accepts Unicode, strings or None. If paremeter is unicode, it’s returned as string encoded in UTF-8. String or None parameter is returned as is.

fbtest.runProgram(args, environment, stdin=None, **kwargs)

Run external program using Popen from subprocess module and capture stdin and stderr.

Parameters:
  • args (list) – List of arguments for subprocess.Popen. First item must be specification of program to run.
  • environment (dict) – Dictionary of environment variables. Ignored in this version.
  • stdin (string) – String to be passed as stdin to external program.
  • kwarg (dict) – Dictionary of additional keyword arguments for Popen.
Returns:

Tuple of (returncode, stdout, stderr)

Classes

TestVersion class

class fbtest.TestVersion(id, platform, firebird_version, test_type, test_script, database='New', expected_stdout='', expected_stderr='', database_name=None, backup_file=None, user_name='SYSDBA', user_password='masterkey', database_character_set=None, connection_character_set=None, page_size=None, sql_dialect=3, init_script='', resources=None, substitutions=None, qmid=None)

Recipe for test execution against specific Firebird version and OS platform.

Parameters:
  • id (string) – Test ID (dot-separated name).
  • platform (string) – List of platform names separated by colon.
  • firebird_version (string) – First Firebird version this test version is designed for.
  • test_type (string) – Test implemetation method: “ISQL” or “Python”.
  • test_script (string) – Test code.
  • database (string) – Database usage specification: None, “New”, “Existing” or “Restore”.
  • expected_stdout (string) – Expected STDOUT content.
  • expected_stderr (string) – Expected STDERR content.
  • database_name (string) – Database file name.
  • backup_file (string) – Backup file name.
  • user_name (string) – User name.
  • user_password (string) – User password.
  • database_character_set (string) – Character set for database.
  • connection_character_set (string) – Character set for connection.
  • page_size (string) – Page size for database.
  • sql_dialect (integer) – SQL dialect for database.
  • init_script (string) – Test initialization script.
  • resources (list) – List of Resource names.
  • substitutions (list) – List of substitution specifications.
  • qmid (string) – Test ID from old system (QMTest).
as_expression()

Return recipe data as string definition of Python dictionary. This string could be evaluated back to Python dictionary and passed to TestVersion constructor to recreate the test version instance. String is encoded in UTF-8 if necessary.

Note

Only attributes that haven’t DEFAULT values are included.

get_platforms()

Returns platforms supported by this test version as list of platform names.

run(context, result)

Execute the recipe.

Important

Test run outcome is stored in Result instance.

Parameters:
FIELDS = ['id', 'qmid', 'firebird_version', 'platform', 'database', 'database_name', 'backup_file', 'user_name', 'user_password', 'database_character_set', 'connection_character_set', 'page_size', 'sql_dialect', 'init_script', 'test_type', 'test_script', 'expected_stdout', 'expected_stderr', 'resources', 'substitutions']

Test class

class fbtest.Test(id, title='', description='', tracker_id='', min_versions=None, versions=None, qmid=None)

Test definition.

Can contain multiple TestVersion recipes for test execution against different Firebird versions on various OS platforms.

Parameters:
  • id (string) – Test ID (dot-separated name).
  • title (string) – Test title.
  • description (string) – test description.
  • tracker_id (string) – JIRA entry ID this tests relates to.
  • min_versions (string) – List of minimal engine versions supported by test (separated by semicolon).
  • versions (list) – List of disctionaries with data for creation of TestVersion instances.
  • qmid (string) – Test ID from old system (QMTest).
add_version(version)

Add TestVersion instance to test.

Parameters:versionTestVersion instance.
as_expression()

Return test data as string definition of Python dictionary. This string could be evaluated back to Python dictionary and passed to Test constructor to recreate the test instance. String is encoded in UTF-8 if necessary.

Note

Only attributes that haven’t DEFAULT values are included.

get_name()

Return test name.

Test name is last part of test ID.

get_version_for(platform, version)

Return test recipe suitable for specified platform and Firebird version.

Parameters:
  • platform (string) – Platform name.
  • version (string) – Firebird version.
Returns:

TestVersion or None.

FIELDS = ['id', 'qmid', 'tracker_id', 'title', 'description', 'min_versions', 'versions']

Resource class

class fbtest.Resource(id)

Base class for test resources.

Parameters:id (string) – Resource ID.
as_expression()

Return resource data as string definition of Python dictionary. This string could be evaluated back to Python dictionary and passed to Resource constructor to recreate the resource instance. String is encoded in UTF-8 if necessary.

cleanup(result)

Resource finalization

Nothing in this class.

classmethod create(kind, **kwargs)

Class method to create right Resource instance from parameters.

Parameters:kind (string) – Resource class specification.
fail_and_annotate_streams(result, outcome, program_name, cause, stdout=None, stderr=None, annotate={})

Helper method to note fail of resource setup/cleanup in Result instance.

Parameters:
  • resultResult instance.
  • outcome (stirng) – Run outcome.
  • program_name (string) – Program that reports the failure.
  • cause (string) – Fail reason.
  • stdout (string) – stdout content.
  • stderr (string) – stderr content.
  • annotate (dict) – Dictionary of annotations.
setup(context, result)

Resource initialization.

Nothing in this class.

UserResource class

class fbtest.UserResource(id, user_name, user_password)

Test resource that provides Firebird USER.

Parameters:
  • id (string) – resource ID.
  • user_name (string) – Firebird user name.
  • user_password (string) – Firebird user password.
cleanup(context, result)

Remove specified Firebird user via GSEC.

setup(context, result)

Create Firebird user with specified name and password via GSEC.

FIELD = ['id', 'user_name', 'user_password']

Suite class

class fbtest.Suite(name, path=None, parent=None)

Container for tests and sub-suites to organize tests in hierarchical structure.

Parameters:
  • name (string) – Suite name.
  • path (string) – Directory with tests and sub-suites.
  • parent (string) – Parent suite name or None.
add_test(test)

Add test to suite.

Parameters:testTest instance.
clear()

Delete all tests and sub-suites from instance.

get_id()

Return suite ID.

Consists from suite names from root to this one separated by dot.

get_parent()

Return parent suite or None

get_tests()

Return all tests for this suite (including tests from sub-suites).

load()

Load tests and sub-suites from path.

reload_test(test_id)

Reload test from disk.

Parameters:test_id (string) – Test ID.
save_test(test_id)

Save test on disk.

Parameters:test_id – Test ID.

Repository class

class fbtest.Repository(repository_path)

Test repository.

All subdirectories must exists except ‘archive’ that’s created if necessary.

Parameters:repository_path (string) – Directory with test repository.
get_suite(suite_id='')

Return suite with specified ID.

Returns:Suite instance or None.
get_test(test_id)

Return test with specified ID.

Returns:Test instance or None.
load()

Load all tests and resources.

Archive class

class fbtest.Archive(archive_path)

Run results archive.

Archive is structured in subdirectories. Each directory is named after Firebird version and contains result collection dump (pickle) files with name:

<number_of_tests_executed>-<platform><cpuarch>-<fbarch>-<person-id>-<sequence>.trf
Parameters:archive_path (string) – Directory with archive.
delete(result)

Remove specified run results from archive.

Parameters:result – File path in archive or RunResult instance stored in archive.
get_archive_file_path(results)

Get File path for archived results. :param RunResult results: Run results. :returns: File path.

get_archive_filename(results)
list_filenames()

Get list of result files in archive. :returns: Sorted list of filenames (including FB version subdirectory name).

retrieve(version=None)

Get list of run results from archive.

Parameters:version (string) – Returns only results for specified Firebird version.
Returns:List of RunResult instances.
store(results)

Store run results into archive. :param RunResult results: Run results.

Result class

class fbtest.Result(kind, id, outcome='PASS', annotations={})

Result from test run or resource setup/cleanup.

Note

Also acts as dictionary of annotations.

Parameters:
  • kind (string) – Result kind.
  • id (string) – Result ID.
  • outcome (string) – Result outcome (default PASS).
  • annotations (dict) – Annotations.
annotate(annotations)

Set annotations.

Parameters:annotations (dict) – Annotations.
error(cause=None, annotations={})

Set ERROR outcome with specified cause and annotations.

Parameters:
  • cause (string) – Error cause.
  • annotations (dict) – Annotations.
fail(cause=None, annotations={})

Set FAIL outcome with specified cause and annotations.

Parameters:
  • cause (string) – Fail cause.
  • annotations (dict) – Annotations.
get(key, default=None)
get_cause()

Return cause.

get_run_time()
has_key(key)
items()
keys()
note_exception(exc_info=None, cause=None)

Set outcome to ERROR and annotate with exception info and traceback.

Parameters:
  • exc_info – Exception info or None to use current exception info.
  • cause (string) – Exception cause or None for default message.
set_cause(cause)

Set cause.

set_outcome(outcome, cause=None, annotations={})

Set Result outcome.

Parameters:
  • outcome (string) – Run outcome.
  • cause (string) – Short description of the outcome,
  • annotations (dict) – Disctionary of result anntotations.
CAUSE = 'cause'
END_TIME = 'end_time'
ERROR = 'ERROR'
EXCEPTION = 'exception'
FAIL = 'FAIL'
PASS = 'PASS'
RESOURCE = 'resource'
RESOURCE_CLEANUP = 'resource_cleanup'
RESOURCE_SETUP = 'resource_setup'
START_TIME = 'start_time'
TEST = 'test'
TRACEBACK = 'traceback'
UNTESTED = 'UNTESTED'
kinds = ['resource_setup', 'resource_cleanup', 'test']
outcomes = ['ERROR', 'FAIL', 'UNTESTED', 'PASS']

RunResults class

class fbtest.RunResults

Collection of test/resource Results.

Designed to hold Results from single QA run. Acts also as dictionary of Results with test/resource ID as key and Result object as value.

add(result)

Add result

clear()

Remove all results.

dump(filename)

Store (pickle) collection of Result instances to file.

get(key, default=None)
get_error_count()

Return number of ERROR outcomes

get_errors()

Return list of Results with ERROR outcome

get_fail_count()

Return number of FAIL outcomes

get_fails()

Return list of Results with FAIL outcome

get_outcomes()

Get list outcomes from stored Results

get_pass_count()

Return number of PASS outcomes

get_passes()

Return list of Results with PASS outcome

get_untested()

Return list of Results with UNTESTED outcome

get_untested_count()

Return number of UNTESTED outcomes

has_key(key)
items()
keys()
classmethod load(filename)

Load results from file.

Parameters:filename (string) – Filename with pickled Result.
print_report()
print_summary()

Print results summary to stdout.

save(filename)

Write text report to file.

Parameters:filename (string) – Filename.
save_xunit(filename)

Write xunit XML report to file.

Parameters:filename (string) – Filename.
values()

Runner class

class fbtest.Runner(repository)

QA Execution engine.

Parameters:repositoryRepository instance.
run(test_list=None, verbosity=1, results=None, no_summary=False, expectations=None)

Run tests.

Parameters:
  • test_list – List of Test objects to run. If not specified, runs all tests in repository.
  • verbosity (int) – Verbosity level (0, 1 or 2).
  • no_summary (bool) – Do not print run summary.
Returns:

RunResults object with results.

Warning

QA environment must be properly configured!

set_person(person)

Set QA person identification.

Parameters:person (string) – QA person name.

Sets person_name and person_id attributes: person_name = person in lowercase person_id = first two letters from uppercased person

If person is “Unknown”, the person_is is set to “XX”.

set_target(arch, host, bin_dir=None, password='masterkey')

Configures the QA environment to run on specified Firebird installation.

Parameters:
  • arch (string) – Firebird architecture (SS,CS, SC or EM).
  • host (string) – ‘LOCAL’ or Firebird host machine identification.
  • password (string) – Password for Firebird access (default ‘masterkey’).
tempdir

ScriptRunner class

class fbtest.ScriptRunner

Helper class to organize script running code. Instantiated as singleton instance script_runner.

analyze(filenames, output_dir, diffs_only=False)

Analyze test run results and produce HTML output.

Parameters:
  • filenames (list) – List of result file filenames.
  • output_dir (string) – Directory for HTML output files.
annotation_filter(annotations)

Filters out annotations we don’t want to compare.

cmd_archive_delete(options)

Called by run_archive() for command execution.

cmd_archive_list(options)

Called by run_archive() for command execution.

cmd_archive_retrieve(options)

Called by run_archive() for command execution.

cmd_archive_save(options)

Called by run_archive() for command execution.

cmd_update_repository(options)

Called by run_update() for command execution.

cmd_update_results(options)

Called by run_update() for command execution.

compare_results(r1, r2)

Compare two results for single test. Compares kind, outcome and annotations.

Parameters:
Returns:

True if both results are equal.

get_repository(options)

Connect to local or remote Test Repository.

Parameters:options – Command-line options.
Returns:Repository instance.
get_result_filenames(directory)

Return list of Result file filenames stored in directory.

Parameters:directory (string) – directory with result files.
Returns:List with filenames.
get_run_tag(platform, cpuarch, arch, sequence)

Return string that could be used as key to group tests by platform, cpu, FB architecture and run sequence.

get_svn_login(realm, username, may_save)

Get Subversion login credentials from user.

print_analysis(version, results, tests, test_details, test_order, output_dir, diffs_only)

Create HTML files with test run analysis.

Parameters:
  • version (string) – Firebird version.
  • result (RunResults) – Run results.
  • tests (dict) – Dictionary of all tests found in results; Test ID: list of results.
  • test_details (dict) – Dictionary with test run results: Run tag: Result.
  • test_order (list) – List that define test order in output.
  • output_dir (string) – Directory for HTML output files.
print_errors(result, cause=False, detail=False)

Print IDs of tests that ended with ERROR.

Parameters:
  • result (RunResults) – Run results.
  • cause (bool) – Print error cause.
  • detail (bool) – Print error details.
print_fails(result, cause=False, detail=False)

Print IDs of tests that ended with FAIL.

Parameters:
  • result (RunResults) – Run results.
  • cause (bool) – Print fail cause.
  • detail (bool) – Print fail details.
print_result_info(result, filename)

Print information from result file.

Parameters:
print_untested(result, cause=False)

Print IDs of tests that ended with UNTESTED.

Parameters:
run_analyze(options)

Called by run_analyze() for command execution.

run_server(options)

Called by run_server() for command execution.

run_tests(options)

Called by run_tests() for command execution.

run_view(options)

Called by run_view() for command execution.

svn_notify(event)

Notification callback from pysvn.

platform_table = {'macos': 'Mac', 'hp-ux': 'HPUX', 'solaris': 'Solaris', 'windows': 'Win', 'freebsd': 'BSD', 'linux': 'Linux'}

Script Functions

fbtest.run_tests()

CLI Script function for test execution.

This is a ‘main’ function called by fbt_run script. Runs all tests in test reporsitory or specified test/suite. Result from test run is stored in current directory:

results.trf : Pickled RunResults instance. Pickled RunResults instance is also stored in ‘archive’.

results.xml : XUNIT XML report (when -x or –xunit is specified).

usage: fbt_run [options] [suite_or_test_name]

-v, --verbose

Be more verbose

--verbosity=NUM

Set verbosity; –verbosity=2 is the same as -v

-q, --quiet

Be less verbose

-b <directory>, --bin-dir=<directory>

Location of Firebird binary tools (like gbak etc.).

-d <directory>, --db-dir=<directory>

Location for temporary databases used by tests.

-k <name_or_file>, --skip=<name_or_file>

Suite or test name or name of file with suite/test names to skip.

--archive

If specified, last run result file is copied to archive.

--rerun

If specified, runs only tests that don’t PASSed in last run.

--remote

If specified, connects to remote fbtest server and runs specified tests on it.

-u, --update

If specified, updates last run results with re-run results.

-w <password>, --password=<password>

SYSDBA password. (default ‘masterkey’)

-o <machine>, --host=<machine>

Firebird or fbtest host machine identification. (default ‘localhost’)

-p <person>, --person=<person>

QA person name. First two letters are used as person ID.

-a <arch>, --arch=<arch>

Firebird architecture (‘SS’, ‘CS’ or ‘SC’). (default ‘SS’ - SuperServer)

-s <number>, --sequence=<number>

Run sequence number. Important to disctinguish repeated runs af the same tests on single ‘target’. Used by fbt_analyze. (default ‘1’)

-x, --xunit

Provides test results also in the standard XUnit XML format.

-e, --expect

Test results file to be used as expeted outcomes

fbtest.run_server()

CLI Script function for test execution.

This is a ‘main’ function called by fbt_server script.

usage: fbt_server [options]

-b <directory>, --bin-dir=<directory>

Location of Firebird binary tools (like gbak etc.).

-d <directory>, --db-dir=<directory>

Location for temporary databases used by tests.

-w <password>, --password=<password>

SYSDBA password. (default ‘masterkey’)

-o <machine>, --host=<machine>

Firebird host machine identification. (default ‘localhost’)

-p <person>, --person=<person>

QA person name. First two letters are used as person ID.

-a <arch>, --arch=<arch>

Firebird architecture (‘SS’, ‘CS’ or ‘SC’). (default ‘SS’ - SuperServer)

--register

If specified, automatically registers RPyC slave server.

fbtest.run_analyze()

CLI Script function for test analysis.

This is a ‘main’ function called by fbt_analyze script. Reads result file or all result files in specified directory and writes HTML report in current or specified directory.

usage: fbt_analyze [options] <file_or_directory>

-o <directory>, --output=<directory>

Location where HTML report should be stored.

-d, --difs-only

Show only diffs on detail pages.

fbtest.run_update()

CLI Script function to update metadata in Result file(s).

This is a ‘main’ function called by fbt_update script. Reads result file or all result files in specified directory and updates metadata with specified value(s).

usage:

fbt_update {result,repository} [options]

result             Change result file metadata.
repository         Update Test Repository.

Options for fbt_update result:

name

Results file or directory with result files.

-a <arch>, --arch=<arch>

Firebird architecture (‘SS’, ‘CS’ or ‘SC’).

-p <person>, --person=<person>

QA person name. First two letters are used as person ID.

-s <number>, --sequence=<number>

Run sequence number. Important to disctinguish repeated runs of the same tests on single ‘target’. Used by fbt_analyze.

Options for fbt_update repository:

None.
fbtest.run_view()

CLI Script function to update metadata in Result file(s).

This is a ‘main’ function called by fbt_view script. Reads result file or all result files in specified directory and prints information about stored results.

usage: fbt_view <file_or_directory>

-x, --xunit

Store processed test results in the standard XUnit XML format.

-c, --cause

Print cause of fails and errors.

-d, --details

Print details for fails and errors.

fbtest.run_archive()

CLI Script function to maintain archive of Result file(s).

This is a ‘main’ function called by fbt_archive script.

usage:

fbt_update {list,save,retrieve,delete} [options]

list                List result(s) in archive.
save                Save result(s) to archive.
retrieve            Retrieve result(s) from archive.
delete              Delete result(s) from archive.
--remote

If specified, connects to remote fbtest server (and archive).

-o <machine>, --host=<machine>

Remote fbtest host machine identification. (default ‘localhost’)

Options for fbt_archive list:

None.

Options for fbt_archive save:

name

Results file.

Options for fbt_archive retrieve:

-v <version_number>, --version=<version_number>

Retrieve results only for specified Firebird version.

-c, --current

Retrieve results only for currently tested Firebird version.

-o <directory>, --output=<directory>

Location where result file(s) should be copied.

-p <person>, --person=<person>

Retrieve results only for specified QA person name. First two letters are used as person ID.

-a <arch>, --arch=<arch>

Retrieve results only for specified Firebird architecture (‘SS’, ‘CS’ or ‘SC’).

-s <number>, --sequence=<number>

Retrieve results only for specified run sequence number.

Options for fbt_archive delete:

-v <version_number>, --version=<version_number>

Delete results only for specified Firebird version.

-c, --current

Delete results only for currently tested Firebird version.

-p <person>, --person=<person>

Delete results only for specified QA person name. First two letters are used as person ID.

-a <arch>, --arch=<arch>

Delete results only for specified Firebird architecture (‘SS’, ‘CS’ or ‘SC’).

-s <number>, --sequence=<number>

Delete results only for specified run sequence number.