fbtest Reference¶
Globals¶
-
fbtest.
DB_NEW
= 'New'¶ str(object=’‘) -> string
Return a nice string representation of the object. If the argument is a string, the return value is the same object.
-
fbtest.
DB_EXISTING
= 'Existing'¶ str(object=’‘) -> string
Return a nice string representation of the object. If the argument is a string, the return value is the same object.
-
fbtest.
DB_RESTORE
= 'Restore'¶ str(object=’‘) -> string
Return a nice string representation of the object. If the argument is a string, the return value is the same object.
-
fbtest.
DB_ACCESS
= [None, 'New', 'Existing', 'Restore']¶ list() -> new empty list list(iterable) -> new list initialized from iterable’s items
-
fbtest.
CHARACTER_SETS
= [None, 'NONE', 'ASCII', 'BIG_5', 'CYRL', 'DOS437', 'DOS737', 'DOS775', 'DOS850', 'DOS852', 'DOS857', 'DOS858', 'DOS860', 'DOS861', 'DOS862', 'DOS863', 'DOS864', 'DOS865', 'DOS866', 'DOS869', 'EUCJ_0208', 'GBK', 'GB_2312', 'ISO8859_1', 'ISO8859_2', 'ISO8859_3', 'ISO8859_4', 'ISO8859_5', 'ISO8859_6', 'ISO8859_7', 'ISO8859_8', 'ISO8859_9', 'ISO8859_13', 'KOI8R', 'KOI8U', 'KSC_5601', 'NEXT', 'OCTETS', 'SJIS_0208', 'TIS620', 'UNICODE_FSS', 'UTF8', 'WIN1250', 'WIN1251', 'WIN1252', 'WIN1253', 'WIN1254', 'WIN1255', 'WIN1256', 'WIN1257', 'WIN1258', 'LATIN2']¶ list() -> new empty list list(iterable) -> new list initialized from iterable’s items
-
fbtest.
PAGE_SIZES
= [None, '1024', '2048', '4096', '8192', '16384']¶ list() -> new empty list list(iterable) -> new list initialized from iterable’s items
-
fbtest.
TYPE_ISQL
= 'ISQL'¶ str(object=’‘) -> string
Return a nice string representation of the object. If the argument is a string, the return value is the same object.
-
fbtest.
TYPE_PYTHON
= 'Python'¶ str(object=’‘) -> string
Return a nice string representation of the object. If the argument is a string, the return value is the same object.
-
fbtest.
TEST_TYPES
= ['ISQL', 'Python']¶ list() -> new empty list list(iterable) -> new list initialized from iterable’s items
-
fbtest.
PLATFORMS
= ['Windows', 'Linux', 'MacOS', 'FreeBSD', 'Solaris', 'HP-UX']¶ list() -> new empty list list(iterable) -> new list initialized from iterable’s items
-
fbtest.
UNKNOWN
= 'Unknown'¶ str(object=’‘) -> string
Return a nice string representation of the object. If the argument is a string, the return value is the same object.
-
fbtest.
script_runner
¶ ScriptRunner
instance.
Functions¶
-
fbtest.
xml_safe
(value)¶ Replaces invalid XML characters with ‘?’.
-
fbtest.
escape_cdata
(cdata)¶ Escape a string for an XML CDATA section.
-
fbtest.
trim_value
(value)¶ Return string with trailing whitespaces from each line removed.
-
fbtest.
quote
(value)¶ Return properly quoted string according its content.
-
fbtest.
compare_versions
(lver, rver)¶ Compare two version strings, returns: 0 for lver = rver -1 for lver < rver 1 for lver > rver
-
fbtest.
as_unicode
(value)¶ Make sure that string is unicode. Accepts Unicode, strings encoded in UTF-8 or None and returns Unicode string or None.
-
fbtest.
as_utf8
(value)¶ Make sure that string is encoded in UTF-8. Accepts Unicode, strings or None. If paremeter is unicode, it’s returned as string encoded in UTF-8. String or None parameter is returned as is.
-
fbtest.
runProgram
(args, environment, stdin=None, **kwargs)¶ Run external program using
Popen
from subprocess module and capture stdin and stderr.Parameters: - args (list) – List of arguments for
subprocess.Popen
. First item must be specification of program to run. - environment (dict) – Dictionary of environment variables. Ignored in this version.
- stdin (string) – String to be passed as stdin to external program.
- kwarg (dict) – Dictionary of additional keyword arguments for Popen.
Returns: Tuple of (returncode, stdout, stderr)
- args (list) – List of arguments for
Classes¶
TestVersion class¶
-
class
fbtest.
TestVersion
(id, platform, firebird_version, test_type, test_script, database='New', expected_stdout='', expected_stderr='', database_name=None, backup_file=None, user_name='SYSDBA', user_password='masterkey', database_character_set=None, connection_character_set=None, page_size=None, sql_dialect=3, init_script='', resources=None, substitutions=None, qmid=None)¶ Recipe for test execution against specific Firebird version and OS platform.
Parameters: - id (string) – Test ID (dot-separated name).
- platform (string) – List of platform names separated by colon.
- firebird_version (string) – First Firebird version this test version is designed for.
- test_type (string) – Test implemetation method: “ISQL” or “Python”.
- test_script (string) – Test code.
- database (string) – Database usage specification: None, “New”, “Existing” or “Restore”.
- expected_stdout (string) – Expected STDOUT content.
- expected_stderr (string) – Expected STDERR content.
- database_name (string) – Database file name.
- backup_file (string) – Backup file name.
- user_name (string) – User name.
- user_password (string) – User password.
- database_character_set (string) – Character set for database.
- connection_character_set (string) – Character set for connection.
- page_size (string) – Page size for database.
- sql_dialect (integer) – SQL dialect for database.
- init_script (string) – Test initialization script.
- resources (list) – List of
Resource
names. - substitutions (list) – List of substitution specifications.
- qmid (string) – Test ID from old system (QMTest).
-
as_expression
()¶ Return recipe data as string definition of Python dictionary. This string could be evaluated back to Python dictionary and passed to
TestVersion
constructor to recreate the test version instance. String is encoded in UTF-8 if necessary.Note
Only attributes that haven’t DEFAULT values are included.
-
get_platforms
()¶ Returns platforms supported by this test version as list of platform names.
-
run
(context, result)¶ Execute the recipe.
Important
Test run outcome is stored in
Result
instance.Parameters:
-
FIELDS
= ['id', 'qmid', 'firebird_version', 'platform', 'database', 'database_name', 'backup_file', 'user_name', 'user_password', 'database_character_set', 'connection_character_set', 'page_size', 'sql_dialect', 'init_script', 'test_type', 'test_script', 'expected_stdout', 'expected_stderr', 'resources', 'substitutions']¶
Test class¶
-
class
fbtest.
Test
(id, title='', description='', tracker_id='', min_versions=None, versions=None, qmid=None)¶ Test definition.
Can contain multiple
TestVersion
recipes for test execution against different Firebird versions on various OS platforms.Parameters: - id (string) – Test ID (dot-separated name).
- title (string) – Test title.
- description (string) – test description.
- tracker_id (string) – JIRA entry ID this tests relates to.
- min_versions (string) – List of minimal engine versions supported by test (separated by semicolon).
- versions (list) – List of disctionaries with data for creation of
TestVersion
instances. - qmid (string) – Test ID from old system (QMTest).
-
add_version
(version)¶ Add
TestVersion
instance to test.Parameters: version – TestVersion
instance.
-
as_expression
()¶ Return test data as string definition of Python dictionary. This string could be evaluated back to Python dictionary and passed to
Test
constructor to recreate the test instance. String is encoded in UTF-8 if necessary.Note
Only attributes that haven’t DEFAULT values are included.
-
get_name
()¶ Return test name.
Test name is last part of test ID.
-
get_version_for
(platform, version)¶ Return test recipe suitable for specified platform and Firebird version.
Parameters: Returns: TestVersion
or None.
-
FIELDS
= ['id', 'qmid', 'tracker_id', 'title', 'description', 'min_versions', 'versions']¶
Resource class¶
-
class
fbtest.
Resource
(id)¶ Base class for test resources.
Parameters: id (string) – Resource ID. -
as_expression
()¶ Return resource data as string definition of Python dictionary. This string could be evaluated back to Python dictionary and passed to
Resource
constructor to recreate the resource instance. String is encoded in UTF-8 if necessary.
-
cleanup
(result)¶ Resource finalization
Nothing in this class.
-
classmethod
create
(kind, **kwargs)¶ Class method to create right
Resource
instance from parameters.Parameters: kind (string) – Resource class specification.
-
fail_and_annotate_streams
(result, outcome, program_name, cause, stdout=None, stderr=None, annotate={})¶ Helper method to note fail of resource setup/cleanup in
Result
instance.Parameters:
-
setup
(context, result)¶ Resource initialization.
Nothing in this class.
-
UserResource class¶
-
class
fbtest.
UserResource
(id, user_name, user_password)¶ Test resource that provides Firebird USER.
Parameters: -
cleanup
(context, result)¶ Remove specified Firebird user via GSEC.
-
setup
(context, result)¶ Create Firebird user with specified name and password via GSEC.
-
FIELD
= ['id', 'user_name', 'user_password']¶
-
Suite class¶
-
class
fbtest.
Suite
(name, path=None, parent=None)¶ Container for tests and sub-suites to organize tests in hierarchical structure.
Parameters: -
clear
()¶ Delete all tests and sub-suites from instance.
-
get_id
()¶ Return suite ID.
Consists from suite names from root to this one separated by dot.
-
get_parent
()¶ Return parent suite or None
-
get_tests
()¶ Return all tests for this suite (including tests from sub-suites).
-
load
()¶ Load tests and sub-suites from
path
.
-
save_test
(test_id)¶ Save test on disk.
Parameters: test_id – Test ID.
-
Repository class¶
Archive class¶
-
class
fbtest.
Archive
(archive_path)¶ Run results archive.
Archive is structured in subdirectories. Each directory is named after Firebird version and contains result collection dump (pickle) files with name:
<number_of_tests_executed>-<platform><cpuarch>-<fbarch>-<person-id>-<sequence>.trf
Parameters: archive_path (string) – Directory with archive. -
delete
(result)¶ Remove specified run results from archive.
Parameters: result – File path in archive or RunResult
instance stored in archive.
-
get_archive_file_path
(results)¶ Get File path for archived results. :param
RunResult
results: Run results. :returns: File path.
-
get_archive_filename
(results)¶
-
list_filenames
()¶ Get list of result files in archive. :returns: Sorted list of filenames (including FB version subdirectory name).
-
retrieve
(version=None)¶ Get list of run results from archive.
Parameters: version (string) – Returns only results for specified Firebird version. Returns: List of RunResult
instances.
-
store
(results)¶ Store run results into archive. :param
RunResult
results: Run results.
-
Result class¶
-
class
fbtest.
Result
(kind, id, outcome='PASS', annotations={})¶ Result from test run or resource setup/cleanup.
Note
Also acts as dictionary of annotations.
Parameters: -
error
(cause=None, annotations={})¶ Set ERROR outcome with specified cause and annotations.
Parameters:
-
fail
(cause=None, annotations={})¶ Set FAIL outcome with specified cause and annotations.
Parameters:
-
get
(key, default=None)¶
-
get_cause
()¶ Return cause.
-
get_elapsed
()¶ Return test run time as (float) number of secods.
-
get_run_time
()¶ Return test run time as datetime.time.
-
has_key
(key)¶
-
items
()¶
-
keys
()¶
-
note_exception
(exc_info=None, cause=None)¶ Set outcome to ERROR and annotate with exception info and traceback.
Parameters: - exc_info – Exception info or None to use current exception info.
- cause (string) – Exception cause or None for default message.
-
set_cause
(cause)¶ Set cause.
-
set_outcome
(outcome, cause=None, annotations={})¶ Set Result outcome.
Parameters:
-
CAUSE
= 'cause'¶
-
END_TIME
= 'end_time'¶
-
ERROR
= 'ERROR'¶
-
EXCEPTION
= 'exception'¶
-
FAIL
= 'FAIL'¶
-
PASS
= 'PASS'¶
-
RESOURCE
= 'resource'¶
-
RESOURCE_CLEANUP
= 'resource_cleanup'¶
-
RESOURCE_SETUP
= 'resource_setup'¶
-
SKIPPED
= 'SKIPPED'¶
-
START_TIME
= 'start_time'¶
-
TEST
= 'test'¶
-
TRACEBACK
= 'traceback'¶
-
UNTESTED
= 'UNTESTED'¶
-
kinds
= ['resource_setup', 'resource_cleanup', 'test']¶
-
outcomes
= ['ERROR', 'FAIL', 'UNTESTED', 'PASS', 'SKIPPED']¶
-
RunResults class¶
-
class
fbtest.
RunResults
¶ Collection of test/resource Results.
Designed to hold Results from single QA run. Acts also as dictionary of Results with test/resource ID as key and
Result
object as value.-
add
(result)¶ Add result
-
clear
()¶ Remove all results.
-
get
(key, default=None)¶
-
get_error_count
()¶ Return number of ERROR outcomes
-
get_errors
()¶ Return list of Results with ERROR outcome
-
get_fail_count
()¶ Return number of FAIL outcomes
-
get_fails
()¶ Return list of Results with FAIL outcome
-
get_outcomes
()¶ Get list outcomes from stored Results
-
get_pass_count
()¶ Return number of PASS outcomes
-
get_passes
()¶ Return list of Results with PASS outcome
-
get_skipped
()¶ Return list of Results with SKIPPED outcome
-
get_skipped_count
()¶ Return number of SKIPPED outcomes
-
get_untested
()¶ Return list of Results with UNTESTED outcome
-
get_untested_count
()¶ Return number of UNTESTED outcomes
-
has_key
(key)¶
-
items
()¶
-
keys
()¶
-
classmethod
load
(filename)¶ Load results from file.
Parameters: filename (string) – Filename with pickled Result
.
-
print_report
()¶
-
print_summary
()¶ Print results summary to stdout.
-
values
()¶
-
Runner class¶
-
class
fbtest.
Runner
(repository)¶ QA Execution engine.
Parameters: repository – Repository
instance.-
run
(test_list=None, skip_list=None, verbosity=1, results=None, no_summary=False, expectations=None)¶ Run tests.
Parameters: Returns: RunResults
object with results.Warning
QA environment must be properly configured!
-
set_person
(person)¶ Set QA person identification.
Parameters: person (string) – QA person name. Sets person_name and person_id attributes: person_name = person in lowercase person_id = first two letters from uppercased person
If person is “Unknown”, the person_is is set to “XX”.
-
set_target
(arch, host, bin_dir=None, password='masterkey', client_lib=None)¶ Configures the QA environment to run on specified Firebird installation.
Parameters:
-
tempdir
¶
-
ScriptRunner class¶
-
class
fbtest.
ScriptRunner
¶ Helper class to organize script running code. Instantiated as singleton instance script_runner.
-
analyze
(filenames, output_dir, diffs_only=False)¶ Analyze test run results and produce HTML output.
Parameters:
-
annotation_filter
(annotations)¶ Filters out annotations we don’t want to compare.
-
cmd_archive_delete
(options)¶ Called by
run_archive()
for command execution.
-
cmd_archive_list
(options)¶ Called by
run_archive()
for command execution.
-
cmd_archive_retrieve
(options)¶ Called by
run_archive()
for command execution.
-
cmd_archive_save
(options)¶ Called by
run_archive()
for command execution.
-
cmd_update_repository
(options)¶ Called by
run_update()
for command execution.
-
cmd_update_results
(options)¶ Called by
run_update()
for command execution.
-
compare_results
(r1, r2)¶ Compare two results for single test. Compares
kind
,outcome
andannotations
.Parameters: Returns: True if both results are equal.
-
get_repository
(options)¶ Connect to local or remote Test Repository.
Parameters: options – Command-line options. Returns: Repository
instance.
-
get_result_filenames
(directory)¶ Return list of Result file filenames stored in directory.
Parameters: directory (string) – directory with result files. Returns: List with filenames.
-
get_run_tag
(platform, cpuarch, arch, sequence)¶ Return string that could be used as key to group tests by platform, cpu, FB architecture and run sequence.
-
get_svn_login
(realm, username, may_save)¶ Get Subversion login credentials from user.
-
print_analysis
(version, results, tests, test_details, test_order, output_dir, diffs_only)¶ Create HTML files with test run analysis.
Parameters: - version (string) – Firebird version.
- result (
RunResults
) – Run results. - tests (dict) – Dictionary of all tests found in results; Test ID: list of results.
- test_details (dict) – Dictionary with test run results: Run tag: Result.
- test_order (list) – List that define test order in output.
- output_dir (string) – Directory for HTML output files.
-
print_errors
(result, cause=False, detail=False)¶ Print IDs of tests that ended with ERROR.
Parameters: - result (
RunResults
) – Run results. - cause (bool) – Print error cause.
- detail (bool) – Print error details.
- result (
-
print_fails
(result, cause=False, detail=False)¶ Print IDs of tests that ended with FAIL.
Parameters: - result (
RunResults
) – Run results. - cause (bool) – Print fail cause.
- detail (bool) – Print fail details.
- result (
-
print_result_info
(result, filename)¶ Print information from result file.
Parameters: - result (
RunResults
) – Run results. - filename (string) – Results filename.
- result (
-
print_skipped
(result, cause=False)¶ Print IDs of tests that ended with SKIPPED.
Parameters: - result (
RunResults
) – Run results. - cause (bool) – Print cause.
- result (
-
print_untested
(result, cause=False)¶ Print IDs of tests that ended with UNTESTED.
Parameters: - result (
RunResults
) – Run results. - cause (bool) – Print cause.
- result (
-
run_analyze
(options)¶ Called by
run_analyze()
for command execution.
-
run_server
(options)¶ Called by
run_server()
for command execution.
-
run_tests
(options)¶ Called by
run_tests()
for command execution.
-
run_view
(options)¶ Called by
run_view()
for command execution.
-
svn_notify
(event)¶ Notification callback from pysvn.
-
platform_table
= {'macos': 'Mac', 'hp-ux': 'HPUX', 'solaris': 'Solaris', 'windows': 'Win', 'freebsd': 'BSD', 'linux': 'Linux'}¶
-
Script Functions¶
-
fbtest.
run_tests
()¶ CLI Script function for test execution.
This is a ‘main’ function called by fbt_run script. Runs all tests in test reporsitory or specified test/suite. Result from test run is stored in current directory:
results.trf
: PickledRunResults
instance. PickledRunResults
instance is also stored in ‘archive’.results.xml
: XUNIT XML report (when -x or –xunit is specified).usage: fbt_run [options] [suite_or_test_name]
-
-v
,
--verbose
¶
Be more verbose
-
--verbosity
=NUM
¶ Set verbosity; –verbosity=2 is the same as -v
-
-q
,
--quiet
¶
Be less verbose
-
-b
<directory>
,
--bin-dir
=<directory>
¶ Location of Firebird binary tools (like gbak etc.).
-
-d
<directory>
,
--db-dir
=<directory>
¶ Location for temporary databases used by tests.
-
-k
<name_or_file>
,
--skip
=<name_or_file>
¶ Suite or test name or name of file with suite/test names to skip.
-
--archive
¶
If specified, last run result file is copied to archive.
-
--rerun
¶
If specified, runs only tests that don’t PASSed in last run.
-
--remote
¶
If specified, connects to remote fbtest server and runs specified tests on it.
-
-u
,
--update
¶
If specified, updates last run results with re-run results.
-
-w
<password>
,
--password
=<password>
¶ SYSDBA password. (default ‘masterkey’)
-
-o
<machine>
,
--host
=<machine>
¶ Firebird or fbtest host machine identification. (default ‘localhost’)
-
-p
<person>
,
--person
=<person>
¶ QA person name. First two letters are used as person ID.
-
-a
<arch>
,
--arch
=<arch>
¶ Firebird architecture (‘SS’, ‘CS’ or ‘SC’). (default ‘SS’ - SuperServer)
-
-s
<number>
,
--sequence
=<number>
¶ Run sequence number. Important to disctinguish repeated runs af the same tests on single ‘target’. Used by fbt_analyze. (default ‘1’)
-
-x
,
--xunit
¶
Provides test results also in the standard XUnit XML format.
-
-e
,
--expect
¶
Test results file to be used as expeted outcomes
-
-
fbtest.
run_server
()¶ CLI Script function for test execution.
This is a ‘main’ function called by fbt_server script.
usage: fbt_server [options]
-
-b
<directory>
,
--bin-dir
=<directory>
¶ Location of Firebird binary tools (like gbak etc.).
-
-d
<directory>
,
--db-dir
=<directory>
¶ Location for temporary databases used by tests.
-
-w
<password>
,
--password
=<password>
¶ SYSDBA password. (default ‘masterkey’)
-
-o
<machine>
,
--host
=<machine>
¶ Firebird host machine identification. (default ‘localhost’)
-
-p
<person>
,
--person
=<person>
¶ QA person name. First two letters are used as person ID.
-
-a
<arch>
,
--arch
=<arch>
¶ Firebird architecture (‘SS’, ‘CS’ or ‘SC’). (default ‘SS’ - SuperServer)
-
--register
¶
If specified, automatically registers RPyC slave server.
-
-
fbtest.
run_analyze
()¶ CLI Script function for test analysis.
This is a ‘main’ function called by fbt_analyze script. Reads result file or all result files in specified directory and writes HTML report in current or specified directory.
usage: fbt_analyze [options] <file_or_directory>
-
-o
<directory>
,
--output
=<directory>
¶ Location where HTML report should be stored.
-
-d
,
--difs-only
¶
Show only diffs on detail pages.
-
-
fbtest.
run_update
()¶ CLI Script function to update metadata in Result file(s).
This is a ‘main’ function called by fbt_update script. Reads result file or all result files in specified directory and updates metadata with specified value(s).
usage:
fbt_update {result,repository} [options] result Change result file metadata. repository Update Test Repository.
Options for fbt_update result:
-
name
¶
Results file or directory with result files.
-
-a
<arch>
,
--arch
=<arch>
¶ Firebird architecture (‘SS’, ‘CS’ or ‘SC’).
-
-p
<person>
,
--person
=<person>
¶ QA person name. First two letters are used as person ID.
-
-s
<number>
,
--sequence
=<number>
¶ Run sequence number. Important to disctinguish repeated runs of the same tests on single ‘target’. Used by fbt_analyze.
Options for fbt_update repository:
None.-
-
fbtest.
run_view
()¶ CLI Script function to update metadata in Result file(s).
This is a ‘main’ function called by fbt_view script. Reads result file or all result files in specified directory and prints information about stored results.
usage: fbt_view <file_or_directory>
-
-x
,
--xunit
¶
Store processed test results in the standard XUnit XML format.
-
-c
,
--cause
¶
Print cause of fails and errors.
-
-d
,
--details
¶
Print details for fails and errors.
-
-
fbtest.
run_archive
()¶ CLI Script function to maintain archive of Result file(s).
This is a ‘main’ function called by fbt_archive script.
usage:
fbt_update {list,save,retrieve,delete} [options] list List result(s) in archive. save Save result(s) to archive. retrieve Retrieve result(s) from archive. delete Delete result(s) from archive.
-
--remote
¶
If specified, connects to remote fbtest server (and archive).
-
-o
<machine>
,
--host
=<machine>
¶ Remote fbtest host machine identification. (default ‘localhost’)
Options for fbt_archive list:
None.Options for fbt_archive save:
-
name
¶
Results file.
Options for fbt_archive retrieve:
-
-v
<version_number>
,
--version
=<version_number>
¶ Retrieve results only for specified Firebird version.
-
-c
,
--current
¶
Retrieve results only for currently tested Firebird version.
-
-o
<directory>
,
--output
=<directory>
¶ Location where result file(s) should be copied.
-
-p
<person>
,
--person
=<person>
¶ Retrieve results only for specified QA person name. First two letters are used as person ID.
-
-a
<arch>
,
--arch
=<arch>
¶ Retrieve results only for specified Firebird architecture (‘SS’, ‘CS’ or ‘SC’).
-
-s
<number>
,
--sequence
=<number>
¶ Retrieve results only for specified run sequence number.
Options for fbt_archive delete:
-
-v
<version_number>
,
--version
=<version_number>
¶ Delete results only for specified Firebird version.
-
-c
,
--current
¶
Delete results only for currently tested Firebird version.
-
-p
<person>
,
--person
=<person>
¶ Delete results only for specified QA person name. First two letters are used as person ID.
-
-a
<arch>
,
--arch
=<arch>
¶ Delete results only for specified Firebird architecture (‘SS’, ‘CS’ or ‘SC’).
-
-s
<number>
,
--sequence
=<number>
¶ Delete results only for specified run sequence number.
-