Tool: mp_c2mpt

Overview

C to MicroprobeTest (mpt) tool provides a command-line interface (CLI) to generate MPT files (Microprobe test (mpt) format) from C source code by following certain guidelines (C test format). Then, from the MPT format, the user can use existing tools (Command line tools) to produce test cases for different environments and formats .

Note

This tool relies on third-party tools (compilers and object code inspectors). The microbenchmark generation policy implemented in this tool reproduces exactly the code provided by the compiler and the object code inspector. The framework resolves any symbolic references (references to data, or branch targets) and adds necessary code to initialize the required registers so that it does not break the target Application Binary Interface.

Basic usage

> mp_objdump2mpt -T TARGET -i OBJDUMP_FILE -O MPT_OUTPUT_FILE

where:

Flag/Argument Description
-T TARGET, --target TARGET Target definition string. Check: Command line target definition scheme
-i INPUT_C_FILE, --input-c-file INPUT_C_FILE C test file following the format in C test format
-O MPT_OUTPUT_FILE, --mpt-output-file MPT_OUTPUT_FILE Output file name

Tool work-flow

The figure below summarizes the high-level work-flow of the C to MPT command line tool:

C to MPT workflow

Fig. 1 C to MPT high level work-flow

From a given C file that complies with the C test format, the process is two-fold (see top and bottom flows in the figure).

  1. The C file is compiled using the local host compiler and executed in the local host. This process is necessary to obtain the variable values at run-time just before the test main function c2mpt_function (check C test format) . Also, the host execution with an exit code of 0 confirms the correctness of the code that is provided.
  2. The C file is compiled using the target host compiler (which can be the same as the local) and then the generated instructions are obtained. The instruction sequence is extended to add the necessary initialization to ensure a correct execution of the code. We call this process as environment construction in the figure. For instance, the stack is defined and the stack pointer is initialized.

Finally, all the information (instructions, variables and their contents) is dumped into a MPT file which can then be used to generate test cases in different formats.

The user can control some features of this work-flow using the following flags:

Flag/Argument Description
–no-data-initialization Do not run the compiled code locally to get the contents of registered variables before executing the c2mpt_function. Only statically specified variable contents will be dumped in the MPT generated
--save-temps Store the generated intermediate files permanently; place them in the input source file directory and name them based on the source file

Note

It is up to the user to control the compiler behavior in order to generate a valid test case. For instance, the compiler might consider a function to be useless and therefore, remove it from the generated code.

C test format

The input file has to be compliant with the C test case format in order to ensure the generation of a correct test case. Check the section C test format for details. For your convenience, the tool provides a flag to dump an empty C file template, which you can use to start the specification of the C-based test:

Flag/Argument Description
--dump-c2mpt-template Dump a template C file, which can be used afterwards as an input file

Third-party tool customization

As explained in the previous section, this tool relies on third-party tools to generate the final MPT. Three external tools are required: a local host compiler, a target host compiler and the objdump utility. By default, cc and objdump commands are assumed, but in case that the user needs to customize these commands, a set of customization flags are provided:

Flag/Argument Description
--host-c-compiler HOST_C_COMPILER Local C compiler (Default:’cc’)
--host-c-compiler-flags HOST_C_COMPILER_FLAGS Local C compiler flags (Default:’-Wall -Werror -m64 -O3 -Wpedantic -pedantic -pedantic-errors -std=c99’)
--target-c-compiler TARGET_C_COMPILER Target C compiler (Default:’cc’)
--target-objdump TARGET_OBJDUMP Target objdump utility (Default:’objdump’)
--target-c-compiler-flags TARGET_C_COMPILER_FLAGS Target C compiler flags (Default:’-Wall -Werror -m64 -O3 -Wpedantic -pedantic -pedantic-errors -std=c99’)

Note

Only GNU GCC tool-chains (gcc, objdump) are currently supported.

Customizing Application Binary Interface parameters

The ABI (Application Binary Interface) for a given environment (defined in Linux Foundation - Referenced Specifications ) specifies the semantics of registers and the calling conventions. For instance, usually a register is reserved to point to the stack. So, the code being extracted from the objdump output requires that certain registers contain appropriate values. This tool supports the automatic definition of the stack, the automatic initialization of the stack pointer and the definition of the start symbol.

In case the default values are not appropriate, the user can use the following flags to change the different options:

Flag/Argument Description
--stack-size STACK_SIZE Stack size in bytes (default: 4096)
--stack-name STACK_NAME Stack name (Default: microprobe_stack)
--stack-address STACK_ADDRESS Stack address (Default: allocated in the data area)
--end-branch-to-itself End the code with a branch to itself instruction

Note

The necessary instructions required for initializing the stack pointer will be added at the beginning of the code (on lower addresses) and the start code address will be modified accordingly.

Current limitations

  • Code constraints: Do not call library functions in any of the functions to be included in the final MPT. Notice that this restriction does not affect the initialization functions that are not included in the MPT.
  • Variable initialization constraints: As explained in previous section, the program is executed locally to obtain the variable values. As a consequence, in the case that the variable type is not a basic type or a fixed width type as specified here and if the local host and the target host endianess and/or struct padding differ, the initial values of the variables will not be correct. A warning is triggered to alert the user about this issue.
  • Code/Data segment addresses: One should be careful when specifying the addresses for the code and data regions. Sometimes, the user may face some limitations for some ranges that are not valid in some environments. For instance, if you get a segmentation fault when running the test locally and you are sure that the code is correct and gdb is telling you that the fault is generated during the program loading phase, then it is very likely that the code/data segment addresses are conflicting with others or they are out of the valid range for that environment.

Full usage

mp_c2mpt.py: INFO: Processing input arguments...
usage: mp_c2mpt.py [-h] [-P SEARCH_PATH [SEARCH_PATH ...]] [-V] [-v] [-d]
                   [-c CONFIG_FILE [CONFIG_FILE ...]] [-C FORCE_CONFIG_FILE]
                   [--dump-configuration-file OUTPUT_CONFIG_FILE]
                   [--dump-full-configuration-file OUTPUT_CONFIG_FILE]
                   [-A ARCHITECTURE_PATHS] [-M MICROARCHITECTURE_PATHS]
                   [-E ENVIRONMENT_PATHS] -T TARGET [--list-architectures]
                   [--list-microarchitectures] [--list-environments]
                   [--traceback] [--profile PROFILE_OUTPUT]
                   [--host-c-compiler HOST_C_COMPILER]
                   [--host-cxx-compiler HOST_CXX_COMPILER]
                   [--target-c-compiler TARGET_C_COMPILER]
                   [--target-cxx-compiler TARGET_CXX_COMPILER]
                   [--target-objdump TARGET_OBJDUMP]
                   [--host-c-compiler-flags HOST_C_COMPILER_FLAGS]
                   [--host-cxx-compiler-flags HOST_CXX_COMPILER_FLAGS]
                   [--target-c-compiler-flags TARGET_C_COMPILER_FLAGS]
                   [--target-cxx-compiler-flags TARGET_CXX_COMPILER_FLAGS]
                   [-i INPUT_C_FILE] -O OUTPUT_MPT_FILE [-S]
                   [-X DEFAULT_CODE_ADDRESS] [-D DEFAULT_DATA_ADDRESS]
                   [--stack-size STACK_SIZE]
                   [--host-displacement HOST_DISPLACEMENT]
                   [--fix-displacement] [--stack-name STACK_NAME]
                   [--stack-address STACK_ADDRESS] [--no-data-initialization]
                   [--save-temps] [--dump-c2mpt-template]
                   [--end-branch-to-itself]

Microprobe C to MPT tool

optional arguments:
  -h, --help            show this help message and exit
  -P SEARCH_PATH [SEARCH_PATH ...], --default_paths SEARCH_PATH [SEARCH_PATH ...]
                        Default search paths for microprobe target definitions
  -V, --version         Show Microprobe version and exit
  -v, --verbosity       Verbosity level (Values: [0,1,2,3,4]). Each time this
                        argument is specified the verbosity level is
                        increased. By default, no logging messages are shown.
                        These are the four levels available:
                        
                          -v (1): critical messages
                          -v -v (2): critical and error messages
                          -v -v -v (3): critical, error and warning messages
                          -v -v -v -v (4): critical, error, warning and info messages
                        
                        Specifying more than four verbosity flags, will
                        default to the maximum of four. If you need extra
                        information, enable the debug mode (--debug or -d
                        flags).
  -d, --debug           Enable debug mode in Microprobe framework. Lots of
                        output messages will be generated

Configuration arguments:

  Command arguments related to configuration file handling

  -c CONFIG_FILE [CONFIG_FILE ...], --configuration CONFIG_FILE [CONFIG_FILE ...]
                        Configuration file. The configuration files will be
                        readed in order of appearance. Values are reset by the
                        last configuration file in case of non-list values.
                        List values will be appended (not reset)
  -C FORCE_CONFIG_FILE, --force-configuration FORCE_CONFIG_FILE
                        Force configuration file. Use this configuration file
                        as the default start configuration. This disables any
                        system-wide, or user-provided configuration.
  --dump-configuration-file OUTPUT_CONFIG_FILE
                        Dump a configuration file with the actual
                        configuration used
  --dump-full-configuration-file OUTPUT_CONFIG_FILE
                        Dump a configuration file with the actual
                        configuration used plus all the configuration options
                        not set

Target path arguments:

  Command arguments related to target paths

  -A ARCHITECTURE_PATHS, --architecture-paths ARCHITECTURE_PATHS
                        Search path for architecture definitions. Microprobe
                        will search in these paths for architecture
                        definitions
  -M MICROARCHITECTURE_PATHS, --microarchitecture-paths MICROARCHITECTURE_PATHS
                        Search path for microarchitecture definitions.
                        Microprobe will search in these paths for
                        microarchitecture definitions
  -E ENVIRONMENT_PATHS, --environment-paths ENVIRONMENT_PATHS
                        Search path for environment definitions. Microprobe
                        will search in these paths for environment definitions

Target arguments:

  Command arguments related to target specification and queries

  -T TARGET, --target TARGET
                        Target tuple. Microprobe follows a GCC-like target
                        definition scheme, where a target is defined by a
                        tuple as following:
                        
                          <arch-name>-<uarch-name>-<env-name>
                        
                        where:
                        
                          <arch-name>: is the name of the architecture
                          <uarch-name>: is the name of the microarchitecture
                          <env-name>: is the name of the environment
                        
                        One can use --list-* options to get the list of
                        definitions available in the default search paths or
                        the paths specified by the different --*-paths options
  --list-architectures  Generate a list of architectures available in the
                        defined search paths and exit
  --list-microarchitectures
                        Generate a list of microarchitectures available in the
                        defined search paths and exit
  --list-environments   Generate a list of environments available in the
                        defined search paths and exit

Debug arguments:

  Command arguments related to debugging facilities

  --traceback           show a traceback and starts a python debugger (pdb)
                        when an error occurs. 'pdb' is an interactive python
                        shell that facilitates the debugging of errors
  --profile PROFILE_OUTPUT
                        dump profiling information into given file (see
                        'pstats' module)

Compilation arguments:

  Command arguments related to compilation options

  --host-c-compiler HOST_C_COMPILER
                        Local C compiler (Default:'cc')
  --host-cxx-compiler HOST_CXX_COMPILER
                        Local C++ compiler (Default:'c++')
  --target-c-compiler TARGET_C_COMPILER
                        Target C compiler (Default:'cc')
  --target-cxx-compiler TARGET_CXX_COMPILER
                        Target C++ compiler (Default:'c++')
  --target-objdump TARGET_OBJDUMP
                        Target objdump utility (Default:'objdump')
  --host-c-compiler-flags HOST_C_COMPILER_FLAGS
                        Local C compiler flags (Default:'-Wall -Werror -m64
                        -O3 -pedantic -pedantic-errors -std=c99')
  --host-cxx-compiler-flags HOST_CXX_COMPILER_FLAGS
                        Local C++ compiler flags (Default:'-Wall -Werror -m64
                        -O3 -pedantic -pedantic-errors -std=c99')
  --target-c-compiler-flags TARGET_C_COMPILER_FLAGS
                        Target C compiler flags (Default:'-Wall -Werror -m64
                        -O3 -pedantic -pedantic-errors -std=c99')
  --target-cxx-compiler-flags TARGET_CXX_COMPILER_FLAGS
                        Target C++ compiler flags (Default:'-Wall -Werror -m64
                        -O3 -pedantic -pedantic-errors -std=c99')

C to MPT arguments:

  Command arguments related to C to MPT tool

  -i INPUT_C_FILE, --input-c-file INPUT_C_FILE
                        C file to process
  -O OUTPUT_MPT_FILE, --output-mpt-file OUTPUT_MPT_FILE
                        Output file name
  -S, --strict          Be strict when parsing objdump input, if not set,
                        silently skip unparsed elements
  -X DEFAULT_CODE_ADDRESS, --default-code-address DEFAULT_CODE_ADDRESS
                        Default code address (default: 0x10030000)
  -D DEFAULT_DATA_ADDRESS, --default-data-address DEFAULT_DATA_ADDRESS
                        Default data address (default: 0x10040000)
  --stack-size STACK_SIZE
                        Stack size in bytes (Default: 4096)
  --host-displacement HOST_DISPLACEMENT
                        Displacement between static objdump code and loaded
                        image on the host. Default computed automatically
  --fix-displacement    If data contains addresses (such as pointers) to code
                        or data regions, the tool will try to fix them adding
                        the necessary displacement
  --stack-name STACK_NAME
                        Stack name (Default: microprobe_stack)
  --stack-address STACK_ADDRESS
                        Stack address (Default: allocated in the data area)
  --no-data-initialization
                        Do not run the compiled code locally to get the
                        contents of registered variables before executing the
                        *c2mpt_function*. Only statically specified variable
                        contents will be dumped in the MPT generated
  --save-temps          Store the generated intermediate files permanently;
                        place them in the input source file directory and name
                        them based on the source file
  --dump-c2mpt-template
                        Dump a template C file, which can be used afterwards
                        as an input file
  --end-branch-to-itself
                        A branch to itself instruction will be added at the
                        end of the test

Environment variables:

  MICROPROBETEMPLATES    Default path for microprobe templates
  MICROPROBEDEBUG        If set, enable debug
  MICROPROBEDEBUGPASSES  If set, enable debug during passes
  MICROPROBEASMHEXFMT    Assembly hexadecimal format. Options:
                         'all' -> All immediates in hex format
                         'address' -> Address immediates in hex format (default)
                         'none' -> All immediate in integer format
  CC                     Default C compiler
  CPP                    Default C++ compiler
  TARGET_OBJDUMP         Default target objdump utility
  CFLAGS                 Default C compiler flags
  CXFLAGS                Default C++ compiler flags

Example outputs

Example 1:

Command:

> mp_c2mpt.py -T z14-z14-z64_linux_gcc -i input.c -O output.mpt

Input file input.c:

Output file output.mpt:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
; Microprobe Test Definition File
[MPT]
mpt_version = 0.5 ;  Format version of this MPT file.

[REGISTERS] ; Section to specify the initial register values

; Format: register = value. E.g.:

; Set GR0, GR1 and GR2 register to 0, 1, 2 values respectively
;GR0 = 0x0


[DATA] ; Section to specify the variables

; Data section default address. Variables will be placed from this address
; if their address is not specified

default_address = 0x0000000010040000

; Variable Declaration
; Format: var_name = [ "type", nelems, address, alignment, init_values ]
; where:
;   - "type": is a string specifying the type of elements in the variable
;   - nelems: is the number of elements in the variable
;   - address : is the address of the variable, if set the address will be
;               fixed, otherwise, it will be computer by microprobe
;   - alignment : alignment requirements of the variable. It should not
;                 conflict with address if specified. It can be set to None
;   - init_values : if it is a single value, all the elements will be
;                   initialized to that value, if it is an array, elements
;                   will be initialized to the values specified in a round-
;                   robin fashion. Two special keywords can be specified:
;                   RNDFP and RNDINT to initialize the elements to random FP
;                   and random INT values
;
; Note that variable names ARE NOT case sensitive. I.e. VAR = Var = var

count = ["int64_t", 00000001, 0x0000000010040000, 0x0008, 3405695742]
linkedlist = ["uint8_t", 00000080, 0x0000000010040010, 0x0000, [8, 7, 6, 5, 4, 3, 2, 1, 32, 208, 58, 185, 36, 86, 0, 0, 8, 7, 6, 5, 4, 3, 2, 1, 48, 208, 58, 185, 36, 86, 0, 0, 8, 7, 6, 5, 4, 3, 2, 1, 64, 208, 58, 185, 36, 86, 0, 0, 8, 7, 6, 5, 4, 3, 2, 1, 80, 208, 58, 185, 36, 86, 0, 0, 8, 7, 6, 5, 4, 3, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0]]
microprobe_stack = ["uint8_t", 00004096, None, 0x0010, None]

[CODE] ; Section to specify the code

; Code section default address. Code will be placed from this address
; if the instruction address is not specified

default_address = 0x000000001002fff0

; The code specified after 'instructions' entry (below) is the code that will be
; processed by microprobe. The instruction format is similar to GNU assembler
; format, it also allows the specification of labels (NOT case sensitive) and
; references to the declared variables. It is also possible to specify instruction
; addresses and to do code expansion by referencing other user
; defined entries. Check the example below to see examples of these features.
;
; *****************************************************************************
; ******  Although Microprobe performs some sanity checks, it is the   ********
; ******  responsibility of the user to define correct code.           ********
; ******                                                               ********
; *****************************************************************************

instructions =
  <ELF_ABI_START>:
    LUI x2, %hi(microprobe_stack)                     
    ADDI x2, x2, %lo(microprobe_stack)                
    JAL x1, c2mpt_function                            
    JAL x1, ELF_ABI_EXIT                              
  0x0000000010030000 <C2MPT_FUNCTION>:
    LUI x12, 65600                                    
    ADDI x12, x12, 0                                  
    LD x15, 8(x12)                                    
    LD x14, 80(x12)                                   
    BEQ x15, x0, C2MPT_FUNCTION+0X24                  
    LD x13, 0(x15)                                    
    LD x15, 8(x15)                                    
    ADD x14, x14, x13                                 
    BNE x15, x0, C2MPT_FUNCTION+0X14                  
    SLLI x14, x14, 1                                  
    SD x14, 80(x12)                                   
    JALR x0, x1, 0                                    
  0x0000000010030030 <MY_SUBROUTINE>:
    SLLI x10, x10, 1                                  
    LUI x15, 65600                                    
    SD x10, 80(x15)                                   
    JALR x0, x1, 0                                    
  <ELF_ABI_EXIT>:
    ADDI x0, x0, 0