jube2 package

jube2 package

Subpackages

• jube2.result_types package
  • Submodules
  • jube2.result_types.keyvaluesresult module
    • KeyValuesResult
      • KeyValuesResult.DataKey
        • KeyValuesResult.DataKey.etree_repr()
        • KeyValuesResult.DataKey.format
        • KeyValuesResult.DataKey.name
        • KeyValuesResult.DataKey.resulting_name
        • KeyValuesResult.DataKey.title
        • KeyValuesResult.DataKey.unit
      • KeyValuesResult.KeyValuesData
        • KeyValuesResult.KeyValuesData.add_id_information()
        • KeyValuesResult.KeyValuesData.add_key_value_data()
        • KeyValuesResult.KeyValuesData.add_result_data()
        • KeyValuesResult.KeyValuesData.benchmark_ids
        • KeyValuesResult.KeyValuesData.create_result()
        • KeyValuesResult.KeyValuesData.data
        • KeyValuesResult.KeyValuesData.data_dict
        • KeyValuesResult.KeyValuesData.keys
      • KeyValuesResult.add_key()
      • KeyValuesResult.create_result_data()
  • jube2.result_types.syslog module
    • SysloggedResult
      • SysloggedResult.SyslogData
        • SysloggedResult.SyslogData.create_result()
      • SysloggedResult.create_result_data()
      • SysloggedResult.etree_repr()
  • jube2.result_types.table module
    • Table
      • Table.Column
        • Table.Column.colw
        • Table.Column.etree_repr()
      • Table.TableData
        • Table.TableData.create_result()
        • Table.TableData.separator
        • Table.TableData.style
      • Table.add_column()
      • Table.add_key()
      • Table.create_result_data()
      • Table.etree_repr()
• jube2.util package
  • Submodules
  • jube2.util.output module
    • element_tree_tostring()
    • format_value()
    • print_loading_bar()
    • text_boxed()
    • text_line()
    • text_table()
  • jube2.util.util module
    • CompType
      • CompType.value
    • Queue
      • Queue.empty()
      • Queue.get_nowait()
      • Queue.put()
      • Queue.put_first()
    • WorkStat
      • WorkStat.empty()
      • WorkStat.get()
      • WorkStat.push_back()
      • WorkStat.put()
      • WorkStat.update_queues()
    • check_and_get_group_id()
    • consistency_check()
    • convert_type()
    • ensure_list()
    • eval_bool()
    • expand_dollar_count()
    • get_current_id()
    • get_tree_element()
    • get_tree_elements()
    • id_dir()
    • now_str()
    • read_timestamps()
    • resolve_depend()
    • safe_split()
    • script_evaluation()
    • substitution()
    • update_timestamps()
    • valid_tags()
  • jube2.util.yaml_converter module
    • YAML_Converter
      • YAML_Converter.allowed_tags
      • YAML_Converter.close()
      • YAML_Converter.create_headtags()
      • YAML_Converter.create_tag()
      • YAML_Converter.is_parseable_yaml_file()
      • YAML_Converter.read()

Submodules

jube2.analyser module

The Analyser class handles the analyse process

class jube2.analyser.Analyser(name, reduce_iteration=True)

Bases: object

The Analyser handles the analyse process and store all important data to run a new analyse.

class AnalyseFile(path)

Bases: object

A file which should be analysed

add_uses(use_names)

Add an addtional patternset name

etree_repr()

Return etree object representation

property path

Get file path

property use

Return uses

add_analyse(step_name, analyse_file)

Add an addtional analyse file

add_uses(use_names)

Add an addtional patternset name

analyse()

Run the analyser

analyse_etree_repr()

Create an etree representation of a analyse dict: stepname -> workpackage_id -> filename -> patternname -> value

property analyse_result

Return analyse result

property analyser

Return analyse dict

property benchmark

Get benchmark information

etree_repr()

Return etree object representation

property name

Get analyser name

property use

Return uses

jube2.benchmark module

The Benchmark class manages the benchmark process

class jube2.benchmark.Benchmark(name, outpath, parametersets, substitutesets, filesets, patternsets, steps, analyser, results, results_order, comment='', tags=None, file_path_ref='.')

Bases: object

The Benchmark class contains all data to run a benchmark

add_tags(other_tags)

analyse(show_info=True, specific_analyser_name=None)

Run analyser

property analyser

Return analyser

property bench_dir

Return benchmark directory

property benchmark_status

Retun global workpackage information dict

property comment

Return comment string

create_result(only=None, show=False, data_list=None, style=None, select=None, exclude=None)

Show benchmark result

delete_bench_dir()

Delete all data inside benchmark directory

etree_repr(new_cwd=None)

Return etree object representation

property file_path_ref

Get file path reference

property filesets

Return filesets

get_jube_parameterset()

Return parameterset which contains benchmark related information

property id

Return benchmark id

property name

Return benchmark name

new_run()

Create workpackage structure and run benchmark

property outpath

Return outpath

property parametersets

Return parametersets

property patternsets

Return patternsets

remove_workpackage(workpackage_to_delete)

Remove a specifc workpackage

reset_all_workpackages()

Reset workpackage state

property results

Return results

property results_order

Return results_order

run()

Run benchmark

set_workpackage_information(workpackages, work_stat)

Set new workpackage information

property steps

Return steps

property substitutesets

Return substitutesets

property tags

Return set of tags

update_analyse_and_result(new_patternsets, new_analyser, new_results, new_results_order, new_cwd)

Update analyser and result data

property work_stat

Return work queue

workpackage_by_id(wp_id)

Search and return a benchmark workpackage by its wp_id

property workpackage_status

Retun workpackage information dict

property workpackages

Return workpackages

wp_post_run_config(workpackage)

additional processing of workpackage: - update status bar - build up queue after restart

write_analyse_data(filename)

All analyse data will be written to given file using xml representation

write_benchmark_configuration(filename, outpath=None)

The current benchmark configuration will be written to given file using xml representation

write_workpackage_information(filename)

All workpackage information will be written to given file using xml representation

jube2.completion module

Shell Completions

jube2.completion.complete_function_bash(args)

Print completion function for bash.

jube2.conf module

Configuration

jube2.fileset module

Fileset related classes

class jube2.fileset.Copy(path, name=None, is_internal_ref=False, active='true', source_dir='', target_dir='')

Bases: File

A file or directory given by path. Which can be copied to the work_dir inside steps.

create_action(path, name, new_file_path)

Copy file/directory to work_dir

etree_repr()

Return etree object representation

class jube2.fileset.File(path, name=None, is_internal_ref=False, active='true', source_dir='', target_dir='')

Bases: object

Generic file access

create(work_dir, parameter_dict, alt_work_dir=None, file_path_ref='', environment=None)

Create file access

create_action(path, name, new_file_path)

File access type specific creation

etree_repr()

Return etree object representation

property file_path_ref

Get file path reference

property is_internal_ref

Return path is internal ref

property path

Return filepath

class jube2.fileset.Fileset(name)

Bases: list

Container for file copy, link and prepare operations

create(work_dir, parameter_dict, alt_work_dir=None, environment=None, file_path_ref='')

Copy/load/prepare all files in fileset

etree_repr()

Return etree object representation

property name

Return fileset name

class jube2.fileset.Link(path, name=None, is_internal_ref=False, active='true', source_dir='', target_dir='')

Bases: File

A link to a given path. Which can be used inside steps.

create_action(path, name, new_file_path)

Create link to file in work_dir

etree_repr()

Return etree object representation

class jube2.fileset.Prepare(cmd, stdout_filename=None, stderr_filename=None, work_dir=None, active='true')

Bases: Operation

Prepare the workpackage work directory

etree_repr()

Return etree object representation

execute(parameter_dict, work_dir, only_check_pending=False, environment=None)

Execute the prepare command

jube2.help module

User help

jube2.help.load_help()

Load additional documentation out of help file and add these data to global help dictionary.

jube2.info module

Gives benchmark related info

jube2.info.print_benchmark_info(benchmark)

Print information concerning a single benchmark

jube2.info.print_benchmark_status(benchmark)

Print overall workpackage status in the following order RUNNING: At least one WP is still active ERROR: At least one WP raised an errror FINISHED: All WPs are finalized and no error was raised

jube2.info.print_benchmarks_info(path)

Print list of all benchmarks, found in given directory

jube2.info.print_step_info(benchmark, step_name, parametrization_only=False, parametrization_only_csv=False)

Print information concerning a single step in a specific benchmark

jube2.info.print_workpackage_info(benchmark, workpackage)

Print information concerning a single workpackage in a specific benchmark

jube2.jubeio module

Basic I/O module

class jube2.jubeio.Parser(filename, tags=None, include_path=None, force=False, strict=False)

Bases: object

JUBE XML input file parser

analyse_result_from_xml()

Read existing analyse out of xml-file

benchmark_info_from_xml()

Return name, comment and available tags of first benchmark found in file

benchmarks_from_xml()

Return a dict of benchmarks

Here parametersets are global and accessible to all benchmarks defined in the corresponding XML file.

property file_path_ref

Return file path given by config file

workpackages_from_xml(benchmark)

Read existing workpackage data out of a xml-file

jube2.log module

Logging Support

class jube2.log.JubeLogger(name, level=0)

Bases: Logger, object

Overwrite logging to handle multi line messages.

jube2.log.change_logfile_name(filename)

Change log file name if not in debug mode.

jube2.log.get_logger(name=None)

Return logger given by name

jube2.log.log_print(text)

Output text

jube2.log.matching_logs(commands, available_logs)

Find intersection between requested logs and available logs.

Returns tuple (matching, not_matching), containing the intersection and its complement.

Only compares basenames.

jube2.log.only_console_log()

Change to console log if not in debug mode.

jube2.log.reset_logging()

Reset logging to default.

jube2.log.safe_output_logfile(filename)

Try to print logfile. If try fails, fail gracefully.

jube2.log.search_for_logs(path=None)

Search for files matching in path with .log extension

jube2.log.setup_logging(mode=None, filename=None, verbose=None)

Setup the logging configuration.

Available modes are

default log to console and file console only console output

filename can be given optionally.

verbose: enable verbose console output

The setup includes setting the handlers and formatters. Calling this function multiple times causes old handlers to be removed before new ones are added.

jube2.main module

CLI program

jube2.main.analyse_benchmarks(args)

Analyse benchmarks

jube2.main.benchmarks_results(args)

Show benchmark results

jube2.main.command_help(args)

Show command help

jube2.main.complete(args)

Handle shell completion

jube2.main.continue_benchmarks(args)

Continue benchmarks

jube2.main.gen_parser_conf()

Generate dict with parser information

jube2.main.gen_subparser_conf()

Generate dict with subparser information

jube2.main.info(args)

Benchmark information

jube2.main.main(command=None)

Parse the command line and run the requested command.

jube2.main.manipulate_comments(args)

Manipulate benchmark comment

jube2.main.output(args)

Show output filename

jube2.main.remove_benchmarks(args)

Remove benchmarks or workpackages

jube2.main.run_new_benchmark(args)

Start a new benchmark run

jube2.main.search_for_benchmarks(args)

Search for existing benchmarks

jube2.main.search_for_workpackage(args, search_for_step=False)

Search for existing workpackages

jube2.main.show_log(args)

Show logs for benchmarks

jube2.main.show_log_single(args, benchmark_folder)

Show logs for a single benchmark

jube2.main.status(args)

Show benchmark status

jube2.main.update_check(args)

Check if a newer JUBE version is available.

jube2.parameter module

Parameter related classes

class jube2.parameter.FixedParameter(name, value, separator=None, parameter_type='string', parameter_mode='text', unit='', export=False, update_mode='never', idx=-1, eval_helper=None, duplicate='none')

Bases: StaticParameter

A FixedParameter is a parameter which can not be evaluated anymore. It represents a fixed value.

substitute_and_evaluate(parametersets=None, final_sub=False, no_templates=False, force_evaluation=False)

No substitute

class jube2.parameter.Parameter(name, value, separator=None, parameter_type='string', parameter_mode='text', unit='', export=False, update_mode='never', idx=-1, eval_helper=None, duplicate='none')

Bases: object

Contains data for single Parameter. This Parameter can be a constant value, a template or a specific value out of a given template

property based_on

The base of the current Parameter

property based_on_mode

Return the root parameter mode inside the based_on graph

property based_on_root

Return the root parameter inside the based_on graph

property based_on_value

Return the root value inside the based_on graph

copy()

Returns Parameter copy (flat copy)

static create_parameter(name, value, separator=None, parameter_type='string', selected_value=None, parameter_mode='text', unit='', export=False, no_templates=False, update_mode='never', idx=-1, eval_helper=None, fixed=False, duplicate='none')

Parameter constructor. Return a Static- or TemplateParameter based on the given data.

property duplicate

Return duplicate option

etree_repr(use_current_selection=False)

Return etree object representation

property eval_helper

Return evaluation helper function

property export

Return if parameter should be exported

property idx

Return template idx

is_equivalent(parameter, first_check=True)

Checks whether the given and the current Parameter based on equivalent templates and (if expanded) contain the same value and were on the same place within the original template

property is_fixed

Return whether the parameter is fixed

property is_jube_parameter

Parameter is handled by JUBE automatically

property is_template

Return whether the parameter is a template

property mode

Return parameter mode

property name

Returns the Parameter name

parameter_regex = re.compile('(?<!\\$)(?:\\$\\$)*\\$(?!\\$)(\\{)?(\\w+?)(?(1)\\}|(?=\\W|$))')

property parameter_type

Return parametertype

search_method(propertyString, recursiveProperty=None)

Searches, potentially recursively, for a method and returns True in case of a success

property unit

Return unit

update_allowed(mode)

Check wether the parameter can be updated using the given update mode

property update_mode

Returns the update mode of the parameter

property value

Return parameter value

class jube2.parameter.Parameterset(name='', duplicate='replace')

Bases: object

A parameterset represent a template or a specific product space. It can be combined with other Parametersets.

add_parameter(parameter)

Add a new parameter

add_parameterset(parameterset)

Add all parameters from given parameterset, existing ones will be overwritten

property all_parameter_names

Return list of all parameter names

property all_parameters

Return list of all parameters

check_parameter_options(parameter, only_duplicate=True)

Check whether both parameters have identical options and throw an error if this is not the case

clear()

Remove all stored parameters

concat_parameter(parameter)

Concatenate a new parameter to a potentially existing one.

property constant_parameter_dict

Return dictionary representation of all constant parameters

copy()

Returns a deepcopy of the Parameterset

delete_parameter(parameter)

Delete a parameter

property duplicate

Return the duplicate property of the Parameterset

etree_repr(use_current_selection=False)

Return etree object representation

expand_templates()

Expand all remaining templates in the Parameterset and returns the resulting parametersets

property export_parameter_dict

Return dictionary representation of all export parameters

get_incompatible_parameter(parameterset, update_mode='never')

Return a set of incompatible parameter names between the current and the given parameterset

get_updatable_parameter(mode, keep_index=False)

Returns a parameterset containing all updatable parameter for a specific mode, the root parameter is added

property has_templates

This Parameterset contains template paramters?

is_compatible(parameterset, update_mode='never')

Two Parametersets are compatible, if the intersection only contains equivilant parameters

property name

Return name of the Parameterset

property parameter_dict

Return dictionary name -> parameter

parameter_substitution(additional_parametersets=None, final_sub=False)

Substitute all parameter inside the parameterset. Parameters from additional_parameterset will be used for substitution but will not be added to the set. final_sub marks the last substitution process.

remove_jube_parameter()

Remove JUBE update mode parameter from the parameterset

property template_parameter_dict

Return dictionary representation of all template parameters

update_parameterset(parameterset)

Overwrite existing parameters. Do not add new parameters

class jube2.parameter.StaticParameter(name, value, separator=None, parameter_type='string', parameter_mode='text', unit='', export=False, update_mode='never', idx=-1, eval_helper=None, duplicate='none')

Bases: Parameter

A StaticParameter can be substituted and evaluated.

can_substitute_and_evaluate(parameterset)

A parameter can be substituted and evaluated if there are no depending templates or unevaluated parameter inside

depends_on(parameter)

Checks the parameter depends on an other parameter.

static fix_export_string(value)

Add missing quotes to jube_wp_envstr if needed

substitute_and_evaluate(parametersets=None, final_sub=False, no_templates=False, force_evaluation=False)

Substitute all variables inside the parameter value by using the parameters inside the given parameterset. final_sub marks the last substitution.

Return the new parameter and a boolean value which represent a change of value

class jube2.parameter.TemplateParameter(name, value, separator=None, parameter_type='string', parameter_mode='text', unit='', export=False, update_mode='never', idx=-1, eval_helper=None, duplicate='none')

Bases: Parameter

A TemplateParameter represent a set of possible parameter values, which can be accessed by a single name. To use the template in a specific environment, it must be expanded.

expand()

Expand Template and produce set of static parameter

property value

Return Template values

jube2.pattern module

Patternset definition

class jube2.pattern.Pattern(name, value, pattern_mode='pattern', content_type='string', unit='', default=None, dotall=False)

Bases: StaticParameter

A pattern can be used to scan a result file, using regular expression, or to represent a derived pattern.

property content_type

Return pattern type

property default_value

Return pattern default value

property derived

pattern is a derived pattern

property dotall

Return pattern dot regex handling

etree_repr(use_current_selection=False)

Return etree object representation

substitute_and_evaluate(parametersets=None, final_sub=False, no_templates=True, force_evaluation=False)

Substitute all variables inside the pattern value by using the parameter inside the given parameterset and additional_parameterset. final_sub marks the last substitution.

Return the new pattern and a boolean value which represent a change of value

property unit

Return unit

class jube2.pattern.Patternset(name='')

Bases: object

A Patternset stores a set of pattern and derived pattern.

add_pattern(pattern)

Add a additional pattern to the patternset. Existing pattern using the same name will be overwritten

add_patternset(patternset)

Add all pattern from given patternset to the current one

copy()

Returns a copy of the Parameterset

property derived_pattern_storage

Return the derived pattern storage

derived_pattern_substitution(parametersets=None)

Run derived pattern substitution using additional parameterset

etree_repr()

Return etree object representation

get_incompatible_pattern(patternset)

Return a set of incompatible pattern names between the current and the given parameterset

is_compatible(patternset)

Two Patternsets are compatible, if all pattern storages are compatible

property name

Get patternset name

property pattern_storage

Return the pattern storage

pattern_substitution(parametersets=None)

Run pattern substitution using additional parameterset

jube2.pattern.get_jube_pattern()

Return jube internal patternset

jube2.result module

Resulttype definition

class jube2.result.Result(name, res_filter=None)

Bases: object

A generic result type

class ResultData(name)

Bases: object

A gerneric result data type

add_result_data(result_data)

Add additional result data

create_result(show=True, filename=None, **kwargs)

Create result output

property name

Return the result name

add_uses(use_names)

Add an addtional analyser name

property benchmark

Return the benchmark

create_result_data()

Create result representation

etree_repr()

Return etree object representation

property name

Return the result name

property result_dir

Return the result_dir

jube2.step module

Step contains the commands for steps

class jube2.step.DoLog(log_dir, log_file, initial_env, cycle=0)

Bases: object

A DoLog class containing the operations and information for setting up the do log.

property initial_env

Get initial env

initialiseFile(shell)

Initialise file if not yet existent.

property log_file

Get log file

property log_path

Get log path

store_do(do, shell, work_dir, parameter_dict=None, shared=False)

Store the current execution directive to the do log and set up the environment if file does not yet exist.

property work_dir

Get last work directory

class jube2.step.Operation(do, async_filename=None, stdout_filename=None, stderr_filename=None, active='true', shared=False, work_dir=None, break_filename=None, error_filename=None)

Bases: object

The Operation-class represents a single instruction, which will be executed in a shell environment.

active(parameter_dict)

Return active status of the current operation depending on the given parameter_dict

property async_filename

Get async filename

property error_filename

Get error filename

etree_repr()

Return etree object representation

execute(parameter_dict, work_dir, only_check_pending=False, environment=None, pid=None, dolog=None)

Execute the operation. work_dir must be set to the given context path. The parameter_dict used for inline substitution. If only_check_pending is set to True, the operation will not be executed, only the async_file will be checked. Return operation status: True => operation finished False => operation pending

static read_process_environment(work_dir, remove_after_read=True, pid=None)

Read standard environment info file in given directory.

property shared

Shared operation?

property stderr_filename

Get stderr filename

property stdout_filename

Get stdout filename

class jube2.step.Step(name, depend, iterations=1, alt_work_dir=None, shared_name=None, export=False, max_wps='0', active='true', suffix='', cycles=1, procs=1, do_log_file=None)

Bases: object

A Step represent one execution step. It contains a list of Do-operations and multiple parametersets, substitutionsets and filesets. A Step is a template for Workpackages.

property active

Return active state

add_operation(operation)

Add operation

add_uses(use_names)

Add use

property alt_work_dir

Return alternativ work directory

create_workpackages(benchmark, global_parameterset, local_parameterset=None, used_sets=None, iteration_base=0, parents=None, incompatible_parameters=None)

Create workpackages for current step using given benchmark context

property cycles

Return number of cycles

property depend

Return dependencies

property do_log_file

Return do log file name

etree_repr()

Return etree object representation

property export

Return export behaviour

get_depend_history(benchmark)

Creates a set of all dependent steps in history for given benchmark

get_jube_parameterset()

Return parameterset which contains step related information

get_used_sets(available_sets, parameter_dict=None)

Get list of all used sets, which can be found in available_sets

property iterations

Return iterations

property max_wps

Return maximum number of simultaneous workpackages

property name

Return step name

property operations

Return operations

property procs

Return number of procs

shared_folder_path(benchdir, parameter_dict=None)

Return shared folder name

property shared_link_name

Return shared link name

property suffix

Return directory suffix

property use

Return parameters and substitutions

jube2.substitute module

Substitution related classes

class jube2.substitute.Sub(source, sub_mode, dest)

Bases: object

property dest

Return dest of Sub

property mode

Return type of Sub

property source

Return source of Sub

class jube2.substitute.Substituteset(name, file_data, substitute_dict)

Bases: object

A Substituteset contains all information

etree_repr()

Return etree object representation

property name

Return name of Substituteset

substitute(parameter_dict=None, work_dir=None)

Do substitution. The work_dir can be set to a given context path. The parameter_dict used for inline substitution of destination-variables.

update_files(file_data)

Update iofiles

update_substitute(substitute_dict)

Update substitute_dict

jube2.workpackage module

The Workpackage class handles a step and its parameter space

class jube2.workpackage.Workpackage(benchmark, step, local_parameter_names, parameterset, workpackage_id=None, iteration=0, cycle=0)

Bases: object

A Workpackage contains all information to run a specific step with its given parameterset.

property active

Check active state

add_children(workpackage)

Add a children workpackage

add_parent(workpackage)

Add a parent Workpackage

allow_workpackage_dir_caching()

Enable workpackage dir cache

alt_work_dir(parameter_dict=None)

Return location of alternative working_dir

property benchmark

Return benchmark of this workpackage

property children

Return list of child workpackages

property children_future

Create a list of all children in the future of this workpackage

create_abspath(value)

Create absolute path representation

create_relpath(value)

Create relative path representation

create_shared_folder_link(parameter_dict=None)

Create shared folder connection

create_workpackage_dir()

Create work directory

property cycle

Return current loop cycle

property done

Workpackage done?

property env

Return workpackage environment

property error

Workpackage error?

etree_repr()

Return etree object representation

get_jube_cycle_parameterset()

Return parameterset which contains cycle related information

get_jube_parameterset()

Return parameterset which contains workpackage related information

property id

Return workpackage id

id_counter = 0

property iteration

Return workpackage iteration number

property iteration_siblings

Return set of iteration siblings

property local_parameterset

Return local parameterset

operation_done(operation_number, set_done=None)

Mark/checks operation status

operation_done_but_pending(operation_number)

Check if an operation was executed, but the result is still pending (because it is a async do)

property parameter_dict

get all available parameter inside a dict

property parameterset

Return parameterset

property parent_history

Create a list of all parents in the history of this workpackage

property parents

Return list of parent workpackages

property queued

Workpackage queued?

static reduce_workpackage_id_counter()

remove(remove_config_from_benchmark=False)

Remove all data of this workpackage

run(mode='s')

Run step and use current parameter space mode: s = seriell (default); p = parallel

set_error(set_error, msg='')

Set/reset Workpackage error

property started

Workpackage started?

status()

return FINISHED, RUNNING or DONE dependign on the workpackage status

property step

Return Step data

update_status()

Update status in jube parameter

property work_dir

Return working directory (user space)

property workpackage_dir

Return workpackage directory