jube package¶
jube package
Subpackages¶
- jube.result_types package
- Submodules
- jube.result_types.keyvaluesresult module
KeyValuesResult
KeyValuesResult.DataKey
KeyValuesResult.KeyValuesData
KeyValuesResult.KeyValuesData.add_id_information()
KeyValuesResult.KeyValuesData.add_key_value_data()
KeyValuesResult.KeyValuesData.add_result_data()
KeyValuesResult.KeyValuesData.benchmark_ids
KeyValuesResult.KeyValuesData.create_result()
KeyValuesResult.KeyValuesData.data
KeyValuesResult.KeyValuesData.data_dict
KeyValuesResult.KeyValuesData.keys
KeyValuesResult.add_key()
KeyValuesResult.create_result_data()
KeyValuesResult.sort
- jube.result_types.syslog module
- jube.result_types.table module
- jube.util package
- Submodules
- jube.util.output module
- jube.util.util module
CompType
Queue
WorkStat
check_and_get_benchmark_outpath()
check_and_get_group_id()
check_and_get_verbose_level()
consistency_check()
convert_type()
ensure_list()
eval_bool()
expand_dollar_count()
get_current_id()
get_tree_element()
get_tree_elements()
id_dir()
now_str()
read_timestamps()
resolve_depend()
safe_split()
script_evaluation()
substitution()
update_timestamps()
valid_tags()
- jube.util.yaml_converter module
Submodules¶
jube.analyser module¶
The Analyser class handles the analyse process
- class jube.analyser.Analyser(name, reduce_iteration=True)[source]¶
Bases:
object
The Analyser handles the analyse process and store all important data to run a new analyse.
- class AnalyseFile(path)[source]¶
Bases:
object
A file which should be analysed
- property path¶
Get file path
- property use¶
Return uses
- analyse_etree_repr()[source]¶
Create an etree representation of a analyse dict: stepname -> workpackage_id -> filename -> patternname -> value
- property analyse_result¶
Return analyse result
- property analyser¶
Return analyse dict
- property benchmark¶
Get benchmark information
- property name¶
Get analyser name
- property reduce¶
Get analyser reduce
- property use¶
Return uses
jube.benchmark module¶
The Benchmark class manages the benchmark process
- class jube.benchmark.Benchmark(name, outpath, parametersets, substitutesets, filesets, patternsets, steps, analyser, results, results_order, comment='', tags=None, tag_docu={}, file_path_ref='.')[source]¶
Bases:
object
The Benchmark class contains all data to run a benchmark
- property analyser¶
Return analyser
- property bench_dir¶
Return benchmark directory
- property benchmark_status¶
Retun global workpackage information dict
- property comment¶
Return comment string
- create_result(only=None, show=False, data_list=None, style=None, select=None, exclude=None)[source]¶
Show benchmark result
- property file_path_ref¶
Get file path reference
- property filesets¶
Return filesets
- property id¶
Return benchmark id
- property name¶
Return benchmark name
- property parametersets¶
Return parametersets
- property patternsets¶
Return patternsets
- property results¶
Return results
- property results_order¶
Return results_order
- property steps¶
Return steps
- property substitutesets¶
Return substitutesets
- property tag_docu¶
Return dict of tag documentation
- property tags¶
Return set of tags
- update_analyse_and_result(new_patternsets, new_analyser, new_results, new_results_order, new_cwd)[source]¶
Update analyser and result data
- property work_stat¶
Return work queue
- property workpackage_status¶
Retun workpackage information dict
- property workpackages¶
Return workpackages
- wp_post_run_config(workpackage)[source]¶
additional processing of workpackage: - update status bar - build up queue after restart
- write_analyse_data(filename)[source]¶
All analyse data will be written to given file using xml representation
jube.completion module¶
Shell Completions
jube.conf module¶
Configuration
jube.fileset module¶
Fileset related classes
- class jube.fileset.Copy(path, name=None, is_internal_ref=False, active='true', source_dir='', target_dir='')[source]¶
Bases:
File
A file or directory given by path. Which can be copied to the work_dir inside steps.
- class jube.fileset.File(path, name=None, is_internal_ref=False, active='true', source_dir='', target_dir='')[source]¶
Bases:
object
Generic file access
- property active¶
Return file active status
- create(work_dir, parameter_dict, alt_work_dir=None, file_path_ref='', environment=None)[source]¶
Create file access
- property file_path_ref¶
Get file path reference
- property file_type¶
Return file type
- property is_internal_ref¶
Return path is internal ref
- property path¶
Return filepath
- property source_dir¶
Return file source dir
- property target_dir¶
Return file source dir
- class jube.fileset.Fileset(name)[source]¶
Bases:
list
Container for file copy, link and prepare operations
- create(work_dir, parameter_dict, alt_work_dir=None, environment=None, file_path_ref='')[source]¶
Copy/load/prepare all files in fileset
- property name¶
Return fileset name
- class jube.fileset.Link(path, name=None, is_internal_ref=False, active='true', source_dir='', target_dir='')[source]¶
Bases:
File
A link to a given path. Which can be used inside steps.
jube.help module¶
User help
jube.info module¶
Gives benchmark related info
- jube.info.print_benchmark_status(benchmark)[source]¶
Print overall workpackage status in the following order RUNNING: At least one WP is still active ERROR: At least one WP raised an errror FINISHED: All WPs are finalized and no error was raised
- jube.info.print_benchmark_tag_documentation(benchmark)[source]¶
Print tag documentation concerning a specific benchmark
- jube.info.print_benchmarks_info(path)[source]¶
Print list of all benchmarks, found in given directory
- jube.info.print_step_info(benchmark, step_name, parametrization_only=False, parametrization_only_csv=False)[source]¶
Print information concerning a single step in a specific benchmark
jube.jubeio module¶
Basic I/O module
- class jube.jubeio.Parser(filename, tags=None, include_path=None, outpath=None, force=False, strict=False, command_name=None)[source]¶
Bases:
object
JUBE XML input file parser
- benchmark_info_from_xml()[source]¶
Return name, comment and available tags of first benchmark found in file
- benchmarks_from_xml(check_tags=True)[source]¶
Return a dict of benchmarks
Here parametersets are global and accessible to all benchmarks defined in the corresponding XML file.
- property file_path_ref¶
Return file path given by config file
jube.log module¶
Logging Support
- class jube.log.JubeLogger(name, level=0)[source]¶
Bases:
Logger
,object
Overwrite logging to handle multi line messages.
- jube.log.matching_logs(commands, available_logs)[source]¶
Find intersection between requested logs and available logs.
Returns tuple (matching, not_matching), containing the intersection and its complement.
Only compares basenames.
- jube.log.safe_output_logfile(filename)[source]¶
Try to print logfile. If try fails, fail gracefully.
- jube.log.setup_logging(mode=None, filename=None, verbose=None)[source]¶
Setup the logging configuration.
- Available modes are
default log to console and file console only console output
filename can be given optionally.
verbose: enable verbose console output
The setup includes setting the handlers and formatters. Calling this function multiple times causes old handlers to be removed before new ones are added.
jube.main module¶
CLI program
jube.parameter module¶
Parameter related classes
- class jube.parameter.FixedParameter(name, value, separator=None, parameter_type='string', parameter_mode='text', unit='', export=False, update_mode='never', idx=-1, eval_helper=None, duplicate='none')[source]¶
Bases:
StaticParameter
A FixedParameter is a parameter which can not be evaluated anymore. It represents a fixed value.
- class jube.parameter.Parameter(name, value, separator=None, parameter_type='string', parameter_mode='text', unit='', export=False, update_mode='never', idx=-1, eval_helper=None, duplicate='none')[source]¶
Bases:
object
Contains data for single Parameter. This Parameter can be a constant value, a template or a specific value out of a given template
- property based_on¶
The base of the current Parameter
- property based_on_mode¶
Return the root parameter mode inside the based_on graph
- property based_on_root¶
Return the root parameter inside the based_on graph
- property based_on_value¶
Return the root value inside the based_on graph
- static create_parameter(name, value, separator=None, parameter_type='string', selected_value=None, parameter_mode='text', unit='', export=False, no_templates=False, update_mode='never', idx=-1, eval_helper=None, fixed=False, duplicate='none')[source]¶
Parameter constructor. Return a Static- or TemplateParameter based on the given data.
- property duplicate¶
Return duplicate option
- property eval_helper¶
Return evaluation helper function
- property export¶
Return if parameter should be exported
- property idx¶
Return template idx
- is_equivalent(parameter, first_check=True)[source]¶
Checks whether the given and the current Parameter based on equivalent templates and (if expanded) contain the same value and were on the same place within the original template
- property is_fixed¶
Return whether the parameter is fixed
- property is_jube_parameter¶
Parameter is handled by JUBE automatically
- property is_template¶
Return whether the parameter is a template
- property mode¶
Return parameter mode
- property name¶
Returns the Parameter name
- parameter_regex = re.compile('(?<!\\$)(?:\\$\\$)*\\$(?!\\$)(\\{)?(\\w+?)(?(1)\\}|(?=\\W|$))')¶
- property parameter_type¶
Return parametertype
- search_method(propertyString, recursiveProperty=None)[source]¶
Searches, potentially recursively, for a method and returns True in case of a success
- property separator¶
Return seperator
- property unit¶
Return unit
- property update_mode¶
Returns the update mode of the parameter
- property value¶
Return parameter value
- class jube.parameter.Parameterset(name='', duplicate='replace')[source]¶
Bases:
object
A parameterset represent a template or a specific product space. It can be combined with other Parametersets.
- add_parameterset(parameterset)[source]¶
Add all parameters from given parameterset, existing ones will be overwritten
- property all_parameter_names¶
Return list of all parameter names
- property all_parameters¶
Return list of all parameters
- check_parameter_options(parameter, only_duplicate=True)[source]¶
Check whether both parameters have identical options and throw an error if this is not the case
- property constant_parameter_dict¶
Return dictionary representation of all constant parameters
- property duplicate¶
Return the duplicate property of the Parameterset
- expand_templates()[source]¶
Expand all remaining templates in the Parameterset and returns the resulting parametersets
- property export_parameter_dict¶
Return dictionary representation of all export parameters
- get_incompatible_parameter(parameterset, update_mode='never')[source]¶
Return a set of incompatible parameter names between the current and the given parameterset
- get_updatable_parameter(mode, keep_index=False)[source]¶
Returns a parameterset containing all updatable parameter for a specific mode, the root parameter is added
- property has_templates¶
This Parameterset contains template paramters?
- is_compatible(parameterset, update_mode='never')[source]¶
Two Parametersets are compatible, if the intersection only contains equivilant parameters
- property name¶
Return name of the Parameterset
- property parameter_dict¶
Return dictionary name -> parameter
- parameter_substitution(additional_parametersets=None, final_sub=False)[source]¶
Substitute all parameter inside the parameterset. Parameters from additional_parameterset will be used for substitution but will not be added to the set. final_sub marks the last substitution process.
- property template_parameter_dict¶
Return dictionary representation of all template parameters
- class jube.parameter.StaticParameter(name, value, separator=None, parameter_type='string', parameter_mode='text', unit='', export=False, update_mode='never', idx=-1, eval_helper=None, duplicate='none')[source]¶
Bases:
Parameter
A StaticParameter can be substituted and evaluated.
- can_substitute_and_evaluate(parameterset)[source]¶
A parameter can be substituted and evaluated if there are no depending templates or unevaluated parameter inside
- substitute_and_evaluate(parametersets=None, final_sub=False, no_templates=False, force_evaluation=False)[source]¶
Substitute all variables inside the parameter value by using the parameters inside the given parameterset. final_sub marks the last substitution.
Return the new parameter and a boolean value which represent a change of value
- class jube.parameter.TemplateParameter(name, value, separator=None, parameter_type='string', parameter_mode='text', unit='', export=False, update_mode='never', idx=-1, eval_helper=None, duplicate='none')[source]¶
Bases:
Parameter
A TemplateParameter represent a set of possible parameter values, which can be accessed by a single name. To use the template in a specific environment, it must be expanded.
- property value¶
Return Template values
jube.pattern module¶
Patternset definition
- class jube.pattern.Pattern(name, value, pattern_mode='pattern', content_type='string', unit='', default=None, dotall=False)[source]¶
Bases:
StaticParameter
A pattern can be used to scan a result file, using regular expression, or to represent a derived pattern.
- property content_type¶
Return pattern type
- property default_value¶
Return pattern default value
- property derived¶
pattern is a derived pattern
- property dotall¶
Return pattern dot regex handling
- substitute_and_evaluate(parametersets=None, final_sub=False, no_templates=True, force_evaluation=False)[source]¶
Substitute all variables inside the pattern value by using the parameter inside the given parameterset and additional_parameterset. final_sub marks the last substitution.
Return the new pattern and a boolean value which represent a change of value
- property unit¶
Return unit
- class jube.pattern.Patternset(name='')[source]¶
Bases:
object
A Patternset stores a set of pattern and derived pattern.
- add_pattern(pattern)[source]¶
Add a additional pattern to the patternset. Existing pattern using the same name will be overwritten
- property derived_pattern_storage¶
Return the derived pattern storage
- derived_pattern_substitution(parametersets=None)[source]¶
Run derived pattern substitution using additional parameterset
- get_incompatible_pattern(patternset)[source]¶
Return a set of incompatible pattern names between the current and the given parameterset
- is_compatible(patternset)[source]¶
Two Patternsets are compatible, if all pattern storages are compatible
- property name¶
Get patternset name
- property pattern_storage¶
Return the pattern storage
jube.result module¶
Resulttype definition
- class jube.result.Result(name, res_filter=None)[source]¶
Bases:
object
A generic result type
- class ResultData(name)[source]¶
Bases:
object
A gerneric result data type
- property name¶
Return the result name
- property benchmark¶
Return the benchmark
- property name¶
Return the result name
- property res_filter¶
Return the result filter
- property result_dir¶
Return the result_dir
- property result_type¶
Return class name
- property use¶
Return the result name
jube.step module¶
Step contains the commands for steps
- class jube.step.DoLog(log_dir, log_file, initial_env, cycle=0)[source]¶
Bases:
object
A DoLog class containing the operations and information for setting up the do log.
- property initial_env¶
Get initial env
- property log_file¶
Get log file
- property log_path¶
Get log path
- store_do(do, shell, work_dir, parameter_dict=None, shared=False)[source]¶
Store the current execution directive to the do log and set up the environment if file does not yet exist.
- property work_dir¶
Get last work directory
- class jube.step.Operation(do, async_filename=None, stdout_filename=None, stderr_filename=None, active='true', shared=False, work_dir=None, break_filename=None, error_filename=None)[source]¶
Bases:
object
The Operation-class represents a single instruction, which will be executed in a shell environment.
- active(parameter_dict)[source]¶
Return active status of the current operation depending on the given parameter_dict
- property active_string¶
Get active
- property async_filename¶
Get async filename
- property break_filename¶
Get break filename
- property do¶
Get do
- property error_filename¶
Get error filename
- execute(parameter_dict, work_dir, only_check_pending=False, environment=None, pid=None, dolog=None)[source]¶
Execute the operation. work_dir must be set to the given context path. The parameter_dict used for inline substitution. If only_check_pending is set to True, the operation will not be executed, only the async_file will be checked. Return operation status: True => operation finished False => operation pending
- static read_process_environment(work_dir, remove_after_read=True, pid=None)[source]¶
Read standard environment info file in given directory.
Shared operation?
- property stderr_filename¶
Get stderr filename
- property stdout_filename¶
Get stdout filename
- property work_dir¶
Get work directory
- class jube.step.Step(name, depend, iterations=1, alt_work_dir=None, shared_name=None, export=False, max_wps='0', active='true', suffix='', cycles=1, procs=1, do_log_file=None)[source]¶
Bases:
object
A Step represent one execution step. It contains a list of Do-operations and multiple parametersets, substitutionsets and filesets. A Step is a template for Workpackages.
- property active¶
Return active state
- property alt_work_dir¶
Return alternativ work directory
- create_workpackages(benchmark, global_parameterset, local_parameterset=None, used_sets=None, iteration_base=0, parents=None, incompatible_parameters=None)[source]¶
Create workpackages for current step using given benchmark context
- property cycles¶
Return number of cycles
- property depend¶
Return dependencies
- property do_log_file¶
Return do log file name
- property export¶
Return export behaviour
- get_depend_history(benchmark)[source]¶
Creates a set of all dependent steps in history for given benchmark
- get_used_sets(available_sets, parameter_dict=None)[source]¶
Get list of all used sets, which can be found in available_sets
- property iterations¶
Return iterations
- property max_wps¶
Return maximum number of simultaneous workpackages
- property name¶
Return step name
- property operations¶
Return operations
- property procs¶
Return number of procs
Return shared folder name
Return shared link name
- property suffix¶
Return directory suffix
- property use¶
Return parameters and substitutions
- property work_dir¶
Return alternative working directory
jube.substitute module¶
Substitution related classes
- class jube.substitute.Sub(source, sub_mode, dest)[source]¶
Bases:
object
- property dest¶
Return dest of Sub
- property mode¶
Return type of Sub
- property source¶
Return source of Sub
- class jube.substitute.Substituteset(name, file_data, substitute_dict)[source]¶
Bases:
object
A Substituteset contains all information
- property files¶
Return iofiles
- property name¶
Return name of Substituteset
- property subs¶
Return subs
jube.workpackage module¶
The Workpackage class handles a step and its parameter space
- class jube.workpackage.Workpackage(benchmark, step, local_parameter_names, parameterset, workpackage_id=None, iteration=0, cycle=0)[source]¶
Bases:
object
A Workpackage contains all information to run a specific step with its given parameterset.
- property active¶
Check active state
- property benchmark¶
Return benchmark of this workpackage
- property children¶
Return list of child workpackages
- property children_future¶
Create a list of all children in the future of this workpackage
Create shared folder connection
- property cycle¶
Return current loop cycle
- property done¶
Workpackage done?
- property env¶
Return workpackage environment
- property error¶
Workpackage error?
- property id¶
Return workpackage id
- id_counter = 0¶
- property iteration¶
Return workpackage iteration number
- property iteration_siblings¶
Return set of iteration siblings
- property local_parameterset¶
Return local parameterset
- operation_done_but_pending(operation_number)[source]¶
Check if an operation was executed, but the result is still pending (because it is a async do)
- property parameter_dict¶
get all available parameter inside a dict
- property parameterset¶
Return parameterset
- property parent_history¶
Create a list of all parents in the history of this workpackage
- property parents¶
Return list of parent workpackages
- property queued¶
Workpackage queued?
- run(mode='s')[source]¶
Run step and use current parameter space mode: s = seriell (default); p = parallel
- property started¶
Workpackage started?
- property step¶
Return Step data
- property work_dir¶
Return working directory (user space)
- property workpackage_dir¶
Return workpackage directory