jube2 package¶
jube2 package
Subpackages¶
Submodules¶
jube2.analyser module¶
The Analyser class handles the analyse process
-
class
jube2.analyser.
Analyser
(name, reduce_iteration=True)[source]¶ Bases:
object
The Analyser handles the analyse process and store all important data to run a new analyse.
-
class
AnalyseFile
(path)[source]¶ Bases:
object
A file which should be analysed
-
property
path
¶ Get file path
-
property
use
¶ Return uses
-
property
-
analyse_etree_repr
()[source]¶ Create an etree representation of a analyse dict: stepname -> workpackage_id -> filename -> patternname -> value
-
property
analyse_result
¶ Return analyse result
-
property
analyser
¶ Return analyse dict
-
property
benchmark
¶ Get benchmark information
-
property
name
¶ Get analyser name
-
property
use
¶ Return uses
-
class
jube2.benchmark module¶
The Benchmark class manages the benchmark process
-
class
jube2.benchmark.
Benchmark
(name, outpath, parametersets, substitutesets, filesets, patternsets, steps, analyser, results, results_order, comment='', tags=None, file_path_ref='.')[source]¶ Bases:
object
The Benchmark class contains all data to run a benchmark
-
property
analyser
¶ Return analyser
-
property
bench_dir
¶ Return benchmark directory
-
property
benchmark_status
¶ Retun global workpackage information dict
-
property
comment
¶ Return comment string
-
property
file_path_ref
¶ Get file path reference
-
property
filesets
¶ Return filesets
-
property
id
¶ Return benchmark id
-
property
name
¶ Return benchmark name
-
property
outpath
¶ Return outpath
-
property
parametersets
¶ Return parametersets
-
property
patternsets
¶ Return patternsets
-
property
results
¶ Return results
-
property
results_order
¶ Return results_order
-
property
steps
¶ Return steps
-
property
substitutesets
¶ Return substitutesets
Return set of tags
-
update_analyse_and_result
(new_patternsets, new_analyser, new_results, new_results_order, new_cwd)[source]¶ Update analyser and result data
-
property
work_stat
¶ Return work queue
-
property
workpackage_status
¶ Retun workpackage information dict
-
property
workpackages
¶ Return workpackages
-
wp_post_run_config
(workpackage)[source]¶ additional processing of workpackage: - update status bar - build up queue after restart
-
write_analyse_data
(filename)[source]¶ All analyse data will be written to given file using xml representation
-
property
jube2.completion module¶
Shell Completions
jube2.conf module¶
Configuration
jube2.fileset module¶
Fileset related classes
-
class
jube2.fileset.
Copy
(path, name=None, is_internal_ref=False, active='true', source_dir='', target_dir='')[source]¶ Bases:
jube2.fileset.File
A file or directory given by path. Which can be copied to the work_dir inside steps.
-
class
jube2.fileset.
File
(path, name=None, is_internal_ref=False, active='true', source_dir='', target_dir='')[source]¶ Bases:
object
Generic file access
-
create
(work_dir, parameter_dict, alt_work_dir=None, file_path_ref='', environment=None)[source]¶ Create file access
-
property
file_path_ref
¶ Get file path reference
-
property
is_internal_ref
¶ Return path is internal ref
-
property
path
¶ Return filepath
-
-
class
jube2.fileset.
Fileset
(name)[source]¶ Bases:
list
Container for file copy, link and prepare operations
-
create
(work_dir, parameter_dict, alt_work_dir=None, environment=None, file_path_ref='')[source]¶ Copy/load/prepare all files in fileset
-
property
name
¶ Return fileset name
-
-
class
jube2.fileset.
Link
(path, name=None, is_internal_ref=False, active='true', source_dir='', target_dir='')[source]¶ Bases:
jube2.fileset.File
A link to a given path. Which can be used inside steps.
-
class
jube2.fileset.
Prepare
(cmd, stdout_filename=None, stderr_filename=None, work_dir=None, active='true')[source]¶ Bases:
jube2.step.Operation
Prepare the workpackage work directory
jube2.help module¶
User help
jube2.info module¶
Gives benchmark related info
-
jube2.info.
print_benchmark_status
(benchmark)[source]¶ Print overall workpackage status in the following order RUNNING: At least one WP is still active ERROR: At least one WP raised an errror FINISHED: All WPs are finalized and no error was raised
jube2.jubeio module¶
Basic I/O module
-
class
jube2.jubeio.
Parser
(filename, tags=None, include_path=None, force=False, strict=False)[source]¶ Bases:
object
JUBE XML input file parser
-
benchmark_info_from_xml
()[source]¶ Return name, comment and available tags of first benchmark found in file
-
benchmarks_from_xml
()[source]¶ Return a dict of benchmarks
Here parametersets are global and accessible to all benchmarks defined in the corresponding XML file.
-
property
file_path_ref
¶ Return file path given by config file
-
jube2.log module¶
Logging Support
-
class
jube2.log.
JubeLogger
(name, level=0)[source]¶ Bases:
logging.Logger
,object
Overwrite logging to handle multi line messages.
-
jube2.log.
matching_logs
(commands, available_logs)[source]¶ Find intersection between requested logs and available logs.
Returns tuple (matching, not_matching), containing the intersection and its complement.
Only compares basenames.
-
jube2.log.
safe_output_logfile
(filename)[source]¶ Try to print logfile. If try fails, fail gracefully.
-
jube2.log.
setup_logging
(mode=None, filename=None, verbose=None)[source]¶ Setup the logging configuration.
- Available modes are
default log to console and file console only console output
filename can be given optionally.
verbose: enable verbose console output
The setup includes setting the handlers and formatters. Calling this function multiple times causes old handlers to be removed before new ones are added.
jube2.main module¶
CLI program
jube2.parameter module¶
Parameter related classes
-
class
jube2.parameter.
FixedParameter
(name, value, separator=None, parameter_type='string', parameter_mode='text', export=False, update_mode='never', idx=- 1, eval_helper=None, duplicate='none')[source]¶ Bases:
jube2.parameter.StaticParameter
A FixedParameter is a parameter which can not be evaluated anymore. It represents a fixed value.
-
class
jube2.parameter.
Parameter
(name, value, separator=None, parameter_type='string', parameter_mode='text', export=False, update_mode='never', idx=- 1, eval_helper=None, duplicate='none')[source]¶ Bases:
object
Contains data for single Parameter. This Parameter can be a constant value, a template or a specific value out of a given template
-
property
based_on
¶ The base of the current Parameter
-
property
based_on_mode
¶ Return the root parameter mode inside the based_on graph
-
property
based_on_root
¶ Return the root parameter inside the based_on graph
-
property
based_on_value
¶ Return the root value inside the based_on graph
-
static
create_parameter
(name, value, separator=None, parameter_type='string', selected_value=None, parameter_mode='text', export=False, no_templates=False, update_mode='never', idx=- 1, eval_helper=None, fixed=False, duplicate='none')[source]¶ Parameter constructor. Return a Static- or TemplateParameter based on the given data.
-
property
duplicate
¶ Return duplicate option
-
property
eval_helper
¶ Return evaluation helper function
-
property
export
¶ Return if parameter should be exported
-
property
idx
¶ Return template idx
-
is_equivalent
(parameter, first_check=True)[source]¶ Checks whether the given and the current Parameter based on equivalent templates and (if expanded) contain the same value and were on the same place within the original template
-
property
is_fixed
¶ Return whether the parameter is fixed
-
property
is_jube_parameter
¶ Parameter is handled by JUBE automatically
-
property
is_template
¶ Return whether the parameter is a template
-
property
mode
¶ Return parameter mode
-
property
name
¶ Returns the Parameter name
-
parameter_regex
= re.compile('(?<!\\$)(?:\\$\\$)*\\$(?!\\$)(\\{)?(\\w+?)(?(1)\\}|(?=\\W|$))')¶
-
property
parameter_type
¶ Return parametertype
-
search_method
(propertyString, recursiveProperty=None)[source]¶ Searches, potentially recursively, for a method and returns True in case of a success
-
property
update_mode
¶ Returns the update mode of the parameter
-
property
value
¶ Return parameter value
-
property
-
class
jube2.parameter.
Parameterset
(name='', duplicate='replace')[source]¶ Bases:
object
A parameterset represent a template or a specific product space. It can be combined with other Parametersets.
-
add_parameterset
(parameterset)[source]¶ Add all parameters from given parameterset, existing ones will be overwritten
-
property
all_parameter_names
¶ Return list of all parameter names
-
property
all_parameters
¶ Return list of all parameters
-
check_parameter_options
(parameter, only_duplicate=True)[source]¶ Check whether both parameters have identical options and throw an error if this is not the case
-
property
constant_parameter_dict
¶ Return dictionary representation of all constant parameters
-
property
duplicate
¶ Return the duplicate property of the Parameterset
-
expand_templates
()[source]¶ Expand all remaining templates in the Parameterset and returns the resulting parametersets
-
property
export_parameter_dict
¶ Return dictionary representation of all export parameters
-
get_incompatible_parameter
(parameterset, update_mode='never')[source]¶ Return a set of incompatible parameter names between the current and the given parameterset
-
get_updatable_parameter
(mode, keep_index=False)[source]¶ Returns a parameterset containing all updatable parameter for a specific mode, the root parameter is added
-
property
has_templates
¶ This Parameterset contains template paramters?
-
is_compatible
(parameterset, update_mode='never')[source]¶ Two Parametersets are compatible, if the intersection only contains equivilant parameters
-
property
name
¶ Return name of the Parameterset
-
property
parameter_dict
¶ Return dictionary name -> parameter
-
parameter_substitution
(additional_parametersets=None, final_sub=False)[source]¶ Substitute all parameter inside the parameterset. Parameters from additional_parameterset will be used for substitution but will not be added to the set. final_sub marks the last substitution process.
-
property
template_parameter_dict
¶ Return dictionary representation of all template parameters
-
-
class
jube2.parameter.
StaticParameter
(name, value, separator=None, parameter_type='string', parameter_mode='text', export=False, update_mode='never', idx=- 1, eval_helper=None, duplicate='none')[source]¶ Bases:
jube2.parameter.Parameter
A StaticParameter can be substituted and evaluated.
-
can_substitute_and_evaluate
(parameterset)[source]¶ A parameter can be substituted and evaluated if there are no depending templates or unevaluated parameter inside
-
substitute_and_evaluate
(parametersets=None, final_sub=False, no_templates=False, force_evaluation=False)[source]¶ Substitute all variables inside the parameter value by using the parameters inside the given parameterset. final_sub marks the last substitution.
Return the new parameter and a boolean value which represent a change of value
-
-
class
jube2.parameter.
TemplateParameter
(name, value, separator=None, parameter_type='string', parameter_mode='text', export=False, update_mode='never', idx=- 1, eval_helper=None, duplicate='none')[source]¶ Bases:
jube2.parameter.Parameter
A TemplateParameter represent a set of possible parameter values, which can be accessed by a single name. To use the template in a specific environment, it must be expanded.
-
property
value
¶ Return Template values
-
property
jube2.pattern module¶
Patternset definition
-
class
jube2.pattern.
Pattern
(name, value, pattern_mode='pattern', content_type='string', unit='', default=None, dotall=False)[source]¶ Bases:
jube2.parameter.StaticParameter
A pattern can be used to scan a result file, using regular expression, or to represent a derived pattern.
-
property
content_type
¶ Return pattern type
-
property
default_value
¶ Return pattern default value
-
property
derived
¶ pattern is a derived pattern
-
property
dotall
¶ Return pattern dot regex handling
-
substitute_and_evaluate
(parametersets=None, final_sub=False, no_templates=True, force_evaluation=False)[source]¶ Substitute all variables inside the pattern value by using the parameter inside the given parameterset and additional_parameterset. final_sub marks the last substitution.
Return the new pattern and a boolean value which represent a change of value
-
property
unit
¶ Return unit
-
property
-
class
jube2.pattern.
Patternset
(name='')[source]¶ Bases:
object
A Patternset stores a set of pattern and derived pattern.
-
add_pattern
(pattern)[source]¶ Add a additional pattern to the patternset. Existing pattern using the same name will be overwritten
-
property
derived_pattern_storage
¶ Return the derived pattern storage
-
derived_pattern_substitution
(parametersets=None)[source]¶ Run derived pattern substitution using additional parameterset
-
get_incompatible_pattern
(patternset)[source]¶ Return a set of incompatible pattern names between the current and the given parameterset
-
is_compatible
(patternset)[source]¶ Two Patternsets are compatible, if all pattern storages are compatible
-
property
name
¶ Get patternset name
-
property
pattern_storage
¶ Return the pattern storage
-
jube2.result module¶
Resulttype definition
-
class
jube2.result.
Result
(name, res_filter=None)[source]¶ Bases:
object
A generic result type
-
class
ResultData
(name)[source]¶ Bases:
object
A gerneric result data type
-
property
name
¶ Return the result name
-
property
-
property
benchmark
¶ Return the benchmark
-
property
name
¶ Return the result name
-
property
result_dir
¶ Return the result_dir
-
class
jube2.step module¶
Step contains the commands for steps
-
class
jube2.step.
DoLog
(log_dir, log_file, initial_env, cycle=0)[source]¶ Bases:
object
A DoLog class containing the operations and information for setting up the do log.
-
property
initial_env
¶ Get initial env
-
property
log_file
¶ Get log file
-
property
log_path
¶ Get log path
-
store_do
(do, shell, work_dir, parameter_dict=None, shared=False)[source]¶ Store the current execution directive to the do log and set up the environment if file does not yet exist.
-
property
work_dir
¶ Get last work directory
-
property
-
class
jube2.step.
Operation
(do, async_filename=None, stdout_filename=None, stderr_filename=None, active='true', shared=False, work_dir=None, break_filename=None, error_filename=None)[source]¶ Bases:
object
The Operation-class represents a single instruction, which will be executed in a shell environment.
-
active
(parameter_dict)[source]¶ Return active status of the current operation depending on the given parameter_dict
-
property
async_filename
¶ Get async filename
-
property
error_filename
¶ Get error filename
-
execute
(parameter_dict, work_dir, only_check_pending=False, environment=None, pid=None, dolog=None)[source]¶ Execute the operation. work_dir must be set to the given context path. The parameter_dict used for inline substitution. If only_check_pending is set to True, the operation will not be executed, only the async_file will be checked. Return operation status: True => operation finished False => operation pending
-
static
read_process_environment
(work_dir, remove_after_read=True, pid=None)[source]¶ Read standard environment info file in given directory.
Shared operation?
-
property
stderr_filename
¶ Get stderr filename
-
property
stdout_filename
¶ Get stdout filename
-
-
class
jube2.step.
Step
(name, depend, iterations=1, alt_work_dir=None, shared_name=None, export=False, max_wps='0', active='true', suffix='', cycles=1, procs=1, do_log_file=None)[source]¶ Bases:
object
A Step represent one execution step. It contains a list of Do-operations and multiple parametersets, substitutionsets and filesets. A Step is a template for Workpackages.
-
property
active
¶ Return active state
-
property
alt_work_dir
¶ Return alternativ work directory
-
create_workpackages
(benchmark, global_parameterset, local_parameterset=None, used_sets=None, iteration_base=0, parents=None, incompatible_parameters=None)[source]¶ Create workpackages for current step using given benchmark context
-
property
cycles
¶ Return number of cycles
-
property
depend
¶ Return dependencies
-
property
do_log_file
¶ Return do log file name
-
property
export
¶ Return export behaviour
-
get_depend_history
(benchmark)[source]¶ Creates a set of all dependent steps in history for given benchmark
-
get_used_sets
(available_sets, parameter_dict=None)[source]¶ Get list of all used sets, which can be found in available_sets
-
property
iterations
¶ Return iterations
-
property
max_wps
¶ Return maximum number of simultaneous workpackages
-
property
name
¶ Return step name
-
property
operations
¶ Return operations
-
property
procs
¶ Return number of procs
Return shared folder name
Return shared link name
-
property
suffix
¶ Return directory suffix
-
property
use
¶ Return parameters and substitutions
-
property
jube2.substitute module¶
Substitution related classes
-
class
jube2.substitute.
Substituteset
(name, file_data, substitute_dict)[source]¶ Bases:
object
A Substituteset contains all information
-
property
name
¶ Return name of Substituteset
-
property
jube2.workpackage module¶
The Workpackage class handles a step and its parameter space
-
class
jube2.workpackage.
Workpackage
(benchmark, step, local_parameter_names, parameterset, workpackage_id=None, iteration=0, cycle=0)[source]¶ Bases:
object
A Workpackage contains all information to run a specific step with its given parameterset.
-
property
active
¶ Check active state
-
property
benchmark
¶ Return benchmark of this workpackage
-
property
children
¶ Return list of child workpackages
-
property
children_future
¶ Create a list of all children in the future of this workpackage
Create shared folder connection
-
property
cycle
¶ Return current loop cycle
-
property
done
¶ Workpackage done?
-
property
env
¶ Return workpackage environment
-
property
error
¶ Workpackage error?
-
property
id
¶ Return workpackage id
-
id_counter
= 0¶
-
property
iteration
¶ Return workpackage iteration number
-
property
iteration_siblings
¶ Return set of iteration siblings
-
property
local_parameterset
¶ Return local parameterset
-
operation_done_but_pending
(operation_number)[source]¶ Check if an operation was executed, but the result is still pending (because it is a async do)
-
property
parameter_dict
¶ get all available parameter inside a dict
-
property
parameterset
¶ Return parameterset
-
property
parent_history
¶ Create a list of all parents in the history of this workpackage
-
property
parents
¶ Return list of parent workpackages
-
property
queued
¶ Workpackage queued?
-
run
(mode='s')[source]¶ Run step and use current parameter space mode: s = seriell (default); p = parallel
-
property
started
¶ Workpackage started?
-
property
step
¶ Return Step data
-
property
work_dir
¶ Return working directory (user space)
-
property
workpackage_dir
¶ Return workpackage directory
-
property