Advanced tutorial¶
This tutorial demonstrates more detailed functions and tools of JUBE. If you want a basic overview you should read the general JUBE tutorial first.
Schema validation¶
To validate your XML based input files you can use DTD or schema validation. You will find jube.dtd
, jube.xsd
and jube.rnc
inside the schema
folder. You have to add these schema information to your input files
which you want to validate.
DTD usage:
1 2 3 4 | <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE jube SYSTEM "<jube.dtd path>"> <jube> ... |
Schema usage:
1 2 3 4 | <?xml version="1.0" encoding="UTF-8"?> <jube xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="<jube.xsd path>"> ... |
RELAX NG Compact Syntax (RNC for emacs nxml-mode) usage:
In order to use the provided rnc schema file schema/jube.rnc
in
emacs open an xml file and use C-c C-s C-f
or M-x
rng-set-schema-file-and-validate
to choose the rnc file. You can
also use M-x customize-variable rng-schema-locating-files
after
you loaded nxml-mode to customize the default search paths to include
jube.rnc
. After successful parsing emacs offers to automatically
create a schema.xml
file which looks like
1 2 3 4 | <?xml version="1.0"?> <locatingRules xmlns="http://thaiopensource.com/ns/locating-rules/1.0"> <uri resource="jube-file.xml" uri="../schema/jube.rnc"/> </locatingRules> |
The next time you open the same xml file emacs will find the correct
rnc for the validation based on schema.xml
.
Example validation tools:
eclipse (using DTD or schema)
emacs (using RELAX NG)
xmllint:
For validation (using the DTD):
>>> xmllint --noout --valid <xml input file>
For validation (using the DTD and Schema):
>>> xmllint --noout --valid --schema <schema file> <xml input file>
Scripting parameter¶
In some cases it is needed to create a parameter which is based on the value of another parameter. In this case you can use a scripting parameter.
The files used for this example can be found inside examples/scripting_parameter
.
The input file scripting_parameter.xml
:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 | <?xml version="1.0" encoding="UTF-8"?> <jube> <benchmark name="scripting_parameter" outpath="bench_run"> <comment>A scripting parameter example</comment> <!-- Configuration --> <parameterset name="param_set"> <!-- Normal template --> <parameter name="number" type="int">1,2,4</parameter> <!-- A template created by a scripting parameter--> <parameter name="additional_number" mode="python" type="int"> ",".join(str(a*${number}) for a in [1,2]) </parameter> <!-- A scripting parameter --> <parameter name="number_mult" mode="python" type="float"> ${number}*${additional_number} </parameter> <!-- Reuse another parameter --> <parameter name="text">Number: $number</parameter> </parameterset> <!-- Operation --> <step name="operation"> <use>param_set</use> <!-- use existing parameterset --> <!-- shell commands --> <do>echo "number: $number, additional_number: $additional_number"</do> <do>echo "number_mult: $number_mult, text: $text"</do> </step> </benchmark> </jube> |
The input file scripting_parameter.yaml
:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 | name: scripting_parameter outpath: bench_run comment: A scripting parameter example #Configuration parameterset: name: param_set parameter: #Normal template - {name: number, type: int, _: "1,2,4"} #A template created by a scripting parameter - {name: additional_number, mode: python, type: int, _: '",".join(str(a*${number}) for a in [1,2])'} #A scripting parameter - {name: number_mult, mode: python, type: float, _: "${number}*${additional_number}"} #Reuse another parameter - {name: text, _: "Number: $number"} #Operation step: name: operation use: param_set #use existing parameterset do: - 'echo "number: $number, additional_number: $additional_number"' - 'echo "number_mult: $number_mult, text: $text"' |
In this example we see four different parameters.
number
is a normal template which will be expanded to three different workpackages.additional_number
is a scripting parameter which creates a new template and bases onnumber
. Themode
is set to the scripting language (python
,perl
andshell
are allowed). The additionaltype
is optional and declares the result type after evaluating the expression. The type is only used by the sort algorithm in the result step. It is not possible to create a template of different scripting parameters. Because of this second template we will get six different workpackages.number_mult
is a small calculation. You can use any other existing parameters (which are used inside the same step).text
is a normal parameter which uses the content of another parameter. For a simple concatenation parameter you do not need a scripting parameter.
For this example we will find the following output inside the run.log
-file:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | ====== operation ====== >>> echo "number: 1, additional_number: 1" >>> echo "number_mult: 1, text: Number: 1" ====== operation ====== >>> echo "number: 1, additional_number: 2" >>> echo "number_mult: 2, text: Number: 1" ====== operation ====== >>> echo "number: 2, additional_number: 2" >>> echo "number_mult: 4, text: Number: 2" ====== operation ====== >>> echo "number: 2, additional_number: 4" >>> echo "number_mult: 8, text: Number: 2" ====== operation ====== >>> echo "number: 4, additional_number: 4" >>> echo "number_mult: 16, text: Number: 4" ====== operation ====== >>> echo "number: 4, additional_number: 8" >>> echo "number_mult: 32, text: Number: 4" |
Implicit Perl or Python scripting inside the <do>
or any other position is not possible.
If you want to use some scripting expressions you have to create a new parameter.
Scripting pattern¶
Similar to the Scripting parameter, also different patterns, or patterns and parameters can be combined.
For this a scripting pattern can be created by using the mode=
attribute in the same way as it is used for the Scripting parameter.
All scripting patterns are evaluated at the end of the analyse part. Each scripting pattern is evaluated once. If there are multiple matches as described in the Statistic pattern values section, only the resulting statistical pattern is available (not each individual value). Scripting pattern do not create statistic values by themselves.
In addition the default=
attribute can be used to set a default pattern value, if the value can’t be found during the analysis.
The files used for this example can be found inside examples/scripting_pattern
.
The input file scripting_pattern.xml
:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 | <?xml version="1.0" encoding="UTF-8"?> <jube> <benchmark name="scripting_pattern" outpath="bench_run"> <comment>A scripting_pattern example</comment> <!-- Configuration --> <parameterset name="param_set"> <parameter name="value" type="int">0,1,2</parameter> </parameterset> <!-- Operation --> <step name="operation"> <use>param_set</use> <do>echo "$value"</do> </step> <!-- Pattern to extract --> <patternset name="pattern_set"> <!-- A normal pattern --> <pattern name="value_pat" type="int">$jube_pat_int</pattern> <!-- A combination of a pattern and a parameter --> <pattern name="dep_pat" type="int" mode="python">$value_pat+$value</pattern> <!-- This pattern is not available --> <pattern name="missing_pat" type="int"> pattern_not_available: $jube_pat_int </pattern> <!-- The combination will fail (create NaN) --> <pattern name="missing_dep_pat" type="int" mode="python"> $missing_pat*$value </pattern> <!-- Default value for missing pattern --> <pattern name="missing_pat_def" type="int" default="0"> pattern_not_available: $jube_pat_int </pattern> <!-- Combination of default value and parameter --> <pattern name="missing_def_dep_pat" type="int" mode="python"> $missing_pat_def*$value </pattern> </patternset> <analyser name="analyse"> <use>pattern_set</use> <analyse step="operation"> <file>stdout</file> </analyse> </analyser> <!-- result table creation --> <result> <use>analyse</use> <table name="result" style="pretty"> <column>value</column> <column>value_pat</column> <column>dep_pat</column> <column>missing_pat</column> <column>missing_dep_pat</column> <column>missing_pat_def</column> <column>missing_def_dep_pat</column> </table> </result> </benchmark> </jube> |
The input file scripting_pattern.yaml
:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 | name: scripting_pattern outpath: bench_run comment: A scripting_pattern example #Configuration parameterset: name: param_set parameter: {name: value, type: int, _: "0,1,2"} #Operation step: name: operation use: param_set do: echo "$value" #Pattern to extract patternset: name: pattern_set pattern: #A normal pattern - {name: value_pat, type: int, _: $jube_pat_int} #A combination of a pattern and a parameter - {name: dep_pat, type: int, mode: python, _: $value_pat+$value} #This pattern is not available - {name: missing_pat, type: int, _: "pattern_not_available: $jube_pat_int"} #The combination will fail (create NaN) - {name: missing_dep_pat, type: int, mode: python, _: $missing_pat*$value} #Default value for missing pattern - {name: missing_pat_def, type: int, default: 0, _: "pattern_not_available: $jube_pat_int"} #Combination of default value and parameter - {name: missing_def_dep_pat, type: int, mode: python, _: $missing_pat_def*$value} analyser: name: analyse use: pattern_set analyse: step: operation file: stdout #result table creation result: use: analyse table: name: result style: pretty column: [value,value_pat,dep_pat,missing_pat,missing_dep_pat,missing_pat_def,missing_def_dep_pat] |
It will create the following output:
1 2 3 4 5 | | value | value_pat | dep_pat | missing_pat | missing_dep_pat | missing_pat_def | missing_def_dep_pat | |-------|-----------|---------|-------------|-----------------|-----------------|---------------------| | 0 | 0 | 0 | | nan | 0 | 0 | | 1 | 1 | 2 | | nan | 0 | 0 | | 2 | 2 | 4 | | nan | 0 | 0 | |
Statistic pattern values¶
Normally a pattern should only match a single entry in your result files. But sometimes there are multiple similar entries (e.g. if the benchmark uses some iteration feature).
JUBE will create the statistical values last
, min
, max
, avg
, std
, cnt
and sum
automatically.
To use these values, the user have to specify the pattern name followed by _<statistic_option>
,
e.g. pattern_name_last
(the pattern_name itself will always be the first match).
An example for multiple matches and the statistic values can be found in examples/statistic
.
The input file statistic.xml
:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 | <?xml version="1.0" encoding="UTF-8"?> <jube> <benchmark name="reduce_example" outpath="bench_run"> <comment>A result reduce example</comment> <!-- Regex pattern --> <patternset name="pattern"> <pattern name="number_pat" type="int">$jube_pat_int</pattern> </patternset> <!-- Operation --> <step name="write_some_numbers"> <do>echo "1 2 3 4 5 6 7 8 9 10"</do> <!-- shell command --> </step> <!-- Analyse --> <analyser name="analyse"> <use>pattern</use> <!-- use existing patternset --> <analyse step="write_some_numbers"> <file>stdout</file> <!-- file which should be scanned --> </analyse> </analyser> <!-- Create result table --> <result> <use>analyse</use> <!-- use existing analyser --> <table name="result" style="pretty"> <column>number_pat</column> <!-- first match --> <column>number_pat_first</column> <!-- first match --> <column>number_pat_last</column> <!-- last match --> <column>number_pat_min</column> <!-- min of all matches --> <column>number_pat_max</column> <!-- max of all matches --> <column>number_pat_sum</column> <!-- sum of all matches --> <column>number_pat_cnt</column> <!-- number of matches --> <column>number_pat_avg</column> <!-- avg of all matches --> <column format=".2f">number_pat_std</column> <!-- std of all matches --> </table> </result> </benchmark> </jube> |
The input file statistic.yaml
:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 | name: reduce_example outpath: bench_run comment: A result reduce example #Regex pattern patternset: name: pattern pattern: {name: number_pat, type: int, _: $jube_pat_int} #Operation step: name: write_some_numbers do: echo "1 2 3 4 5 6 7 8 9 10" #shell command #Analyse analyser: name: analyse use: pattern #use existing patternset analyse: step: write_some_numbers file: stdout #file which should be scanned #Create result table result: use: analyse #use existing analyser table: name: result style: pretty column: - number_pat #first match - number_pat_first #first match - number_pat_last #last match - number_pat_min #min of all matches - number_pat_max #max of all matches - number_pat_sum #sum of all matches - number_pat_cnt #number of matches - number_pat_avg #avg of all matches - {_: number_pat_std, format: .2f} #std of all matches |
It will create the following output:
| number_pat | number_pat_last | number_pat_min | number_pat_max | number_pat_sum | number_pat_cnt | number_pat_avg | number_pat_std |
|------------|-----------------|----------------|----------------|----------------|----------------|----------------|----------------|
| 1 | 10 | 1 | 10 | 55 | 10 | 5.5 | 3.03 |
Jobsystem¶
In most cases you want to submit jobs by JUBE to your local jobsystem. You can use the normal file access and substitution system to prepare your jobfile and send it to the jobsystem. JUBE also provide some additional features.
The files used for this example can be found inside examples/jobsystem
.
The input jobsystem file job.run.in
for Torque/Moab (you can easily adapt your personal jobscript):
1 2 3 4 5 6 7 8 9 10 11 12 | #!/bin/bash -x #MSUB -l nodes=#NODES#:ppn=#PROCS_PER_NODE# #MSUB -l walltime=#WALLTIME# #MSUB -e #ERROR_FILEPATH# #MSUB -o #OUT_FILEPATH# #MSUB -M #MAIL_ADDRESS# #MSUB -m #MAIL_MODE# ### start of jobscript #EXEC# touch #READY# |
The JUBE input file jobsystem.xml
:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 | <?xml version="1.0" encoding="UTF-8"?> <jube> <benchmark name="jobsystem" outpath="bench_run"> <comment>A jobsystem example</comment> <!-- benchmark configuration --> <parameterset name="param_set"> <parameter name="number" type="int">1,2,4</parameter> </parameterset> <!-- Job configuration --> <parameterset name="executeset"> <parameter name="submit_cmd">msub</parameter> <parameter name="job_file">job.run</parameter> <parameter name="nodes" type="int">1</parameter> <parameter name="walltime">00:01:00</parameter> <parameter name="ppn" type="int">4</parameter> <parameter name="ready_file">ready</parameter> <parameter name="mail_mode">abe</parameter> <parameter name="mail_address"></parameter> <parameter name="err_file">stderr</parameter> <parameter name="out_file">stdout</parameter> <parameter name="exec">echo $number</parameter> </parameterset> <!-- Load jobfile --> <fileset name="files"> <copy>${job_file}.in</copy> </fileset> <!-- Substitute jobfile --> <substituteset name="sub_job"> <iofile in="${job_file}.in" out="$job_file" /> <sub source="#NODES#" dest="$nodes" /> <sub source="#PROCS_PER_NODE#" dest="$ppn" /> <sub source="#WALLTIME#" dest="$walltime" /> <sub source="#ERROR_FILEPATH#" dest="$err_file" /> <sub source="#OUT_FILEPATH#" dest="$out_file" /> <sub source="#MAIL_ADDRESS#" dest="$mail_address" /> <sub source="#MAIL_MODE#" dest="$mail_mode" /> <sub source="#EXEC#" dest="$exec" /> <sub source="#READY#" dest="$ready_file" /> </substituteset> <!-- Operation --> <step name="submit" work_dir="$$SCRATCH/jobsystem_bench_${jube_benchmark_id}_${jube_wp_id}" > <use>param_set</use> <use>executeset</use> <use>files,sub_job</use> <do done_file="$ready_file">$submit_cmd $job_file</do> <!-- shell command --> </step> </benchmark> </jube> |
The JUBE input file jobsystem.yaml
:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 | name: jobsystem outpath: bench_run comment: A jobsystem example parameterset: #benchmark configuration - name: param_set parameter: {name: number, type: int, _: "1,2,4"} #comma separated integer must be quoted #Job configuration - name: executeset parameter: - {name: submit_cmd, "_": msub} - {name: job_file, "_": job.run} - {name: nodes, type: int, "_": 1} - {name: walltime, "_": "00:01:00"} #: must be quoted - {name: ppn, type: int, "_": 4} - {name: ready_file, "_": ready} - {name: mail_mode, "_": abe} - {name: mail_address} - {name: err_file, "_": stderr} - {name: out_file, "_": stdout} - {name: exec, "_": echo $number} #Load jobfile fileset: name: files copy: ${job_file}.in substituteset: name: sub_job iofile: {in: "${job_file}.in", out: $job_file} #attributes with {} must be quoted sub: - {source: "#NODES#", dest: $nodes} - {source: "#PROCS_PER_NODE#", dest: $ppn} - {source: "#WALLTIME#", dest: $walltime} - {source: "#ERROR_FILEPATH#", dest: $err_file} - {source: "#OUT_FILEPATH#", dest: $out_file} - {source: "#MAIL_ADDRESS#", dest: $mail_address} - {source: "#MAIL_MODE#", dest: $mail_mode} - {source: "#EXEC#", dest: $exec} - {source: "#READY#", _: $ready_file } # _ can be used here as well instead of dest (should be used for multiline output) #Operation step: name: submit work_dir: "$$WORK/jobsystem_bench_${jube_benchmark_id}_${jube_wp_id}" use: [param_set,executeset,files,sub_job] do: done_file: $ready_file _: $submit_cmd $job_file #shell command |
As you can see the jobfile is very general and several parameters will be used for replacement. By using a general jobfile and the substitution mechanism
you can control your jobsystem directly out of your JUBE input file.
$$
is used for Shell substitutions instead of JUBE substitution (see Environment handling).
The submit command is a normal Shell command so there are no special JUBE tags to submit a job.
There are two new attributes:
done_file
inside the<do>
allows you to set a filename/path to a file which should be used by the jobfile to mark the end of execution. JUBE does not know when the job ends. Normally it will return when the Shell command was finished. When using a jobsystem the user usually have to wait until the jobfile is executed. If JUBE found a<do>
containing adone_file
attribute JUBE will return directly and will not continue automatically until thedone_file
exists. If you want to check the current status of your running steps and continue the benchmark process if possible you can type:>>> jube continue bench_run
This will continue your benchmark execution (
bench_run
is the benchmarks directory in this example). The position of thedone_file
is relatively seen towards the work directory.
work_dir
can be used to change the sandbox work directory of a step. In normal cases JUBE checks that every work directory gets a unique name. When changing the directory the user must select a unique name by his own. For example he can use$jube_benchmark_id
and$jube_wp_id
, which are JUBE internal parameters and will be expanded to the current benchmark and workpackage ids. Files and directories out of a given<fileset>
will be copied into the new work directory. Other automatic links, like the dependency links, will not be created!
You will see this Output after running the benchmark:
| stepname | all | open | wait | error | done |
|----------|-----|------|------|-------|------|
| submit | 3 | 0 | 3 | 0 | 0 |
and this output after running the continue
command (after the jobs where executed):
| stepname | all | open | wait | error | done |
|----------|-----|------|------|-------|------|
| submit | 3 | 0 | 0 | 0 | 3 |
You have to run continue
multiple times if not all done_file
were written when running continue
for the first time.
Include external data¶
As you have seen in the example before a benchmark can become very long. To structure your benchmark you can use multiple files and reuse existing sets. There are three different include features available.
The files used for this example can be found inside examples/include
.
The include file include_data.xml
:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | <?xml version="1.0" encoding="UTF-8"?> <jube> <parameterset name="param_set"> <parameter name="number" type="int">1,2,4</parameter> </parameterset> <parameterset name="param_set2"> <parameter name="text">Hello</parameter> </parameterset> <dos> <do>echo Test</do> <do>echo $number</do> </dos> </jube> |
The include file include_data.yaml
:
1 2 3 4 5 6 7 8 9 | parameterset: - name: param_set parameter: {name: number, type: int, _: "1,2,4"} - name: param_set2 parameter: {name: text, _: Hello} dos: - echo Test - echo $number |
All files which contain data to be included must use the XML-format. The include files can have a user specific structure (there can be no valid
JUBE tags like <dos>
), but the structure must be allowed by the searching mechanism (see below). The resulting file must have a valid JUBE structure.
The main file main.xml
:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | <?xml version="1.0" encoding="UTF-8"?> <jube> <benchmark name="include" outpath="bench_run"> <comment>A include example</comment> <!-- use parameterset out of an external file and add a additional parameter --> <parameterset name="param_set" init_with="include_data.xml"> <parameter name="foo">bar</parameter> </parameterset> <!-- Operation --> <step name="say_hello"> <use>param_set</use> <!-- use existing parameterset --> <use from="include_data.xml">param_set2</use> <!-- out of an external file --> <do>echo $foo</do> <!-- shell command --> <include from="include_data.xml" path="dos/do" /> <!-- include all available tag --> </step> </benchmark> </jube> |
The main file main.yaml
:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 | name: include outpath: bench_run comment: A include example #use parameterset out of an external file and add a additional parameter parameterset: name: param_set init_with: include_data.yaml parameter: {name: foo, _: bar} #Operation step: name: say_hello use: - param_set #use existing parameterset - from: include_data.yaml _: param_set2 #out of an external file do: - echo $foo - !include include_data.yaml:["dos"] #include all available tag |
In these file there are three different include types:
The init_with
can be used inside any set definition. Inside the given file the search mechanism will search for the same set (same type, same name), will parse its structure (this must be JUBE valid) and copy the content to
main.xml
. Inside main.xml
you can add additional values or overwrite existing ones. If your include-set uses a different name inside your include file you can use init_with="filename.xml:old_name"
. It is possible to mix YAML based
include files with XML files and vice versa.
The second method is the <use from="...">
. This is mostly the same like the init_with
structure, but in this case you are not able to add or overwrite some values. The external set will be used directly. There is no set-type inside the <use>
, because of that, the set’s name must
be unique inside the include-file. The remote file can use the YAML or the XML format.
The last method is the most generic include. The include mechanic is the only element in JUBE which works slightly different in YAML and XML based files.
In XML based files by using <include />
you can copy any XML-nodes you want to your main-XML file. The included file can provide tags which are not JUBE-conform but it must be a valid XML-file (e.g. only one root node allowed). The
resulting main configuration file must be completely JUBE valid.
The path
is optional and can be used to select a specific node set (otherwise the root-node itself will be included). The <include />
is the only
include-method that can be used to include any tag you want. The <include />
will copy all parts without any changes. The other include types will update path names, which were relative to the include-file position.
In YAML based files the prefix ! include
is used followed by the file name. The file must be a YAML file, which will be opened and parsed. The second block :[“dos”] can be used to select any subset of data of the full dictionary, any Python syntax is allowed for this selection.
Finally it is possible to also specify a third block which allows full Python list comprehensions. _
is the match of the selection before, e.g.: !include include_data.yaml:["dos"]:[i for i in _ if "Test" in i]
. In contrast to the XML based include it isn’t possible to mix lists
or dictionaries out of different files, each key can only handle a single include.
To run the benchmark you can use the normal command:
>>> jube run main.xml
It will search for the files to include inside four different positions, in the following order:
inside a directory given over the command line interface:
>>> jube run --include-path some_path another_path -- main.xml
inside any path given by an
<include-path>
-tag:1 2 3 4 5 6 7
<?xml version="1.0" encoding="UTF-8"?> <benchmarks> <include-path> <path>some_path</path> <path>another_path</path> </include-path> ...
inside any path given with the
JUBE_INCLUDE_PATH
environment variable (see Configuration):>>> export JUBE_INCLUDE_PATH=some_path:another_path
inside the same directory of your
main.xml
JUBE stops searching as soon as it finds the file to include, or gives an error if the file is not found.
Tagging¶
Tagging is an easy way to hide selectable parts of your input file.
The files used for this example can be found inside examples/tagging
.
The input file tagging.xml
:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | <?xml version="1.0" encoding="UTF-8"?> <jube> <benchmark name="tagging" outpath="bench_run"> <comment>Tags as logical combination</comment> <!-- Configuration --> <parameterset name="param_set"> <parameter name="hello_str" tag="!deu+eng">Hello</parameter> <parameter name="hello_str" tag="deu|!eng">Hallo</parameter> <parameter name="world_str" tag="eng">World</parameter> </parameterset> <!-- Operation --> <step name="say_hello"> <use>param_set</use> <!-- use existing parameterset --> <do>echo '$hello_str $world_str'</do> <!-- shell command --> </step> </benchmark> </jube> |
The input file tagging.yaml
:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | name: tagging outpath: bench_run comment: Tags as logical combination #Configuration parameterset: name: param_set parameter: - {name: hello_str, tag: "!deu+eng", _: Hello} - {name: hello_str, tag: deu|!eng, _: Hallo} - {name: world_str, tag: eng, _: World} #Operation step: name: say_hello use: param_set #use existing parameterset do: echo '$hello_str $world_str' #shell command |
When running this example:
>>> jube run tagging.xml
All <tags>
which contain a special tag="..."
attribute will be hidden if the tag results to false
. !deu
stands for not deu
.
To connect the tags
|
can be used as the oprator OR and +
for the operator AND. Also brackets are allowed.
The result (if no tag
is set on the commandline) inside the stdout
file will be
Hallo $world_str
because !deu+eng
and eng
will be false
and there is no other input available for $world_str
. deu|!eng
will be true
.
When running the same example using a specific tag
:
>>> jube run tagging.xml --tag eng
the result inside the stdout
file will be
Hello World
A tag
which results to false
will trigger to complety ignore the corresponding <tag>
! If there is no alternative this can produce a wrong execution behaviour!
Also a list of tags, separated by spaces, can be provided on the commandline.
The tag
attribute can be used inside every <tag>
inside the input file (except the <jube>
).
Platform independent benchmarking¶
If you want to create platform independent benchmarks you can use the include features inside of JUBE.
All platform related sets must be declared in an includable file e.g. platform.xml
. There can be multiple platform.xml
in different
directories to allow different platforms. By changing the include-path
the benchmark changes its platform specific data.
An example benchmark structure is based on three include files:
The main benchmark include file which contain all benchmark specific but platform independent data
A mostly generic platform include file which contain benchmark independent but platform specific data (this can be created once and placed somewhere central on the system, it can be easily accessed using the
JUBE_INCLUDE_PATH
environment variable.A platform specific and benchmark specific include file which must be placed in a unique directory to allow inlcude-path usage
Inside the platform
directory you will find some example benchmark independent platform configuration files for the supercomputers at
Forschungszentrum Jülich.
To avoid writing long include-paths every time you run a platform independent benchmark, you can store the include-path inside your input file. This can be mixed using the tagging-feature:
1 2 3 4 5 6 7 8 9 | <?xml version="1.0" encoding="UTF-8"?> <jube> <include-path> <path tag="plat1">some path</path> <path tag="plat2">another path</path> ... </include-path> ... </jube> |
Now you can run your benchmark using:
>>> jube run filename.xml --tag plat1
Multiple benchmarks¶
Often you only have one benchmark inside your input file. But it is also possible to store multiple benchmarks inside the same input file:
1 2 3 4 5 6 | <?xml version="1.0" encoding="UTF-8"?> <jube> <benchmark name="a" outpath="bench_runs">...</benchmark> <benchmark name="b" outpath="bench_runs">...</benchmark> ... </jube> |
1 2 3 4 | - name: a # data for benchmark a - name: b # data for benchmark b |
All benchmarks can use the same global (as a child of <jube>
) declared sets. Often it might be better to use an include feature instead.
JUBE will run every benchmark in the given order. Every benchmark gets a unique benchmark id.
To select only one benchmark you can use:
>>> jube run filename.xml --only-bench a
or:
>>> jube run filename.xml --not-bench b
This information can also be stored inside the input file:
1 2 3 4 5 6 7 8 | <?xml version="1.0" encoding="UTF-8"?> <jube> <selection> <only>a</only> <not>b</not> </selection> ... </jube> |
Environment handling¶
Shell environment handling can be very important to configure paths or parameter of your program.
The files used for this example can be found inside examples/environment
.
The input file environment.xml
:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 | <?xml version="1.0" encoding="UTF-8"?> <jube> <benchmark name="environment" outpath="bench_run"> <comment>An environment handling example</comment> <!-- Configuration --> <parameterset name="param_set"> <parameter name="EXPORT_ME" export="true">VALUE</parameter> </parameterset> <!-- Operations --> <step name="first_step" export="true"> <do>export SHELL_VAR=Hello</do> <!-- export a Shell var --> <do>echo "$$SHELL_VAR world"</do><!-- use exported Shell var --> </step> <!-- Create a dependency between both steps --> <step name="second_step" depend="first_step"> <use>param_set</use> <do>echo $$EXPORT_ME</do> <do>echo "$$SHELL_VAR again"</do> <!-- use exported Shell var out of privious step --> </step> </benchmark> </jube> |
The input file environment.yaml
:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 | name: environment outpath: bench_run comment: An environment handling example #Configuration parameterset: name: param_set parameter: {name: EXPORT_ME, export: true, _: VALUE} step: #Operation - name: first_step export: true do: - export SHELL_VAR=Hello #export a Shell var - echo "$$SHELL_VAR world" #use exported Shell var #Create a dependency between both steps - name: second_step depend: first_step use: param_set do: - echo $$EXPORT_ME - echo "$$SHELL_VAR again" #use exported Shell var out of privious step |
In normal cases all <do>
within one <step>
shares the same environment. All exported variables of one <do>
will be available inside the next <do>
within the same <step>
.
By using export="true"
inside of a <parameter>
you can export additional variables to your Shell environment. Be aware that this example uses
$$
to explicitly use Shell substitution instead of JUBE substitution.
You can also export the complete environment of a step to a dependent step by using export="true"
inside of <step>
.
Parameter dependencies¶
Sometimes you need parameters which are based on other parameters or only a specific parameter combination makes sense and other combinations are useless or wrong. For this there are several techniques inside of JUBE to create such a more complex workflow.
The files used for this example can be found inside examples/parameter_dependencies
.
The input file parameter_dependencies.xml
:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 | <?xml version="1.0" encoding="UTF-8"?> <jube> <benchmark name="parameter_dependencies" outpath="bench_run"> <comment>A parameter_dependencies example</comment> <!-- Configuration --> <parameterset name="param_set"> <parameter name="index" type="int">0,1</parameter> <parameter name="text" mode="python">["hello","world"][$index]</parameter> </parameterset> <parameterset name="depend_param_set0"> <parameter name="number" type="int">3,5</parameter> </parameterset> <parameterset name="depend_param_set1"> <parameter name="number" type="int">1,2,4</parameter> </parameterset> <!-- Operation --> <step name="operation"> <use>param_set</use> <!-- use basic parameterset --> <use>depend_param_set$index</use> <!-- use dependent parameterset --> <use from="include_file.xml:depend_param_set0:depend_param_set1"> depend_param_set$index </use> <do>echo "$text $number $number2"</do> </step> </benchmark> </jube> |
The input file parameter_dependencies.yaml
:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | name: parameter_dependencies outpath: bench_run comment: A parameter_dependencies example #Configuration parameterset: - name: param_set parameter: - {name: index, type: int, _: "0,1"} #comma separated integer must be in quotations - {name: text, mode: python, _: '["hello","world"][$index]'} #attributes with " and [] must be in quotations - name: depend_param_set0 parameter: {name: number, type: int, _: "3,5"} #comma separated integer must be in quotations - name: depend_param_set1 parameter: {name: number, type: int, _: "1,2,4"} #comma separated integer must be in quotations #Operation step: name: operation use: - param_set #use basic parameterset - depend_param_set$index #use dependent parameterset - {from: 'include_file.yaml:depend_param_set0:depend_param_set1', _: depend_param_set$index} do: echo "$text $number $number2" |
The include file include_file.xml
:
1 2 3 4 5 6 7 8 9 10 | <?xml version="1.0" encoding="UTF-8"?> <jube> <parameterset name="depend_param_set0"> <parameter name="number2" type="int">10</parameter> </parameterset> <parameterset name="depend_param_set1"> <parameter name="number2" type="int">20</parameter> </parameterset> </jube> |
The include file include_file.yaml
:
1 2 3 4 5 | parameterset: - name: depend_param_set0 parameter: {name: number2, type: int, _: 10} - name: depend_param_set1 parameter: {name: number2, type: int, _: 20} |
The easiest way to handle dependencies is to define an index-parameter which can be used in other scripting parameters to combine all dependent parameter combinations.
Also complete sets can be marked as dependent towards a specific parameter by using this parameter in the <use>
-tag. When using parametersets
out of an other file the correct set-name must be given within the from
attribute, because these sets will be loaded in a pre-processing step before the
corresponding parameter will be evaluated. Also sets out of different files can be combined within the same <use>
by using the
file1:set1,file2:set2
syntax. The sets names must be unique.
Parameter update¶
Once a parameter is specified and evaluated the first time, its value will not change. Sometimes this behaviour can produce the wrong behaviour:
<parameter name="foo">$jube_wp_id</parameter>
In this example foo
should hold the $jube_wp_id
. If you have two steps, where one step depends on the other one foo
will be available in both, but it will only be evaluated in
the first one.
There is a simple workaround to change the update behaviour of a parameter by using the attribute update_mode
:
update_mode="never"
No update (default behaviour)update_mode="use"
Re-evaluate the parameter if the parameterset is explicitly usedupdate_mode="step"
Re-evaluate the parameter for each new stepupdate_mode="cycle"
Re-evaluate the parameter for each new cycleloop, but not at the begin of a new stepupdate_mode="always"
Combine step and cycle
Within a cycle loop no new workpackages can be created. Templates will be reevaluated, but they can not increase the number of existing workpackages within a cycle.
Within the result generation, the parameter value, which is presented in the result table is the value of the selected analysed step. If another parameter representation is needed as well,
all other steps can be reached by using <parameter_name>_<step_name>
.
The files used for this example can be found inside examples/parameter_update
.
The input file parameter_update.xml
:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 | <?xml version="1.0" encoding="UTF-8"?> <jube> <benchmark name="parameter_updates" outpath="bench_run"> <comment>A parameter_update example</comment> <!-- Configuration --> <parameterset name="foo"> <parameter name="bar_never" mode="text" update_mode="never"> iter_never: $jube_wp_id </parameter> <parameter name="bar_use" mode="text" update_mode="use"> iter_use: $jube_wp_id </parameter> <parameter name="bar_step" mode="text" update_mode="step"> iter_step: $jube_wp_id </parameter> </parameterset> <!-- Operation --> <step name="step1"> <use>foo</use> <do>echo $bar_never</do> <do>echo $bar_use</do> <do>echo $bar_step</do> </step> <step name="step2" depend="step1"> <use>foo</use> <do>echo $bar_never</do> <do>echo $bar_use</do> <do>echo $bar_step</do> </step> <step name="step3" depend="step2"> <do>echo $bar_never</do> <do>echo $bar_use</do> <do>echo $bar_step</do> </step> </benchmark> </jube> |
The input file parameter_update.yaml
:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 | name: parameter_updates outpath: bench_run comment: A parameter_update example #Configuration parameterset: name: foo parameter: - {name: bar_never, mode: text, update_mode: never, _: "iter_never: $jube_wp_id"} - {name: bar_use, mode: text, update_mode: use, _: "iter_use: $jube_wp_id"} - {name: bar_step, mode: text, update_mode: step, _: "iter_step: $jube_wp_id"} #Operation step: - name: step1 use: foo do: - echo $bar_never - echo $bar_use - echo $bar_step - name: step2 depend: step1 use: foo do: - echo $bar_never - echo $bar_use - echo $bar_step - name: step3 depend: step2 do: - echo $bar_never - echo $bar_use - echo $bar_step |
The use and influence of the three update modes update_mode="never"
, update_mode="use"
and update_mode="step"
is shown here. Keep in mind, that the steps have to be dependent
from each other leading to identical outputs otherwise.
Step iteration¶
Especially in the context of benchmarking, an application should be executed multiple times to generate some meaningful statistical values. The handling of statistical values is described in Statistic pattern values. This allows you to aggregate multiple result lines if your application automatically support to run multiple times.
In addition there is also an iteration feature within JUBE to run a specific step and its parametrisation multiple times.
The files used for this example can be found inside examples/iterations
.
The input file iterations.xml
:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 | <?xml version="1.0" encoding="UTF-8"?> <jube> <benchmark name="iterations" outpath="bench_run"> <comment>A Iteration example</comment> <!-- Configuration --> <parameterset name="param_set"> <parameter name="foo" type="int">1,2,4</parameter> <parameter name="bar" mode="text" update_mode="step">$foo iter:$jube_wp_iteration</parameter> </parameterset> <step name="first_step" iterations="2"> <use>param_set</use> <!-- use existing parameterset --> <do>echo $bar</do> <!-- shell command --> </step> <step name="second_step" depend="first_step" iterations="2"> <do>echo $bar</do> <!-- shell command --> </step> <!-- analyse without reduce --> <analyser name="analyse_no_reduce" reduce="false"> <analyse step="second_step" /> </analyser> <!-- Analyse with reduce --> <analyser name="analyse" reduce="true"> <analyse step="second_step" /> </analyser> <result> <use>analyse</use> <use>analyse_no_reduce</use> <table name="result" style="pretty"> <column>jube_res_analyser</column> <column>jube_wp_id_first_step</column> <column>jube_wp_id</column> <column>jube_wp_iteration_first_step</column> <column>jube_wp_iteration</column> <column>foo</column> </table> </result> </benchmark> </jube> |
The input file iterations.yaml
:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 | name: iterations outpath: bench_run comment: A Iteration example #Configuration parameterset: name: param_set parameter: - {name: foo, type: int, _: "1,2,4"} - {name: bar, update_mode: step, _: '$foo iter:$jube_wp_iteration'} step: - name: first_step iterations: 2 use: param_set #use existing parameterset do: echo $bar #shell command - name: second_step depend: first_step iterations: 2 do: echo $bar #shell command analyser: #analyse without reduce - name: analyse_no_reduce reduce: false analyse: step: second_step #analyse with reduce - name: analyse reduce: true analyse: step: second_step result: use: [analyse,analyse_no_reduce] table: name: result style: pretty column: - jube_res_analyser - jube_wp_id_first_step - jube_wp_id - jube_wp_iteration_first_step - jube_wp_iteration - foo |
In this example, both steps 1 and 2 are executed 2 times for each parameter and dependency configuration. Because of the given parameter, step 1 is executed 6 times in total (3 parameter combinations x 2). Step 2 is executed 12 times (6 from the dependent step x 2). Each run will be executed in the normal way using its individual sandbox folder.
$jube_wp_iteration
holds the individual iteration id. The update_mode
is needed here to reevaluate the parameter bar
in step 2.
In the analyser reduce=true
or reduce=false
can be enabled, to allow you to see all individual results or to aggregate all results of the same parameter combination.
for the given step. If reduce=true
is enabled (the default behaviour) the output of the individual runs, which uses the same parametrisation, are treated like a big continuous file
before applying the statistical patterns.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 | | jube_res_analyser | jube_wp_id_first_step | jube_wp_id | jube_wp_iteration_first_step | jube_wp_iteration | foo | |-------------------|-----------------------|------------|------------------------------|-------------------|-----| | analyse_no_reduce | 0 | 6 | 0 | 0 | 1 | | analyse_no_reduce | 0 | 7 | 0 | 1 | 1 | | analyse_no_reduce | 1 | 8 | 1 | 2 | 1 | | analyse_no_reduce | 1 | 9 | 1 | 3 | 1 | | analyse_no_reduce | 2 | 10 | 0 | 0 | 2 | | analyse_no_reduce | 2 | 11 | 0 | 1 | 2 | | analyse_no_reduce | 3 | 12 | 1 | 2 | 2 | | analyse_no_reduce | 3 | 13 | 1 | 3 | 2 | | analyse_no_reduce | 4 | 14 | 0 | 0 | 4 | | analyse_no_reduce | 4 | 15 | 0 | 1 | 4 | | analyse_no_reduce | 5 | 16 | 1 | 2 | 4 | | analyse_no_reduce | 5 | 17 | 1 | 3 | 4 | | analyse | 5 | 16 | 1 | 2 | 4 | | analyse | 0 | 7 | 0 | 1 | 1 | | analyse | 1 | 8 | 1 | 2 | 1 | | analyse | 2 | 10 | 0 | 0 | 2 | | analyse | 3 | 12 | 1 | 2 | 2 | | analyse | 4 | 15 | 0 | 1 | 4 | |
Step cycle¶
Instead of having a new workpackage you can also redo the <do>
commands inside a step using the cycle-feature.
In contrast to the iterations, all executions for the cycle feature take place inside the same folder.
The files used for this example can be found inside examples/cycle
.
The input file cycle.xml
:
1 2 3 4 5 6 7 8 9 10 11 12 | <?xml version="1.0" encoding="UTF-8"?> <jube> <benchmark name="cycle" outpath="bench_run"> <comment>A cycle example</comment> <step name="a_step" cycles="5"> <do break_file="done">echo $jube_wp_cycle</do> <do active="$jube_wp_cycle==2">touch done</do> </step> </benchmark> </jube> |
The cycles
attribute allows to repeat all <do>
commands within a step multiple times. The break_file
can be used to cancel the loop and all following commands in the current cycle (the command
itself is still executed). In the given example the output will be:
0
1
2
3
In contrast to the iterations, all executions for the cycle feature take place inside of the same folder.
Parallel workpackages¶
In a standard jube run
a queue is filled with workpackages and then
processed in serial. To enable parallel execution of independent workpackages,
which belong to the expansions of a step, the argument procs
of <step>
can be used.
The files used for this example can be found inside examples/parallel_workpackages
.
The input file parallel_workpackages.xml
:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | <?xml version="1.0" encoding="UTF-8"?> <jube> <benchmark name="parallel_workpackages" outpath="bench_run"> <comment>A parallel workpackages demo</comment> <parameterset name="param_set"> <parameter name="i" type="int" mode="python">",".join([ str(i) for i in range(0,10)])</parameter> </parameterset> <step name="parallel_execution" suffix="${i}" procs="4"> <use>param_set</use> <do>echo "${i}"</do> <do>N=1000000 ; a=1 ; for (( k = 0 ; k < $N ; ++k )); do a=$(( 2*k + 1 + $a )) ; done</do> </step> </benchmark> </jube> |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | name: parallel_workpackages outpath: bench_run comment: A parallel workpackages demo parameterset: name: param_set parameter: {name: i, type: int, mode: python, _: "\",\".join([ str(i) for i in range(0,10)])"} step: name: parallel_execution suffix: ${i} procs: 4 use: param_set do: - "echo \"${i}\"" - "N=1000000 ; a=1 ; for (( k = 0 ; k < $N ; ++k )); do a=$(( 2*k + 1 + $a )) ; done" |
In the example above the expansion of the parameter i
will lead to the
creation of 10 workpackages of the step parallel_execution
. Due to the
given argument procs="4"
JUBE will start 4 worker processes which will
distribute the execution of the workpackages among themselves. N
within the JUBE script represents the number of computation iterations to
simulate a computational workload at hand. The parameters N
, procs
and the upper bound of range
within this prototypical example can be
alternated to study runtime, memory usage and load of CPUs.
Important hints:
<do shared="true">
is not supported ifprocs
is set for the corresponding step.If
<step shared="...">
is set, then the user is responsible to avoid data races within the shared directory.Switching to an alternative
work_dir
for a step can also lead to data races if all expansions of the step access the samework_dir
. Recommendation: Don’t use a sharedwork_dir
in combination withprocs
.This feature is implemented based on the Python package
multiprocessing
and doesn’t support inter-node communication. That’s why the parallelisation is limited to a single shared memory compute node.Be considerate when working on a multi-user system with shared resources. The parallel feature of JUBE can easily exploit a whole compute node.
Parallel execution of a JUBE script can lead to much higher memory demand compared to serial execution with
procs=1
. In this case it is advised to reduceprocs
leading to reduced memory usage.
Result database¶
Results can also be stored into a database to simplify result management.
The files used for this example can be found inside examples/result_database
.
The input file result_database.xml
:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 | <?xml version="1.0" encoding="UTF-8"?> <jube> <benchmark name="result_database" outpath="bench_run"> <comment>result database creation</comment> <parameterset name="param_set"> <parameter name="number" type="int">1,2,4</parameter> </parameterset> <patternset name="pattern"> <pattern name="number_pat" type="int">Number: $jube_pat_int</pattern> </patternset> <step name="write_number"> <use>param_set</use> <do>echo "Number: $number"</do> </step> <analyser name="analyse"> <use>pattern</use> <analyse step="write_number"> <file>stdout</file> </analyse> </analyser> <result> <use>analyse</use> <!-- creating a database containing the columns "number" and "number_pat" --> <!-- one table of the name "results" is created within the database --> <!-- a database file "result_database.dat" is created in the current working directory --> <database name="results" file="result_database.dat" primekeys="number,number_pat"> <key>number</key> <key>number_pat</key> </database> </result> </benchmark> </jube> |
The input file result_database.yaml
:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 | name: result_database outpath: bench_run comment: result database creation parameterset: name: param_set parameter: {name: number, type: int, _: "1,2,4"} patternset: name: pattern pattern: {name: number_pat, type: int, _: "Number: $jube_pat_int"} step: name: write_number use: param_set do: "echo \"Number: $number\"" analyser: name: analyse use: pattern analyse: step: write_number file: stdout result: use: analyse database: # creating a database containing the columns "number" and "number_pat" # one table of the name "results" is created within the database # a database file "result_database.dat" is created in the current working directory name: results file: result_database.dat primekeys: "number,number_pat" key: - number - number_pat |
The default database will be located as follows and has the database
tag name, which is here results
, as root name concatenated with the appendix .dat
:
1 2 3 4 5 6 7 | bench_run | +- 000000 | +- result | +- results.dat |
The database
tag takes the argument name
. name
is also the name of the table created within a database. If sqlite3
is installed the contents of the database can be shown with the following command line.
1 2 3 4 5 6 7 8 | >>> sqlite3 -header -table bench_run/000000/result/results.dat 'SELECT * FROM results' +--------+------------+ | number | number_pat | +--------+------------+ | 1 | 1 | | 2 | 2 | | 4 | 4 | +--------+------------+ |
The argument file
states the full path of a second copy of the database file. The path can be stated relative or absolute. In this case the database file result_database.dat
is created within the current working directory in which jube result
was invoked.
Invocating jube result
a second time updates the database given by the file
parameter. Without the parameter primekeys
three additional lines to the results
table would have been added which are completely identical to the previous three lines. Adding the argument primekeys
ensures that only if the column values stated within primekeys
are not exactly the same in the database table, a new line is added to the database table. In this example no new lines are added. All the primekeys
also need to be stated as key
. Updating the primekeys
is not supported.
The key
tag adds columns to the database table having the same type as the corresponding parameter
or pattern
. Information of columns of the database table results
can be shown as follows.
1 2 3 4 5 6 7 | >>> sqlite3 -header -table bench_run/000000/result/results.dat 'PRAGMA table_info(results)' +-----+------------+------+---------+------------+----+ | cid | name | type | notnull | dflt_value | pk | +-----+------------+------+---------+------------+----+ | 0 | number | int | 0 | | 1 | | 1 | number_pat | int | 0 | | 2 | +-----+------------+------+---------+------------+----+ |
To have a look into a database within a python script the python modules sqalchemy or pandas can be used.
Creating a do log¶
To increase reproducibility of the do statements within a workpackage of a step and to archive the environment during execution, a do log can be printed. A do log tries to mimic an executable script recreating the environment at execution time. The files used for this example can be found inside examples/do_log
.
The input file do_log.xml
:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | <?xml version="1.0" encoding="UTF-8"?> <jube> <benchmark name="do_log_example" outpath="bench_run"> <parameterset name="param_set"> <parameter name="number">1,2,3,4,5</parameter> </parameterset> <step name="execute" shared="shared" do_log_file="do_log"> <use>param_set</use> <do>cp ../../../../loreipsum${number} shared</do> <do shared="true">grep -r -l "Hidden!" loreipsum*</do> </step> </benchmark> </jube> |
The input file do_log.yaml
:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | - name: do_log_example outpath: bench_run parameterset: - name: "param_set" parameter: - {name: "number", _: "1,2,3,4,5"} step: name: execute use: - param_set do_log_file: "do_log" shared: "shared" do: - cp ../../../../loreipsum${number} shared - {shared: "true", _: "grep -r -l \"Hidden!\" loreipsum*"} |
In this example a hidden string is searched for within 5 files and the name of the file containing the hidden string is printed.
For the initial execution of this example within bench_run/000000/00000[0-4]_execute
each can be found a do_log
file. These files can be executed manually by prefixing it with /bin/sh
. The scripts will reproduce the environment at execution time, the execution and the result output. Keep in mind that the shared grep
will be executed by the benchmark with id 4 only.
The duplicate option¶
To simplify advanced tagging and parameter concatenation the duplicate
option within parametersets or parameters can be stated.
The input file duplicate.xml
:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | <?xml version="1.0" encoding="UTF-8"?> <jube> <benchmark name="parameter_duplicate_example" outpath="bench_run"> <comment>parameter duplicate example</comment> <parameterset name="options" duplicate="concat"> <parameter name="iterations" >1</parameter> <parameter name="iterations" tag="few" >2,3,4</parameter> <parameter name="iterations" tag="many">20,30,40</parameter> </parameterset> <parameterset name="result"> <parameter name="sum" mode="python">int(${iterations}*(${iterations}+1)/2)</parameter> </parameterset> <step name="perform_iterations"> <use>options,result</use> <do>echo $sum</do> </step> </benchmark> </jube> |
The input file duplicate.yaml
:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | name: parameter_duplicate_example outpath: bench_run comment: parameter duplicate example parameterset: - name: options duplicate: concat parameter: - {name: iterations, _: "1"} - {name: iterations, tag: few, _: "2,3,4"} - {name: iterations, tag: many, _: "20,30,40"} - name: result parameter: - {name: sum, mode: "python", _: "int(${iterations}*(${iterations}+1)/2)"} step: name: perform_iterations use: "options,result" do: "echo $sum" |
In this example the duplicate
option with the value concat
is stated for a parameterset. This leads to a concatenation of parameter values of the same name. In combination with the tagging option for parameters the user can specify which options are included into the parameters. If the user states the tags few
and many
the parameter iterations
takes the values 1,2,3,4,20,30,40
.
The default option of duplicate
for parametersets is replace
which leads to a replacing of parameters if they are mentioned more than once. A third option for the duplicate
option for parametersets is error
. In this case the execution is aborted if a parameter is defined more than once.
The option duplicate
can also be stated for parameters. In this case the parameters duplicate
option is prioritized over the parametersets one. The possible values for parameters duplicate
option are none
, replace
, concat
and error
. none
is the default value and leads to the duplicate
option being ignored for this parameter such that the parametersets duplicate
option is taking precedence. The other three options have the same effect as in the parameterset.