Skip to content

Continuous Benchmarking Module Architecture

The Continuous Benchmarking (CB) module acts as a bridge between raw artifact data (CSV/JSON) stored in Git repositories and the LLview visualization frontend. It utilizes the git.py plugin to collect information from the repo, and generate the LML files with the data to be added to the databases as well as the required internal configuration files for LLview.

Git plugin

Data Flow

  1. Data Ingestion (Python): The BenchRepo class parses the users's YAML configuration files and the corresponding data files. Examples of configuration files for the Git plugin are given here.
  2. Logic Processing:
    • Metrics are collected from filename, content, metadata and cast to their correct types.
    • Derived metrics (calculated via formulas) are computed.
    • _status is determined based on data integrity.
  3. Configuration Generation: The Python module generates the internal LLview YAML files to configure:
    • Databases
    • Tabs on the web portal
    • Tables
    • Footer Graphs
  4. Database Update (Perl): LLview will use the generated configuration files to add the information from the CB.xml into the corresponding CB database in the corrected expected description (the data is added to the database in the LMLDBupdate step).
  5. Web Portal input generation (Perl): The configuration files also describe the files LLview needs to generate for the web portal in within the jobreport action.

Plugin Usage

These are the options for the ${LLVIEW_HOME}/da/rms/git/git.py plugin script: 

${PYTHON} ${LLVIEW_HOME}/da/rms/git/git.py --help
usage: git.py [-h] [--config CONFIG] [--loglevel LOGLEVEL] [--singleLML SINGLELML] [--tsfile TSFILE] [--outfolder OUTFOLDER] [--repofolder REPOFOLDER] [--outconfigfolder OUTCONFIGFOLDER] [--skipupdate]

Prometheus Plugin for LLview

options:
  -h, --help            show this help message and exit
  --config CONFIG       YAML config file (or folder with YAML configs) containing the information to be gathered and converted to LML
  --loglevel LOGLEVEL   Select log level: 'DEBUG', 'INFO', 'WARNING', 'ERROR' (more to less verbose)
  --singleLML SINGLELML
            Merge all sections into a single LML file
  --tsfile TSFILE       File to read/write timestamp
  --outfolder OUTFOLDER
            Reference output folder for LML files
  --repofolder REPOFOLDER
            Folders where the repos will be cloned
  --outconfigfolder OUTCONFIGFOLDER
            Folder to generate config files
  --skipupdate          Skip updating the repos (if they don't exist, they will still be cloned)

Continuous Benchmarking

First run, Restarting or Manual Run

Since LLview requires the configuration files that are generated by the Git plugin to know how the databases will be designed, the process to (re)start a CB requires a few steps. The CB module may also need to he restarted due to updates, changes on the databases, etc. - or you may even want to run the steps "by hand" (to test configurations). Since this module usually does not involve too much data and computation, the following steps can be run "by hand" (Note: you may need to adapt the folders, if not using the default environment variables):

  • [For restart only] Remove old CB files: 
    rm -r $LLVIEW_DATA/$LLVIEW_SYSTEMNAME/perm/db/LLmonDB_CB.sqlite $LLVIEW_DATA/$LLVIEW_SYSTEMNAME/perm/git_last_timestamp.dat $LLVIEW_DATA/$LLVIEW_SYSTEMNAME/tmp/CB.xml $LLVIEW_CONF/server/LLgenDB/conf_cb/*_cb.yaml $LLVIEW_DATA/$LLVIEW_SYSTEMNAME/tmp/jobreport/data/cb/*
  • [Optional] Remove repos: 
    rm -rf $LLVIEW_DATA/$LLVIEW_SYSTEMNAME/tmp/repos
  • Create new LML files and configurations:
    • Without timestamp recording (i.e., next run will generate the LML file with all the points again) 
      $PYTHON ${LLVIEW_HOME}/da/rms/git/git.py --config $LLVIEW_CONF/plugins/cb/ --singleLML $LLVIEW_DATA/$LLVIEW_SYSTEMNAME/tmp/CB.xml --repofolder $LLVIEW_DATA/$LLVIEW_SYSTEMNAME/tmp/repos --outconfigfolder $LLVIEW_CONF/server/LLgenDB/conf_cb/
    • With timestamp of last update (following runs will generate LML with new points only) 
      $PYTHON ${LLVIEW_HOME}/da/rms/git/git.py --config $LLVIEW_CONF/plugins/cb/  --tsfile $LLVIEW_DATA/$LLVIEW_SYSTEMNAME/perm/git_last_timestamp.dat --singleLML $LLVIEW_DATA/$LLVIEW_SYSTEMNAME/tmp/CB.xml --repofolder $LLVIEW_DATA/$LLVIEW_SYSTEMNAME/tmp/repos --outconfigfolder $LLVIEW_CONF/server/LLgenDB/conf_cb/
  • Update the database: 
    updatedb
  • Important: When running these steps by hand, the execution of git.py will create the LML file $LLVIEW_DATA/$LLVIEW_SYSTEMNAME/tmp/CB.xml with all the points collected from the configured repositories. If this LML file is already added to the LMLDBupdate step (as explained below), the data will be added to the databases in every loop. To avoid multiple additions, you may re-run the command (with the timestamp!) after the data was added - usually 1 minute later (or, instead of waiting 1 minute, you may check if the data is already added to the database, for example, by running sqlite3 $LLVIEW_DATA/$LLVIEW_SYSTEMNAME/perm/db/LLmonDB_CB.sqlite and then sqlite> select * from cb_benchmarks; and confirming if the result is not empty).

Integration on LLview

Updates on the DBupdate action

The git.py plugin can be run as a step in the workflow of the dbupdate action. Since LLview updates are done every minute and the benchmarks are done in larger intervals (hourly, daily, etc.), the script $LLVIEW_HOME/da/utils/exec_every_n_step_or_empty.pl <file_to_store_steps> X (...) may be used to run this script every X times only. The step to run the Git plugin every hour is then: 

<!--   STEP: plugin to get info from git repos  -->
<step active="1" id="da_git" exec_after="start" type="execute">
<cmd  exec="$perl $instdir/utils/exec_every_n_step_or_empty.pl $permdir/stepcount_git.dat 
                                60
                                $instdir/utils/empty_LML.xml 
                                $tmpdir/CB.xml
                                $python $instdir/rms/git/git.py
                                                    --config $LLVIEW_CONF/plugins/cb/ 
                                                    --tsfile $permdir/git_last_timestamp.dat
                                                    --singleLML $tmpdir/CB.xml
                                                    --repofolder $tmpdir/repos
                                                    --outconfigfolder $LLVIEW_CONF/server/LLgenDB/conf_cb/
                                            "/>
</step>
The plugin will generate the single LML file CB.xml containing the data of all repos defined in $LLVIEW_CONF/server/LLgenDB/conf_cb/.

The CB.xml must then be added to the list of LML files in the LMLDBupdate step to be added to the databases (it will be done only after the configuration of the DBs are added - see below), as well as in the combineLML_all step.

Adding the configuration files

The configuration files need also to be added to describe the database, pages, footers, etc., such that the database scheme is described and the pages are generated correctly. The following files must be modified by adding (or uncommenting) the lines below:

  • $LLVIEW_CONF/server/LLgenDB/LLgenDB.yaml: 
    %include "conf_cb/CB.yaml"
  • $LLVIEW_CONF/server/LLgenDB/conf_jobreport/jobreport_footer.yaml: 
    %include "../conf_cb/footer_cb.yaml"
  • $LLVIEW_CONF/server/LLgenDB/conf_jobreport/jobreport_vars.yaml: 
    %include "../conf_cb/vars_cb.yaml"
  • $LLVIEW_CONF/configs/server/LLgenDB/conf_jobreport/data_json/jobreport_datafiles_json_support.yaml: 
    %include "../../conf_cb/tablecsv_cb.yaml"
%include "../../conf_cb/csv_cb.yaml"
  • $LLVIEW_CONF/server/LLgenDB/conf_jobreport/data_templates/jobreport_datafiles_table_templates.yaml: 
    %include "../../conf_cb/template_cb.yaml"
  • $LLVIEW_CONF/server/LLgenDB/conf_jobreport/views/jobreport_view_support.yaml: 
    %include "../../conf_cb/tab_cb.yaml"