Accounting

This document explains how to obtain information about the accounting mode and the current quota-test status.

Job accounting is done via a central database in JSC and the information about JURECA jobs is updated once per day, around midnight, based on information obtained from Slurm database about jobs that ended that day. Users can query information about their current quota status or the usage of single jobs by using the command:

jutil user cpuquota [flags] [options]

which retrieves the information from the database.

All Data quota is immediately made available once the project starts on the system.

Compute Quota General Rules

  • Users are charged for wall-clock time, i.e. for the time the nodes are occupied.

  • Jobs charge the associated account time for all CPU cores on the node, regardless of how many are utilised effectively by that job, or how many GPUs are used. E.g. for a compute node with 128 CPU cores, 128 core-h will be charged per hour of use.

  • There is no compensation for lost compute time in the event of job cancellations due to system issues.

Compute Quota Accounting Modes

Monthly Quota

Most normal projects are given CPU quota monthly, where the granted compute time is split equally amongst the months the project runs, in order to address uneven utilisation and keep queueing and wait times short.

Warning

Compute time can be lost if not used within the appropriate window. Please ensure you understand the below rules to avoid losing compute time.

The following rules apply to monthly quotas:

  • Compute time is in general split equally between each month of the project

  • Each month, a project can access the compute time from a three-month window:

    • The previous month (unless it is the first month of a project)

    • The current month

    • The following month (unless it is the last month of the project)

  • Unused compute time from the previous month will automatically expire when the month changes, and can no longer be used.

    • This reduces the total budget of the project.

  • If the quota of all months in the three month window is consumed, the project can still submit jobs, but is put into a status where jobs are limited to 6hrs walltime and given low priority.

    • If consumption is within the new three-month window at the change of month, the project returns to normal status.

    • If the consumption is still above the new three month window at the change of month, the project remains in the limited status.

    • If a project remains in limited status after the change of month, any usage over the new three-month window is set to 1% consumption, indepent of the time used. The difference in compute time between the actual value and the new reduced usage value is added as a bonus on top of the granted budget. This provides a mechanism for projects to return to a normal status after a period of very high usage, and adds a bonus for projects that can exploit gaps in scheduling, and, as such, help optimise the resource usage on our systems.

Fixed Quota

Small compute time projects are given their entire granted compute time budget at the beginning of the allocation period, i.e. the entirety of the compute resources can be called up and used at any time during the allocation period of the project.

Exceptional Workloads

There are a few exceptional circumstances where the above restrictions on compute quota are somewhat lifted:

  • Big Days, which are specific days where the full system is reserved for large jobs, and projects full budgets can be accessed, if necessary.

    • Big Days broadly require availability of sufficient core-h, and demonstration that the code can already run at larger scales in normal queueing.

  • For campaigns (periods of heightened compute requirements that might extend to large fractions of the system) the monthly budget can be increased for a limited period of time.

    • Campaigns will only be possible with strong justification, as the fair usage of the system between all projects is paramount. In general, projects should plan in the first instance to conduct their calculations within their normal monthly quota.

In either case, this requires more in-depth communication with JSC. If you believe your project needs to access either route, please contact SC Support (sc@fz-juelich.de).

Command description

Command

Description

jutil user cpuquota [flags] [options]

Show values of granted and used CPU quota

available flags:

-a, --all

Query all entries

-Z, --currsys

Query cpuquota only for current/local system

-h, --help

Prints help information

-n, --noheader

Do not print output header

-v

Increase output verbosity

-V, --version

Prints version information

available options:

-A, --budget <BUDGET>

Filter results with given budget account ID

-c, --contpart <CONTPART>

Filter results with given contingent partition

-o, --format <FORMAT>
Output format. Can be retrieved from env var JUTIL_OUTPUT_FORMAT
default: rows, possible values: rows, columns,parsable, json

-p, --project <PROJECT>

Specify project

-s, --system <SYS>

Query cpuquota only for given system

-u, --username <USER>

Specify user name