Information about $DATA incident from January 2021
This page collects information about the
$DATA incident that occurred on 2021-01-26. Until 2021-02-02 an improbable sequence of unrelated hardware failures came together with a firmware bug and has unfortunately led to data loss. This page contains information about the next steps and relevant answers to questions related to the incident and its aftermath. A description of the event time line is provided at the bottom of the page.
Which of my data is affected?
A list of all potentially affected files have been communicated to the persons of contact (PI or PA) of each data project on 2021-02-14. The provided list contained all files that had blocks on the affected disk group. Please note that this list was generated based on the location of the data storage for the files at the time of the incident and we expect the list of actually affected files to be much shorter.
At this point, we cannot reliably assess which of these files are actually affected. Statistics gathered by comparison of restored files and files in
$DATA suggests that ca. 1-2% of the potentially affected files contain corrupted data blocks. The percentage will vary between projects in dependance on file size.
Is my data corrupted or missing entirely?
All i-nodes on the file system are intact and hence all files are visible in the filesystem with correct metadata and are fully accessible.
How can I detect data corruption in my files?
The completeness and accuracy of metadata (file size, time stamps, etc.) does not indicate the correctness of the data in the file!
Based on the analysis so far, we know that all faulty blocks are 512 KiB (524288 bytes) long due to the underlying RAID stripe layout (in rare cases multiple faulty blocks are adjacent). In most cases, the faulty blocks are zeroed out and detectable in this way. Unfortunately, not all affected blocks are zero.
In order to detect zero blocks with this size, we provide the tool
find0s.el8.x located at
/p/largedata/bin for use on JUDAC, JUWELS and JURECA.
find0s.el8.x --quick <filename> searched
<filename> for zero blocks which are a multiple in size of 512 KiB.
--quick) will scan the file for all zero regions. Please note that, depending on the file format, zero blocks are common.
find0s.el8.x will detect corruption but files for which
find0s.el8.x does not report a matching region are not necessarily corruption-free.
Why did the RAID metadata recovery not work fully?
The recovery procedure used the metadata stored in disks in
dg_b01 that were excluded from the rebuild process (and hence not subject to the wiping of the metadata). Due to this, the metadata on these drives was not updated during the rebuild process and hence reflects the stripe allocation at the start of the rebuild. It does not capture any relocation of RAID stripes that happened during the rebuild due to the algorithm.
Is the firmware bug fixed?
We are currently waiting for a released firmware fix. The bug fix is currently undergoing testing. Stop-gap measures are in place. We have been assured that re-occurrence of the problem is not possible without manual actions which are prohibited at this time.
There used to be snapshots, why can’t we use them?
Snapshots protect data on the file systems from accidental file deletion, overwrites and similar events but do not allow to recover data in case of a system-internal corruption such as the one that unfortunately occurred now.
What will be the future measures to avoid such an incident?
$DATAwas available as a storage service generally without out-of-system backup. We are working to enable a regular full backup of the file system and, once operational, will announce the general availability of this feature for all projects on
$DATA. For technical reasons, we can only start providing backup functionality for $DATA once the restore to
/p/largedata_restore/is finalized. Hence, please expect that backup will be available only several weeks in the future. Due to the large volume in the file system and its continuously changing data modification rates the number of backup runs/week will be variable and cannot be predicted at the moment. The backup will currently be performed on tape but the technology may change without further notice. We intend to provide a mechanism for you to assess which files have an off-system copy for incorporation with workflows. Please note that, given the file system size, a backup provides data safety (for data that has been injected before the last backup run) but does not guarantee data availability in case of an incident, since recalls of large volumes may take a very long time.
- Temporary duplication
As a temporary measure until the full system backup is in place and has been validated, we intend to perform an in-system duplication of data to ensure that data blocks are available in more than one RAID system. This will cut in half the available storage space and hence is not a long term option. It also does not provide the same safety as an off-system backup. Please note that the necessary duplication will take several days to complete and may lower access times to the system while it is running.
- File system reorganization
Over the course of the next months we will reorganize the $DATA structure and distribute projects over more file systems. This will improve the manageability of the file systems but also reduce the available bandwidth.
For how long will the largedata_restore filesystem be available?
We plan to leave largedata_restore available to users for six months. It will be removed on 2021-09-30.
What is the reason for my data volume being about twice as large as I expect it to be?
As a consequence of the duplication of data mentioned above, the volume of each file is counted twice. The duplication is a temporary measure until a proper out-of-system backup has been established. It is rather difficult to prevent this effect from showing to the user. Therefore, we decided to also increase the volume quota for each project to compensate for the effect. Quotas will be reset to normal once the duplication will be disabled and the full backup has been established.
The number of files/inodes is not affected by this measure, so you will not see twice the number of files.
Event time line
Failure of one storage enclosure and loss of access to 14 drives. An I/O error on
$DATAwas visible when accessing files with blocks on these drives. Replacement of a seemingly unrelated drive.
Attempts to revive enclosure were not successful.
$DATAwas taken fully into maintenance.
Successful replacement of the enclosure. Disk group
dg_b01started rebuild but
dg_a01remained in quarantine due to lack of disks.
Failure of controller
bforced rebuild to migrate to controller
a. A dequarantine instruction for
dg_a01triggered a previously unknown firmware bug in the form of a race condition between the reconfiguration and the rebuild for disk group
dg_b01. The bug resulted in RAID stripe metadata being cleared on drives.
Loss of RAID metadata confirmed. Start of analysis of affected files and recovery options. Start of migration of data from
dg_a01to other disks.
Start of test for RAID metadata recovery from drives originally in
dg_b01that were excluded from the rebuild and have not been cleared by the bug.
Reduplication of meta data was triggered to ensure two copies are available.
Custom firmware for recovery has gone through testing.
List of files with blocks on
dg_b01has been identified and cross-checked against the available copies on tape.
Start of restore of files onto new file system
List of affected files per project communicated to the contact persons for the data project.
Start of recovery actions.
RAID metadata recovery finalized. Work moves to file system level. After verification of several checksums, the migration of the data from the disks in
dg_b01to other disks in the system was triggered since
dg_b01was still in critical state and additional data loss in case of a drive failure was possible.
dg_b01were successfully removed from the system and two file system checks did not show problems (note: file system checks do not verify correctness of data blocks).
Start of assessment of the exposure of data on the file system to corruption by comparison with tape copies.
$DATAmade available in read-only mode on login and data access nodes allowing users to access data and start validation. The file system
/p/largedata_restoreis made available in read-only mode while the restore of data from tape continues. We are working on making markers in the project available to allow you to assess the completeness of the restore for your project.
$DATAmade available read-write.
Start data duplication for
$DATA(on file system layer).
The restore of