Many of you have already heard about the following in various meetings and slack channels, but things are now complete enough to make the official announcement.
There is now a production filesystem on which teststand data will reside at NCSA (/lsstdata/offline/teststand). This filesystem is accessible from both login and compute nodes in the lsst-dev cluster as well as all the LSP nodes. Paths will be the same on all LSST systems.
Teststand sets (read-only):
- auxTel from L1 Archiver: /lsstdata/offline/teststand/auxTel/L1Archiver
- auxTel from ACCS/DAQ: /lsstdata/offline/teststand/auxTel/DAQ
- BOT: /lsstdata/offline/teststand/BOT
Gen2 Ingestion Notes:
- Mass ingestion of data prior to 2019-05-30 was done with w_2019_21 into fresh production repo
- Output of ingestImages.py located in /log//ingest_output.log
- Since code is still changing regularly, will use latest weekly (not a normal production procedure)
- NCSA is not actively trying to ingest images that previously failed to ingest using each new weekly stack
- K-T has an updated LSST Dev Guide with the new paths ready to merge to master
- LDF documentation on teststand data is available at:
- If files are in the storage directory, but not in the gen2repo directory/database, check the ingest_output.log in the log/ directory for ingestion errors.
- If there are images that failed to ingest that are important, please make a specific request via JIRA (assignee: Michelle Gower)
- If /lsstdata/offline/teststand isn’t visible on a particular machine/container, please report via an IHS ticket in JIRA.
- Requests for new sets of teststand data should also be requested via JIRA (assignee: Michelle Butler)
While waiting for “global” access to this production filesystem, auxTel L1 Archiver images have been duplicated in a temporary location (/project/production/tmpdataloc).
For those using this temporary location via chained Gen2 repositories, the _parent link will need to be updated from /project/production/tmpdataloc/… to /lsstdata/offline/teststand/… .
(e.g., K-T is currently working with folks to determine the time of the switchover of /project/shared/auxTel/_parent)
Tucson L1 Archiver & ACCS/DAQ Image Schedule:
July 22nd: Both temp and production location actively getting any new Tucson L1 Archiver images
Aug 5th: Halt new data arrival in temp location
Sep 5th: Delete data in temp location (or earlier with users’ permission)
SLAC BOT Image Schedule:
July 22nd: Both temp and production location actively getting any new SLAC BOT raw images
Prior to next large BOT batch (early Aug): Halt new data arrival in temp location
Sep 5th: Delete data in temp location (or earlier with users’ permission since takes up a lot of /project space)