Hi, all
I am currently focusing on the process HSC data( emmm, left for a month due to other thing, now I am back). I’m trying to arrange the data structure as possiable as like rc2_subset, which looks like:
Name Type
------------------------------------------- -----------
HSC/calib/DM-28636/curated/20130131T000000Z RUN
HSC/calib/DM-28636/curated/20140601T000000Z RUN
HSC/calib/DM-28636/unbounded RUN
HSC/calib/gen2/20180117/20140131T000000Z RUN
HSC/calib/gen2/20180117/20140325T000000Z RUN
HSC/calib/gen2/20180117/20140328T000000Z RUN
HSC/calib/gen2/20180117/20140329T000000Z RUN
HSC/calib/gen2/20180117/20140402T000000Z RUN
HSC/calib/gen2/20180117/20140404T000000Z RUN
HSC/calib/gen2/20180117/20140704T000000Z RUN
HSC/calib/gen2/20180117/20140714T000000Z RUN
HSC/calib/gen2/20180117/20141102T000000Z RUN
HSC/calib/gen2/20180117/20141104T000000Z RUN
HSC/calib/gen2/20180117/20141112T000000Z RUN
HSC/calib/gen2/20180117/20141114T000000Z RUN
HSC/calib/gen2/20180117/20141115T000000Z RUN
HSC/calib/gen2/20180117/20141121T000000Z RUN
HSC/calib/gen2/20180117/20141123T000000Z RUN
HSC/calib/gen2/20180117/20141221T000000Z RUN
HSC/calib/gen2/20180117/20150116T000000Z RUN
HSC/calib/gen2/20180117/20150121T000000Z RUN
HSC/calib/gen2/20180117/20150122T000000Z RUN
HSC/calib/gen2/20180117/20150127T000000Z RUN
HSC/calib/gen2/20180117/20150216T000000Z RUN
HSC/calib/gen2/20180117/20150323T000000Z RUN
HSC/calib/gen2/20180117/20150326T000000Z RUN
HSC/calib/gen2/20180117/20150331T000000Z RUN
HSC/calib/gen2/20180117/20150407T000000Z RUN
HSC/calib/gen2/20180117/20150417T000000Z RUN
HSC/calib/gen2/20180117/20150512T000000Z RUN
HSC/calib/gen2/20180117/20150513T000000Z RUN
HSC/calib/gen2/20180117/20150514T000000Z RUN
HSC/calib/gen2/20180117/20150518T000000Z RUN
HSC/calib/gen2/20180117/20150701T000000Z RUN
HSC/calib/gen2/20180117/unbounded RUN
HSC/fgcmcal/lut/RC2/DM-28636 RUN
HSC/masks/s18a RUN
HSC/raw/all RUN
refcats/DM-28636 RUN
skymaps RUN
HSC/calib/DM-28636 CALIBRATION
HSC/calib/gen2/20180117 CALIBRATION
HSC/raw/RC2/9615 TAGGED
HSC/raw/RC2/9697 TAGGED
HSC/raw/RC2/9813 TAGGED
HSC/calib CHAINED
HSC/calib/gen2/20180117 CALIBRATION
HSC/calib/DM-28636 CALIBRATION
HSC/calib/gen2/20180117/unbounded RUN
HSC/calib/DM-28636/unbounded RUN
HSC/fgcmcal/lut/RC2 CHAINED
HSC/fgcmcal/lut/RC2/DM-28636 RUN
HSC/masks CHAINED
HSC/masks/s18a RUN
HSC/raw/RC2 CHAINED
HSC/raw/RC2/9615 TAGGED
HSC/raw/RC2/9697 TAGGED
HSC/raw/RC2/9813 TAGGED
refcats CHAINED
refcats/DM-28636 RUN
HSC/RC2/defaults CHAINED
HSC/raw/RC2 CHAINED
HSC/raw/RC2/9615 TAGGED
HSC/raw/RC2/9697 TAGGED
HSC/raw/RC2/9813 TAGGED
HSC/calib CHAINED
HSC/calib/gen2/20180117 CALIBRATION
HSC/calib/DM-28636 CALIBRATION
HSC/calib/gen2/20180117/unbounded RUN
HSC/calib/DM-28636/unbounded RUN
HSC/masks CHAINED
HSC/masks/s18a RUN
HSC/fgcmcal/lut/RC2 CHAINED
HSC/fgcmcal/lut/RC2/DM-28636 RUN
refcats CHAINED
refcats/DM-28636 RUN
skymaps RUN
HSC/calib/unbounded CHAINED
HSC/calib/DM-28636/unbounded RUN
HSC/calib/gen2/20180117/unbounded RUN
Now I know how to ingest data type and collection chain. (In case of bias data)
# ingest bias
butler register-dataset-type $REPO bias ExposureF detector --is-calibration
# butler collection-chain REPO PARENT child1,child2 child3
butler collection-chain $REPO newCollectionName commaSeparatedCollections
- and I run butler query-collections $REPO --chains TREE. results show below:
Name Type
--------------------------------------------------- -----------
refcats/gen2 RUN
DECam/raw/all RUN
DECam/calib CALIBRATION
DECam/calib/unbounded RUN
DECam/calib/curated/19700101T000000Z RUN
DECam/calib/curated/20130115T013000Z RUN
DECam/calib/curated/20130916T092600Z RUN
DECam/calib/curated/20140117T012900Z RUN
DECam/calib/curated/20141020T103000Z RUN
DECam/calib/curated/20150105T011500Z RUN
DECam/calib/curated/20131130T000000Z RUN
skymaps RUN
HSC/calib CALIBRATION
HSC/calib/unbounded RUN
HSC/calib/curated/19700101T000000Z RUN
HSC/calib/curated/20130131T000000Z RUN
HSC/calib/curated/20140403T000000Z RUN
HSC/calib/curated/20140601T000000Z RUN
HSC/calib/curated/20151106T000000Z RUN
HSC/calib/curated/20160401T000000Z RUN
HSC/calib/curated/20161122T000000Z RUN
HSC/calib/curated/20161223T000000Z RUN
HSC/raw/all RUN
u/czw/DM-28920/biasGen.20150326a CHAINED
u/czw/DM-28920/biasGen.20150326a/20220406T081940Z RUN
HSC/raw/all RUN
HSC/calib CALIBRATION
u/czw/DM-28920/biasGen.20150326a/20220406T081940Z RUN
HSC/biasGen.20150326a CHAINED
HSC/biasGen.20150326a/20220406T083611Z RUN
HSC/biasGen.20150326a/20220406T083107Z RUN
HSC/raw/all RUN
HSC/calib CALIBRATION
HSC/biasGen.20150326a/20220406T083107Z RUN
HSC/biasGen.20150326a/20220406T083611Z RUN
HSC/Bell_bias/20150312 RUN
HSC/Bell_bias CALIBRATION
HSC/Bell_flat/20150326 RUN
HSC/Bell_flat CALIBRATION
HSC/default/yibo CHAINED
HSC/raw/all RUN
HSC/calib CALIBRATION
HSC/calib/curated/19700101T000000Z RUN
HSC/calib/unbounded RUN
skymaps RUN
HSC/Bell_bias CALIBRATION
HSC/Bell_flat CALIBRATION
refcats/gen2 RUN
HSC/calib/yibo CALIBRATION
test_target CHAINED
HSC/default/yibo CHAINED
HSC/raw/all RUN
HSC/calib CALIBRATION
HSC/calib/curated/19700101T000000Z RUN
HSC/calib/unbounded RUN
skymaps RUN
HSC/Bell_bias CALIBRATION
HSC/Bell_flat CALIBRATION
refcats/gen2 RUN
Here I encounter an confusion that If arrange my data structure as possible as like rc2_subset, can I use the command below to processess my data?
pipetask run -b $RC2_SUBSET_DIR/SMALL_HSC/butler.yaml \
-p $RC2_SUBSET_DIR/pipelines/DRP.yaml#singleFrame \
-i HSC/RC2/defaults \
-o u/$USER/single_frame \
--register-dataset-types
Question 2：What data type ‘TAGGED’ means? What kind of data should I ingest into ‘TAGGED’, and which command I should ues?