Encounter "No registry for lookup" when I process flat in hscPipe 8.3

Support
#1

I am playing new hscPipe 8.3 with my previous codes, but I immediately encounter a error. Here are my codes:

hscIngestImages.py $WORKDIR  /array2/RAW/run_201507/20150709-flati/*.fits
constructFlat.py $WORKDIR --rerun $CALVRERUN --calib=$WORKDIR/CALIB  --id visit=33138..33156:2 --config isr.doBias=False isr.doDark=False isr.doFringe=False isr.doAttachTransmissionCurve=False --job $CALVRERUN-g --time 300 --clobber-config --cores $RCHP5

But I got following error:

111236 WARN 2020-02-24T08:22:11.763-0500 bias: Unable to process DataId(initialdata={‘visit’: 33396, ‘field’: ‘BIAS’, ‘dateObs’: ‘2015-07-10’, ‘pointing’: 1286, ‘filter’: ‘HSC-I’, ‘ccd’: 3, ‘taiObs’: ‘2015-07-10’, ‘expTime’: 0.0}, tag=set()): No registry for lookup
111237 WARN 2020-02-24T08:22:11.770-0500 bias: Unable to process DataId(initialdata={‘visit’: 33398, ‘field’: ‘BIAS’, ‘dateObs’: ‘2015-07-10’, ‘pointing’: 1286, ‘filter’: ‘HSC-I’, ‘ccd’: 3, ‘taiObs’: ‘2015-07-10’, ‘expTime’: 0.0}, tag=set()): No registry for lookup
[mpiexec@ichiro.asiaa.sinica.edu.tw] HYDT_bscd_pbs_wait_for_completion (tools/bootstrap/external/pbs_wait.c:67): tm_poll(obit_event) failed with TM error 17002
[mpiexec@ichiro.asiaa.sinica.edu.tw] HYDT_bsci_wait_for_completion (tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion
[proxy:0:0@ichiro.asiaa.sinica.edu.tw] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:887): assert (!closed) failed
[proxy:0:0@ichiro.asiaa.sinica.edu.tw] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status
[proxy:0:0@ichiro.asiaa.sinica.edu.tw] main (pm/pmiserv/pmip.c:202): demux engine error waiting for event

(I guess) Because I didn’t apply BIAS and DARK, no calibRegistry.sqlite3 in -calib=$WORKDIR/CALIB when I try to process FLAT. However, I ran the same procedure in hscPipe 7.9.1 without issue, so I have no idea what happened here.

#2

This is due to changes in how defects are handled. Here’s how to start a calib repo these days:

mkdir -p /path/to/calibs
# Grab some products directly from a previous calib repo
cp -r /path/to/otherCalibs/BFKERNEL /path/to/calibs
cp -r /path/to/otherCalibs/STRAY_LIGHT /path/to/calibs
# Install defects
cp -r $OBS_SUBARU_DIR/hsc/CALIB/defects /path/to/calibs
cp -r $OBS_SUBARU_DIR/hsc/CALIB/calibRegistry.sqlite3 /path/to/calibs

You needn’t worry about turning off the attachment of transmission curves if you install them in the data repo:

# Install transmission curves
installTransmissionCurves.py /path/to/dataRepo
#3

Thanks, this solution works.
I have additional two relative questions.
(1) I usually turn off the by isr.doAttachTransmissionCurve=False because I don’t sure if my process really need that. I followed your suggestion to use installTransmissionCurves.py and remove isr.doAttachTransmissionCurve=False, but the process was still ended by this error:

138596 WARN 2020-02-24T18:12:02.227-0500 bias: Unable to process DataId(initialdata={‘visit’: 33394, ‘field’: ‘BIAS’, ‘dateObs’: ‘2015-07-10’, ‘pointing’: 1286, ‘filter’: ‘HSC-I’, ‘ccd’: 1, ‘taiObs’: ‘2015-07-10’, ‘expTime’: 0.0}, tag=set()): No locations for get: datasetType:transmission_optics dataId:DataId(initialdata={‘visit’: 33394, ‘field’: ‘BIAS’, ‘dateObs’: ‘2015-07-10’, ‘pointing’: 1286, ‘filter’: ‘HSC-I’, ‘ccd’: 1, ‘taiObs’: ‘2015-07-10’, ‘expTime’: 0.0}, tag=set())

Do you have a sample to set up TransmissionCurve appropriately?

(2) I did setup BFKERNEL in previous process by:

ln -s $OBS_SUBARU_DIR/8.3-hsc/hsc/brighter_fatter_kernel.pkl  $WORKDIR/CALIB/BFKERNEL/brighter_fatter_kernel.pkl

But I have no idea about STRAY_LIGHT, any suggestion?

#4

Oh! Sorry, I got the transmission curve installation wrong. It gets installed into the data repo, not the calib repo. I’ll edit the instructions above.

The stray light correction is only required for y-band data taken earlier than 2018-01-01.