Issues at BIAS and DARK subtractions with hscPipe 6.7, on private cluster and SMOKA Shot data

Support
,
#1

Hi everyone!

This is my first time posting here, and I am on a fairly specific project using old hscPipe 6.7, so I will give a bit of context, then indicate the errors I am getting, and explain what I do not understand. Sorry if this is a very basic issue, but I have narrowed it down as much as I have been able to.

Context: (not very important for the issue itself, but just to explain why all of this) I want to make my own data reduction of some public HSC raw data I extracted from SMOKA (NEP Deep field 2014 observations) originally reduced with hscPipe 6, which I want to reproduce. Another reason why I am using hscPipe6 is because the CentOS of the cluster I have access to is CentOS 6, and according to the documentation in the hscPipe 6 and 8 tutorials, this is the correct pipeline-CentOS correspondence.

Errors:

The main error I am getting, when doing the DARK subtraction after performing the BIAS one, is the following:
Unable to process DataId(initialdata={'visit': 4746, 'field': 'DARK', 'dateObs': '2014-07-05', 'pointing': 916, 'filter': 'HSC-R', 'ccd': 0, 'taiObs': '2014-07-05', 'expTime': 300.0}, tag=set()): Unable to retrieve bias for DataId(initialdata={'visit': 4746, 'field': 'DARK', 'dateObs': '2014-07-05', 'pointing': 916, 'filter': 'HSC-R', 'ccd': 0, 'taiObs': '2014-07-05', 'expTime': 300.0}, tag=set()): No registry for lookup

The main reason why I think this is happening is because I am not obtaining at the BIAS subtraction step the calibRegistry.sqlite3 file as indicated in https://hsc.mtk.nao.ac.jp/pipedoc/pipedoc_6_e/tutorial_e/calib_data.html#dark

However, I am not entirely sure why this is happening. It might be that, after ingesting the images, instead of getting my BIAS/ DARK/ DOMEFLAT/ files within ~/root-directory/CALIB/, I am just having them in ~/root-directory/. Nonetheless, moving these calibration files from the folder inside CALIB/ does not seem to help. If for example, I move the DARK/ folder into CALIB/, then the constructDark.py does not find it.

What I think is happening: There might be some issues with the installation, as I installed it through my --user but not on the cluster’s root; and I didn’t do it through any PBS but through the usual terminal commands. If I try to run the install.sh at the folder where I installed it, I get the following:

[lsuelves@usrint2 HSCPipe_folder]$ ./install.sh This script must not be run twice. (Previous run ended in success)

I am not sure if the issue is in the installation, in the registry creation, or in something I might be missing.

I hope this issue is not something too basic, as I am afraid it can be just some trivial step I am missing. I also sent the same information to the hsc -software helpdesk, so I also hope this is not redundant. If there is some issue with installation or with our cluster, I can easily contact my cluster’s helpdesk, but I wanted to ask here first in case I am missing the point too much.

Thank you for your time,
Cheers,
Luis

#2

Hi Luis,

I have no experience with hscPipe6.7, but I wonder if your issue is related with this stray light on the dark frames with the r-band (https://www.naoj.org/Observing/Instruments/HSC/hsc_announcement_2020apr.pdf). Do you have the same issue if you try the pipeline on data obtained from different observations?

Best,
Yumi

#3

Hi Yumi,

I am currently trying to check different data, as it turned out I was actually using some Test data. Regarding stray light, I have only the STRAY_LIGHT/ y-background data for the Y band, as it is the one included for hscPipe 6.7

Let’s see what happens with the new data, maybe the registry was not working well due to using this Test data and not actual exposures. I will update here with the outcome (and hopefully close the Thread as solved). Thanks for the reply!

Cheers,
Luis

#4

Sounds good. Look forward to your update!

Best,
Yumi

#5

So it turned out the source of the problem was not the data, but the fact that I was simply skipping one step from the tutorial: between Bias and Dark subtraction, the data has to be registered for detrending through the ingestCalibs.py function.
It’s nice to see it solved, but I need to apologice because this was a very simple and careless error on my side :S

Nonetheless, thank you for the help!
Cheers,
Luis

1 Like
#6

No worries! Glad you hunt down the root cause of the problem.

Best,
Yumi