In DECam data reduction, I faced a FileNotFoundError in "Ingest Raw Images" step

Support
,
#1

Hello,

I have initiated DECam data reduction by adhering to the guidelines provided in this tutorial: The Gen3 workflow — LSST Science Pipelines

I faced an error when I ran this code
: pipetask run -b REPO -i DECam/raw/all -o DECam/raw/crosstalk-sources -p $CP_PIPE_DIR/pipelines/DarkEnergyCamera/RunIsrForCrosstalkSources.yaml --register-dataset-types

This is an error message: FileNotFoundError: Not enough datasets (0) found for non-optional connection overscan.camera (camera) with minimum=1 for quantum data ID {instrument: ‘DECam’, detector: 1, exposure: 516081, …}.

I speculate that the error might have been due to the fact that I skipped executing line B and only ran line A.
line A: butler ingest-raws REPO /path/to/raw/science/.fits.fz --transfer link
line B: butler ingest-raws REPO /path/to/raw/calibs/.fits.fz --transfer link

Since I only possess the original image and lack a bias, flat image, I was unable to run line B.

*As a trial, I ran the process with 3 files of r filter.

Being my first attempt, I am uncertain if I have stated my query correctly. Any corrections or feedback would be greatly appreciated.

Best regards,
Boseong

#2

Did you run the butler write-curated-calibrations line? That will register the camera dataset.

1 Like
#3

You will also need the image calibrations (flats, bias etc) otherwise you will not be able to run single frame processing.

1 Like
#4

Thank you for replying.

Yes, I realized that I had not included any calibration files, so I attempted to download these files.

However, I could not find any file corresponding to the raw file in the NOIRLAB astro data archive.

For example, In NOIRLAB astro data archive, I set a coordinate (104.611, -55.948) and filter(r DECam SDSS c0002 6415.0 1480.0) and instrument(ct4m, decam) only.

Among the raw files, I found only object files. I don’t believe that flat and bias files are categorized under ‘instcal’. Therefore, it appears that there are no matching flat and bias files for the specific raw object file.

Best regards,
Boseong

#5

You are going to need to either construct the flats and biases yourselves or else find some valid ones from somewhere. If you are bringing in external flats/biases you have to be careful that they match the form expected by the LSST science pipelines. Maybe @lskelvin can give you some pointers.

See also this post and the discussion about Merian:

1 Like
#6

Thank you for providing such informative tutorial page!

Fortunately, I resolved the issue using the information from this community page: https://community.lsst.org/t/issues-with-ingestcalib-py-using-decam-images/5602/13?u

I only had to change two things:

  1. As you advised, I used the following command to input the zero(bias) and flat(dome flat) data (which I had downloaded from an archive): butler ingest-raws REPO /path/to/raw/calibs/*.fits.fz --transfer link

  2. I appended “,/DECam/calib” right after “DECam/raw/all” in the following command:
    Before: pipetask run -b REPO -i DECam/raw/all -o DECam/raw/crosstalk-sources -p $CP_PIPE_DIR/pipelines/DarkEnergyCamera/RunIsrForCrosstalkSources.yaml --register-dataset-types
    After: pipetask run -b REPO -i DECam/raw/all,DECam/calib -o DECam/raw/crosstalk-sources -p $CP_PIPE_DIR/pipelines/DarkEnergyCamera/RunIsrForCrosstalkSources.yaml --register-dataset-types

I will continue to follow the tutorial page that you provided.

Thank you for your help.

Best regards,
Boseong