Something wrong when I ingest-raws $REPO $BIASFILES

Data Management
#1

Hi guys, sorry for my poor english. Hope u can understand my problem

env: g4eaec17cb9+1418e6867f current w_latest w_2022_13 setup
I successfuly ingest the scientific raw image by :
‘’’’
$ LOGFILE=$LOGDIR/ingest_science_202102.log;
$ SCIFILES=/data/share/yyb/Subaru_data/Bell_g_data/*.fits;
$ date | tee $LOGFILE;
$ butler ingest-raws $REPO $SCIFILES --transfer link
‘’’

Then I try to constuct Bias file by constructBias.py:
‘’’
$constructBias.py HSC/ --calib HSC/CALIB/ --rerun calib --id visit=26162 ccd=0…8^10…111 --batch-type=smp --cores=8

‘’’
But when I ingest Bias data, something wrong happend. But I don’t know how to solve it

(lsst-scipipe-3.0.0) [yyb@raku HSC]$ butler ingest-raws $REPO $BIASFILES --transfer link
WARNING: version mismatch between CFITSIO header (v4.000999999999999) and linked library (v4.01).

WARNING: version mismatch between CFITSIO header (v4.000999999999999) and linked library (v4.01).

WARNING: version mismatch between CFITSIO header (v4.000999999999999) and linked library (v4.01).

lsst.ingest INFO: Successfully extracted metadata from 111 files with 0 failures
lsst.ingest WARNING: Exposure HSC:HSCA02616200 could not be registered: Conflict in sync for table exposure on column(s) exposure_time: 30.0 != 0.0, dark_time: 30.0 != 0.0.
lsst.ingest INFO: Successfully processed data from 0 exposures with 1 failure from exposure registration and 0 failures from file ingest.
lsst.ingest INFO: Ingested 0 distinct Butler datasets
lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback:
Traceback (most recent call last):
File “/data/share/yyb/lsst_stack/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/obs_base/g7a69c27ea0+5c56388c9f/python/lsst/obs/base/cli/cmd/commands.py”, line 141, in ingest_raws
script.ingestRaws(*args, **kwargs)
File “/data/share/yyb/lsst_stack/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/obs_base/g7a69c27ea0+5c56388c9f/python/lsst/obs/base/script/ingestRaws.py”, line 92, in ingestRaws
ingester.run(
File “/data/share/yyb/lsst_stack/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/utils/gc5abf31359+9633a190c8/python/lsst/utils/timer.py”, line 345, in timeMethod_wrapper
res = func(self, *args, **keyArgs)
File “/data/share/yyb/lsst_stack/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/obs_base/g7a69c27ea0+5c56388c9f/python/lsst/obs/base/ingest.py”, line 1281, in run
raise RuntimeError(“Some failures encountered during ingestion”)
RuntimeError: Some failures encountered during ingestion

#2

The problem here is that it seems you’ve already ingested either one of these files, or a different file from one of the same exposures, and the exposure time and dark time are inconsistent between the two.

If you’ve never ingesting anything into this data repository before, that probably means the FITS metadata for one of the files is bad and needs to be fixed; there is a patch system in the astro_metadata_translator package that could be used to do that without modifying any of the files (which is what we would recommend). If you have ingested files into this data repository, there are more possibilities, but it’d be helpful to know what you’ve tried to do so far before speculating on them.

1 Like
#3

This is a Gen2 middleware command. The new Gen3 infrastructure uses an entirely different approach to running pipelines. It uses the pipetask command to run pipelines that are constructed using YAML specifications. The bias construction pipeline is in the cp_pipe package.

As @jbosch says, there is a header problem. A previous file for that observation_id was calculated to have a different exposure time than this file you are trying to ingest. You can use astrometadata translate on the collection of files to inspect the specific translation output.

1 Like
#4

If you are interested in using the Gen3 infrastructure to construct calibrations (including a combined bias from individual biases), there are some examples on this page: Constructing calibrations — LSST Science Pipelines

1 Like