Error at ingest-raws step

Data Management
#1

Hello,
I am working for Euclid and I have in charge the data processing interface between Rubin and Euclid.
I am currently focusing on the migration to gen3 butler and I encounter an error at the butler ingest-raws step.
Another important information is that we are using our own simulations of Rubin (based on galsim) and there will certainly be some improvements to do to our simulator.

butler ingest-raws butler lsst_a_138098_R22_S01.fits

ingest INFO: Successfully extracted metadata from 1 file with 0 failures
ingest WARNING: Exposure LSSTCam-imSim:138098 could not be registered: Conflict in sync for table exposure on column(s) sky_angle: 318.763674255571 != 318.764306085962.
ingest INFO: Successfully processed data from 0 exposures with 1 failure from exposure registration and 0 failures from file ingest.
ingest INFO: Ingested 0 distinct Butler datasets
lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback:
Traceback (most recent call last):
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2021_41/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/obs_base/22.0.1-20-g59bf897+ddb5b0fb80/python/lsst/obs/base/cli/cmd/commands.py", line 118, in ingest_raws
    script.ingestRaws(*args, **kwargs)
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2021_41/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/obs_base/22.0.1-20-g59bf897+ddb5b0fb80/python/lsst/obs/base/script/ingestRaws.py", line 71, in ingestRaws
    ingester.run(locations, run=output_run, processes=processes, file_filter=regex)
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2021_41/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/pipe_base/22.0.1-26-ge0dcd2c+2fb927f3cf/python/lsst/pipe/base/timer.py", line 181, in wrapper
    res = func(self, *args, **keyArgs)
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2021_41/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/obs_base/22.0.1-20-g59bf897+ddb5b0fb80/python/lsst/obs/base/ingest.py", line 1137, in run
    raise RuntimeError("Some failures encountered during ingestion")
RuntimeError: Some failures encountered during ingestion

Does somebody know what is this sky_angle and how it is computed ? Is there a way to ignore this error ?
Thank you very much for your help !
Rémi

#2

There are two reasons for this sort of error:

  1. You are using an old version of Postgres that does not have enough accuracy in the floating point support. I can’t remember which version has the problem but we have definitely seen this in the past where storing a double precision float turns it into single precision.
  2. The headers relating to this angle are inconsistent for different detectors within the same exposure. Usually this is difficult to achieve.

It’s unlikely the imsim files are bad since I assume we have successfully ingested them here, so option 1 seems the most likely.

So the header to check is ROTANGLE.

#3

Ok thank you very much.
Is it correct to say that sky_angle is the angle between the pixel Y axis and the sky North axis ?

sky_angle