Error in single frame processing of HSC data

[HJKim181213 (HJ Kim)]

Hi all,

I’m trying to reduce some HSC data and I couldn’t find a proper way to handle these type of data.
But this guideline for DECAM (https://hackmd.io/@lsk/merian#Merian-Data-Processing-Using-the-LSST-Science-Pipelines) was extremely useful to have overview the entire process.

The problem is here:
I excuted the following Butler command for DRP-Prod.yaml#Step1
which is analogous to the single frame processing,

INPUT=HSC/raw/all,HSC/calib,HSC/calib/20191226_calibs,HSC/calib/20200117_calibs,refcats,skymaps

OUTPUT=/HSC/runs/20230822

DATAQUERY="exposure.observation_type=‘science’ AND detector.id!=9 AND detector.id<104 AND physical_filter=‘HSC-G’ AND exposure.day_obs=20191229”

pipetask run --register-dataset-types -j 20 \

-b $REPO --instrument lsst.obs.subaru.HyperSuprimeCam \

-i $INPUT \

-o $OUTPUT \

-p $DRP_PIPE_DIR/pipelines/HSC/DRP-Prod.yaml#step1 \

-d "instrument='HSC' AND $DATAQUERY”

And after consuming few hours, some type of error messages repeatedly appeared in log.
The following is one of them:

lsst.calibrate.deblend WARNING: Parent 85501795157148017: skipping large footprint (area: 17434)
lsst.calibrate.deblend WARNING: Parent 85501795157148115: skipping large footprint (area: 89756)
lsst.calibrate.deblend WARNING: Parent 85501795157148177: skipping large footprint (area: 22897)
lsst.calibrate.deblend WARNING: Parent 85501795157148218: skipping large footprint (area: 11585)
lsst.calibrate.deblend WARNING: Parent 85501795157148349: skipping large footprint (area: 10861)
lsst.calibrate.deblend WARNING: Parent 85501795157148393: skipping masked footprint (area: 751)
lsst.calibrate.deblend WARNING: Parent 85501795157148402: skipping large footprint (area: 31494)
lsst.calibrate.deblend WARNING: Parent 85501795157148497: skipping large footprint (area: 45454)
lsst.calibrate.deblend INFO: Deblended: of 1018 sources, 142 were deblended, creating 351 children, total 1369 sources
lsst.calibrate.measurement INFO: Measuring 1369 sources (1018 parents, 351 children) 
lsst.ctrl.mpexec.singleQuantumExecutor INFO: Preparing execution of quantum for label=calibrate dataId={instrument: 'HSC', detector: 99, visit: 199062, ...}.
lsst.ctrl.mpexec.singleQuantumExecutor INFO: Constructing task and executing quantum for label=calibrate dataId={instrument: 'HSC', detector: 99, visit: 199062, ...}.
lsst.calibrate.detection INFO: Setting factor for negative detections equal to that for positive detections: 1.000000
lsst.calibrate.detection INFO: Detected 1679 positive peaks in 640 footprints and 6 negative peaks in 6 footprints to 5 +ve and 5 -ve sigma
lsst.calibrate.detection INFO: Resubtracting the background after object detection
lsst.calibrate.skySources INFO: Added 100 of 100 requested sky sources (100%)
lsst.calibrate.deblend INFO: Deblending 740 sources
lsst.calibrate.deblend WARNING: Parent 85496690588516390: skipping large footprint (area: 133999)
lsst.calibrate.deblend WARNING: Parent 85496690588516394: skipping large footprint (area: 21112)
lsst.calibrate.deblend WARNING: Parent 85496690588516510: skipping masked footprint (area: 8996)
lsst.calibrate.deblend WARNING: Parent 85496690588516534: skipping masked footprint (area: 1155)
lsst.calibrate.deblend WARNING: Parent 85496690588516688: skipping large footprint (area: 14140)
lsst.calibrate.deblend WARNING: Parent 85496690588516697: skipping masked footprint (area: 4471)
lsst.calibrate.deblend WARNING: Parent 85496690588516752: skipping large footprint (area: 19203)
lsst.calibrate.deblend WARNING: Parent 85496690588516754: skipping large footprint (area: 343184)
lsst.calibrate.deblend WARNING: Parent 85496690588516823: skipping masked footprint (area: 3264)
lsst.calibrate.deblend WARNING: Parent 85496690588516878: skipping masked footprint (area: 2574)
lsst.calibrate.deblend WARNING: Parent 85496690588516915: skipping masked footprint (area: 1536)
lsst.calibrate.deblend WARNING: Parent 85496690588516934: skipping large footprint (area: 17566)
lsst.calibrate.deblend WARNING: Parent 85496690588516962: skipping large footprint (area: 13628)
lsst.calibrate.deblend WARNING: Parent 85496690588516966: skipping large footprint (area: 366071)
lsst.calibrate.deblend WARNING: Parent 85496690588516984: skipping large footprint (area: 11421)
lsst.calibrate.deblend WARNING: Parent 85496690588516990: skipping masked footprint (area: 2296)
lsst.calibrate.deblend INFO: Deblended: of 740 sources, 101 were deblended, creating 243 children, total 983 sources
lsst.calibrate.measurement INFO: Measuring 983 sources (740 parents, 243 children) 
lsst.calibrate.applyApCorr INFO: Applying aperture corrections to 3 instFlux fields
lsst.calibrate INFO: Copying flags from icSourceCat to sourceCat for 160 sources
lsst.calibrate.astrometry INFO: Purged 1165 sources, leaving 204 good sources
lsst.calibrate INFO: Loading reference objects from ps1_pv3_3pi_20170110 in region bounded by [336.81825179, 337.07469450], [17.59343796, 17.82537638] RA Dec
lsst.ctrl.mpexec.singleQuantumExecutor ERROR: Execution of task 'calibrate' on quantum {instrument: 'HSC', detector: 76, visit: 199074, ...} failed. Exception ValueError: No version number found in refcat header metadata. Version 0 refcats are no longer supported: refcat fluxes must have nJy units.
Process task-{instrument: 'HSC', detector: 76, visit: 199074, ...}:
Traceback (most recent call last):
  File "/Volumes/JBOD/hjkim/lsst_stack/stack/miniconda3-py38_4.9.2-7.0.1/DarwinX86/meas_algorithms/g6f40f426a6+c40c173807/python/lsst/meas/algorithms/loadReferenceObjects.py", line 66, in getFormatVersionFromRefCat
    version = md.getScalar("REFCAT_FORMAT_VERSION")
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Volumes/JBOD/hjkim/lsst_stack/stack/miniconda3-py38_4.9.2-7.0.1/DarwinX86/daf_base/gf041782ebf+a8f2e22b1e/python/lsst/daf/base/propertyContainer/propertyContainerContinued.py", line 877, in getScalar
    return _propertyContainerGet(self, name, returnStyle=ReturnStyle.SCALAR)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Volumes/JBOD/hjkim/lsst_stack/stack/miniconda3-py38_4.9.2-7.0.1/DarwinX86/daf_base/gf041782ebf+a8f2e22b1e/python/lsst/daf/base/propertyContainer/propertyContainerContinued.py", line 214, in _propertyContainerGet
    raise KeyError(name + " not found")
KeyError: 'REFCAT_FORMAT_VERSION not found'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Volumes/JBOD/hjkim/lsst_stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-7.0.1/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/Volumes/JBOD/hjkim/lsst_stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-7.0.1/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Volumes/JBOD/hjkim/lsst_stack/stack/miniconda3-py38_4.9.2-7.0.1/DarwinX86/ctrl_mpexec/g8a7d0b7f27+3adfb552c9/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 162, in _executeJob
    quantumExecutor.execute(taskDef, quantum)
  File "/Volumes/JBOD/hjkim/lsst_stack/stack/miniconda3-py38_4.9.2-7.0.1/DarwinX86/ctrl_mpexec/g8a7d0b7f27+3adfb552c9/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 161, in execute
    result = self._execute(taskDef, quantum)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Volumes/JBOD/hjkim/lsst_stack/stack/miniconda3-py38_4.9.2-7.0.1/DarwinX86/ctrl_mpexec/g8a7d0b7f27+3adfb552c9/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 258, in _execute
    self.runQuantum(task, quantum, taskDef, limited_butler)
  File "/Volumes/JBOD/hjkim/lsst_stack/stack/miniconda3-py38_4.9.2-7.0.1/DarwinX86/ctrl_mpexec/g8a7d0b7f27+3adfb552c9/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 460, in runQuantum
    task.runQuantum(butlerQC, inputRefs, outputRefs)
  File "/Volumes/JBOD/hjkim/lsst_stack/stack/miniconda3-py38_4.9.2-7.0.1/DarwinX86/pipe_tasks/ga35e63b1f0+af69103581/python/lsst/pipe/tasks/calibrate.py", line 481, in runQuantum
    outputs = self.run(**inputs)
              ^^^^^^^^^^^^^^^^^^
  File "/Volumes/JBOD/hjkim/lsst_stack/stack/miniconda3-py38_4.9.2-7.0.1/DarwinX86/utils/gbd07d1bd26+81bc2a20b4/python/lsst/utils/timer.py", line 295, in timeMethod_wrapper
    res = func(self, *args, **keyArgs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Volumes/JBOD/hjkim/lsst_stack/stack/miniconda3-py38_4.9.2-7.0.1/DarwinX86/pipe_tasks/ga35e63b1f0+af69103581/python/lsst/pipe/tasks/calibrate.py", line 600, in run
    astromRes = self.astrometry.run(
                ^^^^^^^^^^^^^^^^^^^^
  File "/Volumes/JBOD/hjkim/lsst_stack/stack/miniconda3-py38_4.9.2-7.0.1/DarwinX86/utils/gbd07d1bd26+81bc2a20b4/python/lsst/utils/timer.py", line 295, in timeMethod_wrapper
    res = func(self, *args, **keyArgs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Volumes/JBOD/hjkim/lsst_stack/stack/miniconda3-py38_4.9.2-7.0.1/DarwinX86/meas_astrom/g0a0026dc87+46ae74ba8d/python/lsst/meas/astrom/astrometry.py", line 179, in run
    res = self.solve(exposure=exposure, sourceCat=sourceCat)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Volumes/JBOD/hjkim/lsst_stack/stack/miniconda3-py38_4.9.2-7.0.1/DarwinX86/utils/gbd07d1bd26+81bc2a20b4/python/lsst/utils/timer.py", line 295, in timeMethod_wrapper
    res = func(self, *args, **keyArgs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Volumes/JBOD/hjkim/lsst_stack/stack/miniconda3-py38_4.9.2-7.0.1/DarwinX86/meas_astrom/g0a0026dc87+46ae74ba8d/python/lsst/meas/astrom/astrometry.py", line 225, in solve
    loadRes = self.refObjLoader.loadPixelBox(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Volumes/JBOD/hjkim/lsst_stack/stack/miniconda3-py38_4.9.2-7.0.1/DarwinX86/meas_algorithms/g6f40f426a6+c40c173807/python/lsst/meas/algorithms/loadReferenceObjects.py", line 603, in loadPixelBox
    return self.loadRegion(outerSkyRegion, filterName, filtFunc=_filterFunction, epoch=epoch)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Volumes/JBOD/hjkim/lsst_stack/stack/miniconda3-py38_4.9.2-7.0.1/DarwinX86/meas_algorithms/g6f40f426a6+c40c173807/python/lsst/meas/algorithms/loadReferenceObjects.py", line 695, in loadRegion
    version = getFormatVersionFromRefCat(refCat)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Volumes/JBOD/hjkim/lsst_stack/stack/miniconda3-py38_4.9.2-7.0.1/DarwinX86/meas_algorithms/g6f40f426a6+c40c173807/python/lsst/meas/algorithms/loadReferenceObjects.py", line 72, in getFormatVersionFromRefCat
    raise ValueError(f"No version number found in refcat header metadata. {errMsg}")
ValueError: No version number found in refcat header metadata. Version 0 refcats are no longer supported: refcat fluxes must have nJy units.

I guess that this is about the reference catalog, but I thought that I properly ingested the one (ps1_pv3_3pi_20170110)
I have no idea what “Version 0 refcats are no longer supported” means

Thank you for your help.

[ameisner (Aaron Meisner)]

Hi HJ,

Thanks for your message.

Might you perhaps be able to specify the version of the LSST pipelines that you’re using when running this command? I was not immediately able to determine the LSST pipelines version from the traceback provided.

I would also note that the LSST pipelines documentation includes a getting started tutorial based on HSC data rather than DECam:

https://pipelines.lsst.io/#getting-started

@HJKim181213

And after consuming few hours

Maybe you could iterate faster by further restricting the data query? It looks like your command is trying to process a full night of exposures (?).

Lastly, I found this commit from a couple months ago described as “Remove support for version 0 refcats”:

Remove support for version 0 refcats

I’m tagging the commit’s author @parejkoj in case they can provide any further insight.

[jbosch (Jim Bosch)]

I’m pretty sure that the commit @ameisner identified is indeed the issue: your refcat files are too old, and you’ll need to either upgrade them or get new ones. I suspect there are instructions for upgrading them somewhere, but I don’t personally know where. And if you originally downloaded your current refcats from some location maintained by Rubin people, there may be new ones there by now.

[parejkoj (John Parejko)]

The ps1 refcats at NSCA (and copied to USDF) were converted to version 1 in April 2019. I don’t know if we still have a public place to download them from (131k files, 400GB), but we probably should (for these and our htm-sharded Gaia DR2/DR3 files as well). We used to host a download link for PS1 on NCSA.

To run the convert script, you’ll have to use an older science pipelines release: I don’t know how far back you need to go to get a working version of the convert_refcat_to_nJy.py script, but I believe it had bit-rotted before I removed that script at the end of June on DM-34793.

Previous Community posts related to this:

PhotoCalib has replaced Calib: welcoming our nanojansky overlords

The converted refcats are available on lsst machines at /datasets/refcats/htm/v1/ Please update your symlink if you use one. The symlink at /datasets/hsc/repo/ref_cats has been updated to point there.

Pan-STARRS reference catalog in LSST format

The Pan-STARRS 3pi stack data (images and catalogs) were released to the world as DR1 on 2016 December 19. This provides a high-quality network of calibration sources over three quarters of the sky to a modest depth. Telescopes both large and small (depths overlapping i ~ 17 to 21 mag) observing fields north of Dec=-30 deg in the optical will have good PS1 calibration sources in the field, observed through the same column as the target sources, allowing simple relative calibration for astromet…

[ryanlau]

Hi @HJKim181213 , just following up on your issue to see if you’ve had any success with any of the suggestions above.

[HJKim181213 (HJ Kim)]

Hi @ameisner, @jbosch and @parejkoj.

It took some time to specify the problem, but finally I fixed it and your suggestions were right.
First of all, currently I’m using 25.0.0 version, which was probably the latest release when I installed it.
My reference catalog was too old to be supported. What I tried is to install older version of the pipeline(24.0.0) and run the convert_refcat_to_nJy.pyscript to convert it to have nJy units.

I tried running the pipetask command(step 1) again, it ran successfully without any problems.

Thank you again for your help.
HJ Kim

[ameisner (Aaron Meisner)]

Thanks for the update @HJKim181213 ! Glad that step1 is running successfully now. I’ll mark the response from @parejkoj as the solution.