Error in single frame processing of HSC data

Hi all,

I’m trying to reduce some HSC data and I couldn’t find a proper way to handle these type of data.
But this guideline for DECAM ( was extremely useful to have overview the entire process.

The problem is here:
I excuted the following Butler command for DRP-Prod.yaml#Step1
which is analogous to the single frame processing,



DATAQUERY="exposure.observation_type=‘science’ AND!=9 AND<104 AND physical_filter=‘HSC-G’ AND exposure.day_obs=20191229”

pipetask run --register-dataset-types -j 20 \

-b $REPO --instrument lsst.obs.subaru.HyperSuprimeCam \

-i $INPUT \

-o $OUTPUT \

-p $DRP_PIPE_DIR/pipelines/HSC/DRP-Prod.yaml#step1 \

-d "instrument='HSC' AND $DATAQUERY”

And after consuming few hours, some type of error messages repeatedly appeared in log.
The following is one of them:

lsst.calibrate.deblend WARNING: Parent 85501795157148017: skipping large footprint (area: 17434)
lsst.calibrate.deblend WARNING: Parent 85501795157148115: skipping large footprint (area: 89756)
lsst.calibrate.deblend WARNING: Parent 85501795157148177: skipping large footprint (area: 22897)
lsst.calibrate.deblend WARNING: Parent 85501795157148218: skipping large footprint (area: 11585)
lsst.calibrate.deblend WARNING: Parent 85501795157148349: skipping large footprint (area: 10861)
lsst.calibrate.deblend WARNING: Parent 85501795157148393: skipping masked footprint (area: 751)
lsst.calibrate.deblend WARNING: Parent 85501795157148402: skipping large footprint (area: 31494)
lsst.calibrate.deblend WARNING: Parent 85501795157148497: skipping large footprint (area: 45454)
lsst.calibrate.deblend INFO: Deblended: of 1018 sources, 142 were deblended, creating 351 children, total 1369 sources
lsst.calibrate.measurement INFO: Measuring 1369 sources (1018 parents, 351 children) 
lsst.ctrl.mpexec.singleQuantumExecutor INFO: Preparing execution of quantum for label=calibrate dataId={instrument: 'HSC', detector: 99, visit: 199062, ...}.
lsst.ctrl.mpexec.singleQuantumExecutor INFO: Constructing task and executing quantum for label=calibrate dataId={instrument: 'HSC', detector: 99, visit: 199062, ...}.
lsst.calibrate.detection INFO: Setting factor for negative detections equal to that for positive detections: 1.000000
lsst.calibrate.detection INFO: Detected 1679 positive peaks in 640 footprints and 6 negative peaks in 6 footprints to 5 +ve and 5 -ve sigma
lsst.calibrate.detection INFO: Resubtracting the background after object detection
lsst.calibrate.skySources INFO: Added 100 of 100 requested sky sources (100%)
lsst.calibrate.deblend INFO: Deblending 740 sources
lsst.calibrate.deblend WARNING: Parent 85496690588516390: skipping large footprint (area: 133999)
lsst.calibrate.deblend WARNING: Parent 85496690588516394: skipping large footprint (area: 21112)
lsst.calibrate.deblend WARNING: Parent 85496690588516510: skipping masked footprint (area: 8996)
lsst.calibrate.deblend WARNING: Parent 85496690588516534: skipping masked footprint (area: 1155)
lsst.calibrate.deblend WARNING: Parent 85496690588516688: skipping large footprint (area: 14140)
lsst.calibrate.deblend WARNING: Parent 85496690588516697: skipping masked footprint (area: 4471)
lsst.calibrate.deblend WARNING: Parent 85496690588516752: skipping large footprint (area: 19203)
lsst.calibrate.deblend WARNING: Parent 85496690588516754: skipping large footprint (area: 343184)
lsst.calibrate.deblend WARNING: Parent 85496690588516823: skipping masked footprint (area: 3264)
lsst.calibrate.deblend WARNING: Parent 85496690588516878: skipping masked footprint (area: 2574)
lsst.calibrate.deblend WARNING: Parent 85496690588516915: skipping masked footprint (area: 1536)
lsst.calibrate.deblend WARNING: Parent 85496690588516934: skipping large footprint (area: 17566)
lsst.calibrate.deblend WARNING: Parent 85496690588516962: skipping large footprint (area: 13628)
lsst.calibrate.deblend WARNING: Parent 85496690588516966: skipping large footprint (area: 366071)
lsst.calibrate.deblend WARNING: Parent 85496690588516984: skipping large footprint (area: 11421)
lsst.calibrate.deblend WARNING: Parent 85496690588516990: skipping masked footprint (area: 2296)
lsst.calibrate.deblend INFO: Deblended: of 740 sources, 101 were deblended, creating 243 children, total 983 sources
lsst.calibrate.measurement INFO: Measuring 983 sources (740 parents, 243 children) 
lsst.calibrate.applyApCorr INFO: Applying aperture corrections to 3 instFlux fields
lsst.calibrate INFO: Copying flags from icSourceCat to sourceCat for 160 sources
lsst.calibrate.astrometry INFO: Purged 1165 sources, leaving 204 good sources
lsst.calibrate INFO: Loading reference objects from ps1_pv3_3pi_20170110 in region bounded by [336.81825179, 337.07469450], [17.59343796, 17.82537638] RA Dec
lsst.ctrl.mpexec.singleQuantumExecutor ERROR: Execution of task 'calibrate' on quantum {instrument: 'HSC', detector: 76, visit: 199074, ...} failed. Exception ValueError: No version number found in refcat header metadata. Version 0 refcats are no longer supported: refcat fluxes must have nJy units.
Process task-{instrument: 'HSC', detector: 76, visit: 199074, ...}:
Traceback (most recent call last):
  File "/Volumes/JBOD/hjkim/lsst_stack/stack/miniconda3-py38_4.9.2-7.0.1/DarwinX86/meas_algorithms/g6f40f426a6+c40c173807/python/lsst/meas/algorithms/", line 66, in getFormatVersionFromRefCat
    version = md.getScalar("REFCAT_FORMAT_VERSION")
  File "/Volumes/JBOD/hjkim/lsst_stack/stack/miniconda3-py38_4.9.2-7.0.1/DarwinX86/daf_base/gf041782ebf+a8f2e22b1e/python/lsst/daf/base/propertyContainer/", line 877, in getScalar
    return _propertyContainerGet(self, name, returnStyle=ReturnStyle.SCALAR)
  File "/Volumes/JBOD/hjkim/lsst_stack/stack/miniconda3-py38_4.9.2-7.0.1/DarwinX86/daf_base/gf041782ebf+a8f2e22b1e/python/lsst/daf/base/propertyContainer/", line 214, in _propertyContainerGet
    raise KeyError(name + " not found")
KeyError: 'REFCAT_FORMAT_VERSION not found'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Volumes/JBOD/hjkim/lsst_stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-7.0.1/lib/python3.11/multiprocessing/", line 314, in _bootstrap
  File "/Volumes/JBOD/hjkim/lsst_stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-7.0.1/lib/python3.11/multiprocessing/", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Volumes/JBOD/hjkim/lsst_stack/stack/miniconda3-py38_4.9.2-7.0.1/DarwinX86/ctrl_mpexec/g8a7d0b7f27+3adfb552c9/python/lsst/ctrl/mpexec/", line 162, in _executeJob
    quantumExecutor.execute(taskDef, quantum)
  File "/Volumes/JBOD/hjkim/lsst_stack/stack/miniconda3-py38_4.9.2-7.0.1/DarwinX86/ctrl_mpexec/g8a7d0b7f27+3adfb552c9/python/lsst/ctrl/mpexec/", line 161, in execute
    result = self._execute(taskDef, quantum)
  File "/Volumes/JBOD/hjkim/lsst_stack/stack/miniconda3-py38_4.9.2-7.0.1/DarwinX86/ctrl_mpexec/g8a7d0b7f27+3adfb552c9/python/lsst/ctrl/mpexec/", line 258, in _execute
    self.runQuantum(task, quantum, taskDef, limited_butler)
  File "/Volumes/JBOD/hjkim/lsst_stack/stack/miniconda3-py38_4.9.2-7.0.1/DarwinX86/ctrl_mpexec/g8a7d0b7f27+3adfb552c9/python/lsst/ctrl/mpexec/", line 460, in runQuantum
    task.runQuantum(butlerQC, inputRefs, outputRefs)
  File "/Volumes/JBOD/hjkim/lsst_stack/stack/miniconda3-py38_4.9.2-7.0.1/DarwinX86/pipe_tasks/ga35e63b1f0+af69103581/python/lsst/pipe/tasks/", line 481, in runQuantum
    outputs =**inputs)
  File "/Volumes/JBOD/hjkim/lsst_stack/stack/miniconda3-py38_4.9.2-7.0.1/DarwinX86/utils/gbd07d1bd26+81bc2a20b4/python/lsst/utils/", line 295, in timeMethod_wrapper
    res = func(self, *args, **keyArgs)
  File "/Volumes/JBOD/hjkim/lsst_stack/stack/miniconda3-py38_4.9.2-7.0.1/DarwinX86/pipe_tasks/ga35e63b1f0+af69103581/python/lsst/pipe/tasks/", line 600, in run
    astromRes =
  File "/Volumes/JBOD/hjkim/lsst_stack/stack/miniconda3-py38_4.9.2-7.0.1/DarwinX86/utils/gbd07d1bd26+81bc2a20b4/python/lsst/utils/", line 295, in timeMethod_wrapper
    res = func(self, *args, **keyArgs)
  File "/Volumes/JBOD/hjkim/lsst_stack/stack/miniconda3-py38_4.9.2-7.0.1/DarwinX86/meas_astrom/g0a0026dc87+46ae74ba8d/python/lsst/meas/astrom/", line 179, in run
    res = self.solve(exposure=exposure, sourceCat=sourceCat)
  File "/Volumes/JBOD/hjkim/lsst_stack/stack/miniconda3-py38_4.9.2-7.0.1/DarwinX86/utils/gbd07d1bd26+81bc2a20b4/python/lsst/utils/", line 295, in timeMethod_wrapper
    res = func(self, *args, **keyArgs)
  File "/Volumes/JBOD/hjkim/lsst_stack/stack/miniconda3-py38_4.9.2-7.0.1/DarwinX86/meas_astrom/g0a0026dc87+46ae74ba8d/python/lsst/meas/astrom/", line 225, in solve
    loadRes = self.refObjLoader.loadPixelBox(
  File "/Volumes/JBOD/hjkim/lsst_stack/stack/miniconda3-py38_4.9.2-7.0.1/DarwinX86/meas_algorithms/g6f40f426a6+c40c173807/python/lsst/meas/algorithms/", line 603, in loadPixelBox
    return self.loadRegion(outerSkyRegion, filterName, filtFunc=_filterFunction, epoch=epoch)
  File "/Volumes/JBOD/hjkim/lsst_stack/stack/miniconda3-py38_4.9.2-7.0.1/DarwinX86/meas_algorithms/g6f40f426a6+c40c173807/python/lsst/meas/algorithms/", line 695, in loadRegion
    version = getFormatVersionFromRefCat(refCat)
  File "/Volumes/JBOD/hjkim/lsst_stack/stack/miniconda3-py38_4.9.2-7.0.1/DarwinX86/meas_algorithms/g6f40f426a6+c40c173807/python/lsst/meas/algorithms/", line 72, in getFormatVersionFromRefCat
    raise ValueError(f"No version number found in refcat header metadata. {errMsg}")
ValueError: No version number found in refcat header metadata. Version 0 refcats are no longer supported: refcat fluxes must have nJy units.

I guess that this is about the reference catalog, but I thought that I properly ingested the one (ps1_pv3_3pi_20170110) :frowning:
I have no idea what “Version 0 refcats are no longer supported” means

Thank you for your help.

Hi HJ,

Thanks for your message.

Might you perhaps be able to specify the version of the LSST pipelines that you’re using when running this command? I was not immediately able to determine the LSST pipelines version from the traceback provided.

I would also note that the LSST pipelines documentation includes a getting started tutorial based on HSC data rather than DECam:

Maybe you could iterate faster by further restricting the data query? It looks like your command is trying to process a full night of exposures (?).

Lastly, I found this commit from a couple months ago described as “Remove support for version 0 refcats”:

I’m tagging the commit’s author @parejkoj in case they can provide any further insight.

1 Like

I’m pretty sure that the commit @ameisner identified is indeed the issue: your refcat files are too old, and you’ll need to either upgrade them or get new ones. I suspect there are instructions for upgrading them somewhere, but I don’t personally know where. And if you originally downloaded your current refcats from some location maintained by Rubin people, there may be new ones there by now.


The ps1 refcats at NSCA (and copied to USDF) were converted to version 1 in April 2019. I don’t know if we still have a public place to download them from (131k files, 400GB), but we probably should (for these and our htm-sharded Gaia DR2/DR3 files as well). We used to host a download link for PS1 on NCSA.

To run the convert script, you’ll have to use an older science pipelines release: I don’t know how far back you need to go to get a working version of the script, but I believe it had bit-rotted before I removed that script at the end of June on DM-34793.

Previous Community posts related to this:


Hi @HJKim181213 , just following up on your issue to see if you’ve had any success with any of the suggestions above.

Hi @ameisner, @jbosch and @parejkoj.

It took some time to specify the problem, but finally I fixed it and your suggestions were right.
First of all, currently I’m using 25.0.0 version, which was probably the latest release when I installed it.
My reference catalog was too old to be supported. What I tried is to install older version of the pipeline(24.0.0) and run the convert_refcat_to_nJy.pyscript to convert it to have nJy units.

I tried running the pipetask command(step 1) again, it ran successfully without any problems.

Thank you again for your help.
HJ Kim

1 Like

Thanks for the update @HJKim181213 ! Glad that step1 is running successfully now. I’ll mark the response from @parejkoj as the solution.