Converting Pan-STARRS reference catalogue to LSST format

Support
#1

Hi all,

I am trying to run ProcessCcd on some HSC data and am running into the following error:

Traceback (most recent call last):
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/meas_algorithms/g0e4bf8285c+96dd2c2ea9/python/lsst/meas/algorithms/loadReferenceObjects.py", line 66, in getFormatVersionFromRefCat
    version = md.getScalar("REFCAT_FORMAT_VERSION")
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/daf_base/gf041782ebf+171108ac67/python/lsst/daf/base/propertyContainer/propertyContainerContinued.py", line 877, in getScalar
    return _propertyContainerGet(self, name, returnStyle=ReturnStyle.SCALAR)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/daf_base/gf041782ebf+171108ac67/python/lsst/daf/base/propertyContainer/propertyContainerContinued.py", line 214, in _propertyContainerGet
    raise KeyError(name + " not found")
KeyError: 'REFCAT_FORMAT_VERSION not found'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/ctrl_mpexec/g218a3a8f53+ca4789321c/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 473, in _executeQuantaInProcess
    self.quantumExecutor.execute(qnode.taskDef, qnode.quantum)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/ctrl_mpexec/g218a3a8f53+ca4789321c/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 161, in execute
    result = self._execute(taskDef, quantum)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/ctrl_mpexec/g218a3a8f53+ca4789321c/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 258, in _execute
    self.runQuantum(task, quantum, taskDef, limited_butler)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/ctrl_mpexec/g218a3a8f53+ca4789321c/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 460, in runQuantum
    task.runQuantum(butlerQC, inputRefs, outputRefs)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/pipe_tasks/g4a941329ef+4f7594a38e/python/lsst/pipe/tasks/calibrate.py", line 481, in runQuantum
    outputs = self.run(**inputs)
              ^^^^^^^^^^^^^^^^^^
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/utils/g34fdc7a0c9+81bc2a20b4/python/lsst/utils/timer.py", line 295, in timeMethod_wrapper
    res = func(self, *args, **keyArgs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/pipe_tasks/g4a941329ef+4f7594a38e/python/lsst/pipe/tasks/calibrate.py", line 600, in run
    astromRes = self.astrometry.run(
                ^^^^^^^^^^^^^^^^^^^^
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/utils/g34fdc7a0c9+81bc2a20b4/python/lsst/utils/timer.py", line 295, in timeMethod_wrapper
    res = func(self, *args, **keyArgs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/meas_astrom/g0a0026dc87+17526d298f/python/lsst/meas/astrom/astrometry.py", line 179, in run
    res = self.solve(exposure=exposure, sourceCat=sourceCat)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/utils/g34fdc7a0c9+81bc2a20b4/python/lsst/utils/timer.py", line 295, in timeMethod_wrapper
    res = func(self, *args, **keyArgs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/meas_astrom/g0a0026dc87+17526d298f/python/lsst/meas/astrom/astrometry.py", line 225, in solve
    loadRes = self.refObjLoader.loadPixelBox(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/meas_algorithms/g0e4bf8285c+96dd2c2ea9/python/lsst/meas/algorithms/loadReferenceObjects.py", line 603, in loadPixelBox
    return self.loadRegion(outerSkyRegion, filterName, filtFunc=_filterFunction, epoch=epoch)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/meas_algorithms/g0e4bf8285c+96dd2c2ea9/python/lsst/meas/algorithms/loadReferenceObjects.py", line 695, in loadRegion
    version = getFormatVersionFromRefCat(refCat)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/meas_algorithms/g0e4bf8285c+96dd2c2ea9/python/lsst/meas/algorithms/loadReferenceObjects.py", line 72, in getFormatVersionFromRefCat
    raise ValueError(f"No version number found in refcat header metadata. {errMsg}")
ValueError: No version number found in refcat header metadata. Version 0 refcats are no longer supported: refcat fluxes must have nJy units.
lsst.ctrl.mpexec.mpGraphExecutor INFO: Executed 2 quanta successfully, 1 failed and 0 remain out of total 3 quanta.
lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback:
Traceback (most recent call last):
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/ctrl_mpexec/g218a3a8f53+ca4789321c/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 208, in run
    script.run(qgraphObj=qgraph, **kwargs)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/ctrl_mpexec/g218a3a8f53+ca4789321c/python/lsst/ctrl/mpexec/cli/script/run.py", line 214, in run
    f.runPipeline(qgraphObj, taskFactory, args)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/ctrl_mpexec/g218a3a8f53+ca4789321c/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 800, in runPipeline
    executor.execute(graph)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/ctrl_mpexec/g218a3a8f53+ca4789321c/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 405, in execute
    self._executeQuantaInProcess(graph, self.report)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/ctrl_mpexec/g218a3a8f53+ca4789321c/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 529, in _executeQuantaInProcess
    raise MPGraphExecutorError("One or more tasks failed during execution.")
lsst.ctrl.mpexec.mpGraphExecutor.MPGraphExecutorError: One or more tasks failed during execution.

I realise now it is because my Pan-STARRS (ps1_pv3_3pi_20170110) reference catalogues haven’t been converted to the proper LSST format. Following this page: How to generate an LSST reference catalog I tried converting them, however, my reference catalogue files are in .fits format, not a text file format and therefore running convertReferenceCatalog doesn’t work. Does anyone know how I might be able to convert my files? Or is there anywhere I can download the LSST formatted version from? I’m working with v26 of the software. Thanks!

1 Like
#2

Hi, are those .fits refcats in gen2? If so, you could try this method
https://pipelines.lsst.io/modules/lsst.meas.algorithms/creating-a-reference-catalog.html#ingesting-pre-existing-gen2-reference-catalogs
Merian Data Processing Using the LSST Science Pipelines - HackMD (option 1)

1 Like
#3

Thanks @sfu. @ess20clsi if that worked for you, can you mark @sfu’s post as the solution for this topic?

#4

That looks like your PS1 refcat files are very old: this is not a gen2/gen3 butler problem. We removed support for version 0 (which unversioned refcats implicitly are) refcats last summer on DM-34793, to finish implementing RFC-851.

The version 1 updated PS1 catalog is available on the USDF, but I don’t think LSST hosts it anywhere for download. I believe Paul Price has it on his website, though you’ll have to use wget or something to pull all the files down: http://tigress-web.princeton.edu/~pprice/ps1_pv3_3pi_20170110/

Alternately, you should be able to use the since-removed conversion script to convert your catalog to version 1, but it may require setting up an older version of Science Pipelines. This older version of meas_algorithms has the relevant code, in bin/convert_refcat_to_nJy.py; you’ll have to follow the usual directions for setting up and building that package. You could instead download Science Pipelines v24 or older, which should have that script available as well.

#5

Hi everyone. Thank you all for your helpful suggestions. @sfu I tried those things, but unfortunately couldn’t get any to work. @parejkoj I’ve tried the convert_refcat_to_nJy.py and was still getting the same issue. I’ll try downloading the new catalogue at the link you gave, and keep you updated on how it worked.

#6

Hi all. After downloading the new reference catalogue, I am still getting the same error message: KeyError: 'REFCAT_FORMAT_VERSION not found'. Does anyone have any ideas on how to resolve this issue?

#7

Hi Claire. I think I am facing a similar issue here:
I have the PS1 refcats downloaded from Paul’s link. It works for gen2 versions and SOME gen3 versions – it is good for v23.0 but failed in v26.0. I believe the error I got in v26.0 was caused by the wrong flux unit which should be in nJy.

Hence, I have 2 questions here:

(1) What are the flux units in the original PS1 refcat? Are they Jy? (because I saw the description of convert_refcat_to_nJy.py states that it can only convert Jy → nJy)

(2) It seems that convert_refcat_to_nJy.py requires a master_schema.fits file. Since the PS1 refcats are already in FITS format (“the LSST format”) without this master_schema.fits file – how should I generate it?

Also, if you found any solutions to your issues, I’d appreciate it if you could post them here. Thanks!

#8

Hi @rbliu . The master_schema.fits and config.py files are actually in the ps1_pv3_3pi_20170110 directory when you download the PS1 refcats from Paul’s link. Please let us know if that solves your problem, and how you solved it. Thanks!

#9

The original PS1 refcat files are in fact in units of Jy, but they do not have a units field, so you have to know that by having talked with the person who made them. This is why version>=1 catalogs require units!

And as Claire says, the master schema should have come with the refcat you downloaded.

I’ve asked internally about whether we can host catalogs like this for external download. They’re large, in both total size and number of files (>65k files!), so rather unwieldy to handle with normal download methods (e.g. wget).

1 Like
#10

Hi all, thanks for contributing to this thread. I’ve been reviewing it to see if a[solution to the original post’s questions can be marked: “Does anyone know how I might be able to convert my files? Or is there anywhere I can download the LSST formatted version from?”

Is the interim solution that the files can be downloaded from Paul Price’s website, such that @parejkoj’s first response can be marked as the solution?

#11

Yes, I have since discovered that my butler was still looking at the old files, so downloading the new ones was the solution. Thanks everyone for the help!

2 Likes