Using gaia_dr2_20200414 in jointcal?

Support
#1

Hi all,

I am currently running through some G band HSC data with lsst_stack. Everything has been going great, until I hit the jointcal calibration stage. I am still using gen2 butler here, since this is using legacy scripts from a couple of semesters back.

Since I am only reducing one filter, I am running with --config doPhotometry=True. It seems jointcal fails pretty quickly, spitting out errors regarding the gaia_dr2 ref_cat.

root INFO: Loading config overrride file '/data/ipphome1.0/home/tdeboer/lsst_stack/conda/envs/lsst-scipipe-0.8.1/share/eups/Linux64/obs_suba$
CameraMapper INFO: Loading exposure registry from /data/ipp138.0/tdeboer/WHIGS/datared/registry.sqlite3
CameraMapper INFO: Loading calib registry from /data/ipp138.0/tdeboer/WHIGS/datared/CALIB/calibRegistry.sqlite3
CameraMapper INFO: Loading calib registry from /data/ipp138.0/tdeboer/WHIGS/datared/CALIB/calibRegistry.sqlite3
root INFO: Running: /data/ipphome1.0/home/tdeboer/lsst_stack/conda/envs/lsst-scipipe-0.8.1/share/eups/Linux64/jointcal/gf6ad1f1eeb+1390f6aef$
Traceback (most recent call last):
  File "/data/ipphome1.0/home/tdeboer/lsst_stack/conda/envs/lsst-scipipe-0.8.1/share/eups/Linux64/jointcal/gf6ad1f1eeb+1390f6aefb/bin/jointc$
    JointcalTask.parseAndRun()
  File "/data/ipphome1.0/home/tdeboer/lsst_stack/conda/envs/lsst-scipipe-0.8.1/share/eups/Linux64/pipe_base/g5c83ca0194+970dd35637/python/ls$
    resultList = taskRunner.run(parsedCmd)
  File "/data/ipphome1.0/home/tdeboer/lsst_stack/conda/envs/lsst-scipipe-0.8.1/share/eups/Linux64/pipe_base/g5c83ca0194+970dd35637/python/ls$
    if self.precall(parsedCmd):
  File "/data/ipphome1.0/home/tdeboer/lsst_stack/conda/envs/lsst-scipipe-0.8.1/share/eups/Linux64/pipe_base/g5c83ca0194+970dd35637/python/ls$
    task = self.makeTask(parsedCmd=parsedCmd)
  File "/data/ipphome1.0/home/tdeboer/lsst_stack/conda/envs/lsst-scipipe-0.8.1/share/eups/Linux64/pipe_base/g5c83ca0194+970dd35637/python/ls$
    return self.TaskClass(config=self.config, log=self.log, butler=butler)
  File "/data/ipphome1.0/home/tdeboer/lsst_stack/conda/envs/lsst-scipipe-0.8.1/share/eups/Linux64/jointcal/gf6ad1f1eeb+1390f6aefb/python/lss$
    self.makeSubtask('astrometryRefObjLoader', butler=butler)
  File "/data/ipphome1.0/home/tdeboer/lsst_stack/conda/envs/lsst-scipipe-0.8.1/share/eups/Linux64/pipe_base/g5c83ca0194+970dd35637/python/ls$
    subtask = taskField.apply(name=name, parentTask=self, **keyArgs)
  File "/data/ipphome1.0/home/tdeboer/lsst_stack/conda/envs/lsst-scipipe-0.8.1/share/eups/Linux64/pex_config/g87000a63fb+536b1ee016/python/l$
    return self.target(*args, config=self.value, **kw)
  File "/data/ipphome1.0/home/tdeboer/lsst_stack/conda/envs/lsst-scipipe-0.8.1/share/eups/Linux64/meas_algorithms/g8d527e0710+15f63ea384/pyt$
    self.dataset_config = butler.get("ref_cat_config", name=self.config.ref_dataset_name, immediate=True)
  File "/data/ipphome1.0/home/tdeboer/lsst_stack/conda/envs/lsst-scipipe-0.8.1/share/eups/Linux64/daf_persistence/gaaec9c0f9d+c5c846a8b1/pyt$
    raise NoResults("No locations for get:", datasetType, dataId)
lsst.daf.persistence.butlerExceptions.NoResults: No locations for get: datasetType:ref_cat_config dataId:DataId(initialdata={'name': 'gaia_d$

The thing is, I did download the ref_cat data from http://tigress-web.princeton.edu/~HSC/refcats/ and symlinked that to my HSC/ref_cats folder with the proper permissions and everything. I am wondering what I am missing here to make jointcal not find the ref_cat. Is there an extra step needed to set gaia as the ref_cat in some way?

#2

Gaia is a bad choice for photometric calibration: it’s filters do not resemble any standard filters at all.

What version of the science pipelines are you using? We switched jointcal to default to using PS1 for photomery and gaia for astrometry in May 2020: DM-17597: Update testdata_jointcal catalogs to be processed with gaia+ps1 by parejkoj · Pull Request #157 · lsst/jointcal · GitHub

#3

I am using lsst-scipipe-0.8.1 at the moment. I am fine with using the PS1 photometry for photcal, but how do I switch from one to the other?

#4

That’s the conda environment: I think that goes with Science Pipelines v23 (these are the available versions LSST Science Pipelines: LSST the Docs Editions). If that’s correct, you should already have those defaults set correctly. What refcats did you use for photometry/astrometry for single frame processing?

If you’re in gen2, set config.astrometryRefObjLoader.ref_dataset_name = "gaia_dr2_20200414" and self.photometryRefObjLoader.ref_dataset_name = "ps1_pv3_3pi_20170110"

#5

yes, I am indeed using LSSTpipe V23. Thank you for the provided config options. That does help to set things the way they should be. However, it seems the real problem is simply that the gaia_dr2_20200414 ref_cat does not seem to work for me. With the provided configs, jointcal still tries to use gaia dr2 for the astrometry and then encounters the same error as before.

#6

We don’t support gen2 any more, but I’ll try. Please list the contents of your directories, including the main repository path, the path to your refcats, and the contents of the refcats directory (not the refcat files, just the contents of HSC/ref_cats). ls -al output might be helpful here.

#7

I think I figured this out. It turns out, I was missing one of the files in the gaia_dr2_2020414 folder (the master_schema file) and that was causing jointcal to fail reading in the ref_cat. I did not realise all the files were required to be in the folder for things to work. Now, things are working, and my data reduction is chugging along.

1 Like