Reducing Suprime-cam Data

(Kyle Pittman) #1

I have some Subaru Suprime-cam data that I need to reduce. I’m interested in using the LSST software for this, given the success in the HSC pipeline. This repo ( indicates that there is a Suprime-cam configuration, but I wanted to check if anyone has actually used it for reducing Suprime-cam data. There are some Suprime-cam issues that I’m concerned about, such as the scattered light on the detector, that I’m not sure is an issue with the HSC data. I’d appriciate any advice that might help me get started.
Thank you,

(Colin Slater) #2

I successfully reduced some Suprime-cam data circa May 2016, though it was data with the “old” chips and I did not pay close attention to anything like scattered light or the reduction quality. So it does have most of the requisite pieces in place. It wouldn’t surprise me if there was some mild bitrot in the past year though; some tweaking might be required.

(Paul Price) #3

DM-13148 has some changes to reduce the bitrot. Though I haven’t gotten around to merging it yet, it’s not far off being done and could make its way into the next weekly release if you tell me you need it. However, that only gets you through running; to my knowledge, no-one has taken Suprime-Cam data past that point (i.e., mosaic, coadd, multiband) for a couple of years.

Given the order of magnitude size difference, I think there’s actually now more sky in the archive from HSC than Suprime-Cam; of course, that doesn’t help you if you’re interested in a specific target that was observed with Suprime-Cam.

Also, we were never really happy with the quality of the photometry we got out of Suprime-Cam (especially with the old CCDs), so be aware that you may not achieve the same quality as we’re currently getting from HSC.

But if, after all those warnings, you’re still keen on processing Suprime-Cam data and you’re willing to get your hands a bit dirty, I’m happy to help you out along the way.

(Kyle Pittman) #4

I believe that, yes, I will be needing the DM-13148 changes you mentioned. I will need to mosaic and coadd the images, as well. I checked to see if my targets are in the HSC archive, but sadly they are not. What about the Suprime-Cam photometry were you unhappy with? Also, what are you and Colin refering to that might have mild bitrot? Thanks again, your help is greatly appreciated.

(Paul Price) #5

Once upon a time, we reprocessed the original Suprime-Cam Cosmos observations, and we were never happy with the quality of the photometry; unfortunately, I don’t recall the numbers (maybe 3%???). That was with the old CCDs, so you might have more success if you’re using the new CCDs.

The obs_subaru package. Some of that is being addressed with DM-13148, enough to get you through the single frame operations, but more fixes will likely be required for mosaicking and coadding. I suggest you start off using the tickets/DM-13148 branch of obs_subaru with a modern weekly release of the LSST stack, post here any errors you encounter, and I’ll extend DM-13148 as we go along. If you’re uncertain how to work on a branch of a package, please have a look at the Developer and Pipeline documentation sites, and post a new question on this site if you can’t find anything.

(Robert Lupton) #6

I don’t know how bad things will be with the old SuprimeCam (MIT or HSC chips – we processed both once upon a time). We stopped working on SuprimeCam because it was so much easier to learn about things like PSF estimation with the new camera, but many of the improvements should apply to SuprimeCam too.

One word of warning: One of the glories of SuprimeCam was the number of NB filters, so you may have to add some filter definitions (easy enough – ask if you need help).

(Kyle Pittman) #7

Hi again,

I am currently using the 7-stack-lsst_distrib-w_2018_11 release of the LSST stack with the tickets/DM-13148 branch of obs subaru, running in docker. I have had issues setting up the obs_subaru package, following the running with docker directions. When I run: setup -r . I am given this error: setup: in file /home/lsst/obs_subaru/ups/obs_subaru.table: Product skypix not found and I am wondering if skypix is something separate I need to get setup.

Thank you

(Paul Price) #8

Sorry, I started that branch a while ago, so it’s not compatible with the latest version of the stack (which you’re using). I’ve just now made the branch compatible with w_2018_11. You’ll need to git fetch && git reset --hard origin/tickets/DM-13148.

Let me know if you start using w_2018_12, as I’ll need to upgrade the branch again.

(Kyle Pittman) #9

Thank you, Paul. That fixed it. I also had some issues specifying the mapper. Following the getting started tutorial, I run the command:
echo “lsst.obs.suprimecam.suprimecamMapper” > DATA/_mapper
and when I follow that up with: DATA $OBS_SUBARU_DIR/scam/raw/*.fits --mode=link
I get the following error:

Traceback (most recent call last):
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/14.0-54-gb4e84373/bin/", line 3, in <module>
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/14.0-54-gb4e84373/python/lsst/pipe/tasks/", line 396, in parseAndRun
    args = parser.parse_args(config)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/14.0-6-ge2c9487+58/python/lsst/pipe/base/", line 566, in parse_args
    mapperClass = dafPersist.Butler.getMapperClass(namespace.input)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/14.0-17-g2e0c876/python/lsst/daf/persistence/", line 1135, in getMapperClass
    return Storage.getMapperClass(root)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/14.0-17-g2e0c876/python/lsst/daf/persistence/", line 123, in getMapperClass
    ret = Storage.storages[parseRes.scheme].getMapperClass(uri)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/14.0-17-g2e0c876/python/lsst/daf/persistence/", line 216, in getMapperClass
    pkg = importlib.import_module(".".join(components[:-1]))
  File "/opt/lsst/software/stack/python/miniconda3-4.3.21/lib/python3.6/importlib/", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 978, in _gcd_import
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load
  File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed
  File "/home/lsst/obs_subaru/python/lsst/obs/subaru/", line 27, in <module>
    from .version import *
ModuleNotFoundError: No module named 'lsst.obs.subaru.version'

This happens when trying to reduce data for both Suprimecam and HSC using obs_subaru. Do you have an idea of why? Thanks again, I really appreciate it!

(Paul Price) #10

You need to build your obs_subaru by running scons.

Notice also that SuprimecamMapper in your /path/to/DATA/_mapper file should have a leading capital S.

(Kyle Pittman) #11

I was able to get the mapper set up and run ingestImages, but have been having trouble running ingestCalibs. Perhaps I am not using it correctly. I am receiving the error:

bash-4.2$ DATA scam/CALIB/BIAS/*.fits --mode=link --validity 1000 --calibType bias
root INFO: Loading config overrride file '/home/lsst/obs_subaru/config/'
CameraMapper INFO: Loading exposure registry from /home/lsst/DATA/registry.sqlite3
Traceback (most recent call last):
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/14.0-54-gb4e84373/bin/", line 3, in <module>
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/14.0-54-gb4e84373/python/lsst/pipe/tasks/", line 398, in parseAndRun
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/14.0-54-gb4e84373/python/lsst/pipe/tasks/", line 219, in run
    with self.register.openRegistry(calibRoot, create=args.create, dryrun=args.dryrun) as registry:
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/14.0-54-gb4e84373/python/lsst/pipe/tasks/", line 86, in openRegistry
    return RegisterTask.openRegistry(self, directory, create, dryrun, name)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/14.0-54-gb4e84373/python/lsst/pipe/tasks/", line 279, in openRegistry
    registryName = os.path.join(directory, name)
  File "/opt/lsst/software/stack/python/miniconda3-4.3.21/lib/python3.6/", line 78, in join
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType

I believe the problem is happening in

"/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/14.0-54-gb4e84373/python/lsst/pipe/tasks/", line 217

though I am not 100% sure.

(Paul Price) #12 works on “master” calibs created by our code. Unless you’ve run, you’re probably pointing at raw data (which should have been ingested with or a master calib created by different code which may not have the expected headers.

In any case, you didn’t specify a calibration directory (with --calib) in which to put the ingested calibs, which I suspect is the problem causing the TypeError.

(Kyle Pittman) #13

I’m not sure I understand how to get the reference catalog working. Following these instructions, I have grabbed the appropriate FITS files (plus and README.txt), and have installed them into my data repository using
ln -s scam/ps1_pv3_3pi_20170110/ DATA/ref_cats/. I’ve also adjusted my configuration, adding

from lsst.meas.algorithms import LoadIndexedReferenceObjectsTask
config.calibrate.astromRefObjLoader.ref_dataset_name = "ps1_pv3_3pi_20170110"
config.calibrate.photoRefObjLoader.ref_dataset_name = "ps1_pv3_3pi_20170110"
config.calibrate.photoCal.photoCatName = "ps1_pv3_3pi_20170110"

to the existing config/ in the obs_subaru package (I have also tried specifying it with as a seperate file with --configfile). Biases and flats are constructed and ingested. When I run DATA --rerun processCcdOutput --id, I get the error:

bash-4.2$ DATA --rerun processCcdOutput --id
root INFO: Loading config overrride file '/home/lsst/obs_subaru/config/'
root INFO: Loading config overrride file '/home/lsst/obs_subaru/config/suprimecam/'
CameraMapper INFO: Loading exposure registry from /home/lsst/DATA/registry.sqlite3
CameraMapper INFO: Loading calib registry from /home/lsst/DATA/CALIB/calibRegistry.sqlite3
root INFO: Running: /opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/14.0-54-gb4e84373/bin/ DATA --rerun processCcdOutput --id
Traceback (most recent call last):
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/14.0-54-gb4e84373/bin/", line 25, in <module>
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/14.0-6-ge2c9487+58/python/lsst/pipe/base/", line 586, in parseAndRun
    resultList =
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/14.0-6-ge2c9487+58/python/lsst/pipe/base/", line 233, in run
    if self.precall(parsedCmd):
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/14.0-6-ge2c9487+58/python/lsst/pipe/base/", line 347, in precall
    task = self.makeTask(parsedCmd=parsedCmd)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/14.0-6-ge2c9487+58/python/lsst/pipe/base/", line 472, in makeTask
    return self.TaskClass(config=self.config, log=self.log, butler=butler)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/14.0-54-gb4e84373/python/lsst/pipe/tasks/", line 161, in __init__
    astromRefObjLoader=astromRefObjLoader, photoRefObjLoader=photoRefObjLoader)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/14.0-6-ge2c9487+58/python/lsst/pipe/base/", line 299, in makeSubtask
    subtask = taskField.apply(name=name, parentTask=self, **keyArgs)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pex_config/14.0-4-gae1598d+1/python/lsst/pex/config/", line 83, in apply
    return*args, config=self.value, **kw)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/14.0-54-gb4e84373/python/lsst/pipe/tasks/", line 365, in __init__
    self.makeSubtask('astromRefObjLoader', butler=butler)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/14.0-6-ge2c9487+58/python/lsst/pipe/base/", line 299, in makeSubtask
    subtask = taskField.apply(name=name, parentTask=self, **keyArgs)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pex_config/14.0-4-gae1598d+1/python/lsst/pex/config/", line 83, in apply
    return*args, config=self.value, **kw)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/meas_algorithms/14.0-19-gbbfecc2c/python/lsst/meas/algorithms/", line 49, in __init__
    dataset_config = butler.get("ref_cat_config", name=self.config.ref_dataset_name, immediate=True)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/14.0-17-g2e0c876/python/lsst/daf/persistence/", line 1412, in get
    raise NoResults("No locations for get:", datasetType, dataId)
lsst.daf.persistence.butlerExceptions.NoResults: No locations for get: datasetType:ref_cat_config dataId:DataId(initialdata={'name': 'ps1_pv3_3pi_20170110'}, tag=set())

I searched the forums, but didn’t find anything involving errors with ref_cat_config. Also, when I am running should I only be running it on the object exposures (i.e. exclude the bias/flats/focusing)? Thank you!

(Paul Price) #14

You’ll also need the master_schema.fits file (which I think I neglected to mention in the README.txt file).

Can you list the contents of DATA/ref_cats for me (tree -l DATA/refcats)?

See also here for a similar question.

(Kyle Pittman) #15
bash-4.2$ tree -l DATA/ref_cats/
`-- ps1_pv3_3pi_20170110 -> scam/ps1_pv3_3pi_20170110

0 directories, 1 file

the contents of scam/ps1_pv3_3pi_20170110 are:

bash-4.2$ tree -l scam/ps1_pv3_3pi_20170110/
|-- 215042.fits
|-- 215049.fits
|-- 215050.fits
|-- 215051.fits
|-- 215052.fits
|-- 215053.fits
|-- 215055.fits
|-- 226305.fits
|-- 226310.fits
|-- 226318.fits
|-- README.txt

0 directories, 12 files

Is the master_schema.fits specific to the FITS files chosen, or is it more generic?

(Paul Price) #16

I suspect that your link isn’t working. Try ln -s /full/path/to/scam/ps1_pv3_3pi_20170110 DATA/ref_cats, and verify the link with ls -l.

master_schema.fits is an element of the catalog format, so you need it regardless of which shard files you have.

(Kyle Pittman) #17

Thank you. Specifying the full path and adding master_schema.fits fixed the ref_cat_config error, though now I am getting an error that is unable to retrieve bias for the given exposures. This is what I am getting:

bash-4.2$ DATA --rerun processCcdOutput --id field=N5846_1
root INFO: Loading config overrride file '/home/lsst/obs_subaru/config/'
root INFO: Loading config overrride file '/home/lsst/obs_subaru/config/suprimecam/'
CameraMapper INFO: Loading exposure registry from /home/lsst/DATA/registry.sqlite3
CameraMapper INFO: Loading calib registry from /home/lsst/DATA/CALIB/calibRegistry.sqlite3
root INFO: Running: /opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/14.0-54-gb4e84373/bin/ DATA --rerun processCcdOutput --id field=N5846_1
processCcd INFO: Processing {'field': 'N5846_1', 'dateObs': '2009-04-21', 'pointing': 1937, 'filter': 'W-S-G+', 'visit': 109045, 'ccd': 0, 'taiObs': '2009-04-21', 'expTime': 460.0}
processCcd.isr INFO: Performing ISR on sensor {'field': 'N5846_1', 'dateObs': '2009-04-21', 'pointing': 1937, 'filter': 'W-S-G+', 'visit': 109045, 'ccd': 0, 'taiObs': '2009-04-21', 'expTime': 460.0}
daf.persistence.butler INFO: NoResults or IOError (continuing search) while evaluating bypass function for Dataset type:raw_md Data ID:DataId(initialdata={'field': 'N5846_1', 'dateObs': '2009-04-21', 'pointing': 1937, 'filter': 'W-S-G+', 'visit': 109045, 'ccd': 0, 'taiObs': '2009-04-21', 'expTime': 460.0}, tag=set()) at location lsst.afw.image.DecoratedImageU at FitsStorage(N5846_1/2009-04-21/01937/W-S-G+/SUPA01090450.fits)
processCcd FATAL: Failed on dataId={'field': 'N5846_1', 'dateObs': '2009-04-21', 'pointing': 1937, 'filter': 'W-S-G+', 'visit': 109045, 'ccd': 0, 'taiObs': '2009-04-21', 'expTime': 460.0}: RuntimeError: Unable to retrieve bias for {'field': 'N5846_1', 'dateObs': '2009-04-21', 'pointing': 1937, 'filter': 'W-S-G+', 'visit': 109045, 'ccd': 0, 'taiObs': '2009-04-21', 'expTime': 460.0}: No registry for lookup
Traceback (most recent call last):
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/ip_isr/14.0-10-g2094653+11/python/lsst/ip/isr/", line 719, in getIsrExposure
    exp = dataRef.get(datasetType, immediate=immediate)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/14.0-17-g2e0c876/python/lsst/daf/persistence/", line 198, in get
    return self.butlerSubset.butler.get(datasetType, self.dataId, **rest)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/14.0-17-g2e0c876/python/lsst/daf/persistence/", line 1410, in get
    location = self._locate(datasetType, dataId, write=False)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/14.0-17-g2e0c876/python/lsst/daf/persistence/", line 1329, in _locate
    location =, dataId, write=write)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/14.0-17-g2e0c876/python/lsst/daf/persistence/", line 246, in map
    loc =*args, **kwargs)
  File "/home/lsst/obs_subaru/python/lsst/obs/suprimecam/", line 61, in map
    return super(SuprimecamMapperBase, self).map(datasetType, copyId, write=write)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/14.0-17-g2e0c876/python/lsst/daf/persistence/", line 166, in map
    return func(self.validate(dataId), write)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/obs_base/14.0-28-gd4c92d7+3/python/lsst/obs/base/", line 389, in mapClosure
    return, dataId, write)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/obs_base/14.0-28-gd4c92d7+3/python/lsst/obs/base/", line 466, in map
    location =, mapper, dataId, write=write)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/obs_base/14.0-28-gd4c92d7+3/python/lsst/obs/base/", line 158, in map
    actualId = self.need(iter(self.keyDict.keys()), dataId)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/obs_base/14.0-28-gd4c92d7+3/python/lsst/obs/base/", line 322, in need
    lookups = self.lookup(newProps, newId)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/obs_base/14.0-28-gd4c92d7+3/python/lsst/obs/base/", line 512, in lookup
    return Mapping.lookup(self, properties, newId)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/obs_base/14.0-28-gd4c92d7+3/python/lsst/obs/base/", line 210, in lookup
    raise RuntimeError("No registry for lookup")
RuntimeError: No registry for lookup

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/14.0-6-ge2c9487+58/python/lsst/pipe/base/", line 408, in __call__
    result =, **kwargs)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/14.0-6-ge2c9487+58/python/lsst/pipe/base/", line 150, in wrapper
    res = func(self, *args, **keyArgs)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/14.0-54-gb4e84373/python/lsst/pipe/tasks/", line 184, in run
    exposure = self.isr.runDataRef(sensorRef).exposure
  File "/home/lsst/obs_subaru/python/lsst/obs/subaru/", line 296, in runDataRef
    biasExposure = self.getIsrExposure(sensorRef, "bias")
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/ip_isr/14.0-10-g2094653+11/python/lsst/ip/isr/", line 722, in getIsrExposure
    raise RuntimeError("Unable to retrieve %s for %s: %s" % (datasetType, dataRef.dataId, exc1))
RuntimeError: Unable to retrieve bias for {'field': 'N5846_1', 'dateObs': '2009-04-21', 'pointing': 1937, 'filter': 'W-S-G+', 'visit': 109045, 'ccd': 0, 'taiObs': '2009-04-21', 'expTime': 460.0}: No registry for lookup

I’m a bit confused, as and ran fine.

(Paul Price) #18

So what’s in the calibration registry? Please post the results of running:

sqlite3 /home/lsst/DATA/CALIB/calibRegistry.sqlite3
.header on

If that’s empty, please post the command you used.

(Kyle Pittman) #19
bash-4.2$ sqlite3 /home/lsst/DATA/CALIB/calibRegistry.sqlite3
SQLite version 3.13.0 2016-05-18 10:57:30
Enter ".help" for usage hints.
sqlite> .header on
sqlite> SELECT * FROM bias;

The command I ran was: DATA --rerun n5846 --calib /home/lsst/DATA/CALIB --validity 1000 /home/lsst/DATA/rerun/n5846/BIAS/2009-04-21/NONE/BIAS-*.fits

(Paul Price) #20

Oh, maybe it’s not related to the calib registry, but pointing to it. Please try: DATA --calib DATA/CALIB --rerun processCcdOutput --id field=N5846_1