I’ve been struggling to figure out the correct use of ingestImages.py to ingest HSC calibration data. I have raw images in $CI_HSC_DIR/raw and the ingest works perfectly with the command listed in the tutorial.
I have flats and biases in $CI_HSC_DIR/cals, but not sure what flags to provide for ingest to create the CALIBS directory. Obviously run as
ingestImages.py DATA $CI_HSC_DIR/cals/*.fits --mode=link
is wrong because processCcd.py throws the warning: HscMapper WARN: Unable to find calib root directory before failing catastrophically.
I think you want ingestCalibs.py, if the flats and biases are “master” coadded images produced by the HSC/LSST code and not “raw” telescope images, but an HSC expert should probably weigh in here.
What I have are the raw calib frames and not masters. As it is, I need the code to produce those masters, and I can’t ingest the frames in the first place.
If these are raw calibration frames, then they should indeed be ingested with ingestImages. I’d have to see what the actual error message is; the WARN you mention is exactly that: merely a warning and not an actual problem.
Oh: the error is coming from processCcd? Then yes, you need to run master calibration generation code based on your raw calibrations (or get masters from somewhere else) and ingest them with ingestCalibs. Doing that calibration processing is far beyond me
I see. Thanks. Yes I only have the raw calibration frames. I know Gemini automatically creates calibration masters on a ~monthly basis, but I am unaware of any service that would provide something different for the HSC.
FWIW I tried ingestCalibs on the raw frames, with no luck. The following error is thrown, suggesting that it is expecting fits files of a different format (data arrays in different locations in the fits file).
(lsst-scipipe-4d7b902) ubuntu@lsst-test:/media/NHproc$ ingestCalibs.py DATA $CI_HSC_DIR/CALIB/smoka/P*_data/*fits --mode=link --validity 60
root INFO: Loading config overrride file '/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/obs_subaru/19.0.0+2/config/ingestCalibs.py'
HscMapper WARN: Unable to find calib root directory
CameraMapper INFO: Loading exposure registry from /media/NHproc/DATA/registry.sqlite3
Traceback (most recent call last):
File "/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_tasks/19.0.0+2/bin/ingestCalibs.py", line 3, in <module>
IngestCalibsTask.parseAndRun()
File "/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_tasks/19.0.0+2/python/lsst/pipe/tasks/ingest.py", line 425, in parseAndRun
task.run(args)
File "/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_tasks/19.0.0+2/python/lsst/pipe/tasks/ingestCalibs.py", line 220, in run
with self.register.openRegistry(calibRoot, create=args.create, dryrun=args.dryrun) as registry:
File "/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_tasks/19.0.0+2/python/lsst/pipe/tasks/ingestCalibs.py", line 90, in openRegistry
return RegisterTask.openRegistry(self, directory, create, dryrun, name)
File "/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_tasks/19.0.0+2/python/lsst/pipe/tasks/ingest.py", line 294, in openRegistry
registryName = os.path.join(directory, name)
File "/home/ubuntu/lsst_stack/python/miniconda3-4.7.10/envs/lsst-scipipe-4d7b902/lib/python3.7/posixpath.py", line 80, in join
a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType
Yes, ingestCalibs on the raw frames definitely won’t work properly. It fails in your case because the CALIB directory doesn’t exist. If you created it manually, it might appear to work but not actually do the right thing, or it might fail. In any case, apologies for the inscrutable error messages.
The data I am currently playing with are June 2014, but we have multiple upcoming runs that I am trying to get ready for aka. familiarize myself with all parts of the pipeline.
@price Paul, do you have a machine automatically producing flats, biases, and such?
