Hi forum members,
As part of our work preparing for LSST-Euclid joint data processing at the DK-IDAC, @rajuaman and I have been preparing an LSST Gen3 butler for public Euclid-Q1 images.
I have created a skymap based on Euclid “tiles”.
I can symlink-ingest the Euclid data products which have datatype float32 using the storage class ExposureF (ie. the coadds, background model maps, RMS maps and gridded PSF maps) and datatype int32 using storage class ExposureI.
However, the segmentation maps product from Euclid are datatype uint64.
Ordinarily, just trying to regiester the euclidSegMap dataset as “ExposureL”, there is an error.
I have tried using the ExposureL class, using a “seed_config” when I initialise the repo with Butler.makeRepo(repo_path, config=seed_config)
with:
ExposureL_storageClass = {
"pytype": "lsst.afw.image.ExposureL",
"delegate": "lsst.daf.butler.delegates.ExposureStorageClassDelegate",
"parameters": ["bbox", "origin", "conformMasks", "allowUnsafe"],
}
ExposureL_formatter = {
"formatter": "lsst.obs.base.formatters.fitsExposure.FitsExposureFormatter",
}
seed_config = {
"storageClasses": {
"ExposureL": ExposureL_storageClass,
},
"datastore": {
"formatters": {
"ExposureL": ExposureL_formatter
}
}
}
There are no errors when I now register euclidSegMap dataset with ExposureL, or use Butler.ingest() to ingest the data.
However, when I try to get() the data, there is a cfitsio error: “Incompatible type for FITS image: on disk is int64 (HDU 0), in-memory is uint64. Read with allowUnsafe=True”, and I am unsure how to provide the allowUnsafe parameter to the FitsExposureFormatter.
In my datastore.formatters.ExposureL seed config, I have tried using:
- “readRecipes”: {“default”: {“allowUnsafe”: True}}
- “parameters”: {“allowUnsafe”: True}
- “parameters”: {“read_parameters”: {“allowUnsafe”: True}}}
My current solution is to use NumpyArray storage class, which works fine.
However, I would prefer in the long run to not have to create duplicates of the data in two formats.
Has anyone else tried to ingest uint64 data into an LSST Gen3 butler? Is there a `storageClass that I am missing?
Many thanks,
Aidan Sedgewick (for DK-IDAC)