Ingest int64 image data into Gen3 butler

Support
, ,
#1

Hi forum members,

As part of our work preparing for LSST-Euclid joint data processing at the DK-IDAC, @rajuaman and I have been preparing an LSST Gen3 butler for public Euclid-Q1 images.
I have created a skymap based on Euclid “tiles”.
I can symlink-ingest the Euclid data products which have datatype float32 using the storage class ExposureF (ie. the coadds, background model maps, RMS maps and gridded PSF maps) and datatype int32 using storage class ExposureI.

However, the segmentation maps product from Euclid are datatype uint64.
Ordinarily, just trying to regiester the euclidSegMap dataset as “ExposureL”, there is an error.
I have tried using the ExposureL class, using a “seed_config” when I initialise the repo with Butler.makeRepo(repo_path, config=seed_config)

with:

ExposureL_storageClass = {
    "pytype": "lsst.afw.image.ExposureL",
    "delegate": "lsst.daf.butler.delegates.ExposureStorageClassDelegate",
    "parameters": ["bbox", "origin", "conformMasks", "allowUnsafe"],
}
ExposureL_formatter = {
    "formatter": "lsst.obs.base.formatters.fitsExposure.FitsExposureFormatter",
}
seed_config = {
    "storageClasses": {
        "ExposureL": ExposureL_storageClass,
    },
    "datastore": {
        "formatters": {
            "ExposureL": ExposureL_formatter
        }
    }
}

There are no errors when I now register euclidSegMap dataset with ExposureL, or use Butler.ingest() to ingest the data.
However, when I try to get() the data, there is a cfitsio error: “Incompatible type for FITS image: on disk is int64 (HDU 0), in-memory is uint64. Read with allowUnsafe=True”, and I am unsure how to provide the allowUnsafe parameter to the FitsExposureFormatter.

In my datastore.formatters.ExposureL seed config, I have tried using:

  • “readRecipes”: {“default”: {“allowUnsafe”: True}}
  • “parameters”: {“allowUnsafe”: True}
  • “parameters”: {“read_parameters”: {“allowUnsafe”: True}}}

My current solution is to use NumpyArray storage class, which works fine.
However, I would prefer in the long run to not have to create duplicates of the data in two formats.

Has anyone else tried to ingest uint64 data into an LSST Gen3 butler? Is there a `storageClass that I am missing?

Many thanks,
Aidan Sedgewick (for DK-IDAC)

#2

Hi @aidansedgewick, thanks for posting your question here.

We don’t have a quick fix for this, but we wanted to let you know the team is looking into it. Thanks for your patience!

#3

This is great to hear – thanks for the quick response, @galaxyumi331 !

#4

I don’t have a complete solution to this, but I do have some thoughts that might help us come up with one.

  • Are you really writing a FITS HDU with uint64 pixels right now? I was under the impression that was impossible, because FITS only supports unsigned integers via a BZERO offset, and the necessary BZERO for uint64 doesn’t fit within a double-precision float. It’s possible that this has just gotten fixed in the standard since I last looked into it - which was at least a decade ago - but that’s part of why we haven’t put much effort into supporting 64-bit integer images (the other reason is that we just haven’t needed it).

  • This really seems like a case where you’d want to invent your own storage class and formatter. There are some environment variables you can set to make the butler aware of those, and they would let the butler those files read into whatever Python type you’d use naturally in Euclid work (I’m guessing Euclid has an in-memory Python type they’d use for these files?).

  • If you don’t have a natural Python type for these from the Euclid side, or you want to invent one for some other reason, we’re actually trying to move (slowly!) away from lsst.afw.image types like Exposure in favor of the new lsst.images package. That’s not really stable yet - we are in a hurry stabilizing a subset of it for the Early DP2 release - but if you can handle a little instability now, it will be a better foundation to build on in the long term than lsst.afw.image. Nothing in lsst.images would be able to read the files you’ve described out-of-the-box, but it sounds pretty doable to write a function that reads those files with astropy into a data structure built out of lsst.images primitives, and that’s the kind of thing that would back a “formatter” you’d need to talk to the butler.

#5

Hi @aidansedgewick ! Just wanted to check in on this issue, in case the answers to Jim’s questions can help us converge on a solution? Please let us know!

#6

Hi - sorry for my slow reply on this @ChristinaWilliams . Thanks @jbosch for the advice above.

I think that I’ve misunderstood the images that I’m working with - I’m sure they are 64-bit integers (header key BITPIX=64, but with no BZERO keyword, and the max value is much less than 2^63 – so the they must be signed. I’m not part of the Euclid development team, but am working with some colleagues who plan to do joint Rubin-Euclid analysis.

Perhaps the issue is with the way I’m trying to use butler.ingest(): the error message that I pasted above even says that on-disk is int64, so the issue may lie in the way that I’m trying to ingest the FITS files.

Thanks for the update about the plan to move away from lsst.afw.image types like Exposure. This is worth knowing for future planning anyway, and I will work on getting something to work with the new lsst.images primatives you’ve described, and will report back if I have something useful to share.

Using the NumpyArray storage class and butler.put() is a usable solution right now for the public Euclid Q1 data. But when DR.1 becomes publicly available in the future, I’d really like to avoid duplicating all of the segmentaion maps. And it will be worth us keeping up with the latest Butler implementations anyway.

Many thanks,
Aidan

#7

Signed int64 makes more sense, at least in terms of me understanding what you’re trying to do.

I think you’re still stuck trying to ingest these as-is without a custom Python reader, though; our Exposure class just isn’t built to read files it didn’t write except under very limited circumstances, and the same is true of the lsst.images types. It does sound like it ought to be possible for a modest extension to lsst.images.MaskedImageto serve as your in-memory object, along with that custom reader.