Thanks for doing this. I am confused by some of the results.
write-curated-calibrations should be no different with or without composite disassembly because none of the curated calibrations are disassembled and yet somehow it’s 10% slower. I may well have to implement a
butler.put that allows multiple datasets to be stored at once so that the datastore can parallelize uploads.
Coaddition is going to be slow because each component is downloaded separately from S3 and then combined into a single
Exposure. It’s good to see that there is no slowdown with any of the runs with disassembly that used a “local” filesystem. I think I may have to implement an asyncio parallel file retrieval option (and storage option) so that we can be sending these files to S3 simultaneously. It is interesting how
make-discrete-skymap is barely any faster despite it downloading significantly less data.
Lots of separate FITS files is going to be bigger than one file, although I wasn’t expecting a factor of 3.