Processing LSST simulated images

Support
#1

Hello, all

I’m trying to process PhoSim LSST raw images with the pipeline and obtain calibrated exposures (calExps). I intend to do some further processing with calExps and experiment with some algorithms. I could also use HSC or DECam images (if I could obtain them, haven’t investigated yet), but LSST images are preferred.

So I’m first trying to produce some calExps but I have several problems. I downloaded flatr, bias and dark fits files from the simulation done at Purdue (https://bitbucket.org/phosim/phosim_release/wiki/Example%20Images). I ingested the flatr images into a local registry using ingestImages.py task. The next step, I believe, is to ingest bias and dark FITS files using ingestCalibs.py. But there I have some problems.
The command:

ingestCalibs.py /data/folder /bias/folder/*.fits.gz --calibType bias --validity 1000 --create

fails with:

"KeyError: 'taiObs'"

It seems that I’m missing “translations” from HDU field names to registry column names. How and where do I specify those?
I’m using the LsstSimMapper for ingesting biases. Should I be using a different one?

But even if this is solved, I believe I will have a problem down the road anyway because the dataset from Purdue is missing flats for some reason. Can you please tell me how to obtain the flats?

By the way, what do constructBias.py, constructDark.py and others do exactly? I wasn’t able to find any real documentation.

Regarding the reference database, I downloaded ps1_pv3_3pi_20170110 fits files for the relevant area (using the instructions here: https://github.com/lsst/ci_hsc/tree/master/ps1_pv3_3pi_20170110). But again, I think I’ll have problems down the road because I think I will need the Astrometry.net format, and buildAndCatalog.py command fails by reporting that some intermediary files contain no data.

I’m a bit stuck here so I wonder if I could just avoid all of this and obtain calExps directly. Are there any simulated LSST calExp images available for download anywhere?

Are there any other publicly available simulated LSST images? (Purdue dataset contains just one snap, I believe)

Is the stack ready to process LSST simulated images at all or would you recommend I switch to HSC or DECam?
Is there any detailed documentation available for processing simulated LSST images?

Lot’s of questions, but any help is appreciated.
Thank you.

#2

I suggest you contact the DESC (@cwalter) about accessing the data from their recent production run. They have processed a lot of PhoSim data, and have calexps that they should be able to share.

1 Like
#3

Great, thank you

#4

There are two small data sets of raw + reduced data we keep around for CFHT and DECam data.

git clone https://github.com/lsst/validation_data_cfht

git clone https://github.com/lsst/validation_data_decam

These might also just be quick ways to see an output data repository set up and read some calexps.

(There’s also one for HSC, but it’s 700 GB).

This direction will send you off in a mild tangent in exchange for very quick access to a calexp.
@price 's suggestion about asking the DESC Survey Sims group is more in the longer term direction you want.

#5

Thank you. I’ll check those out too