What is the latest adopted Raw FITS File Format?

Data Management
,
#1

What is the latest Raw FITS format of LSST Stack v21 for ITL and e2V sensors? Is the document: LCA-13501: Specification for FITS Image Files from Raft and Focal Plane-Level Electro-Optical Tests dated March 13, 2017 up-to-date in particularly Tables 4 and 5?

My reasons for asking is that I am a new member both in LSST and Euclid’s simulation group working on the joint calibration of LSST and Euclid astrometry and photometry. I’ve taken over a code developed within Euclid that produces simulated images of LSST single exposures. Currently, it produces FITS according to Tables 4 and 5 in LCA-13501 as well as postISR (which I know is no longer used within LSST). I have yet to test the Raw output with the latest LSST stack as I am preoccupied with the Euclid code and other tasks. It would be extremely helpful to know which FITS format to follow for processing with the latest LSST stack v21. Thank you.

#2

I don’t actually know whether LCA-13501 is still correct.

What I can say is that headers are specified in LSE-400 but that situation is actively evolving and being tracked on confluence.

There is a test file from the BOT in obs_lsst which matches the extension layout we know is being used.

#3

Thanks for your reply. I do not have access to the link on confluence. Hoping to resolve that soon in order to view the document LSE-400.

However, I was able to access the test file and will review it further.

#4

Viewing LSE-400 unfortunately requires a DocuShare account, which is separate from a Confluence account.

#5

@ktl do you know how I can acquire a DocuShare account?

#6

We’re about to do the opposite: make LSE-400 accessible without an account.

1 Like
#7

Will you make an announcement when the document is accessible? Also, is it in the process of being updated?

#8

Yes, we will announce its availability (not sure what is taking so long), and yes, updates are unfortunately expected although the basic information should be correct.

#9

Hi @timj I see that both test files are for the ITL CCD. Is it possible to get a test file for the e2V? Also, how do you test your simulations?

#10

One could be made and added but for the purposes of the tests in obs_lsst it’s not really relevant because those files are mostly in there for metadata testing. Since the files have nulled out data it can’t easily tell you if assembly is correct.

Which simulations? I’m not involved in imsim or phosim simulations if that’s what you mean.

#11

I see. I need to verify simulated raw single exposures for LSST e2V CCD, so I would presume that would be imsim. I would like to check the WCS keywords.

#12

@jchiang is the person to talk to regarding imsim and image assembly. I’m not sure from your comment whether you are writing your own simulator or you are using imsim.

#13

Thanks. In my initial post I explained that I am modifying an external code developed within the Euclid Consortium’s simulation group.

#14

I’m also having issues with the ingestion if my image fits into the Butler.

#15

gen2 or gen3 ingestion? What sort of problem?

Ingestion requires that the headers conform. You can test this by running

$ translate_header.py -p lsst.obs.lsst.translators yourfile.fits

and seeing if you get errors.

#16

Hi, thanks. I resolved my ingestion issues. One had to do with using the incorrect _mapper:

before: lsst.obs.lsst.imsim.ImsimMapper
now: lsst.obs.lsst.LsstCamMapper.

The second issue was due to missing certain keywords. I adopted most of the FITS format in the test file from BOT not all, so I began adding keywords and testing iteratively until I could successfully ingest my file without errors.

As I said earlier this was for the ITL sensor, it would be nice to have a test file example for the e2V sensor. I want to check the values of the WCS coordinates. The external code uses the expressions in LCA-13501 Section 9.6. I would like to avoid going into detail here, but I noticed that the expressions for CRVAL## produce rather different values between the two sensors. Again this goes back to my original question as to whether the LCA-13501 document is up-to-date. If I could be provided a FITS file for e2V that would also help.

#17

@timj and @jchiang I managed to ingest my Rubin FITS file and reference catalog into Butler, but when I try to processCcd.py I get the following main warning and error:

CameraMapper WARN: Cannot create SkyWcs using VisitInfo and Detector, using metadata-based SkyWcs: Cannot create SkyWcs from camera geometry: rotator angle defined using RotType=RotType.UNKNOWN instead of SKY.

lsst::meas::base::FatalAlgorithmError: ‘Exposure has no Wcs’

The FITs format of the external code replicates the test fits file you provided as well as what’s been described in LCA-13501. I am a novice, so I am rather confused as to what the issue is. Do you have any ideas?

#18

FYI, although it may not help with the issues you’re having now, LSE-400 is now accessible to the world.

Note that it does specify in Appendix A:
ROTCOORD: Coordinate system of rotator (the string "sky")

Also, as mentioned previously, updates are expected to this document. For example, among other things, I believe HASTART/END will become decimal hours rather than degrees.

1 Like
#19

None of those example FITS files are on sky so if you are simulating a sky observation you may have missed the headers used to work out the tracking ra dec of the boresight. Does translate_header.py show you an RA/Dec?

#20

The processing code ignores the WCS in the headers and reads a boresight RA/DEC and the detector information and then uses that to calculate a WCS based on the camera geometry.