I want to do some testing, but /project/mrawls/hits2015/slurm6 does not exist.
Ah sorry, I forgot “rerun” between “hits2015” and “slurm6”! Will update in original post.
I don’t know how to check to see if I’m using a version of ds9 that supports the version of WCS distortions that we write, but I will try the newest beta in addition to the two versions I have already tried. If there is an “LSST tool” that allows me to quickly display ~50-100 calexps at once in their correct relative sky locations, I would love to know about it.
It’s a little scary to hear this kind of behavior has been observed over years(!). I strongly suspect LSST users will someday try using ds9 to mosaic a bunch of images at once. This seems like a basic functionality we and ds9 should support.
If you can’t get ds9 to work, you could always warp images to a common DiscreteSkyMap tract. ds9 will handle those much better (no non-linear WCSes).
1 Like
Well that was fun… the latest ds9 v8.0 rc2 has a slightly different result! I’ll contact the ds9 folks about this.
Hmm… we went through a discussion of this before. (Check out Erin Sheldon’s slack avatar! That came from when @laurenam(?) saw this ). I thought I remembered that we thought this came down to ds9 not knowing how to handle TAN-SIP.
But, I thought we are also now doing this now with our DC2 outputs (but I can’t remember at what phase). So perhaps there are headers missing?
@jchiang do you have anything to add?
Yeah, I saw this on DM-1573, where I concluded that
The strange arrangement in the CCDs seen in FocalPlane_0001228a.png seems to be an issue with ds9 (getting confused with the wcs?) rather than an erroneous astrometric solution…
Inability to interpret TAN-SIP seems the likely source of DS9”s “confusion”.
1 Like
DS9 uses AST for its WCS handling, so it absolutely should handle TAN-SIP, and should also handle our WCS outputs.
Bruce Berriman (Caltech) doesn’t have an account here (yet…? 
) but said it was OK to post what he emailed to me about this.
The problem you describe is, it turns out, one of the reasons we built the Montage mosaic engine (http://montage.ipac.caltech.edu); we have come across similar problems but in different contexts. The solution to your conundrum lies in the fact that you have to do full reproductions and (if needed, resampling) of every image. DS9 isn’t intended to do this - it treats one image as a template and forces the rest of the images to comply with it. This is actually OK for looking at a few images, but is not an approach intended to visualize substantial collections of images.
The simplest approach might be for us to process the data and use the workflow to illustrate for you how the processing is done. Would you be able to make the data accessible to us?
Please note that Montage is written in C for performance, and we are testing Python binary extensions that will be released at the end of October.
For the record, I put some of the images I am working on in this publicly accessible location for now.
1 Like
So things would be a little better if DS9 was a bit cleverer about picking an image from the middle?
1 Like
Given that response, this seems like something they should call out in the user guide and/or FAQ. I couldn’t find anything about “don’t try to mosaic a dozen images” in there:
http://ds9.si.edu/doc/ref/file.html#FITSMosaic
http://ds9.si.edu/doc/faq.html
1 Like
It’s not the number of images that’s the root of the problem (I’ve successfully used -mosaicwcs with several tens of inputs): it’s the non-linearity in the astrometry. If the WCSes are non-linear with different CRVALn, then having more images means you’re less likely to be able to use simple transforms to the template it chooses.
1 Like
Montage is an awesome program(s), but setting up the workflow is a bit of a hassle. It’s definitely not designed for “quick look” processing.
Yes indeed @timj; feeding ds9 one of the more central DECam chips first results in slightly less wonkiness.
I now think it is somewhat understandable for ds9 to behave this way, but it should absolutely be mentioned in the documentation/FAQ that trying to load more than a ~few images with differing non-linear WCSs will result in some really bad mosaicing. Otherwise, the naive user may think this indicates an actual problem with the astrometry!
Here is the result on the same images with ds9 -mosaic diffexp-2*.fits diffexp-0*.fits diffexp-1*.fits diffexp-3*.fits diffexp-4*.fits diffexp-5*.fits diffexp-6*.fits (the central two DECam CCDs are numbers 28 and 35).
Hi All,
Sorry… I wasn’t getting notifications on the discussion. I was talking to Jim Chiang and trying to understand more about this and looking more carefully at the instructions.
Go here: http://ds9.si.edu/doc/ref/file.html
down to “FITS Mosaic”.
It looks like you didn’t tell it to use a wcs when doing the mosaic.
I think you want something like:
ds9 -mosaic wcs foo.fits bar.fits wow.fits # load mosaic wcs from 3 files
-Chris
Thanks @cwalter, but it turns out the issue is deeper than this. Mosaics in ds9 are one of wcs, iraf, or wfpc2, and while it might be clearer/safer to explicitly specify I want a wcs mosaic, that is the type created by default in the absence of the iraf-y header keywords DETSEC and DETSIZE and the presence of a valid WCS.
I’ve exchanged several emails with John Good (Caltech) about how the Montage tool can do a better job than ds9 (with the caveat that it is a multi-step file-based process). He was able to create a much more accurate reprojection of my images that uses each one’s WCS properly. I also have an open query with the ds9 folks who said they’d get back to me within about a week. I’ll follow up here when they do.
OK thanks. I’m interested to hear what the ds9 people have to say.
Got a response late last week from ds9 expert Bill Joye at the Chandra X-ray Center. It’s pasted below. I intend to reply and suggest they document some of this so others don’t try to do what I did and naively expect it to work. But the gist is that ds9 was never intended to handle a wide field, non-linear WCS situation well.
One takeaway I got is, updating to ds9v8 is a good idea if you want “approximately correct” astrometry for a quick look at a small portion of a focal plane!
Thanks for your patience. I’ve had time to take a look at your issue. I’ve followed the discussion on the forum you listed and downloaded your sample files. The discussions on how DS9 handles mosaics are more or less correct.
To generate a mosaic visualization, DS9 will read the first fits file, and then for each additional file, orient, scale, and rotate to match the WCS of the first file.
This functionality is only meant as a quick look capability. Back in the days when DS9 was first written, re-projection of mosaics took hours. The field of view was typically small and the WCS used were a simple linear system, such as TAN. In these cases, a simple matrix could generate a reasonable representation of the mosaic. The size or number of mosaic segments really did not matter.
In fact, most mosaics at the time did not use a WCS, but instead used IRAF DETSEC, DETSIZE keywords to define a simple linear system. (and we will not talk about WFPC2
In your case, you have a very wide field of view and a non-linear instrument projection plane. Re-projection is the best method to properly visualize your data.
Aside, up until version 7.6, the DS9 code tried to create a simple matrix by reading the CRVAL, CRPIX, CD, CDELT, and CROT keywords directly from the headers. Now, version 8.0 uses AST’s support to build a proper mapping between 2 defined WCS. Hence, the scrambled mess of version 7.6 vs the more or less correct placement of version 8.0.
Down the road DS9 might support some form of re-projection directly, but thats another major research endeavor.
3 Likes
One final comment from Bill Joye. They’re going to explore on-the-fly reprojections for future ds9 releases for rendering mosaics thanks to this discussion, so thank you, all!
Just to followup up on the discussion. One item I did not truly appreciate is the fact that LSST is using a non-linear WCS for each chip, valid for only that chip. Up until now, all mosaics data products from other projects used a linear WCS, which was valid for all of the chips. (actually, there is a WCS defined per chip, but the WCS was linear and superimposed nicely on top of the other WCSs)
Unique to your case at LSST, now you have a non-linear WCS per chip, and very important, that WCS is only valid for that chip, as the WCS solution falls apart as you leave the chip. Therefore, the strategy used by DS9 for rendering the mosaic no longer is valid.
For the next major release of DS9, I will explore the possibility of “on the fly re-projection” for rendering mosaics, as I’m sure LSST will not be the last project to take this approach.
2 Likes
Thanks for following up on this.
It might be worth letting them know that any large (tens to hundreds of chips) mosaic camera is going to have the same problems (non linear WCS for every chip outside the middle), because of optical distortions. For example, the same is true for the DECam astrometry solution of Bernstein et al.