Sometimes it is handy to visualize an entire visit or focal plane at once. We have a stack tool for this in pipe_drivers, called visualizeVisit.py, but it uses camera geometry rather than WCS information to line up the CCDs. So I decided to try ds9.
The handy command ds9 -mosaic filename*.fits is intended to do just this: display several images at once and align them with their WCS information. However:
It looks to me like the WCS for each consecutive file that ds9 accesses is even worse than the previous one. So if you are just displaying a few CCDs at at time, you probably won’t notice, but if you do more than about 10, things start to get weird.
So far, I have tried this and gotten similar garbled patterns for a variety of DECam visits for both calexps and diffims. I have been able to reproduce the view exactly with lsst-dev’s ds9 version 7.5 and my own machine’s ds9 version 7.6b8. I have also done a cursory look at the WCS-related header values and compared them to what ds9 thinks it has read in; they seem to agree.
If you would like to examine the particular images I have shown above, they are the difference image files which live in /project/mrawls/hits2015/rerun/slurm6/deepDiff/v411420.
I should note that the utility of the ds9 -mosaic command was most recently brought to my attention by Shenming Fu and a colleague at PCW 2018. They found a similar issue when displaying DECam images in ds9 and originally thought their astrometry had failed in some way, but we did not have time to get to the root of the issue.
Are you using a version of ds9 that supports the version of WCS distortions that we write?
In general I strongly recommend using LSST tools; it’s not hard to produce a shell command that can absorb files specified on the command line (and it’ll understand e.g. Mask too)
I’ve observed this kind of behaviour from ds9 over a very long time (a decade or more). I’ve always thought it was due to nonlinearities in the transformations. It may make a difference which image you give it first (i.e., it’s trying to convert all the WCSes into the projection of the first), so maybe try to specify an image from the center of the focal plane as the first one, or put all the WCSes into the same projection yourself (this used to be a natural outcome from meas_mosaic).
I don’t know how to check to see if I’m using a version of ds9 that supports the version of WCS distortions that we write, but I will try the newest beta in addition to the two versions I have already tried. If there is an “LSST tool” that allows me to quickly display ~50-100 calexps at once in their correct relative sky locations, I would love to know about it.
It’s a little scary to hear this kind of behavior has been observed over years(!). I strongly suspect LSST users will someday try using ds9 to mosaic a bunch of images at once. This seems like a basic functionality we and ds9 should support.
If you can’t get ds9 to work, you could always warp images to a common DiscreteSkyMap tract. ds9 will handle those much better (no non-linear WCSes).
Hmm… we went through a discussion of this before. (Check out Erin Sheldon’s slack avatar! That came from when @laurenam(?) saw this ). I thought I remembered that we thought this came down to ds9 not knowing how to handle TAN-SIP.
But, I thought we are also now doing this now with our DC2 outputs (but I can’t remember at what phase). So perhaps there are headers missing?
Yeah, I saw this on DM-1573, where I concluded that
The strange arrangement in the CCDs seen in FocalPlane_0001228a.png seems to be an issue with ds9 (getting confused with the wcs?) rather than an erroneous astrometric solution…
Inability to interpret TAN-SIP seems the likely source of DS9”s “confusion”.
Bruce Berriman (Caltech) doesn’t have an account here (yet…? ) but said it was OK to post what he emailed to me about this.
The problem you describe is, it turns out, one of the reasons we built the Montage mosaic engine (http://montage.ipac.caltech.edu); we have come across similar problems but in different contexts. The solution to your conundrum lies in the fact that you have to do full reproductions and (if needed, resampling) of every image. DS9 isn’t intended to do this - it treats one image as a template and forces the rest of the images to comply with it. This is actually OK for looking at a few images, but is not an approach intended to visualize substantial collections of images.
The simplest approach might be for us to process the data and use the workflow to illustrate for you how the processing is done. Would you be able to make the data accessible to us?
Please note that Montage is written in C for performance, and we are testing Python binary extensions that will be released at the end of October.
Given that response, this seems like something they should call out in the user guide and/or FAQ. I couldn’t find anything about “don’t try to mosaic a dozen images” in there:
It’s not the number of images that’s the root of the problem (I’ve successfully used -mosaicwcs with several tens of inputs): it’s the non-linearity in the astrometry. If the WCSes are non-linear with different CRVALn, then having more images means you’re less likely to be able to use simple transforms to the template it chooses.
Yes indeed @timj; feeding ds9 one of the more central DECam chips first results in slightly less wonkiness.
I now think it is somewhat understandable for ds9 to behave this way, but it should absolutely be mentioned in the documentation/FAQ that trying to load more than a ~few images with differing non-linear WCSs will result in some really bad mosaicing. Otherwise, the naive user may think this indicates an actual problem with the astrometry!
Here is the result on the same images with ds9 -mosaic diffexp-2*.fits diffexp-0*.fits diffexp-1*.fits diffexp-3*.fits diffexp-4*.fits diffexp-5*.fits diffexp-6*.fits (the central two DECam CCDs are numbers 28 and 35).
Sorry… I wasn’t getting notifications on the discussion. I was talking to Jim Chiang and trying to understand more about this and looking more carefully at the instructions.
Thanks @cwalter, but it turns out the issue is deeper than this. Mosaics in ds9 are one of wcs, iraf, or wfpc2, and while it might be clearer/safer to explicitly specify I want a wcs mosaic, that is the type created by default in the absence of the iraf-y header keywords DETSEC and DETSIZE and the presence of a valid WCS.
I’ve exchanged several emails with John Good (Caltech) about how the Montage tool can do a better job than ds9 (with the caveat that it is a multi-step file-based process). He was able to create a much more accurate reprojection of my images that uses each one’s WCS properly. I also have an open query with the ds9 folks who said they’d get back to me within about a week. I’ll follow up here when they do.
) but said it was OK to post what he emailed to me about this.