what I’m really trying to do is take some postage-stamp type images of blended sources, run them through the deblender implementation as it currently stands, and examine the measurements that the deblender gets out, whatever they may be. I’m not sure what the easiest path to this is. It seems like passing arbitrary images into the deblending task requires a fair bit of mocking of interfaces, and I found processFile while looking to see if someone had already done this mocking.
The images all have some known angular scale, and filter transmission curve information. Some of them have WCS information, but others are from simulations (processed into simulated observations). I’m not sure that’s enough in common to make building a data store worthwhile. (When I looked at it, it seemed to include things like specifying camera geometry, which due to the heterogeneity of the data doesn’t make much sense. I might be misremembering!)
I am trying to do pretty much the same. I’m trying to test the deblender implementation with simulated images from GalSim/WeakLensingDeblending. I added WCS information in the headers but I am having trouble trying to use processFile. I also tried processCcd and processEimage with no success.
@josePhoenix and @fjaviersanchez: Thanks for the added info. I am looking into how to resurrect processFile. In your particular use case, I’d like to be able to run just the bits you need. I’ll look into that too. Is it possible to attach an example image that I can play with?
Sure, here’s a typical blended object postage stamp from my simulation/analysis pipeline. This one is from a CANDELS mosaic, degraded to 0.7" seeing. example_blend_EGS_F606W_0.7arcsec.fits (14.1 KB)
I believe gcc is used to build the conda-distributed binaries, but only the runtime gets installed when you install the stack with conda (which is the proper behavior, IMHO – you shouldn’t need the whole toolchain to just run the code).
If you want to also build your own code using the conda-delivered gcc, run conda install gcc. Note that this is not an officially supported way to do it; very few people use it for daily development, so I’m not sure how well we’d be able to help you if you run into problems. It’s easy to try, though, so it may be worth a shot.
@josePhoenix: I don’t think I can make the pipeline work on such small postage stamps. The problem is that the deblender requires a list of peaks and the list of peaks comes from the detection phase. The detection phase depends on background and psf estimation. With such a small postage stamp it’s hard to estimate the background and there are no PSF source, so things fall apart.
The options are to use larger cutouts of the mosaic so that we get a few stars in there or to try to feed these postage stamps to the deblender directly. If we go with the latter, we’ll need a catalog of peaks to go with each postage stamp. With the former, I think we can just run things through the way we do normal processing.
@fjaviersanchez: I now have a solution that I hope will work for you. It needs some more work, but I thought I’d put it out there now so I can get comments. This solution makes it fairly trivial to ingest arbitrary files into LSST data repositories so you can treat them like any other image in the stack. In order to use it, you’ll need the tickets/DM-6924 branch of both obs_file and pipe_tasks.
To run processFile is a three step process. The first two are one time only if your input data are static. The following assumes you have a relatively recent master build and the ticket branches setup.
Create a repository to feed the ingest. This involves making a directory containing a file called _mapper. I made a directory called test_out. The content of the file is the single line lsst.obs.file.FileMapper.
Ingest the files. This puts the files in the location expected by the rest of the machinery and builds a database to lookup the available images. $> ingestFiles.py test_out test_imgs/test.fits.gz
The first argument is a valid repository and the rest of the arguments will be treated as images to ingest. If no output is specified the images will be ingested into the repository specified (i.e. test_out).
Run processCcd $> processCcd.py test_out/ --id filename='test.fits.gz' --config isr.noise=100000 isr.addNoise=True --output test_out
Just specify the file you want to process. In the case of your data, I also had to add background noise, otherwise certain algorithms, e.g. cosmic ray detection, do not behave well. The output argument is required.
Below is an example produced by the displaySources.py utility. If you have display_ds9 setup, you can specify a repository and a file and it will plot the parent sources as blue circles and the children as red plusses. Looking at this image, there are some things to tune. For example, I see some faint sources I would have expected to be detected, but maybe that is an artifact of my arbitrary choice of background noise. FYI, the blue patches are the pixels flagged as detected.
Thanks Simon! That’s good to know. Would it be totally crazy to just stick our synthetic PSF and a bit of representative background on the side of a tile? Depending on the extent that the deblender performance is dependent on the PSF estimation, we might mislead ourselves in our analysis, I suppose.
We included some sources below the detection level on purpose in order to see how they affect so, maybe it’s a combination of your choice of noise and these sources.
I don’t think it’s crazy, but it might be a little tricky to do. The background should be fairly easy to attach, but your PSF will have to be represented using our classes. Maybe @jbosch has more to say about how to mock the exposure so it can be fed directly to the deblender.
I think it’ll take a lot of trial-and-error work to convince our PSF modeling code to work well on fairly artificial inputs like this; it’s really designed to expect a full CCD of star and galaxy images. Over in the other thread on this topic, I’ve posted some instructions for trying to run the relevant pieces of the pipeline on data like this, which doesn’t use processCcd.py at all.
Another option would be to follow those instructions for how to attach a PSF to one of our Exposure objects, save that to FITS, and then run processCcd.py on that with all of the PSF modeling steps disabled, but I still think this is likely to be harder than just starting from the snippet I’ve linked to.
Hi Simon. I was trying to follow your steps to run processFile on a simulation image, and I may have some problem when I tried to build pipe_tasks-tickets-DM-6924:
cd pipe_tasks-tickets-DM-6924 setup -k -r . scons opt=3
… The following tests failed: ~/miniconda2/opt/lsst/pipe_tasks-tickets-DM-6924/tests/.tests/testProcessCcd.py.failed ~/miniconda2/opt/lsst/pipe_tasks-tickets-DM-6924/tests/.tests/testTransform.py.failed ~/miniconda2/opt/lsst/pipe_tasks-tickets-DM-6924/tests/.tests/testPhotoCal.py.failed ~/miniconda2/opt/lsst/pipe_tasks-tickets-DM-6924/tests/.tests/testCoadds.py.failed 4 tests failed scons: *** [checkTestStatus] Error 1 scons: building terminated because of errors.
I am not sure where the problem is (obs_file-tickets-DM-6924 could be built without error). FYI, I installed Stack v12.0 with miniconda on a CentOS6 machine. Please let me know if you need any other information. Any hints would be appreciated!
It is possible that v12.0 is too old. Things on master move along pretty quickly and I made no attempt to be compatible with the most recent release. You could install master of everything and try again. Is that an option for you?
If you look in the .failed files, it will give you the traceback. That would also be useful information.
Chiming in here, it looks like following master would make sense for me as well, but the last time I tried the lsstsw installation I got this:
You are attempting to run "setup" which requires administrative
privileges, but more information is needed in order to do so.
Authenticating as "root"
Password:
Since I don’t have root access to the compute servers, I couldn’t use it. Is there a switch to do per-user installs?
Unfortunately, the eups setup command shadows a shell command. This results in the above error if you do not have the eups bin directory in your path. @josePhoenix what steps did you follow the ended up in that error. If you first run $> bin/deploy in your lsstsw install, it should put all the right things in your path.
Thanks for your reply!
I just tried to reinstall the master with miniconda, but the same test failures came out again. I looked into the .failed files:
testProcessCcd and testTransform both have the same error: AttributeError: 'module' object has no attribute 'starSelectorRegistry'
testPhotoCal has the error: TypeError: __init__() got an unexpected keyword argument 'refObjLoader'
and testCoadds has: ERROR: testAlgMetadataOutput (__main__.CoaddsTestCase) Test to see if algMetadata is persisted correctly from MeasureMergedCoaddSourcesTask. ... ERROR: testForcedPhotCcdTask (__main__.CoaddsTestCase) ... ERROR: testForcedPhotCoaddTask (__main__.CoaddsTestCase) ... FAIL: testLeaks (lsst.utils.tests.MemoryTestCase) !Check for memory leaks in the preceding tests AssertionError: Leaked 74 blocks
Would you suggest any clue about these errors? I was wondering if it is possible to install the version of Stack that is compatible with tickets-DM-6924.
Hmmm. It looks like your stack is somehow inconsistent. It’s possible the ticket has fallen behind master. I will take a look and see if I can reproduce. Sorry for the issues you are having.
@rbliu Sorry to take so long to get back to you. I think there may be a couple of things here. One is that I think you need a newer stack. The second is that I found that there are some commits I never pushed.
If you get a new master stack and pull the ticket branches for obs_file and pipe_tasks, you should be able to essentially follow the instructions in this post with the small change that processFile.py should be called processCcd.py. You can find the file to run this example linked from this post.
