Pipetask on images generated with ImSim

Support
#1

Dear all

I am having a problem in running the isr+characterizeImage+calibrate tasks, using pipetask, on some images that I have generated with ImSim.

First of all, let me point out my goal. With my inkind group, we are analyzing the ellipticity effects over the CCDs and, to start, we want to run the pipelines on perfectly clean images with only a grid of stars. By “perfectly clean” I mean no dark current, no atmosphere, no noise, no cosmic rays, etc… This generates images with a perfectly flat background at 0 counts everywhere, plus a grid of 15x15 stars, equally spaced, all with the same brightness and with a 0.7 arcsec fwhm, regulated by the flux parameter in the imsim config file.

By running pipetask on these images, the log displays an error, at the characterizeImage step, because the calculated PSF construction has a sigma=nan.

The error is also showed by setting output.readout.bias_level=50, so a non-zero, perfectly flat background at 50 counts.
By simply adding again the dark current, this error is removed and all the steps are performed.

Now, I have two questions

  1. Is there some workaround to make all the task work also with a perfectly flat background (zero or non-zero)?
  2. There is a strange behaviour concerning the cosmic-ray count. Despite the stars have all the same brightness and the same fwhm, and cosmic_ray_rate= 0.0, several cosmic rays are found, and the number decreases with decreasing fwhm of the initial guess of the PSF (characterizeImage:installSimplePsf.fwhm). Strangely, in an image with a 5x5 grid, 26 cosmic rays are found over 25 sources, and the star count is 24.

I have already received a lot of help on the slack channels, solving many other errors but at this point, the suggestion was to ask here.

#2

That is indeed very strange! Could you post the processing logs from a sample image please? It might make the problem clear, and debugging things without information as to the specifics is quite hard.

#3

It would also be helpful to share the specific LSST pipelines version being used, if that info isn’t readily available from the processing log(s). For instance, in the Rubin/CST tutorial material about running pipetask on DP0.2 simulated images, it’s quite important to be using a version of the LSST pipelines compatible with the relevant ImSim simulations.

#5

Sure, first of all here is the log for the task:


TIMESTAMP="$(date +'%Y%m%d_%H%M%S')"
pipetask --long-log run -p "${PWD}/drp_230727.yaml#isr,characterizeImage,calibrate"          -i "LSSTCam/raw/all,LSSTCam/calib,u/jchiang/calib_imSim2.0_w_2023_04"          -o pipetask_output          -b repo/          -d "instrument='LSSTCam' AND visit=5023071800004 AND detector=94"  -c characterizeImage:installSimplePsf.fwhm=2.5        --register-dataset-types  >& ./run_tasks012_$TIMESTAMP.log

INFO 2023-08-07T01:01:12.638-07:00 lsst.ctrl.mpexec.cmdLineFwk ()(cmdLineFwk.py:605) - QuantumGraph contains 3 quanta for 3 tasks, graph ID: ‘1691395272.636816-931505’
Quanta Tasks

 1               isr
 1 characterizeImage
 1         calibrate

WARNING 2023-08-07T01:01:24.350-07:00 lsst.isr (isr:{instrument: ‘LSSTCam’, detector: 94, exposure: 5023071800004, …})(isrTask.py:1056) - Making DETECTOR level kernel from AMP based brighter fatter kernels.
INFO 2023-08-07T01:01:24.358-07:00 lsst.isr (isr:{instrument: ‘LSSTCam’, detector: 94, exposure: 5023071800004, …})(isrTask.py:1319) - Converting exposure to floating point values.
INFO 2023-08-07T01:01:24.592-07:00 lsst.isr (isr:{instrument: ‘LSSTCam’, detector: 94, exposure: 5023071800004, …})(isrTask.py:1395) - Assembling CCD from amplifiers.
INFO 2023-08-07T01:01:25.020-07:00 lsst.isr (isr:{instrument: ‘LSSTCam’, detector: 94, exposure: 5023071800004, …})(isrTask.py:1462) - Masking non-finite (NAN, inf) value pixels.
INFO 2023-08-07T01:01:25.041-07:00 lsst.isr (isr:{instrument: ‘LSSTCam’, detector: 94, exposure: 5023071800004, …})(isrTask.py:1466) - Widening saturation trails.
INFO 2023-08-07T01:01:25.488-07:00 lsst.isr (isr:{instrument: ‘LSSTCam’, detector: 94, exposure: 5023071800004, …})(isrTask.py:1493) - Applying brighter-fatter correction using kernel type <class ‘numpy.ndarray’> / gains <class ‘dict’>.
INFO 2023-08-07T01:01:37.445-07:00 lsst.isr (isr:{instrument: ‘LSSTCam’, detector: 94, exposure: 5023071800004, …})(isrTask.py:1517) - Finished brighter-fatter correction in 2 iterations.
INFO 2023-08-07T01:01:37.466-07:00 lsst.isr (isr:{instrument: ‘LSSTCam’, detector: 94, exposure: 5023071800004, …})(isrTask.py:1530) - Ensuring image edges are masked as EDGE to the brighter-fatter kernel size.
INFO 2023-08-07T01:01:37.468-07:00 lsst.isr (isr:{instrument: ‘LSSTCam’, detector: 94, exposure: 5023071800004, …})(isrTask.py:1574) - Applying fringe correction after flat.
INFO 2023-08-07T01:01:37.468-07:00 lsst.isr.fringe (isr:{instrument: ‘LSSTCam’, detector: 94, exposure: 5023071800004, …})(fringe.py:155) - Filter not found in FringeTaskConfig.filters. Skipping fringe correction.
INFO 2023-08-07T01:01:37.662-07:00 lsst.isr (isr:{instrument: ‘LSSTCam’, detector: 94, exposure: 5023071800004, …})(isrTask.py:1626) - Interpolating masked pixels.
WARNING 2023-08-07T01:01:38.012-07:00 lsst.isr (isr:{instrument: ‘LSSTCam’, detector: 94, exposure: 5023071800004, …})(isrTask.py:2504) - No rough magnitude zero point defined for filter r.
INFO 2023-08-07T01:01:38.012-07:00 lsst.isr (isr:{instrument: ‘LSSTCam’, detector: 94, exposure: 5023071800004, …})(isrTask.py:2512) - Setting rough magnitude zero point for filter r: 31.692803
INFO 2023-08-07T01:01:38.944-07:00 lsst.ctrl.mpexec.singleQuantumExecutor (isr:{instrument: ‘LSSTCam’, detector: 94, exposure: 5023071800004, …})(singleQuantumExecutor.py:313) - Execution of task ‘isr’ on quantum {instrument: ‘LSSTCam’, detector: 94, exposure: 5023071800004, …} took 16.634 seconds
INFO 2023-08-07T01:01:39.118-07:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:524) - Executed 1 quanta successfully, 0 failed and 2 remain out of total 3 quanta.
INFO 2023-08-07T01:01:40.072-07:00 lsst.characterizeImage (characterizeImage:{instrument: ‘LSSTCam’, detector: 94, visit: 5023071800004, …})(characterizeImage.py:521) - PSF estimation initialized with ‘simple’ PSF
INFO 2023-08-07T01:01:40.342-07:00 lsst.characterizeImage.repair (characterizeImage:{instrument: ‘LSSTCam’, detector: 94, visit: 5023071800004, …})(repair.py:228) - Identified 0 cosmic rays.
INFO 2023-08-07T01:01:40.551-07:00 lsst.characterizeImage.detection (characterizeImage:{instrument: ‘LSSTCam’, detector: 94, visit: 5023071800004, …})(detection.py:551) - Setting factor for negative detections equal to that for positive detections: 1.000000
INFO 2023-08-07T01:01:41.219-07:00 lsst.characterizeImage.detection (characterizeImage:{instrument: ‘LSSTCam’, detector: 94, visit: 5023071800004, …})(detection.py:641) - Detected 225 positive peaks in 225 footprints and 0 negative peaks in 0 footprints to 50 +ve and 50 -ve sigma
INFO 2023-08-07T01:01:41.491-07:00 lsst.characterizeImage.detection (characterizeImage:{instrument: ‘LSSTCam’, detector: 94, visit: 5023071800004, …})(detection.py:666) - Resubtracting the background after object detection
INFO 2023-08-07T01:01:41.898-07:00 lsst.characterizeImage.measurement (characterizeImage:{instrument: ‘LSSTCam’, detector: 94, visit: 5023071800004, …})(sfm.py:292) - Measuring 225 sources (225 parents, 0 children)
INFO 2023-08-07T01:01:42.053-07:00 lsst.characterizeImage.measurePsf (characterizeImage:{instrument: ‘LSSTCam’, detector: 94, visit: 5023071800004, …})(measurePsf.py:166) - Measuring PSF
INFO 2023-08-07T01:01:42.166-07:00 lsst.characterizeImage.measurePsf (characterizeImage:{instrument: ‘LSSTCam’, detector: 94, visit: 5023071800004, …})(measurePsf.py:182) - PSF star selector found 141 candidates
INFO 2023-08-07T01:01:42.167-07:00 lsst.characterizeImage.measurePsf.reserve (characterizeImage:{instrument: ‘LSSTCam’, detector: 94, visit: 5023071800004, …})(reserveSourcesTask.py:110) - Reserved 28/141 sources
INFO 2023-08-07T01:01:42.167-07:00 lsst.characterizeImage.measurePsf (characterizeImage:{instrument: ‘LSSTCam’, detector: 94, visit: 5023071800004, …})(measurePsf.py:193) - Sending 113 candidates to PSF determiner
INFO 2023-08-07T01:01:42.642-07:00 lsst.characterizeImage.measurePsf (characterizeImage:{instrument: ‘LSSTCam’, detector: 94, visit: 5023071800004, …})(measurePsf.py:206) - PSF determination using 113/113 stars.
INFO 2023-08-07T01:01:42.945-07:00 lsst.characterizeImage (characterizeImage:{instrument: ‘LSSTCam’, detector: 94, visit: 5023071800004, …})(characterizeImage.py:431) - iter 1; PSF sigma=nan, dimensions=(41, 41); median background=0.00
ERROR 2023-08-07T01:01:42.946-07:00 lsst.ctrl.mpexec.singleQuantumExecutor (characterizeImage:{instrument: ‘LSSTCam’, detector: 94, visit: 5023071800004, …})(singleQuantumExecutor.py:300) - Execution of task ‘characterizeImage’ on quantum {instrument: ‘LSSTCam’, detector: 94, visit: 5023071800004, …} failed. Exception RuntimeError: PSF sigma is NaN, cannot continue PSF determination.
ERROR 2023-08-07T01:01:42.976-07:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:508) - Task <TaskDef(lsst.pipe.tasks.characterizeImage.CharacterizeImageTask, label=characterizeImage) dataId={instrument: ‘LSSTCam’, detector: 94, visit: 5023071800004, …}> failed; processing will continue for remaining tasks.
Traceback (most recent call last):
File “/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2023_19/conda/envs/lsst-scipipe-6.0.0-exact-ext/share/eups/Linux64/ctrl_mpexec/g45d0adf2d3+61755f3ad1/python/lsst/ctrl/mpexec/mpGraphExecutor.py”, line 478, in _executeQuantaInProcess
self.quantumExecutor.execute(qnode.taskDef, qnode.quantum)
File “/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2023_19/conda/envs/lsst-scipipe-6.0.0-exact-ext/share/eups/Linux64/ctrl_mpexec/g45d0adf2d3+61755f3ad1/python/lsst/ctrl/mpexec/singleQuantumExecutor.py”, line 173, in execute
result = self._execute(taskDef, quantum)
File “/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2023_19/conda/envs/lsst-scipipe-6.0.0-exact-ext/share/eups/Linux64/ctrl_mpexec/g45d0adf2d3+61755f3ad1/python/lsst/ctrl/mpexec/singleQuantumExecutor.py”, line 298, in _execute
self.runQuantum(runTask, quantum, taskDef, limited_butler)
File “/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2023_19/conda/envs/lsst-scipipe-6.0.0-exact-ext/share/eups/Linux64/ctrl_mpexec/g45d0adf2d3+61755f3ad1/python/lsst/ctrl/mpexec/singleQuantumExecutor.py”, line 592, in runQuantum
task.runQuantum(butlerQC, inputRefs, outputRefs)
File “/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2023_19/conda/envs/lsst-scipipe-6.0.0-exact-ext/share/eups/Linux64/pipe_tasks/g147c7b1cba+32d6b977c5/python/lsst/pipe/tasks/characterizeImage.py”, line 352, in runQuantum
outputs = self.run(**inputs)
File “/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2023_19/conda/envs/lsst-scipipe-6.0.0-exact-ext/share/eups/Linux64/utils/ga34f01d9d8+f0a1800e64/python/lsst/utils/timer.py”, line 347, in timeMethod_wrapper
res = func(self, *args, **keyArgs)
File “/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2023_19/conda/envs/lsst-scipipe-6.0.0-exact-ext/share/eups/Linux64/pipe_tasks/g147c7b1cba+32d6b977c5/python/lsst/pipe/tasks/characterizeImage.py”, line 434, in run
raise RuntimeError(“PSF sigma is NaN, cannot continue PSF determination.”)
RuntimeError: PSF sigma is NaN, cannot continue PSF determination.
INFO 2023-08-07T01:01:43.120-07:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:524) - Executed 1 quanta successfully, 1 failed and 1 remain out of total 3 quanta.
ERROR 2023-08-07T01:01:43.121-07:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:464) - Upstream job failed for task <TaskDef(lsst.pipe.tasks.calibrate.CalibrateTask, label=calibrate) dataId={instrument: ‘LSSTCam’, detector: 94, visit: 5023071800004, …}>, skipping this task.
ERROR 2023-08-07T01:01:43.121-07:00 lsst.daf.butler.cli.utils ()(utils.py:1110) - Caught an exception, details are in traceback:
Traceback (most recent call last):
File “/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2023_19/conda/envs/lsst-scipipe-6.0.0-exact-ext/share/eups/Linux64/ctrl_mpexec/g45d0adf2d3+61755f3ad1/python/lsst/ctrl/mpexec/cli/cmd/commands.py”, line 229, in run
script.run(qgraphObj=qgraph, **kwargs)
File “/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2023_19/conda/envs/lsst-scipipe-6.0.0-exact-ext/share/eups/Linux64/ctrl_mpexec/g45d0adf2d3+61755f3ad1/python/lsst/ctrl/mpexec/cli/script/run.py”, line 199, in run
f.runPipeline(qgraphObj, taskFactory, args)
File “/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2023_19/conda/envs/lsst-scipipe-6.0.0-exact-ext/share/eups/Linux64/ctrl_mpexec/g45d0adf2d3+61755f3ad1/python/lsst/ctrl/mpexec/cmdLineFwk.py”, line 745, in runPipeline
executor.execute(graph)
File “/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2023_19/conda/envs/lsst-scipipe-6.0.0-exact-ext/share/eups/Linux64/ctrl_mpexec/g45d0adf2d3+61755f3ad1/python/lsst/ctrl/mpexec/mpGraphExecutor.py”, line 410, in execute
self._executeQuantaInProcess(graph, self.report)
File “/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2023_19/conda/envs/lsst-scipipe-6.0.0-exact-ext/share/eups/Linux64/ctrl_mpexec/g45d0adf2d3+61755f3ad1/python/lsst/ctrl/mpexec/mpGraphExecutor.py”, line 534, in _executeQuantaInProcess
raise MPGraphExecutorError(“One or more tasks failed during execution.”)
lsst.ctrl.mpexec.mpGraphExecutor.MPGraphExecutorError: One or more tasks failed during execution.

#6

Concerning the version, I cannot find how to check that neither within pipetask -h, nor in the pipetask — LSST Science Pipelines page.

#7

OK, that’s a very good start, at least we can see where it’s going wrong, thanks! Which PSF determiner are you using? Is it the default, or are you setting something specific?

And regarding the versions (which will also determine the default config) perhaps just saying how you installed/setup the stack would be a good start to understanding what you’re running.

#8

I believe your output collection should have a packages dataset that will give the versions of all software used.

butler retrieve-artifacts $REPO ./out -d packages --collections $OUTPUT_COLLECTION

will copy that dataset into an out subdirectory of the current directory; you can then look at the YAML file to see, among other things, the conda_env and ip_isr lines.

But describing how you set up the stack might be easier :slight_smile:

#9

Hi Vittorio,

I’m not certain, but I think it’s quite possible that processing an image with identically zero background is breaking some assumptions about pixel statistics within the science pipelines. I think you can probably get what you want though by simulating sufficiently bright stars with a small but nonzero background where the errors from signal-to-noise should still be quite small.

#10

My first reply is to @jmeyers314 : I have tried to change the background to a flat, but non-zero one, and the sigma of the PSF is still a nan. Only by introducing a noise (e.g., by setting dark_current to 3.0)

To @merlin: I am using the default PSF determiner. I have used the butler command to retrieve the .yaml file and I found, among the others,

conda: 23.3.1
conda-package-handling: 2.0.2
conda-package-streaming: 0.7.0
conda_env: lsst-scipipe-6.0.0
ip_isr: gcba2a62c5a+038a6eaf3f with boost=unknown eigen=unknown
stack-data: 0.6.2
stack_data: 0.6.2

#11

It looks like that version of ip_isr is w_2023_21.

#12

I’m just spitballing here, but something which is totally flat but non-zero still has zero standard deviation, so would still mess up the sigma calculation. So it would depend on how the dark current is implemented as to whether that would be expected to fix that.

#13

Well, is that fine or should I update it?

#14

Oh, that’s fine, if a little old. I was just pointing it out for anyone who wanted to try to reproduce.

It does sound like you will have to add some amount of noise to make the existing code work.