Would like to do photometry on Auxtel Spectroscopic images

sylvielsstfr · November 3, 2022, 2:50pm

I am running a the atmospec pipeline (GitHub - lsst/atmospec: Package containing elements used for the atmospheric spectroscopy) that process raw spectroscopic images from Auxtel (LATISS) at USDF.
I am using Stack version w_2022_39.

Inside the spectroscopic images, I would-like to make the photometry of a few stars in the field of view in order to monitor the photometric stability along the night as I do with the spectra produced by atmospec.
For the photometry, I don’t care of the unit, but I want to be sure that I identifies the same stars from one exposure from another, even if the field rotate during the night (the same star has not the same pixel position).

Atmospec generate postISRCCD but not CALEXP.
It means I have the following data products generated among others:

icSrc
icExp ,
icExpBackground,
postISRCCD
I would like to have the calibrated photometry of a few not saturated stars in the field of view.

From what I understand, I have two options:

a) run a calibration pipeline like it is explained in this link : Log In - Confluence
b) process icSrc or postISRCCD myself in a notebook.
In fact my both attempts failed, that is why I am asking for some help from the community.

In the attempt a) I run the command:
pipetask run -b /sdf/group/rubin/repo/main -p /sdf/home/d/dagoret/repos/repos_w_2022_39/drp_pipe/pipelines/LATISS/DRP.yaml#isr,characterizeImage,calibrate -i LATISS/raw/all,refcats,LATISS/calib -o u/dagoret/photo/empty~holo4/20220630 -d “exposure.day_obs=20220630 and exposure.seq_num=45 and instrument=‘LATISS’” --register-dataset-types --clobber-outputs -j 8

and I get the following error:

lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback:
Traceback (most recent call last):
File “/sdf/group/rubin/sw/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g5106228df7+8f386c43b7/python/lsst/ctrl/mpexec/cli/cmd/commands.py”, line 159, in run
qgraph = script.qgraph(pipelineObj=pipeline, **kwargs, show=show)
File “/sdf/group/rubin/sw/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g5106228df7+8f386c43b7/python/lsst/ctrl/mpexec/cli/script/qgraph.py”, line 196, in qgraph
qgraph = f.makeGraph(pipelineObj, args)
File “/sdf/group/rubin/sw/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/ctrl_mpexec/g5106228df7+8f386c43b7/python/lsst/ctrl/mpexec/cmdLineFwk.py”, line 563, in makeGraph
qgraph = graphBuilder.makeGraph(
File “/sdf/group/rubin/sw/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/pipe_base/g646899f3d4+9b4274c3ef/python/lsst/pipe/base/graphBuilder.py”, line 1304, in makeGraph
return scaffolding.makeQuantumGraph(metadata=metadata, datastore=self.datastore)
File “/sdf/group/rubin/sw/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/pipe_base/g646899f3d4+9b4274c3ef/python/lsst/pipe/base/graphBuilder.py”, line 1147, in makeQuantumGraph
qset = task.makeQuantumSet(unresolvedRefs=self.unfoundRefs, datastore_records=datastore_records)
File “/sdf/group/rubin/sw/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/pipe_base/g646899f3d4+9b4274c3ef/python/lsst/pipe/base/graphBuilder.py”, line 485, in makeQuantumSet
raise exc
File “/sdf/group/rubin/sw/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/pipe_base/g646899f3d4+9b4274c3ef/python/lsst/pipe/base/graphBuilder.py”, line 463, in makeQuantumSet
tmpQuanta = q.makeQuantum(datastore_records)
File “/sdf/group/rubin/sw/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/pipe_base/g646899f3d4+9b4274c3ef/python/lsst/pipe/base/graphBuilder.py”, line 333, in makeQuantum
helper.adjust_in_place(self.task.taskDef.connections, self.task.taskDef.label, self.dataId)
File “/sdf/group/rubin/sw/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/pipe_base/g646899f3d4+9b4274c3ef/python/lsst/pipe/base/connections.py”, line 750, in adjust_in_place
adjusted_inputs_by_connection, adjusted_outputs_by_connection = connections.adjustQuantum(
File “/sdf/group/rubin/sw/conda/envs/lsst-scipipe-4.1.0/share/eups/Linux64/pipe_base/g646899f3d4+9b4274c3ef/python/lsst/pipe/base/connections.py”, line 634, in adjustQuantum
raise FileNotFoundError(
FileNotFoundError: Not enough datasets (0) found for non-optional connection isr.flat (flat) with minimum=1 for quantum data ID {instrument: ‘LATISS’, detector: 0, exposure: 2022063000045, …}.

For the option b) I found the example in StackClub the BrighterFatter example here

github.com

LSSTScienceCollaborations/StackClub/blob/master/ImageProcessing/BrighterFatterCorrection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Analysis of Beam Simulator Images and Brighter-fatter Correction\n",
    "<br>Developer(s): **Andrew Bradshaw** ([@andrewkbradshaw](https://github.com/LSSTScienceCollaborations/StackClub/issues/new?body=@andrewkbradshaw))\n",
    "<br>Maintainer(s): **Andrew Bradshaw** ([@andrewkbradshaw](https://github.com/LSSTScienceCollaborations/StackClub/issues/new?body=@andrewkbradshaw))\n",
    "<br>Level: **Advanced**\n",
    "<br>Last Verified to Run: **2021-09-10**\n",
    "<br>Verified Stack Release: **w_2021_33**\n",
    "\n",
    "This notebook demonstrates the [brighter-fatter systematic error](https://arxiv.org/abs/1402.0725) on images of stars and galaxies illuminated on an ITL-3800C-002 CCD at the [UC Davis LSST beam simulator laboratory](https://arxiv.org/abs/1411.5667). Using a series of images at increasing exposure times, we demonstrate the broadening of image profiles on DM stack shape measurements, and a [possible correction method](https://arxiv.org/abs/1711.06273) which iteratively applies a kernel to restore electrons to the pixels from which they were deflected. To keep things simple, for now we skip most DM stack instrument signature removal (ISR) and work on a subset of images which are already processed arrays (500x500) of electrons.\n",
    "\n",
    "### Learning Objectives:\n",
    "\n",
    "After working through this tutorial you should be able to: \n",
    "1. Characterize and measure objects (stars/galaxies) in LSST beam simulator images\n",
    "2. Test the Brighter-Fatter kernel correction method on those images\n",

This file has been truncated. show original

(Examples in DP0 start not before CALEXP).

The interest is that this example start from an afwImage.ExposureF and my postISRCCD is an ExposureF.
Thus I have implemented my notebook according BrighterFatter example here:

github.com

sylvielsstfr/AuxTelComm/blob/main/notebooks_usdf/ExtractSources/ConvertExposuresToSources.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0b8c2138-5194-4d03-8b77-be24e2cc8ee3",
   "metadata": {},
   "source": [
    "# Convert postISRCCD into sources"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "740e34fb-3fef-45a8-941b-c229742ab016",
   "metadata": {},
   "source": [
    " work with Weakly_2022_39\n",
    "- use jupyter kernel LSST\n",
    "\n",
    "\n",
    "- author : Sylvie Dagoret-Campagne\n",

This file has been truncated. show original

In my notebook, I run

charResult = charTask.run(postisrccd)
calTask = CalibrateTask

But I wonder what means the detected sources table obtained ?

id	coord_ra	coord_dec	Xpix	Ypix	abs_chld_instFlux	abs_paren_instFlux

0 12 1.510130 -0.563972 2432.993317 365.016924 106265.039866 106265.039866
1 16 1.510073 -0.563925 2337.996199 476.001635 121169.724625 121169.724625
2 19 1.510687 -0.563800 3476.978856 640.938890 32335.204833 32335.204833
3 26 1.510329 -0.563703 2846.006750 908.993812 376733.309820 376733.309820
4 28 1.509605 -0.563650 1544.033540 1142.953559 24901.989343 24901.989343
5 31 1.510847 -0.563429 3841.019240 1409.957118 64877.781231 64877.781231
6 41 1.510784 -0.563360 3740.016618 1567.993269 590321.563767 590321.563767
7 58 1.510565 -0.563151 3383.984781 2053.985284 105584.434579 105584.434579
8 61 1.510608 -0.563134 3466.967400 2082.050870 21445.474154 21445.474154
9 64 1.510405 -0.563138 3098.033523 2108.056988 52053.675664 52053.675664
10 67 1.509898 -0.563071 2191.000000 2336.026168 101769.663866 101769.663866
11 74 1.509168 -0.563048 870.000423 2510.018892 254916.210549 254916.210549
12 77 1.509140 -0.562994 828.989260 2630.033419 37466.136018 37466.136018
13 87 1.510791 -0.562751 3874.052338 2873.020801 22994.292936 22994.292936
14 105 1.510278 -0.562524 2989.001825 3448.040981 69675.145451 69675.145451
15 107 1.510210 -0.562500 2869.106049 3510.168873 13262.364749 13262.364749

But in the above table, I had
calConfig.doAstrometry = False

This what coord_ra and coord_dec means ?

Because I would like to make some astrometry with

calConfig.doAstrometry = True

But in that case, I get an error:

RuntimeError Traceback (most recent call last)
Input In [27], in <cell line: 2>()
1 # for stack v16.0+22, change to calTask.run(charResult.exposure)
----> 2 calResult = calTask.run(charResult.exposure, background=charResult.background,
3 icSourceCat = charResult.sourceCat)
6 print("Detected “,len(calResult.sourceCat),” objects ")

File /opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-4.1.0/Linux64/utils/g6f6f670c95+16108f2031/python/lsst/utils/timer.py:347, in timeMethod..decorator_timer..timeMethod_wrapper(self, *args, **keyArgs)
338 logInfo(
339 obj=self,
340 prefix=func.name + “Start”,
(…)
344 stacklevel=stacklevel,
345 )
346 try:
→ 347 res = func(self, *args, **keyArgs)
348 finally:
349 logInfo(
350 obj=self,
351 prefix=func.name + “End”,
(…)
355 stacklevel=stacklevel,
356 )

File /opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-4.1.0/Linux64/pipe_tasks/g369a80f31c+437b643543/python/lsst/pipe/tasks/calibrate.py:603, in CalibrateTask.run(self, exposure, exposureIdInfo, background, icSourceCat)
601 if self.config.doAstrometry:
602 try:
→ 603 astromRes = self.astrometry.run(
604 exposure=exposure,
605 sourceCat=sourceCat,
606 )
607 astromMatches = astromRes.matches
608 matchMeta = astromRes.matchMeta

File /opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-4.1.0/Linux64/utils/g6f6f670c95+16108f2031/python/lsst/utils/timer.py:347, in timeMethod..decorator_timer..timeMethod_wrapper(self, *args, **keyArgs)
338 logInfo(
339 obj=self,
340 prefix=func.name + “Start”,
(…)
344 stacklevel=stacklevel,
345 )
346 try:
→ 347 res = func(self, *args, **keyArgs)
348 finally:
349 logInfo(
350 obj=self,
351 prefix=func.name + “End”,
(…)
355 stacklevel=stacklevel,
356 )

File /opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-4.1.0/Linux64/meas_astrom/gb4acbdd2b2+ef66f08a2a/python/lsst/meas/astrom/astrometry.py:173, in AstrometryTask.run(self, sourceCat, exposure)
134 “”“Load reference objects, match sources and optionally fit a WCS.
135
136 This is a thin layer around solve or loadAndMatch, depending on
(…)
170 (lsst.daf.base.PropertyList)
171 “””
172 if self.refObjLoader is None:
→ 173 raise RuntimeError(“Running matcher task with no refObjLoader set in init or setRefObjLoader”)
174 if self.config.forceKnownWcs:
175 res = self.loadAndMatch(exposure=exposure, sourceCat=sourceCat)

RuntimeError: Running matcher task with no refObjLoader set in init or setRefObjLoader

So it Is not working !

Would someone have a suggestion ?

sylvielsstfr · November 3, 2022, 4:29pm

I just had more information:

if I run : pipetask run -b /repo/main -i LATISS/raw/all,refcats,LATISS/calib -o u/ dagoret / photom/test0 -d "exposure=2022050400274 AND instrument='LATISS' AND detector=0" -p \$DRP_PIPE_DIR/pipelines/LATISS/DRP.yaml#isr,characterizeImage,calibrate --register-dataset-type s

it works

If I run

pipetask run -b /repo/main -i LATISS/raw/all,refcats,LATISS/calib -o u/ dagoret / photom/test0 -d "exposure=20220 630 00 045 AND instrument='LATISS' AND detector=0" -p \$DRP_PIPE_DIR/pipelines/LATISS/DRP.yaml#isr,characterizeImage,calibrate --register-dataset-type s

it fails.

ktl · November 3, 2022, 4:34pm

I believe we only have flat calibrations for SDSS{i,g,r} filters. empty~holo4 will need to have a calibration generated.

sylvielsstfr · November 3, 2022, 6:57pm

OK, thanks, I understand why !