Generation 3 run (LSST science pipeline tutorial) using v21_0_0

---

Some of the alterations I had to do in this run were due to the fact that I'm current using the v21_0_0 stack. I don't know how these would relate to v22_0_0 stack which is going to be released soon

Contents

---

• Useful links
• Setting up butler repository
  • Creating a butler repository
  • Registering a instrument to the butler repository
  • Importing the refcats into the butler repository
  • Ingesting the raws in the butler repository
  • Defining the visits
  • Writing the curated calibrations
• Processing the LSST pipeline tutorial data
  • ProcessCcd
  • Coaddtions
  • Multiband
    • Detection task
    • mergeDetection task
    • deblend task
    • Measure task

Useful links

---

• https://github.com/lsst/pipe_tasks/blob/master/pipelines/DRP.yaml [1]
• https://pipelines.lsst.io/v/weekly/modules/lsst.daf.butler/scripts/butler.html [2]
• https://github.com/lsst [3]
• https://community.lsst.org/t/generation-3-butler-tutorial/4965/8 [4]

---

Setting up butler repository

---

Creating a butler repository

• Note: the butler create command line task creates a directory and populates it with a butler.yaml file (base config) and a registry (sqlite3 file).

! butler create GEN3_run

Registering a instrument to the butler repository

! butler register-instrument GEN3_run/ lsst.obs.subaru.HyperSuprimeCam

Importing the refcats into the butler repository

• I used a hack to get the refcats into this butler repository

  • I used the butler convert command line task to convert the gen 2 repository (LSST Science pipeline tutorial) to a gen 3 repository
  • Then I used the butler import command link task to import into the new butler repository (GEN3_run)

• Question - Is there a way of ingestiing the refcats in a gen 3 format ??

! butler import GEN3_run/ ~/lsst_stack/DATA_gen3/ --export-file exports.yaml

Ingesting the raws in the butler repository

! butler ingest-raws GEN3_run/ DATA_gen3/HSC/raw/

Defining the visits

! butler define-visits GEN3_run/ HSC 

Writing the curated calibrations

butler write-curated-calibrations GEN3_run/ HSC

Processing the LSST pipeline tutorial data

• I'm using the latest version on the pipe_task file"DRP.yaml" thats found on github [1]

---

ProcessCcd

• Useful links

  • https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/calibrate.py
  • https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/characterizeImage.py
  • https://github.com/lsst/ip_isr/blob/master/python/lsst/ip/isr/isrTask.py
    
    

• These links are very helpful when defining the configs for each task for your data (path within your conda environment (starting point is within lsst_stact directory)
  "stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_tasks/21.0.0+44ca056b81/python/lsst/pipe/tasks/" and
  "stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ip_isr/21.0.0+ee58a624b3/python/lsst/ip/isr/")
  
  

• Question - Are the configs correct?

pipetask run  -b GEN3_run/ --input HSC/raw/all --register-dataset-types -p "${PIPE_TASKS_DIR}/pipelines/DRP.yaml":processCcd
    --instrument lsst.obs.subaru.HyperSuprimeCam --output-run demo_collection -c isr:doBias=False -c 
    isr:doBrighterFatter=False -c isr:doDark=False -c isr:doFlat=False -c isr:doDefect=False -c 
    calibrate:doPhotoCal=False -c calibrate:doAstrometry=False

• The butler repository now looks like this

• One thing i noticed is that this created more directories than in the gen 2 pipeline tutorial
  
• I'm guessing that I'm probably running more tasks using this method ??

Coaddtions

• Useful links

  • https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/makeCoaddTempExp.py
  • https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/assembleCoadd.py
  • https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/healSparseMapping.py
    
    

• These links are very helpful when defining the configs for each task for your data (path within your conda environment (starting point is within lsst_stact directory)
  "stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_tasks/21.0.0+44ca056b81/python/lsst/pipe/tasks/"
  
  

• The healSparsePropertyMaps was commented out. As a EUPS couldn't recognise new files added into the python pipetask directory. (These file weren't already in the stack unlike the other files I've edited)
  
  

• I'm assuming that the "healSparsePropertyMaps" deals with the images that have 0 good pixels in "directWarp" and "psfMatchedWarp"

• The solution I used to overcome this error was to look at my fully converted gen 2 pipeline and select the patches and visit that where created there. (This fix the error but I'm guessing using the healSparsePropertyMaps is a more efficient way of doing what I've done)

• Question - Are the configs correct?

Before you can run the coaddtions pipe task you have to run the make-discrete-skymap command line task

Potential Issue

• https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/script/makeDiscreteSkyMap.py

• line 78 was giving this error

• I altered this file to get it to work and i realised two things you can't use getdirect and the metadata and wcs are in the wrong order

• I also altered line 83-85

• I commented out the "if config.doAppend:" and put the "result.skyMap.register(skymap_id, butler.registry)" in line. Similarly to how the latest version on github is

! butler make-discrete-skymap test1/GEN3_run HSC --collections demo_collection

coaddtions pipetask

! pipetask run  -b GEN3_run/ --input demo_collection --input skymaps  --register-dataset-types -p DRP.yaml:coaddition
    --instrument lsst.obs.subaru.HyperSuprimeCam --output-run coadd -c makeWarp:doApplySkyCorr=False -c
    makeWarp:doApplyExternalSkyWcs=False -c makeWarp:doApplyExternalPhotoCalib=False -d "(patch=0 AND visit=903988) OR 
    (patch=1 AND visit IN (903986,903988,903990,904014,903334,903336,903338,903344,903346)) OR 
    (patch=2 AND visit IN (903986,903988,903990,904014,903334,903336,903344,903346)) OR 
    (patch=3 AND visit IN (903986,903988,903334)) OR 
    (patch=4 AND visit IN (903986,903988,903990,904010,904014,903334,903336,903338,903342,903344,903346)) OR
    (patch=5 AND visit IN (903986,903988,903990,904010,904014,903334,903336,903338,903342,903344,903346)) OR
    (patch=7 AND visit IN (904010,904014,903342,903344,903346)) OR
    (patch=8 AND visit IN (904010,904014,903342,903344,903346)) "

• The skymap command creates this directory within the butler repository
  
  

• The coaddtions pipetask directory sturture looks like this

• Very similar to the gen 2 repository which was converted into gen3. (assuming that the name convention of files has cahnge from gen2 to gen3 format)

Multiband

• Useful links

  • https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/multiBand.py
  • https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/mergeDetections.py
  • https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/deblendCoaddSourcesPipeline.py
  • https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/mergeMeasurements.py
  • https://github.com/lsst/meas_base/blob/master/python/lsst/meas/base/forcedPhotCcd.py
    
    

• These links are very helpful when defining the configs for each task for your data (path within your conda environment (starting point is within lsst_stact directory)
  "stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_tasks/21.0.0+44ca056b81/python/lsst/pipe/tasks/" and
  "stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/meas_base/21.0.0+087873070b/python/lsst/meas/base/")
  
  

Detection task

! pipetask run  -b test1/GEN3_run --input demo_collection --input skymaps --input coadd --register-dataset-types 
-p DRP.yaml:detection --instrument lsst.obs.subaru.HyperSuprimeCam --output-run dec

• This work without any changes to the code

mergeDetections task

• I had edit 2 files to get this command to work
  
  

  • exposureidinfo.py in the lsst.obs.base
  • multiband.py in the lsst.pipe.tasks
    
    

• with the exposureidinfo.py I updated that file to the latest version that is found on the github (https://github.com/lsst/obs_base/blob/master/python/lsst/obs/base/exposureIdInfo.py)
  
  

• Then I imported this in the mulitband.py (from lsst.obs.base import ExposureIdInfo)
  
  

• The issue i was initally getting was specficly in the "runQuantum"
  
  

• I copyed and pasted the latest runQuantum from the github (https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/mergeMeasurements.py)
  
  

• After these changes the command worked

pipetask run  -b test1/GEN3_run --input demo_collection --input skymaps --input coadd --input dec  
    --register-dataset-types -p DRP.yaml:mergeDetections --instrument lsst.obs.subaru.HyperSuprimeCam 
    --output-run coaddPhot_merge_12 --debug

deblend task

• Similarly to the task above i had to Import the exposureidinfo and change runQuantum (to the latest version that found on the github
  https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/deblendCoaddSourcesPipeline.py)
  (from lsst.obs.base import ExposureIdInfo)
  
  

• In addtion to this i had to edit line 205 as outputs.templateCatalogs

• As you can see from the image that the templateCatalogs is a tuple of (None, dict)
  
  

• To solve this error i simply choosed the second term in the tuple which was the dict (Catalog by adding "[1]" before "[band]"on line 205 )
  
  

• After this was change the command was able to be run
  
  

pipetask run  -b test1/GEN3_run --input demo_collection --input skymaps --input coadd --input dec 
    --input coaddPhot_merge_12   --register-dataset-types -p DRP.yaml:deblend --instrument lsst.obs.subaru.HyperSuprimeCam 
    --output-run coaddPhot_deblend_2

• However in the output messages there were alot of errors
  
  

• Here is a example:
  
  
  
  

• Question - is this error message expected ???

• Here is a look at the directory
  
  

Measure task

• I haven't got this working
  
  

• I am getting this error