Generation 3 run (LSST science pipeline tutorial) using v22 (w_2021_30)

---

Contents

---

• Useful links
• Setting up butler repository
  • Creating a butler repository
  • Registering a instrument to the butler repository
  • Importing the refcats into the butler repository
  • Ingesting the raws in the butler repository
  • Defining the visits
  • Writing the curated calibrations
• Processing the LSST pipeline tutorial data
  • ProcessCcd
  • Coaddtions
  • Multiband
    • Detection task
    • mergeDetection task
    • deblend task
    • Measure task

Useful links

---

• https://github.com/lsst/pipe_tasks/blob/master/pipelines/DRP.yaml [1]
• https://pipelines.lsst.io/v/weekly/modules/lsst.daf.butler/scripts/butler.html [2]
• https://github.com/lsst [3]
• https://community.lsst.org/t/generation-3-butler-tutorial/4965/8 [4]

---

Setting up butler repository

---

Creating a butler repository

• Note: the butler create command line task creates a directory and populates it with a butler.yaml file (base config) and a registry (sqlite3 file).

! butler create GEN3_run

Registering a instrument to the butler repository

! butler register-instrument GEN3_run/ lsst.obs.subaru.HyperSuprimeCam

Importing the refcats into the butler repository

• I used a hack to get the refcats into this butler repository

  • I used the butler convert command line task to convert the gen 2 repository (LSST Science pipeline tutorial) to a gen 3 repository
  • Then I used the butler import command link task to import into the new butler repository (GEN3_run)

• Question - Is there a way of ingestiing the refcats in a gen 3 format ??

! butler import GEN3_run/ ~/lsst_stack/DATA_gen3/ --export-file exports.yaml

Ingesting the raws in the butler repository

! butler ingest-raws GEN3_run/ DATA_gen3/HSC/raw/

Defining the visits

! butler define-visits GEN3_run/ HSC 

Writing the curated calibrations

butler write-curated-calibrations GEN3_run/ HSC

Processing the LSST pipeline tutorial data

• I'm using the latest version on the pipe_task file"DRP.yaml" thats found on github [1]

---

ProcessCcd

• Useful links

  • https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/calibrate.py
  • https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/characterizeImage.py
  • https://github.com/lsst/ip_isr/blob/master/python/lsst/ip/isr/isrTask.py
    
    

• These links are very helpful when defining the configs for each task for your data (path within your conda environment (starting point is within lsst_stact directory)
  "stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_tasks/21.0.0+44ca056b81/python/lsst/pipe/tasks/" and
  "stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/ip_isr/21.0.0+ee58a624b3/python/lsst/ip/isr/")
  
  

• Question - Are the configs correct?

pipetask run  -b GEN3_run/ --input HSC/raw/all --register-dataset-types -p "${PIPE_TASKS_DIR}/pipelines/DRP.yaml"#processCcd
    --instrument lsst.obs.subaru.HyperSuprimeCam --output-run demo_collection -c isr:doBias=False -c 
    isr:doBrighterFatter=False -c isr:doDark=False -c isr:doFlat=False -c isr:doDefect=False -c 
    calibrate:doPhotoCal=False -c calibrate:doAstrometry=False

• The butler repository now looks like this

• One thing i noticed is that this created more directories than in the gen 2 pipeline tutorial
  
• I'm guessing that I'm probably running more tasks using this method ??

Coaddtions

• Useful links

  • https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/makeCoaddTempExp.py
  • https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/assembleCoadd.py
  • https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/healSparseMapping.py
    
    

• These links are very helpful when defining the configs for each task for your data (path within your conda environment (starting point is within lsst_stact directory)
  "stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_tasks/21.0.0+44ca056b81/python/lsst/pipe/tasks/"
  
  

• The healSparsePropertyMaps was commented out. (Not relevant to the tutorial )
  
  

• Question - Are the configs correct?

Before you can run the coaddtions pipe task you have to run the make-discrete-skymap command line task

• https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/script/makeDiscreteSkyMap.py

! butler make-discrete-skymap test1/GEN3_run HSC --collections demo_collection

coaddtions pipetask

! pipetask run  -b GEN3_run/ --input demo_collection --input skymaps  --register-dataset-types -p DRP.yaml#coaddition
    --instrument lsst.obs.subaru.HyperSuprimeCam --output-run coadd -c makeWarp:doApplySkyCorr=False -c
    makeWarp:doApplyExternalSkyWcs=False -c makeWarp:doApplyExternalPhotoCalib=False -c assembleCoadd:doMaskBrightObjects=False

Potential Issue

• The "assembleCoadd:doMaskBrightObjects=False" wasn't needed when i ran this test before. I think what I've had done is a work around to get pipeline working. If "assembleCoadd:doMaskBrightObjects=True" I get this error
  
  

• The skymap command creates this directory within the butler repository
  
  

• The coaddtions pipetask directory sturture looks like this

• Very similar to the gen 2 repository which was converted into gen3. (assuming that the name convention of files has cahnge from gen2 to gen3 format)

Multiband

• Useful links

  • https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/multiBand.py
  • https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/mergeDetections.py
  • https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/deblendCoaddSourcesPipeline.py
  • https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/mergeMeasurements.py
  • https://github.com/lsst/meas_base/blob/master/python/lsst/meas/base/forcedPhotCcd.py
    
    

• These links are very helpful when defining the configs for each task for your data (path within your conda environment (starting point is within lsst_stact directory)
  "stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_tasks/21.0.0+44ca056b81/python/lsst/pipe/tasks/" and
  "stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/meas_base/21.0.0+087873070b/python/lsst/meas/base/")
  
  

Detection task

! pipetask run  -b test1/GEN3_run --input demo_collection --input skymaps --input coadd --register-dataset-types 
-p DRP.yaml:detection --instrument lsst.obs.subaru.HyperSuprimeCam --output-run dec

• This work without any changes to the code

mergeDetections task

pipetask run  -b test1/GEN3_run --input demo_collection --input skymaps --input coadd --input dec  
    --register-dataset-types -p DRP.yaml:mergeDetections --instrument lsst.obs.subaru.HyperSuprimeCam 
    --output-run coaddPhot_merge_12 --debug

deblend task

pipetask run  -b test1/GEN3_run --input demo_collection --input skymaps --input coadd --input dec 
    --input coaddPhot_merge_12   --register-dataset-types -p DRP.yaml:deblend --instrument lsst.obs.subaru.HyperSuprimeCam 
    --output-run coaddPhot_deblend

• Here is a look at the directory
  
  

Measure task

• I haven't got this working
  
  

• I am getting this error

• I assume it has something to with what I've done in thecoaddtions steps with the assemblecoadd (Coaddtions)