Recreating the LSST Science pipeline tutorial (gen 2) only using Generation 3 command line tasks and the pipetasks

Data Management
#141

Simon,
I found another anomaly that you may already be aware of.
In tutorial 7 the parm tract is spelled as trct in the “Note” section.
Cheers KSK

Also, Simon, I’m working through Step 7 of the Tutorial. I’m not clear if meas_coadd is accurate. I see that my execution has created a directory actually named coad_meas . Can you advise if this is what we want here?
See cut/paste below:

import os
from lsst.daf.butler import Butler
collections = [f"u/{os.environ[‘USER’]}/objects", f"u/{os.environ[‘USER’]}/meas_coadd"]
butler = Butler(‘SMALL_HSC’, collections=collections)

#142

OK. I’ll take care of those typos

#143

Thank you Simon…

#144

I just closely watched the Delegate assembly recordings from 09/24.
Is it convenient to locate and download the notebook that Yusra A. used , entitled 05_Intro_to_Source_Detection…
Thanks

#145

This notebook is at tutorial-notebooks/05_Intro_to_Source_Detection.ipynb at main · rubin-dp0/tutorial-notebooks · GitHub ; the other tutorial notebooks are in the same repo.

#146

thanks KTL. I’ve got it functioning locally using Jupyter notebook.
We know it uses test data at:
repo = ‘s3://butler-us-central1-dp01’
collection = ‘2.2i/runs/DP0.1’
Is there a convenient zip file or tarball with this data? or is that notion unrealistic (due to size)

#147

…even though I probably can use some of the gen3 tutorial raw fits files. Do you agree?

#148

You will not be able to access the DESC data but you can use the notebooks as a guide to help you experiment with the test repositories you do have access to.

#149

okay Tim, thanks.
And,
I presume there is no convenient zip/tarballs/etc of the S3 files…

#150

Correct. It’s a huge dataset.

#151

So far I have been successful in readadapting the Jupyter notebooks from the 09/24 delegate assembly to the smaller tutorial gen3 datasets. I keep working through the notebooks some each day…
I’m examining the output datasets from the coadd processing and do not know what a .hsp filetype is…While I get some hits on open web search, can someone explain what this is for LSST …?
Happy Turkey day.

#152

It’s a FITS file – you can tell this by using the file command on a file. hsp stands for healsparse. Running fitsinfo on a file I have here reports:

No.    Name      Ver    Type      Cards   Dimensions   Format
  0  COV           1 PrimaryHDU       8   (12288,)   int64   
  1  SPARSE        1 CompImageHDU     11   (3145728,)   float64
#153

great… did not see the association with healpix and healpy. good deal.
thanks Tim

#154

I’m circling back to run the 02 Intermediate TAP Query exercise. I’m familiar with astropy.table and astropy.io for TAP and VOTable access, however, here I’m trying to perform the steps as shown in this LSST Tap Query exercise.
I’m attempting to accomplish these imports without success:
# Import the Rubin TAP service utilities
from lsst.rsp import get_tap_service, retrieve_query
In my Gen3 environment, can you suggest what I can install (with pip) to allow these directives to work ?
Thanks,
Fred

#155

I believe that’s pip install lsst-rsp. But note that I think these utilities were designed for execution in a Rubin Science Platform context and may not function in a generic Python, iPython, or Jupyter instance.

#156

ktl: your suggestion worked fine. As I expected, I am unable to establish the get_tap_service connection without an ACCESS_TOKEN, correlating to EXTERNAL_TAP_URL and EXTERNAL_INSTANCE_UTL.
Is there a READ-ONLY account that can be used to connect to the baseurl [“https://data.lsst.cloud/api/tap”] for persons not part of the formal delegate group?
thanks, Fred

#157

No there is not. If you want to practice with ADQL and TAP there are plenty of public services for that include those at CADC and IPAC. You can install the pyvo package (which you might already have).

#158

thanks Tim. This workbook has marginal value for me, as a professional Oracle DBA,moreso curious of any uniqueness in accessing data.lsst.cloud services. In past years I’ve worked with the archives for Hubble, Spitzer, PanStarrs, WISE, 2Mass, Sloan and other archives at Strasbourg; My first tries were with IPAC years ago.
Any chance the Tucson team will be in Salt Lake City in January?
thanks again.

#159

I’m seeking a clarification about Tracts. In the Gen3 tutorial and sqlite registry, I see only one tract, specifically #9813. I can understand having just one tract in the tutorial, as we only have raw images for 6 CCD/Detectors.
My question is , when we process all the raws from the LSST detectors [or even the HSC instrument, how many “tracts” will there be?
All the discussions from docs and YouTube LSST videos suggest a “tract” is a subpart of the whole; I’m asking how many tracts will be defined for the entire LSST/HSC focal plane.
Any docs (or pictures) would sure help. I watched Yusra AlSayyad video several times and was missing something…confused by the comments that one tract is 9 patches “wide” and 9 patches “high”…suggesting that a Tract was the entire focal plane… With THAT, I still got a LOT our of her presentation.
Fred , Dallas

#160

@yusra
I’m seeking a clarification about Tracts. In the Gen3 tutorial and sqlite registry, I see only one tract, specifically #9813. I can understand having just one tract in the tutorial, as we only have raw images for 6 CCD/Detectors.
My question is , when we process all the raws from the LSST detectors [or even the HSC instrument, how many “tracts” will there be?
All the discussions from docs and YouTube LSST videos suggest a “tract” is a subpart of the whole; I’m asking how many tracts will be defined for the entire LSST/HSC focal plane.
Any docs (or pictures) would sure help. I watched Yusra AlSayyad video several times and was missing something…confused by the comments that one tract is 9 patches “wide” and 9 patches “high”…suggesting that a Tract was the entire focal plane…while the slide only showed 8X8 “patch squares”! With THAT, I still got a LOT our of her presentation.
Fred , Dallas
image