DM Pipelines Bootcamp 2022

Save the date! We are happy to announce a 2022 DM Science Pipelines Boot camp June 28-30
EDIT: Full Program: DM Pipelines Bootcamp 2022 - Data Management - Confluence

What:

Content will be similar to (DM Boot camp 2019)., (DM Boot Camp Announcement), with new updated material.

Why:

The DM Science Pipelines team has a few new team members as of this summer. The talks and docs from the last boot camp have served as valuable reference material for the DM team, and it’s time for a refresh.

Where:

  • Lectures will be remotely streamed: Zoom
  • Hands-on tutorials will be at University of Washington and Princeton University, however other institutions are welcome to locally organize as well.

Who:

Tutorials and lectures will be lead by current DM team members. They are geared towards the new scientists who have joined the Science Pipelines Team. Attendance is open to everyone from all corners of Rubin, but note that the material will be aimed at the level of new DM team members.

When: All times PDT

  • Tuesday June 28: Talks 10-12:30
    • hands on cowork (Tues afternoon PDT or Wed morning EDT)
  • Wednesday June 29: Talks 10-12:30
    • hands on cowork (Wed afternoon PDT or Thurs morning EDT)
  • Thursday June 30: Talks 10-12:30

On the West Coast hands-on co-work will take place in the afternoon, on the East coast hands-on co-work will take place in the morning.

Topics:

  • Welcome to DM
  • System Architecture
  • Documentation: How to find it and how to write it
  • Where do I look stuff up?
  • How to write a metric
  • How to generate a plot with a Pipeline
  • Gen3 Butler (incl dataset management, collection management,)
  • How your code is tested (release cycle, CI, datasets that are readily available)
  • How to test your code (incl setting up containerized stack builds)
  • How to run a Pipeline (incl chaining off of existing collections)
  • Using the databases at the data-facility
  • How to write a PipelineTask (incl. warning of gen2-historical remnants in Tasks that are being deleted)

Program: DM Pipelines Bootcamp 2022 - Data Management - Confluence

12 Likes

Thank you for organizing this event! I saw the attendance is open to everyone. But does it require registration?
Thanks.

1 Like

Registration will not be required. Zoom link will be open.

2 Likes

Hello,
Could I learn how to generate a Master bias and a Master Flat and how to ingest it in the Butler ?
Thanks for your answer.

Sylvie

1 Like

There are instructions for calibration construction available here: Constructing calibrations — LSST Science Pipelines I’m happy to present an introduction to that during the bootcamp if there is interest and time.

1 Like

Many thanks, Yes please, I am interested in this topic right now.

There’s time during the Thursday morning 10am EDT pair coding session for a tutorial on constructing calibrations.

Hello,
I am very interested by that time Thursday morning 10 am EDT (which make 4 PM CET - Paris time if I am not wrong).
I will attend this coding session.
Thanks you and see you soon.

A little question about yesterday’s talk: Intro to PipelineTask and Pipelines, There are many config file in the folder obs_package. For example obs_subaru/config, in which files like bias.py, calibrate.py. What’s the difference between the config like in DRP.yaml or that in the obs_subaru/config?
And will the slides be available? some link named here may be useful.
Thank you!

The configs in the obs_packages get applied before the ones in the Pipeline yamls. In other words, the ones in the pipeline yamls can overwrite the ones in the obs_package configs.

If you have a config that should be applied every time a task is run (i.e. both AP and DRP flavored pipelines) with data for that camera, you should put it in the obs_package/config. If it should only be applied in a particular pipeline (say the HSC-RC2 reruns), then it should go in the Pipeline yaml.

2 Likes

I’ve added a link to the slides for the PipelineTask/Pipelines session to the program Confluence page in the original post.

1 Like