Stack release 14.0 - Status and discussion

Data Management DM Notifications
#1

Here is where we currently are in the release process. Current step in bold.

Summary

Release is out: LSST stack release version 14.0 now available

CernVM FS contributed binaries are in preparation.

Release-specific Precursor Steps

  1. Identify any pre-release blockers (“must-have features”) :tools:
  2. Wait for them to clear
  3. Note: the seed for this release pre-dates the pytest changes

Release Engineering Steps

  1. Eups publish rc1 candidate (based on w_2017_33)
  2. Git Tag v14.0-rc1
  3. Github release lsst_demo 14.0
  4. Branch 14.0 of newinstall.sh (lsst)
  5. Wait for first round of bugs to clear
  6. Repeat last 2 steps, -rcN candidates <-- final candidate is rc1]
  7. Confirm DM Externals are at stable tags
  8. Tag DM Auxilliary (non-lsst_distrib) repos
  9. Full OS testing (see https://ls.st/faq )
  10. Git Tag 14.0, rebuild, eups publish

Binary release steps

  1. Produce factory binaries
  2. Test factory binaries
  3. Gather contributed binaries

Documentation Steps

  1. Update Prereqs/Install
  2. Update Known Issues
  3. Gather Release notes
  4. Gather Metrics report
  5. Email announcement
#2

What’s the target release date?

#3

I’m some issues and questions from the documentation perspective:

  • @frossie and @josh, can we update the v14 branch of to https://github.com/lsst/lsst to incorporate the warning fixes for macOS that @timj put in place?

  • I noticed that weekly tags are no longer being made in https://github.com/lsst/lsst/tags. Is this on purpose? It would help the docs if newinstall.sh tagged for each weekly build.

  • Also, going forward, can we change the branch and tag format in https://github.com/lsst/lsst to match EUPS tags (v14_0 instead of v14.0, and w_2017_33 instead of w.2017.33). Was this tag format a Conda requirement? If so, it would help the docs if Git refs in https://github.com/lsst/lsst matched the EUPS tags.

  • Both @ksk and I have had issues getting binaries when we think we should be. @ksk did you make progress on understanding this? I haven’t had time to track it yet.

  • Can we tag https://github.com/lsst/lsst_dm_stack_demo with each weekly build? This would help the docs. Similarly, the existing tags in lsst_dm_stack_demo aren’t formatted to match EUPS tags.

#4

This would be much simpler if demo was recast as a proper EUPS package.

#5

Any reason https://github.com/lsst/lsst_dm_stack_demo isn’t an EUPS package? I don’t know the history of it.

The other, orthogonal approach, is to drop the demo step from our installation instructions. Are we ready to do this? Are we getting value from the demo’s verification that outweighs the support issues?

#6

Yes. I added the -S switch in calling newinstall.sh and it is now getting binaries. I think that working is a bug.

#7

DM-1357.

#8

Where is the Python 2 deprecation notice going?

#9

That should go in the release notes. I believe @swinbank is working on that (there isn’t a Git branch for that yet as far as I know).

#10

The term “release notes” could cover a variety of sins, but my intention is simply to produce some sort of glorified changelog covering the Science Pipelines code, rather than a statement of policy or procedures for covering future releases. That is, the equivalent of this, but probably in a little less detail. I was expecting anything which doesn’t need specialist Pipelines knowledge to come from SQuaRE.

#11

That’s fine. After you merge those release notes I can add an announcement to both the v14 release notes and a small mention on the homepage where we link to the release notes. Does that sound good?

#12

:+1:

#13

Is @mwv doing the characterization test report? I can issue the DMTR number for it.

#14

Is @mwv doing the characterization test report?

He produced some numbers that I put in the document. See the 14.0 branch of pipelines_lsst_io.

#15

Ok. DMTR-41 will be where this ends up.

#16

@KSK I’m done with the conversion to DMTR-41. Can you take a look before I upload it to Docushare? I assume you know this but we should include a note where we report a metric is significantly worse than the previous release’s value.

I’m not entirely convinced by the performance argument that we don’t expect any difference from a year ago. Firstly, the performance target has changed in the past year. Secondly, we have received reports of performance changes over the year.

If you need to edit the characterization report please edit the Latex in https://github.com/lsst-dm/DMTR-41

#17

Thanks for the conversion. I’m in discussions with @mwv to see if we can compose explanations for the changes in KPM numbers between v13 and the release candidate.

#18

I suppose it’s too late to ask to wait on DM-11905 (in review) and DM-9195 (should go into review tomorrow) to get into the build? This would get the new jointcal photometric model and a better photoCalib into 14.0, which would be a big win.

#19

I believe the answer is no. Critical hot fixes only. 14 release candidate is based off weekly 33 I think so is from quite a long time ago. You might be able to make the case for it if you could cherry pick these changes back to jointcal from that epoch but that would still trigger an entirely new round of operating system build validations and you’d have to make a really good argument. The release is essentially done pending the release notes.

#20

Got it, no worries. 33 is far enough back that this would be complicated to pick back in.