Task metadata now written as YAML

dm-dev
Tags: #<Tag:0x00007f7f71e28220>

(Jim Bosch) #1

I’ve just merged DM-15082, which changes all* obs packages (full list below) to use YAML to store Task metadata instead of Boost.Serialization.

Task metadata is currently rarely inspected, and it’s most useful (or at least most general) content is probably timing and resource use information, for Task methods decorated with the lsst.pipe.base.timeMethod decorator. At least at one point, reportTaskTiming.py from pipe_tasks could be used to put that information in a more human-readable form, though it is itself rarely used and may have suffered slight bit-rot.

In any case, because these datasets are infrequently used (and are never used as inputs to any Task), this ticket does not provide any backwards compatibility for reading existing metadata files written with Boost.Serialization. To read those, please just set up the 16.0 version of the appropriate obs package (any weekly prior to the upcoming w_2018_35 should also work).

Please also note that some existing metadata files written with Boost.Serialization are intrinsically corrupted and unreadable (DM-4927), including some written quite recently. That should no longer be a problem for new metadata files written with YAML, but this ticket does nothing to make the old ones recoverable.

Finally, note that while the low-level support for reading and writing via Boost.Serialization will still be around after this change, we plan to remove it soon as well (DM-14504). If I have missed any obs packages or datasets written with Boost.Serialization in this change, they will not be broken until that change is made. I’m intending for there to be at least one weekly release between these changes.

*Full list of obs packages modified (yes, including some that are probably dead already):

  • auxTel
  • cfht
  • comCam
  • ctio0m9
  • decam
  • lsstCam
  • lsstSim
  • monocam
  • noao_mosaic
  • sdss
  • subaru
  • wiyn

(as well as obs_base and obs_test, of course).