Changes to Command-Line Task Behavior

Data Management DM Notifications
#1

I’ve just merged two tickets (DM-12450 and DM-12531, implementing RFC-407 and RFC-409) that make some major changes to how CmdLineTasks and especially the highest-level “driver” tasks work.

The quick summary:

  • You should almost never have to use the --clobber-config, --clobber-versions, or --no-versions command-line options anymore.

  • The clobber* and doOverwrite configuration options have been removed from multibandDriver.py and coaddDriver.py, and the default behavior for both tasks is to always reprocess and write new outputs.

  • To skip steps for which the outputs already exist (the old default behavior), use the same output repository and pass --reuse-outputs-from all on the command line.

More precisely:

  • We now only compare and require consistency in configurations, catalog schemas, and software versions when those are present in the output data repository, not the input repository. It’s now expected that a new output repository will have different configurations (etc) from its input. The only times you’re expected to need --clobber-config, --clobber-versions, or --no-versions is when you’re using an output repository as a sort of scratch space for repeated throwaway runs with different configurations (etc).

  • The multibandDriver.py and coaddDriver.py tasks only skip steps when both the --reuse-outputs-from option is used and the outputs of those steps are present in the output repository. There is no longer a way to skip steps when outputs are present only in an input repository (this seems to have only ever happened by accident in the past).

  • The --reuse-outputs-from option accepts a single argument, which is the name of a subtask (the same name used in the configuration tree, e.g. measureCoaddSources or makeCoaddTempExp) or the special value all. All steps up to and including that step will be skipped when their outputs are present. Passing all is equivalent to passing the last subtask in the driver (i.e. forcedPhotCoadd for multibandDriver.py or detectCoaddSources for coaddDriver.py).

These changes were not included in the w_2017_45 weekly.

2 Likes