Upcoming config change to AP Pipelines

With the merging of DM-43416 later today, ApPipe.yaml and all related pipelines will use the new APDB configuration system developed by @andy_slac . The system will make it easier to support Cassandra databases (coming very soon!) and additional APDB-sensitive tasks in the pipeline. The new pipeline will first appear in Science Pipelines releases d_2024_05_01 and w_2024_18.

If you are using ApPipe, ApPipeWithFakes, ApVerify, etc., you will no longer be able to set up the database with make_apdb.py (which was previously deprecated). Instead, you must use the new apdb-cli utility to create a config file, then pass it to the pipeline through parameters:apdb_config:

apdb-cli create-sql sqlite:///apdb.db my_apdb_config.py
pipetask run -p ApPipe.yaml -c parameters:apdb_config=my_apdb_config.py

Like the diaPipe:apdb.db_url task config field that it replaces, apdb_config is mandatory, but this is the only pipeline parameter you have to set, now or in the future.

Note: in this example, my_apdb_config.py is created, not read, by create-sql. create-sql (and create-cassandra) take command-line keywords for extra APDB options like namespaces. Starting tomorrow, you can read the ap_pipe docs for more examples.

If you are running DiaPipelineTask in your own pipeline, you can continue using the old system, or migrate to the new one by setting diaPipe.doConfigureApdb=False and diaPipe.apdb_config_url.

1 Like