We’ll be doing our first big schema migration for the Gen3 data repositories at NCSA (those in /repo) next Thursday, June 3, starting at 7am PDT. That will involve cutting off database access to everybody who isn’t doing the migration work for about 5 hours. One of us will announce when we’re done here and on the #dm-middleware-support Slack channel.
We will be changing the type of the central dataset_id column from an autoincrement integer to 128-bit UUID, which (among other things) will help a lot with our plans for avoiding per-job database connections in batch processing. The migrated repos will be compatible with stack versions w_2021_16 and newer. QuantumGraphs created before the migration will not be usable after the migration. Export YAML files created before the migration will be usable after the migration.
New repos created with w_2021_22 or later will automatically use the UUID configuration, and we will provide migration scripts and instructions for maintainers of other major data repositories after we’ve learned from this experience. We are not currently planning to migrate the IDF DP0.1 repository, and have no immediate plans to deprecate the old autoincrement integer configuration (though I expect we will someday, and future functionality may require the UUID configuration).
As the Condor cluster used by BPS doesn’t require us to provide total job runtime estimates, we don’t have a good way to gracefully drain the queues in advance and prevent potentially-affected jobs from starting, so it’ll be up to users to either stop their work in advance themselves or allow their jobs to die when database access is cut off (I imagine most other Science Pipelines devs will agree with me that this is a good tradeoff).