Remove or insert new certified calibrations from/into a collection

Hi, I am trying to add new certified calibrations to a collection, but am running into issues:

$ butler certify-calibrations ${REPO} master-flat/210321 master-flat-certified/210321 flat --begin-date 2019-01-01T00:00:00 --end-date 2023-01-01T23:59:59
lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback:
Traceback (most recent call last):
  File "/gscratch/astro/stevengs/lsst_stacks/stacks/w.2022.06/stack/miniconda3-py38_4.9.2-1.0.0/Linux64/daf_butler/ge8d8315da9+47328fee21/python/lsst/daf/butler/cli/cmd/commands.py", line 530, in certify_calibrations
    script.certifyCalibrations(*args, **kwargs)
  File "/gscratch/astro/stevengs/lsst_stacks/stacks/w.2022.06/stack/miniconda3-py38_4.9.2-1.0.0/Linux64/daf_butler/ge8d8315da9+47328fee21/python/lsst/daf/butler/script/certifyCalibrations.py", line 75, in certifyCalibrations
    registry.certify(output_collection, refs, timespan)
  File "/gscratch/astro/stevengs/lsst_stacks/stacks/w.2022.06/stack/miniconda3-py38_4.9.2-1.0.0/Linux64/daf_butler/ge8d8315da9+47328fee21/python/lsst/daf/butler/core/utils.py", line 56, in inner
    return func(self, *args, **kwargs)
  File "/gscratch/astro/stevengs/lsst_stacks/stacks/w.2022.06/stack/miniconda3-py38_4.9.2-1.0.0/Linux64/daf_butler/ge8d8315da9+47328fee21/python/lsst/daf/butler/registries/sql.py", line 621, in certify
    storage.certify(collectionRecord, refsForType, timespan)
  File "/gscratch/astro/stevengs/lsst_stacks/stacks/w.2022.06/stack/miniconda3-py38_4.9.2-1.0.0/Linux64/daf_butler/ge8d8315da9+47328fee21/python/lsst/daf/butler/registry/datasets/byDimensions/_storage.py", line 249, in certify
    raise ConflictingDefinitionError(
lsst.daf.butler.registry._exceptions.ConflictingDefinitionError: 186 validity range conflicts certifying datasets of type flat into master-flat-certified/210321 for range [2019-01-01 00:00:00.000000, 2023-01-01 23:59:59.000000).

The input collection master-flat/210321 contains new flats produced for DECam z-band, and I’d like to insert them into the collection master-flat-certified/210321, which already exists and contains certified flats for the same time range for DECam g/i/r-band. I would just like to add the z-band flats into this collection. (Here 210321 means 2021-03-21, and I don’t set specific --begi and --end dates because it seems easier to me to define nightly collections than try to put them all into the same collection with different ranges)

I can’t remove and re-create the collection with butler remove-collections because I’ve already processed data with the old g/i/r-band flats and have many collections that depend on master-flat-certified/210321. It seems like it would be a mess to delete all of these collections and then re-create them manually with butler collection-chain.

$ butler remove-collections ${REPO} master-flat-certified/210321

The following collections will be removed:
         Collection          Collection Type
---------------------------- ---------------
master-flat-certified/210321 CALIBRATION    
Continue? [y/N]: y
lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback:
Traceback (most recent call last):
  File "/gscratch/astro/stevengs/lsst_stacks/stacks/w.2022.06/env/lsst-w.2022.06/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1802, in _execute_context
    self.dialect.do_execute(
  File "/gscratch/astro/stevengs/lsst_stacks/stacks/w.2022.06/env/lsst-w.2022.06/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.ForeignKeyViolation: update or delete on table "collection" violates foreign key constraint "fkey_collection_chain_collection_collection_id_child" on table "collection_chain"
DETAIL:  Key (collection_id)=(311) is still referenced from table "collection_chain".


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/gscratch/astro/stevengs/lsst_stacks/stacks/w.2022.06/stack/miniconda3-py38_4.9.2-1.0.0/Linux64/daf_butler/ge8d8315da9+47328fee21/python/lsst/daf/butler/cli/cmd/_remove_collections.py", line 71, in remove_collections
    result.onConfirmation()
  File "/gscratch/astro/stevengs/lsst_stacks/stacks/w.2022.06/stack/miniconda3-py38_4.9.2-1.0.0/Linux64/daf_butler/ge8d8315da9+47328fee21/python/lsst/daf/butler/script/removeCollections.py", line 133, in doRemove
    butler.registry.removeCollection(name)
  File "/gscratch/astro/stevengs/lsst_stacks/stacks/w.2022.06/stack/miniconda3-py38_4.9.2-1.0.0/Linux64/daf_butler/ge8d8315da9+47328fee21/python/lsst/daf/butler/core/utils.py", line 56, in inner
    return func(self, *args, **kwargs)
  File "/gscratch/astro/stevengs/lsst_stacks/stacks/w.2022.06/stack/miniconda3-py38_4.9.2-1.0.0/Linux64/daf_butler/ge8d8315da9+47328fee21/python/lsst/daf/butler/registries/sql.py", line 345, in removeCollection
    self._managers.collections.remove(name)
  File "/gscratch/astro/stevengs/lsst_stacks/stacks/w.2022.06/stack/miniconda3-py38_4.9.2-1.0.0/Linux64/daf_butler/ge8d8315da9+47328fee21/python/lsst/daf/butler/registry/collections/_base.py", line 438, in remove
    self._db.delete(
  File "/gscratch/astro/stevengs/lsst_stacks/stacks/w.2022.06/stack/miniconda3-py38_4.9.2-1.0.0/Linux64/daf_butler/ge8d8315da9+47328fee21/python/lsst/daf/butler/registry/interfaces/_database.py", line 1582, in delete
    rowcount += connection.execute(newsql).rowcount
  File "/gscratch/astro/stevengs/lsst_stacks/stacks/w.2022.06/env/lsst-w.2022.06/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1289, in execute
    return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
  File "/gscratch/astro/stevengs/lsst_stacks/stacks/w.2022.06/env/lsst-w.2022.06/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 325, in _execute_on_connection
    return connection._execute_clauseelement(
  File "/gscratch/astro/stevengs/lsst_stacks/stacks/w.2022.06/env/lsst-w.2022.06/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1481, in _execute_clauseelement
    ret = self._execute_context(
  File "/gscratch/astro/stevengs/lsst_stacks/stacks/w.2022.06/env/lsst-w.2022.06/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1845, in _execute_context
    self._handle_dbapi_exception(
  File "/gscratch/astro/stevengs/lsst_stacks/stacks/w.2022.06/env/lsst-w.2022.06/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2026, in _handle_dbapi_exception
    util.raise_(
  File "/gscratch/astro/stevengs/lsst_stacks/stacks/w.2022.06/env/lsst-w.2022.06/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
    raise exception
  File "/gscratch/astro/stevengs/lsst_stacks/stacks/w.2022.06/env/lsst-w.2022.06/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1802, in _execute_context
    self.dialect.do_execute(
  File "/gscratch/astro/stevengs/lsst_stacks/stacks/w.2022.06/env/lsst-w.2022.06/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (psycopg2.errors.ForeignKeyViolation) update or delete on table "collection" violates foreign key constraint "fkey_collection_chain_collection_collection_id_child" on table "collection_chain"
DETAIL:  Key (collection_id)=(311) is still referenced from table "collection_chain".

[SQL: DELETE FROM public.collection WHERE public.collection.collection_id IN (%(collection_id_1_1)s)]
[parameters: {'collection_id_1_1': 311}]
(Background on this error at: https://sqlalche.me/e/14/gkpj)

I’ve also tried removing the datasets from this collection in a few ways with no success:

$ butler prune-datasets ${REPO} master-flat-certified/210321 --purge master-flat-certified/210321 --find-all
Error: Can not prune a collection that is not a RUN collection: master-flat-certified/210321
$ butler prune-datasets ${REPO} master-flat-certified/210321 --disassociate master-flat-certified/210321 --find-all
The following datasets will be removed:

type                 run                                  id                  band instrument detector         physical_filter         
---- ----------------------------------- ------------------------------------ ---- ---------- -------- --------------------------------
flat master-flat/210321/20220215T135047Z ae74ecb5-8535-437d-86f1-1366f9f3c36e    g      DECam        1 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z 34959a36-d68a-4a00-a74b-e1e9be778029    g      DECam        2 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z 7165b1b1-437f-4a79-8977-d35a5a906501    g      DECam        3 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z 0d23f981-7059-4758-b427-6e028477182d    g      DECam        4 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z a6fe5c14-9630-4d74-b0c7-91cb8a2bf5e0    g      DECam        5 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z ab6605d8-852f-4ccc-87c1-9bffdd58d5d1    g      DECam        6 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z 38f45d8e-1e1b-43d4-aeff-c5e053ab169e    g      DECam        7 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z ff3fddba-ad4c-4b5c-a0ae-6f057a25a74c    g      DECam        8 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z cc588596-f95a-4fdb-b5fc-5e515ae0a0d1    g      DECam        9 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z ec0bce83-c09b-4b4d-b3ac-1716ec5481cf    g      DECam       10 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z 2018d1ef-70e6-4f3a-a614-3f0aeda2ec80    g      DECam       11 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z cdd2ea3f-2e9b-415f-aea4-3829462ed1cf    g      DECam       12 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z 565c6927-9fd2-4b30-a4b5-b598222fc2bf    g      DECam       13 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z 8ae93bc8-4813-4042-926c-8e8719af9c1e    g      DECam       14 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z ebe7cc91-ea3a-43c0-ac0d-f337df81c6fe    g      DECam       15 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z fc66799d-0907-4b0c-a118-d76f9e2d812c    g      DECam       16 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z 7cfa1ecc-d87e-433d-8ee9-4cd2b08c3d01    g      DECam       17 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z 4fa7068d-a957-4c2b-ba48-d70aaa2e2499    g      DECam       18 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z c99b8fa9-a9ea-4a08-a91a-8c702cb047ef    g      DECam       19 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z f70731bd-7c00-4c41-ad35-14e28ecfe813    g      DECam       20 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z ce8d9686-2ca6-4a83-82e7-db6be3a48c96    g      DECam       21 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z 1528390b-c6b4-4290-ada1-0d5dcf61847a    g      DECam       22 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z 937b1f92-001c-45f1-a21f-af72d6a4e269    g      DECam       23 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z e3bd6c6f-f7ab-4f91-aa13-7f3d7a1cce44    g      DECam       24 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z d98bc020-9330-4647-adce-fe310abf058c    g      DECam       25 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z 9b717d88-e796-482c-9a4b-4a25cb1cb765    g      DECam       26 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z f5ad0285-a105-4ec2-87fd-4cfa83b16587    g      DECam       27 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z 97abe33f-37f5-4ec0-94cc-bacb61da7e55    g      DECam       28 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z cf7ac221-0822-4cc3-ac7a-dcb932ab5949    g      DECam       29 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z 58e9b508-92f3-4640-a277-9ba53a54261d    g      DECam       30 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z 5681c5d6-a404-4408-bb3b-36b14d36d2fc    g      DECam       31 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z 022983ee-42c0-4284-b147-a02ce93e3b83    g      DECam       32 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z 86b9466e-fbc6-4c8f-aa27-b299c7791b2c    g      DECam       33 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z b08dd23a-2c9b-499d-a5f8-682e2c80ba65    g      DECam       34 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z 684351ca-2cc1-4677-b30b-c61cd8c1335a    g      DECam       35 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z 79a7206a-4ab5-4262-be88-6f6eecce91c6    g      DECam       36 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z 1e14b631-0e28-4899-9b29-0894105848d9    g      DECam       37 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z 99b2ed2d-31af-4c54-9281-da33a196be74    g      DECam       38 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z b0a7bbd1-fed8-45b1-a82e-1369aaa4fefd    g      DECam       39 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z f91c355a-219b-42e7-b2b9-6b2f7eb68fa7    g      DECam       40 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z 3729a4f1-d681-4e1d-85cb-4005a5257554    g      DECam       41 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z b4ac4517-fd2d-4f47-8109-fd87c9448f1a    g      DECam       42 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z 33277fe9-aa5e-435d-a7bc-d0d0bb1cd7b9    g      DECam       43 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z db577e1e-ed74-46b3-a1bc-ddacbf32dfe6    g      DECam       44 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z bf6c6ce8-e9f4-46a9-ba94-ef32aa6efb51    g      DECam       45 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z 2c618f23-8201-4ad0-8af3-49c3e3712eba    g      DECam       46 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z 8d80c064-6c47-4ffb-baed-53c7714f748b    g      DECam       47 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z 6a6c009e-a479-4945-9c5e-0ea11152a539    g      DECam       48 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z 50f5dd2e-f58d-4095-8099-1f1cdac7ad19    g      DECam       49 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z 5dd98623-25b6-4a00-8ee8-05f24cb51caa    g      DECam       50 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z 32a743e7-152b-4e20-9485-25012a75ea3c    g      DECam       51 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z e9715f97-fc38-4961-9b33-478fad59f068    g      DECam       52 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z 289361b1-1145-48cd-85ec-fd09d7d60839    g      DECam       53 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z 3209dad1-2a7a-4c50-9264-7b06594e9ff2    g      DECam       54 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z 3362ed1e-165d-4b79-9a22-93817e439106    g      DECam       55 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z c389bfda-b41d-4a93-bd83-69cd8c5f65e3    g      DECam       56 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z c1e13fd3-597b-4fc8-934e-e256022218d8    g      DECam       57 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z 7e700f80-19db-4f8d-9e05-9911b98c6051    g      DECam       58 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z 65d2c3ac-599a-4a68-83ff-7c2514eba910    g      DECam       59 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z 366938e5-4c34-445b-bd45-1c68efaef084    g      DECam       60 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z 733c42e7-202a-42f0-8f32-ed06edf622b6    g      DECam       61 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z 43385e81-65ec-42cd-be56-b57e2028810c    g      DECam       62 g DECam SDSS c0001 4720.0 1520.0
flat master-flat/210321/20220215T135047Z 11cbc43e-293b-4989-a23e-41ff2c4c9e00    i      DECam        1 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z 206f8ece-6f75-4911-b4d8-641010fafa3d    i      DECam        2 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z d132a9c2-c44f-4243-8443-6a57531a06ea    i      DECam        3 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z 264d762f-1ebb-4936-8c29-d6396ba6bc55    i      DECam        4 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z 214df209-6dd6-4b1a-aa32-3c711d4150df    i      DECam        5 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z 2234d2fd-077a-4b9c-bf28-6aa2cdc05ad9    i      DECam        6 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z e8b624ba-2198-42a6-bf62-61cca55730c2    i      DECam        7 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z 1b5556c0-b0ba-4946-9103-f83b0f026af3    i      DECam        8 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z b38ec67b-ed55-46a8-b34a-228986af4225    i      DECam        9 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z c014c5f2-8e4a-4135-9bb1-aa341905f165    i      DECam       10 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z 7bd22694-00dc-4c6b-90d8-e3bf3381b87c    i      DECam       11 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z b0cf7082-5a35-41d4-ab64-b664c10eb912    i      DECam       12 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z 4223ea54-2564-48aa-858c-7515cee9cba1    i      DECam       13 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z bb747f65-fe41-40be-a060-c20c4af04b15    i      DECam       14 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z 2f730cc0-0bb0-4bb8-a4fa-81a553f85611    i      DECam       15 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z 18d7a22d-ce11-48e0-9b4a-42b6159d1657    i      DECam       16 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z a7f4397b-8471-46d7-a837-0fed25f393ec    i      DECam       17 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z d6f8161e-9bb1-46d9-9074-35e09a7b6764    i      DECam       18 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z c419b615-4168-4b16-a8d7-0eab81110b12    i      DECam       19 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z 9ab18fa4-3ba8-408d-a039-43ac549073c3    i      DECam       20 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z 793896fe-30e3-4e0f-a467-a82e141bd26b    i      DECam       21 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z 69aafdb4-55c7-44ed-a5c6-49f297ac0525    i      DECam       22 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z 92474741-f6c8-441d-a5cb-28e2b5f3f211    i      DECam       23 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z 1ff6e102-188b-4464-bf4f-5be84c794588    i      DECam       24 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z 55ab03c9-5e83-4fdf-900d-ae64635d3d55    i      DECam       25 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z dae1647f-6709-4af5-8004-df974eed4427    i      DECam       26 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z c1d97858-73fb-4f6a-be47-a12a69abe034    i      DECam       27 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z b3a4ea79-d9c8-4664-a963-a3cbaae67e82    i      DECam       28 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z c154d6b5-2ca0-41e2-91d3-739098612fb7    i      DECam       29 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z 66bc075b-4fc6-4247-857c-eedff5cf13d1    i      DECam       30 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z ef870675-4d87-433c-9d9f-5f82007d966c    i      DECam       31 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z a3309e1b-54af-478a-8745-1125d828f030    i      DECam       32 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z fd8f847d-3945-475b-99a0-3c0933e0eb3a    i      DECam       33 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z 00e52436-4231-4425-9234-0ed2e42b6b36    i      DECam       34 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z 4895f4f9-8ed4-49c1-9439-3e1258b4ab8e    i      DECam       35 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z a8773976-66f4-4e4b-94b4-d19aa4961430    i      DECam       36 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z c3d81b99-85ea-4c39-89c9-d1248c752ddd    i      DECam       37 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z 3eaf861c-8b67-4a44-a848-35b4cf3c84e0    i      DECam       38 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z 38db38f4-f6a1-449f-babc-ee97d7225091    i      DECam       39 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z 936146db-5cc8-47f2-a94d-57cf192ad78a    i      DECam       40 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z f68df310-3e7f-454f-a4a2-2e17fbd3c102    i      DECam       41 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z 015c8c71-46a3-4dd1-8606-a8e9749c9c23    i      DECam       42 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z cbb4b346-2ed2-4d7e-896c-68fab466d1f5    i      DECam       43 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z 629e7eec-392e-4c90-8392-aa5899361e27    i      DECam       44 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z 24efb1e7-006b-41db-ac5b-dc1b141e335d    i      DECam       45 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z 8b43095d-ca5e-4db8-b156-815466cb15f5    i      DECam       46 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z 8f769440-16a2-414d-b56c-fddc61c3b986    i      DECam       47 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z af02270f-79e8-4396-9986-b70a7e493680    i      DECam       48 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z cd155d81-774f-486b-a785-9a9254c9b3db    i      DECam       49 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z 99e1c2c3-d98e-48f4-9ea0-ac00cf001c6b    i      DECam       50 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z 410d2c27-8c24-4257-bea8-cccebee2cdda    i      DECam       51 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z 5f015f1a-f4a4-4821-b426-a910599d20b5    i      DECam       52 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z 9f06f636-cbe3-46c3-ae2d-bf678eaee79a    i      DECam       53 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z 099eb3a3-fc5c-46c3-8c7b-036937f88a1b    i      DECam       54 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z f702f500-5b6b-4927-838a-916898c97aed    i      DECam       55 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z 5fe4cdee-f9d2-4e36-8b27-a6fb9484213f    i      DECam       56 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z a797e142-aff7-423e-bfb0-e64ae189c3d9    i      DECam       57 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z 28e6a4a6-25ac-45d2-94d8-cfa2e470a270    i      DECam       58 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z b01e3cbe-c82b-4665-8f9e-0b8217eb5645    i      DECam       59 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z ee4f6087-39f9-46df-8e77-0c5d486c4802    i      DECam       60 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z 465ce4e4-4169-4562-a9d7-fa31f5b03181    i      DECam       61 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z fa2448ea-ecf4-49a2-be18-e5f8b93b2200    i      DECam       62 i DECam SDSS c0003 7835.0 1470.0
flat master-flat/210321/20220215T135047Z 201dc1eb-eb41-4eb0-a786-29c94a05c3b4    r      DECam        1 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z a21b52fb-f957-4d97-bb81-baaa919dda02    r      DECam        2 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z 3986cfd5-064d-40a7-b91c-aad0359b1113    r      DECam        3 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z e791d1d1-7259-40e1-bcdc-15789232ea00    r      DECam        4 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z d3eb2483-063a-4734-9c1a-a72cbbb9dd45    r      DECam        5 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z fc70ea6a-a07c-4569-96c6-c1ba7a231bff    r      DECam        6 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z 00e1479d-fb4f-4663-8f57-1d8b7143403e    r      DECam        7 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z a772e1d2-8804-480a-8e2d-cc28e5f7202a    r      DECam        8 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z f2b0fbfe-ef65-4e46-829e-2defec19ba14    r      DECam        9 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z 05874a49-ca45-4649-a5cf-42edaaed7cc6    r      DECam       10 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z cdbb8ca8-ab5c-4d12-9192-e9aee3527661    r      DECam       11 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z 01181264-d4af-44a4-b70a-9d15e92921c4    r      DECam       12 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z 30e89a5e-17a4-4f89-9f3a-e3ebff9f2e92    r      DECam       13 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z 94c1360d-7c14-4402-8cae-a69a22cdb9d2    r      DECam       14 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z c2292651-5aa3-44f2-afcc-9fc69f0aab98    r      DECam       15 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z 50c679ba-72d3-4a9d-8784-236bfec85f2d    r      DECam       16 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z 52ab3614-8def-425f-b700-3d9cdbbd6678    r      DECam       17 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z b9b7850e-e74c-43b2-9aff-99ebac5a7619    r      DECam       18 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z aeef99d6-37e2-4e7c-8d09-19ab40f1e4a6    r      DECam       19 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z 9ea0ae08-f978-4e36-bb1e-39e40e323ff4    r      DECam       20 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z 8b5416c5-d843-4098-b344-a796173c4b1d    r      DECam       21 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z c3795b5e-a25a-47e8-a05d-8d78fc6865bf    r      DECam       22 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z 7c4626bf-e424-4228-b158-2f782ba5f281    r      DECam       23 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z ebecd5fc-3167-44e8-898a-10bc794086e1    r      DECam       24 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z 319b022e-3e39-47ce-964f-763c06806c28    r      DECam       25 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z 2078cc48-cf4d-4cc8-9c4f-d0cd1c950f5d    r      DECam       26 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z 98164861-549c-41d0-94dc-fc3a2f0ca111    r      DECam       27 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z f7bb97de-6114-4637-ad73-54c58589c6e8    r      DECam       28 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z 4695774c-815f-4b8b-8754-2153dbc651d3    r      DECam       29 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z 5d4a6d40-f117-4dbe-ba31-cb285eda698b    r      DECam       30 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z cbdb2e47-6985-4673-81db-e55b8378cea1    r      DECam       31 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z 8147f761-8b9d-4ab6-829f-df63a3d20c6a    r      DECam       32 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z abea3ed9-763b-4c41-b0bf-3e7bc41f49aa    r      DECam       33 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z 02bb3791-e9de-4f5f-87e0-0a521d79f524    r      DECam       34 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z c364ce8b-7960-404c-af24-186eba5142b0    r      DECam       35 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z 00efbb33-41cd-4de8-8134-88ac1a594b7e    r      DECam       36 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z d1e75377-8ffc-401e-aae9-a3534f1466cc    r      DECam       37 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z 9c9c2488-d813-4ad7-9030-80f8da505916    r      DECam       38 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z e67e2e0c-7f96-4e2c-85e1-f4ef8db8c714    r      DECam       39 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z d0aa73af-c583-4e3f-a9fd-9cc8736ab107    r      DECam       40 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z 5af5d80b-a01b-4f0a-b43c-9a536d77d854    r      DECam       41 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z d53f17d3-dce3-418f-b987-c31d4f1c862f    r      DECam       42 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z 3c1e7a17-4569-4c60-8b45-26420aa8512d    r      DECam       43 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z 83404453-a4f7-4a0b-81a9-2b45fe03e135    r      DECam       44 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z a6573fe8-07ec-49ed-9d55-b9034374329c    r      DECam       45 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z 2f386b06-2cf3-4366-8ce1-5d55649ebbc0    r      DECam       46 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z 6894c8ff-2351-401e-adea-1c0b154857e7    r      DECam       47 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z 0a4e858e-d3ca-46e3-9c7a-1948bbc1f813    r      DECam       48 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z 47ff2fff-8159-44cf-a1e8-6c8f575452a4    r      DECam       49 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z 755b5d8f-4097-4f98-89c9-21bdcb86242b    r      DECam       50 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z 667d794b-d5e2-4a8a-baef-cf398611e008    r      DECam       51 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z c9e04148-f1c1-4c79-af98-2e7cdf62b8ef    r      DECam       52 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z a0ce01a8-ef03-4868-9bd4-24546579bb89    r      DECam       53 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z 53c0f1fe-6b92-4273-b1c1-ecf3e9e9cf1e    r      DECam       54 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z fb0ead66-98a0-467f-9b44-3ed928e444bd    r      DECam       55 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z 6ad5cae6-d487-48cd-a6f5-5054777835ab    r      DECam       56 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z dcd1c6e6-dbde-43ec-bac7-78355602e975    r      DECam       57 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z e83cfeee-87ef-418b-bc03-551cda30ca72    r      DECam       58 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z a27c1ee7-3850-4416-92f0-20a444beb0cb    r      DECam       59 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z 70a98e3b-604d-408b-80eb-2295b586e59e    r      DECam       60 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z 702fe023-9755-4e72-a7d1-a7fab6aa39da    r      DECam       61 r DECam SDSS c0002 6415.0 1480.0
flat master-flat/210321/20220215T135047Z c98a16ba-2e55-460f-99d4-b9ca19577d81    r      DECam       62 r DECam SDSS c0002 6415.0 1480.0

Continue? [y/N]: y
lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback:
Traceback (most recent call last):
  File "/gscratch/astro/stevengs/lsst_stacks/stacks/w.2022.06/stack/miniconda3-py38_4.9.2-1.0.0/Linux64/daf_butler/ge8d8315da9+47328fee21/python/lsst/daf/butler/cli/cmd/commands.py", line 414, in prune_datasets
    result.onConfirmation()
  File "/gscratch/astro/stevengs/lsst_stacks/stacks/w.2022.06/stack/miniconda3-py38_4.9.2-1.0.0/Linux64/daf_butler/ge8d8315da9+47328fee21/python/lsst/daf/butler/script/_pruneDatasets.py", line 222, in doPruneDatasets
    butler.pruneDatasets(
  File "/gscratch/astro/stevengs/lsst_stacks/stacks/w.2022.06/stack/miniconda3-py38_4.9.2-1.0.0/Linux64/daf_butler/ge8d8315da9+47328fee21/python/lsst/daf/butler/_butler.py", line 1701, in pruneDatasets
    raise TypeError(
TypeError: Cannot disassociate from collection 'master-flat-certified/210321' of non-TAGGED type CALIBRATION.

Any suggestions what I can do to get the new flats certified? Thanks.

I think my workaround right now will be to create new certified flat collections:

$ butler certify-calibrations ${REPO} master-flat/210321 master-flat-certified-new/210321 flat --begin-date 2019-01-01T00:00:00 --end-date 2023-01-01T23:59:59

and then replace the existence of the old collection with the new one in all CHAINED-type collections in my repo. It’s easier to do this than I thought (with Python at least – what I was really avoiding was messing around with finding which collections contain a child etc in Bash). I have a script that is doing this “collection renaming” I can share:

import argparse

def rename_collection(repo, old, new):
    from lsst.daf.butler import Butler
    from lsst.daf.butler import registry
    from lsst.daf.butler import script as butler_script

    butler = Butler(repo)
    collections = list(butler.registry.queryCollections(collectionTypes=[registry.CollectionType.CHAINED]))
    for parent in collections:
        children = list(butler.registry.getCollectionChain(parent))
        new_children = list(children)
        redefine = False
        for i, child in enumerate(children):
            if old == child:
                print("redefining chilren of", parent)
                new_children[i] = new
                redefine = True
        if redefine:
            redefined_children = butler_script.collectionChain(
                repo, 
                "redefine", 
                parent, 
                new_children,
                None,
                "False",
            )
            print("redefined", children, "to", redefined_children)

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("repo", type=str, help="The butler repository path")
    parser.add_argument("old_name", type=str, help="The old collection name to replace")
    parser.add_argument("new_name", type=str, help="The new collection name")

    args, _ = parser.parse_known_args()
    repo = args.repo
    old_name = args.old_name
    new_name = args.new_name

    rename_collection(repo, old_name, new_name)

if __name__ == "__main__":
    main()

I’ll update in case this doesn’t work out.

1 Like

Thank you for posting your workaround solution, @stevenstetzler !