Best way, if any, to modify/undo a butler certify-calibrations call?

Support
#1

On a few occasions I’ve accidentally specified the wrong start/end date(s) when certifying a calibration. The certification commands that I use look like:

butler certify-calibrations \
$REPO DECam/calib/bias DECam/calib bias \
--begin-date 2012-01-01T00:00:00 --end-date 2020-12-31T23:59:59

I would like to be able to “edit” the certification start/end dates after having already run butler certify-calibrations one or more times. When I try to simply re-run the same above command with different start/end dates, I get an error:

lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback:
Traceback (most recent call last):
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.8.1/Linux64/daf_butler/g952c50c189+d18c45d440/python/lsst/daf/butler/cli/cmd/commands.py", line 421, in certify_calibrations
    script.certifyCalibrations(*args, **kwargs)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.8.1/Linux64/daf_butler/g952c50c189+d18c45d440/python/lsst/daf/butler/script/certifyCalibrations.py", line 72, in certifyCalibrations
    registry.certify(output_collection, refs, timespan)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.8.1/Linux64/daf_butler/g952c50c189+d18c45d440/python/lsst/daf/butler/core/utils.py", line 272, in inner
    return func(self, *args, **kwargs)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.8.1/Linux64/daf_butler/g952c50c189+d18c45d440/python/lsst/daf/butler/registries/sql.py", line 586, in certify
    storage.certify(collectionRecord, refsForType, timespan)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.8.1/Linux64/daf_butler/g952c50c189+d18c45d440/python/lsst/daf/butler/registry/datasets/byDimensions/_storage.py", line 243, in certify
    raise ConflictingDefinitionError(
lsst.daf.butler.registry._exceptions.ConflictingDefinitionError: 62 validity range conflicts certifying datasets of type bias into DECam/calib for range [2012-01-01 00:00:00.000000, 2021-12-31 23:59:59.000000).

I checked butler certify-calibrations --help but don’t see any available options along the lines of --clobber to override previously supplied start/end dates. Currently, the best workaround I’ve come up with is to make a new/different output collection, for instance the DECam/calib_alt input parameter in:

butler certify-calibrations \
$REPO DECam/calib/bias DECam/calib_alt bias \
--begin-date 2012-01-01T00:00:00 --end-date 2021-12-31T23:59:59

Is there a better/recommended way to redefine the certification start/end dates of a calibration without needing to create a new output collection with a different name? My question seems at least somewhat similar/related to this one from 2022 April. I am using LSST pipelines v23_0_2 on the Rubin Science Platform. Thanks very much.

1 Like
#2

I think your best bet is to use the Python Registry.decertify interface to undo the original certification that you want to change, and then try again with the certify command. We just don’t have a command-line interface for that yet.

1 Like
#3

Thanks! :slightly_smiling_face:

I’ll try out Registry.decertify and then mark this as the solution.

#4

I tried the following:

import lsst.daf.butler as dafButler                                                                                            
butler = dafButler.Butler('repo')

qda = butler.registry.queryDatasetAssociations
coll = "DECam/calib_alt"

biases = qda("bias", collections=coll)

butler.registry.decertify(coll, 'bias', next(biases).timespan)

But I end up with an error message:

  File "try_decertify_alt.py", line 9, in <module>
    butler.registry.decertify(coll, 'bias', next(biases).timespan)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.8.1/Linux64/daf_butler/g952c50c189+d18c45d440/python/lsst/daf/butler/core/utils.py", line 272, in inner
    return func(self, *args, **kwargs)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.8.1/Linux64/daf_butler/g952c50c189+d18c45d440/python/lsst/daf/butler/registries/sql.py", line 601, in decertify
    storage.decertify(collectionRecord, timespan, dataIds=standardizedDataIds)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.8.1/Linux64/daf_butler/g952c50c189+d18c45d440/python/lsst/daf/butler/registry/datasets/byDimensions/_storage.py", line 297, in decertify
    self._db.delete(self._calibs, ["id"], *rowsToDelete)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.8.1/Linux64/daf_butler/g952c50c189+d18c45d440/python/lsst/daf/butler/registry/interfaces/_database.py", line 1491, in delete
    self.assertTableWriteable(table, f"Cannot delete from read-only table {table}.")
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.8.1/Linux64/daf_butler/g952c50c189+d18c45d440/python/lsst/daf/butler/registry/interfaces/_database.py", line 601, in assertTableWriteable
    raise ReadOnlyDatabaseError(msg)
lsst.daf.butler.registry.interfaces._database.ReadOnlyDatabaseError: Cannot delete from read-only table dataset_calibs_00000004.

I was also a little surprised that the Timespan argument for Registry.decertify seemed to be required, even though my read of the documentation was that it’s optional. Without the Timespan argument included the error I get is:

TypeError: decertify() missing 1 required positional argument: 'timespan'

#5

I added writeable=True to my script’s Butler instantiation call, and now it appears to be running the decertify command/operation:

butler = dafButler.Butler('repo', writeable=True)

decertify seems to be taking a while (hasn’t finished yet as of this writing).

#6

It ended up giving an error about the database being locked (traceback appended at end of this post). The code I’m currently using is:

import lsst.daf.butler as dafButler                                                                                            
butler = dafButler.Butler('repo', writeable=True)

qda = butler.registry.queryDatasetAssociations
coll = "DECam/calib_alt"

biases = qda("bias", collections=coll)

butler.registry.decertify(coll, 'bias', next(biases).timespan)

My only guess would be that this might have something to do with the biases generator. Will continue trying to make this work…

Traceback (most recent call last):
  File "/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.8.1/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1015, in _commit_impl
    self.engine.dialect.do_commit(self.connection)
  File "/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.8.1/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 685, in do_commit
    dbapi_connection.commit()
sqlite3.OperationalError: database is locked

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "try_decertify_alt.py", line 9, in <module>
    butler.registry.decertify(coll, 'bias', next(biases).timespan)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.8.1/Linux64/daf_butler/g952c50c189+d18c45d440/python/lsst/daf/butler/core/utils.py", line 272, in inner
    return func(self, *args, **kwargs)
  File "/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.8.1/lib/python3.8/contextlib.py", line 120, in __exit__
    next(self.gen)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.8.1/Linux64/daf_butler/g952c50c189+d18c45d440/python/lsst/daf/butler/registries/sql.py", line 237, in transaction
    yield
  File "/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.8.1/lib/python3.8/contextlib.py", line 120, in __exit__
    next(self.gen)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-0.8.1/Linux64/daf_butler/g952c50c189+d18c45d440/python/lsst/daf/butler/registry/interfaces/_database.py", line 523, in transaction
    trans.commit()
  File "/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.8.1/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2378, in commit
    self._do_commit()
  File "/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.8.1/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2568, in _do_commit
    self._connection_commit_impl()
  File "/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.8.1/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2539, in _connection_commit_impl
    self.connection._commit_impl()
  File "/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.8.1/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1017, in _commit_impl
    self._handle_dbapi_exception(e, None, None, None, None)
  File "/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.8.1/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2043, in _handle_dbapi_exception
    util.raise_(
  File "/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.8.1/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
    raise exception
  File "/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.8.1/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1015, in _commit_impl
    self.engine.dialect.do_commit(self.connection)
  File "/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-0.8.1/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 685, in do_commit
    dbapi_connection.commit()
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked
(Background on this error at: https://sqlalche.me/e/14/e3q8)
#7

The following worked, having pre-written a pickle file with the desired Timespan object in it so as to avoid the database being locked:

import lsst.daf.butler as dafButler                                                                                            
butler = dafButler.Butler('repo', writeable=True)
import pickle

with open('timespan.pickle', 'rb') as f:
    timespan = pickle.load(f)

coll = "DECam/calib_alt"

butler.registry.decertify(coll, 'bias', timespan)
#8

It would also work to turn your query result into a list or set. This would terminate the database query, allowing your subsequent modification to go through.

1 Like
#9

Thanks, K-T! I just tried the list version of your suggestion and it worked:

import lsst.daf.butler as dafButler                                                                                            
butler = dafButler.Butler('repo', writeable=True)

qda = butler.registry.queryDatasetAssociations
coll = "DECam/calib_alt"

biases = list(qda("bias", collections=coll))

butler.registry.decertify(coll, 'bias', biases[0].timespan)
2 Likes