Incompatible dimension universe versions - database universe (r30+)

Support Data Products
#1

Hi,

Recently I developed a pipeline to work with modified injected stamps. The pipeline works correctly with releases prior to r30 (e.g. r29).

The problem appears when I try to create a local repository. During the transfer process, when the pipeline reaches the step:

> Registering DatasetType ‘skyMap’ in local repo before transfer.

I get the following error:

ValueError: Incompatible dimension universe versions - database universe: DimensionUniverse(8, daf_butler), dataset type universe: DimensionUniverse(7, daf_butler).

To check whether this was an issue with my code, I ran exactly the same notebook using the latest r29 release, and everything worked as expected. This makes me think that the problem is related to changes introduced in r30, possibly involving the Butler dimension universe or dataset type compatibility.

Has anyone encountered this issue before, or is there any migration step required when creating a local repository with r30+? Any suggestions would be greatly appreciated.

Thank you!

1 Like
#2

If you are creating the local repository using v30 from a DP1 butler this should work since I fixed a bug for exactly that issue in v30.0.10 (there was a DP1 tutorial notebook doing this). Can you give me more explicit instructions on how to reproduce?

1 Like
#3

Dear Tim,

Thank you for your reply, and I apologize for not providing enough information.

The error appears in the following environment:

  • Release: r30.0.10 (RSP Build 2991)
  • lsst_distrib: gdfb3db0272+4407f27008 (current, o_latest, v30_0_10, v30_0_10_rc4)
  • Python: 3.13.9

And yes, I’m using the LSSTComCam/DP1 collection.

The error originates from the following function:

def reg_transfer_skyMap(self, remote_collection: str = "LSSTComCam/DP1") -> None:
    """
    First: Registering SkyMap DatasetType and copying skyMap dataset(s)
    (preserving UUIDs). Ensure skyMap DatasetType exists locally so
    transfer-datasets can attach refs.

    Second: Transfer skyMap(s).
    """
    remote_butler = Butler(self.remote_repo, collections=remote_collection)
    remote_dt = [dt for dt in remote_butler.registry.queryDatasetTypes()
                 if dt.name == "skyMap"]

    if remote_dt:
        self.logger.info("Registering DatasetType 'skyMap' in local repo before transfer.")
        register_datasetTypes(self.local_repo, remote_dt, logger=self.logger)
    else:
        self.logger.warning("Remote repo does NOT contain datasetType 'skyMap'.")
        raise

    try:
        self.logger.info("Copying skyMap(s) from remote to local (preserving UUIDs)...")
        skymap_register_from_remote(
            self.remote_repo,
            self.local_repo,
            remote_collections=remote_collection,
            logger=self.logger,
        )
        self.logger.info("SkyMap copy finished.")
    except Exception:
        self.logger.exception("SkyMap copy failed.")
        raise

More specifically, the exception is raised at:

register_datasetTypes(self.local_repo, remote_dt, logger=self.logger)

where I get:

ValueError: Incompatible dimension universe versions - database universe: DimensionUniverse(8, daf_butler), dataset type universe: DimensionUniverse(7, daf_butler).

The same code works correctly with r29, so I suspect this is related to a change in the Butler dimension universe between r29 and r30. Does register_datasetTypes require any changes in r30, or is there now a different recommended way to register dataset types from a remote repository?

Thank you for your help.

1 Like
#4

You have your own API wrappers so it’s hard for me to say exactly what you are doing but if you are trying to register a dataset type from one butler in another butler and they do not have identical dimension universes you need to conform the source to the destination. Something like:

 butler.registry.registerDatasetType(dataset_type.conform_to(butler.dimensions))

If you use dest_butler.transfer_from() then this is all handled for you, including registering of dataset types and dimension records.

If you are running a pipeline task you can also use executor.use_local_butler(local_repo_path) to copy what you need from the graph into a local butler (see DP1 tutorial 105_6).

2 Likes
#5

Thanks, this is very helpful. I think I understand where the problem is now.

In my wrapper I create a new local Butler repository and then manually register the DatasetTypes before transferring the datasets. In particular, I currently have something like:

def register_datasetTypes(local_repo, datasettypes, ...):
    lbutler = Butler(local_repo, writeable=True)
    lreg = lbutler.registry

    for dt in datasettypes:
        lreg.registerDatasetType(dt)

The DatasetType objects in this case come directly from the remote Butler. Therefore, I think this is exactly where the dimension-universe mismatch occurs: I am trying to register the remote DatasetType directly into the newly created local Butler.

Based on your suggestion, I think the minimal fix should be:

for dt in datasettypes:
    local_dt = dt.conform_to(lbutler.dimensions)
    lreg.registerDatasetType(local_dt)

At the moment, after this manual registration I use butler transfer-datasets to transfer the actual datasets and preserve their UUIDs.

I also see your point about using dest_butler.transfer_from() instead. It seems that this could simplify my wrapper considerably, since I would not need to manually register the DatasetTypes and dimension records before the transfer.

I will first test the conform_to() change to confirm that this is the source of the r30 problem, and then I will look at replacing part of my manual transfer logic with transfer_from().

Thanks also for pointing me to executor.use_local_butler(). My current code is a more general wrapper for constructing a local Butler before running the pipeline, but I will check tutorial 105_6 as well.

1 Like
#6

Hi again!,

I tested the change you recommended, using dataset_type.conform_to(local_butler.dimensions) before registering the DatasetType , and it solved the problem. The dimension universe mismatch disappeared, so that was indeed the issue. Thanks for pointing me in the right direction!

I will also take a closer look at your recommendation of using dest_butler.transfer_from() . It seems like a much cleaner approach than my current implementation, so I’ll evaluate whether I can simplify my wrapper around it.

Thanks again for your help. I really appreciate it.

1 Like