Butler and CameraMapper have been refactored so that CameraMapper can be used with repositories that are not on the local file system. The draft version of LDM-463 has been updated. Major changes are outlined below. As always, if this breaks something for you please reach out to me for help as needed.
General User Visible Changes
Butler should init mapper
Butler establishes repository relationships based on inputs, some of this affects mapper configuration. When passing a mapper into butler init, the mapper should be an importable string (e.g.
lsst.daf.persistence.CameraMapper) or a class object, NOT a class instance.
At some point this may change from "should" to "must".
Bypass functions do not participate in the Butler’s deferred-read mechanism.
**until the draft version of LDM-463 builds, you can replace
draft in the URL with
DM-8686 e.g. https://ldm-463.lsst.io/v/DM-8686/#bypass-functions. This is true for all the following links to LDM-463.
Init RepositoryArgs via dict
It can be very verbose to type lsst.daf.persistence.RepositoryArgs (or some abbreviation e.g. dafPersist.RepostioryArgs is common) for every item in inputs and outputs in Butler init. To help with this, a dict can be used to initialize a RepositoryArgs, and can be used as the value passed to inputs and outputs in the Butler initializer.
Changes that Affect Mapper Authors
ButlerLocation requires and uses a Storage Interface.
Mappers Should Provide Access to their Registry
Mappers should provide access to their registry via the method
getRegistry(self). This allows mappers in child repositories to use the registry from a parent.
CameraMapper input args
The Butler needs to be able to pass arguments to CameraMapper during init. The two that it currently needs to pass are the RepositoryCfg of the associated repository as
repositoryCfg, and the parent Registry as
parentRegistry. I've modified subclass mappers to take these args and pass them on to the CameraMapper superclass.
Butler Power-User Changes
Storage and Storage Interface Classes
We wanted to allow Butler (and CameraMapper) to work with files that are not in the local posix filesystem (e.g. in a Swift storage).
In the next story, DM-7468 (newer than DM-8686, implements a SwiftStorage interface class) StorageInterface is an abc, and the API is in the docstrings of that file
There are a few Posix-only methods in the PosixStorage class to support old-butler repositories:
- v1RepoExists(root): given a filesystem path see if a butler “version 1 repository” exists.
- getParentSymlinkPath(root): For Butler V1 Repositories only, if a _parent symlink exists, get the location pointed to by the symlink.
The Storage class is a helper for accessing storage interface functions, although I think maybe this is misguided. See documentation, I’m not sure if this is good syntatic sugar, or making it do too much.
New Butler init Signature Being Used More.
New Butler initialization api is being used in more places. This is
Butler(inputs=..., outputs=...)`. The arguments describe repositories. More information is in https://ldm-463.lsst.io/v/draft/#id1
Repositories do not Access their Parents Directly
In New Butler the CameraMapper does not access parents directly (via _parent
symlinks). In the cases where child repositories need to access elements of
parent repositories, those elements are passed to the children: Parent Registry, Parent mapper, and mapper init.
Relative and Absolute Paths for Repository Root and Parents
Using relative paths can become confusing when trying to identify a specific repository in Butler logic. Internally Butler converts all relative paths to absolute paths.
However, we want to be able to move Repositories around as a group (e.g. within a top level folder) on one computer and from computer to computer. So when it is possible to store a relative path from one repository to its parent, the relative path is kept in the “RepositoryCfg.
When copying a repository from one Storage type to another (e.g. from a developer to a Swift location) it’s possible the parent URIs will have to be adjusted. When we add Storage locations this should be considered, and it’s possible we should write a helper script to support this.