Problem constructing butler

Support
#1

I am having issues with the tutorial involving constructing the bulter. The problem doesn’t seem to be in my software but in the Butler repository in the tutorial. As this seems unlikely I imagine that I am doing something wrong but can’t determine what.

I am working through: “Getting started tutorial part 1: setting up the Butler data repository”

I am in python and have given the below commands:

from lsst.daf.butler import Butler
import os
repo_path = os.path.join(os.environ[‘RC2_SUBSET_DIR’], ‘SMALL_HSC’)
butler = Butler(repo_path)

I then get the following error message:

Failed to instantiate Butler from config file:///home/plah/programs/lsst_stack/rc2_subset/SMALL_HSC/butler.yaml.
Traceback (most recent call last):
File “/home/plah/programs/lsst_stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.1.0/lib/python3.10/site-packages/sqlalchemy/engine/base.py”, line 1900, in _execute_context
self.dialect.do_execute(
File “/home/plah/programs/lsst_stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.1.0/lib/python3.10/site-packages/sqlalchemy/engine/default.py”, line 736, in do_execute
cursor.execute(statement, parameters)
sqlite3.DatabaseError: file is not a database

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “”, line 1, in
File “/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.1.0/Linux64/daf_butler/g3ee7dd65e6+24e5aa5155/python/lsst/daf/butler/_butler.py”, line 281, in init
self.registry = Registry.fromConfig(
File “/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.1.0/Linux64/daf_butler/g3ee7dd65e6+24e5aa5155/python/lsst/daf/butler/registry/_registry.py”, line 230, in fromConfig
return registry_cls.fromConfig(config, butlerRoot, writeable, defaults)
File “/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.1.0/Linux64/daf_butler/g3ee7dd65e6+24e5aa5155/python/lsst/daf/butler/registries/sql.py”, line 195, in fromConfig
managers = managerTypes.loadRepo(database)
File “/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.1.0/Linux64/daf_butler/g3ee7dd65e6+24e5aa5155/python/lsst/daf/butler/registry/managers.py”, line 194, in loadRepo
with database.declareStaticTables(create=False) as context:
File “/home/plah/programs/lsst_stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.1.0/lib/python3.10/contextlib.py”, line 135, in enter
return next(self.gen)
File “/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.1.0/Linux64/daf_butler/g3ee7dd65e6+24e5aa5155/python/lsst/daf/butler/registry/interfaces/_database.py”, line 654, in declareStaticTables
context = StaticTablesContext(self)
File “/home/plah/programs/lsst_stack/stack/miniconda3-py38_4.9.2-4.1.0/Linux64/daf_butler/g3ee7dd65e6+24e5aa5155/python/lsst/daf/butler/registry/interfaces/_database.py”, line 105, in init
self._tableNames = frozenset(self._inspector.get_table_names(schema=self._db.namespace))
File “/home/plah/programs/lsst_stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.1.0/lib/python3.10/site-packages/sqlalchemy/engine/reflection.py”, line 266, in get_table_names
return self.dialect.get_table_names(
File “”, line 2, in get_table_names
File “/home/plah/programs/lsst_stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.1.0/lib/python3.10/site-packages/sqlalchemy/engine/reflection.py”, line 55, in cache
ret = fn(self, con, *args, **kw)
File “/home/plah/programs/lsst_stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.1.0/lib/python3.10/site-packages/sqlalchemy/dialects/sqlite/base.py”, line 1992, in get_table_names
rs = connection.exec_driver_sql(s)
File “/home/plah/programs/lsst_stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.1.0/lib/python3.10/site-packages/sqlalchemy/engine/base.py”, line 1760, in exec_driver_sql
return self._exec_driver_sql(
File “/home/plah/programs/lsst_stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.1.0/lib/python3.10/site-packages/sqlalchemy/engine/base.py”, line 1669, in _exec_driver_sql
ret = self._execute_context(
File “/home/plah/programs/lsst_stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.1.0/lib/python3.10/site-packages/sqlalchemy/engine/base.py”, line 1943, in _execute_context
self.handle_dbapi_exception(
File “/home/plah/programs/lsst_stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.1.0/lib/python3.10/site-packages/sqlalchemy/engine/base.py”, line 2124, in handle_dbapi_exception
util.raise(
File “/home/plah/programs/lsst_stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.1.0/lib/python3.10/site-packages/sqlalchemy/util/compat.py”, line 208, in raise
raise exception
File “/home/plah/programs/lsst_stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.1.0/lib/python3.10/site-packages/sqlalchemy/engine/base.py”, line 1900, in _execute_context
self.dialect.do_execute(
File “/home/plah/programs/lsst_stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-4.1.0/lib/python3.10/site-packages/sqlalchemy/engine/default.py”, line 736, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.DatabaseError: (sqlite3.DatabaseError) file is not a database
[SQL: SELECT name FROM sqlite_master WHERE type=‘table’ ORDER BY name]
(Background on this error at: Error Messages — SQLAlchemy 1.4 Documentation)

The error message seems to be saying that the tutorial data file is not a database which makes no sense as it would not have been put in the tutorial if it didn’t work. My research has not turned up anything useful. Does anyone have any ideas what is going wrong.

#2

It’s highly likely that you have not configured your git lfs properly. If you look at the gen3.sqlite3 file it likely is very small and just has a ref to a dataset in it rather than a proper database file.

#3

The entire contents of the:

rc2_subset/SMALL_HSC/gen3.sqlite3

is:

version ttps://git-lfs.github.com/spec/v1
oid sha256:063885e3c431808c5fe23d2e34b3489e819edc3046f9e06635b1885925296263
size 5738496

I removed the ‘h’ from ttp so you can see the link. From what you said this is too small and is not a proper database file. This is the file that comes with the tutorial. Do you know where I can get a gen3.sqlite3 file that will work?

#4

You likely need to execute git lfs install once, then do a git lfs checkout (I think) to replace that file with the proper contents.

#5

When I run:

git lfs checkout

I get the below message:

Not in a Git repository.

Is there another way to replace the file?

#6

Were you in the /home/plah/programs/lsst_stack/rc2_subset/ directory when you executed the git lfs checkout command?

#7

I was not so I ran it in that directory.

It then goes through what looks like each file in the directory and gives a message similar to the below:

Skipped checkout for “central_six_coadd_9813.qgraph”, content not local. Use fetch to download.

I assume it did not work. Do you have any other ideas?

#8

You’ll need to run git lfs fetch as it said.

#9

I ran:

git lfs fetch

and it spent a while downloading stuff. However when I tried making the butler in python it still failed. The gen3.sqlite3 is unchanged. From what you have said this file needs to much larger. Is there anywhere else that I can get a modified version of this file?

Getting started tutorial part 2 (and 4): qgraph problems
#10

It’s not just that file, it’s all the (LFS) files in rc2_subset.

Let’s make this very explicit. Assuming you started with a git clone of the rc2_subset repo without git lfs installed properly, here are the steps for recovery, once you have git lfs available (which it should be in any rubin-env conda environment):

$ git lfs install     # needs to be done once per user home directory
                      # but doesn't really hurt to run it again
$ cd rc2_subset       # must be in the git clone
$ git lfs fetch       # downloads the LFS objects
$ git lfs checkout    # replaces the "pointer" files with LFS objects

I’m sorry I neglected the fetch command in my original response.

3 Likes
Issue in running `pipetask` command
#11

That worked. Thanks for your help.