Getting started tutorial part 1: setting up the Butler data repository

Hello,

I’m trying to run Getting started tutorial part 1: setting up the Butler data repository.

However, after I run:

from lsst.daf.butler import Butler
import os
repo_path = os.path.join(os.environ['RC2_SUBSET_DIR'], 'SMALL_HSC')
butler = Butler(repo_path)

I get error message:

Failed to instantiate Butler from config file:///home/venil360/lsstsw/rc2_subset/SMALL_HSC/butler.yaml.
Traceback (most recent call last):
  File "/home/venil360/lsstsw/miniconda/envs/lsst-scipipe-9.0.0/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
    self.dialect.do_execute(
  File "/home/venil360/lsstsw/miniconda/envs/lsst-scipipe-9.0.0/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 941, in do_execute
    cursor.execute(statement, parameters)
sqlite3.DatabaseError: file is not a database

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/venil360/lsstsw/stack/lsst-scipipe-9.0.0/Linux64/daf_butler/g09cb37a6c2+f7ddf18cb7/python/lsst/daf/butler/_butler.py", line 142, in __new__
    return Butler.from_config(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/venil360/lsstsw/stack/lsst-scipipe-9.0.0/Linux64/daf_butler/g09cb37a6c2+f7ddf18cb7/python/lsst/daf/butler/_butler.py", line 306, in from_config
    return DirectButler.create_from_config(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/venil360/lsstsw/stack/lsst-scipipe-9.0.0/Linux64/daf_butler/g09cb37a6c2+f7ddf18cb7/python/lsst/daf/butler/direct_butler/_direct_butler.py", line 195, in create_from_config
    registry = _RegistryFactory(config).from_config(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/venil360/lsstsw/stack/lsst-scipipe-9.0.0/Linux64/daf_butler/g09cb37a6c2+f7ddf18cb7/python/lsst/daf/butler/registry/_registry_factory.py", line 138, in from_config
    return self._registry_cls.fromConfig(self._config, butlerRoot, writeable, defaults)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/venil360/lsstsw/stack/lsst-scipipe-9.0.0/Linux64/daf_butler/g09cb37a6c2+f7ddf18cb7/python/lsst/daf/butler/registry/sql_registry.py", line 237, in fromConfig
    managers = managerTypes.loadRepo(database)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/venil360/lsstsw/stack/lsst-scipipe-9.0.0/Linux64/daf_butler/g09cb37a6c2+f7ddf18cb7/python/lsst/daf/butler/registry/managers.py", line 269, in loadRepo
    with database.declareStaticTables(create=False) as context:
  File "/home/venil360/lsstsw/miniconda/envs/lsst-scipipe-9.0.0/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/home/venil360/lsstsw/stack/lsst-scipipe-9.0.0/Linux64/daf_butler/g09cb37a6c2+f7ddf18cb7/python/lsst/daf/butler/registry/interfaces/_database.py", line 766, in declareStaticTables
    context = StaticTablesContext(self, connection)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/venil360/lsstsw/stack/lsst-scipipe-9.0.0/Linux64/daf_butler/g09cb37a6c2+f7ddf18cb7/python/lsst/daf/butler/registry/interfaces/_database.py", line 141, in __init__
    self._tableNames = frozenset(self._inspector.get_table_names(schema=self._db.namespace))
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/venil360/lsstsw/miniconda/envs/lsst-scipipe-9.0.0/lib/python3.11/site-packages/sqlalchemy/engine/reflection.py", line 405, in get_table_names
    return self.dialect.get_table_names(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 2, in get_table_names
  File "/home/venil360/lsstsw/miniconda/envs/lsst-scipipe-9.0.0/lib/python3.11/site-packages/sqlalchemy/engine/reflection.py", line 106, in cache
    ret = fn(self, con, *args, **kw)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/venil360/lsstsw/miniconda/envs/lsst-scipipe-9.0.0/lib/python3.11/site-packages/sqlalchemy/dialects/sqlite/base.py", line 2124, in get_table_names
    names = connection.exec_driver_sql(query).scalars().all()
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/venil360/lsstsw/miniconda/envs/lsst-scipipe-9.0.0/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1779, in exec_driver_sql
    ret = self._execute_context(
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/venil360/lsstsw/miniconda/envs/lsst-scipipe-9.0.0/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context
    return self._exec_single_context(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/venil360/lsstsw/miniconda/envs/lsst-scipipe-9.0.0/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1986, in _exec_single_context
    self._handle_dbapi_exception(
  File "/home/venil360/lsstsw/miniconda/envs/lsst-scipipe-9.0.0/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 2355, in _handle_dbapi_exception
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "/home/venil360/lsstsw/miniconda/envs/lsst-scipipe-9.0.0/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
    self.dialect.do_execute(
  File "/home/venil360/lsstsw/miniconda/envs/lsst-scipipe-9.0.0/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 941, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.DatabaseError: (sqlite3.DatabaseError) file is not a database
[SQL: SELECT name FROM sqlite_master WHERE type='table' AND name NOT LIKE 'sqlite~_%' ESCAPE '~' ORDER BY name]
(Background on this error at: https://sqlalche.me/e/20/4xp6)

What can be done about this?

Thank you for help.

Hi @VenIl, if you simply want to learn how to use Butler, I recommend trying these RSP tutorials instead: DP02_04a_Introduction_to_the_Butler and DP02_04b_Intermediate_Butler_Queries.

In case you want to try the original tutorial you are following, the error message suggests that the butler.yaml file may be referencing an invalid or non-existent database file. Please make sure if the downloading HSC data part has been done properly.

If you have another issues, please open a new post. I will mark this as a solution.

Best,
Yumi

This means you do not have git LFS enabled on your system.

See this previous post: