Testing the Science Pipelines installation with a demo

Support
#1

Hello,

I’m trying to test my installation of pipeline but 6 tests are failed.I get following message:
===================== short test summary info ============================
(chain=‘demo_collection_exe’) SUBFAIL tests/test_butler.py::PiplinesCheckTestCase::testExecutionButler - lsst.daf.butler._exceptions.MissingCollectionError: No collection with name…
(chain=‘demo_collection_qbb’) SUBFAIL tests/test_butler.py::PiplinesCheckTestCase::testExecutionButler - lsst.daf.butler._exceptions.MissingCollectionError: No collection with name…
(chain=‘demo_collection_exe’) SUBFAIL tests/test_butler.py::PiplinesCheckTestCase::testExecutionExistence - lsst.daf.butler._exceptions.MissingCollectionError: No collection with name…
(chain=‘demo_collection_qbb’) SUBFAIL tests/test_butler.py::PiplinesCheckTestCase::testExecutionExistence - lsst.daf.butler._exceptions.MissingCollectionError: No collection with name…
(chain=‘demo_collection_exe’) SUBFAIL tests/test_butler.py::PiplinesCheckTestCase::testLogDataset - lsst.daf.butler._exceptions.MissingCollectionError: No collection with name…
(chain=‘demo_collection_qbb’) SUBFAIL tests/test_butler.py::PiplinesCheckTestCase::testLogDataset - lsst.daf.butler._exceptions.MissingCollectionError: No collection with name…

What is the source of my problem?Is it dependencies or problem with data base?

Thank you.

#2

There is usually a more detailed error message further up the test output. Is this pipelines_check that is failing? What version of the software are you testing? Is this a specific weekly or is it lsstsw using current main?

#3

With pipelines_check this sort of error usually means that the actual test script in the bin directory failed in a very bad way and you’d have to look at the output of the script to work out the real error message.

#4

It’s a recommended installation.

#5

I get missingcollection error. There is traceback: PiplinesCheckTestCase.testExecutionButler (chain=‘demo_collection_exe’) ____

self = <test_butler.PiplinesCheckTestCase testMethod=testExecutionButler>

def testExecutionButler(self):
    """Check outputs match in both runs."""

    for chain in (EXE_CHAIN, QBB_CHAIN):
        with self.subTest(chain=chain):
            # Check that we have identical datasets in both collections
            # except for the dataset.id
            main_datasets = self._get_datasets_from_chain(MAIN_CHAIN)

          datasets = self._get_datasets_from_chain(chain)

tests/test_butler.py:96:

tests/test_butler.py:51: in _get_datasets_from_chain
collections = list(self.butler.registry.queryCollections(chain, flattenChains=True))
…/…/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/daf_butler/gb745877958+0176e2a6b4/python/lsst/daf/butler/_registry_shim.py:300: in queryCollections
return self._registry.queryCollections(
…/…/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/daf_butler/gb745877958+0176e2a6b4/python/lsst/daf/butler/registry/sql_registry.py:1815: in queryCollections
for record in self._managers.collections.resolve_wildcard(
…/…/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/daf_butler/gb745877958+0176e2a6b4/python/lsst/daf/butler/registry/collections/_base.py:356: in resolve_wildcard
for record in self._find_many(wildcard.strings):

self = <lsst.daf.butler.registry.collections.synthIntKey.SynthIntKeyCollectionManager object at 0x7f588c983e90>
names = [‘demo_collection_exe’]

def _find_many(self, names: Iterable[str]) -> list[CollectionRecord[K]]:
    """Return multiple records given their names."""
    names = list(names)
    # To protect against potential races in cache updates.
    records: dict[str, CollectionRecord | None] = {}
    if self._caching_context.collection_records is not None:
        for name in names:
            records[name] = self._caching_context.collection_records.get_by_name(name)
        fetch_names = [name for name, record in records.items() if record is None]
    else:
        fetch_names = list(names)
        records = {name: None for name in fetch_names}
    if fetch_names:
        for record in self._fetch_by_name(fetch_names):
            records[record.name] = record
            self._addCachedRecord(record)
    missing_names = [name for name, record in records.items() if record is None]
    if len(missing_names) == 1:

      raise MissingCollectionError(f"No collection with name '{missing_names[0]}' found.")

E lsst.daf.butler._exceptions.MissingCollectionError: No collection with name ‘demo_collection_exe’ found.

…/…/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/daf_butler/gb745877958+0176e2a6b4/python/lsst/daf/butler/registry/collections/_base.py:302: MissingCollectionError
___ PiplinesCheckTestCase.testExecutionButler (chain=‘demo_collection_qbb’) ____

self = <test_butler.PiplinesCheckTestCase testMethod=testExecutionButler>

def testExecutionButler(self):
    """Check outputs match in both runs."""

    for chain in (EXE_CHAIN, QBB_CHAIN):
        with self.subTest(chain=chain):
            # Check that we have identical datasets in both collections
            # except for the dataset.id
            main_datasets = self._get_datasets_from_chain(MAIN_CHAIN)

          datasets = self._get_datasets_from_chain(chain)

tests/test_butler.py:96:

tests/test_butler.py:51: in _get_datasets_from_chain
collections = list(self.butler.registry.queryCollections(chain, flattenChains=True))
…/…/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/daf_butler/gb745877958+0176e2a6b4/python/lsst/daf/butler/_registry_shim.py:300: in queryCollections
return self._registry.queryCollections(
…/…/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/daf_butler/gb745877958+0176e2a6b4/python/lsst/daf/butler/registry/sql_registry.py:1815: in queryCollections
for record in self._managers.collections.resolve_wildcard(
…/…/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/daf_butler/gb745877958+0176e2a6b4/python/lsst/daf/butler/registry/collections/_base.py:356: in resolve_wildcard
for record in self._find_many(wildcard.strings):

self = <lsst.daf.butler.registry.collections.synthIntKey.SynthIntKeyCollectionManager object at 0x7f588c983e90>
names = [‘demo_collection_qbb’]

def _find_many(self, names: Iterable[str]) -> list[CollectionRecord[K]]:
    """Return multiple records given their names."""
    names = list(names)
    # To protect against potential races in cache updates.
    records: dict[str, CollectionRecord | None] = {}
    if self._caching_context.collection_records is not None:
        for name in names:
            records[name] = self._caching_context.collection_records.get_by_name(name)
        fetch_names = [name for name, record in records.items() if record is None]
    else:
        fetch_names = list(names)
        records = {name: None for name in fetch_names}
    if fetch_names:
        for record in self._fetch_by_name(fetch_names):
            records[record.name] = record
            self._addCachedRecord(record)
    missing_names = [name for name, record in records.items() if record is None]
    if len(missing_names) == 1:

      raise MissingCollectionError(f"No collection with name '{missing_names[0]}' found.")

E lsst.daf.butler._exceptions.MissingCollectionError: No collection with name ‘demo_collection_qbb’ found.

…/…/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/daf_butler/gb745877958+0176e2a6b4/python/lsst/daf/butler/registry/collections/_base.py:302: MissingCollectionError
__ PiplinesCheckTestCase.testExecutionExistence (chain=‘demo_collection_exe’) __

self = <test_butler.PiplinesCheckTestCase testMethod=testExecutionExistence>

def testExecutionExistence(self):
    """Check that the execution butler files are really there."""

    for chain in (EXE_CHAIN, QBB_CHAIN):
        with self.subTest(chain=chain):

          datasets = self._get_datasets_from_chain(chain)

tests/test_butler.py:112:

tests/test_butler.py:51: in _get_datasets_from_chain
collections = list(self.butler.registry.queryCollections(chain, flattenChains=True))
…/…/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/daf_butler/gb745877958+0176e2a6b4/python/lsst/daf/butler/_registry_shim.py:300: in queryCollections
return self._registry.queryCollections(
…/…/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/daf_butler/gb745877958+0176e2a6b4/python/lsst/daf/butler/registry/sql_registry.py:1815: in queryCollections
for record in self._managers.collections.resolve_wildcard(
…/…/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/daf_butler/gb745877958+0176e2a6b4/python/lsst/daf/butler/registry/collections/_base.py:356: in resolve_wildcard
for record in self._find_many(wildcard.strings):

self = <lsst.daf.butler.registry.collections.synthIntKey.SynthIntKeyCollectionManager object at 0x7f588c37cbd0>
names = [‘demo_collection_exe’]

def _find_many(self, names: Iterable[str]) -> list[CollectionRecord[K]]:
    """Return multiple records given their names."""
    names = list(names)
    # To protect against potential races in cache updates.
    records: dict[str, CollectionRecord | None] = {}
    if self._caching_context.collection_records is not None:
        for name in names:
            records[name] = self._caching_context.collection_records.get_by_name(name)
        fetch_names = [name for name, record in records.items() if record is None]
    else:
        fetch_names = list(names)
        records = {name: None for name in fetch_names}
    if fetch_names:
        for record in self._fetch_by_name(fetch_names):
            records[record.name] = record
            self._addCachedRecord(record)
    missing_names = [name for name, record in records.items() if record is None]
    if len(missing_names) == 1:

      raise MissingCollectionError(f"No collection with name '{missing_names[0]}' found.")

E lsst.daf.butler._exceptions.MissingCollectionError: No collection with name ‘demo_collection_exe’ found.

…/…/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/daf_butler/gb745877958+0176e2a6b4/python/lsst/daf/butler/registry/collections/_base.py:302: MissingCollectionError
__ PiplinesCheckTestCase.testExecutionExistence (chain=‘demo_collection_qbb’) __

self = <test_butler.PiplinesCheckTestCase testMethod=testExecutionExistence>

def testExecutionExistence(self):
    """Check that the execution butler files are really there."""

    for chain in (EXE_CHAIN, QBB_CHAIN):
        with self.subTest(chain=chain):

          datasets = self._get_datasets_from_chain(chain)

tests/test_butler.py:112:

tests/test_butler.py:51: in _get_datasets_from_chain
collections = list(self.butler.registry.queryCollections(chain, flattenChains=True))
…/…/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/daf_butler/gb745877958+0176e2a6b4/python/lsst/daf/butler/_registry_shim.py:300: in queryCollections
return self._registry.queryCollections(
…/…/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/daf_butler/gb745877958+0176e2a6b4/python/lsst/daf/butler/registry/sql_registry.py:1815: in queryCollections
for record in self._managers.collections.resolve_wildcard(
…/…/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/daf_butler/gb745877958+0176e2a6b4/python/lsst/daf/butler/registry/collections/_base.py:356: in resolve_wildcard
for record in self._find_many(wildcard.strings):

self = <lsst.daf.butler.registry.collections.synthIntKey.SynthIntKeyCollectionManager object at 0x7f588c37cbd0>
names = [‘demo_collection_qbb’]

def _find_many(self, names: Iterable[str]) -> list[CollectionRecord[K]]:
    """Return multiple records given their names."""
    names = list(names)
    # To protect against potential races in cache updates.
    records: dict[str, CollectionRecord | None] = {}
    if self._caching_context.collection_records is not None:
        for name in names:
            records[name] = self._caching_context.collection_records.get_by_name(name)
        fetch_names = [name for name, record in records.items() if record is None]
    else:
        fetch_names = list(names)
        records = {name: None for name in fetch_names}
    if fetch_names:
        for record in self._fetch_by_name(fetch_names):
            records[record.name] = record
            self._addCachedRecord(record)
    missing_names = [name for name, record in records.items() if record is None]
    if len(missing_names) == 1:

      raise MissingCollectionError(f"No collection with name '{missing_names[0]}' found.")

E lsst.daf.butler._exceptions.MissingCollectionError: No collection with name ‘demo_collection_qbb’ found.

…/…/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/daf_butler/gb745877958+0176e2a6b4/python/lsst/daf/butler/registry/collections/_base.py:302: MissingCollectionError
______ PiplinesCheckTestCase.testLogDataset (chain=‘demo_collection_exe’) ______

self = <test_butler.PiplinesCheckTestCase testMethod=testLogDataset>

def testLogDataset(self):
    """Ensure that the logs are captured in both modes."""

    log_datasets = self._get_datasets_from_chain(MAIN_CHAIN, datasetType="isr_log")
    self.assertEqual(len(log_datasets), 1)

    isr_log_ref = log_datasets.pop()

    # Get the logs from both main and exe/qbb collections.
    main_isr_log = self.butler.get("isr_log", dataId=isr_log_ref.dataId, collections=MAIN_CHAIN)
    for chain in (EXE_CHAIN, QBB_CHAIN):
        with self.subTest(chain=chain):

          isr_log = self.butler.get("isr_log", dataId=isr_log_ref.dataId, collections=chain)

tests/test_butler.py:128:

…/…/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/daf_butler/gb745877958+0176e2a6b4/python/lsst/daf/butler/direct_butler/_direct_butler.py:1146: in get
ref = self._findDatasetRef(
…/…/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/daf_butler/gb745877958+0176e2a6b4/python/lsst/daf/butler/direct_butler/_direct_butler.py:883: in _findDatasetRef
ref = self.find_dataset(
…/…/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/daf_butler/gb745877958+0176e2a6b4/python/lsst/daf/butler/direct_butler/_direct_butler.py:1256: in find_dataset
ref = self._registry.findDataset(
…/…/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/daf_butler/gb745877958+0176e2a6b4/python/lsst/daf/butler/registry/sql_registry.py:909: in findDataset
matched_collections = backend.resolve_collection_wildcard(collection_wildcard)
…/…/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/daf_butler/gb745877958+0176e2a6b4/python/lsst/daf/butler/registry/queries/_sql_query_backend.py:102: in resolve_collection_wildcard
return self._managers.collections.resolve_wildcard(
…/…/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/daf_butler/gb745877958+0176e2a6b4/python/lsst/daf/butler/registry/collections/_base.py:356: in resolve_wildcard
for record in self._find_many(wildcard.strings):

self = <lsst.daf.butler.registry.collections.synthIntKey.SynthIntKeyCollectionManager object at 0x7f588c4f0290>
names = [‘demo_collection_exe’]

def _find_many(self, names: Iterable[str]) -> list[CollectionRecord[K]]:
    """Return multiple records given their names."""
    names = list(names)
    # To protect against potential races in cache updates.
    records: dict[str, CollectionRecord | None] = {}
    if self._caching_context.collection_records is not None:
        for name in names:
            records[name] = self._caching_context.collection_records.get_by_name(name)
        fetch_names = [name for name, record in records.items() if record is None]
    else:
        fetch_names = list(names)
        records = {name: None for name in fetch_names}
    if fetch_names:
        for record in self._fetch_by_name(fetch_names):
            records[record.name] = record
            self._addCachedRecord(record)
    missing_names = [name for name, record in records.items() if record is None]
    if len(missing_names) == 1:

      raise MissingCollectionError(f"No collection with name '{missing_names[0]}' found.")

E lsst.daf.butler._exceptions.MissingCollectionError: No collection with name ‘demo_collection_exe’ found.

…/…/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/daf_butler/gb745877958+0176e2a6b4/python/lsst/daf/butler/registry/collections/_base.py:302: MissingCollectionError
______ PiplinesCheckTestCase.testLogDataset (chain=‘demo_collection_qbb’) ______

self = <test_butler.PiplinesCheckTestCase testMethod=testLogDataset>

def testLogDataset(self):
    """Ensure that the logs are captured in both modes."""

    log_datasets = self._get_datasets_from_chain(MAIN_CHAIN, datasetType="isr_log")
    self.assertEqual(len(log_datasets), 1)

    isr_log_ref = log_datasets.pop()

    # Get the logs from both main and exe/qbb collections.
    main_isr_log = self.butler.get("isr_log", dataId=isr_log_ref.dataId, collections=MAIN_CHAIN)
    for chain in (EXE_CHAIN, QBB_CHAIN):
        with self.subTest(chain=chain):

          isr_log = self.butler.get("isr_log", dataId=isr_log_ref.dataId, collections=chain)

tests/test_butler.py:128:

…/…/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/daf_butler/gb745877958+0176e2a6b4/python/lsst/daf/butler/direct_butler/_direct_butler.py:1146: in get
ref = self._findDatasetRef(
…/…/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/daf_butler/gb745877958+0176e2a6b4/python/lsst/daf/butler/direct_butler/_direct_butler.py:883: in _findDatasetRef
ref = self.find_dataset(
…/…/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/daf_butler/gb745877958+0176e2a6b4/python/lsst/daf/butler/direct_butler/_direct_butler.py:1256: in find_dataset
ref = self._registry.findDataset(
…/…/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/daf_butler/gb745877958+0176e2a6b4/python/lsst/daf/butler/registry/sql_registry.py:909: in findDataset
matched_collections = backend.resolve_collection_wildcard(collection_wildcard)
…/…/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/daf_butler/gb745877958+0176e2a6b4/python/lsst/daf/butler/registry/queries/_sql_query_backend.py:102: in resolve_collection_wildcard
return self._managers.collections.resolve_wildcard(
…/…/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/daf_butler/gb745877958+0176e2a6b4/python/lsst/daf/butler/registry/collections/_base.py:356: in resolve_wildcard
for record in self._find_many(wildcard.strings):

self = <lsst.daf.butler.registry.collections.synthIntKey.SynthIntKeyCollectionManager object at 0x7f588c4f0290>
names = [‘demo_collection_qbb’]

def _find_many(self, names: Iterable[str]) -> list[CollectionRecord[K]]:
    """Return multiple records given their names."""
    names = list(names)
    # To protect against potential races in cache updates.
    records: dict[str, CollectionRecord | None] = {}
    if self._caching_context.collection_records is not None:
        for name in names:
            records[name] = self._caching_context.collection_records.get_by_name(name)
        fetch_names = [name for name, record in records.items() if record is None]
    else:
        fetch_names = list(names)
        records = {name: None for name in fetch_names}
    if fetch_names:
        for record in self._fetch_by_name(fetch_names):
            records[record.name] = record
            self._addCachedRecord(record)
    missing_names = [name for name, record in records.items() if record is None]
    if len(missing_names) == 1:

      raise MissingCollectionError(f"No collection with name '{missing_names[0]}' found.")

E lsst.daf.butler._exceptions.MissingCollectionError: No collection with name ‘demo_collection_qbb’ found.

…/…/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/daf_butler/gb745877958+0176e2a6b4/python/lsst/daf/butler/registry/collections/_base.py:302: MissingCollectionError

#6

Since this is pipelines_check, the test failures are telling you that the test script did not run properly (in a catastrophic manner that meant that nothing populated the butler). The error is going to be in the output before the test ran. If you are running this from rebuild then it will be in _build.log in the pipelines_check folder. It seems you might be running manually so you will have to search up higher.

It looks like you are running v27. Is that correct?

#7

Yes it’s v27 .

#8

Seems like I don’t have in_build.log file.

#9

How are you running this? pipelines_check isn’t part of the formal release.

Did you git clone it? Did you check out the 27.0.0 tag for the package?

#10

I’m trying to run this.