The number of visits is more than that of exposure while define-visits when using version 24.0.0

Support
#1

Hello, I am using a newer version 24.0.0, when I ingest fits, the number of visits is more than that of exposures, for example, after I have ingest 2 files and run the command:

butler define-visits

The visits is more than exposures, and the output is:

lsst.obs.base.defineVisits INFO: Defaulting to searching for raw exposures in collection WFST/raw/all
lsst.defineVisits INFO: Preprocessing data IDs.
lsst.defineVisits INFO: Registering visit_system 2: by-seq-start-end.
lsst.defineVisits INFO: Registering visit_system 0: one-to-one.
lsst.defineVisits INFO: Grouping 2 exposure(s) into visits.
lsst.defineVisits INFO: Computing regions and other metadata for 3 visit(s).

In the gen3.sqlite3, there are a visit id named 92 which shouldn’t appear:

instrument	id	physical_filter	name	day_obs	seq_num	exposure_time	target_name	observation_reason	science_program	azimuth	zenith_angle	region	timespan_begin	timespan_end
WFST	92	WFST-G	WFST00000020_first	20220427	0	30	9999	science	9999	0	90	cF1WkPk6A+q/ZUAg7kANmr/LX1Xvjp7iP3eEP0RWuOm/WKNeA+8Gmr/RGo+ifwXjP8snxQ/aa+m/R2YT2CeCmb/iNk1JbWvjP1bXE9isbem/05txgFUshL8V/3utkmzjP0W8VSkCbum/sNdTXLhNdz8V/3utkmzjP0VH88HUbOm/Dk72ias/lT/iNk1JbWvjPxCth0FWuem/5jnvj6G3lT/RGo+ifwXjPwwBVvc6BOq/k4m8DWexlT/LX1Xvjp7iP2+aIkvCT+q/gyT0wjmmlT8dqAO8OjPiP4cEpRINUeq/TIHmy6v3dj/HbkSOWjTiP+CkyiyoUOq/xByuUR9ZhL8DTu9ocDTiPz8TdjWETOq/elY3kpwOmr/zSRV6eDbiPw==	1651047067277000000	1651047097277000000
WFST	1	WFST-G	WFST00000010	20220427	0	30	9999	science	9999	0	90	cFO7983kBeq/1kvtYEOxlb+Tg0O9O5ziPwk/WksJu+m/tgXRHX23lb/AsV8hMwPjP1lA0ueQbum/pHZ5VIY/lb/75KedJ2njPx7iamm+b+m/dd+XfSNNd7+C0ILfTGrjPy79KBhpb+m/2ZfP758shD+C0ILfTGrjP9ggpDWWbem/UT2QDU2CmT/85KedJ2njP2gWEk4Juum/ntd8dRMHmj/AsV8hMwPjP50QMtDkBOq/gH3vmmQNmj+Tg0O9O5ziP+8VaMBiUOq/dS35Mt4Omj/qYYnK4DDiP+Qh3DRYUuq/tiRFN6pNhD9r35p3ADLiPyKD/PydUuq/qTIRbqAOd791qKlTFjLiPxDXJ4kkT+q/x5RmVVimlb/oiem7HjTiPw==	1651047042024000000	1651047072024000000
WFST	2	WFST-G	WFST00000020	20220427	0	60	9999	science	9999	0	90	cF1WkPk6A+q/ZUAg7kANmr/LX1Xvjp7iP3eEP0RWuOm/WKNeA+8Gmr/RGo+ifwXjP8snxQ/aa+m/R2YT2CeCmb/iNk1JbWvjP1bXE9isbem/05txgFUshL8V/3utkmzjP0W8VSkCbum/sNdTXLhNdz8V/3utkmzjP0VH88HUbOm/Dk72ias/lT/iNk1JbWvjPxCth0FWuem/5jnvj6G3lT/RGo+ifwXjPwwBVvc6BOq/k4m8DWexlT/LX1Xvjp7iP2+aIkvCT+q/gyT0wjmmlT8dqAO8OjPiP4cEpRINUeq/TIHmy6v3dj/HbkSOWjTiP+CkyiyoUOq/xByuUR9ZhL8DTu9ocDTiPz8TdjWETOq/elY3kpwOmr/zSRV6eDbiPw==	1651047042024000000	1651047097277000000

When I use the older version 23.0.1, this situation didn’t happen:

py.warnings WARNING: /home/yu/lsst_stack/23.0.1/stack/miniconda3-py38_4.9.2-0.8.1/Linux64/daf_butler/g6b22db343a+d18c45d440/python/lsst/daf/butler/registry/databases/sqlite.py:444: SAWarning: Class _Ensure will not make use of SQL compilation caching as it does not set the 'inherit_cache' attribute to ``True``.  This can have significant performance implications including some performance degradations in comparison to prior SQLAlchemy versions.  Set this attribute to True if this object can make use of the cache key generated by the superclass.  Alternatively, this attribute may be set to False which will disable this warning. (Background on this error at: https://sqlalche.me/e/14/cprf)
  return connection.execute(_Ensure(table), rows).rowcount

defineVisits INFO: Defaulting to searching for raw exposures in collection WFST/raw/all
defineVisits INFO: Preprocessing data IDs.
defineVisits INFO: Registering visit_system 0: one-to-one.
defineVisits INFO: Grouping 2 exposure(s) into visits.
defineVisits INFO: Computing regions and other metadata for 2 visit(s).

And if I run the command butler define-visits twice in version 24.0.0, the output is:

lsst.obs.base.defineVisits INFO: Defaulting to searching for raw exposures in collection WFST/raw/all
lsst.defineVisits INFO: Preprocessing data IDs.
lsst.defineVisits INFO: Registering visit_system 2: by-seq-start-end.
lsst.defineVisits INFO: Registering visit_system 0: one-to-one.
lsst.defineVisits INFO: Grouping 2 exposure(s) into visits.
lsst.defineVisits INFO: Computing regions and other metadata for 3 visit(s).
lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback:
Traceback (most recent call last):
  File "/home/yu/lsst_stack/24.0.0/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/obs_base/gbaa45dfa32+2718a75a08/python/lsst/obs/base/cli/cmd/commands.py", line 107, in define_visits
    script.defineVisits(*args, **kwargs)
  File "/home/yu/lsst_stack/24.0.0/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/obs_base/gbaa45dfa32+2718a75a08/python/lsst/obs/base/script/defineVisits.py", line 95, in defineVisits
    task.run(
  File "/home/yu/lsst_stack/24.0.0/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/obs_base/gbaa45dfa32+2718a75a08/python/lsst/obs/base/defineVisits.py", line 674, in run
    inserted_or_updated = self.butler.registry.syncDimensionData(
  File "/home/yu/lsst_stack/24.0.0/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/daf_butler/gdde7253329+1ed234098f/python/lsst/daf/butler/registries/sql.py", line 797, in syncDimensionData
    return storage.sync(record, update=update)
  File "/home/yu/lsst_stack/24.0.0/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/daf_butler/gdde7253329+1ed234098f/python/lsst/daf/butler/registry/dimensions/table.py", line 251, in sync
    _, inserted_or_updated = self._db.sync(
  File "/home/yu/lsst_stack/24.0.0/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/daf_butler/gdde7253329+1ed234098f/python/lsst/daf/butler/registry/interfaces/_database.py", line 1325, in sync
    raise DatabaseConflictError(
lsst.daf.butler.registry.interfaces._database.DatabaseConflictError: Conflict in sync for table visit on column(s) exposure_time: 60.0 != 30.0, timespan_begin: 1651047042024000000 != 1651047067277000000.

seems like there are something wrong with time, (it may read two different exposure time from one exposure?)
but the astrometadata -p work well:

(lsst-scipipe-4.0.5) [yu@localhost mock_data]$ astrometadata -p wfst translate ./WFST00000010.fits 
Analyzing ./WFST00000010.fits...
HDU 1 was not found in file ./WFST00000010.fits. Ignoring request.
instrument: WFST
telescope: WFST
datetime_begin: 2022-04-27T08:10:05.024
altaz_begin: <AltAz Coordinate (obstime=2022-04-27T08:10:05.024, location=(-339424.21484631, 4982414.0010383, 3960862.59046989) m, pressure=0.0 hPa, temperature=0.0 deg_C, relative_humidity=0.0, obswl=1.0 micron): (az, alt) in deg
    (0., 0.)>
boresight_airmass: 1.0
boresight_rotation_angle: 0.0 deg
boresight_rotation_coord: sky
dark_time: 30.0 s
datetime_end: 2022-04-27T08:10:35.024
detector_exposure_id: 14
detector_group: 0
detector_name: 4
detector_num: 4
detector_serial: 4
detector_unique_name: 4
exposure_group: 1
exposure_id: 1
exposure_time: 30.0 s
focus_z: 0.0 mm
group_counter_end: 0
group_counter_start: 0
has_simulated_content: False
location: (-339424.21484631, 4982414.0010383, 3960862.59046989) m
object: 9999
observation_counter: 0
observation_id: WFST00000010
observation_reason: science
observation_type: science
observing_day: 20220427
physical_filter: WFST-G
pressure: 621.6 hPa
relative_humidity: 20.0
science_program: 9999
temperature: 20.0 deg_C
tracking_radec: <SkyCoord (ICRS): (ra, dec) in deg
    (179.85, 35.99)>
visit_id: 1

I also test the HSC’s data, and the error didn’t happen, what may caused this?

Thank you!

#2

We allow visits to be defined in different ways for the same set of base exposures. This is primarily designed to handle the baseline two-exposures-per-visit mode of operations of the LSST, where we may sometimes want to process a pair of exposures (“snaps”) and sometimes want to process the exposures individually. Each of these ways of defining visits is a “visit system”.

You have two visit systems, so you can have up to twice as many visits as exposures.

The visit definition code bases visit ids on the id of the exposure. For some kinds of visits, it prefixes the id with the number 9 to distinguish them from other types. Unfortunately, this assumes that all visit (and exposure) ids have the same number of decimal digits, which may not be the case if you’re numbering them as 1, 2, 3, …

The best thing in your case is probably to use only visit system 0, in which each exposure is a separate visit. You can do this by configuring the DefineVisitsTask to have groupExposures = "one-to-one".

1 Like
#3

You will see that in v23 you are only defining one-to-one visits. In v24 there is the additional by-seq-start-end definition. The latter visit definition should only make a difference if your metadata translator defines group_counter_end different to group_counter_start (which will only happen if you take snaps).

It might be informative for you to dump the visit dimensions (butler query-dimension-records $REPO visit) and also the visit_definition records which map visit to exposure ID.

If you are developing a new camera for use with butler you probably should be using far newer versions of the software such as w_2023_16, even better if you try to keep up with weeklies.

1 Like