py.warnings WARNING: /home/eric/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/fgcm/g941b12d670/lib/python/fgcm/fgcmNumbaUtilities.py:35: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  def numba_test(value):

lsst.ctrl.mpexec.cmdLineFwk INFO: QuantumGraph contains 3 quanta for 3 tasks, graph ID: '1686347985.6231873-20624'
lsst.fgcmBuildStarsTable INFO: Running with 12 sourceTable_visit handles
lsst.fgcmBuildStarsTable INFO: Assembling visitCatalog from 12 visits
lsst.fgcmBuildStarsTable INFO: Retrieving metadata for visit 1206 (0/12)
lsst.fgcmBuildStarsTable INFO:   Found 406 good stars in visit 1206 (deltaAper = 0.031)
lsst.fgcmBuildStarsTable INFO:   Found 448 good stars in visit 1242 (deltaAper = 0.022)
lsst.fgcmBuildStarsTable INFO:   Found 349 good stars in visit 11696 (deltaAper = 0.021)
lsst.fgcmBuildStarsTable INFO:   Found 383 good stars in visit 11704 (deltaAper = 0.022)
lsst.fgcmBuildStarsTable INFO:   Found 367 good stars in visit 11710 (deltaAper = 0.026)
lsst.fgcmBuildStarsTable INFO:   Found 556 good stars in visit 17900 (deltaAper = 0.014)
lsst.fgcmBuildStarsTable INFO:   Found 569 good stars in visit 17904 (deltaAper = 0.009)
lsst.fgcmBuildStarsTable INFO:   Found 572 good stars in visit 17906 (deltaAper = 0.009)
lsst.fgcmBuildStarsTable INFO:   Found 580 good stars in visit 17950 (deltaAper = 0.012)
lsst.fgcmBuildStarsTable INFO:   Found 506 good stars in visit 23704 (deltaAper = 0.012)
lsst.fgcmBuildStarsTable INFO:   Found 442 good stars in visit 23706 (deltaAper = 0.014)
lsst.fgcmBuildStarsTable INFO:   Found 507 good stars in visit 23718 (deltaAper = 0.015)
lsst.fgcmBuildStarsTable INFO: Found all good star observations in 2.58 s
lsst.fgcmBuildStarsTable INFO: Using the following reference filters: HSC-G, HSC-R, HSC-I, HSC-Z
lsst.fgcmBuildStarsTable INFO: Using htm for matching.
lsst.fgcmBuildStarsTable INFO: Matching primary stars in 1 pixels
lsst.fgcmBuildStarsTable WARNING:   No matches found for pixel 381, i band
lsst.fgcmBuildStarsTable INFO:  Found 1062 primary stars in r band
lsst.fgcmBuildStarsTable INFO:  Found 420 new primary stars in g band
lsst.fgcmBuildStarsTable INFO:  Found 571 new primary stars in z band
lsst.fgcmBuildStarsTable INFO:   No observations in pixel 381, y band
lsst.fgcmBuildStarsTable INFO:   No observations in pixel 381, N921 band
lsst.fgcmBuildStarsTable INFO: Found 2053 unique objects in pixel 381 (0 of 0).
lsst.fgcmBuildStarsTable INFO: Found 2053 unique objects with >= 1 observations.
lsst.fgcmBuildStarsTable INFO: Matching stars to neighbors...
lsst.fgcmBuildStarsTable INFO: Matching positions to observations...
lsst.fgcmBuildStarsTable INFO: Collating observations
lsst.fgcmBuildStarsTable INFO: There are 2053 stars with at least 1 observations in each required band.
lsst.fgcmBuildStarsTable INFO: There are 0/6 pixels with high stellar density
lsst.fgcmBuildStarsTable INFO: Spooling out 5642 observation indices.
lsst.fgcmBuildStarsTable INFO: Loading reference objects from ps1_pv3_3pi_20170110 in region bounded by [150.17419359, 150.27427475], [2.18298352, 2.28298695] RA Dec
lsst.fgcmBuildStarsTable INFO: Loaded 547 reference objects
lsst.fgcmBuildStarsTable INFO: Loading reference objects from ps1_pv3_3pi_20170110 in region bounded by [149.56002936, 150.88843898], [1.56928567, 2.89668480] RA Dec
lsst.fgcmBuildStarsTable INFO: Loaded 101425 reference objects
py.warnings WARNING: /home/eric/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/meas_algorithms/gc655b1545f+78fc91a3d9/python/lsst/meas/algorithms/sourceSelector.py:343: RuntimeWarning: divide by zero encountered in true_divide
  signalToNoise = flux/err

lsst.fgcmBuildStarsTable.fgcmLoadReferenceCatalog.referenceSelector INFO: Selected 36529/101425 references
lsst.fgcmBuildStarsTable INFO: Found 1300 reference matches in pixel 381 (0 of 0).
lsst.ctrl.mpexec.singleQuantumExecutor INFO: Execution of task 'fgcmBuildStarsTable' on quantum {instrument: 'HSC'} took 8.085 seconds
lsst.ctrl.mpexec.mpGraphExecutor INFO: Executed 1 quanta successfully, 0 failed and 2 remain out of total 3 quanta.
lsst.fgcmFitCycle INFO: Logging to external logger.
lsst.fgcmFitCycle INFO: Will reset atmosphere parameters
lsst.fgcmFitCycle INFO: Found 112 CCDs and 6 bands (g r i z N921 y)
lsst.fgcmFitCycle INFO: AB offset (w/o throughput) estimated as 29.7737
lsst.fgcmFitCycle INFO: Loading info on 12 exposures.
lsst.fgcmFitCycle INFO: Exposures taken on 4 nights.
lsst.fgcmFitCycle INFO: No exposures in filter HSC-Y
lsst.fgcmFitCycle INFO: No exposures in filter NB0921
lsst.fgcmFitCycle INFO: Non-required band N921 has no exposures.
lsst.fgcmFitCycle INFO: Non-required band y has no exposures.
lsst.fgcmFitCycle INFO: Band g first used in wash index 0
lsst.fgcmFitCycle INFO: Band r first used in wash index 0
lsst.fgcmFitCycle INFO: Band i first used in wash index 0
lsst.fgcmFitCycle INFO: Band z first used in wash index 0
lsst.fgcmFitCycle INFO: Compiling indices for 1 mirror coating(s).
lsst.fgcmFitCycle INFO: PSF Candidate Flags found
lsst.fgcmFitCycle INFO: Delta-mag from local background found
lsst.fgcmFitCycle INFO: No observations in filter HSC-Y
lsst.fgcmFitCycle INFO: No observations in filter NB0921
lsst.fgcmFitCycle INFO: Observations matched in 0.0 seconds.
lsst.fgcmFitCycle INFO: Reserving 205 stars from the fit.
lsst.fgcmFitCycle INFO: Observations matched in 0.0 seconds.
lsst.fgcmFitCycle INFO: Done indexing in 0.0 seconds.
lsst.fgcmFitCycle INFO: 12.56698% stars have a reference match in the g band.
lsst.fgcmFitCycle INFO: 24.79299% stars have a reference match in the r band.
lsst.fgcmFitCycle INFO: 26.30297% stars have a reference match in the i band.
lsst.fgcmFitCycle INFO: 20.84754% stars have a reference match in the z band.
lsst.fgcmFitCycle INFO: Done matching reference stars in 0.0 seconds.
lsst.fgcmFitCycle INFO: Computed secZenith in 0.0 seconds.
lsst.fgcmFitCycle INFO: Flagging 1215 of 2053 stars with TOO_FEW_OBS
lsst.fgcmFitCycle INFO: Default: fit 40 parameters.
lsst.fgcmFitCycle INFO: Memory usage at FitCycle Prepared: 6722 MB current; 17118 MB peak.
lsst.fgcmFitCycle INFO: Fit initial cycle starting...
lsst.fgcmFitCycle INFO: Computed ccdDeltaBkg and expDeltaBkg in 0.01 seconds.
lsst.fgcmFitCycle INFO: Flagging 1215 of 2053 stars with TOO_FEW_OBS
lsst.fgcmFitCycle INFO: Map plotting not available.  Sorry!
lsst.fgcmFitCycle INFO: Found 838 good stars for chisq
lsst.fgcmFitCycle INFO: Chisq computation took 0.37 seconds.
lsst.fgcmFitCycle INFO: Selecting good stars from Bright Observations
lsst.fgcmFitCycle INFO: Running BrightObs on 4 cores
lsst.fgcmFitCycle INFO: Finished BrightObs in 0.11 seconds.
lsst.fgcmFitCycle INFO: Median SED slope in g band (0.00000000)
lsst.fgcmFitCycle INFO: Median SED slope in r band (0.00000000)
lsst.fgcmFitCycle INFO: Median SED slope in i band (0.00000000)
lsst.fgcmFitCycle INFO: Median SED slope in z band (0.00000000)
lsst.fgcmFitCycle INFO: Median SED slope in N921 band (0.00000000)
lsst.fgcmFitCycle INFO: Median SED slope in y band (0.00000000)
lsst.fgcmFitCycle INFO: Memory usage at FitCycle Post Bright-Obs: 6774 MB current; 17118 MB peak.
lsst.fgcmFitCycle INFO: Applying color cut: -0.250 < g-r < 2.250
lsst.fgcmFitCycle INFO: Flag 1 stars of 2053 with BAD_COLOR
lsst.fgcmFitCycle INFO: Applying color cut: -0.500 < r-i < 2.250
lsst.fgcmFitCycle INFO: Flag 0 stars of 2053 with BAD_COLOR
lsst.fgcmFitCycle INFO: Applying color cut: -0.500 < i-z < 1.000
lsst.fgcmFitCycle INFO: Flag 0 stars of 2053 with BAD_COLOR
lsst.fgcmFitCycle INFO: Applying color cut: 0.000 < g-i < 3.500
lsst.fgcmFitCycle INFO: Flag 0 stars of 2053 with BAD_COLOR
lsst.fgcmFitCycle INFO: Applying reference star color cut: 0.000 < g-i < 1.500
lsst.fgcmFitCycle INFO: Flag 106 stars with REFSTAR_BAD_COLOR (ref colors)
lsst.fgcmFitCycle INFO: Flag 0 stars with REFSTAR_BAD_COLOR (std colors)
lsst.fgcmFitCycle INFO: ExpGray for initial selection computed for 11 exposures.
lsst.fgcmFitCycle INFO: Computed ExpGray for initial selection in 0.00 seconds.
lsst.fgcmFitCycle INFO: Flagged 2 bad exposures with too few stars
lsst.fgcmFitCycle INFO:  TOO_FEW_STARS: g band: 1 of 3 exposures
lsst.fgcmFitCycle INFO:  TOO_FEW_STARS: r band: 0 of 4 exposures
lsst.fgcmFitCycle INFO:  TOO_FEW_STARS: i band: 1 of 1 exposures
lsst.ctrl.mpexec.singleQuantumExecutor ERROR: Execution of task 'fgcmFitCycle' on quantum {instrument: 'HSC'} failed. Exception RuntimeError: FATAL: All observations in i band have been cut with TOO_FEW_STARS
lsst.ctrl.mpexec.mpGraphExecutor ERROR: Task <TaskDef(lsst.fgcmcal.fgcmFitCycle.FgcmFitCycleTask, label=fgcmFitCycle) dataId={instrument: 'HSC'}> failed; processing will continue for remaining tasks.
Traceback (most recent call last):
  File "/home/eric/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/ctrl_mpexec/g4d64b21cde+4a4456e283/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 470, in _executeQuantaInProcess
    self.quantumExecutor.execute(qnode.taskDef, qnode.quantum, butler)
  File "/home/eric/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/ctrl_mpexec/g4d64b21cde+4a4456e283/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 140, in execute
    result = self._execute(taskDef, quantum, butler)
  File "/home/eric/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/ctrl_mpexec/g4d64b21cde+4a4456e283/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 218, in _execute
    self.runQuantum(runTask, quantum, taskDef, butler)
  File "/home/eric/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/ctrl_mpexec/g4d64b21cde+4a4456e283/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 582, in runQuantum
    task.runQuantum(butlerQC, inputRefs, outputRefs)
  File "/home/eric/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/fgcmcal/ge448b5faa6+bb1360196b/python/lsst/fgcmcal/fgcmFitCycle.py", line 946, in runQuantum
    fgcmDatasetDict, config = self._fgcmFitCycle(camera, handleDict, config=config)
  File "/home/eric/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/fgcmcal/ge448b5faa6+bb1360196b/python/lsst/fgcmcal/fgcmFitCycle.py", line 1168, in _fgcmFitCycle
    fgcmFitCycle.run()
  File "/home/eric/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/fgcm/g941b12d670/lib/python/fgcm/fgcmFitCycle.py", line 339, in run
    self.expSelector.selectGoodExposuresInitialSelection(self.fgcmGray)
  File "/home/eric/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/fgcm/g941b12d670/lib/python/fgcm/fgcmExposureSelector.py", line 106, in selectGoodExposuresInitialSelection
    logFlaggedExposuresPerBand(self.fgcmLog, self.fgcmPars,
  File "/home/eric/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/fgcm/g941b12d670/lib/python/fgcm/fgcmUtilities.py", line 676, in logFlaggedExposuresPerBand
    raise RuntimeError("FATAL: All observations in %s band have been cut with %s" % (band, flagName))
RuntimeError: FATAL: All observations in i band have been cut with TOO_FEW_STARS
lsst.ctrl.mpexec.mpGraphExecutor INFO: Executed 1 quanta successfully, 1 failed and 1 remain out of total 3 quanta.
lsst.ctrl.mpexec.mpGraphExecutor ERROR: Upstream job failed for task <TaskDef(lsst.fgcmcal.fgcmOutputProducts.FgcmOutputProductsTask, label=fgcmOutputProducts) dataId={instrument: 'HSC'}>, skipping this task.
lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback:
Traceback (most recent call last):
  File "/home/eric/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/ctrl_mpexec/g4d64b21cde+4a4456e283/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 131, in run
    script.run(qgraphObj=qgraph, **kwargs)
  File "/home/eric/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/ctrl_mpexec/g4d64b21cde+4a4456e283/python/lsst/ctrl/mpexec/cli/script/run.py", line 187, in run
    f.runPipeline(qgraphObj, taskFactory, args)
  File "/home/eric/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/ctrl_mpexec/g4d64b21cde+4a4456e283/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 749, in runPipeline
    executor.execute(graph, butler)
  File "/home/eric/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/ctrl_mpexec/g4d64b21cde+4a4456e283/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 401, in execute
    self._executeQuantaInProcess(graph, butler, self.report)
  File "/home/eric/lsst_stack/stack/miniconda3-py38_4.9.2-4.0.5/Linux64/ctrl_mpexec/g4d64b21cde+4a4456e283/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 526, in _executeQuantaInProcess
    raise MPGraphExecutorError("One or more tasks failed during execution.")
lsst.ctrl.mpexec.mpGraphExecutor.MPGraphExecutorError: One or more tasks failed during execution.