Problem running calibTask.py example script from $PIPE_TASKS_DIR/examples

Support
#1

hi all- as part of my writing documentation for various DM Tasks, I’m working through some of the examples in the Tasks (as listed on the doxygen pages), and though some work straight off, others are not doing so, and I’m posting one of the first of those here in case it’s a real issue. The calibTask.py example script is supposed to exercise both CalibTask and CharacterizeImageTask.

Also NB: I’m testing on 2 versions of the stack, a very old local one (from April 2016) and a recent one, which is the one that matters to other users, so that’s the one for which I’ll report the error. (In this case, the example actually runs fine locally.)

So, on the Princeton Tiger-Sumire machine, after logging in and doing:

source /tigress/HSC/LSST/stack_20160915/eups/bin/setups.sh
setup miniconda2
setup lsst_apps

then going to $PIPE_TASKS_DIR/examples and running out of the box python photoCalTask.py (which I checked is using by default python ver 2.7.11 btw), yields the below, which indicates some problem with a file possibly being moved? (The schema subdir in fact doesn’t exist at all).

thx.

------------------ Output:

Traceback (most recent call last):
  File "calibrateTask.py", line 175, in <module>
    run(display=args.display)
  File "calibrateTask.py", line 128, in run
    calibrateTask = CalibrateTask(butler, config=config)
  File "/tigress/HSC/LSST/stack_20160915/Linux64/pipe_tasks/12.1-10-gcf06047+5/python/lsst/pipe/tasks/calibrate.py", line 279, in __init__
    icSourceSchema = butler.get("icSrc_schema", immediate=True).schema
  File "/tigress/HSC/LSST/stack_20160915/Linux64/daf_persistence/12.1-11-gb70df55+1/python/lsst/daf/persistence/butler.py", line 749, in get
    return callback()
  File "/tigress/HSC/LSST/stack_20160915/Linux64/daf_persistence/12.1-11-gb70df55+1/python/lsst/daf/persistence/butler.py", line 744, in <lambda>
    callback = lambda: self._read(location)
  File "/tigress/HSC/LSST/stack_20160915/Linux64/daf_persistence/12.1-11-gb70df55+1/python/lsst/daf/persistence/butler.py", line 888, in _read
    results = location.repository.read(location)
  File "/tigress/HSC/LSST/stack_20160915/Linux64/daf_persistence/12.1-11-gb70df55+1/python/lsst/daf/persistence/repository.py", line 152, in read
    return self._storage.read(butlerLocation)
  File "/tigress/HSC/LSST/stack_20160915/Linux64/daf_persistence/12.1-11-gb70df55+1/python/lsst/daf/persistence/posixStorage.py", line 276, in read
    raise RuntimeError("No such FITS catalog file: " + logLoc.locString())
RuntimeError: No such FITS catalog file: /tigress/HSC/LSST/stack_20160915/Linux64/obs_test/12.1-9-g3e397f1+2/data/input/schema/icSrc.fits
{tiger-sumire:/tigress/HSC/LSST/stack_20160915/Linux64/pipe_tasks/12.1-10-gcf06047+5/examples}py calibrateTask.py 
Traceback (most recent call last):
  File "calibrateTask.py", line 175, in <module>
    run(display=args.display)
  File "calibrateTask.py", line 128, in run
    calibrateTask = CalibrateTask(butler, config=config)
  File "/tigress/HSC/LSST/stack_20160915/Linux64/pipe_tasks/12.1-10-gcf06047+5/python/lsst/pipe/tasks/calibrate.py", line 279, in __init__
    icSourceSchema = butler.get("icSrc_schema", immediate=True).schema
  File "/tigress/HSC/LSST/stack_20160915/Linux64/daf_persistence/12.1-11-gb70df55+1/python/lsst/daf/persistence/butler.py", line 749, in get
    return callback()
  File "/tigress/HSC/LSST/stack_20160915/Linux64/daf_persistence/12.1-11-gb70df55+1/python/lsst/daf/persistence/butler.py", line 744, in <lambda>
    callback = lambda: self._read(location)
  File "/tigress/HSC/LSST/stack_20160915/Linux64/daf_persistence/12.1-11-gb70df55+1/python/lsst/daf/persistence/butler.py", line 888, in _read
    results = location.repository.read(location)
  File "/tigress/HSC/LSST/stack_20160915/Linux64/daf_persistence/12.1-11-gb70df55+1/python/lsst/daf/persistence/repository.py", line 152, in read
    return self._storage.read(butlerLocation)
  File "/tigress/HSC/LSST/stack_20160915/Linux64/daf_persistence/12.1-11-gb70df55+1/python/lsst/daf/persistence/posixStorage.py", line 276, in read
    raise RuntimeError("No such FITS catalog file: " + logLoc.locString())
RuntimeError: No such FITS catalog file: /tigress/HSC/LSST/stack_20160915/Linux64/obs_test/12.1-9-g3e397f1+2/data/input/schema/icSrc.fits
#2

It looks as though there are a couple of problems in this example.

First, the issue resulting in the traceback above. This is happening because the example explicitly passes a butler to CalibrateTask, which then attempts to use it to look up table schemas which a) it doesn’t strictly need, and b) don’t exist. You can work around it by editing line 128 of the example like so:

--- a/examples/calibrateTask.py
+++ b/examples/calibrateTask.py
@@ -125,7 +125,7 @@ def run(display=False):

     config = CalibrateTask.ConfigClass()
     config.astrometry.retarget(MyAstrometryTask)
-    calibrateTask = CalibrateTask(butler, config=config)
+    calibrateTask = CalibrateTask(config=config)

     # load the data
     # Exposure ID and the number of bits required for exposure IDs are usually obtained from a data repo,

However, having done that, you’ll hit another failure with a very lengthy error message which goes something like:

Traceback (most recent call last):
  File "calibrateTask.py", line 175, in <module>
    run(display=args.display)
  File "calibrateTask.py", line 133, in run
    exposure = loadData()
  File "calibrateTask.py", line 59, in loadData
    exposure = afwImage.ExposureF(imFile, visitInfo)
  File "/ssd/lsstsw/stack/Linux64/afw/12.1-24-g22f008f+1/python/lsst/afw/image/imageLib.py", line 11633, in __init__
    this = _imageLib.new_ExposureF(*args)
NotImplementedError: Wrong number or type of arguments for overloaded function 'new_ExposureF'.
  Possible C/C++ prototypes are:

A naïve workaround is to change line 59 of the example like so:

--- a/examples/calibrateTask.py
+++ b/examples/calibrateTask.py
@@ -56,7 +56,10 @@ def loadData(pixelScale=1.0):
     imFile = os.path.join(mypath, "CFHT", "D4", "cal-53535-i-797722_small_1.fits")

     visitInfo = afwImage.makeVisitInfo(exposureTime=1.0)
-    exposure = afwImage.ExposureF(imFile, visitInfo)
+    exposureInfo = afwImage.ExposureInfo()
+    exposureInfo.setVisitInfo(visitInfo)
+    exposure = afwImage.ExposureF(imFile)
+    exposure.setInfo(exposureInfo)
     # add a filter
     afwImage.Filter.define(afwImage.FilterProperty(FilterName, 600, True))
     exposure.setFilter(afwImage.Filter(FilterName))

That all seems surprisingly verbose to me — I wonder if @rowen knows of a better way?

I’ll file a JIRA ticket to record that this example needs fixing.

#3

DM-8390.

#4

Thx @swinbank – i made both modifications, reran, and get the below - afwdata is a core pkg, so i don’t get why it’s not being found…?

Traceback (most recent call last):
  File "calibrateTask.py", line 181, in <module>
    run(display=args.display)
  File "calibrateTask.py", line 139, in run
    exposure = loadData()
  File "calibrateTask.py", line 55, in loadData
    mypath = lsst.utils.getPackageDir('afwdata')
  File "/tigress/HSC/LSST/stack_20160915/Linux64/utils/12.1-2-gd71c75d+4/python/lsst/utils/utilsLib.py", line 156, in getPackageDir
    return _utilsLib.getPackageDir(packageName)
lsst.pex.exceptions.wrappers.NotFoundError: 
  File "src/Utils.cc", line 42, in std::string lsst::utils::getPackageDir(const string&)
    Package afwdata not found {0}
lsst::pex::exceptions::NotFoundError: 'Package afwdata not found'
#5

I’m not sure exactly what you mean by “core pkg”, but afwdata isn’t available in the shared stack on Tiger (or lsst-dev01) because we don’t distribute it through eups distrib:

[swinbank@tiger-sumire ~]$ . /tigress/HSC/LSST/stack/loadLSST.bash
[swinbank@tiger-sumire ~]$ eups list afwdata
eups list: Unable to find product afwdata

You’ll need to clone it and set it up yourself.

#6

Ah! Ok, will work on this then, thx…