coaddDriver.py error (HSC data processing)

Support
#1

So I have been processing the data from HSC with visit=007808-007818 for Abell 2319. The output of singleFrameDrive seems fine with corr/BKGD, CORR and output/ICSRC, SRC, SRCMATCH, SRCMATCHFULL.
However, when I proceed to coaddDriver, there is no output product. Warnings like

WARN  2017-07-17T02:24:20.020 coaddDriver.makeCoaddTempExp ()(makeCoaddTempExp.py:186)- 
  Error processing calexp DataId(initialdata={'taiObs': '2014-09-24', u'pointing': 997, 'visit': 7814, 'dateObs': '2014-09-24', u'filter': 'HSC-I', 'field': 'ABELL2319', 'tract': 0, u'ccd': 70, 'expTime': 240.0}, tag=set([])); skipping it: No locations for get: datasetType:wcs_md dataId:DataId(initialdata={'taiObs': '2014-09-24', u'pointing': 997, 'visit': 7814, 'dateObs': '2014-09-24', u'filter': 'HSC-I', 'field': 'ABELL2319', 'tract': 0, u'ccd': 70, 'expTime': 240.0}, tag=set([]))

are found inside log, not sure if it is the cause though.
Has anyone encountered such problem?
Thank you.

1 Like
Problem using skyCorrection.py and buliding meas_mosaic
#2

The HSC configuration for coaddDriver.py by default tries to use outputs from meas_mosaic, a joint calibration script that can only be used on HSC data. This includes the wcs dataset (improved image registation) and the fcr dataset (improve photometric calibration). You can either run meas_mosaic (you’ll have to install it from git source, as it’s not part of the main LSST pipeline), or you can tell coaddDriver.py to just use the original calibrations from singleFrameDriver.py with the following options. You can find out how to do either here:

https://dmtn-023.lsst.io/#joint-calibration

`

1 Like
#3

I didn’t see any instructions on how to run meas_mosaic in DMTN-023, so here are some:

# Build meas_mosaic
cd /work
git clone git://github.com/LSST/meas_mosaic
cd meas_mosaic
scons  # Or 'python $(which scons)' on OSX

# Run meas_mosaic
mosaic.py /path/to/DATA --rerun myRerun -c allowMixedFilters=True --numCoresForRead=12 --diagnostics --diagDir=/path/to/diagnostics --id tract=0 field=MY_FIELD filter=HSC-I^HSC-I2 ccd=0..8^10..103

Here, I’m combining data from the i and i2 filters, and I drop ccd=9 because it usually doesn’t have good data (half the amps are dead).

1 Like
Problem building meas_mosaic
Problem using skyCorrection.py and buliding meas_mosaic
#4

Thank you so much. I had installed meas_mosaic, yet I have no idea I need to run mosaic.py and how. It is clear now.
For clarification, does field means visit? As I tried patch=0,0^... , it cannot identify it as ID. Also, I reckon meas_mosaic should be run after makeDiscreteSkyMap.py? Then the order in https://dmtn-023.lsst.io/#joint-calibration is a bit confusing.

Besides, as I run
mosaic.py DATA --rerun output8 --numCoresForRead=4 --diagnostics --diagDir=DATA/diagnostics --id tract=0 visit=007802..007806:2 filter=HSC-I ccd=0..8^10..69^71..103
( ccd=70 has an error of lsst::pex::exceptions::InvalidParameterError: 'Mismatch between yaw (-409.047253 deg) and nQuarter (3)) ,
the following error pops out:

Mosaic INFO: Output Flux Diagnostic Figures...
/home/verawu/my_packages/meas_mosaic/python/lsst/meas/mosaic/utils.py:827: RuntimeWarning: invalid value encountered in divide
  ra /= numbers
/home/verawu/my_packages/meas_mosaic/python/lsst/meas/mosaic/utils.py:828: RuntimeWarning: invalid value encountered in divide
  dec /= numbers
/home/verawu/my_packages/meas_mosaic/python/lsst/meas/mosaic/utils.py:829: RuntimeWarning: invalid value encountered in divide
  mag /= err
/home/verawu/my_packages/meas_mosaic/python/lsst/meas/mosaic/utils.py:830: RuntimeWarning: invalid value encountered in sqrt
  err = numpy.sqrt((var - mag*mag*err)/err)

As I checked, basically all the fcr and wcs fits are produced, though.

#5

Those RuntimeWarnings from utils.py are normal. You should now be able to run coaddDriver.py.

I don’t understand the problem with ccd=70; could you elaborate? What command did you run that produced that error? Can you post the entire output from the command?

field doesn’t mean visit. It’s a different way of specifying data. For example, you can specify field=M31 dateObs=2017-07-18 filter=HSC-I or you can give a list of visits, visit=1234..1248:2. Think of it like a database query that will provide the CCDs of interest.

#6

The error output is like this:

Number good: 72672, 1025
Traceback (most recent call last):
  File "/home/verawu/my_packages/meas_mosaic/bin/mosaic.py", line 5, in <module>
    MosaicTask.parseAndRun()
  File "/../lsstsw/stack/Linux64/pipe_base/13.0-9-g1c7d9c5+11/python/lsst/pipe/base/cmdLineTask.py", line 514, in parseAndRun
    resultList = taskRunner.run(parsedCmd)
  File "/../lsstsw/stack/Linux64/pipe_base/13.0-9-g1c7d9c5+11/python/lsst/pipe/base/cmdLineTask.py", line 223, in run
    resultList = list(mapFunc(self, targetList))
  File "/home/verawu/my_packages/meas_mosaic/python/lsst/meas/mosaic/mosaicTask.py", line 79, in __call__
    result = task.run(*args)
  File "/home/verawu/my_packages/meas_mosaic/python/lsst/meas/mosaic/mosaicTask.py", line 1029, in run
    numCoresForReadSource, readTimeout, verbose)
  File "/home/verawu/my_packages/meas_mosaic/python/lsst/meas/mosaic/mosaicTask.py", line 918, in mosaic
    snapshots, self.outputDir)
lsst.pex.exceptions.wrappers.InvalidParameterError: 
  File "src/shimCameraGeom.cc", line 25, in lsst::afw::geom::Angle lsst::meas::mosaic::getYaw(std::shared_ptr<const lsst::afw::cameraGeom::Detector>)
    Mismatch between yaw (-409.047253 deg) and nQuarter (3) for detector 70: abs(yaw - 90*nQuarter) = 679.047253 is > 90 deg {0}
lsst::pex::exceptions::InvalidParameterError: 'Mismatch between yaw (-409.047253 deg) and nQuarter (3) for detector 70: abs(yaw - 90*nQuarter) = 679.047253 is > 90 deg'

When I tried to include more visits, the error is gone.

#7

That looks like a bug in meas_mosaic. Could you please post the command that you ran to get that output, so we can fix it?

#8

The command is:

mosaic.py DATA --rerun output8 --numCoresForRead=4 --diagnostics --diagDir=DATA/diagnostics --id tract=0 visit=007802..007806:2 filter=HSC-I ccd=0..8^10..103

When i picked visit=007802..007818:2 for the same command, the error is gone and the data product is in place. Though calexp-[filter]-[tract]-[patch].fits is missing, it should be the problem mentioned in DM-10413. Hope the information is helpful.

Thank you.

#9

Sorry for brining questions again.
As I ran mosaic.py for another filter, HSC-G for 007852 - 007868 of ABELL2319, the warning appeared for many of the ID, e.g.
afw.table WARN: Persisted match record with ID 160992895358036012 not found in catalog 1.
I suspected that this might be something of the processCcd task? Though the output seems normal.

Also, the following error comes out:

Traceback (most recent call last):
  File "/home/verawu/my_packages/meas_mosaic/bin/mosaic.py", line 5, in <module>
    MosaicTask.parseAndRun()
  File "/../lsstsw/stack/Linux64/pipe_base/13.0-9-g1c7d9c5+11/python/lsst/pipe/base/cmdLineTask.py", line 514, in parseAndRun
    resultList = taskRunner.run(parsedCmd)
  File "/../lsstsw/stack/Linux64/pipe_base/13.0-9-g1c7d9c5+11/python/lsst/pipe/base/cmdLineTask.py", line 223, in run
    resultList = list(mapFunc(self, targetList))
  File "/home/verawu/my_packages/meas_mosaic/python/lsst/meas/mosaic/mosaicTask.py", line 79, in __call__
    result = task.run(*args)
  File "/home/verawu/my_packages/meas_mosaic/python/lsst/meas/mosaic/mosaicTask.py", line 1029, in run
    numCoresForReadSource, readTimeout, verbose)
  File "/home/verawu/my_packages/meas_mosaic/python/lsst/meas/mosaic/mosaicTask.py", line 918, in mosaic
    snapshots, self.outputDir)
RuntimeError: solving linear equation failed: dgesv returned 1093
#10

That warning means it couldn’t find the reference source in the catalog, and the RuntimeError means it couldn’t solve the matrix. I suppose you’re using the PS1 reference catalog and the latest weekly release?

I think I’m going to have to try to reproduce this myself. I was already working on the previous problem when we hit cluster maintenance time. Can you please let me know the full range of data you’re working on, so I can download it? If you can make the data available in the US that would save some time. And could you please post the list of commands that you’ve run as part of this reduction so I can duplicate?

#11

Yes, I have retrieved the shards with ra=290.2833, dec=43.3078 and radius=1.6deg from https://lsst-web.ncsa.illinois.edu/~pprice/ps1_pv3_3pi_20170110/. For the DM stack and packages, it should be latest.

I am working on the data of ABELL2319 from HSC, which I downloaded from SMOKA. The FRAME ID of SHOTs is HSCA00780[0,1]XX-00781[8,9]XX which is for filter=HSC-I and HSCA00785[0,1]XX-00786[8,9]XX which is for filter=HSC-G.

I have set up meas_extensions_photometryKron and meas_extensions_convolved
The commands I ran for this part is:

ingestImage.py ...    
singleFrameDriver.py DATA --rerun output --id visit=007852..007868:2 --cores=8
makeDiscreteSkyMap.py DATA --rerun output:output9 --id visit=007852..007868:2 --config skyMap.projection="TAN"
mosaic.py DATA --rerun output9 --numCoresForRead=8 --diagnostics --diagDir=DATA/diagnostics3 --id tract=0 visit=007852..007868:2 filter=HSC-G ccd=0..8^10..103

For the numCoresForRead, I have tried 4 and 12. All of them have the same error.

The following works fine with calexp outputs:

ingestImage.py  ...
singleFrameDriver.py DATA --rerun output --id visit=007802..007818:2 --cores=8
makeDiscreteSkyMap.py DATA --rerun output:output9 --id visit=007802..007818:2 --config skyMap.projection="TAN"
mosaic.py DATA --rerun output9 --numCoresForRead=8 --diagnostics --diagDir=DATA/diagnostics2 --id tract=0 visit=007802..007818:2 filter=HSC-I ccd=0..8^10..103
coadd.py ....

I ran the process separately for different filters just as I wanted to try out if it works. Hope the information is clear and helpful.

#12

I’ve downloaded the data, and will try it out.

Here’s a thought: I believe you’re using a relatively recent weekly release of the LSST pipeline (not sure which?). It’s possible you’re running into little problems that pop up and get fixed as we go along in the course of development. If you want a more stable release that is known to work pretty well on HSC data, you could try the HSC pipeline, which is based on the LSST pipeline as of a couple of months ago. Instructions for building it are here. The latest version is hscPipe 5.2.

#13

I’ve tracked down the problem causing the Persisted match record with ID 160992895358036012 not found in catalog 1 warnings: DM-11356. I’ll fix that and see if it helps meas_mosaic do better.

#14

I have a solution for you, @Vera.

Firstly, I think DM-11356 turned out to be mostly a red herring for this case (though I’m glad I fixed it). The main problem is bad astrometry coming out of singleFrameDriver.py. Unfortunately, the astrometry has been a bit unstable recently. My recommendation, below, combines two attempts to improve this situation:

I used the following configuration with singleFrameDriver.py:

if hasattr(config, "processCcd"):
    config = config.processCcd

# Improves astrometric matching
from lsst.meas.astrom import MatchPessimisticBTask
for cfg in (config.calibrate.astrometry,
            config.charImage.ref_match,
            config.calibrate.photoCal):
   cfg.matcher.retarget(MatchPessimisticBTask)
   cfg.matcher.maxOffsetPix = 250  # not required for w_2017_29 and following

Then meas_mosaic completes successfully with the g-band data.

#15

Hi Paul -

I’m trying to process HSC (MADCASH) data on lsst-dev, and am having troubles getting meas_mosaic to work. I’m following the steps you outlined above, but get the following:

[jcarlin@lsst-dev01 meas_mosaic]$ scons
scons: Reading SConscript files ...
EUPS integration: enabled
Checking who built the CC compiler...gcc=4.8.5
CC is gcc version 4.8.5
Checking for C++14 support
Checking whether the C++ compiler works... no
C++14 extensions could not be enabled for compiler 'gcc'

Which I assume is the reason why when I setup meas_mosaic, it is not found in my path?
[jcarlin@lsst-dev01 meas_mosaic]$ cd …/
[jcarlin@lsst-dev01 madcash]$ setup -jr meas_mosaic
[jcarlin@lsst-dev01 madcash]$ mosaic.py
bash: mosaic.py: command not found

Any tips?

-Jeff

#16

You need a more modern C++ compiler. I think you want at least gcc 5.0.

#17

This is on lsst-dev. Is there not a more modern compiler on lsst-dev? (And can I install things on lsst-dev?)

#18

Dunno where you’re getting your ancient compiler:

pprice@lsst-dev01:~ $ g++ --version
g++ (GCC) 6.3.1 20170216 (Red Hat 6.3.1-3)
Copyright (C) 2016 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Maybe something’s cached? Try running scons -c and then scons again.

#19

Still got the same (old) version:

[jcarlin@lsst-dev01 meas_mosaic]$ scons -c
scons: Reading SConscript files ...
EUPS integration: enabled
Setting up environment to build package 'meas_mosaic'.
find . \( -name .svn -prune -o -name \* \) \( -name \*~ -o -name core -o -name \*.so -o -name \*.os -o -name \*.o -o -name \*.pyc -o -name \*.pkgc \) -exec rm -f {} \; ; rm -rf .sconf_temp .sconsign.dblite .sconsign.tmp config.log
Running pytest with 1 process
pytest: running on 2 Python test files.
scons: done reading SConscript files.
scons: Cleaning targets ...
scons: done cleaning targets.
[jcarlin@lsst-dev01 meas_mosaic]$ scons
scons: Reading SConscript files ...
EUPS integration: enabled
Checking who built the CC compiler...gcc=4.8.5
CC is gcc version 4.8.5
Checking for C++14 support
Checking whether the C++ compiler works... no
C++14 extensions could not be enabled for compiler 'gcc'
[jcarlin@lsst-dev01 meas_mosaic]$ g++ --version
g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-16)
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
#20

Oh, you need to do source scl_source enable devtoolset-6. See here.