Error of using obs_decam

mbl · September 21, 2016, 6:46pm

Hello,

I encountered a few problems when I tried processing data with obs_decam.

When I tried to import the data to the repository, I received the warning messages.

CameraMapper: Unable to locate registry registry in root: /home/mleng/lsst/Linux64/obs_decam/repo1/registry.sqlite3
CameraMapper: Loading registry registry from ./registry.sqlite3
CameraMapper: Unable to locate calibRegistry registry in root: /home/mleng/lsst/Linux64/obs_decam/repo1/calibRegistry.sqlite3
CameraMapper: Unable to locate calibRegistry registry in current dir: ./calibRegistry.sqlite3
CameraMapper: Loading Posix registry from /home/mleng/lsst/Linux64/obs_decam/repo1
CameraMapper: Unable to locate registry registry in root: /home/mleng/lsst/Linux64/obs_decam/repo1/registry.sqlite3
CameraMapper: Loading registry registry from ./registry.sqlite3
CameraMapper: Unable to locate calibRegistry registry in root: /home/mleng/lsst/Linux64/obs_decam/repo1/calibRegistry.sqlite3
CameraMapper: Unable to locate calibRegistry registry in current dir: ./calibRegistry.sqlite3
CameraMapper: Loading Posix registry from /home/mleng/lsst/Linux64/obs_decam/repo1
ingest.parse WARNING: Unable to find value for ccdnum (derived from CCDNUM)

And when I tried to process the image using processCcdDecam.py, it turned out the command was not found. It seems that this file has been removed from the package already.

How can I get the obs_decam working? Any help would be very much appreciated.

Thank you

ktl · September 21, 2016, 7:19pm

Most of those messages (the “Unable to locate” ones) can be ignored. (They ought to go away relatively soon.) The message about CCDNUM is indicating that the file you tried to ingest didn’t have that header.

I think that processCcdDecam.py has been superseded by using the standard processCcd.py. See Backward-incompatible changes to ProcessCcdTask and subtasks

price · September 21, 2016, 7:26pm

It would be helpful if you could identify which document you’re looking at that told you to use processCcdDecam.py, as that document needs to be updated.

mbl · September 21, 2016, 8:07pm

Hi,

Thanks for letting me know that. But when I tried using processCcd.py instead and replacing isr with DecamNullIsrTask to process the images, something is wrong with config override file.

I received an error:

raise AttributeError("%s has no attribute %s"%(_typeStr(self), attr))
AttributeError: lsst.obs.decam.decamNullIsr.DecamNullIsrConfig has no attribute doDark

I’d be grateful if you can let me know how to fix this error.

Thanks

mbl · September 21, 2016, 8:09pm

Hi,

I am using the following two documents as guidance (which I realized that they might be obsolete).

https://confluence.lsstcorp.org/display/LSWUG/Process+DECam+Images

price · September 21, 2016, 8:29pm

@hsinfang, are you the right person to update these?

rowen · September 21, 2016, 8:34pm

I think ccdNum has been replaced by ccd. I can’t answer your other questions, but encourage you to look at the config overrides and the available ISR tasks (of which I see two)

hsinfang · September 21, 2016, 11:10pm

It’s in DM-7283 which is waiting on the DMLT decisions…

Isn’t the confluence User Guide no longer updated because the information is being moved to somewhere else or that was for an older version of the stack,? (@jsick do you know something about that, or have advices on how to proceed with obs_* documentations? I seem to recall conversations that obs_decam README may not the right place; should I remove the outdated info in obs_decam README to avoid further confusion? )

@mbl may you please post here what command did you run for getting the error? I haven’t run processCcd with DECam instcal data for a while, but I think that is CI-ed by validate_drp so it should run. It is true that DecamNullIsrConfig does not have doDark (code here).

I may not worry about the WARNING from ingest because I guess it comes from HDU0; if so, just ignore it. It depends on what data were ingested.

ccdnum is not really replaced by ccd in obs_decam. ccdnum is still the proper dataId for those datasets needing the ccd-like dataId in obs_decam, but ccd may be used as an aliased dataId for getting raw and calexp.

jsick · September 22, 2016, 12:10am

@hsinfang Feel free to put documentation in the obs_decam README for now. We’ll get https://pipelines.lsst.io up to speed soon, but the READMEs are a good interim medium.

The Confluence page is (or will be) deprecated.

mbl · September 24, 2016, 12:17am

Hi, here is what I did and received

 >$ processCcd.py /home/mleng/lsst/Linux64/obs_decam/repo --id visit=283453 ccdnum=10 --config calibrate.doPhotoCal=False calibrate.doAstrometry=False calibrate.measurePsf.starSelector.name="seconMoment" doWriteCalibrateMatches=False --clobber-                     -show config
> : Loading config overrride file '/home/mleng/lsst/Linux64/obs_decam/config/processCcd.py'
> Traceback (most recent call last):
>   File "/home/mleng/lsst/Linux64/pipe_tasks/2016_01.0-48-g7bf2411/bin/processCcd.py", line 25, in <module>
>     ProcessCcdTask.parseAndRun()
>   File "/home/mleng/lsst/Linux64/pipe_base/2016_01.0-9-gbdeabf2+8/python/lsst/pipe/base/cmdLineTask.py", line 450, in parseAndRun
>     parsedCmd = argumentParser.parse_args(config=config, args=args, log=log, override=cls.applyOverrides)
>   File "/home/mleng/lsst/Linux64/pipe_base/2016_01.0-9-gbdeabf2+8/python/lsst/pipe/base/argumentParser.py", line 436, in parse_args
>     self._applyInitialOverrides(namespace)
>   File "/home/mleng/lsst/Linux64/pipe_base/2016_01.0-9-gbdeabf2+8/python/lsst/pipe/base/argumentParser.py", line 599, in _applyInitialOverrides
>     namespace.config.load(filePath)
>   File "/home/mleng/lsst/Linux64/pex_config/2016_01.0-1-g6fbf654+11/python/lsst/pex/config/config.py", line 532, in load
>     self.loadFromStream(stream=code, root=root)
>   File "/home/mleng/lsst/Linux64/pex_config/2016_01.0-1-g6fbf654+11/python/lsst/pex/config/config.py", line 552, in loadFromStream
>     exec stream in {}, local
>   File "/home/mleng/lsst/Linux64/obs_decam/config/processCcd.py", line 5, in <module>
>     config.isr.doDark = False
>   File "/home/mleng/lsst/Linux64/pex_config/2016_01.0-1-g6fbf654+11/python/lsst/pex/config/configurableField.py", line 120, in __setattr__
>     self._value.__setattr__(name, value, at=at, label=label)
>   File "/home/mleng/lsst/Linux64/pex_config/2016_01.0-1-g6fbf654+11/python/lsst/pex/config/config.py", line 694, in __setattr__
>     raise AttributeError("%s has no attribute %s"%(_typeStr(self), attr))
> AttributeError: lsst.obs.decam.decamNullIsr.DecamNullIsrConfig has no attribute doDark

Here is config file:

from lsst.obs.decam.decamNullIsr import DecamNullIsrTask

config.isr.retarget(DecamNullIsrTask)

config.isr.doDark = False
config.isr.fringe.filters=['z', 'y']
config.isr.assembleCcd.keysToRemove = ['DATASECA', 'DATASECB',
                                       'TRIMSECA', 'TRIMSECB',
                                       'BIASSECA', 'BIASSECB',
                                       'PRESECA', 'PRESECB',
                                       'POSTSECA', 'POSTSECB']

config.charImage.repair.cosmicray.nCrPixelMax = 100000

hsinfang · September 26, 2016, 5:09pm

If isr is retargeted to DecamNullIsrTask as in your config file, there is no doDark, fringe, etc in its config, so lines such as config.isr.doDark = False do not make sense and should be removed.

mbl · October 7, 2016, 8:23pm

Hello,

@hsinfang Thank you for updating document. The processCcd.py is working now.

But I have got one question regarding the information the output table contains.

The following is the list of the output columns; however I am interested in getting to know the sizes of objects (for instance, the radius for a specified fraction of flux), which is not in the list. What can I do to have it returned the output of parameter I would like to have?

Thanks

ColDefs(
    name = 'flags'; format = '73X'
    name = 'id'; format = '1K'
    name = 'coord_ra'; format = '1D'
    name = 'coord_dec'; format = '1D'
    name = 'parent'; format = '1K'
    name = 'deblend_nChild'; format = '1J'
    name = 'deblend_psfCenter_x'; format = '1D'; unit = 'pixel'
    name = 'deblend_psfCenter_y'; format = '1D'; unit = 'pixel'
    name = 'deblend_psfFlux'; format = '1D'
    name = 'base_GaussianCentroid_x'; format = '1D'; unit = 'pixel'
    name = 'base_GaussianCentroid_y'; format = '1D'; unit = 'pixel'
    name = 'base_NaiveCentroid_x'; format = '1D'; unit = 'pixel'
    name = 'base_NaiveCentroid_y'; format = '1D'; unit = 'pixel'
    name = 'base_SdssCentroid_x'; format = '1D'; unit = 'pixel'
    name = 'base_SdssCentroid_y'; format = '1D'; unit = 'pixel'
    name = 'base_SdssCentroid_xSigma'; format = '1E'; unit = 'pixel'
    name = 'base_SdssCentroid_ySigma'; format = '1E'; unit = 'pixel'
    name = 'base_SdssShape_xx'; format = '1D'; unit = 'pixel^2'
    name = 'base_SdssShape_yy'; format = '1D'; unit = 'pixel^2'
    name = 'base_SdssShape_xy'; format = '1D'; unit = 'pixel^2'
    name = 'base_SdssShape_xxSigma'; format = '1E'; unit = 'pixel^2'
    name = 'base_SdssShape_yySigma'; format = '1E'; unit = 'pixel^2'
    name = 'base_SdssShape_xySigma'; format = '1E'; unit = 'pixel^2'
    name = 'base_SdssShape_x'; format = '1D'; unit = 'pixel'
    name = 'base_SdssShape_y'; format = '1D'; unit = 'pixel'
    name = 'base_SdssShape_flux'; format = '1D'; unit = 'count'
    name = 'base_SdssShape_fluxSigma'; format = '1D'; unit = 'count'
    name = 'base_SdssShape_flux_xx_Cov'; format = '1E'; unit = 'count*pixel^2'
    name = 'base_SdssShape_flux_yy_Cov'; format = '1E'; unit = 'count*pixel^2'
    name = 'base_SdssShape_flux_xy_Cov'; format = '1E'; unit = 'count*pixel^2'
    name = 'base_CircularApertureFlux_3_0_flux'; format = '1D'; unit = 'count'
    name = 'base_CircularApertureFlux_3_0_fluxSigma'; format = '1D'; unit = 'count'
    name = 'base_CircularApertureFlux_4_5_flux'; format = '1D'; unit = 'count'
    name = 'base_CircularApertureFlux_4_5_fluxSigma'; format = '1D'; unit = 'count'
    name = 'base_CircularApertureFlux_6_0_flux'; format = '1D'; unit = 'count'
    name = 'base_CircularApertureFlux_6_0_fluxSigma'; format = '1D'; unit = 'count'
    name = 'base_CircularApertureFlux_9_0_flux'; format = '1D'; unit = 'count'
    name = 'base_CircularApertureFlux_9_0_fluxSigma'; format = '1D'; unit = 'count'
    name = 'base_CircularApertureFlux_12_0_flux'; format = '1D'; unit = 'count'
    name = 'base_CircularApertureFlux_12_0_fluxSigma'; format = '1D'; unit = 'count'
    name = 'base_CircularApertureFlux_17_0_flux'; format = '1D'; unit = 'count'
    name = 'base_CircularApertureFlux_17_0_fluxSigma'; format = '1D'; unit = 'count'
    name = 'base_CircularApertureFlux_25_0_flux'; format = '1D'; unit = 'count'
    name = 'base_CircularApertureFlux_25_0_fluxSigma'; format = '1D'; unit = 'count'
    name = 'base_CircularApertureFlux_35_0_flux'; format = '1D'; unit = 'count'
    name = 'base_CircularApertureFlux_35_0_fluxSigma'; format = '1D'; unit = 'count'
    name = 'base_CircularApertureFlux_50_0_flux'; format = '1D'; unit = 'count'
    name = 'base_CircularApertureFlux_50_0_fluxSigma'; format = '1D'; unit = 'count'
    name = 'base_CircularApertureFlux_70_0_flux'; format = '1D'; unit = 'count'
    name = 'base_CircularApertureFlux_70_0_fluxSigma'; format = '1D'; unit = 'count'
    name = 'base_GaussianFlux_flux'; format = '1D'; unit = 'count'
    name = 'base_GaussianFlux_fluxSigma'; format = '1D'; unit = 'count'
    name = 'base_PsfFlux_flux'; format = '1D'; unit = 'count'
    name = 'base_PsfFlux_fluxSigma'; format = '1D'; unit = 'count'
    name = 'base_Variance_value'; format = '1D'
    name = 'base_ClassificationExtendedness_value'; format = '1D'
    name = 'base_PsfFlux_apCorr'; format = '1D'
    name = 'base_PsfFlux_apCorrSigma'; format = '1D'
    name = 'base_GaussianFlux_apCorr'; format = '1D'
    name = 'base_GaussianFlux_apCorrSigma'; format = '1D'
    name = 'footprint'; format = '1J'
)

price · October 12, 2016, 7:29pm

For the size of objects, you can look at the base_SdssShape_{xx,yy,xy} values, which are adaptive moments. I expect you can convert that to “the radius for a specified fraction of flux” if you assume a Gaussian profile.

I recommend using the HSM adaptive moments, as we’ve found they are a bit better behaved than the SDSS versions at the faint end, but it doesn’t look like you’re currently set up to get those. Try putting the following in your processCcd.py configuration:

from lsst.utils import getPackageDir
def activateHsm(cfg):
    cfg.load(os.path.join(getPackageDir("meas_extensions_shapeHSM"), "config", "enable.py"))
    cfg.plugins["ext_shapeHSM_HsmShapeRegauss"].deblendNChild = "deblend_nChild"
activateHsm(config.charImage.measurement)
activateHsm(config.calibrate.measurement)

You might also try using the Kron radius by putting the following in your processCcd.py configuration:

import lsst.meas.extensions.photometryKron
config.charImage.measurement.plugins.names |= ["ext_photometryKron_KronFlux"]
config.calibrate.measurement.plugins.names |= ["ext_photometryKron_KronFlux"]