Hi everyone
I’m currently starting to use the lsst pipeline, and I’ve been following the Getting started tutorial (Getting started with the LSST Science Pipelines — LSST Science Pipelines). In part 2 to calibrate the data, the command processCcd.py is used; I can do the ‘dry-run’ to see what data will be processed in the Butler repository:
(lsst) jahumada@leftraru2:/mnt/flock/jahumada/Paula$ processCcd.py DATA --rerun processCcdOutputs --id --show data
root INFO: Loading config overrride file '/home/jahumada/.conda/envs/lsst/opt/lsst/obs_subaru/config/processCcd.py'
Cannot import lsst.meas.extensions.convolved (No module named convolved): disabling convolved flux measurements
root INFO: Loading config overrride file '/home/jahumada/.conda/envs/lsst/opt/lsst/obs_subaru/config/hsc/processCcd.py'
id dataRef.dataId = {'taiObs': '2013-06-17', 'pointing': 533, 'visit': 903334, 'dateObs': '2013-06-17', 'filter': 'HSC-R', 'field': 'STRIPE82L', 'ccd': 23, 'expTime': 30.0}
id dataRef.dataId = {'taiObs': '2013-06-17', 'pointing': 533, 'visit': 903334, 'dateObs': '2013-06-17', 'filter': 'HSC-R', 'field': 'STRIPE82L', 'ccd': 22, 'expTime': 30.0}
id dataRef.dataId = {'taiObs': '2013-06-17', 'pointing': 533, 'visit': 903334, 'dateObs': '2013-06-17', 'filter': 'HSC-R', 'field': 'STRIPE82L', 'ccd': 16, 'expTime': 30.0}
id dataRef.dataId = {'taiObs': '2013-06-17', 'pointing': 533, 'visit': 903334, 'dateObs': '2013-06-17', 'filter': 'HSC-R', 'field': 'STRIPE82L', 'ccd': 100, 'expTime': 30.0}
id dataRef.dataId = {'taiObs': '2013-06-17', 'pointing': 533, 'visit': 903336, 'dateObs': '2013-06-17', 'filter': 'HSC-R', 'field': 'STRIPE82L', 'ccd': 24, 'expTime': 30.0}
id dataRef.dataId = {'taiObs': '2013-06-17', 'pointing': 533, 'visit': 903336, 'dateObs': '2013-06-17', 'filter': 'HSC-R', 'field': 'STRIPE82L', 'ccd': 17, 'expTime': 30.0}
id dataRef.dataId = {'taiObs': '2013-06-17', 'pointing': 533, 'visit': 903338, 'dateObs': '2013-06-17', 'filter': 'HSC-R', 'field': 'STRIPE82L', 'ccd': 25, 'expTime': 30.0}
id dataRef.dataId = {'taiObs': '2013-06-17', 'pointing': 533, 'visit': 903338, 'dateObs': '2013-06-17', 'filter': 'HSC-R', 'field': 'STRIPE82L', 'ccd': 18, 'expTime': 30.0}
id dataRef.dataId = {'taiObs': '2013-06-17', 'pointing': 533, 'visit': 903342, 'dateObs': '2013-06-17', 'filter': 'HSC-R', 'field': 'STRIPE82L', 'ccd': 100, 'expTime': 30.0}
id dataRef.dataId = {'taiObs': '2013-06-17', 'pointing': 533, 'visit': 903342, 'dateObs': '2013-06-17', 'filter': 'HSC-R', 'field': 'STRIPE82L', 'ccd': 10, 'expTime': 30.0}
id dataRef.dataId = {'taiObs': '2013-06-17', 'pointing': 533, 'visit': 903342, 'dateObs': '2013-06-17', 'filter': 'HSC-R', 'field': 'STRIPE82L', 'ccd': 4, 'expTime': 30.0}
id dataRef.dataId = {'taiObs': '2013-06-17', 'pointing': 533, 'visit': 903344, 'dateObs': '2013-06-17', 'filter': 'HSC-R', 'field': 'STRIPE82L', 'ccd': 11, 'expTime': 30.0}
id dataRef.dataId = {'taiObs': '2013-06-17', 'pointing': 533, 'visit': 903344, 'dateObs': '2013-06-17', 'filter': 'HSC-R', 'field': 'STRIPE82L', 'ccd': 5, 'expTime': 30.0}
id dataRef.dataId = {'taiObs': '2013-06-17', 'pointing': 533, 'visit': 903344, 'dateObs': '2013-06-17', 'filter': 'HSC-R', 'field': 'STRIPE82L', 'ccd': 0, 'expTime': 30.0}
id dataRef.dataId = {'taiObs': '2013-06-17', 'pointing': 533, 'visit': 903346, 'dateObs': '2013-06-17', 'filter': 'HSC-R', 'field': 'STRIPE82L', 'ccd': 12, 'expTime': 30.0}
id dataRef.dataId = {'taiObs': '2013-06-17', 'pointing': 533, 'visit': 903346, 'dateObs': '2013-06-17', 'filter': 'HSC-R', 'field': 'STRIPE82L', 'ccd': 6, 'expTime': 30.0}
id dataRef.dataId = {'taiObs': '2013-06-17', 'pointing': 533, 'visit': 903346, 'dateObs': '2013-06-17', 'filter': 'HSC-R', 'field': 'STRIPE82L', 'ccd': 1, 'expTime': 30.0}
id dataRef.dataId = {'taiObs': '2013-11-02', 'pointing': 671, 'visit': 903986, 'dateObs': '2013-11-02', 'filter': 'HSC-I', 'field': 'STRIPE82L', 'ccd': 23, 'expTime': 30.0}
id dataRef.dataId = {'taiObs': '2013-11-02', 'pointing': 671, 'visit': 903986, 'dateObs': '2013-11-02', 'filter': 'HSC-I', 'field': 'STRIPE82L', 'ccd': 22, 'expTime': 30.0}
id dataRef.dataId = {'taiObs': '2013-11-02', 'pointing': 671, 'visit': 903986, 'dateObs': '2013-11-02', 'filter': 'HSC-I', 'field': 'STRIPE82L', 'ccd': 16, 'expTime': 30.0}
id dataRef.dataId = {'taiObs': '2013-11-02', 'pointing': 671, 'visit': 903986, 'dateObs': '2013-11-02', 'filter': 'HSC-I', 'field': 'STRIPE82L', 'ccd': 100, 'expTime': 30.0}
id dataRef.dataId = {'taiObs': '2013-11-02', 'pointing': 671, 'visit': 903988, 'dateObs': '2013-11-02', 'filter': 'HSC-I', 'field': 'STRIPE82L', 'ccd': 24, 'expTime': 30.0}
id dataRef.dataId = {'taiObs': '2013-11-02', 'pointing': 671, 'visit': 903988, 'dateObs': '2013-11-02', 'filter': 'HSC-I', 'field': 'STRIPE82L', 'ccd': 23, 'expTime': 30.0}
id dataRef.dataId = {'taiObs': '2013-11-02', 'pointing': 671, 'visit': 903988, 'dateObs': '2013-11-02', 'filter': 'HSC-I', 'field': 'STRIPE82L', 'ccd': 17, 'expTime': 30.0}
id dataRef.dataId = {'taiObs': '2013-11-02', 'pointing': 671, 'visit': 903988, 'dateObs': '2013-11-02', 'filter': 'HSC-I', 'field': 'STRIPE82L', 'ccd': 16, 'expTime': 30.0}
id dataRef.dataId = {'taiObs': '2013-11-02', 'pointing': 671, 'visit': 903990, 'dateObs': '2013-11-02', 'filter': 'HSC-I', 'field': 'STRIPE82L', 'ccd': 25, 'expTime': 30.0}
id dataRef.dataId = {'taiObs': '2013-11-02', 'pointing': 671, 'visit': 903990, 'dateObs': '2013-11-02', 'filter': 'HSC-I', 'field': 'STRIPE82L', 'ccd': 18, 'expTime': 30.0}
id dataRef.dataId = {'taiObs': '2013-11-02', 'pointing': 671, 'visit': 904010, 'dateObs': '2013-11-02', 'filter': 'HSC-I', 'field': 'STRIPE82L', 'ccd': 100, 'expTime': 30.0}
id dataRef.dataId = {'taiObs': '2013-11-02', 'pointing': 671, 'visit': 904010, 'dateObs': '2013-11-02', 'filter': 'HSC-I', 'field': 'STRIPE82L', 'ccd': 10, 'expTime': 30.0}
id dataRef.dataId = {'taiObs': '2013-11-02', 'pointing': 671, 'visit': 904010, 'dateObs': '2013-11-02', 'filter': 'HSC-I', 'field': 'STRIPE82L', 'ccd': 4, 'expTime': 30.0}
id dataRef.dataId = {'taiObs': '2013-11-02', 'pointing': 671, 'visit': 904014, 'dateObs': '2013-11-02', 'filter': 'HSC-I', 'field': 'STRIPE82L', 'ccd': 12, 'expTime': 30.0}
id dataRef.dataId = {'taiObs': '2013-11-02', 'pointing': 671, 'visit': 904014, 'dateObs': '2013-11-02', 'filter': 'HSC-I', 'field': 'STRIPE82L', 'ccd': 6, 'expTime': 30.0}
id dataRef.dataId = {'taiObs': '2013-11-02', 'pointing': 671, 'visit': 904014, 'dateObs': '2013-11-02', 'filter': 'HSC-I', 'field': 'STRIPE82L', 'ccd': 1, 'expTime': 30.0}
But once I try to actually run it, I get the following error:
(lsst) jahumada@leftraru2:/mnt/flock/jahumada/Paula$ processCcd.py DATA --rerun processCcdOutputs --id
root INFO: Loading config overrride file '/home/jahumada/.conda/envs/lsst/opt/lsst/obs_subaru/config/processCcd.py'
Cannot import lsst.meas.extensions.convolved (No module named convolved): disabling convolved flux measurements
root INFO: Loading config overrride file '/home/jahumada/.conda/envs/lsst/opt/lsst/obs_subaru/config/hsc/processCcd.py'
root INFO: Running: /home/jahumada/.conda/envs/lsst/opt/lsst/pipe_tasks/bin/processCcd.py DATA --rerun processCcdOutputs --id
processCcd INFO: Processing {'taiObs': '2013-06-17', 'pointing': 533, 'visit': 903334, 'dateObs': '2013-06-17', 'filter': 'HSC-R', 'field': 'STRIPE82L', 'ccd': 23, 'expTime': 30.0}
processCcd.isr INFO: Performing ISR on sensor {'taiObs': '2013-06-17', 'pointing': 533, 'visit': 903334, 'dateObs': '2013-06-17', 'filter': 'HSC-R', 'field': 'STRIPE82L', 'ccd': 23, 'expTime': 30.0}
processCcd FATAL: Failed on dataId={'taiObs': '2013-06-17', 'pointing': 533, 'visit': 903334, 'dateObs': '2013-06-17', 'filter': 'HSC-R', 'field': 'STRIPE82L', 'ccd': 23, 'expTime': 30.0}: disk I/O error
Traceback (most recent call last):
File "/home/jahumada/.conda/envs/lsst/opt/lsst/pipe_base/python/lsst/pipe/base/cmdLineTask.py", line 347, in __call__
result = task.run(dataRef, **kwargs)
File "/home/jahumada/.conda/envs/lsst/opt/lsst/pipe_base/python/lsst/pipe/base/timer.py", line 121, in wrapper
res = func(self, *args, **keyArgs)
File "/home/jahumada/.conda/envs/lsst/opt/lsst/pipe_tasks/python/lsst/pipe/tasks/processCcd.py", line 181, in run
exposure = self.isr.runDataRef(sensorRef).exposure
File "/home/jahumada/.conda/envs/lsst/opt/lsst/obs_subaru/python/lsst/obs/subaru/isr.py", line 230, in runDataRef
defects = sensorRef.get("defects", immediate=True)
File "/home/jahumada/.conda/envs/lsst/opt/lsst/daf_persistence/python/lsst/daf/persistence/butlerSubset.py", line 198, in get
return self.butlerSubset.butler.get(datasetType, self.dataId, **rest)
File "/home/jahumada/.conda/envs/lsst/opt/lsst/daf_persistence/python/lsst/daf/persistence/butler.py", line 699, in get
location = self._locate(datasetType, dataId, write=False)
File "/home/jahumada/.conda/envs/lsst/opt/lsst/daf_persistence/python/lsst/daf/persistence/butler.py", line 649, in _locate
location = repoData.repo.map(datasetType, dataId, write=write)
File "/home/jahumada/.conda/envs/lsst/opt/lsst/daf_persistence/python/lsst/daf/persistence/repository.py", line 183, in map
loc = self._mapper.map(*args, **kwargs)
File "/home/jahumada/.conda/envs/lsst/opt/lsst/obs_subaru/python/lsst/obs/hsc/hscMapper.py", line 156, in map
location = super(HscMapper, self).map(datasetType, copyId, write=write)
File "/home/jahumada/.conda/envs/lsst/opt/lsst/daf_persistence/python/lsst/daf/persistence/mapper.py", line 144, in map
return func(self.validate(dataId), write)
File "/home/jahumada/.conda/envs/lsst/opt/lsst/obs_base/python/lsst/obs/base/cameraMapper.py", line 688, in map_defects
defectFitsPath = self._defectLookup(dataId=dataId)
File "/home/jahumada/.conda/envs/lsst/opt/lsst/obs_base/python/lsst/obs/base/cameraMapper.py", line 985, in _defectLookup
(ccdVal, taiObs))
File "/home/jahumada/.conda/envs/lsst/opt/lsst/daf_persistence/python/lsst/daf/persistence/registries.py", line 365, in executeQuery
c = self.conn.execute(cmd, values)
OperationalError: disk I/O error
Its an Input/Output error, but I have no idea why is wrong or how it could be solved, maybe there is something that I missed along the way!
Just in case, these are the commands I use to get to the point of using processccd.py
jahumada@leftraru2:/mnt/flock/jahumada/Paula$ source activate lsst
(lsst) jahumada@leftraru2:/mnt/flock/jahumada/Paula$ source eups-setups.sh
(lsst) jahumada@leftraru2:/mnt/flock/jahumada/Paula$ setup lsst_distrib
(lsst) jahumada@leftraru2:/mnt/flock/jahumada/Paula$ ml git-lfs
(lsst) jahumada@leftraru2:/mnt/flock/jahumada/Paula$ git lfs install
Git LFS initialized.
I apply the git-lfs related commands every time I log in since I’m using the lsst pipeline in a cluster called ‘leftraru’ from NLHPC.
Thanks a lot!