Trouble with daf_persistence 12.1 build at NERSC

Support
#1

I was trying to build the sims_2_3_1 tag of lsst_apps (created by @danielsf for the Twinkles work) at NERSC on Edison, we did this successfully at SLAC last month, but I ran into an error at NERSC:

   [ 46/70 ]  daf_persistence 12.1 ...                                             

***** error: from /global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/EupsBuildDir/Linux64/daf_persistence-12.1/build.log:                               
  File "/global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/Linux64/pyyaml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 599, in construct_python_object_apply                                                                        
    instance = self.make_python_instance(suffix, node, args, kwds, newobj)        
  File "/global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/Linux64/pyyaml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 540, in make_python_instance                                                                                 
    cls = self.find_python_name(suffix, node.start_mark)                          
  File "/global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/Linux64/pyyaml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 513, in find_python_name   
    module.__name__), mark)                                                       
ConstructorError: while constructing a Python object                              
cannot find 'ImgMapper' in the module '__main__'                                  
  in "./repositoryCfg.yaml", line 3, column 10                                    

----------------------------------------------------------------------
Ran 4 tests in 0.151s                                                 

FAILED (errors=1)
The following tests failed:
/global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/EupsBuildDir/Linux64/daf_persistence-12.1/daf_persistence-12.1/tests/.tests/butlerPickle.py.failed
1 tests failed
scons: *** [checkTestStatus] Error 1
scons: building terminated because of errors.
+ exit -4
eups distrib: Failed to build daf_persistence-12.1.eupspkg: Command:
        source /global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/eups/bin/setups.sh; export EUPS_PATH=/global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1; (/global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/EupsBuildDir/Linux64/daf_persistence-12.1/build.sh) >> /global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/EupsBuildDir/Linux64/daf_persistence-12.1/build.log 2>&1 4>/global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/EupsBuildDir/Linux64/daf_persistence-12.1/build.msg
exited with code 252

Poking into the build.log I see

Failed test output:
tests/butlerPickle.py

/global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/Linux64/miniconda2/
3.19.0.lsst4/lib/python2.7/site-packages/astropy/config/configuration.py:687: Conf
igurationMissingWarning: Configuration defaults will be used due to OSError:Could 
not find unix home directory to search for astropy config dir on None             
  warn(ConfigurationMissingWarning(msg))                                          
E...                                                                              
======================================================================            
ERROR: testIO (__main__.ButlerPickleTestCase)                                     
----------------------------------------------------------------------            
Traceback (most recent call last):                                                
  File "tests/butlerPickle.py", line 50, in setUp                                 
    self.butler = dafPersist.Butler(root='.', mapper=MinMapper())                 
  File "/global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/EupsBuildDi
r/Linux64/daf_persistence-12.1/daf_persistence-12.1/python/lsst/daf/persistence/bu
tler.py", line 285, in __init__                                                   
    self._addRepo(args, inout='out', defaultMapper=defaultMapper, butlerIOParents=
butlerIOParents)                                                                  
  File "/global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/EupsBuildDi
r/Linux64/daf_persistence-12.1/daf_persistence-12.1/python/lsst/daf/persistence/bu
tler.py", line 345, in _addRepo                                                   
    cfg = Storage.getRepositoryCfg(args.cfgRoot)                                  
  File "/global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/EupsBuildDi
r/Linux64/daf_persistence-12.1/daf_persistence-12.1/python/lsst/daf/persistence/st
orage.py", line 75, in getRepositoryCfg                                           
    ret = Storage.storages[parseRes.scheme].getRepositoryCfg(uri)                 
  File "/global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/EupsBuildDi
r/Linux64/daf_persistence-12.1/daf_persistence-12.1/python/lsst/daf/persistence/po
sixStorage.py", line 78, in getRepositoryCfg                                      
    repositoryCfg = PosixStorage._getRepositoryCfg(uri)                           
  File "/global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/EupsBuildDi
r/Linux64/daf_persistence-12.1/daf_persistence-12.1/python/lsst/daf/persistence/po
sixStorage.py", line 71, in _getRepositoryCfg                                     
    repositoryCfg = yaml.load(f)                                                  
  File "/global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/Linux64/pyy
aml/3.11.lsst1+2/lib/python/yaml/__init__.py", line 71, in load                   
    return loader.get_single_data()                                               
  File "/global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/Linux64/pyy
aml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 39, in get_single_data     
    return self.construct_document(node)                                          
  File "/global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/Linux64/pyy
aml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 43, in construct_document  
    data = self.construct_object(node)                                            
  File "/global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/Linux64/pyy
aml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 88, in construct_object    
    data = constructor(self, node)                                                
  File "/global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/EupsBuildDi
r/Linux64/daf_persistence-12.1/daf_persistence-12.1/python/lsst/daf/persistence/re
positoryCfg.py", line 61, in v1Constructor                                        
    d = loader.construct_mapping(node)                                            
  File "/global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/Linux64/pyy
aml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 208, in construct_mapping  
    return BaseConstructor.construct_mapping(self, node, deep=deep)               
  File "/global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/Linux64/pyy
aml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 133, in construct_mapping  
    value = self.construct_object(value_node, deep=deep)                          
  File "/global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/Linux64/pyy
aml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 90, in construct_object    
    data = constructor(self, tag_suffix, node)                                    
  File "/global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/Linux64/pyy
aml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 610, in construct_python_ob
ject_new                                                                          
    return self.construct_python_object_apply(suffix, node, newobj=True)          
  File "/global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/Linux64/pyy
aml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 599, in construct_python_ob
ject_apply                                                                        
    instance = self.make_python_instance(suffix, node, args, kwds, newobj)        
  File "/global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/Linux64/pyy
aml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 540, in make_python_instanc
e                                                                                 
    cls = self.find_python_name(suffix, node.start_mark)                          
  File "/global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/Linux64/pyy
aml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 513, in find_python_name
    module.__name__), mark)
ConstructorError: while constructing a Python object
cannot find 'ImgMapper' in the module '__main__'
  in "./repositoryCfg.yaml", line 3, column 10

----------------------------------------------------------------------
Ran 4 tests in 0.151s

FAILED (errors=1)
The following tests failed:
/global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/EupsBuildDir/Linux6
4/daf_persistence-12.1/daf_persistence-12.1/tests/.tests/butlerPickle.py.failed
1 tests failed
scons: *** [checkTestStatus] Error 1
scons: building terminated because of errors.
+ exit -4

I’m trying again with v12_1 for fun - but I have no reason to believe the result will change. I see something similar about
Configuration defaults will be used due to OSError:Could not find unix home directory to search for astropy config dir on None
here


but I don’t know how that was resolved. Can someone offer any suggestions?
Thanks,
Heather

#2

Hi Heather,

What does the test fail log say? You should find it in
$DAF_PERSISTENCE_DIR/tests/.tests/butlerPickle.py.failed

#3

Hi Scott,
Does this help?

more EupsBuildDir/Linux64/daf_persistence-12.1/daf_persistence-12.1/tests/.tests/butlerPickle.py.failed                                                             
tests/butlerPickle.py                                                             

/global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/Linux64/miniconda2/
3.19.0.lsst4/lib/python2.7/site-packages/astropy/config/configuration.py:687: Conf
igurationMissingWarning: Configuration defaults will be used due to OSError:Could 
not find unix home directory to search for astropy config dir on None             
  warn(ConfigurationMissingWarning(msg))                                          
E...                                                                              
======================================================================            
ERROR: testIO (__main__.ButlerPickleTestCase)                                     
----------------------------------------------------------------------            
Traceback (most recent call last):                                                
  File "tests/butlerPickle.py", line 50, in setUp                                 
    self.butler = dafPersist.Butler(root='.', mapper=MinMapper())                 
  File "/global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/EupsBuildDi
r/Linux64/daf_persistence-12.1/daf_persistence-12.1/python/lsst/daf/persistence/bu
tler.py", line 285, in __init__                                                   
    self._addRepo(args, inout='out', defaultMapper=defaultMapper, butlerIOParents=
butlerIOParents)                                                                  
  File "/global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/EupsBuildDi
r/Linux64/daf_persistence-12.1/daf_persistence-12.1/python/lsst/daf/persistence/bu
tler.py", line 345, in _addRepo                                                   
    cfg = Storage.getRepositoryCfg(args.cfgRoot)                                  
  File "/global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/EupsBuildDi
r/Linux64/daf_persistence-12.1/daf_persistence-12.1/python/lsst/daf/persistence/st
orage.py", line 75, in getRepositoryCfg                                           
    ret = Storage.storages[parseRes.scheme].getRepositoryCfg(uri)                 
  File "/global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/EupsBuildDi
r/Linux64/daf_persistence-12.1/daf_persistence-12.1/python/lsst/daf/persistence/po
sixStorage.py", line 78, in getRepositoryCfg                                      
    repositoryCfg = PosixStorage._getRepositoryCfg(uri)                           
  File "/global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/EupsBuildDi
r/Linux64/daf_persistence-12.1/daf_persistence-12.1/python/lsst/daf/persistence/po
sixStorage.py", line 71, in _getRepositoryCfg                                     
    repositoryCfg = yaml.load(f)                                                  
  File "/global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/Linux64/pyy
aml/3.11.lsst1+2/lib/python/yaml/__init__.py", line 71, in load                   
    return loader.get_single_data()                                               
  File "/global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/Linux64/pyy
aml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 39, in get_single_data     
    return self.construct_document(node)                                          
  File "/global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/Linux64/pyy
aml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 43, in construct_document  
    data = self.construct_object(node)                                            
  File "/global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/Linux64/pyy
aml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 88, in construct_object    
    data = constructor(self, node)                                                
  File "/global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/EupsBuildDi
r/Linux64/daf_persistence-12.1/daf_persistence-12.1/python/lsst/daf/persistence/re
positoryCfg.py", line 61, in v1Constructor                                        
    d = loader.construct_mapping(node)                                            
  File "/global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/Linux64/pyy
aml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 208, in construct_mapping  
    return BaseConstructor.construct_mapping(self, node, deep=deep)               
  File "/global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/Linux64/pyy
aml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 133, in construct_mapping  
    value = self.construct_object(value_node, deep=deep)                          
  File "/global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/Linux64/pyy
aml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 90, in construct_object    
    data = constructor(self, tag_suffix, node)                                    
  File "/global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/Linux64/pyy
aml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 610, in construct_python_ob
ject_new                                                                          
    return self.construct_python_object_apply(suffix, node, newobj=True)          
  File "/global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/Linux64/pyy
aml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 599, in construct_python_ob
ject_apply
    instance = self.make_python_instance(suffix, node, args, kwds, newobj)
  File "/global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/Linux64/pyy
aml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 540, in make_python_instanc
e
    cls = self.find_python_name(suffix, node.start_mark)
  File "/global/common/edison/contrib/lsst/lsstDM/w.2016.40-sims_2.3.1/Linux64/pyy
aml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 513, in find_python_name
    module.__name__), mark)
ConstructorError: while constructing a Python object
cannot find 'ImgMapper' in the module '__main__'
  in "./repositoryCfg.yaml", line 3, column 10

----------------------------------------------------------------------
Ran 4 tests in 0.151s

FAILED (errors=1)
#4

and interestingly, I tried a fresh build using v12_1 and it got beyond the daf_persistence 12.1 build…
Which is great - but I’m wondering if that sims_2_3_1 tag is either not reproducible or is it just not working in the NERSC case, while it did at SLAC?

#5

(Sorry; I didn’t see your responses yesterday)

Did you try re-starting the sims_2_3_1 build after the 12_1 build completed? If daf_persistence is there because of 12_1, sims_2_3_1 might not need to rebuild it. I know that’s not an answer, but I’m curious to see what happens.

#6

I actually tend to create completely separate installations for each DMstack release or build in a new directory (but in the same directory area) - perhaps that’s archaic! :slight_smile: I could try the test you suggest though and see what happens, I still have the sims_2_3_1 install set up, it’s just not complete. I’ll see if I can do that and report back.

#7

I wanted to mention that I just reproduced this exact same error with daf_persistence 13.0 on a CentOS 7.3.1611 system. The error occurred when the permissions on my home directory were 700. I changed the permissions to 755 since there was the complaint

OSError:Could not find unix home directory

The build subsequently succeeded after that change.

2 Likes