Making detrending data for HSC

hsc
Tags: #<Tag:0x00007f61a37e6ec0>

(Surhud More) #1

Is there a tutorial which I can access to make detrending data for HSC. The ci_hsc package directly has all the calibration files already created and the “Getting started” page directly links those calibrations instead of telling how to create them.

I am using LSST v17.0.1 and trying to follow the HSC tutorial. When constructing bias frames using constructBias command. It complains about No module named lsst.obs.subaru.isr. I have already performed: setup -v obs_subaru. I do not see a file related to isr here:

lsst_stack/stack/miniconda3-4.5.4-fcd27eb/Linux64/obs_subaru/17.0.1/python/lsst/obs/subaru/

An ls command on that directory shows:

arcturus.py    _crosstalk.so     filterFraction.py  __init__.py  SConscript
_crosstalk.cc  crosstalkYagi.py  gen3               masking.py   strayLight
crosstalk.py   dither.py         ingest.py          __pycache__  version.py

Could some one give me pointers regarding this?


(Paul Price) #2

The HSC tutorial should be fairly accurate here, despite being out-of-date.

The only reason I can think of for the No module named lsst.obs.subaru.isr error is that perhaps you are re-using a rerun from an old version, so it’s trying to read an old configuration that won’t work with the new version? Perhaps you could post the command you’re running along with the output?


(Surhud More) #3

Thanks Paul!

Here is what I am doing:

# Source for enabling gcc-6
source devtoolset-6/enable

# Source LSST
source loadLSST.bash

# Setup pipe_tasks, pipe_drivers, obs_subaru 
setup pipe_tasks
setup pipe_drivers
setup obs_subaru

cd lsst_stack
mkdir DATA
echo "lsst.obs.hsc.HscMapper" > DATA/_mapper
installTransmissionCurves.py DATA
ingestImages.py DATA rawdata/*.fits --mode=link --create


# Test using sqlite that the files have been ingested properly
#cd DATA
#sqlite3 registry.sqlite3
#sqlite> .header on
#sqlite> select visit, filter, field, count(visit)
#  ...> from raw
#   ...> group by visit, filter, field;
# This properly shows all the ingested HSC-G images, DOMEFLAT, BIAS and DARK files.

constructBias.py DATA --calib DATA/CALIB --rerun calib --id visit=132732..132740:2 

This last command fails giving the following output:

root INFO: Loading config overrride file '/mnt/home/faculty/csurhud/lsst_stack/stack/miniconda3-4.5.4-fcd27eb/Linux64/obs_subaru/17.0.1/config/hsc/bias.py'
Traceback (most recent call last):
  File "/mnt/home/faculty/csurhud/lsst_stack/stack/miniconda3-4.5.4-fcd27eb/Linux64/pipe_drivers/17.0.1/bin/constructBias.py", line 3, in <module>
    BiasTask.parseAndSubmit()
  File "/mnt/home/faculty/csurhud/lsst_stack/stack/miniconda3-4.5.4-fcd27eb/Linux64/ctrl_pool/17.0.1/python/lsst/ctrl/pool/parallel.py", line 437, in parseAndSubmit
    **kwargs)
  File "/mnt/home/faculty/csurhud/lsst_stack/stack/miniconda3-4.5.4-fcd27eb/Linux64/ctrl_pool/17.0.1/python/lsst/ctrl/pool/parallel.py", line 346, in parse_args
    args.parent = self._parent.parse_args(config, args=leftover, **kwargs)
  File "/mnt/home/faculty/csurhud/lsst_stack/stack/miniconda3-4.5.4-fcd27eb/Linux64/pipe_drivers/17.0.1/python/lsst/pipe/drivers/constructCalibs.py", line 311, in parse_args
    namespace = ArgumentParser.parse_args(self, *args, **kwargs)
  File "/mnt/home/faculty/csurhud/lsst_stack/stack/miniconda3-4.5.4-fcd27eb/Linux64/pipe_base/17.0.1/python/lsst/pipe/base/argumentParser.py", line 639, in parse_args
    self._applyInitialOverrides(namespace)
  File "/mnt/home/faculty/csurhud/lsst_stack/stack/miniconda3-4.5.4-fcd27eb/Linux64/pipe_base/17.0.1/python/lsst/pipe/base/argumentParser.py", line 837, in _applyInitialOverrides
    namespace.config.load(filePath)
  File "/mnt/home/faculty/csurhud/lsst_stack/stack/miniconda3-4.5.4-fcd27eb/Linux64/pex_config/17.0.1/python/lsst/pex/config/config.py", line 938, in load
    self.loadFromStream(stream=code, root=root)
  File "/mnt/home/faculty/csurhud/lsst_stack/stack/miniconda3-4.5.4-fcd27eb/Linux64/pex_config/17.0.1/python/lsst/pex/config/config.py", line 976, in loadFromStream
    exec(stream, {}, local)
  File "/mnt/home/faculty/csurhud/lsst_stack/stack/miniconda3-4.5.4-fcd27eb/Linux64/obs_subaru/17.0.1/config/hsc/bias.py", line 4, in <module>
    from lsst.obs.subaru.isr import SubaruIsrTask
ModuleNotFoundError: No module named 'lsst.obs.subaru.isr'

For confirming that I do not have any other things going on, I deleted my previous DATA directory and reran the above commands again to confirm that I can reproduce this error.


(Meredith Rawls) #4

There is indeed no isr.py in obs_subaru anymore per the recent ISR refactor. This suggests you have an older version of obs_subaru which still refers to SubaruIsrTask when it should be using the standard IsrTask from ip_isr.

Please try updating the obs_subaru repo you are setting up via git pull, or just use the one that is already included when you do source loadLSST.bash (it should already include self-consistent versions of pipe_tasks, pipe_drivers, and obs_subaru without you having to set them up separately).


(Surhud More) #5

I have not installed any other obs_subaru package other than from sourcing loadLSST.bash. This is the first time the LSST pipeline was installed on the cluster. I have copy pasted the exact set of commands I ran.

Is there a way that I can output the version of the obs_subaru package?


(Surhud More) #6

I checked with:

eups list obs_subaru

This outputs

17.0.1     	v17_0_1 o_latest current setup

But,

cat lsst_stack/stack/miniconda3-4.5.4-fcd27eb/Linux64/obs_subaru/17.0.1/config/hsc/bias.py

still shows references to the lsst.obs.subaru.isr.

Am I doing something wrong in the installation? I followed the standard LSST pipeline installation steps outlined here using newinstall.

In this case, how to ask eups to install the latest version of obs_subaru?


(Surhud More) #7

I still face the same problem with the obs_subaru package not being up to date when I did a completely new setup of the pipeline:

mkdir -p lsst_stacknew
cd lsst_stacknew/
ls
source ../soft/devtoolset-6/enable 
ls
curl -OL https://raw.githubusercontent.com/lsst/lsst/17.0.1/scripts/newinstall.sh
bash newinstall.sh -ct
ls
source loadLSST.bash 
ls
eups distrib install -t v17_0_1 lsst_distrib

cd stack/miniconda3-4.5.4-fcd27eb/Linux64/obs_subaru/17.0.1/config/hsc/
cat bias.py 

A more detailed log is here:

[csurhud@ln2 ~]$ mkdir -p lsst_stacknew
[csurhud@ln2 ~]$ cd lsst_stacknew/            
[csurhud@ln2 lsst_stacknew]$ source ../soft/devtoolset-6/enable 
[csurhud@ln2 lsst_stacknew]$ curl -OL https://raw.githubusercontent.com/lsst/lsst/17.0.1/scripts/newinstall.sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 30780  100 30780    0     0  34233      0 --:--:-- --:--:-- --:--:-- 34200
[csurhud@ln2 lsst_stacknew]$ bash newinstall.sh -ct

LSST Software Stack Builder
=======================================================================

######################################################################## 100.0%
!!! This script differs from the official version on the distribution
server.  If this is not intentional, get the current version from here:
https://raw.githubusercontent.com/lsst/lsst/master/scripts/newinstall.sh
Detected git version 2.9.5. OK.

LSST stack requires Python >= 3.6; you seem to have Python 2.7.5 on your
path (/usr/bin/python).

In addition to Python >= 3.6, some LSST packages depend on recent versions
of numpy, matplotlib, and scipy.  If you do not have all of these, the
installation may fail.  Using the Miniconda Python distribution will ensure
all these are set up.

Would you like us to install the Miniconda Python distribution (if unsure,
say yes)?yes
::: Deploying Miniconda3-4.5.4-Linux-x86_64.sh
######################################################################## 100.0%
PREFIX=/mnt/home/faculty/csurhud/lsst_stacknew/python/miniconda3-4.5.4
installing: python-3.6.5-hc3d631a_2 ...
Python 3.6.5 :: Anaconda, Inc.
installing: ca-certificates-2018.03.07-0 ...
installing: conda-env-2.6.0-h36134e3_1 ...
installing: libgcc-ng-7.2.0-hdf63c60_3 ...
installing: libstdcxx-ng-7.2.0-hdf63c60_3 ...
installing: libffi-3.2.1-hd88cf55_4 ...
installing: ncurses-6.1-hf484d3e_0 ...
installing: openssl-1.0.2o-h20670df_0 ...
installing: tk-8.6.7-hc745277_3 ...
installing: xz-5.2.4-h14c3975_4 ...
installing: yaml-0.1.7-had09818_2 ...
installing: zlib-1.2.11-ha838bed_2 ...
installing: libedit-3.1.20170329-h6b74fdf_2 ...
installing: readline-7.0-ha6073c6_4 ...
installing: sqlite-3.23.1-he433501_0 ...
installing: asn1crypto-0.24.0-py36_0 ...
installing: certifi-2018.4.16-py36_0 ...
installing: chardet-3.0.4-py36h0f667ec_1 ...
installing: idna-2.6-py36h82fb2a8_1 ...
installing: pycosat-0.6.3-py36h0a5515d_0 ...
installing: pycparser-2.18-py36hf9f622e_1 ...
installing: pysocks-1.6.8-py36_0 ...
installing: ruamel_yaml-0.15.37-py36h14c3975_2 ...
installing: six-1.11.0-py36h372c433_1 ...
installing: cffi-1.11.5-py36h9745a5d_0 ...
installing: setuptools-39.2.0-py36_0 ...
installing: cryptography-2.2.2-py36h14c3975_0 ...
installing: wheel-0.31.1-py36_0 ...
installing: pip-10.0.1-py36_0 ...
installing: pyopenssl-18.0.0-py36_0 ...
installing: urllib3-1.22-py36hbe7ace6_0 ...
installing: requests-2.18.4-py36he2e5f8d_1 ...
installing: conda-4.5.4-py36_0 ...
installation finished.
WARNING:
    You currently have a PYTHONPATH environment variable set. This may cause
    unexpected behavior when running the Python interpreter in Miniconda3.
    For best results, please verify that your PYTHONPATH only points to
    directories of packages that are compatible with the Python interpreter
    in Miniconda3: /mnt/home/faculty/csurhud/lsst_stacknew/python/miniconda3-4.5.4
######################################################################## 100.0%
Solving environment: done


==> WARNING: A newer version of conda exists. <==
  current version: 4.5.4
  latest version: 4.6.14

Please update conda by running

    $ conda update -n base conda



Downloading and Extracting Packages
cycler-0.10.0        |   13 KB | ####################################### | 100% 
requests-2.19.1      |   96 KB | ####################################### | 100% 
libxml2-2.9.8        |  2.0 MB | ####################################### | 100% 
libgfortran-ng-7.2.0 |  1.2 MB | ####################################### | 100% 
dbus-1.13.2          |  554 KB | ####################################### | 100% 
python-dateutil-2.7. |  260 KB | ####################################### | 100% 
qt-5.9.5             | 84.9 MB | ####################################### | 100% 
pluggy-0.6.0         |   23 KB | ####################################### | 100% 
more-itertools-4.2.0 |   78 KB | ####################################### | 100% 
atomicwrites-1.1.5   |   10 KB | ####################################### | 100% 
libpng-1.6.34        |  334 KB | ####################################### | 100% 
gst-plugins-base-1.1 |  6.3 MB | ####################################### | 100% 
urllib3-1.23         |  152 KB | ####################################### | 100% 
tornado-5.0.2        |  644 KB | ####################################### | 100% 
cython-0.28.3        |  3.4 MB | ####################################### | 100% 
kiwisolver-1.0.1     |   84 KB | ####################################### | 100% 
freetype-2.8         |  804 KB | ####################################### | 100% 
pytest-3.6.2         |  300 KB | ####################################### | 100% 
future-0.16.0        |  634 KB | ####################################### | 100% 
numexpr-2.6.5        |  189 KB | ####################################### | 100% 
pyyaml-3.12          |  159 KB | ####################################### | 100% 
bottleneck-1.2.1     |  127 KB | ####################################### | 100% 
pytest-remotedata-0. |   12 KB | ####################################### | 100% 
scikit-learn-0.19.1  |  5.2 MB | ####################################### | 100% 
attrs-18.1.0         |   43 KB | ####################################### | 100% 
fontconfig-2.12.6    |  283 KB | ####################################### | 100% 
sqlite-3.24.0        |  1.8 MB | ####################################### | 100% 
h5py-2.8.0           |  1.1 MB | ####################################### | 100% 
numpy-base-1.14.5    |  4.1 MB | ####################################### | 100% 
pyqt-5.9.2           |  5.9 MB | ####################################### | 100% 
nomkl-3.0            |   48 KB | ####################################### | 100% 
pytest-openfiles-0.3 |    9 KB | ####################################### | 100% 
expat-2.2.5          |  186 KB | ####################################### | 100% 
libopenblas-0.2.20   |  8.8 MB | ####################################### | 100% 
psutil-5.4.6         |  303 KB | ####################################### | 100% 
matplotlib-2.2.2     |  6.6 MB | ####################################### | 100% 
pyparsing-2.2.0      |   96 KB | ####################################### | 100% 
astropy-3.0.3        |  6.8 MB | ####################################### | 100% 
glib-2.56.1          |  5.0 MB | ####################################### | 100% 
libxcb-1.13          |  502 KB | ####################################### | 100% 
pcre-8.42            |  251 KB | ####################################### | 100% 
icu-58.2             | 22.5 MB | ####################################### | 100% 
py-1.5.3             |  135 KB | ####################################### | 100% 
hdf5-1.10.2          |  5.2 MB | ####################################### | 100% 
sip-4.19.8           |  290 KB | ####################################### | 100% 
numpy-1.14.5         |   94 KB | ####################################### | 100% 
pytest-arraydiff-0.2 |   14 KB | ####################################### | 100% 
scipy-1.1.0          | 18.1 MB | ####################################### | 100% 
idna-2.7             |  132 KB | ####################################### | 100% 
blas-1.0             |   48 KB | ####################################### | 100% 
pytest-astropy-0.4.0 |    5 KB | ####################################### | 100% 
gstreamer-1.14.0     |  3.8 MB | ####################################### | 100% 
python-3.6.6         | 29.4 MB | ####################################### | 100% 
pandas-0.23.1        | 11.9 MB | ####################################### | 100% 
sqlalchemy-1.2.8     |  1.6 MB | ####################################### | 100% 
pytz-2018.5          |  232 KB | ####################################### | 100% 
jpeg-9b              |  248 KB | ####################################### | 100% 
pytest-doctestplus-0 |   19 KB | ####################################### | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use:
# > source activate lsst-scipipe-fcd27eb
#
# To deactivate an active environment, use:
# > source deactivate
#

name: lsst-scipipe-fcd27eb
channels:
  - defaults
dependencies:
  - asn1crypto=0.24.0=py36_0
  - astropy=3.0.3=py36h14c3975_2
  - atomicwrites=1.1.5=py36_0
  - attrs=18.1.0=py36_0
  - blas=1.0=openblas
  - bottleneck=1.2.1=py36haac1ea0_0
  - ca-certificates=2018.03.07=0
  - certifi=2018.4.16=py36_0
  - cffi=1.11.5=py36h9745a5d_0
  - chardet=3.0.4=py36h0f667ec_1
  - cryptography=2.2.2=py36h14c3975_0
  - cycler=0.10.0=py36h93f1223_0
  - cython=0.28.3=py36h14c3975_0
  - dbus=1.13.2=h714fa37_1
  - expat=2.2.5=he0dffb1_0
  - fontconfig=2.12.6=h49f89f6_0
  - freetype=2.8=hab7d2ae_1
  - future=0.16.0=py36_1
  - glib=2.56.1=h000015b_0
  - gst-plugins-base=1.14.0=hbbd80ab_1
  - gstreamer=1.14.0=hb453b48_1
  - h5py=2.8.0=py36ha1f6525_0
  - hdf5=1.10.2=hba1933b_1
  - icu=58.2=h9c2bf20_1
  - idna=2.7=py36_0
  - jpeg=9b=h024ee3a_2
  - kiwisolver=1.0.1=py36h764f252_0
  - libedit=3.1.20170329=h6b74fdf_2
  - libffi=3.2.1=hd88cf55_4
  - libgcc-ng=7.2.0=hdf63c60_3
  - libgfortran-ng=7.2.0=hdf63c60_3
  - libopenblas=0.2.20=h9ac9557_7
  - libpng=1.6.34=hb9fc6fc_0
  - libstdcxx-ng=7.2.0=hdf63c60_3
  - libxcb=1.13=h1bed415_1
  - libxml2=2.9.8=h26e45fe_1
  - matplotlib=2.2.2=py36h0e671d2_1
  - more-itertools=4.2.0=py36_0
  - ncurses=6.1=hf484d3e_0
  - nomkl=3.0=0
  - numexpr=2.6.5=py36_nomklhaa809a4_0
  - numpy=1.14.5=py36h28100ab_3
  - numpy-base=1.14.5=py36h2b20989_3
  - openssl=1.0.2o=h20670df_0
  - pandas=0.23.1=py36h637b7d7_0
  - pcre=8.42=h439df22_0
  - pip=10.0.1=py36_0
  - pluggy=0.6.0=py36hb689045_0
  - psutil=5.4.6=py36h14c3975_0
  - py=1.5.3=py36_0
  - pycparser=2.18=py36hf9f622e_1
  - pyopenssl=18.0.0=py36_0
  - pyparsing=2.2.0=py36hee85983_1
  - pyqt=5.9.2=py36h751905a_0
  - pysocks=1.6.8=py36_0
  - pytest=3.6.2=py36_0
  - pytest-arraydiff=0.2=py36_0
  - pytest-astropy=0.4.0=py36_0
  - pytest-doctestplus=0.1.3=py36_0
  - pytest-openfiles=0.3.0=py36_0
  - pytest-remotedata=0.3.0=py36_0
  - python=3.6.6=hc3d631a_0
  - python-dateutil=2.7.3=py36_0
  - pytz=2018.5=py36_0
  - pyyaml=3.12=py36hafb9ca4_1
  - qt=5.9.5=h7e424d6_0
  - readline=7.0=ha6073c6_4
  - requests=2.19.1=py36_0
  - scikit-learn=0.19.1=py36_nomklh27f7947_0
  - scipy=1.1.0=py36_nomklh9d22d0a_0
  - setuptools=39.2.0=py36_0
  - sip=4.19.8=py36hf484d3e_0
  - six=1.11.0=py36h372c433_1
  - sqlalchemy=1.2.8=py36h14c3975_0
  - sqlite=3.24.0=h84994c4_0
  - tk=8.6.7=hc745277_3
  - tornado=5.0.2=py36_0
  - urllib3=1.23=py36_0
  - wheel=0.31.1=py36_0
  - xz=5.2.4=h14c3975_4
  - yaml=0.1.7=had09818_2
  - zlib=1.2.11=ha838bed_2
prefix: /mnt/home/faculty/csurhud/lsst_stacknew/python/miniconda3-4.5.4/envs/lsst-scipipe-fcd27eb

Configured EUPS_PKGROOT: 
Using python at /mnt/home/faculty/csurhud/lsst_stacknew/python/miniconda3-4.5.4/envs/lsst-scipipe-fcd27eb/bin/python to install EUPS
Installing EUPS (2.1.4)...  done.
Creating startup scripts (bash) ... done.
Creating startup scripts (ksh) ... done.
Creating startup scripts (csh) ... done.
Creating startup scripts (zsh) ... done.

Bootstrap complete. To continue installing (and to use) the LSST stack type
one of:

source "/mnt/home/faculty/csurhud/lsst_stacknew/loadLSST.bash"  # for bash
source "/mnt/home/faculty/csurhud/lsst_stacknew/loadLSST.csh"   # for csh
source "/mnt/home/faculty/csurhud/lsst_stacknew/loadLSST.ksh"   # for ksh
source "/mnt/home/faculty/csurhud/lsst_stacknew/loadLSST.zsh"   # for zsh

Individual LSST packages may now be installed with the usual `eups distrib
install` command.  For example, to install the science pipeline elements
of the LSST stack, use:

eups distrib install lsst_apps

Next, read the documentation at:

https://pipelines.lsst.io

and feel free to ask any questions via the LSST Community forum:

https://community.lsst.org/c/support

                                       Thanks!
                                                -- The LSST Software Teams
                                                       http://dm.lsst.org/
[csurhud@ln2 lsst_stacknew]$ source loadLSST.bash 
(lsst-scipipe-fcd27eb) [csurhud@ln2 lsst_stacknew]$ ls
_build  eupsbuild.log  loadLSST.csh  loadLSST.zsh   python
eups    loadLSST.bash  loadLSST.ksh  newinstall.sh  stack
(lsst-scipipe-fcd27eb) [csurhud@ln2 lsst_stacknew]$ eups distrib install -t v17_0_1 lsst_distrib
   [  1/128 ]  apr 1.5.2 (Linux64)                                       done. 
   [  2/128 ]  cfitsio 3360.lsst6 (Linux64)                              done. 
   [  3/128 ]  doxygen 1.8.13.lsst3 (Linux64)                            done. 
   [  4/128 ]  eigen 3.3.4.lsst1 (Linux64)                               done. 
   [  5/128 ]  fftw 3.3.4.lsst2 (Linux64)                                done. 
   [  6/128 ]  gsl 2.4 (Linux64)                                         done. 
   [  7/128 ]  libyaml 0.1.7 (Linux64)                                   done. 
   [  8/128 ]  minuit2 5.34.14 (Linux64)                                 done. 
   [  9/128 ]  mpich 3.2.1 (Linux64)                                     done. 
   [ 10/128 ]  pybind11 2.2.3.lsst2 (Linux64)                            done. 
   [ 11/128 ]  python 0.0.8 (Linux64)                                    done. 
   [ 12/128 ]  starlink_ast lsst-dev-gd6cc4e835a (Linux64)               done. 
   [ 13/128 ]  xpa 2.1.15.lsst3 (Linux64)                                done. 
   [ 14/128 ]  apr_util 1.5.4.lsst1 (Linux64)                            done. 
   [ 15/128 ]  astropy 3.0.3 (Linux64)                                   done. 
   [ 16/128 ]  boost 1.68 (Linux64)                                      done. 
   [ 17/128 ]  mpi 0.0.1+3 (Linux64)                                     done. 
   [ 18/128 ]  numpy 1.14.5 (Linux64)                                    done. 
   [ 19/128 ]  pycodestyle 2.5.0 (Linux64)                               done. 
   [ 20/128 ]  pyflakes 2.1.0 (Linux64)                                  done. 
   [ 21/128 ]  pykg_config 1.3.0+4 (Linux64)                             done. 
   [ 22/128 ]  python_coverage 4.5.1 (Linux64)                           done. 
   [ 23/128 ]  python_execnet 1.4.1.lsst4+1 (Linux64)                    done. 
   [ 24/128 ]  python_future 0.16.0+3 (Linux64)                          done. 
   [ 25/128 ]  python_psutil 5.4.3+2 (Linux64)                           done. 
   [ 26/128 ]  python_py 1.7.0 (Linux64)                                 done. 
   [ 27/128 ]  pyyaml 3.13 (Linux64)                                     done. 
   [ 28/128 ]  requests 2.9.1.lsst1+4 (Linux64)                          done. 
   [ 29/128 ]  scons 3.0.0.lsst1+3 (Linux64)                             done. 
   [ 30/128 ]  sqlalchemy 1.2.16 (Linux64)                               done. 
   [ 31/128 ]  wcslib 5.13.lsst1+3 (Linux64)                             done. 
   [ 32/128 ]  ws4py 0.4.2+3 (Linux64)                                   done. 
   [ 33/128 ]  astrometry_net 0.67.123ff3e.lsst1+10 (Linux64)            done. 
   [ 34/128 ]  coord 1.1.lsst (Linux64)                                  done. 
   [ 35/128 ]  esutil 0.6.2.5.lsst2+1 (Linux64)                          done. 
   [ 36/128 ]  firefly_client lsst-dev-gaed8ac2231 (Linux64)             done. 
   [ 37/128 ]  healpy 1.10.3.lsst2+1 (Linux64)                           done. 
   [ 38/128 ]  log4cxx 0.10.0.lsst7+1 (Linux64)                          done. 
   [ 39/128 ]  matplotlib 2.2.2 (Linux64)                                done. 
   [ 40/128 ]  mpi4py 3.0.0 (Linux64)                                    done. 
   [ 41/128 ]  ndarray 1.5.2+1 (Linux64)                                 done. 
   [ 42/128 ]  pytest 3.6.2.lsst2 (Linux64)                              done. 
   [ 43/128 ]  scipy 1.1.0 (Linux64)                                     done. 
   [ 44/128 ]  treecorr 3.2.3.lsst4 (Linux64)                            done. 
   [ 45/128 ]  fgcm lsst-dev-g6c5abd46ca (Linux64)                       done. 
   [ 46/128 ]  galsim 2.1.4.lsst+1 (Linux64)                             done. 
   [ 47/128 ]  lmfit 0.9.3+9 (Linux64)                                   done. 
   [ 48/128 ]  proxmin lsst-dev-g87d7c0f1bb+1 (Linux64)                  done. 
   [ 49/128 ]  pytest_cov 2.5.1+4 (Linux64)                              done. 
   [ 50/128 ]  pytest_forked 0.2.lsst4+5 (Linux64)                       done. 
   [ 51/128 ]  pytest_session2file 0.1.9+13 (Linux64)                    done. 
   [ 52/128 ]  python_mccabe 0.6.1+13 (Linux64)                          done. 
   [ 53/128 ]  flake8 3.7.5 (Linux64)                                    done. 
   [ 54/128 ]  pytest_xdist 1.20.1.lsst4+5 (Linux64)                     done. 
   [ 55/128 ]  scarlet lsst-dev-g62a773d79d (Linux64)                    done. 
   [ 56/128 ]  pep8_naming 0.8.2 (Linux64)                               done. 
   [ 57/128 ]  pytest_flake8 1.0.4 (Linux64)                             done. 
   [ 58/128 ]  sconsUtils 17.0.1 (Linux64)                               done. 
   [ 59/128 ]  astrometry_net_data 8.0.0.0+62 (Linux64)                  done. 
   [ 60/128 ]  astshim 17.0.1 (Linux64)                                  done. 
   [ 61/128 ]  base 17.0.1 (Linux64)                                     done. 
   [ 62/128 ]  jointcal_cholmod master-g48cb81d145+66 (Linux64)          done. 
   [ 63/128 ]  psfex 17.0.1 (Linux64)                                    done. 
   [ 64/128 ]  verify_metrics 17.0.1 (Linux64)                           done. 
   [ 65/128 ]  pex_exceptions 17.0.1 (Linux64)                           done. 
   [ 66/128 ]  sphgeom 17.0.1 (Linux64)                                  done. 
   [ 67/128 ]  utils 17.0.1 (Linux64)                                    done. 
   [ 68/128 ]  daf_base 17.0.1 (Linux64)                                 done. 
   [ 69/128 ]  geom 17.0.1 (Linux64)                                     done. 
   [ 70/128 ]  log 17.0.1 (Linux64)                                      done. 
   [ 71/128 ]  astro_metadata_translator master-g6e72ed3b81+1 (Linux64)  done. 
   [ 72/128 ]  pex_policy 17.0.1 (Linux64)                               done. 
   [ 73/128 ]  daf_persistence 17.0.1 (Linux64)                          done. 
   [ 74/128 ]  pex_config 17.0.1 (Linux64)                               done. 
   [ 75/128 ]  afw 17.0.1 (Linux64)                                      done. 
   [ 76/128 ]  ctrl_orca 17.0.1 (Linux64)                                done. 
   [ 77/128 ]  ctrl_execute 17.0.1 (Linux64)                             done. 
   [ 78/128 ]  daf_butler 17.0.1 (Linux64)                               done. 
   [ 79/128 ]  dax_ppdb 17.0.1 (Linux64)                                 done. 
   [ 80/128 ]  display_ds9 17.0.1 (Linux64)                              done. 
   [ 81/128 ]  display_firefly 17.0.1 (Linux64)                          done. 
   [ 82/128 ]  display_matplotlib 17.0.1 (Linux64)                       done. 
   [ 83/128 ]  shapelet 17.0.1 (Linux64)                                 done. 
   [ 84/128 ]  ctrl_platform_lsstvc 17.0.1 (Linux64)                     done. 
   [ 85/128 ]  obs_base 17.0.1 (Linux64)                                 done. 
   [ 86/128 ]  skymap 17.0.1 (Linux64)                                   done. 
   [ 87/128 ]  obs_test 17.0.1 (Linux64)                                 done. 
   [ 88/128 ]  pipe_base 17.0.1 (Linux64)                                done. 
   [ 89/128 ]  coadd_utils 17.0.1 (Linux64)                              done. 
   [ 90/128 ]  cp_pipe 17.0.1 (Linux64)                                  done. 
   [ 91/128 ]  ctrl_mpexec 17.0.1 (Linux64)                              done. 
   [ 92/128 ]  ctrl_pool 17.0.1 (Linux64)                                done. 
   [ 93/128 ]  verify 17.0.1 (Linux64)                                   done. 
   [ 94/128 ]  meas_base 17.0.1 (Linux64)                                done. 
   [ 95/128 ]  meas_algorithms 17.0.1 (Linux64)                          done. 
   [ 96/128 ]  meas_extensions_simpleShape 17.0.1 (Linux64)              done. 
   [ 97/128 ]  ap_association 17.0.1 (Linux64)                           done. 
   [ 98/128 ]  ip_isr 17.0.1 (Linux64)                                   done. 
   [ 99/128 ]  meas_astrom 17.0.1 (Linux64)                              done. 
   [ 100/128 ]  meas_deblender 17.0.1 (Linux64)                          done. 
   [ 101/128 ]  meas_extensions_photometryKron 17.0.1 (Linux64)          done. 
   [ 102/128 ]  meas_extensions_psfex 17.0.1 (Linux64)                   done. 
   [ 103/128 ]  meas_extensions_shapeHSM 17.0.1 (Linux64)                done. 
   [ 104/128 ]  meas_modelfit 17.0.1 (Linux64)                           done. 
   [ 105/128 ]  ip_diffim 17.0.1 (Linux64)                               done. 
   [ 106/128 ]  meas_extensions_astrometryNet 17.0.1 (Linux64)           done. 
   [ 107/128 ]  meas_extensions_convolved 17.0.1 (Linux64)               done. 
   [ 108/128 ]  obs_ctio0m9 17.0.1 (Linux64)                             done. 
   [ 109/128 ]  obs_monocam 17.0.1 (Linux64)                             done. 
   [ 110/128 ]  pipe_tasks 17.0.1 (Linux64)                              done. 
   [ 111/128 ]  ap_pipe 17.0.1 (Linux64)                                 done. 
   [ 112/128 ]  fgcmcal 17.0.1 (Linux64)                                 done. 
   [ 113/128 ]  obs_cfht 17.0.1 (Linux64)                                done. 
   [ 114/128 ]  obs_comCam 17.0.1 (Linux64)                              done. 
   [ 115/128 ]  obs_decam 17.0.1 (Linux64)                               done. 
   [ 116/128 ]  obs_lsst 17.0.1 (Linux64)                                done. 
   [ 117/128 ]  obs_lsstSim 17.0.1 (Linux64)                             done. 
   [ 118/128 ]  obs_sdss 17.0.1 (Linux64)                                done. 
   [ 119/128 ]  obs_subaru 17.0.1 (Linux64)                              done. 
   [ 120/128 ]  pipe_drivers 17.0.1 (Linux64)                            done. 
   [ 121/128 ]  synpipe 17.0.1 (Linux64)                                 done. 
   [ 122/128 ]  ap_verify 17.0.1 (Linux64)                               done. 
   [ 123/128 ]  jointcal 17.0.1 (Linux64)                                done. 
   [ 124/128 ]  lsst_apps 17.0.1 (Linux64)                               done. 
   [ 125/128 ]  lsst_obs 17.0.1 (Linux64)                                done. 
   [ 126/128 ]  meas_mosaic 17.0.1 (Linux64)                             done. 
   [ 127/128 ]  validate_drp 17.0.1 (Linux64)                            done. 
   [ 128/128 ]  lsst_distrib 17.0.1 (Linux64)                            done. 
(lsst-scipipe-fcd27eb) [csurhud@ln2 lsst_stacknew]$ cd stack/miniconda3-4.5.4-fcd27eb/Linux64/obs_subaru/17.0.1/config/hsc/
(lsst-scipipe-fcd27eb) [csurhud@ln2 hsc]$ cat bias.py 
import os.path

from lsst.utils import getPackageDir
from lsst.obs.subaru.isr import SubaruIsrTask

config.isr.retarget(SubaruIsrTask)
config.isr.load(os.path.join(getPackageDir("obs_subaru"), "config", "hsc", "isr.py"))
config.isr.doBrighterFatter = False
config.isr.doStrayLight = False

(Surhud More) #8

@price @mrawls I finally just installed hscPipe 6.7 and got it running out of the box based on instructions here. There seem to be some issues with the current version of LSSTPipe in order to get it working as is with HSC data.


(Paul Price) #9

OK, now I see the traceback, I recognise this as a bug in the LSST 17.0.1 release. It was fixed in DM-18174.

I could offer you multiple suggestions about how to proceed, but I recommend switching to the latest release of hscPipe (7.7.2), which is the version currently being used at NAOJ for the SSP production run:

# Start new installation
wget http://tigress-web.princeton.edu/~HSC/hscPipe7/newinstall.sh
bash newinstall.sh
# Answer "yes" to installing Anaconda unless you really know what you're doing
# Source the appropriate file it tells you to, and then proceed to the next step

# Install this release:
eups distrib install hscPipe 7.7.2

(Surhud More) #10

Got it, thanks @price!