Toshi website blocked

Support
#1

Hi Community

Currently working through the second part of the pipeline tutorial and have run into the funny situation that the toshi.nofs.mil cite is blocked. From processCcd.py the command fails on every image with a

"WARNING: failed to download http://maia.usno.navy.mil/ser7/finals2000A.all and http://toshi.nofs.navy.mil/ser7/finals2000A.all, using local IERS-B: <urlopen error [Errno -2] Name or service not known>;HTTP Error 503: Service Unavailable [astropy.utils.iers.iers]

WARNING:astropy:failed to download http://maia.usno.navy.mil/ser7/finals2000A.all and http://toshi.nofs.navy.mil/ser7/finals2000A.all, using local IERS-B: <urlopen error [Errno -2] Name or service not known>;HTTP Error 503: Service Unavailable"

Doing the dumb thing of loading up the blocked site (http://toshi.nofs.navy.mil/ser7/finals2000A.all) in a browser brings up “WEBSITE BLOCKED” page just dripping with freedom.

This may be an issue with http vs. https on my network, or something harder to deal with.

I am attempting this all from a Canadian NRC network, for what it’s worth.

Anyone seen this before?

Thanks

Wes

#2

You need a new version of Astropy with modified endpoints for IERS tables. Try doing conda update astropy. You will need 3.2.3 I think.

#3

To expand on that a little bit:

  • First, it’s worth noting that you’re looking at an old version of the tutorial. I don’t think it makes any difference to the specific problem you’re seeing, but you can find the most up-to-date version of that page at https://pipelines.lsst.io/getting-started/processccd.html.
  • Second, as Tim correctly points out, this problem is really originating in Astropy, rather than LSST code. It was fixed in Astropy 3.2.3, which was released in October.
  • We don’t formally distribute Astropy as part of the LSST software distribution. However — depending a bit on exactly which installation method you used — you likely had the option to use LSST’s Conda environment when installing. Doing that will pull in version 3.1.2 of Astropy for the last stable release of the LSST Pipelines (18.1.0); if you install a recent weekly (or the upcoming 19.0.0 release) you’ll get Astropy 3.2.3.
  • As Tim suggests, you can likely upgrade Astropy in your existing Conda environment to work around the problem.
#4

Hi all

We had the same problem - our solution was to change in $lsstsw/miniconda/pkgs/astropy-3.1.2-py37h7b6447c_0/lib/python3.7/site-packages/astropy/utils/iers/iers.py

instead of sites with .mil to

https://datacenter.iers.org/data/9/finals2000A.all

Cheers

D.

#5

I strongly argue in favor of doing the conda update rather than hacking the source in your conda installation :smile:

#6

Thanks for the responses.

A few things to note: I followed the instructions to install via the newinstall.sh method, and adopted the miniconda installation with that. So I have no idea why there are version mismatches at this point.

As for the tutorial version, the one I am using is the one linked through a google search, so it sounds like the indexing isn’t keeping up.

As for conda updates, I am getting version update errors regardless of how conda is updated (either update all or update astropy, with reverting using --revisions). The latter for example results in the following:

root INFO: Loading config overrride file ‘/home/ubuntu/lsst_stack/stack/miniconda3-4.5.12-1172c30/Linux64/obs_subaru/18.1.0/config/processCcd.py’

root INFO: Loading config overrride file ‘/home/ubuntu/lsst_stack/stack/miniconda3-4.5.12-1172c30/Linux64/obs_subaru/18.1.0/config/hsc/processCcd.py’

CameraMapper INFO: Loading exposure registry from /home/ubuntu/DATA/registry.sqlite3

CameraMapper INFO: Loading calib registry from /home/ubuntu/DATA/CALIB/calibRegistry.sqlite3

CameraMapper INFO: Loading calib registry from /home/ubuntu/DATA/CALIB/calibRegistry.sqlite3

root INFO: Running: /home/ubuntu/lsst_stack/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/18.1.0/bin/processCcd.py DATA --rerun processCcdOutputs --id

WARNING: You are using OpenBLAS with multiple threads (4), but have not

specified the number of threads using one of the OpenBLAS environment variables:

OPENBLAS_NUM_THREADS, GOTO_NUM_THREADS, OMP_NUM_THREADS.

This may indicate that you are unintentionally using multiple threads, which may

cause problems. WE HAVE THEREFORE DISABLED OpenBLAS THREADING. If you know

what you are doing and want threads enabled implicitly, set the environment

variable LSST_ALLOW_IMPLICIT_THREADS.

processCcd FATAL: Failed in task initialization: Version mismatch (astropy: 3.1.2 vs 3.2.3; astropy.wcs._wcs: 5.19.1 vs 6.2.0; astropy.io.fits.diff: 3.1.2 vs 3.2.3); consider using --clobber-versions or --no-versions

So I tried with --no-versions and get the massive string of errors:

(lsst-scipipe-1172c30) ubuntu@lsst-test:~$ processCcd.py DATA --rerun processCcdOutputs --id --no-versions
root INFO: Loading config overrride file ‘/home/ubuntu/lsst_stack/stack/miniconda3-4.5.12-1172c30/Linux64/obs_subaru/18.1.0/config/processCcd.py’
root INFO: Loading config overrride file ‘/home/ubuntu/lsst_stack/stack/miniconda3-4.5.12-1172c30/Linux64/obs_subaru/18.1.0/config/hsc/processCcd.py’
CameraMapper INFO: Loading exposure registry from /home/ubuntu/DATA/registry.sqlite3
CameraMapper INFO: Loading calib registry from /home/ubuntu/DATA/CALIB/calibRegistry.sqlite3
CameraMapper INFO: Loading calib registry from /home/ubuntu/DATA/CALIB/calibRegistry.sqlite3
root INFO: Running: /home/ubuntu/lsst_stack/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/18.1.0/bin/processCcd.py DATA --rerun processCcdOutputs --id --no-versions
WARNING: You are using OpenBLAS with multiple threads (4), but have not
specified the number of threads using one of the OpenBLAS environment variables:
OPENBLAS_NUM_THREADS, GOTO_NUM_THREADS, OMP_NUM_THREADS.
This may indicate that you are unintentionally using multiple threads, which may
cause problems. WE HAVE THEREFORE DISABLED OpenBLAS THREADING. If you know
what you are doing and want threads enabled implicitly, set the environment
variable LSST_ALLOW_IMPLICIT_THREADS.
processCcd INFO: Processing {‘field’: ‘STRIPE82L’, ‘dateObs’: ‘2013-06-17’, ‘pointing’: 533, ‘filter’: ‘HSC-R’, ‘visit’: 903334, ‘ccd’: 23, ‘taiObs’: ‘2013-06-17’, ‘expTime’: 30.0}
processCcd.isr INFO: Performing ISR on sensor {‘field’: ‘STRIPE82L’, ‘dateObs’: ‘2013-06-17’, ‘pointing’: 533, ‘filter’: ‘HSC-R’, ‘visit’: 903334, ‘ccd’: 23, ‘taiObs’: ‘2013-06-17’, ‘expTime’: 30.0}
Downloading ftp://cddis.gsfc.nasa.gov/pub/products/iers/finals2000A.all
|==============================================================================| 3.3M/3.3M (100.00%) 0s
processCcd FATAL: Failed on dataId={‘field’: ‘STRIPE82L’, ‘dateObs’: ‘2013-06-17’, ‘pointing’: 533, ‘filter’: ‘HSC-R’, ‘visit’: 903334, ‘ccd’: 23, ‘taiObs’: ‘2013-06-17’, ‘expTime’: 30.0}: NoResults: No locations for get: datasetType:transmission_optics dataId:DataId(initialdata={‘field’: ‘STRIPE82L’, ‘dateObs’: ‘2013-06-17’, ‘pointing’: 533, ‘filter’: ‘HSC-R’, ‘visit’: 903334, ‘ccd’: 23, ‘taiObs’: ‘2013-06-17’, ‘expTime’: 30.0}, tag=set())
Traceback (most recent call last):
File “/home/ubuntu/lsst_stack/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py”, line 388, in call
result = self.runTask(task, dataRef, kwargs)
File “/home/ubuntu/lsst_stack/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py”, line 447, in runTask
return task.runDataRef(dataRef, **kwargs)
File “/home/ubuntu/lsst_stack/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/timer.py”, line 150, in wrapper
res = func(self, *args, **keyArgs)
File “/home/ubuntu/lsst_stack/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/18.1.0/python/lsst/pipe/tasks/processCcd.py”, line 184, in runDataRef
exposure = self.isr.runDataRef(sensorRef).exposure
File “/home/ubuntu/lsst_stack/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/timer.py”, line 150, in wrapper
res = func(self, *args, **keyArgs)
File “/home/ubuntu/lsst_stack/stack/miniconda3-4.5.12-1172c30/Linux64/ip_isr/18.1.0/python/lsst/ip/isr/isrTask.py”, line 1429, in runDataRef
isrData = self.readIsrData(sensorRef, ccdExposure)
File “/home/ubuntu/lsst_stack/stack/miniconda3-4.5.12-1172c30/Linux64/ip_isr/18.1.0/python/lsst/ip/isr/isrTask.py”, line 945, in readIsrData
if self.config.doUseOpticsTransmission else None)
File “/home/ubuntu/lsst_stack/stack/miniconda3-4.5.12-1172c30/Linux64/daf_persistence/18.1.0/python/lsst/daf/persistence/butlerSubset.py”, line 206, in get
return self.butlerSubset.butler.get(datasetType, self.dataId, **rest)
File “/home/ubuntu/lsst_stack/stack/miniconda3-4.5.12-1172c30/Linux64/daf_persistence/18.1.0/python/lsst/daf/persistence/butler.py”, line 1380, in get
raise NoResults(“No locations for get:”, datasetType, dataId)
lsst.daf.persistence.butlerExceptions.NoResults: No locations for get: datasetType:transmission_optics dataId:DataId(initialdata={‘field’: ‘STRIPE82L’, ‘dateObs’: ‘2013-06-17’, ‘pointing’: 533, ‘filter’: ‘HSC-R’, ‘visit’: 903334, ‘ccd’: 23, ‘taiObs’: ‘2013-06-17’, ‘expTime’: 30.0}, tag=set())
processCcd INFO: Processing {‘field’: ‘STRIPE82L’, ‘dateObs’: ‘2013-06-17’, ‘pointing’: 533, ‘filter’: ‘HSC-R’, ‘visit’: 903334, ‘ccd’: 22, ‘taiObs’: ‘2013-06-17’, ‘expTime’: 30.0}
processCcd.isr INFO: Performing ISR on sensor {‘field’: ‘STRIPE82L’, ‘dateObs’: ‘2013-06-17’, ‘pointing’: 533, ‘filter’: ‘HSC-R’, ‘visit’: 903334, ‘ccd’: 22, ‘taiObs’: ‘2013-06-17’, ‘expTime’: 30.0}
processCcd FATAL: Failed on dataId={‘field’: ‘STRIPE82L’, ‘dateObs’: ‘2013-06-17’, ‘pointing’: 533, ‘filter’: ‘HSC-R’, ‘visit’: 903334, ‘ccd’: 22, ‘taiObs’: ‘2013-06-17’, ‘expTime’: 30.0}: NoResults: No locations for get: datasetType:transmission_optics dataId:DataId(initialdata={‘field’: ‘STRIPE82L’, ‘dateObs’: ‘2013-06-17’, ‘pointing’: 533, ‘filter’: ‘HSC-R’, ‘visit’: 903334, ‘ccd’: 22, ‘taiObs’: ‘2013-06-17’, ‘expTime’: 30.0}, tag=set())
Traceback (most recent call last):
File “/home/ubuntu/lsst_stack/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py”, line 388, in call
result = self.runTask(task, dataRef, kwargs)
File “/home/ubuntu/lsst_stack/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py”, line 447, in runTask
return task.runDataRef(dataRef, **kwargs)
File “/home/ubuntu/lsst_stack/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/timer.py”, line 150, in wrapper
res = func(self, *args, **keyArgs)
File “/home/ubuntu/lsst_stack/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/18.1.0/python/lsst/pipe/tasks/processCcd.py”, line 184, in runDataRef
exposure = self.isr.runDataRef(sensorRef).exposure
File “/home/ubuntu/lsst_stack/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/timer.py”, line 150, in wrapper
res = func(self, *args, **keyArgs)
File “/home/ubuntu/lsst_stack/stack/miniconda3-4.5.12-1172c30/Linux64/ip_isr/18.1.0/python/lsst/ip/isr/isrTask.py”, line 1429, in runDataRef
isrData = self.readIsrData(sensorRef, ccdExposure)
File “/home/ubuntu/lsst_stack/stack/miniconda3-4.5.12-1172c30/Linux64/ip_isr/18.1.0/python/lsst/ip/isr/isrTask.py”, line 945, in readIsrData
if self.config.doUseOpticsTransmission else None)
File “/home/ubuntu/lsst_stack/stack/miniconda3-4.5.12-1172c30/Linux64/daf_persistence/18.1.0/python/lsst/daf/persistence/butlerSubset.py”, line 206, in get
return self.butlerSubset.butler.get(datasetType, self.dataId, **rest)
File “/home/ubuntu/lsst_stack/stack/miniconda3-4.5.12-1172c30/Linux64/daf_persistence/18.1.0/python/lsst/daf/persistence/butler.py”, line 1380, in get
raise NoResults(“No locations for get:”, datasetType, dataId)
lsst.daf.persistence.butlerExceptions.NoResults: No locations for get: datasetType:transmission_optics dataId:DataId(initialdata={‘field’: ‘STRIPE82L’, ‘dateObs’: ‘2013-06-17’, ‘pointing’: 533, ‘filter’: ‘HSC-R’, ‘visit’: 903334, ‘ccd’: 22, ‘taiObs’: ‘2013-06-17’, ‘expTime’: 30.0}, tag=set())
processCcd INFO: Processing {‘field’: ‘STRIPE82L’, ‘dateObs’: ‘2013-06-17’, ‘pointing’: 533, ‘filter’: ‘HSC-R’, ‘visit’: 903334, ‘ccd’: 16, ‘taiObs’: ‘2013-06-17’, ‘expTime’: 30.0}
processCcd.isr INFO: Performing ISR on sensor {‘field’: ‘STRIPE82L’, ‘dateObs’: ‘2013-06-17’, ‘pointing’: 533, ‘filter’: ‘HSC-R’, ‘visit’: 903334, ‘ccd’: 16, ‘taiObs’: ‘2013-06-17’, ‘expTime’: 30.0}
processCcd FATAL: Failed on dataId={‘field’: ‘STRIPE82L’, ‘dateObs’: ‘2013-06-17’, ‘pointing’: 533, ‘filter’: ‘HSC-R’, ‘visit’: 903334, ‘ccd’: 16, ‘taiObs’: ‘2013-06-17’, ‘expTime’: 30.0}: NoResults: No locations for get: datasetType:transmission_optics dataId:DataId(initialdata={‘field’: ‘STRIPE82L’, ‘dateObs’: ‘2013-06-17’, ‘pointing’: 533, ‘filter’: ‘HSC-R’, ‘visit’: 903334, ‘ccd’: 16, ‘taiObs’: ‘2013-06-17’, ‘expTime’: 30.0}, tag=set())
…

Note i cut the errors off as they seem to be repeating one per input fits file.

#7

Astropy 3.2.3 was released after lsst_distrib 18.1.0 so the conda environment that comes with newinstall will be out of date. I think if you used master newinstall and a weekly install tag this will be fixed automatically.

What did conda update astropy say? What did it try to update?

Somehow you have both 3.1.2 and 3.2.3 of astropy installed.

#8

Thanks for continuing to help debug what I thought was going to be the easy part.

Below is output of conda update. Note this is done from revision 0 of the miniconda install. That is, directly from what is installed my newinstall.sh.

Is it possible that like the tutorial, there are outdated installation instructions that google has provided me? I am not sure where to get a different version of newinstall other than what is provided in the instructions. Also not sure how to get a list of tags from which to choose.

(lsst-scipipe-1172c30) ubuntu@lsst-test:~$ conda update astropy
Solving environment: done

==> WARNING: A newer version of conda exists. <==
current version: 4.5.12
latest version: 4.7.12

Please update conda by running

$ conda update -n base -c defaults conda

Package Plan

environment location: /home/ubuntu/lsst_stack/python/miniconda3-4.5.12/envs/lsst-scipipe-1172c30

added / updated specs:
- astropy

The following packages will be UPDATED:

astropy: 3.1.2-py37h7b6447c_0 --> 3.2.3-py37h7b6447c_0

Proceed ([y]/n)? y

Preparing transaction: done
Verifying transaction: done
Executing transaction: done

#9

Seems right. I don’t understand why astropy still has some 3.1.2 versions lying around after that. I can only suggest you use master newinstall and build a weekly.

#10

use master newinstall and build a weekly

Any chance you could explain what that means? Keep in mind I am not familiar with the dev work, repo layouts, etc.

Thanks

#11

Found it.

https://pipelines.lsst.io/install/newinstall.html#newinstall-other-tags

#12

The “version mismatch” message is coming from the (Gen2) Butler. It is saying that you created an output repository using a particular version of various software packages and are now trying to write to it using different versions of those packages. This is disallowed without explicit user override to protect against inconsistency in production outputs.

The simplest way to avoid this is to use a new output repository or to remove the existing one.

#13

Getting further. Next error! ha

Installed using the master newinstall as suggested by Tim, but using v18_1_0 of the lsst_distrib as using w_latest resulted in its own set of eups install failures… I at least have astropy v3.2.3 as needed. I can also confirm I have actual fits files (aka git lfs worked correctly) including the raw imagery and the PS1 catalogs with appropriate links in ref_cats.

Now I am running into the following error for all images.

(lsst-scipipe-4d7b902) ubuntu@lsst-test:/mnt$ processCcd.py DATA --rerun processCcdOutputs --id
root INFO: Loading config overrride file ‘/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/obs_subaru/18.1.0/config/processCcd.py’
root INFO: Loading config overrride file ‘/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/obs_subaru/18.1.0/config/hsc/processCcd.py’
CameraMapper INFO: Loading exposure registry from /mnt/DATA/registry.sqlite3
CameraMapper INFO: Loading calib registry from /mnt/DATA/CALIB/calibRegistry.sqlite3
CameraMapper INFO: Loading calib registry from /mnt/DATA/CALIB/calibRegistry.sqlite3
root INFO: Running: /home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_tasks/18.1.0/bin/processCcd.py DATA --rerun processCcdOutputs --id
WARNING: You are using OpenBLAS with multiple threads (4), but have not
specified the number of threads using one of the OpenBLAS environment variables:
OPENBLAS_NUM_THREADS, GOTO_NUM_THREADS, OMP_NUM_THREADS.
This may indicate that you are unintentionally using multiple threads, which may
cause problems. WE HAVE THEREFORE DISABLED OpenBLAS THREADING. If you know
what you are doing and want threads enabled implicitly, set the environment
variable LSST_ALLOW_IMPLICIT_THREADS.
processCcd INFO: Processing {‘field’: ‘STRIPE82L’, ‘dateObs’: ‘2013-06-17’, ‘pointing’: 533, ‘filter’: ‘HSC-R’, ‘visit’: 903334, ‘ccd’: 23, ‘taiObs’: ‘2013-06-17’, ‘expTime’: 30.0}
processCcd.isr INFO: Performing ISR on sensor {‘field’: ‘STRIPE82L’, ‘dateObs’: ‘2013-06-17’, ‘pointing’: 533, ‘filter’: ‘HSC-R’, ‘visit’: 903334, ‘ccd’: 23, ‘taiObs’: ‘2013-06-17’, ‘expTime’: 30.0}
processCcd FATAL: Failed on dataId={‘field’: ‘STRIPE82L’, ‘dateObs’: ‘2013-06-17’, ‘pointing’: 533, ‘filter’: ‘HSC-R’, ‘visit’: 903334, ‘ccd’: 23, ‘taiObs’: ‘2013-06-17’, ‘expTime’: 30.0}: NoResults: No locations for get: datasetType:transmission_optics dataId:DataId(initialdata={‘field’: ‘STRIPE82L’, ‘dateObs’: ‘2013-06-17’, ‘pointing’: 533, ‘filter’: ‘HSC-R’, ‘visit’: 903334, ‘ccd’: 23, ‘taiObs’: ‘2013-06-17’, ‘expTime’: 30.0}, tag=set())
Traceback (most recent call last):
File “/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py”, line 388, in call
result = self.runTask(task, dataRef, kwargs)
File “/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py”, line 447, in runTask
return task.runDataRef(dataRef, **kwargs)
File “/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/timer.py”, line 150, in wrapper
res = func(self, *args, **keyArgs)
File “/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_tasks/18.1.0/python/lsst/pipe/tasks/processCcd.py”, line 184, in runDataRef
exposure = self.isr.runDataRef(sensorRef).exposure
File “/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/timer.py”, line 150, in wrapper
res = func(self, *args, **keyArgs)
File “/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/ip_isr/18.1.0/python/lsst/ip/isr/isrTask.py”, line 1429, in runDataRef
isrData = self.readIsrData(sensorRef, ccdExposure)
File “/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/ip_isr/18.1.0/python/lsst/ip/isr/isrTask.py”, line 945, in readIsrData
if self.config.doUseOpticsTransmission else None)
File “/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/daf_persistence/18.1.0/python/lsst/daf/persistence/butlerSubset.py”, line 206, in get
return self.butlerSubset.butler.get(datasetType, self.dataId, **rest)
File “/home/ubuntu/lsst_stack/stack/miniconda3-4.7.10-4d7b902/Linux64/daf_persistence/18.1.0/python/lsst/daf/persistence/butler.py”, line 1380, in get
raise NoResults(“No locations for get:”, datasetType, dataId)
lsst.daf.persistence.butlerExceptions.NoResults: No locations for get: datasetType:transmission_optics dataId:DataId(initialdata={‘field’: ‘STRIPE82L’, ‘dateObs’: ‘2013-06-17’, ‘pointing’: 533, ‘filter’: ‘HSC-R’, ‘visit’: 903334, ‘ccd’: 23, ‘taiObs’: ‘2013-06-17’, ‘expTime’: 30.0}, tag=set())

#14

newinstall shouldn’t be installing anything that uses git lfs so would I be correct in saying you were actually using lsstsw all along? I assume from the fact that you are running processCcd that the lsst_distrib install did work okay?

#15

The second step in the tutorial is running Git LFS, so I don’t think that’s a fair inference.

#16

Thanks for the continued responses.

It seems there’s some confusion as to what I am trying to do, and actually doing. I’m merely trying to get through the tutorial to 1) confirm the lsst_distrib install has worked and 2) make sure I understand the basics of the pipeline operations.

I’ll try and be very specific. Sorry about the pedantry of the following. Here’s a specific list of steps I have followed from a clean install of Ubuntu 18.10.

Following the newinstall installation instructions (https://pipelines.lsst.io/install/newinstall.html) I first make sure I have the prereqs:

> sudo apt update
> sudo apt install bison … big list … zlib1g-dev

I also confirmed that git-lfs was installed and functional at this point as per the instructions in ci_hsc tutorial.

Then, I execute newinstall.sh (downloaded from master) and install the miniconda installation as:

> mkdir -p lsst_stack
> cd lsst_stack
> bash newinstall.sh
> source loadLSST.bash

Then install lsst_distrib as:

> eups distrib install -t v18_1_0 lsst_distrib
> curl -sSL https://raw.githubusercontent.com/lsst/shebangtron/master/shebangtron | python
> setup lsst_distrib

No errors during installation so as far as I can tell, we are good to go off to the tutorial (https://pipelines.lsst.io/v/DM-11391/getting-started/data-setup.html).

As stated above, data setup works fine as:

> git clone https://github.com/lsst/ci_hsc
> cd ci_hsc
> git checkout -b tutorial ffa10de
> cd …
> setup -j -r ci_hsc

Butler setup…
> mkdir DATA
> echo “lsst.obs.hsc.HscMapper” > DATA/_mapper

Ingest…
> ingestImages.py DATA $CI_HSC_DIR/raw/.fits --mode=link*
> ln -s $CI_HSC_DIR/CALIB/ DATA/CALIB
> mkdir -p DATA/ref_cats
> ln -s $CI_HSC_DIR/ps1_pv3_3pi_20170110 DATA/ref_cats/ps1_pv3_3pi_20170110

Then finally off to tutorial part 2 (https://pipelines.lsst.io/v/DM-11391/getting-started/processccd.html):

> processCcd.py DATA --rerun processCcdOutputs --id

This is where things break as I posted in message 13.