Getting started tutorial Butler data repository issue

Support
#1

Hello
I am having some issues with the online tutorial linked here,
https://pipelines.lsst.io/getting-started/data-setup.html

I am using Docker to access the lsst pipeline container to get the example fits images displayed using DS9, and I have been getting through the tutorial up until the creation of the butler repository. When I attempt to load that section:
ingestImages.py DATA $TESTDATA_CI_HSC_DIR/raw/*.fits --mode=link

I get the errors:
Downloading https://hpiers.obspm.fr/iers/bul/bulc/Leap_Second.dat
|================================================================================================================================| 1.3k/1.3k (100.00%) 0s
root INFO: Loading config overrride file ‘/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/obs_subaru/21.0.0+d401af1dcd/config/ingest.py’
HscMapper WARN: Unable to find calib root directory
CameraMapper INFO: Loading Posix exposure registry from /opt/lsst/software/stack/DATA
ingest WARN: Failed to ingest file /opt/lsst/software/stack/testdata_ci_hsc/raw/HSCA90333426.fits: Error parsing /opt/lsst/software/stack/testdata_ci_hsc/raw/HSCA90333426.fits
Traceback (most recent call last):
File “/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_tasks/21.0.0+44ca056b81/python/lsst/pipe/tasks/ingest.py”, line 569, in runFile
fileInfo, hduInfoList = self.parse.getInfo(infile)
File “/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_tasks/21.0.0+44ca056b81/python/lsst/pipe/tasks/ingest.py”, line 80, in getInfo
md = readMetadata(filename, self.config.hdu)
lsst.pex.exceptions.wrappers.FitsError:
File “src/fits.cc”, line 1578, in lsst::afw::fits::Fits::Fits(const string&, const string&, int)
cfitsio error: error reading from FITS file (108) : Opening file ‘/opt/lsst/software/stack/testdata_ci_hsc/raw/HSCA90333426.fits’ with mode ‘r’
cfitsio error stack:
Error reading data buffer from file:
/opt/lsst/software/stack/testdata_ci_hsc/raw/HSCA90333426.fits
ffopen could not interpret primary array header of file:
/opt/lsst/software/stack/testdata_ci_hsc/raw/HSCA90333426.fits
{0}
lsst::afw::fits::FitsError: 'cfitsio error: error reading from FITS file (108) : Opening file ‘/opt/lsst/software/stack/testdata_ci_hsc/raw/HSCA90333426.fits’ with mode ‘r’
cfitsio error stack:
Error reading data buffer from file:
/opt/lsst/software/stack/testdata_ci_hsc/raw/HSCA90333426.fits
ffopen could not interpret primary array header of file:
/opt/lsst/software/stack/testdata_ci_hsc/raw/HSCA90333426.fits
’

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_tasks/21.0.0+44ca056b81/python/lsst/pipe/tasks/ingest.py”, line 605, in run
self.runFile(infile, registry, args, pos)
File “/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_tasks/21.0.0+44ca056b81/python/lsst/pipe/tasks/ingest.py”, line 572, in runFile
raise RuntimeError(f"Error parsing {infile}") from e
RuntimeError: Error parsing /opt/lsst/software/stack/testdata_ci_hsc/raw/HSCA90333426.fits

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_tasks/21.0.0+44ca056b81/bin/ingestImages.py”, line 3, in
IngestTask.parseAndRun()
File “/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_tasks/21.0.0+44ca056b81/python/lsst/pipe/tasks/ingest.py”, line 411, in parseAndRun
task.run(args)
File “/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_tasks/21.0.0+44ca056b81/python/lsst/pipe/tasks/ingest.py”, line 609, in run
raise IngestError(f"Failed to ingest file {infile}", infile, pos) from exc
lsst.pipe.tasks.ingest.IngestError: Failed to ingest file /opt/lsst/software/stack/testdata_ci_hsc/raw/HSCA90333426.fits

I have followed the tutorial as best I can, including adding the gitconfig and gitcredentials files using vi into the container, but I am still getting this issue.
Could anyone please assist with this issue?

Sincerely
Oliver Bartlett

#2

The errors you are getting suggest that you do not have git-lfs properly configured. If you don’t have a [filter "lfs"] section in your .gitconfig, you may need to execute git lfs install. This should only be necessary once per user/home directory.

#3

As a caveat, please note that those tutorials still refer to the now-deprecated “generation 2” butler. The new “generation 3” butler is a complete rewrite with most of the API being changed. We are currently trying to rewrite the tutorials to use gen3 but not all of them are complete yet.

#4

That seems to of done it, thanks! Now onto step 3.

#5

I see. It seems to of worked up until step 3. The issue I am now facing is that the example works up until trying to display the image. but then states theres an error in row 12. The other files can be seen but not called upon for some reason. Like I said I am using docker for all of this, so it does not seem that the tutorials account for this bit. If you know of any way to fix this, or for me to download those fits files (Docker says I cant as its a read-only image environment), That would be really helpful.

#6

This comment I made on gen3 on another post might be useful for you.

#7

Some aspects of the tutorials won’t change with the middleware, however, including image display.

Some questions: Do you have DS9 running inside the container or outside? Are you seeing its UI (before running anything from Python)? What Python code did you run that gave an error (was it display.mtv(calexp)?)? What error did you get exactly? What other files are you referring to?

#8

So here is the error I am getting. I run python inside the container, then I get this:

calexp = butler.get(‘calexp’, dataId={‘filter’: ‘HSC-R’, ‘visit’: 903334, ‘ccd’: 23})
display.mtv(calexp)
XPA$ERROR no ‘xpaset’ access points match template: ds9
Error writing image: : Broken pipe
Traceback (most recent call last):
File “”, line 1, in
File “/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/afw/21.0.0+fdf35455f6/python/lsst/afw/display/interface.py”, line 551, in mtv
self._impl._mtv(data.getImage(), data.getMask(), wcs, title)
File “/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/display_ds9/21.0.0+8829bf0056/python/lsst/display/ds9/ds9.py”, line 426, in _mtv
_i_mtv(image, wcs, title, False)
File “/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/display_ds9/21.0.0+8829bf0056/python/lsst/display/ds9/ds9.py”, line 685, in _i_mtv
raise e
File “/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/display_ds9/21.0.0+8829bf0056/python/lsst/display/ds9/ds9.py”, line 678, in _i_mtv
displayLib.writeFitsImage(pfd.fileno(), data, wcs, title)
lsst.pex.exceptions.wrappers.RuntimeError:
File “python/lsst/afw/display/simpleFits.cc”, line 441, in void lsst::afw::display::writeBasicFits(int, const ImageT&, const lsst::afw::geom::SkyWcs*, const char*) [with ImageT = lsst::afw::image::Image]
Error writing data for row 12 {0}
lsst::pex::exceptions::RuntimeError: ‘Error writing data for row 12’

From what i gathered this should run DS9 in the container. I have DS9 on my own host, but it also says it comes in the container. Im not seeing the DS9 UI come up before hand, but I can’t seem to open it beforehand anyway. And by other files I mean the other example fits files that come in the testdata_ci_hsc.

#9

This means that the python process cannot locate the ds9 process to be able to talk to it. If you are running ds9 outside the container and python inside, that would explain it.

Running ds9 inside the container would allow python to talk to it, but ds9 may then have difficulty talking to your display.

Unfortunately, you’ll need someone more expert in containers and forwarding X display or XPA ports to explain how to connect all of this up. I suspect you will need to forward some ports on the docker run command line.

#10

I see, thank you for the help with all of this.

I am running the python section in the container, but I’m not entirely sure how to connect to the DS9 in the container. I do have DS9 outside the container, but that would not be of use to me since I am running everything up until this point in the container. Is there a way to get the prepared fits files from the container onto my host system? If not, I’ll have to find a way to tackle part 3 of this tutorial