Repeated trouble download Docker image

mwv · March 23, 2018, 3:58pm

I generally have trouble downloading a Docker image of the stack. I have to try a few times before I get a successful clean download. This morning I had my longest series of unsuccessful tries. See below for the boring details.

Is there something I can try separately to download?
Is there something I can enable to provide more verbose logging or other information to aid investigation of the issue?

Previous discussions on Slack with @ktl and @josh on March 13 suggested this was a problem in the downloads from AWS and the real-time compression.

I unfortunately have no ability to run useful things like traceroute or mtr from my machine (I can run them but the useful information is filtered out). I would love for a simple way to get a little more information to take to my local NOC, if the problem is on our end.

I’m using the Docker for MacOSX app install to provide the command-line tools.

[serenity ~] docker run -ti lsstsqre/centos:7-stack-lsst_distrib-w_2018_10
Unable to find image 'lsstsqre/centos:7-stack-lsst_distrib-w_2018_10' locally
7-stack-lsst_distrib-w_2018_10: Pulling from lsstsqre/centos
af4b0a2388c6: Already exists
71d8ff22b9e3: Pull complete
80bba1407a5e: Pull complete
627590907c30: Pull complete
818ff8fc02d0: Pull complete
7f697852a362: Pull complete
28cfba9a2f9c: Pull complete
144a9fa3d421: Pull complete
a587006746df: Extracting [==================================================>]  697.5MB/697.5MB
4faa29fe0351: Download complete
docker: failed to register layer: Error processing tar file(exit status 1): archive/tar: invalid tar header.
See 'docker run --help'.
[serenity ~] docker run -ti lsstsqre/centos:7-stack-lsst_distrib-w_2018_10
 Unable to find image 'lsstsqre/centos:7-stack-lsst_distrib-w_2018_10' locally
7-stack-lsst_distrib-w_2018_10: Pulling from lsstsqre/centos
af4b0a2388c6: Already exists
71d8ff22b9e3: Pull complete
80bba1407a5e: Pull complete
627590907c30: Pull complete
818ff8fc02d0: Pull complete
7f697852a362: Pull complete
28cfba9a2f9c: Pull complete
144a9fa3d421: Pull complete
a587006746df: Extracting [==================================================>]  697.5MB/697.5MB
4faa29fe0351: Download complete
docker: failed to register layer: Error processing tar file(exit status 1): unexpected EOF.
See 'docker run --help'.
[serenity ~] docker run -ti lsstsqre/centos:7-stack-lsst_distrib-w_2018_10
Unable to find image 'lsstsqre/centos:7-stack-lsst_distrib-w_2018_10' locally
7-stack-lsst_distrib-w_2018_10: Pulling from lsstsqre/centos
af4b0a2388c6: Already exists
71d8ff22b9e3: Pull complete
80bba1407a5e: Pull complete
627590907c30: Pull complete
818ff8fc02d0: Pull complete
7f697852a362: Pull complete
28cfba9a2f9c: Pull complete
144a9fa3d421: Extracting [==================================================>]  511.7MB/511.7MB
a587006746df: Download complete
4faa29fe0351: Download complete
docker: failed to register layer: Error processing tar file(exit status 1): archive/tar: invalid tar header.
See 'docker run --help'.
[serenity ~] docker run -ti lsstsqre/centos:7-stack-lsst_distrib-w_2018_10
Unable to find image 'lsstsqre/centos:7-stack-lsst_distrib-w_2018_10' locally
7-stack-lsst_distrib-w_2018_10: Pulling from lsstsqre/centos
af4b0a2388c6: Already exists
71d8ff22b9e3: Pull complete
80bba1407a5e: Pull complete
627590907c30: Pull complete
818ff8fc02d0: Pull complete
7f697852a362: Pull complete
28cfba9a2f9c: Pull complete
144a9fa3d421: Pull complete
a587006746df: Extracting [==================================================>]  697.5MB/697.5MB
4faa29fe0351: Download complete
docker: failed to register layer: Error processing tar file(exit status 1): unexpected EOF.
See 'docker run --help'.
[serenity ~] docker run -ti lsstsqre/centos:7-stack-lsst_distrib-w_2018_10
Unable to find image 'lsstsqre/centos:7-stack-lsst_distrib-w_2018_10' locally
7-stack-lsst_distrib-w_2018_10: Pulling from lsstsqre/centos
af4b0a2388c6: Already exists
71d8ff22b9e3: Pull complete
80bba1407a5e: Pull complete
627590907c30: Pull complete
818ff8fc02d0: Pull complete
7f697852a362: Pull complete
28cfba9a2f9c: Pull complete
144a9fa3d421: Extracting [==================================================>]  511.7MB/511.7MB
a587006746df: Download complete
4faa29fe0351: Download complete
docker: failed to register layer: Error processing tar file(exit status 1): archive/tar: invalid tar header.
See 'docker run --help'.

brianv0 · March 23, 2018, 4:10pm

I’m unable to reproduce using the Docker for Mac, the official version, not the brew version.

I will say that extraction of the docker layers appears slower for that same image compared to other docker containers I’ve used.

I’m on:
High Sierra 10.13.3
Docker Community Edition 17.12.0-ce-mac55
Docker Engine: 17.12.0-ce
Docker Machine - 0.13.0
docker --version => Docker version 17.12.0-ce, build c97c6d6

Can you verify Docker versions and Mac OS versions?

mwv · March 23, 2018, 4:21pm

High Siera 10.13.3
Docker Community Edition:
Version 17.12.0-ce-mac55 (23011)
Engine: 17.12.0-ce
Machine: 0.13.0
[serenity ~] docker --version
Docker version 17.12.0-ce, build c97c6d6

josh · March 23, 2018, 4:37pm

I have not personally seen errors extracting docker layers. Possibly explanations are:

the data being returned from docker hub is corrupt. This seems unlikely.
the data is being corrupted in transmission. While I have seen byte swapping errors make it past the tcp header checksum (and the fits header checksum at the same time…), docker hub transfers are (always?) over https, so there would be robust tls hashes as well. It is possible to offload some parts of tls processing to the nic – I don’t know if this is done on osx.
software bugs involving tls processing or within docker – as @brianv0 can’t reproduce with the same docker version, this also seems unlikely.
os (kernel) bugs
cpu/cpu cache corruption
bus (pcie/sata/etc.) transfer corruption
memory corruption
disk corruption

mwv · March 23, 2018, 5:49pm

I’ve experienced this on two different machine, using the same network. So I’m leaning toward ruling out CPU, bus, memory, disk issues.