Bug 1119265

Summary: Docker daemon non-functional - fails to connect to containerd, containerd not running
Product: [openSUSE] openSUSE Tumbleweed Reporter: Athanasios Zorbas <thanosz>
Component: ContainersAssignee: Containers Team <containers-bugowner>
Status: RESOLVED INVALID QA Contact: E-mail List <qa-bugs>
Severity: Major    
Priority: P2 - High CC: mvedovati, thanosz
Version: Current   
Target Milestone: ---   
Hardware: 64bit   
OS: Other   
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---

Description Athanasios Zorbas 2018-12-12 13:36:45 UTC
I installed docker but it does not run. It seems that containerd is not running. Running docker-containerd manually resolves the problem


NAME="openSUSE Tumbleweed"
# VERSION="20181208"
ID="opensuse-tumbleweed"
ID_LIKE="opensuse suse"
VERSION_ID="20181208"
PRETTY_NAME="openSUSE Tumbleweed"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:opensuse:tumbleweed:20181208"
BUG_REPORT_URL="https://bugs.opensuse.org"
HOME_URL="https://www.opensuse.org/"


root@electra:~ # zypper in docker
Loading repository data...
Reading installed packages...c
Resolving package dependencies...

The following 5 NEW packages are going to be installed:
  containerd docker docker-bash-completion docker-libnetwork docker-runc

5 new packages to install.
Overall download size: 40.3 MiB. Already cached: 0 B. After the operation, additional 195.3 MiB will be used.
Continue? [y/n/...? shows all options] (y): y
Retrieving package docker-libnetwork-0.7.0.1+gitr2704_6da50d197830-33.1.x86_64          (1/5), 559.0 KiB (  1.8 MiB unpacked)
Retrieving: docker-libnetwork-0.7.0.1+gitr2704_6da50d197830-33.1.x86_64.rpm .............................[done (388.4 KiB/s)]
Retrieving package docker-runc-1.0.0rc5+gitr3562_69663f0bd4b6-37.3.x86_64               (2/5),   1.8 MiB (  7.9 MiB unpacked)
Retrieving: docker-runc-1.0.0rc5+gitr3562_69663f0bd4b6-37.3.x86_64.rpm ..................................[done (788.9 KiB/s)]
Retrieving package containerd-1.1.2-84.3.x86_64                                         (3/5),   8.7 MiB ( 40.8 MiB unpacked)
Retrieving: containerd-1.1.2-84.3.x86_64.rpm ............................................................[done (939.4 KiB/s)]
Retrieving package docker-18.09.0_ce-304.2.x86_64                                       (4/5),  29.1 MiB (144.6 MiB unpacked)
Retrieving: docker-18.09.0_ce-304.2.x86_64.rpm ..........................................................[done (964.2 KiB/s)]
Retrieving package docker-bash-completion-18.09.0_ce-304.4.noarch                       (5/5), 157.3 KiB (104.8 KiB unpacked)
Retrieving: docker-bash-completion-18.09.0_ce-304.4.noarch.rpm ..........................................[done (332.3 KiB/s)]
Checking for file conflicts: ..........................................................................................[done]
(1/5) Installing: docker-libnetwork-0.7.0.1+gitr2704_6da50d197830-33.1.x86_64 .........................................[done]
Additional rpm output:
warning: /var/cache/zypp/packages/Virtualization_containers/x86_64/docker-libnetwork-0.7.0.1+gitr2704_6da50d197830-33.1.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 72174fc2: NOKEY


(2/5) Installing: docker-runc-1.0.0rc5+gitr3562_69663f0bd4b6-37.3.x86_64 ..............................................[done]
Additional rpm output:
warning: /var/cache/zypp/packages/Virtualization_containers/x86_64/docker-runc-1.0.0rc5+gitr3562_69663f0bd4b6-37.3.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 72174fc2: NOKEY


(3/5) Installing: containerd-1.1.2-84.3.x86_64 ........................................................................[done]
Additional rpm output:
warning: /var/cache/zypp/packages/Virtualization_containers/x86_64/containerd-1.1.2-84.3.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 72174fc2: NOKEY


(4/5) Installing: docker-18.09.0_ce-304.2.x86_64 ......................................................................[done]
Additional rpm output:
warning: /var/cache/zypp/packages/Virtualization_containers/x86_64/docker-18.09.0_ce-304.2.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 72174fc2: NOKEY
Updating /etc/sysconfig/docker ...
Failed to disable unit: Unit file containerd.service does not exist.
Failed to stop containerd.service: Unit containerd.service not loaded.
Failed to stop containerd.socket: Unit containerd.socket not loaded.


(5/5) Installing: docker-bash-completion-18.09.0_ce-304.4.noarch ......................................................[done]
Additional rpm output:
warning: /var/cache/zypp/packages/Virtualization_containers/noarch/docker-bash-completion-18.09.0_ce-304.4.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID 72174fc2: NOKEY


root@electra:~ # systemctl start docker
Job for docker.service failed because a timeout was exceeded.
See "systemctl status docker.service" and "journalctl -xe" for details.


root@electra:~ # systemctl status docker.service

● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/docker.service.d
           └─http-proxy.conf, https-proxy.conf, no-proxy.conf, options.conf
   Active: activating (start) since Wed 2018-12-12 15:31:38 EET; 25s ago
     Docs: http://docs.docker.com
 Main PID: 24328 (dockerd)
    Tasks: 27
   Memory: 41.1M
   CGroup: /system.slice/docker.service
           └─24328 /usr/bin/dockerd --containerd /run/containerd/containerd.sock --add-runtime oci=/usr/sbin/docker-runc -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock

Dec 12 15:31:58 electra dockerd[24328]: time="2018-12-12T15:31:58.605087780+02:00" level=warning msg="grpc: addrConn.createTransport failed to connect to {unix:///run/containerd/containerd.sock 0  <nil>}. Err :connection error: desc = \"transport: Error while dialing dial unix:///run/containerd/containerd.sock: timeout\". Reconnecting..." module=grpc
Dec 12 15:31:58 electra dockerd[24328]: time="2018-12-12T15:31:58.605280338+02:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420974340, TRANSIENT_FAILURE" module=grpc
Dec 12 15:31:58 electra dockerd[24328]: time="2018-12-12T15:31:58.605345632+02:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420974340, CONNECTING" module=grpc
Dec 12 15:31:58 electra dockerd[24328]: time="2018-12-12T15:31:58.605409716+02:00" level=warning msg="grpc: addrConn.createTransport failed to connect to {unix:///run/containerd/containerd.sock 0  <nil>}. Err :connection error: desc = \"transport: Error while dialing dial unix:///run/containerd/containerd.sock: timeout\". Reconnecting..." module=grpc
Dec 12 15:31:58 electra dockerd[24328]: time="2018-12-12T15:31:58.605501003+02:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc42098c860, TRANSIENT_FAILURE" module=grpc
Dec 12 15:31:58 electra dockerd[24328]: time="2018-12-12T15:31:58.605572488+02:00" level=error msg="failed to get event" error="rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial unix:///run/containerd/containerd.sock: timeout\"" module=libcontainerd namespace=plugins.moby
Dec 12 15:31:58 electra dockerd[24328]: time="2018-12-12T15:31:58.605522444+02:00" level=error msg="Failed to restore container bf6e0a4366e903bdcc03fdb052ab0e61c65b0a6009bc36ddaf17723c634999b7 with containerd: all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial unix:///run/containerd/containerd.sock: timeout\": unavailable"
Dec 12 15:31:58 electra dockerd[24328]: time="2018-12-12T15:31:58.605706766+02:00" level=error msg="failed to get event" error="rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial unix:///run/containerd/containerd.sock: timeout\"" module=libcontainerd namespace=plugins.moby
Dec 12 15:31:58 electra dockerd[24328]: time="2018-12-12T15:31:58.605716151+02:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc42098c860, CONNECTING" module=grpc
Dec 12 15:31:58 electra dockerd[24328]: time="2018-12-12T15:31:58.606000215+02:00" level=error msg="failed to get event" error="rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial unix:///run/containerd/containerd.sock: timeout\"" module=libcontainerd namespace=plugins.moby


If I run docker-containerd, docker is functional again. (Missing systemd unit file?)
root@electra:~ # docker-containerd 



root@electra:~ # systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/docker.service.d
           └─http-proxy.conf, https-proxy.conf, no-proxy.conf, options.conf
   Active: active (running) since Wed 2018-12-12 15:32:56 EET; 25s ago
     Docs: http://docs.docker.com
 Main PID: 24328 (dockerd)
    Tasks: 35
   Memory: 44.8M
   CGroup: /system.slice/docker.service
           ├─24328 /usr/bin/dockerd --containerd /run/containerd/containerd.sock --add-runtime oci=/usr/sbin/docker-runc -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock
           └─24827 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 9001 -container-ip 172.17.0.2 -container-port 80

Dec 12 15:32:54 electra dockerd[24328]: time="2018-12-12T15:32:54.819590159+02:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420974340, READY" module=grpc
Dec 12 15:32:54 electra dockerd[24328]: time="2018-12-12T15:32:54.824285634+02:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc42098c860, READY" module=grpc
Comment 1 Marco Vedovati 2018-12-14 15:10:05 UTC
This may happen because you have specified the `--containerd /run/containerd/containerd.sock`  option in /etc/sysconfig/docker or in the docker unit file (/usr/lib/systemd/system/docker.service).

Can you check in both files and make sure that the option is not there?
Comment 2 Athanasios Zorbas 2018-12-14 19:46:07 UTC
Thanks for your response. That did it. It was a leftover option under /etc/systemd/system/docker.service.d. Removing --containerd /run/containerd/containerd.sock did the trick.