ERROR: No such container: drone_ambassador_bgLn0tfGX7E
-- Logs begin at Mon 2017-05-29 06:24:53 UTC. --
May 29 17:26:14 viziya-drone dockerd[1375]: time="2017-05-29T17:26:14.120375263Z" level=warning msg="failed to retrieve containerd version: rpc error: code = 14 desc = grpc: the connection is unavailable"
May 29 17:26:25 viziya-drone dockerd[1375]: time="2017-05-29T17:26:25.424926920Z" level=warning msg="DEPRECATED: Setting host configuration options when the container starts is deprecated and has been removed in Docker 1.12"
May 29 17:26:25 viziya-drone dockerd[1375]: time="2017-05-29T17:26:25.493673892Z" level=error msg="stream copy error: reading from a closed fifo\ngithub.com/docker/docker/vendor/github.com/tonistiigi/fifo.(*fifo).Read\n\t/usr/src/docker/.gopath/src/github.com/docker/docker/vendor/github.com/tonistiigi/fifo/fifo.go:142\nbufio.(*Reader).fill\n\t/usr/local/go/src/bufio/bufio.go:97\nbufio.(*Reader).WriteTo\n\t/usr/local/go/src/bufio/bufio.go:472\nio.copyBuffer\n\t/usr/local/go/src/io/io.go:380\nio.Copy\n\t/usr/local/go/src/io/io.go:360\ngithub.com/docker/docker/pkg/pools.Copy\n\t/usr/src/docker/.gopath/src/github.com/docker/docker/pkg/pools/pools.go:60\ngithub.com/docker/docker/container/stream.(*Config).CopyToPipe.func1.1\n\t/usr/src/docker/.gopath/src/github.com/docker/docker/container/stream/streams.go:119\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:2086"
May 29 17:26:25 viziya-drone dockerd[1375]: time="2017-05-29T17:26:25.493951780Z" level=error msg="stream copy error: reading from a closed fifo\ngithub.com/docker/docker/vendor/github.com/tonistiigi/fifo.(*fifo).Read\n\t/usr/src/docker/.gopath/src/github.com/docker/docker/vendor/github.com/tonistiigi/fifo/fifo.go:142\nbufio.(*Reader).fill\n\t/usr/local/go/src/bufio/bufio.go:97\nbufio.(*Reader).WriteTo\n\t/usr/local/go/src/bufio/bufio.go:472\nio.copyBuffer\n\t/usr/local/go/src/io/io.go:380\nio.Copy\n\t/usr/local/go/src/io/io.go:360\ngithub.com/docker/docker/pkg/pools.Copy\n\t/usr/src/docker/.gopath/src/github.com/docker/docker/pkg/pools/pools.go:60\ngithub.com/docker/docker/container/stream.(*Config).CopyToPipe.func1.1\n\t/usr/src/docker/.gopath/src/github.com/docker/docker/container/stream/streams.go:119\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:2086"
May 29 17:26:25 viziya-drone dockerd[1375]: time="2017-05-29T17:26:25.494209751Z" level=error msg="Create container failed with error: grpc: the connection is unavailable"
May 29 17:26:25 viziya-drone dockerd[1375]: time="2017-05-29T17:26:25.597674140Z" level=error msg="Handler for POST /v1.15/containers/4651b32d42f32f222a5f2e04d532ff4fa532ca9675d1b36b7967b8ee87828636/start returned error: grpc: the connection is unavailable"
May 29 17:26:25 viziya-drone dockerd[1375]: time="2017-05-29T17:26:25.657240383Z" level=error msg="Handler for POST /v1.15/containers/create returned error: No such container: drone_ambassador_bgLn0tfGX7E"
May 29 17:26:32 viziya-drone dockerd[1375]: time="2017-05-29T17:26:32.504822959Z" level=info msg="Container failed to stop after sending signal 15 to the process, force killing"
May 29 17:26:32 viziya-drone dockerd[1375]: time="2017-05-29T17:26:32.508786376Z" level=error msg="Handler for POST /v1.22/containers/1a58663be2169c0b449a163ddcb9a2502ea090a14146ce07268aa7b97958e374/stop returned error: Cannot stop container 1a58663be2169c0b449a163ddcb9a2502ea090a14146ce07268aa7b97958e374: Cannot kill container 1a58663be2169c0b449a163ddcb9a2502ea090a14146ce07268aa7b97958e374: rpc error: code = 14 desc = grpc: the connection is unavailable"
May 29 17:26:38 viziya-drone dockerd[1375]: time="2017-05-29T17:26:38.032094257Z" level=warning msg="failed to retrieve containerd version: rpc error: code = 14 desc = grpc: the connection is unavailable"
This has only happened as of noon today. We are gearing up our bug hunting for a big deployment in the next week so there is more activity on the server than normal I guess, but I can’t identify what the root cause could have been. My best guess comes from May 29 17:26:32 viziya-drone dockerd[1375]: time="2017-05-29T17:26:32.504822959Z" level=info msg="Container failed to stop after sending signal 15 to the process, force killing" but I’m not sure how I would solve that?
Am I stuck having to rebuild everything or is there a way to recover this? Also is there a way to identify what caused the root issue?
EDIT: To note when I tell it to restart a build I get the same error but with a different ID on the end of drone_ambassador.