[solved] ECR plugin fails with "no space left on device" error message

Drone Support
1

When I attempt to build and publish a Docker image to ECR I receive a “no space left on device” error message:

Step 4 : RUN pip install --no-cache-dir uwsgi
---> Running in 44d6a6f5a2f6
Collecting uwsgi
 Downloading uwsgi-2.0.14.tar.gz (788kB)
Installing collected packages: uwsgi
 Running setup.py install for uwsgi: started
   Running setup.py install for uwsgi: finished with status 'done'
Successfully installed uwsgi-2.0.14
Untar re-exec error: exit status 1: output: write /usr/share/perl/5.20.2/Locale/Codes/Currency_Codes.pm: no space left on device
[info] build failed (exit code 1)

I do not understand why I am receiving this error since I have plenty of disk and i-nodes on the system:

[root@ip-172-31-46-20 ec2-user]# df -ih
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/xvda1        64M   43K   64M    1% /
devtmpfs         996K   541  996K    1% /dev
tmpfs            999K     1  999K    1% /dev/shm
[root@ip-172-31-46-20 ec2-user]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1     1008G  1.4G 1007G   1% /
devtmpfs        3.9G  116K  3.9G   1% /dev
tmpfs           3.9G     0  3.9G   0% /dev/shm

I am running on an Amazon EC2 instance managed by Elastic Beanstalk. Here is the drone.yml file I am running:

build:
image: python:2.7
commands:
- pip install pep8
- pep8 mysite
publish:
ecr:
access_key: "$$AWS_ACCESS_KEY"
secret_key: "$$AWS_SECRET_KEY"
region: us-east-2
repo: django-app-test
tag:
- latest
- “$$DRONE_BRANCH”
- “$$DRONE_COMMIT”
- "$$DRONE_BUILD_NUMBER"
storage_driver: vfs
environment:
- DOCKER_LAUNCH_DEBUG=true

And this is the output of docker info:

docker info

Containers: 2
Running: 2
Paused: 0
Stopped: 0
Images: 2
Server Version: 1.11.2
Storage Driver: devicemapper
Pool Name: docker-docker–pool
Pool Blocksize: 524.3 kB
Base Device Size: 107.4 GB
Backing Filesystem: ext4
Data file:
Metadata file:
Data Space Used: 2.897 GB
Data Space Total: 12.73 GB
Data Space Available: 9.837 GB
Metadata Space Used: 991.2 kB
Metadata Space Total: 16.78 MB
Metadata Space Available: 15.79 MB
Udev Sync Supported: true
Deferred Removal Enabled: true
Deferred Deletion Enabled: true
Deferred Deleted Device Count: 0
Library Version: 1.02.93-RHEL7 (2015-01-28)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge null host
Kernel Version: 4.4.19-29.55.amzn1.x86_64
Operating System: Amazon Linux AMI 2016.03
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.8 GiB
Name: ip-172-31-46-20
ID: QHIS:DZZR:NKVX:YXHW:VGIK:FP4Z:MRVZ:O2E7:SRLC:H2TS:MDQ3:J7N6
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/

2

this is because you are using the vfs storage driver which makes a full copy of every layer when building your image, potentially consuming huge amounts of disk space. Ideally you should be using aufs or overlay, but to do so you need to make sure your host machines supports these filesystems.

1 Like
3

For reference see this stackoverflow thread Why is the Docker vfs storage backend not considered suitable for production? - Stack Overflow

The vfs backend is a very simple fallback that has no copy-on-write support. Each layer is just a separate directory. Creating a new layer based on another layer is done by making a deep copy of the base layer into a new directory.

Since this backend doesn’t share diskspace use between layers, and since creating a new layer is a slow operation this is not a very practical backend. However, it still has its uses, for instance to verify other backends against, or if you need a super robust (if slow) backend that works everywhere.

4

oh and thanks for providing such a detailed description of your issue and including the docker info. If everyone provided this much detail when asking questions it would make support so much easier :slight_smile:

5

Hi there Im having the same issue with devicemapper storage type, this is my docker info

Containers: 9
Running: 8
Paused: 0
Stopped: 1
Images: 7
Server Version: 17.03.2-ce
Storage Driver: devicemapper
Pool Name: docker-docker–pool
Pool Blocksize: 524.3 kB
Base Device Size: 10.74 GB
Backing Filesystem: ext4
Data file:
Metadata file:
Data Space Used: 2.485 GB
Data Space Total: 318.3 GB
Data Space Available: 315.8 GB
Metadata Space Used: 1.724 MB
Metadata Space Total: 323 MB
Metadata Space Available: 321.2 MB
Thin Pool Minimum Free Space: 31.82 GB
Udev Sync Supported: true
Deferred Removal Enabled: true
Deferred Deletion Enabled: true
Deferred Deleted Device Count: 0
Library Version: 1.02.135-RHEL7 (2016-11-16)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 4.9.43-17.39.amzn1.x86_64
Operating System: Amazon Linux AMI 2017.03
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.677 GiB
Name: ip-172-20-130-6
ID: U6ND:BG2N:JWLN:VKFQ:2OL2:GKY5:HO6S:SFP5:2NV7:SXAU:6ZCO:KI2W
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

lvs command output
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
docker-pool docker twi-aot— 296.39g 0.78 0.53

vgs command output
VG #PV #LV #SN Attr VSize VFree
docker 1 1 0 wz–n- 300.00g 3.00g

The only thing that makes me wonder if it’s wrong is the size of each container (which is always set on 10 gb)
What could be wrong?

6

Also changed the Base Device Size from 10.47Gb to 21.47 GB without lucky, any ideas?

7

I believe my response to the original question in this thread still applies.

In general, device mapper does not work with the official docker:dind image, which is the base image used by the ecr plugin. Because device mapper is not supported, the docker daemon will fallback to vfs which is not a space efficient driver.

Please see my troubleshooting guide:
http://docs.drone.io/error-no-space-left-on-device/

Note that issues with device mapper and fallback to VFS are docker defaults. There is nothing that Drone can do to change this. You would need to install the appropriate drivers (overlay) on your host machine.

8

Yeah after changing to overlay2 (which I though wasn;t supported by Amzn Linux) it started working, thanks!

1 Like