Improvements to autoscaler to automatically re-attempt fail pull [pending patch]

[ashwilliams1 (Ash Williams)]

This is my response to comments made in Can I use volumes in Drone Cloud in order to map /var/run/docker.sock?

@trondhindenes

It would be good if more robustness was built-in to the drone-autoscaler instead, so that we didn’t have to do this. For example, perform a docker pull (with appropirate retry logic) and only when that succeeds mark the runner as “ready for service”.

please consider sending a pull request that retries the docker pull, on failure, using a limited backoff.

@trondhindenes

just to note: We’ve also seen other situations where an ec2 instance becomes unhealthy, but drone has no way of checking/discarding unhealthy nodes

drone does have an option to ping unhealthy nodes and then clean them up. You need to enable the following feature flags to get this behavior:

DRONE_ENABLE_PINGER=true
DRONE_ENABLE_REAPER=true

this enables the pinger which will ping the instance every N minutes to check whether or not it is healthy. If it cannot ping the instance, it is placed in an errored state. The reaper then periodically cleans up instances marked in errored states.