# "stopped" agents remain in drone server ls

**URL:** <https://drone.discourse.group/t/stopped-agents-remain-in-drone-server-ls/9133>\
**Category:** Drone Support\
**Created:** [October 7, 2019, 11:03pm UTC](https://drone.discourse.group/t/stopped-agents-remain-in-drone-server-ls/9133 "2019-10-07T23:03:19Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![k2n](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/k2n/32/4414_2.png) [@k2n](https://drone.discourse.group/u/k2n)\
**Post date:** [October 7, 2019, 11:03pm UTC](https://drone.discourse.group/t/stopped-agents-remain-in-drone-server-ls/9133/1 "2019-10-07T23:03:19Z")

</div>

I observe multiple agents are stuck in `stopped` status, and cannot delete even with `drone server destroy --force`.

```auto
👉 bin/drone server ls -a -H -l
Name Address State Created
agent-DnFRUDrY 10.0.12.148 stopped 10 hours ago
agent-CQBQ10mi 10.0.12.208 stopped 10 hours ago
agent-Q9rxIhb3 10.0.12.250 stopped 9 hours ago
agent-iNblnE3h 10.0.12.132 stopped 8 hours ago
agent-jbwphpaw 10.0.12.51 stopped 7 hours ago
agent-pxJXYnsp 10.0.12.206 stopped 7 hours ago
agent-Zj0FPMeC 10.0.12.108 stopped 6 hours ago
agent-SNpn3AAk 10.0.12.122 stopped 6 hours ago
agent-DDuEO8V7 10.0.12.174 stopped 6 hours ago
agent-sjaBShDY 10.0.12.76 stopped 6 hours ago
agent-2Pccjm7k 10.0.12.210 stopped 5 hours ago
agent-fA8Jf6QC 10.0.12.32 stopped 5 hours ago
agent-LjqZQjFL 10.0.12.33 stopped 5 hours ago
agent-MmBZUozk 10.0.12.110 running 2 hours ago
agent-qZ1oW4kw 10.0.12.152 stopped 2 hours ago
agent-QO6awaAd 10.0.12.237 stopped 2 hours ago
agent-ILClFdZI 10.0.12.108 stopped 2 hours ago
agent-JLPdPr5A 10.0.12.249 stopped 2 hours ago
agent-TvZhvFyu 10.0.12.75 running About an hour ago
agent-F8ChaoYK 10.0.12.226 stopped About an hour ago

```

I grep’ed the autoscaler log for agent `agent-DnFRUDrY`. At the end of the log, I tried to kill the agent but it failed with the message “server no longer exists. nothing to destroy”.

Why do they remain `stopped` status? Is there a way to clean them up?

Here is the configuration of autoscaler. Please note I added `DRONE_ENABLE_REPAIR=true` following the advice [Drone-agent in error status](http://discuss.harness.io/t/drone-agent-in-error-status/5852/2)

```auto
sudo docker run -d \
  -e DRONE_POOL_MIN=0 \
  -e DRONE_POOL_MAX=20 \
  -e DRONE_CAPACITY_BUFFER=0 \
  -e DRONE_POOL_MIN_AGE=30m \
  -e DRONE_INTERVAL=10s \
  -e DRONE_SERVER_PROTO=https \
  -e DRONE_INSTALL_CHECK_INTERVAL=60s \
  -e DRONE_ENABLE_REAPER=true \
  -e DRONE_AMAZON_IMAGE=ami- ***** \
  -e DRONE_SERVER_HOST=drone-ci. ******* \
  -e DRONE_SERVER_TOKEN= ****** \
  -e DRONE_AGENT_CONCURRENCY=12 \
  -e DRONE_AGENT_TOKEN= ******** \
  -e DRONE_AMAZON_REGION=us-west-2 \
  -e DRONE_AMAZON_SUBNET_ID=subnet- ***** \
  -e DRONE_AMAZON_SECURITY_GROUP=sg- ****** \
  -e DRONE_AMAZON_INSTANCE=c5.9xlarge \
  -e DRONE_AMAZON_SSHKEY=drone \
  -e DRONE_AMAZON_TAGS=type:drone-agent \
  -e AWS_ACCESS_KEY_ID= ***** \
  -e AWS_SECRET_ACCESS_KEY= ****** \
  -e DRONE_AMAZON_PRIVATE_IP=true \
  -p 8080:8080 \
  --restart=always \
  --name=autoscaler \
  drone/autoscaler

```

Here is the except of the log from `docker logs autoscaler`. I squashed consecutive outputs.

```auto
{"level":"debug","region":"us-west-2","image":"ami- **********","size":"c5.9xlarge","name":"agent-DnFRUDrY","time":"2019-10-07T13:16:54Z","message":"instance create"}
{"level":"info","region":"us-west-2","image":"ami- **********","size":"c5.9xlarge","name":"agent-DnFRUDrY","name":"agent-DnFRUDrY","time":"2019-10-07T13:16:54Z","message":"instance create success"}
{"level":"debug","region":"us-west-2","image":"ami- **********","size":"c5.9xlarge","name":"agent-DnFRUDrY","name":"agent-DnFRUDrY","time":"2019-10-07T13:16:54Z","message":"check instance network"}
{"level":"debug","region":"us-west-2","image":"ami- **********","size":"c5.9xlarge","name":"agent-DnFRUDrY","name":"agent-DnFRUDrY","ip":"10.0.12.148","time":"2019-10-07T13:16:54Z","message":"instance network ready"}
{"level":"debug","server":"agent-DnFRUDrY","time":"2019-10-07T13:16:54Z","message":"provisioned server"}
{"level":"debug","ip":"10.0.12.148","name":"agent-DnFRUDrY","name":"agent-DnFRUDrY","time":"2019-10-07T13:17:02Z","message":"check docker connectivity"}
{"level":"debug","ip":"10.0.12.148","name":"agent-DnFRUDrY","name":"agent-DnFRUDrY","time":"2019-10-07T13:17:02Z","message":"connecting to docker"}
{"level":"debug","ip":"10.0.12.148","name":"agent-DnFRUDrY","error":"Cannot connect to the Docker daemon at https://10.0.12.148:2376. Is the docker daemon running?","name":"agent-DnFRUDrY","time":"2019-10-07T13:17:34Z","message":"cannot connect, retry in 1m0s"}
{"level":"debug","ip":"10.0.12.148","name":"agent-DnFRUDrY","name":"agent-DnFRUDrY","time":"2019-10-07T13:18:34Z","message":"connecting to docker"}
{"level":"debug","ip":"10.0.12.148","name":"agent-DnFRUDrY","image":"drone/agent:1","time":"2019-10-07T13:18:34Z","message":"pull docker image"}
{"level":"debug","ip":"10.0.12.148","name":"agent-DnFRUDrY","image":"drone/agent:1","time":"2019-10-07T13:18:36Z","message":"create agent container"}
{"level":"debug","ip":"10.0.12.148","name":"agent-DnFRUDrY","image":"drone/agent:1","time":"2019-10-07T13:18:36Z","message":"start the agent container"}
{"level":"debug","ip":"10.0.12.148","name":"agent-DnFRUDrY","image":"drone/agent:1","time":"2019-10-07T13:18:37Z","message":"agent container started"}
{"level":"debug","id":"M5IYIZ9F4Edq5bsq","server":"agent-DnFRUDrY","time":"2019-10-07T13:23:05Z","message":"server is busy"}
...
{"level":"debug","id":"UbYimZqr1mTZvtNG","server":"agent-DnFRUDrY","time":"2019-10-07T13:28:05Z","message":"server is busy"}
{"level":"debug","id":"wXo1GZnRLtqxUzbc","server":"agent-DnFRUDrY","age":691833.715306,"min-age":1800000,"time":"2019-10-07T13:28:15Z","message":"server min-age not reached"}
...
{"level":"debug","id":"EvIYmYrRFWm3Htez","server":"agent-DnFRUDrY","age":1392959.755046,"min-age":1800000,"time":"2019-10-07T13:39:56Z","message":"server min-age not reached"}
{"level":"debug","id":"uTuixOnbu64rOSaG","server":"agent-DnFRUDrY","time":"2019-10-07T13:43:37Z","message":"server is busy"}
...
{"level":"debug","id":"mcUApEgSXmA2gKhb","server":"agent-DnFRUDrY","time":"2019-10-07T13:44:27Z","message":"server is busy"}
...
{"level":"debug","id":"n1webanNSJcljaY3","server":"agent-DnFRUDrY","time":"2019-10-07T13:46:47Z","message":"server is idle"}
{"level":"debug","server":"agent-DnFRUDrY","time":"2019-10-07T13:46:52Z","message":"destroying server"}
{"level":"debug","id":"i-00aec8d18ed1422f8","ip":"10.0.12.148","name":"agent-DnFRUDrY","zone":"us-west-2a","time":"2019-10-07T13:46:53Z","message":"terminate instance"}
{"level":"debug","id":"i-00aec8d18ed1422f8","ip":"10.0.12.148","name":"agent-DnFRUDrY","zone":"us-west-2a","time":"2019-10-07T13:46:53Z","message":"terminated"}
{"level":"debug","server":"agent-DnFRUDrY","time":"2019-10-07T13:46:53Z","message":"destroyed server"}
{"level":"debug","ip":"10.0.2.218","path":"/api/servers/agent-DnFRUDrY","method":"DELETE","request_id":"bmdrrabc0uvqfbtt7u40","username":"k2n","username":"k2n","time":"2019-10-07T22:35:21Z","message":"user authorized"}
{"level":"debug","ip":"10.0.1.142","path":"/api/servers/agent-DnFRUDrY","method":"GET","request_id":"bmdrrajc0uvqfbtt7u4g","username":"k2n","username":"k2n","time":"2019-10-07T22:35:22Z","message":"user authorized"}
{"level":"debug","server":"agent-DnFRUDrY","time":"2019-10-07T22:35:24Z","message":"destroying server"}
{"level":"debug","ip":"10.0.1.142","path":"/api/servers/agent-DnFRUDrY?force=true","method":"DELETE","request_id":"bmdrs7rc0uvqfbtt7uu0","username":"k2n","username":"k2n","time":"2019-10-07T22:37:19Z","message":"user authorized"}
{"level":"debug","ip":"10.0.2.218","path":"/api/servers/agent-DnFRUDrY","method":"GET","request_id":"bmdrs7rc0uvqfbtt7uug","username":"k2n","username":"k2n","time":"2019-10-07T22:37:19Z","message":"user authorized"}
{"level":"debug","server":"agent-DnFRUDrY","time":"2019-10-07T22:37:25Z","message":"destroying server"}
{"level":"warn","error":"Cannot connect to the Docker daemon at https://10.0.12.148:2376. Is the docker daemon running?","server":"agent-DnFRUDrY","time":"2019-10-07T22:37:36Z","message":"cannot stop the agent"}
{"level":"debug","id":"i-00aec8d18ed1422f8","ip":"10.0.12.148","name":"agent-DnFRUDrY","zone":"us-west-2a","time":"2019-10-07T22:37:36Z","message":"terminate instance"}
{"level":"debug","id":"i-00aec8d18ed1422f8","ip":"10.0.12.148","name":"agent-DnFRUDrY","zone":"us-west-2a","time":"2019-10-07T22:37:36Z","message":"instance does not exist"}
{"level":"info","state":"error","server":"agent-DnFRUDrY","time":"2019-10-07T22:37:36Z","message":"server no longer exists. nothing to destroy"}
{"level":"warn","error":"Cannot connect to the Docker daemon at https://10.0.12.148:2376. Is the docker daemon running?","server":"agent-DnFRUDrY","time":"2019-10-07T22:39:34Z","message":"cannot stop the agent"}
{"level":"debug","id":"i-00aec8d18ed1422f8","ip":"10.0.12.148","name":"agent-DnFRUDrY","zone":"us-west-2a","time":"2019-10-07T22:39:34Z","message":"terminate instance"}
{"level":"debug","id":"i-00aec8d18ed1422f8","ip":"10.0.12.148","name":"agent-DnFRUDrY","zone":"us-west-2a","time":"2019-10-07T22:39:34Z","message":"instance does not exist"}
{"level":"info","state":"error","server":"agent-DnFRUDrY","time":"2019-10-07T22:39:34Z","message":"server no longer exists. nothing to destroy"}
{"level":"debug","ip":"10.0.2.218","path":"/api/servers/agent-DnFRUDrY","method":"DELETE","request_id":"bmdrtubc0uvqfbtt7vbg","username":"k2n","username":"k2n","time":"2019-10-07T22:40:57Z","message":"user authorized"}
{"level":"debug","ip":"10.0.1.142","path":"/api/servers/agent-DnFRUDrY","method":"GET","request_id":"bmdrtubc0uvqfbtt7vc0","username":"k2n","username":"k2n","time":"2019-10-07T22:40:57Z","message":"user authorized"}
{"level":"debug","server":"agent-DnFRUDrY","time":"2019-10-07T22:41:05Z","message":"destroying server"}
{"level":"debug","ip":"10.0.2.218","path":"/api/servers/agent-DnFRUDrY","method":"DELETE","request_id":"bmdruk3c0uvqfbtt7vgg","username":"k2n","username":"k2n","time":"2019-10-07T22:42:24Z","message":"user authorized"}
{"level":"debug","ip":"10.0.1.142","path":"/api/servers/agent-DnFRUDrY","method":"GET","request_id":"bmdruk3c0uvqfbtt7vh0","username":"k2n","username":"k2n","time":"2019-10-07T22:42:24Z","message":"user authorized"}
{"level":"debug","server":"agent-DnFRUDrY","time":"2019-10-07T22:42:25Z","message":"destroying server"}
{"level":"warn","error":"Cannot connect to the Docker daemon at https://10.0.12.148:2376. Is the docker daemon running?","server":"agent-DnFRUDrY","time":"2019-10-07T22:43:16Z","message":"cannot stop the agent"}
{"level":"debug","id":"i-00aec8d18ed1422f8","ip":"10.0.12.148","name":"agent-DnFRUDrY","zone":"us-west-2a","time":"2019-10-07T22:43:16Z","message":"terminate instance"}
{"level":"debug","id":"i-00aec8d18ed1422f8","ip":"10.0.12.148","name":"agent-DnFRUDrY","zone":"us-west-2a","time":"2019-10-07T22:43:16Z","message":"instance does not exist"}
{"level":"info","state":"error","server":"agent-DnFRUDrY","time":"2019-10-07T22:43:16Z","message":"server no longer exists. nothing to destroy"}
{"level":"warn","error":"Cannot connect to the Docker daemon at https://10.0.12.148:2376. Is the docker daemon running?","server":"agent-DnFRUDrY","time":"2019-10-07T22:44:35Z","message":"cannot stop the agent"}
{"level":"debug","id":"i-00aec8d18ed1422f8","ip":"10.0.12.148","name":"agent-DnFRUDrY","zone":"us-west-2a","time":"2019-10-07T22:44:35Z","message":"terminate instance"}
{"level":"debug","id":"i-00aec8d18ed1422f8","ip":"10.0.12.148","name":"agent-DnFRUDrY","zone":"us-west-2a","time":"2019-10-07T22:44:35Z","message":"instance does not exist"}
{"level":"info","state":"error","server":"agent-DnFRUDrY","time":"2019-10-07T22:44:35Z","message":"server no longer exists. nothing to destroy"}

```

---

<div class="post-metadata">

**Author:** ![bradrydzewski](https://yyz1.discourse-cdn.com/flex003/user_avatar/drone.discourse.group/bradrydzewski/32/3513_2.png) [@bradrydzewski](https://drone.discourse.group/u/bradrydzewski)\
**Post date:** [October 8, 2019, 12:10am UTC](https://drone.discourse.group/t/stopped-agents-remain-in-drone-server-ls/9133/2 "2019-10-08T00:10:51Z")

</div>

> I observe multiple agents are stuck in `stopped` status,

The agents are not stuck … stopped agents are purged from the database every 24 hours [1]. The records are kept in the database for 24 hours for audit trail purposes. Stopped instances are excluded from capacity calculations [2].

If you do not want to see them in the results, you can omit `-a`

[1] [https://github.com/drone/autoscaler/blob/master/engine/engine.go#L242:L258](https://github.com/drone/autoscaler/blob/master/engine/engine.go#L242:L258)  
[2] [https://github.com/drone/autoscaler/blob/master/engine/planner.go#L243](https://github.com/drone/autoscaler/blob/master/engine/planner.go#L243)