Cancelling a buid doesn't stop the containers

Drone Support
1

Hi,

When I cancel a build on the interface, the status is updated in the UI but containers are still running. It keeps the place in the queue so, because I limited the number of simultaneous build, if I cancel 4 build I have to wait them to start a new one. I have to kill containers manually to release the place.

The problem looks similar to this other one : Canceling running builds does not successfully stop container but there’s no answer.

Here is the result of the request when I cancel a buid :

{
  "JSON": {
    "action": "",
      "after": "806aca4d7ded658a087e8be0b053b6bf19c3c726",
      "author_avatar": "https://secure.gravatar.com/avatar/0ab45d63adae19f314668733071e1674?s=80&d=identicon",
      "author_email": "",
      "author_login": "User.Name",
      "author_name": "User.Name",
      "before": "",
      "created": 1569875380,
      "event": "push",
      "finished": 1569876638,
      "id": 530,
      "link": "https://git.mydomain.com/project/project/commit/806aca4d7ded658a087e8be0b053b6bf19c3c726",
      "message": "[MIGRATION] Better: Update  Attachment with deprecated attachable_type 'MobileChat::Message' RD-11323\n",
      "number": 110,
      "ref": "refs/heads/rd-11323-migrate-attachment-for-mobile-chat",
      "repo_id": 2133,
      "sender": "User.Name",
      "source": "rd-11323-migrate-attachment-for-mobile-chat",
      "source_repo": "",
      "stages": [
      {
        "arch": "amd64",
        "build_id": 530,
        "created": 1569875380,
        "errignore": false,
        "exit_code": 0,
        "id": 512,
        "kind": "pipeline",
        "machine": "end01-t01-utl04",
        "name": "default",
        "number": 1,
        "on_failure": false,
        "on_success": true,
        "os": "linux",
        "repo_id": 2133,
        "started": 1569875730,
        "status": "killed",
        "steps": [
        {
          "exit_code": 130,
          "id": 5139,
          "name": "mongo",
          "number": 1,
          "started": 1569875730,
          "status": "killed",
          "step_id": 512,
          "stopped": 1569876638,
          "version": 3
        },
        {
          "exit_code": 130,
          "id": 5140,
          "name": "redis",
          "number": 2,
          "started": 1569875730,
          "status": "killed",
          "step_id": 512,
          "stopped": 1569876638,
          "version": 3
        },
        {
          "exit_code": 130,
          "id": 5141,
          "name": "elasticsearch",
          "number": 3,
          "started": 1569875730,
          "status": "killed",
          "step_id": 512,
          "stopped": 1569876638,
          "version": 3
        },
        {
          "exit_code": 130,
          "id": 5142,
          "name": "memcache",
          "number": 4,
          "started": 1569875730,
          "status": "killed",
          "step_id": 512,
          "stopped": 1569876638,
          "version": 3
        },
        {
          "exit_code": 0,
          "id": 5143,
          "name": "git",
          "number": 5,
          "started": 1569875730,
          "status": "success",
          "step_id": 512,
          "stopped": 1569875750,
          "version": 3
        },
        {
          "exit_code": 0,
          "id": 5144,
          "name": "restore-cache",
          "number": 6,
          "started": 1569875750,
          "status": "success",
          "step_id": 512,
          "stopped": 1569875843,
          "version": 3
        },
        {
          "exit_code": 0,
          "id": 5145,
          "name": "prepare",
          "number": 7,
          "started": 1569875843,
          "status": "success",
          "step_id": 512,
          "stopped": 1569875862,
          "version": 3
        },
        {
          "exit_code": 130,
          "id": 5146,
          "name": "rspec",
          "number": 8,
          "started": 1569875862,
          "status": "killed",
          "step_id": 512,
          "stopped": 1569876638,
          "version": 3
        },
        {
          "exit_code": 130,
          "id": 5147,
          "name": "rspec-features",
          "number": 9,
          "started": 1569875862,
          "status": "killed",
          "step_id": 512,
          "stopped": 1569876638,
          "version": 3
        },
        {
          "exit_code": 0,
          "id": 5148,
          "name": "jasmine",
          "number": 10,
          "started": 1569875862,
          "status": "success",
          "step_id": 512,
          "stopped": 1569876020,
          "version": 3
        },
        {
          "exit_code": 0,
          "id": 5149,
          "name": "assets-precompilation",
          "number": 11,
          "started": 1569875862,
          "status": "success",
          "step_id": 512,
          "stopped": 1569876167,
          "version": 3
        },
        {
          "exit_code": 0,
          "id": 5150,
          "name": "rubocop",
          "number": 12,
          "started": 1569875862,
          "status": "success",
          "step_id": 512,
          "stopped": 1569875892,
          "version": 3
        },
        {
          "exit_code": 130,
          "id": 5151,
          "name": "code-coverage",
          "number": 13,
          "started": 1569876638,
          "status": "skipped",
          "step_id": 512,
          "stopped": 1569876638,
          "version": 2
        },
        {
          "exit_code": 130,
          "id": 5152,
          "name": "rebuild-cache",
          "number": 14,
          "started": 1569876638,
          "status": "skipped",
          "step_id": 512,
          "stopped": 1569876638,
          "version": 2
        }
        ],
        "stopped": 1569876638,
        "type": "docker",
        "updated": 1569875730,
        "version": 4
      }
    ],
    "started": 1569875730,
    "status": "killed",
    "target": "rd-11323-migrate-attachment-for-mobile-chat",
    "timestamp": 0,
    "trigger": "my.name",
    "updated": 1569875730,
    "version": 3
  },
    "Response payload": {
      "EDITOR_CONFIG": {
        "mode": "application/json",
        "text": "{\"id\":530,\"repo_id\":2133,\"trigger\":\"my.namel\",\"number\":110,\"status\":\"killed\",\"event\":\"push\",\"action\":\"\",\"link\":\"https://git.mydomain.com/project/project/commit/806aca4d7ded658a087e8be0b053b6bf19c3c726\",\"timestamp\":0,\"message\":\"[MIGRATION] Better: Update  Attachment with deprecated attachable_type 'MobileChat::Message' RD-11323\\n\",\"before\":\"\",\"after\":\"806aca4d7ded658a087e8be0b053b6bf19c3c726\",\"ref\":\"refs/heads/rd-11323-migrate-attachment-for-mobile-chat\",\"source_repo\":\"\",\"source\":\"rd-11323-migrate-attachment-for-mobile-chat\",\"target\":\"rd-11323-migrate-attachment-for-mobile-chat\",\"author_login\":\"User.Name\",\"author_name\":\"User.Name\",\"author_email\":\"\",\"author_avatar\":\"https://secure.gravatar.com/avatar/0ab45d63adae19f314668733071e1674?s=80\\u0026d=identicon\",\"sender\":\"User.Name\",\"started\":1569875730,\"finished\":1569876638,\"created\":1569875380,\"updated\":1569875730,\"version\":3,\"stages\":[{\"id\":512,\"repo_id\":2133,\"build_id\":530,\"number\":1,\"name\":\"default\",\"kind\":\"pipeline\",\"type\":\"docker\",\"status\":\"killed\",\"errignore\":false,\"exit_code\":0,\"machine\":\"end01-t01-utl04\",\"os\":\"linux\",\"arch\":\"amd64\",\"started\":1569875730,\"stopped\":1569876638,\"created\":1569875380,\"updated\":1569875730,\"version\":4,\"on_success\":true,\"on_failure\":false,\"steps\":[{\"id\":5139,\"step_id\":512,\"number\":1,\"name\":\"mongo\",\"status\":\"killed\",\"exit_code\":130,\"started\":1569875730,\"stopped\":1569876638,\"version\":3},{\"id\":5140,\"step_id\":512,\"number\":2,\"name\":\"redis\",\"status\":\"killed\",\"exit_code\":130,\"started\":1569875730,\"stopped\":1569876638,\"version\":3},{\"id\":5141,\"step_id\":512,\"number\":3,\"name\":\"elasticsearch\",\"status\":\"killed\",\"exit_code\":130,\"started\":1569875730,\"stopped\":1569876638,\"version\":3},{\"id\":5142,\"step_id\":512,\"number\":4,\"name\":\"memcache\",\"status\":\"killed\",\"exit_code\":130,\"started\":1569875730,\"stopped\":1569876638,\"version\":3},{\"id\":5143,\"step_id\":512,\"number\":5,\"name\":\"git\",\"status\":\"success\",\"exit_code\":0,\"started\":1569875730,\"stopped\":1569875750,\"version\":3},{\"id\":5144,\"step_id\":512,\"number\":6,\"name\":\"restore-cache\",\"status\":\"success\",\"exit_code\":0,\"started\":1569875750,\"stopped\":1569875843,\"version\":3},{\"id\":5145,\"step_id\":512,\"number\":7,\"name\":\"prepare\",\"status\":\"success\",\"exit_code\":0,\"started\":1569875843,\"stopped\":1569875862,\"version\":3},{\"id\":5146,\"step_id\":512,\"number\":8,\"name\":\"rspec\",\"status\":\"killed\",\"exit_code\":130,\"started\":1569875862,\"stopped\":1569876638,\"version\":3},{\"id\":5147,\"step_id\":512,\"number\":9,\"name\":\"rspec-features\",\"status\":\"killed\",\"exit_code\":130,\"started\":1569875862,\"stopped\":1569876638,\"version\":3},{\"id\":5148,\"step_id\":512,\"number\":10,\"name\":\"jasmine\",\"status\":\"success\",\"exit_code\":0,\"started\":1569875862,\"stopped\":1569876020,\"version\":3},{\"id\":5149,\"step_id\":512,\"number\":11,\"name\":\"assets-precompilation\",\"status\":\"success\",\"exit_code\":0,\"started\":1569875862,\"stopped\":1569876167,\"version\":3},{\"id\":5150,\"step_id\":512,\"number\":12,\"name\":\"rubocop\",\"status\":\"success\",\"exit_code\":0,\"started\":1569875862,\"stopped\":1569875892,\"version\":3},{\"id\":5151,\"step_id\":512,\"number\":13,\"name\":\"code-coverage\",\"status\":\"skipped\",\"exit_code\":130,\"started\":1569876638,\"stopped\":1569876638,\"version\":2},{\"id\":5152,\"step_id\":512,\"number\":14,\"name\":\"rebuild-cache\",\"status\":\"skipped\",\"exit_code\":130,\"started\":1569876638,\"stopped\":1569876638,\"version\":2}]}]}\n"
      }
    }
}

I tried to send a SIGINT signals to processes I want to close and it closes them.

What can I do?

2

I inspected a bit more and it seems I can kill a process with SIGINT on locally but it doesn’t works on a container on the CI. SIGKILL works.

3

I tried again and SIGINT works. It seems containers doesn’t receive the signal.

4

I’m in a similar situation here. Using a config with multiple pipelines, which mount the Docker socket as a volume. Haven’t had much time to debug this

5

I’m hitting this issue too. Somehow the drone-runner-docker fails to kill containers.

The UI shows the build as “cancelled”, but the containers are still running.
Subsequent builds are stuck in “pending” state for lack of runners.

I’m having pipelines with one service container (running autossh to keep a remote port available locally) and some build steps.

Looking at the drone-runner-docker code, it calls client.DockerKill(), which should be sufficient.

Manually running “docker kill” or sending SIGINT/SIGKILL on the runner kills the container, but drone doesn’t realize.

Are there any requirements on the containers for drone killing them to work?
How can I debug this?

6

Let me add that this only happens sometimes (maybe less than half of the time).

7

My service container was using a bash script, using “exec” to finally start the service. Removing that exec seems to make killing by drone reliable.