Builds not finishing

Drone Support
1

I’ve got a bit of a problem where my builds do not seem to be finishing like ever.

I do see all the build steps exiting in the build logs but still the build stays in “running” state. No relevant logs found either on server or agent. Agent is started with DRONE_DEBUG=true env and I can see the build start event there but nothing more. In some of the cases eventually the UI shows:

ERROR: terminal inactive for 15m0s, build cancelled

Any idea where to look for more info?

The problematic steps seems to be this one:

integration-test:
    image: jnummelin/build-tools
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    commands:
      - docker-compose up -d --build
      - bats tests/
      - docker-compose stop && docker-compose rm -f -v && sleep 5 && echo "Integration testing done"

Weird part is that in the build logs I can see this exiting with:

exit code 0

And on the node where agent runs I don’t see any leftover containers that could even be hanging on any terminal.

2

While doing some debugging on this, it seems that some of the output from docker-compose commands come “out-of-place”. I mean that in the build logs I didn’t see all the output done events that it prints out but still saw the exit code.

This makes me think if the issue could be that agent sees some of the output AFTER the command has already exited and thus thinks the step is still alive?

I was able to workaround the issue by using the commands like so:

commands:
      - docker-compose up -d --build
      - echo "**** Begin integration tests ****"
      - bats tests/
      - echo "**** End integration tests ****"
      - docker-compose stop > /dev/null 2>&1
      - docker-compose rm -f -v > /dev/null 2>&1

So there will be no output and hence agent does not hang on the step.

Any pointers for a real fix would be highly appreciated as this hurts my eye a bit. :slight_smile:

3

This is hanging your shell because it is waiting for docker-compose to complete. If you run the below script on a linux system it will hang indefinitely waiting for the backgrounded process to complete:

#!/bin/bash

python -m SimpleHTTPServer 8000 &
echo started
exit 0

This is unfortunately a bash (on Linux) issue. It won’t exit a script if there are processes still running in the background.

I don’t think the output is the root cause. The difference is that you are stopping the docker-compose processes, allowing the terminal to exit.

4

Thanks for the response.

The issue I don’t get is that how could bash have processes on the background as the containers are spawned on the host side as I’m mounting the hosts docker socket to the build container? See:

bash-4.3# docker-compose up -d --build
Building web
(output omitted)
Successfully built fa8a1d925302
workspace_mongodb_1 is up-to-date
Starting workspace_web_1
bash-4.3# ps faux
PID   USER     TIME   COMMAND
    1 root       0:00 bash
   31 root       0:00 ps faux

As you see, there’s no background processes in the container running at all.

Any tips how I could debug this further?