Hanging at `git fetch` with private repo on gitlab.com

Drone Support
1

Hello, I am having trouble getting DroneCI initially setup. I am using private repos on gitlab.com.

What Works
I can login, sync my account, activeate repos, etc. Most basic functionality seems to be working.

Whats the Problem
When I trigger a build, it hangs on the fetch step. I get output similar to this on every build.

+ git init
Initialized empty Git repository in /drone/src/gitlab.com/<group-name>/<repo-name>/.git/
+ git remote add origin https://gitlab.com/<group-name>/<repo-name>.git
+ git fetch --no-tags origin +refs/heads/develop:
fatal: unable to access 'https://gitlab.com/<group-name>/<repo-name>.git/': LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to gitlab.com:443 
exit status 128

I’m not sure if it is an authentication issue, maybe I don’t have things configured correctly? I have tried a few variations on configurations and can not seem to make any progress on this. I would really appreciate any help.

Enviornment
I am running DroneCI on kubernetes (installed via helm)

// Install Command
helm install 	--wait --name builder -f settings.yaml incubator/drone

// settings.yaml
persistence:
  storageClass: default
ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: "public"
  hosts:
    - <my-domain>
  tls:
  - hosts:
    - <my-domain>
    secretName: <secret-name>
server:
  env:
    DRONE_PROVIDER: gitlab
    DRONE_GITLAB: true
    DRONE_GITLAB_CLIENT: <application-id> 
    DRONE_GITLAB_SECRET: <secret>
    DRONE_GITLAB_URL: https://gitlab.com
    DRONE_OPEN: true
    DRONE_ORGS: <group-name>
service:
  httpPort:  8000
2

This error comes form the standard git client (not Drone) which is running inside a standard alpine container. The error seems to indicate some sort of SSL connection problem, perhaps related to a proxy or network configuration on the host machine? But just a guess. Have you googled the error message to research further?

3

Note that I can run the plugin directly on my machine, and it clones a repository from gitlab.com without any issues. That would lead me to believe the issue is specific to your host machine network configuration. I recommend checking network and proxy settings on the host machine. I would also verify that your configuration supports user-defined networks.

$ docker run --rm \
>   -e DRONE_REMOTE_URL=https://gitlab.com/gitlab-org/gitlab-test.git \
>   -e DRONE_WORKSPACE=/drone/src \
>   -e DRONE_BUILD_EVENT=push \
>   -e DRONE_COMMIT_SHA=60ecb67744cb56576c30214ff52294f8ce2def98 \
>   -e DRONE_COMMIT_REF=refs/heads/master \
>   plugins/git
Unable to find image 'plugins/git:latest' locally
latest: Pulling from plugins/git
b56ae66c2937: Pull complete 
c0fd9486e124: Pull complete 
a76803713564: Pull complete 
aa982143412b: Pull complete 
Digest: sha256:b2c63e1612886d8b9dd5f215f35ed2e7a5916450fd4ca028bd16030e66bb59ff
Status: Downloaded newer image for plugins/git:latest
+ git init
Initialized empty Git repository in /drone/src/.git/
+ git remote add origin https://gitlab.com/gitlab-org/gitlab-test.git
+ git fetch --no-tags origin +refs/heads/master:
From https://gitlab.com/gitlab-org/gitlab-test
 * branch            master     -> FETCH_HEAD
 * [new branch]      master     -> origin/master
+ git reset --hard -q 60ecb67744cb56576c30214ff52294f8ce2def98
+ git submodule update --init --recursive
Submodule 'gitlab-grack' (https://gitlab.com/gitlab-org/gitlab-grack.git) registered for path 'gitlab-grack'
Submodule 'gitlab-shell' (https://github.com/gitlabhq/gitlab-shell.git) registered for path 'gitlab-shell'
Submodule 'six' (git://github.com/randx/six.git) registered for path 'six'
Cloning into '/drone/src/gitlab-grack'...
Cloning into '/drone/src/gitlab-shell'...
Cloning into '/drone/src/six'...
Submodule path 'gitlab-grack': checked out '645f6c4c82fd3f5e06f67134450a570b795e55a6'
Submodule 'tests/example' (git://github.com/schacon/simplegit.git) registered for path 'gitlab-grack/tests/example'
Cloning into '/drone/src/gitlab-grack/tests/example'...
Submodule path 'gitlab-grack/tests/example': checked out 'ca82a6dff817ec66f44342007202690a93763949'
Submodule path 'gitlab-shell': checked out '79bceae69cb5750d6567b223597999bfa91cb3b9'
Submodule path 'six': checked out '409f37c4f05865e4fb208c771485f211a22c4c2d'
4

I’ve done a little bit. Everything I can find seems to indicate that git needs to be upgraded, or that SSL verifications needs to be turned off.

I’m not sure how to do the first… esp with this running in kubernetes. And the second always makes me cringe when I see that as the “solution”. However for the sake of testing I gave it a shot (assuming thats what DRONE_GITLAB_SKIP_VERIFY: true would do) and… I get some different output:

+ git init
Initialized empty Git repository in /drone/src/gitlab.com/minesrv/website/.git/
+ git remote add origin https://gitlab.com/minesrv/website.git
+ git fetch --no-tags origin +refs/heads/develop:
fatal: unable to access 'https://gitlab.com/minesrv/website.git/': Operation timed out after 300033 milliseconds with 0 out of 0 bytes received
exit status 128

So… networking issues? The pod can def reach gitlab though since it can pull a list of projects… if it helps at all kubernetes is running on coreos on digital ocean. I found one reference to limiting the depth… ill try that now.

5

The pod can def reach gitlab though since it can pull a list of project

You are likely running the drone server in a pod using the standard bridge network. Drone pipelines do not use the bridge network; drone creates a user-defined network for each pipeline execution. User-defined networks and bridge networks have different behavior. The server’s ability to communicate with gitlab does not necessarily indicate that user-defined networks are working on your installation.

$ docker network create foo
$ docker run --rm \
>   -network=foo \
>   -e DRONE_REMOTE_URL=https://gitlab.com/gitlab-org/gitlab-test.git \
>   -e DRONE_WORKSPACE=/drone/src \
>   -e DRONE_BUILD_EVENT=push \
>   -e DRONE_COMMIT_SHA=60ecb67744cb56576c30214ff52294f8ce2def98 \
>   -e DRONE_COMMIT_REF=refs/heads/master \
>   plugins/git

I do not believe that skipping ssl verification or upgrading the git installation are going to solve your error. If it were an issue with the git client, I would have gotten the same error when I just ran the plugin locally (not to mention we would be getting A LOT of complaints from our community).

1 Like
6

So the example you gave me works on the worker machine when I ssh into it… is that the context drone would be running it in? Or more than likely is that happening within a docker container… and could that cause an issue? Docker inside of docker?

7

Or more than likely is that happening within a docker container

I believe the helm chart creates a dind container, which means the git clone is happening inside a container inside the dind container.

And could that cause an issue? Docker inside of docker?

I am not aware of any specific issues with this approach, but networking issues are often specific to the host machine configuration and can vary. So it is definitely possible.

I personally use the autoscaler with digital ocean, which installs instances on dedicated machines. You could host the server and autoscaler images on kubernetes, and then let the autoscaler manage agents separately.

8

That is a fantastic solution! I am working on getting that implemented now… I’ve done a little searching but do you know of any documentation on getting the autoscaler setup in kubernetes?

I’m trying to deploy the autoscaler alongside the helm install of drone… I’m starting to guess that may not work exactly… do I need to do a custom drone deploy? I know I eventually need to so I can remove the agent and such but I thought I may be able to try out the autoscaler with the current deploy I have up and running.

The issue I am running into with that is a client error 401: {"message":"Unauthorized"} when I issue drone autoscale pause just to test it out. I can list the version just fine though:

drone autoscale version
Version: latest
Commit: e2ce69fa092c9cffc12f1fd4ee96fa7f8b19fe04
Source: https://github.com/drone/autoscaler.git

Also… maybe my deploy of the auto-scaler is not working properly… would it automatically create a droplet on digital ocean if it deployed correctly? Or does it wait for a job to be queued first?

Thanks again for all your help!

9

Update
I tried to run the autoscaler locally just to see if I could get it to work. I got further… I am now getting the error message client error 403: {"message":"Forbidden"} when I issue a command other than version. When I look at the container logs I see:

{"level":"error","ip":"172.17.0.1","path":"/api/resume","method":"POST","request_id":"bb1frh8rgvn000daac7g","username":"<my-username>","time":"2018-04-03T04:06:29Z","message":"insufficient privileges"}
10

Update 2

I think I have it all mostly working now… I am just running into an issue where the autoscaler based agents can not connect to the grpc port… I most likely just need to dig a little deeper to make sure my drone server is configured correctly in kubernetes and that I have the correct ingress/services setup.

11

this was exactly my problem, and it was all due to ‘iptables: false’ in daemon.json, which i had put because otherwise i cannot block stuff with ufw.

12

Hi!

I have the exact same problem with a fresh install, but I cannot figure out how to fix it…

image
image1221×165 28.6 KB