Unclear on GitHub activation

iancullinane · February 13, 2019, 8:29pm

I have a multimachine deployment which appears to be set up appropriately. However I cannot activate any repos. I have read a good amount on the permissions and I want to know if the following error is stating the user does not have necessary permissions, and thus unable to create the necessary hooks.

I am also curious if GitHub scopes are something which needs to be modified for a deployment. My log error when attempting to activate is the following:

{"admin":true,"level":"debug","msg":"api: access granted","name":"some-service","namespace":"MyOrganization","read":true,"request-id":"123456789","time":"2019-02-13T18:56:48Z","user.login":"a-username","visibility":"public","write":true}

I get this as well the following errors in the browser console:

POST https://custom.myteam.ninja/api/repos/MyOrganization/some-service 500 Uncaught (in promise) SyntaxError: Unexpected token I in JSON at position 0

bradrydzewski · February 13, 2019, 8:51pm

can you look at the networking tools in the browser and snapshot the http response you are receiving, and paste in this thread? also, I would expect more log data for activating a repository in the event of a server-side error … drone logs all error scenarios. Also, do you have a reverse proxy or load balancer in front of drone that could be intercepting or mutating traffic?

Also, what version of Drone are you using? Can you confirm you are using 1.0.0-rc.5?

iancullinane · February 13, 2019, 9:15pm

Yes this is 1.0.0-rc.5

The entire response is:

Internal Server Error

Other requests seem fine and give a more typical response (for instance the request to the service page under apis/repos. Checking the docker logs I am also getting a invalid memory address. Possible this is after the first attempt to activate. I can post that stacktrace.

I do have an AWS alb in front of drone. In addition I am doing service discovery and container scheduling via Hashi Nomad/Consul.

bradrydzewski · February 13, 2019, 9:20pm

yes, the more detail you can provide (logs, stacktrace, etc) would be helpful. A plain text response is abnormal, so I do not think this is related to permissions. But logs and stacktrace may help.

iancullinane · February 13, 2019, 9:30pm

Following is the stacktrace of my error as well as the preceding 3 log messages:

{"fields.time":"2019-02-13T21:07:24Z","latency":14078,"level":"debug","method":"GET","msg":"","remote":"10.218.0.124:10592","request":"/v1/test/ping","request-id":"546579413","time":"2019-02-13T21:07:24Z"}
{"level":"debug","msg":"api: read access granted","name":"some-service","namespace":"MyOrg","request-id":"123456789","time":"2019-02-13T21:07:25Z","user.login":"a-user","visibility":"public"}
{"admin":true,"level":"debug","msg":"api: access granted","name":"some-service","namespace":"MyOrg","read":true,"request-id":"987654321","time":"2019-02-13T21:07:25Z","user.login":"a-user","visibility":"public","write":true}
Panic: runtime error: invalid memory address or nil pointer dereference
goroutine 8074 [running]:
runtime/debug.Stack(0x48, 0x0, 0x0)
    /usr/local/go/src/runtime/debug/stack.go:24 +0xa7
runtime/debug.PrintStack()
    /usr/local/go/src/runtime/debug/stack.go:16 +0x22
github.com/drone/drone/vendor/github.com/go-chi/chi/middleware.Recoverer.func1.1(0xc42013d900, 0x1a48c00, 0xc4206237a0)
    /go/src/github.com/drone/drone/vendor/github.com/go-chi/chi/middleware/recoverer.go:28 +0x1b6
panic(0x167c680, 0x25a9580)
    /usr/local/go/src/runtime/panic.go:502 +0x229
github.com/drone/drone/internal/scm/hook.deleteHook(0x1a49f80, 0xc4200bcfc0, 0xc4203531e0, 0xc420690280, 0x17, 0xc420ada1e0, 0x22, 0x2, 0xc420ada1e0)
    /go/src/github.com/drone/drone/internal/scm/hook/util.go:24 +0x44
github.com/drone/drone/internal/scm/hook.replaceHook(0x1a49f80, 0xc4200bcfc0, 0xc4203531e0, 0xc420690280, 0x17, 0xc4209a2780, 0x22, 0xc4200bcfc0)
    /go/src/github.com/drone/drone/internal/scm/hook/util.go:15 +0x74
github.com/drone/drone/internal/scm/hook.(*service).Create(0xc4200bcf90, 0x1a49f80, 0xc4200bcfc0, 0xc420814280, 0x18785df, 0x4)
    /go/src/github.com/drone/drone/internal/scm/hook/hook.go:63 +0x294
github.com/drone/drone/internal/api/repos.HandleEnable.func1(0x1a48c00, 0xc4206237a0, 0xc420868200)
    /go/src/github.com/drone/drone/internal/api/repos/enable.go:83 +0x33a
net/http.HandlerFunc.ServeHTTP(0xc4200b05a0, 0x1a48c00, 0xc4206237a0, 0xc420868200)
    /usr/local/go/src/net/http/server.go:1947 +0x44
github.com/drone/drone/internal/api/acl.CheckAccess.func1.1(0x1a48c00, 0xc4206237a0, 0xc420868000)
    /go/src/github.com/drone/drone/internal/api/acl/check.go:127 +0x62d
net/http.HandlerFunc.ServeHTTP(0xc42014ec60, 0x1a48c00, 0xc4206237a0, 0xc420868000)
    /usr/local/go/src/net/http/server.go:1947 +0x44
github.com/drone/drone/vendor/github.com/go-chi/chi.(*ChainHandler).ServeHTTP(0xc42008ab00, 0x1a48c00, 0xc4206237a0, 0xc420868000)
    /go/src/github.com/drone/drone/vendor/github.com/go-chi/chi/chain.go:31 +0x52
github.com/drone/drone/vendor/github.com/go-chi/chi.(*Mux).routeHTTP(0xc42008eae0, 0x1a48c00, 0xc4206237a0, 0xc420868000)
    /go/src/github.com/drone/drone/vendor/github.com/go-chi/chi/mux.go:424 +0x289
github.com/drone/drone/vendor/github.com/go-chi/chi.(*Mux).(github.com/drone/drone/vendor/github.com/go-chi/chi.routeHTTP)-fm(0x1a48c00, 0xc4206237a0, 0xc420868000)
    /go/src/github.com/drone/drone/vendor/github.com/go-chi/chi/mux.go:368 +0x48
net/http.HandlerFunc.ServeHTTP(0xc4203416f0, 0x1a48c00, 0xc4206237a0, 0xc420868000)
    /usr/local/go/src/net/http/server.go:1947 +0x44
github.com/drone/drone/internal/api/acl.CheckAccess.func1.1(0x1a48c00, 0xc4206237a0, 0xc420868000)
    /go/src/github.com/drone/drone/internal/api/acl/check.go:89 +0xb11
net/http.HandlerFunc.ServeHTTP(0xc42014e5d0, 0x1a48c00, 0xc4206237a0, 0xc420868000)
    /usr/local/go/src/net/http/server.go:1947 +0x44
github.com/drone/drone/internal/api/acl.InjectRepository.func1.1(0x1a48c00, 0xc4206237a0, 0xc42013df00)
    /go/src/github.com/drone/drone/internal/api/acl/repo.go:123 +0xa8b
net/http.HandlerFunc.ServeHTTP(0xc4200b0550, 0x1a48c00, 0xc4206237a0, 0xc42013df00)
    /usr/local/go/src/net/http/server.go:1947 +0x44
github.com/drone/drone/vendor/github.com/go-chi/chi.(*Mux).ServeHTTP(0xc42008eae0, 0x1a48c00, 0xc4206237a0, 0xc42013df00)
    /go/src/github.com/drone/drone/vendor/github.com/go-chi/chi/mux.go:69 +0x2eb
github.com/drone/drone/vendor/github.com/go-chi/chi.(*Mux).Mount.func1(0x1a48c00, 0xc4206237a0, 0xc42013df00)
    /go/src/github.com/drone/drone/vendor/github.com/go-chi/chi/mux.go:291 +0x107
net/http.HandlerFunc.ServeHTTP(0xc420116ca0, 0x1a48c00, 0xc4206237a0, 0xc42013df00)
    /usr/local/go/src/net/http/server.go:1947 +0x44
github.com/drone/drone/vendor/github.com/go-chi/chi.(*Mux).routeHTTP(0xc42008ea80, 0x1a48c00, 0xc4206237a0, 0xc42013df00)
    /go/src/github.com/drone/drone/vendor/github.com/go-chi/chi/mux.go:424 +0x289
github.com/drone/drone/vendor/github.com/go-chi/chi.(*Mux).(github.com/drone/drone/vendor/github.com/go-chi/chi.routeHTTP)-fm(0x1a48c00, 0xc4206237a0, 0xc42013df00)
    /go/src/github.com/drone/drone/vendor/github.com/go-chi/chi/mux.go:368 +0x48
net/http.HandlerFunc.ServeHTTP(0xc420341a30, 0x1a48c00, 0xc4206237a0, 0xc42013df00)
    /usr/local/go/src/net/http/server.go:1947 +0x44
github.com/drone/drone/vendor/github.com/go-chi/cors.(*Cors).Handler.func1(0x1a48c00, 0xc4206237a0, 0xc42013df00)
    /go/src/github.com/drone/drone/vendor/github.com/go-chi/cors/cors.go:199 +0xfe
net/http.HandlerFunc.ServeHTTP(0xc420116cc0, 0x1a48c00, 0xc4206237a0, 0xc42013df00)
    /usr/local/go/src/net/http/server.go:1947 +0x44
github.com/drone/drone/internal/api/auth.HandleAuthentication.func1.1(0x1a48c00, 0xc4206237a0, 0xc42013da00)
    /go/src/github.com/drone/drone/internal/api/auth/auth.go:41 +0x280
net/http.HandlerFunc.ServeHTTP(0xc4202ad350, 0x1a48c00, 0xc4206237a0, 0xc42013da00)
    /usr/local/go/src/net/http/server.go:1947 +0x44
github.com/drone/drone/internal/logger.Middleware.func1(0x1a48c00, 0xc4206237a0, 0xc42013d900)
    /go/src/github.com/drone/drone/internal/logger/handler.go:22 +0x220
net/http.HandlerFunc.ServeHTTP(0xc420116ce0, 0x1a48c00, 0xc4206237a0, 0xc42013d900)
    /usr/local/go/src/net/http/server.go:1947 +0x44
github.com/drone/drone/vendor/github.com/go-chi/chi/middleware.NoCache.func1(0x1a48c00, 0xc4206237a0, 0xc42013d900)
    /go/src/github.com/drone/drone/vendor/github.com/go-chi/chi/middleware/nocache.go:54 +0x217
net/http.HandlerFunc.ServeHTTP(0xc420116d00, 0x1a48c00, 0xc4206237a0, 0xc42013d900)
    /usr/local/go/src/net/http/server.go:1947 +0x44
github.com/drone/drone/vendor/github.com/go-chi/chi/middleware.Recoverer.func1(0x1a48c00, 0xc4206237a0, 0xc42013d900)
    /go/src/github.com/drone/drone/vendor/github.com/go-chi/chi/middleware/recoverer.go:35 +0x9f
net/http.HandlerFunc.ServeHTTP(0xc420116d20, 0x1a48c00, 0xc4206237a0, 0xc42013d900)
    /usr/local/go/src/net/http/server.go:1947 +0x44
github.com/drone/drone/vendor/github.com/go-chi/chi.(*Mux).ServeHTTP(0xc42008ea80, 0x1a48c00, 0xc4206237a0, 0xc42013d900)
    /go/src/github.com/drone/drone/vendor/github.com/go-chi/chi/mux.go:69 +0x2eb
github.com/drone/drone/vendor/github.com/go-chi/chi.(*Mux).Mount.func1(0x1a48c00, 0xc4206237a0, 0xc42013d900)
    /go/src/github.com/drone/drone/vendor/github.com/go-chi/chi/mux.go:291 +0x107
net/http.HandlerFunc.ServeHTTP(0xc420117ce0, 0x1a48c00, 0xc4206237a0, 0xc42013d900)
    /usr/local/go/src/net/http/server.go:1947 +0x44
github.com/drone/drone/vendor/github.com/go-chi/chi.(*Mux).routeHTTP(0xc42008e8a0, 0x1a48c00, 0xc4206237a0, 0xc42013d900)
    /go/src/github.com/drone/drone/vendor/github.com/go-chi/chi/mux.go:424 +0x289
github.com/drone/drone/vendor/github.com/go-chi/chi.(*Mux).(github.com/drone/drone/vendor/github.com/go-chi/chi.routeHTTP)-fm(0x1a48c00, 0xc4206237a0, 0xc42013d900)
    /go/src/github.com/drone/drone/vendor/github.com/go-chi/chi/mux.go:368 +0x48
net/http.HandlerFunc.ServeHTTP(0xc420341600, 0x1a48c00, 0xc4206237a0, 0xc42013d900)
    /usr/local/go/src/net/http/server.go:1947 +0x44
github.com/drone/drone/vendor/github.com/go-chi/chi.(*Mux).ServeHTTP(0xc42008e8a0, 0x1a48c00, 0xc4206237a0, 0xc42080f500)
    /go/src/github.com/drone/drone/vendor/github.com/go-chi/chi/mux.go:81 +0x221
net/http.serverHandler.ServeHTTP(0xc420353ee0, 0x1a48c00, 0xc4206237a0, 0xc42080f500)
    /usr/local/go/src/net/http/server.go:2694 +0xbc
net/http.(*conn).serve(0xc4205ecb40, 0x1a49ec0, 0xc4203f54c0)
    /usr/local/go/src/net/http/server.go:1830 +0x651
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:2795 +0x27b

bradrydzewski · February 13, 2019, 10:01pm

Thanks, this was helpful. Can you share your Drone server configuration? Specifically your source code management configuration (e.g. GitHub) with the tokens redacted? These lines of code are the source of the stacktrace:

	u, _ := url.Parse(target)
	h, err := findHook(ctx, client, repo, u.Host)

drone assumes your source control management url is valid. This stacktrace tells me that parsing the url failed which caused a nil pointer, which is unexpected. My guess is that this is the root cause of all issues.

iancullinane · February 13, 2019, 10:05pm

This is the config copied from our configs.

"DRONE_SERVER_HOST": "${PRIVATE_IP}",
"DRONE_LOGS_DEBUG": "true",
"DRONE_RPC_SECRET": "789456123",
"DRONE_GITHUB_PRIVATE_MODE": "true",
"DRONE_GITHUB_SERVER": "https://drone.my-organization.ninja",
"DRONE_GITHUB_CLIENT_ID": "123456789",
"DRONE_GITHUB_CLIENT_SECRET": "789456123",
"DRONE_DATABASE_DRIVER": "postgres",
"DRONE_DATABASE_DATASOURCE": "postgres://drone_b:pw123@droneb.abcdefg.us-east-2.rds.amazonaws.com:5432/droneb"

bradrydzewski · February 13, 2019, 10:18pm

Actually, I was wrong, the url.Parse that is failing is parsing your Drone server address. This is the combination of the DRONE_SERVER_HOST and DRONE_SERVER_PROTO. I would need to see these values to advise further.

Alternatively, if you do not want to share publicly you could try plugging them into the following script which could help surface any errors:
https://play.golang.org/p/UHJ4hWFjaJQ

If it is still unclear what the problem is, I would probably need to see the results of docker inspect <container> to see exactly what DRONE_SERVER_HOST and optionally DRONE_SERVER_PROTO are being set to.

iancullinane · February 13, 2019, 10:23pm

Thanks for your help I will be trying those out tomorrow morning.

iancullinane · February 14, 2019, 8:51pm

Checking in to say that the problem was from my environment. It was essentially a problem with Nomad env variables and the specifics of our base container image.

Thanks for your help as I never would have found the error without being pointed to the url.Parse error that stacktrace brought up.

bradrydzewski · February 14, 2019, 9:09pm

awesome, glad to hear you figured it out

Topic		Replies	Views
[RESOLVED] GitHub API Regression, nil pointer error Drone Support	2	419	March 23, 2018
Failed to activate your repository Drone Support	7	4819	March 20, 2019
Oauth_error with GitHub Drone Support	9	2271	October 11, 2017
Troubleshooting "error setting commit status" errors Drone Support	4	403	July 31, 2018
Github authorization loop Drone Support	4	353	October 5, 2019

Unclear on GitHub activation

Related topics