I’ve been reading some threads around logs and they all seem to describe issues similar to this one, but they are not the same. I’ve also seen that there were some logs limits in the agent of 0.5, but I browsed through the code and didn’t find in 0.8.
drone limits the amount of logs to 2MB per pipeline step to prevent a single step from overloading the logs and freezing the browser, and potentially causing OOM issues since logs are streamed from memory while the build is running. Do you have the option to reduce logging verbosity? Or split tests into multiple steps, thus removing the per-step limit?
Thanks for the answer @bradrydzewski. I tried that, but my output is verbose on purpose. I run the integration tests on a huge cluster that gives me important profiles that then I use to analyze.
Is there a way to avoid cutting the logs to 2MB or allowing me to get them directly from the host machine running docker?
Not at the moment. This is something we could make configurable in the future, but it would not be until the next release.
You could pipe the results to a file and upload them somewhere (to S3) in the subsequent step.
pipeline:
build:
image: golang
commands:
- set -o pipefail
- go get -t ./...
- go test -v ./... | tee gotest.out
upload:
image: plugins/s3
secrets: [ aws_access_key_id, aws_secret_access_key ]
source: gotest.out
target: gotest.${DRONE_BUILD_NUMBER}.out
Exposing secrets, however, to public pull requests is not recommended unless you enable the “protected” repository setting (screenshot below). This setting will block pull requests that change the yaml and require manual approval, to prevent tampering with the yaml in an attempt to expose secrets.
Thank you @bradrydzewski, I’ll certainly try that out. I was aware that was a possibility, but wanted to know if there was some way of avoiding it because of the layer of indirection it adds to the build.
In the future, it would be great if Drone was a little bit more configurable than it is right now so that all these use cases can be accommodated. I would be happy to have some kind of link to download the logs if they are too large rather than seeing them (this is to the best of my understanding what Jenkinds does).
Great. A better option than using S3 would be to send the full logs as attachments to slack, but that’s unfortunately not possible by the drone-slack plugin.
a large log makes loading in browser very slow (increasing the database/web server’s overhead)
the log files can not be downloaded for further investigation (I knew you can download via drone cli, but it would be nicer to see a download link in the webpage)
I also have a concern with my database sizes. Would it be better to store logs in the file system? The database only store building status and other metadata.
