I have recently taken over a test install of Qserv from Teng
(One of his previous posts - Installing and testing latest version of Qserv in Docker)
and have run into an issue involving very large (2.5G) core* files being generated locally on the master container. Which then stops the container from starting back up as it is out of disk space.
I was able to get around this by copying out the core files to somewhere local # docker cp master.localdomain:/qserv/core* /somewhere_local
Then I was able to bring up the container and delete the core file. docker start master.localdomain && docker exec -it master.localdomain rm -rf /qserv/core.2277
I was just wondering if there was any way of moving the location of these core files?
There are a few mounted volumes that are present on the container, so an easy solution would be to dump them there if at all possible.
Since Docker containers all share the kernel of the host node, core file location is mediated by the kernel core_pattern setting for the host node and is not per-container. You can investigate how this might be set on your host node (it differs somewhat between Linux distributions, but for many may be set via /proc/sys/kernel/core_pattern.)
We could also investigate options to prevent the Qserv containers from generating core files at all, but this begs the question: how often is the system dumping core at your site, and can we assist in identifying the underlying problem?
One note: wherever the core file is dumped, you can mount a host filesystem volume (or network volume) at that spot to avoid filling the container’s filesystem. But as Fritz says, not having a core dump at all is the best solution
Thank you both for replying.
Thankfully it appears this has just been a once-off, but if I run into it again I can let you know if I need some assistance in narrowing down what the cause might be.
