Quote:
Originally Posted by TronDD
Try `fstat -f /var` and see if there is any obvious large file. Unfortunatly, you'll only get an inode and not a file name. But are we looking for one run away file or are we looking for a bunch of non-obvious smaller files?
|
At this point, no idea. Something simply fills /var, and the regular tools cannot find anything.
Quote:
Originally Posted by TronDD
Since this is a log server, we might be able to assume a log is the problem and yuo can try `fstat /var/log/*` and see if it points you to a named file.
|
It isn't actually a logserver. Logstash receives data from remote machines on port 5044, processes it, and passes it on to an Elasticsearch cluster.
Anyway, it's getting interesting here:
Code:
# /etc/logstash> fstat -f /var
USER CMD PID FD MOUNT INUM MODE R/W SZ|DV
_logstas java 28153 5 /var 26020 -rw-r----- w 36022383
This file is owned by the logstash process (java), and it keeps growing. The size roughly equals the current difference between the output of
du and
df. For some reason I just cannot see a file, and
du cannot see it either.
The file disappears when I stop logstash, and reappears when I start it, growing steadily. And it keeps growing as long as logstash runs, even when it is not receiving and processing any data (I've redirected the input to another machine).
I have set up a similar system with the same config, running OBSD 6.2 and logstash-2.4.0p1v0. Logstash creates a file too, but it does not seem to grow.
I currently think that the problem relates to logstash, but the question remains why
du cannot see the file (the default beviour of du -s is to summarize the contents of all objects and directories).