5.9: /var filling

[MatthiasKoch]

Whatever it is, it is obviously not OS related. I have the system running without any input at all now (input has been diverted to another machine running the current version of OpenBSD and logstash), and the file is still growing. The new system is running fine. Probably it's a bug in an older version of logstash that has been fixed in a more recent version. I consider the issue closed.

Thanks a lot everybody for your input.
Matthias

[J65nko]

So it doesn;t look a named pipe issue then

I wonder whether it a log file rotation issue as TronDD has mentioned,

To rotate a log file the log rotating program sends a signal to the program creating the log. This way it knows that it the log file has been renamed/rotated and thus stop logging to that file and has to open/create a new one.

The standard signal is SIGHUP, but some programs needs a different signal, In OpenBSD's newslog.conf(8) there is a field that allows you to specify an alternative signal i.e. SIGUSR1.

Nginx is an example of such a program that needs a SIGUSR1 because it also has to deal with it's multipe worker processes, and thus can rotate the logs by itself.

From http://article.gmane.org/gmane.comp....nx.english/181 :

From: Igor Sysoev <is-G97k7egY2jIKNkxEY4oc4w <at> public.gmane.org>
Subject: Re: Log rotation
Newsgroups: gmane.comp.web.nginx.english
Date: Tuesday 19th September 2006 22:49:17 UTC (over 11 years ago)

On Tue, 19 Sep 2006, Bob Ippolito wrote:

> It looks like nginx can do log rotation by sending it USR1, but I
> haven't seen any English mention of this outside of the source code.
> Could someone show an example of how log rotation works with nginx? Is
> there a recommended log rotation tool that can handle this style of
> log rotation? I've been using piped log processes (mostly rotatelogs)
> w/ Apache for years to handle this issue.
>
> Currently most of my nginx usage is as a proxy, so I've just turned
> access logs off.. but I would like to replace Apache in a few other
> places, so I need to have rotated logs.

When master process receives -USR1 it repopens all logs, does chown()
and chmod() (to allow unpriviliged worker processes to reopen them),
and notifies workers about reopening. Then the workers reopens its
logs too. So the old logs are available to gzip right away - you will
not lose any line. The sequence is following:

mv access_log  access_log.0
mv error_log  error_log.0
kill -USR1 master
sleep 1
gzip access_log.0
gzip error_log.0

nginx does not support piping logs.

Igor Sysoev

[MatthiasKoch]

Originally Posted by J65nko

I wonder whether it a log file rotation issue as TronDD has mentioned

As a matter of fact, newsyslog IS rotating the logstash logs. I've changed the entry in newsyslog.conf from

/var/log/logstash/logstash.log          640  5     *    24    Z

to

/var/log/logstash/logstash.log          640  5     *    24    Z "/etc/rc.d/logstash restart"

The machine in question is obsolete and will be dumped soon, but I'm curious now.

[jggimi]

Originally Posted by MatthiasKoch

...I'm curious now.

• What did ls(1) tell you about the inode?
• Did you need to use fsdb(8)? If so, what did you learn?

[MatthiasKoch]

I'm getting this:

# fstat -f /var
USER     CMD          PID   FD MOUNT        INUM   MODE       R/W    SZ|DV
_logstas java         14501 5  /var         26020 -rw-r-----    w  197812691

The value under "PID" is the actual PID of the logstash process. The size equals roughly the "missing" diskspace (according to df, ~200M are used, but du /var finds only 11M) and is growing.

ls -iR /var | grep 26020 doesn't find anything.

Still reading man 8 fsdb, I'm absolutely not familiar with it.

[MatthiasKoch]

Atm I concur with TronDD's and J65nko's suspicion that logstash doesn't get notified of the rotated log when newsyslog moves the file, and continues writing to a now unlinked inode. After a restart, the inode gets cleared (and the "missing space" issue is resolved) and logstash writes to the regular logfile again - probably until it's rotated again.

After the logfile rotation, the new logfile doesn't get written to, and consequently is never rotated again. I've changed newsyslog.conf like this

/var/log/logstash/logstash.log          640  5     *    24    Z "/etc/rc.d/logstash restart"

and started logstash over. The logfile is growing. Let's see what's happening.

[MatthiasKoch]

Originally Posted by MatthiasKoch

Atm I concur with TronDD's and J65nko's suspicion that logstash doesn't get notified of the rotated log when newsyslog moves the file.

This was the actual problem. With the new configuration in newsyslog.conf, logstash gets properly restarted, and the logfiles are rotating.

Thanks everybody for your input.
Matthias