DaemonForums  

Go Back   DaemonForums > OpenBSD > OpenBSD General

OpenBSD General Other questions regarding OpenBSD which do not fit in any of the categories below.

Reply
 
Thread Tools Display Modes
Old 23rd November 2017
MatthiasKoch MatthiasKoch is offline
Real Name: Matthias Koch
Port Guard
 
Join Date: Mar 2016
Location: Germany
Posts: 37
Default

Whatever it is, it is obviously not OS related. I have the system running without any input at all now (input has been diverted to another machine running the current version of OpenBSD and logstash), and the file is still growing. The new system is running fine. Probably it's a bug in an older version of logstash that has been fixed in a more recent version. I consider the issue closed.

Thanks a lot everybody for your input.
Matthias
Reply With Quote
Old 23rd November 2017
J65nko J65nko is offline
Administrator
 
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 4,125
Default

So it doesn;t look a named pipe issue then

I wonder whether it a log file rotation issue as TronDD has mentioned,

To rotate a log file the log rotating program sends a signal to the program creating the log. This way it knows that it the log file has been renamed/rotated and thus stop logging to that file and has to open/create a new one.

The standard signal is SIGHUP, but some programs needs a different signal, In OpenBSD's newslog.conf(8) there is a field that allows you to specify an alternative signal i.e. SIGUSR1.

Nginx is an example of such a program that needs a SIGUSR1 because it also has to deal with it's multipe worker processes, and thus can rotate the logs by itself.

From http://article.gmane.org/gmane.comp....nx.english/181 :

Code:
From: Igor Sysoev <is-G97k7egY2jIKNkxEY4oc4w <at> public.gmane.org>
Subject: Re: Log rotation
Newsgroups: gmane.comp.web.nginx.english
Date: Tuesday 19th September 2006 22:49:17 UTC (over 11 years ago)

On Tue, 19 Sep 2006, Bob Ippolito wrote:

> It looks like nginx can do log rotation by sending it USR1, but I
> haven't seen any English mention of this outside of the source code.
> Could someone show an example of how log rotation works with nginx? Is
> there a recommended log rotation tool that can handle this style of
> log rotation? I've been using piped log processes (mostly rotatelogs)
> w/ Apache for years to handle this issue.
>
> Currently most of my nginx usage is as a proxy, so I've just turned
> access logs off.. but I would like to replace Apache in a few other
> places, so I need to have rotated logs.

When master process receives -USR1 it repopens all logs, does chown()
and chmod() (to allow unpriviliged worker processes to reopen them),
and notifies workers about reopening. Then the workers reopens its
logs too. So the old logs are available to gzip right away - you will
not lose any line. The sequence is following:

mv access_log  access_log.0
mv error_log  error_log.0
kill -USR1 master
sleep 1
gzip access_log.0
gzip error_log.0

nginx does not support piping logs.

Igor Sysoev
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump
Reply With Quote
Old 27th November 2017
MatthiasKoch MatthiasKoch is offline
Real Name: Matthias Koch
Port Guard
 
Join Date: Mar 2016
Location: Germany
Posts: 37
Default

Quote:
Originally Posted by J65nko View Post
I wonder whether it a log file rotation issue as TronDD has mentioned
As a matter of fact, newsyslog IS rotating the logstash logs. I've changed the entry in newsyslog.conf from
Code:
/var/log/logstash/logstash.log          640  5     *    24    Z
to
Code:
/var/log/logstash/logstash.log          640  5     *    24    Z "/etc/rc.d/logstash restart"
The machine in question is obsolete and will be dumped soon, but I'm curious now.
Reply With Quote
Old 27th November 2017
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,975
Default

Quote:
Originally Posted by MatthiasKoch View Post
...I'm curious now.
  • What did ls(1) tell you about the inode?
  • Did you need to use fsdb(8)? If so, what did you learn?
Reply With Quote
Old 30th November 2017
MatthiasKoch MatthiasKoch is offline
Real Name: Matthias Koch
Port Guard
 
Join Date: Mar 2016
Location: Germany
Posts: 37
Default

I'm getting this:
Code:
# fstat -f /var
USER     CMD          PID   FD MOUNT        INUM   MODE       R/W    SZ|DV
_logstas java         14501 5  /var         26020 -rw-r-----    w  197812691
The value under "PID" is the actual PID of the logstash process. The size equals roughly the "missing" diskspace (according to df, ~200M are used, but du /var finds only 11M) and is growing.

ls -iR /var | grep 26020 doesn't find anything.

Still reading man 8 fsdb, I'm absolutely not familiar with it.
Reply With Quote
Old 30th November 2017
MatthiasKoch MatthiasKoch is offline
Real Name: Matthias Koch
Port Guard
 
Join Date: Mar 2016
Location: Germany
Posts: 37
Default

Atm I concur with TronDD's and J65nko's suspicion that logstash doesn't get notified of the rotated log when newsyslog moves the file, and continues writing to a now unlinked inode. After a restart, the inode gets cleared (and the "missing space" issue is resolved) and logstash writes to the regular logfile again - probably until it's rotated again.

After the logfile rotation, the new logfile doesn't get written to, and consequently is never rotated again. I've changed newsyslog.conf like this
Code:
/var/log/logstash/logstash.log          640  5     *    24    Z "/etc/rc.d/logstash restart"
and started logstash over. The logfile is growing. Let's see what's happening.
Reply With Quote
Old 4th December 2017
MatthiasKoch MatthiasKoch is offline
Real Name: Matthias Koch
Port Guard
 
Join Date: Mar 2016
Location: Germany
Posts: 37
Default

Quote:
Originally Posted by MatthiasKoch View Post
Atm I concur with TronDD's and J65nko's suspicion that logstash doesn't get notified of the rotated log when newsyslog moves the file.
This was the actual problem. With the new configuration in newsyslog.conf, logstash gets properly restarted, and the logfiles are rotating.

Thanks everybody for your input.
Matthias
Reply With Quote
Reply

Tags
df, du, logstash

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Clientmqueue keeps filling /var and I don't know why beandip FreeBSD General 6 19th November 2008 10:42 PM


All times are GMT. The time now is 05:32 AM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick