5.9: /var filling

[MatthiasKoch]

Good afternoon all,

I'm running a virtual (ESXi 5.5) OpenBSD 5.9 logstash server with logstash-2.1.1p0v0 from packages. It's only listening for remote machines and forwards their data to an Elasticsearch cluster (no other local application running).

The /var partition is small (1.7G) and fills for a reason I cannot find out.

[root@ymir]:/> uname -var
OpenBSD ymir 5.9 GENERIC.MP#1888 amd64

[root@ymir]:/> pkg_info logstash
Information for inst:logstash-2.1.1p0v0

[root@ymir]:/> df -h
Filesystem     Size    Used   Avail Capacity  Mounted on
...
/dev/sd0e      1.7G    1.3G    343M    80%    /var

[root@ymir]:/> du -hs /var
10.8M   /var

This means, 1.3 out of 1.7G are used, but du only finds 10.8M (!) So what about the rest?
I'm a bit stuck atm... any input is welcome.

Matthias

[ibara]

$ ls -lh /var

?

[MatthiasKoch]

Nothing uncommon or unexpected. /var contains only directories, no files.

$ du -hs /var/*

lists the directories and their content size, which is in line with

 $ du -hs /var
10.8M     /var

/var/db being the biggest directory with 4.7M.

$ du -hs /var/*
2.0K    /var/account
2.0K    /var/audit
2.0K    /var/authpf
634K    /var/backups
1.2M    /var/cache
4.0K    /var/crash
22.0K   /var/cron
4.7M    /var/db
2.0K    /var/empty
44.0K   /var/games
450K    /var/log
14.0K   /var/lost+found
1.8M    /var/mail
18.0K   /var/nsd
2.0K    /var/quotas
116K    /var/run
554K    /var/spool
218K    /var/sysmerge
0B      /var/tmp
8.0K    /var/unbound
1.1M    /var/www
32.0K   /var/yp

[jggimi]

If du(1) is not finding the root cause, I will guess there is a filesystem problem.

I recommend running fsck(8). To do this, /var must be dismounted. As it is the storage for most daemons, the easiest way to fsck() /var is to boot in single-user mode (boot> -s). You may want to attempt to back up the filesystem before running fsck(), as its repair procedures may remove files. See fsck_ffs(8), called by fsck(), regarding the repair procedures used for FFS filesystems.

[MatthiasKoch]

fsck_ffs -f says the filesystem is OK... rebooting now.

[MatthiasKoch]

Must have been a filesystem issue. Although fsck and ffsck_ffs found no apparent error, after a reboot the values look reasonable:

# df -h
...
/dev/sd0e      1.7G   10.9M    1.6G     1%    /var

# du -hs /var
11.0M   /var

In other words, the problem is solved for now, but I still don't know what caused it. Quite unpleasant. Need to watch this...

Thanks so far
Matthias

[jggimi]

Do you use the softdep mount option? If so, consider disabling it.

[MatthiasKoch]

No, the mount options are ffs rw,nodev,nosuid 1 2.

[jggimi]

Then no, it wasn't a filesystem problem, as fsck() didn't report an issue and made no repair. Insead, it appears to have been an issue in the running system. Reboot resolved it, though it is possible a remount would have resolved it also.

(The soft dependencies option delays some metadata writes. I thought there might have been some sort of issue with it on your system.)

[TronDD]

Often when I see this, it's because some deleted files were unlinked from the directory but were still held open by a running process and therefor not actually removed from the filesystem.

It can happen if you rotate log files and delete them without restarting the application that is appending to that log.

[MatthiasKoch]

It's filling again. While du reveals 10.8M used space, df reports about 40M are used.

[TronDD]

In linux, you can use lsof to find deleted files, but you can't have that on OpenBSD.

Try `fstat -f /var` and see if there is any obvious large file. Unfortunatly, you'll only get an inode and not a file name. But are we looking for one run away file or are we looking for a bunch of non-obvious smaller files?

Since this is a log server, we might be able to assume a log is the problem and yuo can try `fstat /var/log/*` and see if it points you to a named file.

[jggimi]

The lsof utility was removed about a year ago, for 6.1.

Move lsof to the Attic.

Requires kmem access, is so coupled to the system internals that it
needs a /usr/src/sys checkout, and breaks regularly due to changes in
base.  People used to it should be told to use fstat(1) & friends
instead.

ok landry@ sthen@ dcoppa@

The discussion about it was archived here: https://marc.info/?t=148059362500004&r=1&w=2

[MatthiasKoch]

Originally Posted by TronDD

Try `fstat -f /var` and see if there is any obvious large file. Unfortunatly, you'll only get an inode and not a file name. But are we looking for one run away file or are we looking for a bunch of non-obvious smaller files?

At this point, no idea. Something simply fills /var, and the regular tools cannot find anything.

Originally Posted by TronDD

Since this is a log server, we might be able to assume a log is the problem and yuo can try `fstat /var/log/*` and see if it points you to a named file.

It isn't actually a logserver. Logstash receives data from remote machines on port 5044, processes it, and passes it on to an Elasticsearch cluster.

Anyway, it's getting interesting here:

# /etc/logstash> fstat -f /var
USER     CMD          PID   FD MOUNT        INUM MODE       R/W    SZ|DV
_logstas java       28153    5 /var        26020 -rw-r-----   w 36022383

This file is owned by the logstash process (java), and it keeps growing. The size roughly equals the current difference between the output of du and df. For some reason I just cannot see a file, and du cannot see it either.

The file disappears when I stop logstash, and reappears when I start it, growing steadily. And it keeps growing as long as logstash runs, even when it is not receiving and processing any data (I've redirected the input to another machine).

I have set up a similar system with the same config, running OBSD 6.2 and logstash-2.4.0p1v0. Logstash creates a file too, but it does not seem to grow.

I currently think that the problem relates to logstash, but the question remains why du cannot see the file (the default beviour of du -s is to summarize the contents of all objects and directories).

[jggimi]

You have an inode number, so there is a file assigned to the space being consumed. I will guess that fsdb(8) could tell you more, if ls(1) -i isn't sufficiently forthcoming.

[TronDD]

I currently think that the problem relates to logstash, but the question remains why du cannot see the file (the default beviour of du -s is to summarize the contents of all objects and directories).

It is a problem with logstash.

The file has been unlinked from the filesystem and 'du' operates on the filesystem. It still takes up space and and can still be used by logstash because logstash holds an open file descriptor.

This may be a mistake in logstash, but I have seen temp files used like this where the file is created, opened, then "deleted" (unlinked) and then used by the program. This way if the program exists unexpectedly, the file descriptor goes away and the file is cleaned up automatically.

See the first paragraph of unlink(2)

[J65nko]

Just wild guessing how logstash works....

If if collects data and puts everything (writing) in a named pipe then it needs a mechanism, i.e. an internet socket to read from that named pipe and send it out to the collecting server. A named pipe that is only written to without being read from just accumulates data in the named pipe file and it will become bigger and bigger.

So assuming a named pipe is being used you need to verify whether data is read from that named pipe. Only when this named pipe file has a reader it will not grow bigger and bigger

With tools like netstat or tcpdump you can verify if is there is communication or a socket between the logstash sender and the Elastic Search cluster.
Maybe pf is blocking?


With the inode number you should be able to find the file or named pipe. Something like # ls -liR /var | grep 26020

[MatthiasKoch]

Originally Posted by J65nko

you can verify if is there is communication or a socket between the logstash sender and the Elastic Search cluster.

This has definitely been working, as the machine in question is the only node that collects data and forwards it to ES.

[jggimi]

Originally Posted by J65nko

...Only when this named pipe file has a reader it will not grow bigger and bigger.

This has not been my experience when using them. Instead, I have seen writes will not complete when the reader is disconnected or reads are pending.

[e1-531g]

Originally Posted by jggimi

This has not been my experience when using them. Instead, I have seen writes will not complete when the reader is disconnected or reads are pending.

This behavior would be consistent with what I read about named pipes in Gnu/Linux in some general book ("Zrozumieć programowanie" by Gynvael Coldwind) about OSes from programmer's perspective. There were some buffer in kernel. When reader is not reading and writer is writing, buffer becomes full and writing is not completed.