|
OpenBSD General Other questions regarding OpenBSD which do not fit in any of the categories below. |
|
Thread Tools | Display Modes |
|
|
|||
5.9: /var filling
Good afternoon all,
I'm running a virtual (ESXi 5.5) OpenBSD 5.9 logstash server with logstash-2.1.1p0v0 from packages. It's only listening for remote machines and forwards their data to an Elasticsearch cluster (no other local application running). The /var partition is small (1.7G) and fills for a reason I cannot find out. Code:
[root@ymir]:/> uname -var OpenBSD ymir 5.9 GENERIC.MP#1888 amd64 [root@ymir]:/> pkg_info logstash Information for inst:logstash-2.1.1p0v0 [root@ymir]:/> df -h Filesystem Size Used Avail Capacity Mounted on ... /dev/sd0e 1.7G 1.3G 343M 80% /var [root@ymir]:/> du -hs /var 10.8M /var I'm a bit stuck atm... any input is welcome. Matthias |
|
|||
Code:
$ ls -lh /var |
|
|||
Nothing uncommon or unexpected. /var contains only directories, no files.
Code:
$ du -hs /var/* Code:
$ du -hs /var 10.8M /var Code:
$ du -hs /var/* 2.0K /var/account 2.0K /var/audit 2.0K /var/authpf 634K /var/backups 1.2M /var/cache 4.0K /var/crash 22.0K /var/cron 4.7M /var/db 2.0K /var/empty 44.0K /var/games 450K /var/log 14.0K /var/lost+found 1.8M /var/mail 18.0K /var/nsd 2.0K /var/quotas 116K /var/run 554K /var/spool 218K /var/sysmerge 0B /var/tmp 8.0K /var/unbound 1.1M /var/www 32.0K /var/yp |
|
|||
fsck_ffs -f says the filesystem is OK... rebooting now.
|
|
|||
Must have been a filesystem issue. Although fsck and ffsck_ffs found no apparent error, after a reboot the values look reasonable:
Code:
# df -h ... /dev/sd0e 1.7G 10.9M 1.6G 1% /var # du -hs /var 11.0M /var Thanks so far Matthias |
|
||||
Then no, it wasn't a filesystem problem, as fsck() didn't report an issue and made no repair. Insead, it appears to have been an issue in the running system. Reboot resolved it, though it is possible a remount would have resolved it also.
(The soft dependencies option delays some metadata writes. I thought there might have been some sort of issue with it on your system.) Last edited by jggimi; 21st November 2017 at 11:47 AM. Reason: clarity |
|
|||
Often when I see this, it's because some deleted files were unlinked from the directory but were still held open by a running process and therefor not actually removed from the filesystem.
It can happen if you rotate log files and delete them without restarting the application that is appending to that log. |
|
|||
No, the mount options are ffs rw,nodev,nosuid 1 2.
|
|
|||
It's filling again. While du reveals 10.8M used space, df reports about 40M are used.
|
|
|||
In linux, you can use lsof to find deleted files, but you can't have that on OpenBSD.
Try `fstat -f /var` and see if there is any obvious large file. Unfortunatly, you'll only get an inode and not a file name. But are we looking for one run away file or are we looking for a bunch of non-obvious smaller files? Since this is a log server, we might be able to assume a log is the problem and yuo can try `fstat /var/log/*` and see if it points you to a named file. |
|
|||
Quote:
Quote:
Anyway, it's getting interesting here: Code:
# /etc/logstash> fstat -f /var USER CMD PID FD MOUNT INUM MODE R/W SZ|DV _logstas java 28153 5 /var 26020 -rw-r----- w 36022383 The file disappears when I stop logstash, and reappears when I start it, growing steadily. And it keeps growing as long as logstash runs, even when it is not receiving and processing any data (I've redirected the input to another machine). I have set up a similar system with the same config, running OBSD 6.2 and logstash-2.4.0p1v0. Logstash creates a file too, but it does not seem to grow. I currently think that the problem relates to logstash, but the question remains why du cannot see the file (the default beviour of du -s is to summarize the contents of all objects and directories). |
|
|||
Quote:
The file has been unlinked from the filesystem and 'du' operates on the filesystem. It still takes up space and and can still be used by logstash because logstash holds an open file descriptor. This may be a mistake in logstash, but I have seen temp files used like this where the file is created, opened, then "deleted" (unlinked) and then used by the program. This way if the program exists unexpectedly, the file descriptor goes away and the file is cleaned up automatically. See the first paragraph of unlink(2) |
|
||||
The lsof utility was removed about a year ago, for 6.1.
Code:
Move lsof to the Attic. Requires kmem access, is so coupled to the system internals that it needs a /usr/src/sys checkout, and breaks regularly due to changes in base. People used to it should be told to use fstat(1) & friends instead. ok landry@ sthen@ dcoppa@ |
|
|||
Just wild guessing how logstash works....
If if collects data and puts everything (writing) in a named pipe then it needs a mechanism, i.e. an internet socket to read from that named pipe and send it out to the collecting server. A named pipe that is only written to without being read from just accumulates data in the named pipe file and it will become bigger and bigger. So assuming a named pipe is being used you need to verify whether data is read from that named pipe. Only when this named pipe file has a reader it will not grow bigger and bigger With tools like netstat or tcpdump you can verify if is there is communication or a socket between the logstash sender and the Elastic Search cluster. Maybe pf is blocking? With the inode number you should be able to find the file or named pipe. Something like # ls -liR /var | grep 26020
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump |
|
|||
This has definitely been working, as the machine in question is the only node that collects data and forwards it to ES.
|
|
|||
This behavior would be consistent with what I read about named pipes in Gnu/Linux in some general book ("Zrozumieć programowanie" by Gynvael Coldwind) about OSes from programmer's perspective. There were some buffer in kernel. When reader is not reading and writer is writing, buffer becomes full and writing is not completed.
__________________
Signature: Furthermore, I consider that systemd must be destroyed. Based on Latin oratorical phrase Last edited by e1-531g; 23rd November 2017 at 12:18 PM. |
Tags |
df, du, logstash |
Thread Tools | |
Display Modes | |
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Clientmqueue keeps filling /var and I don't know why | beandip | FreeBSD General | 6 | 19th November 2008 10:42 PM |