DaemonForums  

Go Back   DaemonForums > OpenBSD > OpenBSD General

OpenBSD General Other questions regarding OpenBSD which do not fit in any of the categories below.

Reply
 
Thread Tools Display Modes
  #1   (View Single Post)  
Old 3rd March 2021
roddierod's Avatar
roddierod roddierod is offline
Real Name: Rod Person
VPN Cryptographer
 
Join Date: Apr 2008
Location: Pittsburgh, Pa
Posts: 429
Default Slow NFS transfer with large files

I have a FreeBSD server that with a few NFS shares. The write time to the share from my OpenBSD machine is terribly slow as can be seen in below. the file big.mkv is 7GB:

Move to server:
Code:
atomizer$ time mv big.mkv /mnt/Audio/.tmp
   85m11.01s real     0m00.34s user     1m15.02s system
Move from server:
Code:
atomizer$ time mv /mnt/Audio/.tmp/big.mkv ~/Downloads
   11m54.42s real     0m00.40s user     0m41.44s system
Copy to server:
Code:
atomizer$ time cp big.mkv /mnt/Audio/.tmp
   59m05.32s real     0m00.37s user     1m26.60s system
this is my current /etc/fstab on my OpenBSD machine. I have tried different settings
for -r/-w value 4096,8192,16384
Code:
192.168.1.178:/mnt/AUDIO /mnt/Audio nfs rw,noatime,nodev,nosuid,-r=32768,-w=32768 0 0
192.168.1.178:/mnt/BIGSTORE /mnt/nfs nfs rw,noatime,nodev,nosuid,-r=32768,-w=32768 0 0
192.168.1.178:/mnt/FBSD /mnt/FreeBSD nfs rw,noatime,nodev,nosuid,-r=32768,-w=32768 0 0
192.168.1.178:/usr/local/share/MEDIA /mnt/Video nfs rw,noatime,nodev,nosuid,-r=32768,-w=32768 0 0
Interestingly if I try testing transfer with dd everything looks fine...

Code:
atomizer$ time dd if=/dev/zero of=/mnt/Audio/.tmp/otest bs=1m count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 88.146 secs (11895802 bytes/sec)
    1m29.56s real     0m00.02s user     0m09.46s system
So I tried the same test from my Macbook and here they are...

Code:
Doris:~ rod2$
Doris:~ rod2$ time cp /private/Audio/.tmp/big.mkv /Users/rod2

real	9m0.636s
user	0m0.031s
sys	0m13.321s
Doris:~ rod2$ time mv /Users/rod2/big.mkv /private/Audio/.tmp/big2.mkv

real	10m55.434s
user	0m2.140s
sys	1m28.825s
And dd

Code:
Doris:~ rod2$ time dd if=/dev/zero of=/private/Audio/.tmp/test bs=1m count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 89.905967 secs (11663030 bytes/sec)

real	1m30.420s
user	0m0.005s
sys	0m4.379s
__________________
"The basic tool for the manipulation of reality is the manipulation of words. If you can control the meaning of words, you can control the people who must use the words." -Philip K. Dick
Reply With Quote
  #2   (View Single Post)  
Old 3rd March 2021
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,210
Default

The nfsstat(1) utility may be of help in diagnosing the problem. As would consideration of mount options on the server, such as noatime and -- if your use-case permits it -- async.
  • Per the mount_nfs(8) man page, the -r and -w mount options are knobs you twist when the NFS server reports dropped UDP fragments for read or write ops, respectively. Unless nfsstat(1) is reporting the issue, these knobs are unlikely to make a positive impact.
  • The mount_nfs(8) man page suggests using -a (read ahead) for large sequential reads, particularly over bandwidth delayed networks. You might consider bumping this value up from the default to see if it makes any difference on read operations. Unfortunately, there's no "write ahead" equivalent due to the laws of physics. That is where async might improve performance if the possibility of data loss is acceptable for your use-case.
Reply With Quote
  #3   (View Single Post)  
Old 3rd March 2021
roddierod's Avatar
roddierod roddierod is offline
Real Name: Rod Person
VPN Cryptographer
 
Join Date: Apr 2008
Location: Pittsburgh, Pa
Posts: 429
Default

Thanks. I'll try anything at this point. I'm pretty sure I had async at one point and that didn't seem to do anything but I'll try again.

I first added the -r,-w when I started noticing the problem and changing them does do something...but I haven't tried measuring with them removed so I'll give that a shot.
__________________
"The basic tool for the manipulation of reality is the manipulation of words. If you can control the meaning of words, you can control the people who must use the words." -Philip K. Dick
Reply With Quote
  #4   (View Single Post)  
Old 4th March 2021
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,210
Default

Async might make a difference on write performance, as metadata would not need to be synchronized.
Reply With Quote
  #5   (View Single Post)  
Old 4th March 2021
roddierod's Avatar
roddierod roddierod is offline
Real Name: Rod Person
VPN Cryptographer
 
Join Date: Apr 2008
Location: Pittsburgh, Pa
Posts: 429
Default

it seems that async is not a valid option for nfs mounts. But after a bit more of experimenting and using nfsstat I'm starting to think it might be the OpenBSD machine drive but now sure...

Code:
atomizer# nfsstat -c
Client Info:
  Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
    50398       478      7315         0   1556472   5775282       548       174
   Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
      423         0         0         1         0       519         0      5883
    Mknod    Fsstat    Fsinfo  PathConf    Commit
        0       706        13         0       905
Rpc Info:
 TimedOut   Invalid X Replies   Retries  Requests   FrcSync
        0         0      1049      1295   7399117         0
Cache Info:
Attr Hits    Misses Lkup Hits    Misses BioR Hits    Misses BioW Hits    Misses
  2603760     87675    171087      7315      6372   1556472 18446744073708580777   5775282
BioRLHits    Misses BioD Hits    Misses DirE Hits    Misses
        0         0         0         0       355       361
I not extirely clear but that Cache Infor doesn't look cool. idk?

But I got the write from the server down to what looks normal...but writes TO the server are still extremly slow.
__________________
"The basic tool for the manipulation of reality is the manipulation of words. If you can control the meaning of words, you can control the people who must use the words." -Philip K. Dick
Reply With Quote
  #6   (View Single Post)  
Old 4th March 2021
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,210
Default

I was thinking async on the server side, but clearly there are limits to what can be done with NFSv3.

The cache is used for Metadata, so increasing cache timeout limits per mount_nfs(8) may reduce your cache misses.
Reply With Quote
  #7   (View Single Post)  
Old 5th March 2021
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,210
Default

I happened across an NFSv3/v2 performance guide that may be of interest, though it is Linux-centric. http://nfs.sourceforge.net/nfs-howto/ar01s05.html
Reply With Quote
  #8   (View Single Post)  
Old 5th March 2021
J65nko J65nko is offline
Administrator
 
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 3,660
Default

https://www.admin-magazine.com/HPC/A...and-Management mentions using jumbo Ethernet frames, a Maximum Transmission Unit of 9000 bytes, instead of the standard 1500 byte MTU.

I remember that ajacoutot more than 12 years ago complained on bsdforums.org about the same slow NFS performance of OpenBSD. (Later he became an OpenBSD developer)
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump

Last edited by J65nko; 5th March 2021 at 03:40 AM.
Reply With Quote
  #9   (View Single Post)  
Old 5th March 2021
roddierod's Avatar
roddierod roddierod is offline
Real Name: Rod Person
VPN Cryptographer
 
Join Date: Apr 2008
Location: Pittsburgh, Pa
Posts: 429
Default

j65nko and jggimi thanks I will check both of these out. So far only changing -w is doing anything.
__________________
"The basic tool for the manipulation of reality is the manipulation of words. If you can control the meaning of words, you can control the people who must use the words." -Philip K. Dick
Reply With Quote
Old 9th March 2021
roddierod's Avatar
roddierod roddierod is offline
Real Name: Rod Person
VPN Cryptographer
 
Join Date: Apr 2008
Location: Pittsburgh, Pa
Posts: 429
Default

Quick updates...

I found that OpenBSD wouldn't let me set an MTU over 6000.

The FreeBSD server allowed an MTU of 9000. Still this seemed to do nothing.

On rebooting the OpenBSD machine I noticed RPC portmapper errors in the dmesg. I ended up putting in a 1GB network card I had laying around and those error disappeared. But then I noticed the card wouldn't get over 100MB. So changed network cables and got 1GB. Still this did nothing for nfs write speed to server. But write from server to client was fine.

So I ran smartctl on the server drive and it shows no probablems at all. But I was able to write to another drive on the server at resonable speeds. So even though ever test show that the one drive is fine I ordered a new drive and we'll see when it gets here. But I'm thinking that is the problem.
__________________
"The basic tool for the manipulation of reality is the manipulation of words. If you can control the meaning of words, you can control the people who must use the words." -Philip K. Dick
Reply With Quote
Old 9th March 2021
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,210
Default

Quote:
Originally Posted by roddierod View Post
Quick updates...

I found that OpenBSD wouldn't let me set an MTU over 6000.
That will be dependent on the individual NIC and driver, reported as the "hardmtu" limit, e.g.:
Code:
$ ifconfig em0 hwfeatures
em0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> mtu 1500
        hwfeatures=36<CSUM_TCPv4,CSUM_UDPv4,VLAN_MTU,VLAN_HWTAGGING> hardmtu 9216
...
Quote:
...So even though ever test show that the one drive is fine I ordered a new drive and we'll see when it gets here. But I'm thinking that is the problem.
If so, then any non-NFS "local" writes on the server would show a performance problem if the drive was the bottleneck.
Reply With Quote
Old 9th March 2021
roddierod's Avatar
roddierod roddierod is offline
Real Name: Rod Person
VPN Cryptographer
 
Join Date: Apr 2008
Location: Pittsburgh, Pa
Posts: 429
Default

Quote:
Originally Posted by jggimi View Post
If so, then any non-NFS "local" writes on the server would show a performance problem if the drive was the bottleneck.
Not sure why I didn't think of this so just tried it and yup...perfectly fine. So back to square one.
__________________
"The basic tool for the manipulation of reality is the manipulation of words. If you can control the meaning of words, you can control the people who must use the words." -Philip K. Dick
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Slow rsync backup from zfs pool with 99% of files Septic FreeBSD General 10 29th May 2013 09:45 PM
Best way to transfer files from OpenBSD to Windows 7? guitarscn OpenBSD General 19 22nd December 2010 12:55 AM
Cannot copy large files to Flash Drive sharris FreeBSD General 6 30th July 2010 09:57 AM
Have problem transfer large file bigger 1GB bsdme2 FreeBSD General 9 14th January 2009 05:49 AM
Local transfer is slow. maurobottone OpenBSD General 1 10th January 2009 02:12 PM


All times are GMT. The time now is 07:55 PM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick