Slow NFS transfer with large files

[roddierod]

I have a FreeBSD server that with a few NFS shares. The write time to the share from my OpenBSD machine is terribly slow as can be seen in below. the file big.mkv is 7GB:

Move to server:

atomizer$ time mv big.mkv /mnt/Audio/.tmp
   85m11.01s real     0m00.34s user     1m15.02s system

Move from server:

atomizer$ time mv /mnt/Audio/.tmp/big.mkv ~/Downloads
   11m54.42s real     0m00.40s user     0m41.44s system

Copy to server:

atomizer$ time cp big.mkv /mnt/Audio/.tmp
   59m05.32s real     0m00.37s user     1m26.60s system

this is my current /etc/fstab on my OpenBSD machine. I have tried different settings
for -r/-w value 4096,8192,16384

192.168.1.178:/mnt/AUDIO /mnt/Audio nfs rw,noatime,nodev,nosuid,-r=32768,-w=32768 0 0
192.168.1.178:/mnt/BIGSTORE /mnt/nfs nfs rw,noatime,nodev,nosuid,-r=32768,-w=32768 0 0
192.168.1.178:/mnt/FBSD /mnt/FreeBSD nfs rw,noatime,nodev,nosuid,-r=32768,-w=32768 0 0
192.168.1.178:/usr/local/share/MEDIA /mnt/Video nfs rw,noatime,nodev,nosuid,-r=32768,-w=32768 0 0

Interestingly if I try testing transfer with dd everything looks fine...

atomizer$ time dd if=/dev/zero of=/mnt/Audio/.tmp/otest bs=1m count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 88.146 secs (11895802 bytes/sec)
    1m29.56s real     0m00.02s user     0m09.46s system

So I tried the same test from my Macbook and here they are...

Doris:~ rod2$
Doris:~ rod2$ time cp /private/Audio/.tmp/big.mkv /Users/rod2

real	9m0.636s
user	0m0.031s
sys	0m13.321s
Doris:~ rod2$ time mv /Users/rod2/big.mkv /private/Audio/.tmp/big2.mkv

real	10m55.434s
user	0m2.140s
sys	1m28.825s

And dd

Doris:~ rod2$ time dd if=/dev/zero of=/private/Audio/.tmp/test bs=1m count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 89.905967 secs (11663030 bytes/sec)

real	1m30.420s
user	0m0.005s
sys	0m4.379s

[jggimi]

The nfsstat(1) utility may be of help in diagnosing the problem. As would consideration of mount options on the server, such as noatime and -- if your use-case permits it -- async.

• Per the mount_nfs(8) man page, the -r and -w mount options are knobs you twist when the NFS server reports dropped UDP fragments for read or write ops, respectively. Unless nfsstat(1) is reporting the issue, these knobs are unlikely to make a positive impact.
• The mount_nfs(8) man page suggests using -a (read ahead) for large sequential reads, particularly over bandwidth delayed networks. You might consider bumping this value up from the default to see if it makes any difference on read operations. Unfortunately, there's no "write ahead" equivalent due to the laws of physics. That is where async might improve performance if the possibility of data loss is acceptable for your use-case.

[roddierod]

Thanks. I'll try anything at this point. I'm pretty sure I had async at one point and that didn't seem to do anything but I'll try again.

I first added the -r,-w when I started noticing the problem and changing them does do something...but I haven't tried measuring with them removed so I'll give that a shot.

[jggimi]

Async might make a difference on write performance, as metadata would not need to be synchronized.

[roddierod]

it seems that async is not a valid option for nfs mounts. But after a bit more of experimenting and using nfsstat I'm starting to think it might be the OpenBSD machine drive but now sure...

atomizer# nfsstat -c
Client Info:
  Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
    50398       478      7315         0   1556472   5775282       548       174
   Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
      423         0         0         1         0       519         0      5883
    Mknod    Fsstat    Fsinfo  PathConf    Commit
        0       706        13         0       905
Rpc Info:
 TimedOut   Invalid X Replies   Retries  Requests   FrcSync
        0         0      1049      1295   7399117         0
Cache Info:
Attr Hits    Misses Lkup Hits    Misses BioR Hits    Misses BioW Hits    Misses
  2603760     87675    171087      7315      6372   1556472 18446744073708580777   5775282
BioRLHits    Misses BioD Hits    Misses DirE Hits    Misses
        0         0         0         0       355       361

I not extirely clear but that Cache Infor doesn't look cool. idk?

But I got the write from the server down to what looks normal...but writes TO the server are still extremly slow.

[jggimi]

I was thinking async on the server side, but clearly there are limits to what can be done with NFSv3.

The cache is used for Metadata, so increasing cache timeout limits per mount_nfs(8) may reduce your cache misses.

[jggimi]

I happened across an NFSv3/v2 performance guide that may be of interest, though it is Linux-centric. http://nfs.sourceforge.net/nfs-howto/ar01s05.html

[J65nko]

https://www.admin-magazine.com/HPC/A...and-Management mentions using jumbo Ethernet frames, a Maximum Transmission Unit of 9000 bytes, instead of the standard 1500 byte MTU.

I remember that ajacoutot more than 12 years ago complained on bsdforums.org about the same slow NFS performance of OpenBSD. (Later he became an OpenBSD developer)

[roddierod]

j65nko and jggimi thanks I will check both of these out. So far only changing -w is doing anything.

[roddierod]

Quick updates...

I found that OpenBSD wouldn't let me set an MTU over 6000.

The FreeBSD server allowed an MTU of 9000. Still this seemed to do nothing.

On rebooting the OpenBSD machine I noticed RPC portmapper errors in the dmesg. I ended up putting in a 1GB network card I had laying around and those error disappeared. But then I noticed the card wouldn't get over 100MB. So changed network cables and got 1GB. Still this did nothing for nfs write speed to server. But write from server to client was fine.

So I ran smartctl on the server drive and it shows no probablems at all. But I was able to write to another drive on the server at resonable speeds. So even though ever test show that the one drive is fine I ordered a new drive and we'll see when it gets here. But I'm thinking that is the problem.

[jggimi]

Originally Posted by roddierod

Quick updates...

I found that OpenBSD wouldn't let me set an MTU over 6000.

That will be dependent on the individual NIC and driver, reported as the "hardmtu" limit, e.g.:

$ ifconfig em0 hwfeatures
em0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> mtu 1500
        hwfeatures=36<CSUM_TCPv4,CSUM_UDPv4,VLAN_MTU,VLAN_HWTAGGING> hardmtu 9216
...

...So even though ever test show that the one drive is fine I ordered a new drive and we'll see when it gets here. But I'm thinking that is the problem.

If so, then any non-NFS "local" writes on the server would show a performance problem if the drive was the bottleneck.

[roddierod]

Originally Posted by jggimi

If so, then any non-NFS "local" writes on the server would show a performance problem if the drive was the bottleneck.

Not sure why I didn't think of this so just tried it and yup...perfectly fine. So back to square one.