DaemonForums  

Go Back   DaemonForums > FreeBSD > FreeBSD General

FreeBSD General Other questions regarding FreeBSD which do not fit in any of the categories below.

Reply
 
Thread Tools Display Modes
  #1   (View Single Post)  
Old 26th May 2013
Septic Septic is offline
New User
 
Join Date: Mar 2010
Posts: 9
Thanked 0 Times in 0 Posts
Default Slow rsync backup from zfs pool with 99% of files

I have a zfs pool that I want to create a monthly offsite backup of, storing it on a separate disk, which I'm attaching locally.

When I'm performing the rsync, the time it takes to move the majority of each new file is between 1-8 seconds; once the transfer starts, the speed is perfectly fine, usually over 50MB/s - but it's the actual sitting there with the filename, with no other progress, that is causing the delay.

My rsync command (with/out -h or --progress makes no difference):

Code:
# rsync -turlgxvh --progress /share/ /mnt/
The system is running off an Intel Xeon E3-1220, and has 16GB of ECC RAM (cpu usage < 1%, RAM @ 50% with wired).
The zfs pool consists of 4x 3TB Western Digital Red disks in raidz2:

Code:
# zpool status share
  pool: share
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        share       ONLINE       0     0     0
          raidz2-0  ONLINE       0     0     0
            ada5    ONLINE       0     0     0
            ada3    ONLINE       0     0     0
            ada4    ONLINE       0     0     0
            ada6    ONLINE       0     0     0

errors: No known data errors
The copy is being done to a Western Digital Caviar Green 2TB that was created as such:

Code:
# gpart create -s gpt ada0
# gpart add -t freebsd-ufs ada0
# newfs /dev/ada0p1
# mount /dev/ada0p1/ /mnt/
No problems with the pool bandwidth (except the obvious lack of transfer):

Code:
# zpool iostat share 2
               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
share       2.56T  8.31T     10      0   229K      0
share       2.56T  8.31T      0      0      0      0
share       2.56T  8.31T      0     37      0  68.2K
share       2.56T  8.31T      0      0      0      0
share       2.56T  8.31T      0      0      0      0
share       2.56T  8.31T      0      0      0      0
share       2.56T  8.31T      0      0      0      0
share       2.56T  8.31T     12      6  1.46M  17.1K
share       2.56T  8.31T      0      0      0      0
share       2.56T  8.31T      0      0  12.5K      0
share       2.56T  8.31T      0      0      0      0
share       2.56T  8.31T      2     34  8.24K  58.7K
Note that this is the initial creation of the offsite disk; in future, the file differences will be much smaller and I could live with this speed. For 1.3TB of data though - it's so far been running the whole weekend and it's only done 678GB.

Code:
# df -h share
Filesystem    Size    Used   Avail Capacity  Mounted on
share         5.3T    1.3T    4.1T    24%    /share
Is there anything I can try to see where a bottleneck is (i.e. UFS/ZFS)? I've tried watching the files it has issues on, but they're all variable sizes, types and compressions. The current folder - extracted files from a server 2003 iso - has been going for about 3 hours now!

In future I'll probably just gzip each folder in the share root and transfer it individually.
Reply With Quote
  #2   (View Single Post)  
Old 27th May 2013
J65nko J65nko is offline
Administrator
 
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 3,135
Thanked 182 Times in 149 Posts
Default

To speed things up on the UFS backup disk, you could enable softupdates on the file system and mount with noatime option.

You can check for softupdates by running tunefs:

Code:
#  tunefs -p /dev/ada0s3f
tunefs: POSIX.1e ACLs: (-a)                                disabled
tunefs: NFSv4 ACLs: (-N)                                   disabled
tunefs: MAC multilabel: (-l)                               disabled
tunefs: soft updates: (-n)                                 enabled
tunefs: soft update journaling: (-j)                       disabled
tunefs: gjournal: (-J)                                     disabled
tunefs: trim: (-t)                                         disabled
tunefs: maximum blocks per file in a cylinder group: (-e)  4096
tunefs: average file size: (-f)                            16384
tunefs: average number of files in a directory: (-s)       64
tunefs: minimum percentage of free space: (-m)             8%
tunefs: optimization preference: (-o)                      time
tunefs: volume label: (-L)
Here it already has been enabled but with something like # tunefs -n /dev/ada0p1 you could enable it.

Or to see whether the destination is the bottleneck you could create a test backup tmpfs partition of say 2 or 4 GB and do a partial rsync to that partition. See tmpfs(5)

This an /etc/fstab with a tmpfs mount of /tmp:
Code:
# dev           mount           FStype  Options                                 Dump    Pass#
#-----------    -----           -----   -------------------                     ----    ----
/dev/ada0s3a    /               ufs     rw,noatime                              1       1
/dev/ada0s3b    none            swap    sw                                      0       0
/dev/ada0s3d    /usr            ufs     rw,noatime                              2       2
/dev/ada0s3e    /home           ufs     rw,noatime                              2       2
/dev/ada0s3f    /var            ufs     rw,noatime                              2       2
/dev/ada0s3g    /var/www        ufs     rw,noatime                              2       2
/dev/ada0s3h    /var/db/mysql   ufs     rw,noatime                              2       2
#
tmpfs           /tmp            tmpfs   rw,mode=01777,size=1073741824           0       0

# tmpfs size is in bytes:
# calculate the size with 'echo 'x * 1024^2' | bc' for size in MB
# calculate the size with 'echo 'x * 1024^3' | bc' for size in GB
# NOTE: you cannot use 'noatime' for tmpfs !
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump
Reply With Quote
  #3   (View Single Post)  
Old 27th May 2013
Septic Septic is offline
New User
 
Join Date: Mar 2010
Posts: 9
Thanked 0 Times in 0 Posts
Default

Ah, forgot about tmpfs, good call!

After applying softupdates and noatime, the performance was a little better - it tended to do 4-6 files at 'normal' speed, then hang - rinse and repeat (faster than before though, so already a plus).

I thought I'd do a test on something not yet backed up with lots of little files - I had the linux kernel 2.6.31 sources, so used that:

Code:
# time rsync -turlgxvh --progress /share/Programming/C/linux-2.6.31/ /tmp/tmpfs/

...

sent 343.97M bytes  received 559.85K bytes  76.56M bytes/sec
total size is 342.20M  speedup is 0.99
2.335u 2.950s 0:03.75 140.8%    475+2250k 22+0io 0pf+0w
Clearly the zpool runs nicely, nearly 350MB in under 4 seconds. Back onto the newly remounted backup drive though, it took over 15 minutes, with clear i/o delays:

Code:
# time rsync -turlgxvh --progress /share/Programming/C/linux-2.6.31/ /mnt/Programming/C/linux-2.6.31/

...

sent 343.97M bytes  received 559.87K bytes  379.64K bytes/sec
total size is 342.20M  speedup is 0.99
5.690u 8.376s 15:07.90 1.5%     477+2259k 386+432io 0pf+0w
As it's the destination disk, I thought I'd check its smart details:

Code:
# smartctl -a /dev/ada0
smartctl 6.1 2013-03-16 r3800 [FreeBSD 9.1-RELEASE amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Caviar Green
Device Model:     WDC WD20EADS-00R6B0
Serial Number:    WD-WCAVY1950152
LU WWN Device Id: 5 0014ee 2592e4d98
Firmware Version: 01.00A01
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Size:      512 bytes logical/physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS (minor revision not indicated)
SATA Version is:  SATA 2.6, 3.0 Gb/s
Local Time is:    Mon May 27 11:00:33 2013 BST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

...

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   193   193   051    Pre-fail  Always       -       1778345
  3 Spin_Up_Time            0x0027   182   147   021    Pre-fail  Always       -       7866
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       553
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   069   069   000    Old_age   Always       -       23140
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       63
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       37
193 Load_Cycle_Count        0x0032   187   187   000    Old_age   Always       -       41393
194 Temperature_Celsius     0x0022   118   108   000    Old_age   Always       -       34
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   198   198   000    Old_age   Always       -       754
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       308
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       1
200 Multi_Zone_Error_Rate   0x0008   001   001   000    Old_age   Offline      -       261345

SMART Error Log Version: 1
No Errors Logged

...
While I didn't think the drive had been powered on for nearly 3 years in total, the raw read error rate is a bit concerning - could it be as simple as the drive that's giving grief? All it's usage before now hasn't shown that it's been stalling or having any other form of issues, hence why I hadn't considered it before.

Thanks for the help so far
Reply With Quote
  #4   (View Single Post)  
Old 27th May 2013
J65nko J65nko is offline
Administrator
 
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 3,135
Thanked 182 Times in 149 Posts
Default

According to http://www.expertreviews.co.uk/inter...viar-green-2tb the disk is quite slow with its' 5400 rpm, especially in transferring small files.

Your test with tmpfs seems to rule out your ZFS setup as cause, so you could try with another disk.

Another plausible cause could be the infamous 4K sector alignment issue.
What do gpart and diskinfo report about the disk?
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump
Reply With Quote
  #5   (View Single Post)  
Old 27th May 2013
Septic Septic is offline
New User
 
Join Date: Mar 2010
Posts: 9
Thanked 0 Times in 0 Posts
Default

Yeah, the 'normal' slow speed I can live with - but it was the fact the hang was occurring when it wasn't doing the transfer is what got me.

I've put in a WD Caviar Black 2TB in its place, gone through the same setup procedure (enabling softupdates, noatime), and tried the same folder. This time the transfer speed was a normal-looking 4.5MB/s - not spectacular, but certainly fast enough in comparison to ~380K.

I'm starting the full backup again from scratch now, and all appears to be running normally:

Code:
# zpool iostat share 2
               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
share       2.61T  8.27T    122      8  10.0M  28.0K
share       2.61T  8.27T    358      0  44.5M      0
share       2.61T  8.27T    200      0  24.9M      0
share       2.61T  8.27T    432      0  53.7M      0
share       2.61T  8.27T    352      0  43.7M      0
share       2.61T  8.27T    311      0  38.7M      0
I think I'll give the green drive another go in a linux or windows box, and see if it has any similar issues - it hasn't before now, so I'm curious to see the result. I'll update if it's interesting.

Thanks for the help, consider this solved!
Reply With Quote
  #6   (View Single Post)  
Old 27th May 2013
J65nko J65nko is offline
Administrator
 
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 3,135
Thanked 182 Times in 149 Posts
Default

You ignored the possibility of the 4K partition misalignment so I don't consider it solved

Code:
#  diskinfo -v ada0
ada0
        512             # sectorsize
        400088457216    # mediasize in bytes (372G)
        781422768       # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        775221          # Cylinders according to firmware.
        16              # Heads according to firmware.
        63              # Sectors according to firmware.
        S0NFJDWQ304886  # Disk ident.
If your 'slow' disk reports a sector size of 4096 and if that partition p is not aligned on a 4K boundary it will have a negative impact on disk performance.

From https://en.wikipedia.org/wiki/Advanced_Format :

Quote:
The translation process is more complicated when writing data that is either not a multiple of 4K or not aligned to a 4K boundary. In these instances, the hard drive must read the entire 4,096-byte sector containing the targeted data into internal memory, integrate the new data into the previously existing data and then rewrite the entire 4,096-byte sector onto the disk media. This operation, known as read-modify-write (RMW), can require additional revolution of the magnetic disks, resulting in a perceptible performance impact to the system user. Performance analysis conducted by IDEMA and the hard drive vendors indicates that approximately five to ten percent of all write operations in a typical business PC user environment may be misaligned and a RMW performance penalty incurred.
That is why I asked for the output of diskinfo(8) and gpart(8)

Although not needed for this particular disk because it does not have 4K disk sectors, I aligned the first partition on a 63 x 8 = 504 sectors boundary.

Code:
 # gpart show
=>       63  625142385  ada0  MBR  (298G)
         63        441        - free -  (220k)
        504   20971440     1  freebsd  (10G)
   20971944   83885760     2  !166  (40G)
  104857704  520284744     3  freebsd  [active]  (248G)
So please check the disk sector size and partition alignment
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump
Reply With Quote
  #7   (View Single Post)  
Old 28th May 2013
Septic Septic is offline
New User
 
Join Date: Mar 2010
Posts: 9
Thanked 0 Times in 0 Posts
Default

Yeah, I admit I was more in a rush of getting the backup done so I could actually take it into work after the bank holiday!

Since it completed just in time this morning before I left, when I get back from work this evening I'll put the slow disk in and report back
Reply With Quote
  #8   (View Single Post)  
Old 28th May 2013
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 3,613
Thanked 214 Times in 189 Posts
Default

When J65nko mentioned 4096-byte alignment in post #4 above, I had an epiphany -- in regards to another thread on drive performance. This post of mine describes the steps a drive takes to conduct an RMW operation. Alignment is one way to minimize extra RMWs.

Keep in mind: data transfers of a sector occur in microseconds. Seeks and rotational latency (RPS Miss) occur in milliseconds. They take three orders of magnitude longer to execute. Anything you can do to eliminate seek or rotational delay should be done.

Your WD Red and Green drives are "Advanced Format 512e" devices -- internally they are 4096-byte sector drives, for capacity, which present 512-byte sectors to an OS for compatibility reasons. RMWs will occur when conducting write operations.
Reply With Quote
  #9   (View Single Post)  
Old 28th May 2013
Septic Septic is offline
New User
 
Join Date: Mar 2010
Posts: 9
Thanked 0 Times in 0 Posts
Default

Ok, so here we go:

Code:
 # diskinfo -v ada0
ada0
        512             # sectorsize
        2000398934016   # mediasize in bytes (1.8T)
        3907029168      # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        3876021         # Cylinders according to firmware.
        16              # Heads according to firmware.
        63              # Sectors according to firmware.
        WD-WCAVY1950152 # Disk ident.
Code:
# gpart show ada0
=>        34  3907029101  ada0  GPT  (1.8T)
          34  3907029101     1  freebsd-ufs  (1.8T)
Whilst I'm aware of sector sizes, head movement, parking, etc., it's just the level of slowness - no normal drive should take more than one second to copy two tiny files (< 2KB) when there's no other i/o on the disk - and assuming there's no other faults.

I'll try with an initial partition alignment and see how that goes, unless there's anything else you want me to try while I've still got the original setup?
Reply With Quote
Old 29th May 2013
J65nko J65nko is offline
Administrator
 
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 3,135
Thanked 182 Times in 149 Posts
Default

gpart(8), as well as http://www.wonkity.com/~wblock/docs/html/disksetup.html has some tips on alignment of GPT partitions.

In my previous post I aligned my disk on 63 x 8 = 504 sectors, but that is because I used MBR partitions and for some reasons buried deeply inside the FreeBSD kernel, FreeBSD refuses to accept MBR partitions not aligned on 63 sectors.
For GPT partitions this limitation does not apply.

It still could turn out that your disk has an hardware issue, but checking whether alignment of the partition helps would be helpful to figure that out.
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump
Reply With Quote
Old 29th May 2013
Septic Septic is offline
New User
 
Join Date: Mar 2010
Posts: 9
Thanked 0 Times in 0 Posts
Default

Gotcha

Well - here's what I did (it's now ada6 after rebooting, config):

Code:
gpart delete -i 1 ada6
gpart destroy ada6
gpart create -s gpt ada6
gpart add -t freebsd-ufs -b 1M ada6
newfs ada6p1
Just for kicks, I did a fsck - but no errors

I then proceeded to do the same rsync. After 01:58:51 (~2 hours), it had only reached up to linux-2.6.31/arch/arm/mach-realview/core.c - still in the very first 'root' directory.

I've also spotted that when doing the gpart/umount/newfs/etc. under this disk, it takes significantly longer than the wd black drive did - maybe 5-6x as long. Most noticeable was the newfs - parts of it near the middle raced through, but the majority of the others suffered a delay.

Sometimes even doing an ls or rm -rf hung; in rm's instance, top showed it just sitting in the biowr state for about 4 minutes.

All I can assume is that the disk is therefore faulty
Reply With Quote
Reply

Tags
zfs rsync

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
IPv4 Free Pool Drops Below 10%, 1.0.0.0/8 Allocated J65nko News 4 29th January 2010 12:13 PM
Moving ZFS to a other pool? amscotti Solaris 1 28th July 2009 11:49 PM
backup freeBSD 7.0 using Backup Exec ccc FreeBSD General 2 25th April 2009 09:23 PM
The location of Backup files from portupgrade paul-lkw FreeBSD Ports and Packages 4 21st September 2008 05:53 PM
Rsh/rsync problems ebzzry FreeBSD General 4 15th August 2008 08:09 AM


All times are GMT. The time now is 04:59 AM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick