![]() |
|
FreeBSD General Other questions regarding FreeBSD which do not fit in any of the categories below. |
![]() |
|
Thread Tools | Display Modes |
|
|||
![]()
I have a zfs pool that I want to create a monthly offsite backup of, storing it on a separate disk, which I'm attaching locally.
When I'm performing the rsync, the time it takes to move the majority of each new file is between 1-8 seconds; once the transfer starts, the speed is perfectly fine, usually over 50MB/s - but it's the actual sitting there with the filename, with no other progress, that is causing the delay. My rsync command (with/out -h or --progress makes no difference): Code:
# rsync -turlgxvh --progress /share/ /mnt/ The zfs pool consists of 4x 3TB Western Digital Red disks in raidz2: Code:
# zpool status share pool: share state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM share ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 ada5 ONLINE 0 0 0 ada3 ONLINE 0 0 0 ada4 ONLINE 0 0 0 ada6 ONLINE 0 0 0 errors: No known data errors Code:
# gpart create -s gpt ada0 # gpart add -t freebsd-ufs ada0 # newfs /dev/ada0p1 # mount /dev/ada0p1/ /mnt/ Code:
# zpool iostat share 2 capacity operations bandwidth pool alloc free read write read write ---------- ----- ----- ----- ----- ----- ----- share 2.56T 8.31T 10 0 229K 0 share 2.56T 8.31T 0 0 0 0 share 2.56T 8.31T 0 37 0 68.2K share 2.56T 8.31T 0 0 0 0 share 2.56T 8.31T 0 0 0 0 share 2.56T 8.31T 0 0 0 0 share 2.56T 8.31T 0 0 0 0 share 2.56T 8.31T 12 6 1.46M 17.1K share 2.56T 8.31T 0 0 0 0 share 2.56T 8.31T 0 0 12.5K 0 share 2.56T 8.31T 0 0 0 0 share 2.56T 8.31T 2 34 8.24K 58.7K Code:
# df -h share Filesystem Size Used Avail Capacity Mounted on share 5.3T 1.3T 4.1T 24% /share In future I'll probably just gzip each folder in the share root and transfer it individually. |
|
|||
![]()
To speed things up on the UFS backup disk, you could enable softupdates on the file system and mount with noatime option.
You can check for softupdates by running tunefs: Code:
# tunefs -n /dev/ada0p1 you could enable it.Or to see whether the destination is the bottleneck you could create a test backup tmpfs partition of say 2 or 4 GB and do a partial rsync to that partition. See tmpfs(5) This an /etc/fstab with a tmpfs mount of /tmp: Code:
# dev mount FStype Options Dump Pass# #----------- ----- ----- ------------------- ---- ---- /dev/ada0s3a / ufs rw,noatime 1 1 /dev/ada0s3b none swap sw 0 0 /dev/ada0s3d /usr ufs rw,noatime 2 2 /dev/ada0s3e /home ufs rw,noatime 2 2 /dev/ada0s3f /var ufs rw,noatime 2 2 /dev/ada0s3g /var/www ufs rw,noatime 2 2 /dev/ada0s3h /var/db/mysql ufs rw,noatime 2 2 # tmpfs /tmp tmpfs rw,mode=01777,size=1073741824 0 0 # tmpfs size is in bytes: # calculate the size with 'echo 'x * 1024^2' | bc' for size in MB # calculate the size with 'echo 'x * 1024^3' | bc' for size in GB # NOTE: you cannot use 'noatime' for tmpfs !
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump |
|
|||
![]()
Ah, forgot about tmpfs, good call!
After applying softupdates and noatime, the performance was a little better - it tended to do 4-6 files at 'normal' speed, then hang - rinse and repeat (faster than before though, so already a plus). I thought I'd do a test on something not yet backed up with lots of little files - I had the linux kernel 2.6.31 sources, so used that: Code:
# time rsync -turlgxvh --progress /share/Programming/C/linux-2.6.31/ /tmp/tmpfs/ ... sent 343.97M bytes received 559.85K bytes 76.56M bytes/sec total size is 342.20M speedup is 0.99 2.335u 2.950s 0:03.75 140.8% 475+2250k 22+0io 0pf+0w Code:
# time rsync -turlgxvh --progress /share/Programming/C/linux-2.6.31/ /mnt/Programming/C/linux-2.6.31/ ... sent 343.97M bytes received 559.87K bytes 379.64K bytes/sec total size is 342.20M speedup is 0.99 5.690u 8.376s 15:07.90 1.5% 477+2259k 386+432io 0pf+0w Code:
# smartctl -a /dev/ada0 smartctl 6.1 2013-03-16 r3800 [FreeBSD 9.1-RELEASE amd64] (local build) Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Western Digital Caviar Green Device Model: WDC WD20EADS-00R6B0 Serial Number: WD-WCAVY1950152 LU WWN Device Id: 5 0014ee 2592e4d98 Firmware Version: 01.00A01 User Capacity: 2,000,398,934,016 bytes [2.00 TB] Sector Size: 512 bytes logical/physical Device is: In smartctl database [for details use: -P show] ATA Version is: ATA8-ACS (minor revision not indicated) SATA Version is: SATA 2.6, 3.0 Gb/s Local Time is: Mon May 27 11:00:33 2013 BST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED ... SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 193 193 051 Pre-fail Always - 1778345 3 Spin_Up_Time 0x0027 182 147 021 Pre-fail Always - 7866 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 553 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0 9 Power_On_Hours 0x0032 069 069 000 Old_age Always - 23140 10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 63 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 37 193 Load_Cycle_Count 0x0032 187 187 000 Old_age Always - 41393 194 Temperature_Celsius 0x0022 118 108 000 Old_age Always - 34 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 198 198 000 Old_age Always - 754 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 308 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 1 200 Multi_Zone_Error_Rate 0x0008 001 001 000 Old_age Offline - 261345 SMART Error Log Version: 1 No Errors Logged ... Thanks for the help so far ![]() |
|
|||
![]()
According to http://www.expertreviews.co.uk/inter...viar-green-2tb the disk is quite slow with its' 5400 rpm, especially in transferring small files.
Your test with tmpfs seems to rule out your ZFS setup as cause, so you could try with another disk. Another plausible cause could be the infamous 4K sector alignment issue. What do gpart and diskinfo report about the disk?
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump |
|
|||
![]()
Yeah, the 'normal' slow speed I can live with - but it was the fact the hang was occurring when it wasn't doing the transfer is what got me.
I've put in a WD Caviar Black 2TB in its place, gone through the same setup procedure (enabling softupdates, noatime), and tried the same folder. This time the transfer speed was a normal-looking 4.5MB/s - not spectacular, but certainly fast enough in comparison to ~380K. I'm starting the full backup again from scratch now, and all appears to be running normally: Code:
# zpool iostat share 2 capacity operations bandwidth pool alloc free read write read write ---------- ----- ----- ----- ----- ----- ----- share 2.61T 8.27T 122 8 10.0M 28.0K share 2.61T 8.27T 358 0 44.5M 0 share 2.61T 8.27T 200 0 24.9M 0 share 2.61T 8.27T 432 0 53.7M 0 share 2.61T 8.27T 352 0 43.7M 0 share 2.61T 8.27T 311 0 38.7M 0 Thanks for the help, consider this solved! |
|
|||
![]()
You ignored the possibility of the 4K partition misalignment so I don't consider it solved
![]() Code:
From https://en.wikipedia.org/wiki/Advanced_Format : Quote:
Although not needed for this particular disk because it does not have 4K disk sectors, I aligned the first partition on a 63 x 8 = 504 sectors boundary. Code:
# gpart show => 63 625142385 ada0 MBR (298G) 63 441 - free - (220k) 504 20971440 1 freebsd (10G) 20971944 83885760 2 !166 (40G) 104857704 520284744 3 freebsd [active] (248G) ![]()
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump |
|
|||
![]()
Yeah, I admit I was more in a rush of getting the backup done so I could actually take it into work after the bank holiday!
Since it completed just in time this morning before I left, when I get back from work this evening I'll put the slow disk in and report back ![]() |
|
||||
![]()
When J65nko mentioned 4096-byte alignment in post #4 above, I had an epiphany -- in regards to another thread on drive performance. This post of mine describes the steps a drive takes to conduct an RMW operation. Alignment is one way to minimize extra RMWs.
Keep in mind: data transfers of a sector occur in microseconds. Seeks and rotational latency (RPS Miss) occur in milliseconds. They take three orders of magnitude longer to execute. Anything you can do to eliminate seek or rotational delay should be done. Your WD Red and Green drives are "Advanced Format 512e" devices -- internally they are 4096-byte sector drives, for capacity, which present 512-byte sectors to an OS for compatibility reasons. RMWs will occur when conducting write operations. |
|
|||
![]()
Ok, so here we go:
Code:
# diskinfo -v ada0 ada0 512 # sectorsize 2000398934016 # mediasize in bytes (1.8T) 3907029168 # mediasize in sectors 0 # stripesize 0 # stripeoffset 3876021 # Cylinders according to firmware. 16 # Heads according to firmware. 63 # Sectors according to firmware. WD-WCAVY1950152 # Disk ident. Code:
# gpart show ada0 => 34 3907029101 ada0 GPT (1.8T) 34 3907029101 1 freebsd-ufs (1.8T) I'll try with an initial partition alignment and see how that goes, unless there's anything else you want me to try while I've still got the original setup? |
|
|||
![]()
gpart(8), as well as http://www.wonkity.com/~wblock/docs/html/disksetup.html has some tips on alignment of GPT partitions.
In my previous post I aligned my disk on 63 x 8 = 504 sectors, but that is because I used MBR partitions and for some reasons buried deeply inside the FreeBSD kernel, FreeBSD refuses to accept MBR partitions not aligned on 63 sectors. For GPT partitions this limitation does not apply. It still could turn out that your disk has an hardware issue, but checking whether alignment of the partition helps would be helpful to figure that out. ![]()
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump |
|
|||
![]()
Gotcha
![]() Well - here's what I did (it's now ada6 after rebooting, config): Code:
gpart delete -i 1 ada6 gpart destroy ada6 gpart create -s gpt ada6 gpart add -t freebsd-ufs -b 1M ada6 newfs ada6p1 ![]() I then proceeded to do the same rsync. After 01:58:51 (~2 hours), it had only reached up to linux-2.6.31/arch/arm/mach-realview/core.c - still in the very first 'root' directory. I've also spotted that when doing the gpart/umount/newfs/etc. under this disk, it takes significantly longer than the wd black drive did - maybe 5-6x as long. Most noticeable was the newfs - parts of it near the middle raced through, but the majority of the others suffered a delay. Sometimes even doing an ls or rm -rf hung; in rm's instance, top showed it just sitting in the biowr state for about 4 minutes. All I can assume is that the disk is therefore faulty ![]() |
![]() |
Tags |
zfs rsync |
Thread Tools | |
Display Modes | |
|
|
![]() |
||||
Thread | Thread Starter | Forum | Replies | Last Post |
IPv4 Free Pool Drops Below 10%, 1.0.0.0/8 Allocated | J65nko | News | 4 | 29th January 2010 12:13 PM |
Moving ZFS to a other pool? | amscotti | Solaris | 1 | 28th July 2009 11:49 PM |
backup freeBSD 7.0 using Backup Exec | ccc | FreeBSD General | 2 | 25th April 2009 09:23 PM |
The location of Backup files from portupgrade | paul-lkw | FreeBSD Ports and Packages | 4 | 21st September 2008 05:53 PM |
Rsh/rsync problems | ebzzry | FreeBSD General | 4 | 15th August 2008 08:09 AM |