DaemonForums  

Go Back   DaemonForums > FreeBSD > FreeBSD General

FreeBSD General Other questions regarding FreeBSD which do not fit in any of the categories below.

Reply
 
Thread Tools Display Modes
  #1   (View Single Post)  
Old 20th January 2010
J65nko J65nko is offline
Administrator
 
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 4,125
Default Immense delayed write to file system (ZFS and UFS2), performance issues

From the FreeBSD stable mailing list:
Quote:
I realise a strange behaviour of several FreeBSD 8.0-STABLE/amd64 boxes.
All boxes have the most recent STABLE. One box is a UP system, two
others SMP boxes, one with a Q6600 4-core, another XEON with 2x 4-cores
(Dell Poweredge III).

Symptome: All boxes have ZFS and UFS2 filesystems. Since two weeks or
so, sometimes the I/O performance drops massively when doing 'svn
update', 'make world' or even 'make kernel'. It doesn't matter what
memory and how many cpu the box has, it get stuck for several seconds
and freezing. On the UP box, this is sometimes for 10 - 20 seconds.
A very interesting phenomenon is the massively delayed file writing on
ZFS filesystems I realise. Editing a file in 'vi' running on one XTerm
and having in another Xterminal my shell for compiling this file, it
takes sometimes up to 20 seconds to get the file updated after it has
been written. It's like having an old, slow NFS connection with long
cache delays.
These massively delayed file transactions are not necessarely under
heavy load, sometimes they occur in a relaxed situation. They seem to
occur much more often on the UP box than on the SMP boxes, but this
strange phenomenon also occur on the Dell Poweredge II, which has 16GB
RAM and summa summarum 16 cores. This phenomenon does occur on ZFS- and
UFS2 filesystems as well. It is hardly reproducable.
The discussion then focuses on :
Quote:
The disks involved don't happen to be Western Digital Green Power disks,
do they? The Intelli-Park function in these disks are wrecking havoc
with I/O in Linux-land at least, causing massive stalls and iowait
through the roof during the 25-30 seconds it takes for the heads to
unload after parking. I have two of these disks sitting on my desk now
collecting dust...
Althoug the OP did not use these disks, still a very interesting read
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump

Last edited by J65nko; 20th January 2010 at 10:35 PM. Reason: boldfacing the disk type
Reply With Quote
  #2   (View Single Post)  
Old 20th January 2010
phoenix's Avatar
phoenix phoenix is offline
Risen from the ashes
 
Join Date: May 2008
Posts: 696
Default

We (now) have these disks in our ZFS storage server, and things are starting to go downhill. They were just too good of a bargain to pass up (~$100 CDN for 1.5 TB drives, over $200 CDN for the RE 1.5 TB drives). Now we realise that you get what you pay for.

In under 6 weeks of usage, some of the drives are showing almost 40,000 Load Cycles. At that rate, we'll be past the 300,000 mark within a year, possibly 18 months.

Things were running fine until this week. Out of 8 drives in one raidz2 vdev, we've successfully replaced 5. The resilver process took ~60 hours to complete for each drive.

Last Friday, I replaced the 6th drive, and it still hasn't finished the resilver. It's averaging 1-3 GB per hour. Throughput on the entire pool has dropped to 9 MBps ... even after a reboot. It's just nuts! Any zfs/zpool operations except list/status takes multiple minutes to complete. Even unmounting a snapshot takes upwards of 10 minutes.

Our lovely backup process that used to process ~110 servers in under 11 hours barely complete a third of that now. I'm about ready to start re-replacing these drives with the original 500 GB WD Caviar Blue and Caviar Black drives.
__________________
Freddie

Help for FreeBSD: Handbook, FAQ, man pages, mailing lists.
Reply With Quote
  #3   (View Single Post)  
Old 20th January 2010
Carpetsmoker's Avatar
Carpetsmoker Carpetsmoker is offline
Real Name: Martin
Tcpdump Spy
 
Join Date: Apr 2008
Location: Netherlands
Posts: 2,243
Default

Freddie, these problems occured only with FreeBSD 8 or also with FreeBSD 7?
I'm also curious if you have GP drives or GP RE (Raid Edition) drives?

This is interesting because we sell quite a few WD drives, as of late I've noticed the GP drives are become more and more common ...

Running a few test with this and different OS' just got on my TODO list ...
__________________
UNIX was not designed to stop you from doing stupid things, because that would also stop you from doing clever things.
Reply With Quote
  #4   (View Single Post)  
Old 20th January 2010
J65nko J65nko is offline
Administrator
 
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 4,125
Default

From http://www.nordic-pc.com/node/422
Quote:
Green Power drives spin around 5400rpm or so,

[snip]

Also, and very importantly, this drive is not for RAID systems. To keep with the low-power design, Western Digital chose to automatically park the drive heads at times, which some RAID systems will see as a hard drive failure. So you could be in a constant RAID rebuild if the drives are mirrored, or if the drives are in RAID-5 for parity, you could lose all of your data. WD has acknowledged this, and released the RE4-GP 2TB hard drive for users who need big RAID arrays. They are really pricey though, so you may want to stick with a 1TB Caviar Black for your RAID solutions.
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump
Reply With Quote
  #5   (View Single Post)  
Old 20th January 2010
phoenix's Avatar
phoenix phoenix is offline
Risen from the ashes
 
Join Date: May 2008
Posts: 696
Default

Quote:
Originally Posted by Carpetsmoker View Post
Freddie, these problems occured only with FreeBSD 8 or also with FreeBSD 7?
I'm also curious if you have GP drives or GP RE (Raid Edition) drives?

This is interesting because we sell quite a few WD drives, as of late I've noticed the GP drives are become more and more common ...

Running a few test with this and different OS' just got on my TODO list ...
These are WD Caviar Green 1.5 TB drives. The desktop version.

What's really annoying is the huge jumble of drive variations that are available. Jeremy Chadwick broke it down into a nice table on one of the FreeBSD mailing lists.

We've learnt our lesson, and won't be using any drives with Green in the name from now on, regardless of the manufacturer.
__________________
Freddie

Help for FreeBSD: Handbook, FAQ, man pages, mailing lists.
Reply With Quote
  #6   (View Single Post)  
Old 20th January 2010
phoenix's Avatar
phoenix phoenix is offline
Risen from the ashes
 
Join Date: May 2008
Posts: 696
Default

Quote:
Originally Posted by J65nko View Post
Green drives from WD have a variable speed motor that can go as high as 7200 or as low as ... who knows.

The most annoying thing is that the "idle timeout" to park the heads is ... 8 seconds. In a desktop drive!! Talk about unnecessary wear and tear on the drive. Just browsing the web will causes the drive to thunk about, since Firefox has a 30s (or 10s?) "update the open tabs list" timer. So if you read a page for more than 8s, the drive will park.

I can understand having that feature in a laptop drive, where you don't want to be spinning the drive unless absolutely necessary. But in a desktop drive?

WD is going to be getting a lot of returns on Green drives.
__________________
Freddie

Help for FreeBSD: Handbook, FAQ, man pages, mailing lists.
Reply With Quote
  #7   (View Single Post)  
Old 8th February 2010
phoenix's Avatar
phoenix phoenix is offline
Risen from the ashes
 
Join Date: May 2008
Posts: 696
Default

Contrary to the vast majority of online reports ... one can use the wdidle3 utility to "disable" this horrible idle timeout setting.

While one cannot truly disable it, one can set the timeout to 62 minutes, effectively disabling it. Or select a value that reflects your workload.

I've "disabled" this on the 7 drives that have been put into our storage server. The resilver throughput for that 7th drive has jumped from an average of 7 MBps to an average of 40 MBps!!

Checking the Load_Cycle_Count via smartctl now shows the numbers as not moving. Previously, they would go up quite quickly (1 drive has 50,000 after less than 2 months of use). Even using a twiddle script running every 5 seconds wouldn't completely stop this from increasing.

Unfortunately, the wdtler utility does not work on these drives. But it's still an improvement. So maybe they aren't totally worthless drives.
__________________
Freddie

Help for FreeBSD: Handbook, FAQ, man pages, mailing lists.
Reply With Quote
  #8   (View Single Post)  
Old 27th February 2010
Mr_Gibson Mr_Gibson is offline
New User
 
Join Date: May 2008
Posts: 3
Default

OK, this is interesting information. I've been planning on moving my storage on my home file server to a fairly large raidz2 config for some time but hasn't gotten around buying the drives and doing it.
I guess to aviod future headaches, it's best to stay away from anything with "green" in it.
To bad, they are usually a pretty good bargain.
(I remember seeing part of this discussion in the mailling list archives, but must have missed the last bit.)
Reply With Quote
  #9   (View Single Post)  
Old 27th February 2010
arch arch is offline
Port Guard
 
Join Date: Jun 2008
Posts: 38
Default

Quote:
Originally Posted by phoenix View Post
What's really annoying is the huge jumble of drive variations that are available. Jeremy Chadwick broke it down into a nice table on one of the FreeBSD mailing lists.
can you point link to that table?
__________________
Verbose mode can also be turned on for SSH2 with the (surprise!) VerboseMode keyword.
Reply With Quote
Old 25th August 2010
Carpetsmoker's Avatar
Carpetsmoker Carpetsmoker is offline
Real Name: Martin
Tcpdump Spy
 
Join Date: Apr 2008
Location: Netherlands
Posts: 2,243
Default

I'm considering getting a pair of WD15EADS drives (1.5TB GP/32MB cache), for a gmirror.

There's also the WD15EARS, which has 64MB cache (Which is near-useless AFAIK) and so called "Advanced format", which seems to be some sort of marketing speak for a 4K sector size.

Is there any advantage for a 4K sector size instead of the default 512 bytes? I know about the disk alignment (Start at 64, not 63), but is there are real world benefit or drawback?

(It's also 15 euro's more expensive)

Most importantly however, the WD site mentions:
Quote:
This firmware modifies the behavior of the drive to wait longer before positioning the heads in their park position and turning off unnecessary electronics. This utility is designed to upgrade the firmware of the following hard drives: WD1000FYPS-01ZKB0, WD7500AYPS-01ZKB0, WD7501AYPS-01ZKB0.
CAUTION: Do not attempt to run this software on any hard drives other than what is listed above.
It's not entirely clear to me if this utility only works for listed models or for all GP drives (including the WD15EADS/EARS drives)?
__________________
UNIX was not designed to stop you from doing stupid things, because that would also stop you from doing clever things.
Reply With Quote
Old 25th August 2010
vermaden's Avatar
vermaden vermaden is offline
Administrator
 
Join Date: Apr 2008
Location: pl_PL.lodz
Posts: 1,056
Default

Quote:
Originally Posted by Carpetsmoker View Post
Is there any advantage for a 4K sector size instead of the default 512 bytes?
Here You have detailed comparision between WD10EADS and WD10EARS:
http://tomshardware.com/reviews/wd-4k-sector,2554.html

Quote:
Originally Posted by Carpetsmoker View Post
It's not entirely clear to me if this utility only works for listed models or for all GP drives (including the WD15EADS/EARS drives)?
According to phoenix it does work:
http://forums.freebsd.org/showpost.php?p=98908&postcount=21
__________________
religions, worst damnation of mankind
"If 386BSD had been available when I started on Linux, Linux would probably never had happened." Linus Torvalds

Linux is not UNIX! Face it! It is not an insult. It is fact: GNU is a recursive acronym for “GNU's Not UNIX”.
vermaden's: links resources deviantart spreadbsd
Reply With Quote
Old 5th September 2010
phoenix's Avatar
phoenix phoenix is offline
Risen from the ashes
 
Join Date: May 2008
Posts: 696
Default

Quote:
Originally Posted by Carpetsmoker View Post
I'm considering getting a pair of WD15EADS drives (1.5TB GP/32MB cache), for a gmirror.

There's also the WD15EARS, which has 64MB cache (Which is near-useless AFAIK) and so called "Advanced format", which seems to be some sort of marketing speak for a 4K sector size.

Is there any advantage for a 4K sector size instead of the default 512 bytes? I know about the disk alignment (Start at 64, not 63), but is there are real world benefit or drawback?
Skip all the WD "Advanced Format" drives. They're crap. They report a 512 KB physical sector size to the OS, even though they use a 4 KB physical sector size. This causes all kinds of performance issues for everything.

4 KB drives from other manufacturers correctly report the real physical sector size, so the OS/filesystem will work correctly.
__________________
Freddie

Help for FreeBSD: Handbook, FAQ, man pages, mailing lists.
Reply With Quote
Old 15th September 2010
Carpetsmoker's Avatar
Carpetsmoker Carpetsmoker is offline
Real Name: Martin
Tcpdump Spy
 
Join Date: Apr 2008
Location: Netherlands
Posts: 2,243
Default

So I got the WD15EADS, used wdidle, and that worked great:

Code:
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       60
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       4
As a sidenote, wdidle just runs on *all* disks, I had an old WD 20GB disk connected at the time and it tried to disable the idle timeout on that disk too. meh :-/

I also had the chance to get my hands on some other WD GP disks such as the WD10EADS and WD20EARS, and wdidle worked on those too.

... When I look on the interwebz I get conflicting reports about which drives do and don't work ... Internet sucks.
__________________
UNIX was not designed to stop you from doing stupid things, because that would also stop you from doing clever things.
Reply With Quote
Reply

Tags
intelli-park, iowait, western digital green power

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
File system not properly dismounted rex FreeBSD General 6 12th September 2008 02:45 PM
File system at more than 100% michaelrmgreen FreeBSD General 4 28th July 2008 01:52 PM
Which file system use to share data on Bsd system? aleunix Other BSD and UNIX/UNIX-like 2 1st June 2008 04:14 PM
File system UFS2 gjournal support aleunix Other BSD and UNIX/UNIX-like 7 30th May 2008 05:13 AM
What is good support to the file system ext3? aleunix Solaris 3 29th May 2008 03:50 PM


All times are GMT. The time now is 10:46 AM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick