DaemonForums  

Go Back   DaemonForums > FreeBSD > FreeBSD General

FreeBSD General Other questions regarding FreeBSD which do not fit in any of the categories below.

Reply
 
Thread Tools Display Modes
  #1   (View Single Post)  
Old 30th December 2010
sharris sharris is offline
Package Pilot
 
Join Date: Jun 2010
Posts: 146
Thanked 0 Times in 0 Posts
Default dd'ing went to pot ...

I don't know where to post this question or who to share this information with since it seems to be an technical issue with FreeBSD 8.1 for i386 and AMD. I read at other forum outside this one, where they say "if you have an technical question, take it to the one out tens of thousands of open source NEWS GROUP where you might get a single response within a year or two" ... heehee .. They lost their minds...

Maybe I can ask/show here. I'm not set-up for FreeBSD 8.0 anymore, but if I remember correctly, dd'ing was near equal in speed with Fedora-12 and Arch-2009-08 ... Btw, both went to pot on many newer machines if you don't use a few tricks. Fedora 13 and 14 no good either.

FreeBSD 8.1 lost something and I bet they know it. (New rule "Sorry your thread is an technical issue and has been closed") ... WTH Linux talks about EVERYTHING. They have nothing to hide. For what I been reading lately, we will never, ever learn anything outside getting something just INSTALLED to work.

Anyway, what's up with this? Any ideas for a repair job?

I notices this tooooo many times in the pass but this time I documented it for proof because I thought it was my imagination sometimes. Machine don't matter... old or new P3-800 - 512MB RAM or Phem-965-QUAD - 8GB RAM, same results.

Hands Down!!! ½ hour for Linux vs 4 ½ hours for FreeBSD 8.1

I did it twice, last night as I sleep and today, as I watch!!!

WTH

Code:
......................................................
dd if=/dev/ad4s1 | gzip -c | split -b 3500m - /12/PcBSD-Partiton-dd-By-FreeBSD.gz.

ARCH-2009 - 1,504,432,145,920 bytes - 150GB  =  2118sec  =  35min -  1/2 hour  -  71 MB/s

FBSD  8.1 - 1,504,432,145,920 bytes - 150GB  = 16652sec  = 277min - 4 1/2hour  -  9033694 b/s


293,812,785 + 0  record in
293,812,785 + 0  record out
Reply With Quote
  #2   (View Single Post)  
Old 30th December 2010
BSDfan666 BSDfan666 is offline
Real Name: N/A, this is the interweb.
Helpful companion
 
Join Date: Apr 2008
Location: Ontario, Canada
Posts: 2,223
Thanked 193 Times in 184 Posts
Default

What's the problem? you're not specifying a block size.. so dd(1) is reading/write 512 blocks at a time from an unbuffered device.

Try increasing the block size a bit.

I get around 14MB/s on my OpenBSD system with a blocksize of 64k.
Reply With Quote
  #3   (View Single Post)  
Old 30th December 2010
sharris sharris is offline
Package Pilot
 
Join Date: Jun 2010
Posts: 146
Thanked 0 Times in 0 Posts
Default

Quote:
What's the problem???
Are we on the same page? Did you notice that I did not specify a block size for neither OS and still Arch ran at 71 MB/s and FreeBSD 8.1 ran at approx 9 MB/s. Even at default 512 in the man pages, this had taken way too long (4 ½ HOURS vs ½ HOUR).

If you get around 14MB/s on your OpenBSD system and Arch plus Fedora get 71 MB/s and better ... Don't this tell you something is lagging and there may be a possible fix elsewhere in the system. They both use the same open-source dd code I believe, so it seems like it's a BSD I/O thing or Arch and Fedora both has a higher default, but not likely. I still think for what I remember, this did not happen with FreeBSD 8.0.

I would like to bump it up a bit but this is not an average dd command. It's using gZip and I'm not sure where to place the numbers like bs=1M to this appending type dd command. One screw up and my work would be gone! So I ask.

dd if=/dev/ad4s1 | gzip -c | split -b 3500m - /12/file-name.gz.

You may have a point... Arch and Fedora could have higher default build-in if one do not specify a block size, but if not than what? I don't like overlook anything I use. I'll do an search because after noticing this for over a full year I need to get to the bottom of this.
Reply With Quote
  #4   (View Single Post)  
Old 30th December 2010
Carpetsmoker's Avatar
Carpetsmoker Carpetsmoker is offline
Real Name: Martin
Old man from scene 24
 
Join Date: Apr 2008
Location: Eindhoven, Netherlands
Posts: 2,062
Thanked 198 Times in 156 Posts
Default

I use dd on a regular basis to copy disks on FreeBSD, you *need* to specify a block size otherwise it will *always* be slow.

In general, I just choose 16MB.

Example: # dd if=/dev/ad0 of=/dev/ad1 bs=16M

Using a variety of WD disks, I get about 100MB/s transfer rates.

I just tested your command, and it's also very slow on my system, even with a larger block size.
A "normal" dd without pipes to gzip and split works fine.

Using:
# dd if=/dev/ad1 bs=16M | gzip -c > /data/tmp/img

Is also very slow. If I look at top, I notice that gzip is using 100% CPU.

Note that this is on an Intel Atom system, so that's not particularly fast, but the point is that the problem may be in your pipies to gzip or split and not in dd.

Quote:
I would like to bump it up a bit but this is not an average dd command. It's using gZip and I'm not sure where to place the numbers like bs=1M to this appending type dd command.
Code:
dd if=/dev/ad4s1 bs=16M | gzip -c | split -b 3500m - /12/PcBSD-Partiton-dd-By-FreeBSD.gz.
__________________
UNIX was not designed to stop you from doing stupid things, because that would also stop you from doing clever things.
Reply With Quote
  #5   (View Single Post)  
Old 31st December 2010
sharris sharris is offline
Package Pilot
 
Join Date: Jun 2010
Posts: 146
Thanked 0 Times in 0 Posts
Default

Whenever I do a standard dd and specify the size it does in fact match linux in speed or better. The only thing I can figure in this case (where no size is specified for each); I beleive that ARCH-LINUX must be bumping up the default size with-out the user knowledge when no size has been specified by user (running like a bat–out-of-hell with no size given is strange). This may prove that most of these bench-marks are un-fair. Some Linux distro's might be in-a-since cheating. Out the box, with FreeBSD we have to do our own configuration, there are no hand-holding (favors), as should be. I have seen FreeBSD using raw dd at 124M/s or better when a size is given, in ALL cases I only bs=1M. I never tried bigger before other than bs=4096 which I did not trust for some reason. I Never knew I could say bs=16M. Thanks Carpetsmoker for the information, I can't wait, I'm going to try it tonight. Don't get me wrong, to me Arch and Fedora are #1 Linux. Arch is like minimum and use a BSD init. Fedora running Gnome is like the best so far and don't take over your root like Ubuntu. PcBSD is the future as a desktop and FreeBSD as everything that will never change.

I put my money on the ARCH and Fedora be during users favors. That's great for common users but not for people like me. I dd all the time so I going to start recording all results for now on, but it really make no difference because FreeBSD excel in many more areas anyway. I just like to know if it can be done here also. For now I think this has been solved. I'll post a comparison list in the future for the record.

Thanks again
Reply With Quote
  #6   (View Single Post)  
Old 31st December 2010
BSDfan666 BSDfan666 is offline
Real Name: N/A, this is the interweb.
Helpful companion
 
Join Date: Apr 2008
Location: Ontario, Canada
Posts: 2,223
Thanked 193 Times in 184 Posts
Default

I mentioned this in my initial reply to you, there is a subtle difference.. on Linux people use "block" (..buffered) device nodes to access drives, but BSD's generally use "raw/character" (..unbuffered) devices instead.

As stated in the FreeBSD handbook, block devices don't even exist in a traditional sense anymore:
http://www.freebsd.org/doc/en/books/...ics-block.html

OpenBSD still has block devices, but it's always recommended to use the raw character device, the only exception is when mounting.

Because you're dealing with character devices, you may even wish to trying substituting dd(1) for cat(1) when using BSD.
Reply With Quote
  #7   (View Single Post)  
Old 31st December 2010
sharris sharris is offline
Package Pilot
 
Join Date: Jun 2010
Posts: 146
Thanked 0 Times in 0 Posts
Default

That explains it all BSDfan666!!!

and here's real-time, living proof, to what you just said and with links you posted.

I just used my usual command to restore but this time I am using ARCH first:

Code:
cat /12/PcBSD-Partiton-dd-By-FreeBSD.gz.* | gzip -dc | dd of=/dev/sda3

I never done this before under ARCH "I open another TERMINAL and ran TOP".

Within one minute I get:

Quote:
Kernel panic - not syncing: Out of memory and no killable processes...

I running this on the Gigabyte GA-880GA-UD3H mb, Phem-965 processor with DDR3 8GB1600MHz DUAL Viper. Unbelievable!!! My P3's was more than enough all along and never gave me these kinds of issues.

That goes to show, you can turn top on with BSD but not LINUX when using this kind of command. I never knew this until now.

After pulling the plug I tried again, but with-out turning anything on as usual and it replaced it even faster:

Code:
ARCH-2009 - 1,504,432,145,920 bytes - 150GB  =  1851sec  =  30min -  1/2 hour  - 81 MB/s
I always view ARCH as a great dd,split tool with-out the hassle. I use split to save large files to MsDos partitions but I'm about to start saving to UFS and use ZFS. BLOCKS are fast but now I see why we can trust BSD a whole lot more. If this was reverse my first kernel panic ever could have destroyed all that I worked on in a heart-beat.

THANK-YOU

edit:: I thought it was a problem but it did work just fine.

Last edited by sharris; 31st December 2010 at 08:15 AM.
Reply With Quote
  #8   (View Single Post)  
Old 1st January 2011
sharris sharris is offline
Package Pilot
 
Join Date: Jun 2010
Posts: 146
Thanked 0 Times in 0 Posts
Default

Quote:
I just tested your command, and it's also very slow on my system, even with a larger block size.
Carpetsmoker, this is something that had me puzzled from day1 while during straight dd's with no spare time to investigate. Now I see for a fact that a big buffer sometimes can cause a slow down *AND* in my case sometimes my backup's did not work. I than blame ME until I finally got a clue. Maybe it was me but that's why I always used bs=1M. It's usually faster and it have never fail. Now I know it for a fact and don't have to blame myself anymore. Check results below.

Spilt is fun to use and may even be equal to or faster than straight dd. I notice the longer it runs the CPU usages grows slowly than drop way back down again for gzip on my machine. But for LINUX with TOP OPENED it hit 100% with-in seconds and it crashed again.

FreeBSD 8.1

bs=16:
dd - 2.20% adv - 2.55% max
gzip - 76% adv - 88% max
split - 0

bs=1:
dd - 0.22% adv - 1.67% max
gzip - 62% adv - 96% max
split - 0

Code:
ARCH-2009 --- 1,504,432,145,920 bytes - 150GB  =  1851sec  =  30min -  1/2 hour  - 81 MB/s

FBSD-bs=1M -- 1,504,432,145,920 bytes - 150GB  = 2302sec  =  38min -  1/2 hour  - 62 MB/s

FBSD-bs=16M - 1,504,432,145,920 bytes - 150GB  = 3002sec  = 50min  -  1---hour  - 47 MB/s

Quote:
... there is a subtle difference.. on Linux people use "block" (..buffered) device nodes to access drives, but BSD's generally use "raw/character" (..unbuffered) devices instead.

BSDfan666, what a lesson. I never had a clue. Who would have thunk it.

I did a lot of googling and now have a much better understanding of BLOCK and RAW devices. I found and lost a thread that some believe BLOCK device is still in FreeBSD and used only by the system, (That the kind of stuff I alway find) but was removed from /DEV. Kind of make since. It could break the OS if they strip it all out. It took over 30 year to write that multi-million lines of code, even before it changed hands. It takes a life time to circle Pluto twice.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT. The time now is 08:28 AM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick