DaemonForums  

Go Back   DaemonForums > FreeBSD > FreeBSD General

FreeBSD General Other questions regarding FreeBSD which do not fit in any of the categories below.

Reply
 
Thread Tools Display Modes
  #1   (View Single Post)  
Old 5th November 2008
Bubba_HoTep Bubba_HoTep is offline
New User
 
Join Date: Nov 2008
Posts: 6
Thanked 0 Times in 0 Posts
Default can't access ZFS drives

Hi,

I have a 64 bit FreeBSD7.0 server that I'm using as a backup server. I have a ZFS pool that to appears to be ok:

$ sudo zpool status -x
all pools are healthy

And all the filesystems appear in df output:
$ df
Filesystem Size Used Avail Capacity Mounted on
/dev/ad4s1a 272M 197M 53M 79% /
devfs 1.0K 1.0K 0B 100% /dev
/dev/ad4s1e 161M 22K 148M 0% /tmp
/dev/ad4s1f 2.2G 1.2G 836M 60% /usr
/dev/ad4s1d 210M 7.9M 185M 4% /var
data/corporate 3.0T 941G 2.1T 30% /corporate
data 2.1T 128K 2.1T 0% /data
data/disco 2.3T 231G 2.1T 10% /disco
data/email 2.4T 255G 2.1T 11% /email
data/hphome 2.1T 24G 2.1T 1% /hphome
data/misc 2.1T 14G 2.1T 1% /misc
data/corporate@FriOct17-08:1224266291 3.0T 948G 2.1T 31% /corporate/.zfs/snapshot/FriOct17-08:1224266291
data/corporate@MonOct20-08:1224513013 3.0T 934G 2.1T 30% /corporate/.zfs/snapshot/MonOct20-08:1224513013
data/corporate@TueOct14-08:1224058974 3.0T 938G 2.1T 30% /corporate/.zfs/snapshot/TueOct14-08:1224058974
data/corporate@WedOct15-08:1224082275 3.0T 937G 2.1T 30% /corporate/.zfs/snapshot/WedOct15-08:1224082275
data/corporate@WedOct22-08:1224680388 3.0T 936G 2.1T 30% /corporate/.zfs/snapshot/WedOct22-08:1224680388
data/corporate@ThuOct16-08:1224169692 3.0T 947G 2.1T 30% /corporate/.zfs/snapshot/ThuOct16-08:1224169692

And all the physical disks are present:
$ ls /dev/ | grep ^ad
ad10
ad12
ad4
ad4s1
ad4s1a
ad4s1c
ad4s1d
ad4s1e
ad4s1f
ad6
ad7
ad8
ad9

(ad4, a CF card, is my system disk)

However any time I try to access any of the ZFS filesystems (all in the pool "data") the system just hangs.

So a
$ ls /corporate
or
$ cd /corporate

will hang the system. Obviously I can't get to my data although it appears to be there.

I can perform operations ok on anything on ad4, the system disk.

I added these lines

vm.kmem_size="1536M"
vm.kmem_size_max="1536M"

to my /boot/loader.conf file as per the instructions at http://wiki.freebsd.org/ZFSTuningGuide
and the system is 64 bit with 4G RAM.
All the server has been doing is receiving nightly rsyncs which it did ok for about a month.


I'm new to freeBSD and have kinda run outta ideas of what to try next.


Can anyone help?
Reply With Quote
  #2   (View Single Post)  
Old 5th November 2008
corey_james corey_james is offline
Uber Geek
 
Join Date: Apr 2008
Location: Brisbane, Australia
Posts: 238
Thanked 4 Times in 10 Posts
Default

Have you run a scrub over the FS?

edit:
also - anything in dmesg? When my ZFS partition was dying my system was spewing out messages to /var/log/messages
__________________
"No, that's wrong, Cartman. But don't worry, there are no stupid answers, just stupid people." -- Mr. Garrison

Forum Netiquette
Reply With Quote
  #3   (View Single Post)  
Old 5th November 2008
Bubba_HoTep Bubba_HoTep is offline
New User
 
Join Date: Nov 2008
Posts: 6
Thanked 0 Times in 0 Posts
Default

I thought ZFS didn't have fsck?

dmesg doesn't say anything useful:

$ dmesg |grep ZFS
WARNING: ZFS is considered to be an experimental feature in FreeBSD.
ZFS filesystem version 6
ZFS storage pool version 6

Though it complains about root being improperly unmounted:
$ dmesg |grep -i "warning\|error"
WARNING: / was not properly dismounted
WARNING: /tmp was not properly dismounted
WARNING: /usr was not properly dismounted
WARNING: /var was not properly dismounted
WARNING: ZFS is considered to be an experimental feature in FreeBSD.
nfsd send error 32

And if I do
$ ls /
it hangs.

Which I don't understand - if / were unmounted wouldn't the system be inoperable?
Reply With Quote
  #4   (View Single Post)  
Old 5th November 2008
jb_daefo jb_daefo is offline
Package Pilot
 
Join Date: May 2008
Posts: 196
Thanked 5 Times in 5 Posts
Default

don't know about ZFS.
for a "usual" bsd:
..............
boot to single-user
swapon -a
fsck -p (that takes a long time!)
mount -u /
mount -va
shutdown -p now
.................................
way back in 5.x that would result in lost files.
haven't had lost files with that procedure for years.
..................................
on a post-it on the monitor for easy reference
__________________
using /LOOKAT/ with /var/db/pkg files and portmaster/aliases/pipes/find/grep to meteorically speedup port upgrades/installs...
Reply With Quote
  #5   (View Single Post)  
Old 5th November 2008
phoenix's Avatar
phoenix phoenix is offline
Risen from the ashes
 
Join Date: May 2008
Posts: 699
Thanked 90 Times in 81 Posts
Default

Quote:
Originally Posted by Bubba_HoTep View Post
I thought ZFS didn't have fsck?
It doesn't. But it does have a built-in "scrub" command that checks every bit on every disk against the checksums and repairs any files that don't match (if the pool is mirrored/raidz'd or has copies set to >1).

What's the output of zpool list?

What's the output of dmesg | grep -i zfs or dmesg | grep -i ad?

Can you run top at the console, and ssh to the system from another system, and do the ls and see what's happening in top?

Do you have background fsck disabled in /etc/rc.conf? If not, disable it, reboot into single-user mode, and run fsck -y on all your partitions on ad4. Background fsck can cause all kinds of issues and system lockups. You want fsck to run in the foreground if the system reboots after a lockup.
__________________
Freddie

Help for FreeBSD: Handbook, FAQ, man pages, mailing lists.
Reply With Quote
  #6   (View Single Post)  
Old 6th November 2008
Bubba_HoTep Bubba_HoTep is offline
New User
 
Join Date: Nov 2008
Posts: 6
Thanked 0 Times in 0 Posts
Default

Quote:
But it does have a built-in "scrub" command
Ahhh, ok. I don't want to run scrub just yet cos it's a 5Tb Z2 array, so I'd imagine a scrub would take a bit of time.

Quote:
What's the output of zpool list?
The zpool appears fine:
$ sudo zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
data 5.44T 2.19T 3.25T 40% ONLINE -

Quote:
What's the output of dmesg
dmesg as I said above isn't showing anything that grabs my attention:
$ dmesg |grep -i zfs
WARNING: ZFS is considered to be an experimental feature in FreeBSD.
ZFS filesystem version 6
ZFS storage pool version 6

and looking thru dmesg for the disks:
$ dmesg |grep ^ad
ad4: DMA limited to UDMA33, device found non-ATA66 cable
ad4: 3887MB <TRANSCEND 20070831> at ata2-master UDMA33
ad6: 953869MB <Seagate ST31000340AS SD15> at ata3-master SATA150
ad7: 953869MB <Seagate ST31000340AS SD15> at ata3-slave SATA150
ad8: 953869MB <Seagate ST31000340AS SD15> at ata4-master SATA150
ad9: 953869MB <Seagate ST31000340NS SN05> at ata4-slave SATA300
ad10: 953869MB <Seagate ST31000340AS SD15> at ata5-master SATA150
ad12: 953869MB <Seagate ST31000340AS SD15> at ata6-master SATA150

(there's x6 1Tb disks in the RAIDZ2 array and x1 CF card - the OS disk)


Top runs ok - although there's x4 rsync processes running (which were probably from backups being sent that failed) along with a pile of sshd instances.

It's a headless server so I'm connecting over ssh - which works fine.
I get the feeling that if I reboot it'll come good - but I wanna figure out what happened cos this is my backup server and i get kinda nervous when all the data _just drops out_


Quote:
Do you have background fsck disabled in /etc/rc.conf?
rc.conf didn't have anything explicitly mentioning fsck so I added
background_fsck="NO"
in case the default was YES.

I'm gonna go reboot and fsck now, thanks for the help.

BHT
Reply With Quote
  #7   (View Single Post)  
Old 6th November 2008
Bubba_HoTep Bubba_HoTep is offline
New User
 
Join Date: Nov 2008
Posts: 6
Thanked 0 Times in 0 Posts
Default

Ok I rebooted and ran fsck. Fsck threw up a whole bunch of errors on the system disk, which it fixed.

And it all came good.

I think the problem stems from another problem I have where FreeBSD is only successfully booting every third or fourth boot. I was having to kill the power and reboot multiple times to get the system to come up - obviously that's thrown up some filesystem errors.

Thanks again y'all for the help.



BHT
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Is is possible to access my hard drives from install CD 'Shell' option?... Quaxo OpenBSD Installation and Upgrading 2 17th May 2009 04:37 PM
SATA connected CD/DVD drives, any experience? teig FreeBSD General 13 6th December 2008 01:35 AM
Yet another question about hot swapping drives PeterSteele FreeBSD General 1 13th November 2008 01:54 AM
Adding drives annoys OpenBSD. diw OpenBSD General 4 4th September 2008 02:14 AM
Automounting drives jwhal OpenBSD General 4 12th June 2008 05:00 PM


All times are GMT. The time now is 11:35 PM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick