DaemonForums  

Go Back   DaemonForums > OpenBSD > OpenBSD General

OpenBSD General Other questions regarding OpenBSD which do not fit in any of the categories below.

Reply
 
Thread Tools Display Modes
  #1   (View Single Post)  
Old 1st March 2018
hanzer's Avatar
hanzer hanzer is offline
Real Name: Adam Jensen
just passing through
 
Join Date: Oct 2013
Location: EST USA
Posts: 314
Default How would you set up a 4TB USB drive?

How would you set up a 4TB USB drive for maximum portability? I have an OpenBSD laptop, a FreeBSD server, and a Scientific Linux workstation here. The drive had ZFS on it which worked nicely with FreeBSD. The SL machine also runs ZFS but I didn't risk plugging in the USB drive (it had important data on it during that period). Anyway, now the drive needs to sneaker-net data between machines - most likely including MS Windows machines. What kind of disk/filesystem setup would you, dear reader, recommend for this use of the hard drive?

This is currently running on the FreeBSD machine but it is taking a long time to format and I'm not sure if it makes sense.

Code:
root@minerva:~ # gpart destroy -F da1                                       
da1 destroyed                                             
root@minerva:~ # gpart create -s GPT da1      
da1 created                                                                   
root@minerva:~ # gpart add -t ms-basic-data -a 1M -s 925G -l "WD1" da1       
da1p1 added                                                                 
root@minerva:~ # gpart add -t ms-basic-data -a 1M -s 925G -l "WD2" da1 
da1p2 added                                                               
root@minerva:~ # gpart add -t ms-basic-data -a 1M -s 925G -l "WD3" da1        
da1p3 added                             
root@minerva:~ # gpart add -t ms-basic-data -a 1M -l "WD4" da1                
da1p4 added                                                                  
root@minerva:~ # gpart show da1                                             
=>        40  7813969840  da1  GPT  (3.6T)                                    
          40        2008       - free -  (1.0M)                           
        2048  1939865600    1  ms-basic-data  (925G)                          
  1939867648  1939865600    2  ms-basic-data  (925G)                    
  3879733248  1939865600    3  ms-basic-data  (925G)                          
  5819598848  1994369024    4  ms-basic-data  (951G)                     
  7813967872        2008       - free -  (1.0M)     

root@minerva:~ # newfs_msdos -F32 -L WD1 /dev/da1p1
Does this seem reasonable? Any ideas or recommendations for other approaches?

Many thanks!
Reply With Quote
  #2   (View Single Post)  
Old 1st March 2018
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,977
Default

What's the sector size of the drive? FAT32 is limited to 2TB with 512-byte sectors. With 2KB sectors, the max volume size is 8TB, with 4KB sectors, the max is 16TB. Note that if you need individual files larger than 1-byte-less-than-4GB, or if you need any form of access control, then FAT is not the solution.

Because of Windows, I'd consider NTFS. On OpenBSD, sysutils/ntfs-3g uses a FUSE implementation, and that is slower than a native filesystem. I have never tried using it with large file systems, so that is something you should test, if you consider NTFS.

Last edited by jggimi; 1st March 2018 at 06:34 PM. Reason: clarity
Reply With Quote
  #3   (View Single Post)  
Old 1st March 2018
hanzer's Avatar
hanzer hanzer is offline
Real Name: Adam Jensen
just passing through
 
Join Date: Oct 2013
Location: EST USA
Posts: 314
Default

It's a new, large hard drive so I would guess (Advanced Format) 4k sectors. newfs_msdos is still running.

Very large file size (>4G) isn't needed, nor is access control. This drive will be a mule for multimedia files - nothing sophisticated at all is needed other than reliable multi-platform read/write compatibility.

I hadn't really considered that NTFS would have a sufficiently consistent implementation across the various platforms. I've had some problems with it (and ext2) http://daemonforums.org/showthread.php?t=9572#post58714

Do you think the FAT32 approach would be more reliable? If the FreeBSD test doesn't go well, how would you layout a FAT32 filesystem for a 4TB AF4k drive using OpenBSD? Multiple partitions are okay if needed.
Reply With Quote
  #4   (View Single Post)  
Old 1st March 2018
bsd-keith bsd-keith is offline
Real Name: Keith
Open Source Software user
 
Join Date: Jun 2014
Location: Surrey/Hants Border, England
Posts: 344
Default

My personal largest drive is only 500GB, & I would use Fat32 if I wanted to transfer files between different OSes.

If I had a drive your size I might be tempted to partition it, depending on how much data needs to be transferred at any one time.

I have found problems trying to use ext2, & would not recommend using it.
__________________
Linux since 1999, & also a BSD user.
Reply With Quote
  #5   (View Single Post)  
Old 1st March 2018
hanzer's Avatar
hanzer hanzer is offline
Real Name: Adam Jensen
just passing through
 
Join Date: Oct 2013
Location: EST USA
Posts: 314
Default

Quote:
Originally Posted by bsd-keith View Post
My personal largest drive is only 500GB, & I would use Fat32 if I wanted to transfer files between different OSes.

If I had a drive your size I might be tempted to partition it, depending on how much data needs to be transferred at any one time.

I have found problems trying to use ext2, & would not recommend using it.
Hi!

I've also had problems with ext2 and NTFS on a USB drive in the past. I am mostly interested in the particulars of how to partition the drive for FAT32, what kind of partition table to use, how to lay it all out, any parameters or tweaks and the switches to commands, etc. The current test is starting with FreeBSD and uses a GPT partition table; the partition type I am trying is ms-basic-data simply because a type of fat32 resulted in an error from gpart. If that doesn't go well, I would like to lay out the partitions and filesystem with OpenBSD, so I'm just gearing up for that. Any advice on the sequence of commands and the various knobs and values that might work well for my situation?
Reply With Quote
  #6   (View Single Post)  
Old 1st March 2018
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,977
Default

Quote:
Originally Posted by hanzer View Post
It's a new, large hard drive so I would guess (Advanced Format) 4k sectors. newfs_msdos is still running.
Don't guess. Determine your drive's sector size. If it isn't obvious to you from command outputs, plug the model number into an Internet search and check the published specification.
Quote:
Do you think the FAT32 approach would be more reliable?
It is certainly more well-used, and has a vnode(9) implementation, so is likely to be more reliable than a FUSE implementation, merely because of less links in the chain between the application and the on-disk sector. Certainly faster.
Quote:
If the FreeBSD test doesn't go well, how would you layout a FAT32 filesystem for a 4TB AF4k drive using OpenBSD?
I don't happen to have one. But I would follow the configuration guidance in wikipedia, which seems to indicate 32KB cluster size is optimum for 4K sector sizes.

I'd be tempted to use Windows for formatting, only because it has a "quick" option which newfs_msdos doesn't have. But that may cause issues with GPT partitioning, as Windows makes decisions outside of end-users' control, such as removing MBR/GPT on devices it decides are "removable" based on how the USB firmware identifies the device.

Partitioning depends upon your use. Be aware that while partitioning may limit storage flexibility, it may also limit any impacts from errors to a single filesystem.
Quote:
Originally Posted by hanzer View Post
...what kind of partition table to use, how to lay it all out...
MBRs are limited to 2TB on 512-byte sector drives. If the drive is 512-byte sector, GPT is the correct choice, especially since this is intended to be shared heterogeneously. (OpenBSD-only use would not require a GPT, as the 2TB MBR limit can be ignored when writing the on-disk disklabel.)
Quote:
...Any advice on the sequence of commands and the various knobs and values that might work well for my situation?
Read newfs_msdos(8) and make use of the output from -N. The filesystem's sector size must match the physical sector size, whether 512, 4096, or 8192 bytes.

Last edited by jggimi; 1st March 2018 at 11:30 PM. Reason: a little bit more on MBR
Reply With Quote
  #7   (View Single Post)  
Old 2nd March 2018
hanzer's Avatar
hanzer hanzer is offline
Real Name: Adam Jensen
just passing through
 
Join Date: Oct 2013
Location: EST USA
Posts: 314
Default

It's an AF4k drive. Can OpenBSD's fdisk (or some other utility program) create fat32 partitions? FreeBSD's gpart ms-basic-data partition type shows up as fat12 in OpenBSD.
Reply With Quote
  #8   (View Single Post)  
Old 2nd March 2018
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,977
Default

All I know is that in GPT-land, Microsoft treats all "Basic Data Partitions" equivalently, and apparently inspects leading sectors to determine what they are ... whether NTFS, FAT, or exFAT.

https://en.wikipedia.org/wiki/Micros...data_partition

I don't thing having a partitioning tool reporting "FAT12" is going to matter. But that is only a guess on my part.
Reply With Quote
  #9   (View Single Post)  
Old 2nd March 2018
hanzer's Avatar
hanzer hanzer is offline
Real Name: Adam Jensen
just passing through
 
Join Date: Oct 2013
Location: EST USA
Posts: 314
Default

On FreeBSD, this:
Code:
gpart destroy -F da1
gpart create -s GPT da1
gpart add -t ms-basic-data -a 1M -s 930G -l "WD1" da1
newfs_msdos -F 32 -S 4096 -L WD1 /dev/da1p1
mkdir /mnt/WD1
mount_msdosfs -o large -o longnames /dev/da1p1 /mnt/WD1
Results in this:
mount_msdosfs: /dev/da1p1: Invalid argument

I haven't been able to sort out how to make an attempt on OpenBSD.

The Scientific Linux 7.4 system is surprisingly fussy about this drive.
Code:
[71608.334018] usb 2-5: new high-speed USB device number 11 using ehci-pci
[71608.452102] usb 2-5: New USB device found, idVendor=1058, idProduct=25a1
[71608.452106] usb 2-5: New USB device strings: Mfr=2, Product=3, SerialNumber=1
[71608.452109] usb 2-5: Product: Elements 25A1
[71608.452112] usb 2-5: Manufacturer: Western Digital
[71608.452114] usb 2-5: SerialNumber: 5758523145393734554A4544
[71608.452394] usb-storage 2-5:1.0: USB Mass Storage device detected
[71608.452763] scsi host15: usb-storage 2-5:1.0
[71609.454887] scsi 15:0:0:0: Direct-Access     WD       Elements 25A1    1014 PQ: 0 ANSI: 6
[71609.455441] sd 15:0:0:0: Attached scsi generic sg4 type 0
[71609.459878] sd 15:0:0:0: [sdd] Spinning up disk...
[71610.461013] ......ready
[71615.481605] sd 15:0:0:0: [sdd] Very big device. Trying to use READ CAPACITY(16).
[71615.481977] sd 15:0:0:0: [sdd] 7813969920 512-byte logical blocks: (4.00 TB/3.63 TiB)
[71615.481981] sd 15:0:0:0: [sdd] 4096-byte physical blocks
[71615.483103] sd 15:0:0:0: [sdd] Write Protect is off
[71615.483107] sd 15:0:0:0: [sdd] Mode Sense: 47 00 10 08
[71615.484296] sd 15:0:0:0: [sdd] No Caching mode page found
[71615.484300] sd 15:0:0:0: [sdd] Assuming drive cache: write through
[71615.485604] sd 15:0:0:0: [sdd] Very big device. Trying to use READ CAPACITY(16).
[71615.658135]  sdd: sdd1 sdd2 sdd3 sdd4
[71615.659231] sd 15:0:0:0: [sdd] Very big device. Trying to use READ CAPACITY(16).
[71615.661604] sd 15:0:0:0: [sdd] Attached SCSI disk
This machine doesn't have USB3 but I wouldn't expect that to be the source of these problems. The graphical program GParted reports gibberish for this drive and frequently crashes in a variety shocking ways.

Hmm.
Reply With Quote
Old 2nd March 2018
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,977
Default

On OpenBSD, if you are creating a GPT partition, all the FATs and NTFS appear as "FAT12" which I assume is equivalent to a Microsoft Basic Data Partition.

I have just tested a virtual 4TB (512-byte sector) drive on OpenBSD, using a 4TB sparse file. You can use dd(1) but I use vmctl(8) because it has that function built-in. I created the file with $ vmctl create /tmp/4t -s 4t, attached it to a vnd(4) virtual disk device using # vnconfig vnd0 /tmp/4t, created a GPT with # fdisk -gie vnd0 and in its editor I changed the default OpenBSD 4TB partition into a 1GB FAT32L partition (type x0C), which appeared as "FAT12" in the GPT listing. Disklabel showed a virtual partition "i" as "MSDOS" which I formatted as FAT32 with # newfs_msdos -F 32 vnd0i, and then mounted it successfully with # mount /dev/vnd0i /mnt.

You might test with small partitions, as they are quick to format.
Reply With Quote
Old 2nd March 2018
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,977
Default

Hmm, above, your report stated:
Code:
[71615.481977] sd 15:0:0:0: [sdd] 7813969920 512-byte logical blocks: (4.00 TB/3.63 TiB)
[71615.481981] sd 15:0:0:0: [sdd] 4096-byte physical blocks
The drive has 4096-byte sectors, but apparently is presenting 512-byte emulated sectors to the OS.
Reply With Quote
Old 2nd March 2018
hanzer's Avatar
hanzer hanzer is offline
Real Name: Adam Jensen
just passing through
 
Join Date: Oct 2013
Location: EST USA
Posts: 314
Default

Thanks, jggimi. There are some interesting hints, nifty gizmo's, and useful methods in your posts.

The story so far...

The drive is set up on FreeBSD like this:
Code:
gpart destroy -F da1
gpart create -s GPT da1
gpart add -t ms-basic-data -a 1M -s 930G -l "WD1" da1
gpart add -t ms-basic-data -a 1M -s 930G -l "WD2" da1
gpart add -t ms-basic-data -a 1M -s 930G -l "WD3" da1
gpart add -t ms-basic-data -a 1M -l "WD4" da1
newfs_msdos -F 32 -S 4096 -L WD1 /dev/da1p1
newfs_msdos -F 32 -S 4096 -L WD1 /dev/da1p2
newfs_msdos -F 32 -S 4096 -L WD1 /dev/da1p3
newfs_msdos -F 32 -S 4096 -L WD1 /dev/da1p4
Oddly, it can't be mounted from FreeBSD:
Code:
mkdir /mnt/WD1
mount_msdosfs -o large -o longnames /dev/da1p1 /mnt/WD1
->  mount_msdosfs: /dev/da1p1: Invalid argument
OpenBSD sees this setup as:
Code:
# fdisk sd1
Disk: sd1       Usable LBA: 40 to 7813969879 [7813969920 Sectors]
   #: type                                 [       start:         size ]
------------------------------------------------------------------------
   0: FAT12                                [        2048:   1950351360 ]
   1: FAT12                                [  1950353408:   1950351360 ]
   2: FAT12                                [  3900704768:   1950351360 ]
   3: FAT12                                [  5851056128:   1962911744 ]

# disklabel sd1                                                                           
# /dev/rsd1c:
type: SCSI
disk: SCSI disk
label: Elements 25A1   
duid: 0000000000000000
flags:
bytes/sector: 512
sectors/track: 255
tracks/cylinder: 511
sectors/cylinder: 130305
cylinders: 59966
total sectors: 7813969920
boundstart: 0
boundend: 7813969920
drivedata: 0 

16 partitions:
#                size           offset  fstype [fsize bsize   cpg]
  c:       7813969920                0  unused                    
  i:       1950351360             2048   MSDOS                    
  j:       1950351360       1950353408   MSDOS                    
  k:       1950351360       3900704768   MSDOS                    
  l:       1962911744       5851056128   MSDOS
But can't mount it either:
Code:
# mount_msdos /dev/sd1i /mnt/WD1 
mount_msdos: /dev/sd1i on /mnt/WD1: not an MSDOS filesystem
root@nomad:/root:21 # mount_msdos /dev/sd1j /mnt/WD1 
mount_msdos: /dev/sd1j on /mnt/WD1: not an MSDOS filesystem
root@nomad:/root:22 # mount_msdos /dev/sd1k /mnt/WD1 
mount_msdos: /dev/sd1k on /mnt/WD1: not an MSDOS filesystem
root@nomad:/root:23 # mount_msdos /dev/sd1l /mnt/WD1 
mount_msdos: /dev/sd1l on /mnt/WD1: not an MSDOS filesystem
Scientific Linux auto mounts all four partitions and seems to be able to read and write without problems so far.

Since bytes/sector are reported as 512 (nice catch, jggimi), I might start over with # newfs_msdos -F 32 -S 512 -L WD1 /dev/da1p1 and see if any behavior changes.
Reply With Quote
Old 2nd March 2018
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,977
Default

I'll guess that the BSDs fail to mount due to the sector size mismatch. To prove that hypothesis, you should be able to redefine one of these as a smaller partition for quick testing of a FAT32 partition with a 512-byte sector size.
Reply With Quote
Old 2nd March 2018
hanzer's Avatar
hanzer hanzer is offline
Real Name: Adam Jensen
just passing through
 
Join Date: Oct 2013
Location: EST USA
Posts: 314
Default

Quote:
Originally Posted by hanzer View Post
[snip]
Since bytes/sector are reported as 512 (nice catch, jggimi), I might start over with # newfs_msdos -F 32 -S 512 -L WD1 /dev/da1p1 and see if any behavior changes.
Quote:
Originally Posted by jggimi View Post
I'll guess that the BSDs fail to mount due to the sector size mismatch. To prove that hypothesis, you should be able to redefine one of these as a smaller partition for quick testing of a FAT32 partition with a 512-byte sector size.
Yep, that did. It was the sector size. Using 512 now and all is good, the drive is working nicely on all three platforms. Thanks for the help!

note: each of the four partitions is less than 2TB (~930GB) so that was never an issue except for newfs creation time.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Mount pen drive lucas34 OpenBSD General 12 9th December 2013 06:47 PM
sd/mmc drive phill OpenBSD General 8 22nd March 2011 03:21 PM
Creating USB flash drive drive image from FreeBSD disc1.iso FBSD Guides 1 10th February 2010 04:42 PM
which drive is what? knasbas OpenBSD Installation and Upgrading 2 20th July 2009 09:48 PM
How can I tell if my USB drive is bad? JMJ_coder General Hardware 4 3rd November 2008 04:12 AM


All times are GMT. The time now is 06:02 AM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick