DaemonForums  

Go Back   DaemonForums > Miscellaneous > General Hardware

General Hardware General hardware related questions.

Reply
 
Thread Tools Display Modes
  #1   (View Single Post)  
Old 26th January 2017
hulten hulten is offline
Port Guard
 
Join Date: Dec 2014
Posts: 34
Question Should my USB disk work on BSD?

I have bought a nice little disk to store my backups that require a lot of diskspace (smaller stuff I backup with tarsnap).

It is a Seagate 4 TB Backup Plus Portable, and I bought it here.

I have not yet unsealed the package, because I noticed the text "Use between Windows and Mac". This seemed unlikely to me. Just to be sure, I called Alternate to ask if I can repartion it, put any filesystem on it and just use it on OpenBSD or GNU/Linux. They told me that only Windows and Mac is supported, and they said it was very unlikely it would work on anything else.

I still find it unlikely, but just to be sure: do you have any experience with drives like this that actually give problems on the better operating systems?
Reply With Quote
  #2   (View Single Post)  
Old 26th January 2017
jggimi's Avatar
jggimi jggimi is online now
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,977
Default

This is a mass storage device using USB for its connection. It *must* follow standards, which are OS agnostic. The vendor may not have ever heard of anything but "Windows" and "Mac" but that should not preclude the product's adherence to standards and therefore its use.

See: https://en.wikipedia.org/wiki/USB_ma...e_device_class and https://en.wikipedia.org/wiki/USB_Attached_SCSI for information on the technical standards the disk should adhere to.

I don't know what the 24-month warranty includes, nor what can be implied under the regulations of your country. I don't know EU law, but under directive 1999/44/EC the customer's country's laws and regulations -- if EU -- apply to the transaction, as the seller was in the EU.

As for OpenBSD -- if the drive connects to it -- you can provision an MBR or a GPT. If the sector size is 512 bytes, an MBR will only address 2TB, though the disklabel could still address the entire drive. An MBR or GPT would be protective in nature, as those are recognized by other OSes.

Last edited by jggimi; 26th January 2017 at 06:55 PM. Reason: one typo, and one thinko, and some additional clarity
Reply With Quote
  #3   (View Single Post)  
Old 26th January 2017
hulten hulten is offline
Port Guard
 
Join Date: Dec 2014
Posts: 34
Default

Good to know that I should not expect any problems at the hardware side!

Concerning the partitioning, if I have an MBR and then create an OpenBSD partition in it, I understand that I can only assign 2 TB to it. But then I create a disklabel on it and I can assign the whole 4 TB?! I don't think that I understand it.
Also, GPT is new for me: I understand it is like MBR but it can do more (no limitations like 4 primary partitions, or 2 TB). Is that right?

Then the filesystems, what should I use if I want to read it under both OpenBSD and Linux? At least the first 2 TB? Something like this maybe:

* 1 GPT partition of 1.5 TB with a VFAT filesystem;
* 1 GPT disklabel partition of 2.5 TB with an FFS partition.

Then the first one can be fully used by any operating system.
The second could maybe only easily be used in BSDs.

The VFAT may fragment, and has neither journaling nor softdeps which may be suboptimal for data integrity. But if I take ext3, I only have journaling in Linux but not in OpenBSD where it is used as ext2.

Last edited by hulten; 26th January 2017 at 09:54 PM.
Reply With Quote
  #4   (View Single Post)  
Old 26th January 2017
jggimi's Avatar
jggimi jggimi is online now
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,977
Default

MBRs and the 2TB limit

The OpenBSD FAQ used to have a section on large drives, which has been removed when it went out-of-date (GPTs, 4K sector sizes) and was not being maintained. Here is the applicable procedure:
Quote:
Originally Posted by Old OpenBSD FAQ
To use a disk larger than 2TB, create an OpenBSD partition on the disk using fdisk, whatever size fdisk will let you. When you label the disk with disklabel(8), use the "b" option to set the OpenBSD boundaries (which defaulted to the size of the OpenBSD fdisk partition) to cover the entire disk. Now you can create your partitions as you wish.
These instructions are for a drive which is dedicated to OpenBSD, though with care, one could create foreign MBR partitions that reside below the 2 TB limit. But for your use-case, which is multi-system, I would recommend using GPT.

GPTs and their partitions are managed with fdisk(8), in similar fashion to how MBR partitions are managed. The program can also create a GPT if it does not have one. See the -g option.

GPT provisioning

For your example use-case, you refer to "VFAT", which I understand to indicate FAT with long file names. If correct, I would use fdisk() to create two GPT partitions, one type 0C for FAT, the other type A6 for OpenBSD.

FAT32 is the largest FAT filesystem. FAT32 filesystem size is limited to 2TB, which fits your example, but individual files are limited to 4GB size, which might not meet needs.

Long names ("VFAT") will be used by default on OpenBSD if any are on the drive when mounted. You can mount_msdos(8) an empty filesystem with -l to ensure long file names are used when files are first created.

FFS and FFS2

Your example showed a 2.5TB FFS filesystem. When formatted, this would actually be FFS2, as FFS is limited to 2TB. The newfs(8) program formats both, and switches automatically to FFS2 for filesystems larger than 2 TB.

Note that FFS2 cannot be used for booting. Should you want to use this drive as an OpenBSD boot device, consider a small FFS partition for this purpose. Booting GPT requires an EFI boot partition and hardware that boots EFI. The GPT initialization performed by fdisk() can optionally create an EFI boot partition.

Last edited by jggimi; 27th January 2017 at 01:04 AM. Reason: added mention of boot partition, then correct the wording. :(
Reply With Quote
  #5   (View Single Post)  
Old 27th January 2017
jggimi's Avatar
jggimi jggimi is online now
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,977
Default

I just tried this in a virtual machine. Here is a 4TB (virtual) drive. The disklabel program shows all sectors of the drive:
Code:
# disklabel sd1
# /dev/rsd1c:
type: SCSI
disk: SCSI disk
label: Block Device
duid: 0000000000000000
flags:
bytes/sector: 512
sectors/track: 63
tracks/cylinder: 255
sectors/cylinder: 16065
cylinders: 534698
total sectors: 8589934592
boundstart: 0
boundend: 8589934592
drivedata: 0

16 partitions:
#                size           offset  fstype [fsize bsize   cpg]
  c:       8589934592                0  unused
The fdisk program shows no MBR or GPT, so MBR is assumed. Note the number of sectors reported. This is the 2 TB limit :
Code:
# fdisk sd1
Disk: sd1       geometry: 267349/255/63 [4294961685 Sectors]
Offset: 0       Signature: 0x0
            Starting         Ending         LBA Info:
 #: id      C   H   S -      C   H   S [       start:        size ]
-------------------------------------------------------------------------------
 0: 00      0   0   0 -      0   0   0 [           0:           0 ] unused
 1: 00      0   0   0 -      0   0   0 [           0:           0 ] unused
 2: 00      0   0   0 -      0   0   0 [           0:           0 ] unused
 3: 00      0   0   0 -      0   0   0 [           0:           0 ] unused
Here, I create a GPT. These include a protective MBR just in case an older OS reads sector 0.
Code:
# fdisk -igy sd1
Writing MBR at offset 0.
Writing GPT.
As with -i and MBRs, the default is a single partition of the entire disk. As this is a GPT, I do not have the sector count limit:
Code:
# fdisk sd1
Disk: sd1       Usable LBA: 64 to 8589934528 [8589934592 Sectors]
   #: type                                 [       start:         size ]
------------------------------------------------------------------------
   3: OpenBSD                              [          64:   8589934465 ]
The first step to having two GPT partitions is to disable (erase) the full-disk OpenBSD partition, to make room for the FAT partition:
Code:
# fdisk -e sd1
Enter 'help' for information
fdisk: 1> e 3
   #: type                                 [       start:         size ]
------------------------------------------------------------------------
   3: OpenBSD                              [          64:   8589934465 ]
Partition id ('0' to disable) [01 - FF]: [A6] (? for help) 0
Partition 3 is disabled.
Then I create a 1.5 TB FAT partition:
Code:
fdisk:*1> e 0
   #: type                                 [       start:         size ]
------------------------------------------------------------------------
   0: unused                               [           0:            1 ]
Partition id ('0' to disable) [01 - FF]: [0] (? for help) 0C
Partition offset [64 - 8589934528]: [64]
Partition size [1 - 8589934465]: [8589934465] 1500g
Partition name: [] fat32
Apparently all of the various MS-DOS partition types used in MBR are all consolidated into a single type when in GPT which fdisk() shows as FAT12. I was unaware of this, because I don't have any FAT partitions on my GPT disks:
Code:
fdisk:*1> p
Disk: sd1       Usable LBA: 64 to 8589934528 [8589934592 Sectors]
GUID: 797fb8e9-dc04-4f2b-9eb5-0c9c100925ab
   #: type                                 [       start:         size ]
      guid                                 name
------------------------------------------------------------------------
   0: DOS FAT-12                           [          64:   3145728000 ]
      ae179056-a87a-47d6-86de-813f5b9ffcb4 fat32
Then I add a replacement, smaller OpenBSD partition. I set the start to the end of the FAT partition. Luckily, I can add 64 to 3145728000 without needing a calculator:
Code:
fdisk:*1> e 1
   #: type                                 [       start:         size ]
------------------------------------------------------------------------
   1: unused                               [           0:            1 ]
Partition id ('0' to disable) [01 - FF]: [0] (? for help) a6
Partition offset [64 - 8589934528]: [64] 3145728064
Partition size [1 - 5444206465]: [5444206465] *
Partition name: [] OpenBSD
I note how many GB have been allocated, see that it looks correct, write the revised GPT, and quit:
Code:
fdisk:*1> p g
Disk: sd1       Usable LBA: 64 to 8589934528 [4096 Gigabytes]
GUID: 797fb8e9-dc04-4f2b-9eb5-0c9c100925ab
   #: type                                 [       start:         size ]
      guid                                 name
------------------------------------------------------------------------
   0: DOS FAT-12                           [          64:         1500G]
      ae179056-a87a-47d6-86de-813f5b9ffcb4 fat32
   1: OpenBSD                              [  3145728064:         2596G]
      59b48cf6-3b3f-4ea7-a167-0e1f171aa944 OpenBSD
fdisk:*1> w
Writing MBR at offset 0.
Writing GPT.
fdisk: 1> q
#
There is no physical disklabel on the drive, yet. The OS allocates a virtual disklabel containing any recognized foreign filesystems, starting with partition "i":
Code:
# disklabel sd1                                                         
# /dev/rsd1c:
type: SCSI
disk: SCSI disk
label: Block Device    
duid: 0000000000000000
flags:
bytes/sector: 512
sectors/track: 63
tracks/cylinder: 255
sectors/cylinder: 16065
cylinders: 534698
total sectors: 8589934592
boundstart: 3145728064
boundend: 8589934529
drivedata: 0 

16 partitions:
#                size           offset  fstype [fsize bsize   cpg]
  c:       8589934592                0  unused                    
  i:       3145728000               64   MSDOS
And now I can add a "d" partition for use by large FFS2 filesystem. I write the disklabel, which will save it on disk, creating a physical disklabel:
Code:
# disklabel -E sd1
Label editor (enter '?' for help at any prompt)
> a d
offset: [3145728064] 
size: [5444206465] 
FS type: [4.2BSD] 
Rounding size to bsize (128 sectors): 5444206400
> w
> q
No label changes.
#
and now I can check my work:
Code:
# disklabel -p g sd1
# /dev/rsd1c:
type: SCSI
disk: SCSI disk
label: Block Device    
duid: d2f93a7a514914bd
flags:
bytes/sector: 512
sectors/track: 63
tracks/cylinder: 255
sectors/cylinder: 16065
cylinders: 534698
total sectors: 8589934592 # total bytes: 4096.0G
boundstart: 3145728064
boundend: 8589934529
drivedata: 0 

16 partitions:
#                size           offset  fstype [fsize bsize   cpg]
  c:          4096.0G                0  unused                    
  d:          2596.0G       3145728064  4.2BSD   8192 65536     1 
  i:          1500.0G               64   MSDOS                    
#
I did not format either partition. This was done in a Qemu virtual machine, emulating all hardware.

Last edited by jggimi; 27th January 2017 at 02:13 AM. Reason: a little clarity, lots of little typos, plus one glaring one
Reply With Quote
  #6   (View Single Post)  
Old 2nd February 2017
hulten hulten is offline
Port Guard
 
Join Date: Dec 2014
Posts: 34
Default

@jggimi, thank you for the clear explanation and howto. I just partioned the drive and now formatting the fat32 partition with
Code:
newfs_msdos sd0i
. It takes a long time, but I guess I just have to be patient. It doesn't matter if I do this over USB 2 or 3, since it does not actually send data over the connecting; the OS just needs to tell the drive how to format it, right?

usbdevs(8) tells me it is an EHCI device, so that's USB 2.
I have an extension USB cable between the computer and the official USB cord delivered with the disk; could that make it revert to even lower speeds?
Reply With Quote
  #7   (View Single Post)  
Old 2nd February 2017
jggimi's Avatar
jggimi jggimi is online now
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,977
Default

Quote:
Originally Posted by hulten View Post
It takes a long time, but I guess I just have to be patient.
Indeed, it takes a very long time.
Quote:
It doesn't matter if I do this over USB 2 or 3, since it does not actually send data over the connecting; the OS just needs to tell the drive how to format it, right?
No, that is an incorrect assumption.

A mass storage device presents either 512-byte or 4096-byte blocks to the systems. On a write operation, the OS sends either 512 bytes or 4096 bytes to be written, along with the sector number to write to.

Every filesystem contains a structure for storing and managing data. You know the ones presented to users, such as Files, and file names. Directories, or folders. You also know about access controls, and status flags such as "read only" or "hidden." But filesystems also contain data that is not presented directly to users, such as lists of free space, and structures to manage lists of used space.

All of those structures must be initiated as data stored on the drive before the file system can be established and used. It is this process which we call "formatting."
Quote:
..it is an EHCI device, so that's USB 2.
If you have USB 3 capability, I recommend it be used.
Reply With Quote
  #8   (View Single Post)  
Old 2nd February 2017
hulten hulten is offline
Port Guard
 
Join Date: Dec 2014
Posts: 34
Thumbs up

Formatting was much faster (less than a minute) for the ffs compared with fat32 (more than an hour). This is surprising because you mentioned that every filesystem needs this extra stuff like lists of free space which would take time to put on the disk.

The result
Code:
# disklabel sd0
# /dev/rsd0c:
...
16 partitions:
#                size           offset  fstype [fsize bsize  cpg]
  c:       7814037167                0  unused                   
  d:       4668308928       3145728064  4.2BSD   8192 65536    1 
  i:       3145728000               64   MSDOS                   
# mount
...
/dev/sd0d on /mnt/sd0d type ffs (local)
/dev/sd0i on /mnt/sd0i type msdos (local)
# df -h /mnt/*
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/sd0d      2.2T    8.0K    2.0T     0%    /mnt/sd0d
/dev/sd0i      1.5T    8.0K    1.5T     0%    /mnt/sd0i
Quite some overhead (0.2 TiB) for the FFS.

Of course the total does not add to 4 TiB but rather something like 4 TB, because hard disk vendors use base 10 (or, alternatively, they are bastards).

As mount(8) does not show it is FFS2 and the next thing I saw was that I have "2.0T" available on the FFS filesystem, I thought it was FFS, but I guess it is FFS2.
Reply With Quote
  #9   (View Single Post)  
Old 2nd February 2017
jggimi's Avatar
jggimi jggimi is online now
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,977
Default

FAT and FFS are very different filesystems. FFS needs a superblock, backup superblocks, cylinder group maps with inode structures. FAT requires much more, such as the File Allocation Tables and the Cluster Map.

FFS

While there are many differences between UFS/FFS implementations, this wiki page should help to understand the general structure of FFS/FFS2 on OpenBSD.

FAT

While this is not the only documentation availalble, here is a key wiki page that may help you to understand the structure of a FAT filesystem.

Last edited by jggimi; 2nd February 2017 at 06:44 PM. Reason: there is only one superblock. All the others are merely backups. :)
Reply With Quote
Old 2nd February 2017
jggimi's Avatar
jggimi jggimi is online now
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,977
Default

Quote:
Originally Posted by hulten View Post
Quite some overhead (0.2 TiB) for the FFS.
5% is the standard reserved free space for FFS/FFS2 on OpenBSD. If you are unaware, this can produce confusing reports from df(1), such as showing a drive at greater than 100% capacity.

There's a FAQ entry about it here.

Last edited by jggimi; 2nd February 2017 at 06:03 PM. Reason: clarity
Reply With Quote
Reply

Tags
hard disk, storage, support

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
disk mgt and zfs ananix FreeBSD General 1 26th December 2011 10:37 PM
Hard disk won't work after changing case sepuku OpenBSD General 68 8th September 2011 03:09 PM
Disk > 100% Weaseal FreeBSD General 3 11th February 2009 05:50 PM
Disk geometry for Seagate disk phreud FreeBSD Installation and Upgrading 5 8th November 2008 10:51 PM
Disk I/O Throughput m4rc OpenBSD General 5 10th July 2008 02:50 AM


All times are GMT. The time now is 11:17 AM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick