Formatting MSDOS drive

[gustaf]

I just bought a Seagate Backup Plus 2TB external USB hard drive for backing up my OpenBSD files. The factory-installed file system (not listed on the box) is (or was) NTFS -- read-only in OpenBSD. Obviously, I need to reformat this drive if I want to use it.

Disk formatting in OpenBSD is truly intimidating. I just want to plug in a drive and format it as MSDOS for maximum compatibility with other operating systems. Smaller-sized drives are easy. I plug them into my OS9 Mac, format them as DOS, and I'm good to go. That works until the drive gets too big for the Mac to handle. Hence, my current problem.

Long story short, I plugged this 2TB drive into my OS9 Mac and initialized it as "DOS 1,863GB". The end result was a 7GB capacity drive.

I plugged the drive into OpenBSD and ran:

$ disklabel -h sd1

# /dev/rsd1c:
type: SCSI
disk: SCSI disk
label: BUP Slim BK     
duid: 0000000000000000
flags:
bytes/sector: 512
sectors/track: 63
tracks/cylinder: 255
sectors/cylinder: 16065
cylinders: 243201
total sectors: 3907029167 # total bytes: 1863.0G
boundstart: 0
boundend: 3907029167
drivedata: 0 

16 partitions:
#                size           offset  fstype [fsize bsize   cpg]
  c:          1863.0G                0  unused                    
  i:             7.0G               32   MSDOS

That's mostly what I wanted -- an MSDOS drive with two partitions, c and i -- except that partition i should be 1863.0G.

newfs(8) seemed like the best option for increasing the size of this partition.

Starting with disklabel(8), I ran $ disklabel -e sd1 and changed the size of partition i to 3907029135 (3907029167 minus 32, to account for the offset), then checked the partition size and available free space:

$ df -h

Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/sd1i      7.0G    8.0K    7.0G     0%    /path/to/mount-point

I still needed to rebuild the file system on the disk.

I printed the file system parameters:

$ newfs -N -t msdos sd1i

/dev/rsd1i: 3903217360 sectors in 243951085 FAT32 clusters (8192 bytes/cluster)
bps=512 spc=16 res=32 nft=2 mid=0xf8 spt=63 hds=255 hid=32 bsec=3907029135 bspf=1905868 rdcl=2 infs=1 bkbs=2

and reconstructed the file system:

$ newfs -t msdos sd1i

I think it worked!

$ disklabel -h sd1

# /dev/rsd1c:
type: SCSI
disk: SCSI disk
label: BUP Slim BK     
duid: afb5ddb4eca1df71
flags:
bytes/sector: 512
sectors/track: 63
tracks/cylinder: 255
sectors/cylinder: 16065
cylinders: 243201
total sectors: 3907029167 # total bytes: 1.8T
boundstart: 0
boundend: 3907029167
drivedata: 0 

16 partitions:
#                size           offset  fstype [fsize bsize   cpg]
  c:             1.8T                0  unused                    
  i:             1.8T               32   MSDOS

$ df -h

Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/sd1i      1.8T    8.0K    1.8T     0%    /path/to/mount-point

$ fdisk sd1

Disk: sd1       geometry: 243201/255/63 [3907029167 Sectors]
Offset: 0       Signature: 0xAA55
            Starting         Ending         LBA Info:
 #: id      C   H   S -      C   H   S [       start:        size ]
-------------------------------------------------------------------------------
*0: 0C      0   0  33 -    915 244  33 [          32:    14714848 ] FAT32L      
 1: 00      0   0   0 -      0   0   0 [           0:           0 ] unused      
 2: 00      0   0   0 -      0   0   0 [           0:           0 ] unused      
 3: 00      0   0   0 -      0   0   0 [           0:           0 ] unused

Did I do this correctly and is there anything I should be aware of before writing a backup to this drive?

Also, does anyone know if there is a good reference on the nuts and bolts of disk formatting?

Thanks.

[ibara]

sysutils/ntfs-3g

[jggimi]

Two things to keep in mind:

• There there are three distinct partitioning systems in use on MBR/GPT architectures such as amd64. Either an MBR or a GPT are managed by fdisk(8), and then the OpenBSD disklabel is managed by disklabel(8).

• When a physical disk does not have an OpenBSD disklabel on it, the MBR/GPT partition table is scanned looking for recognized foreign filesystems, and OpenBSD assigns them virtually to disklabel partitions starting with partition "i". If that virtual table is written to disk, the disklabel will be used for partitioning information, without scanning the MBR or GPT table.

I'm going to focus on that second item. There are five possible states:

1. No partition table of any kind on a disk.
2. Only a disklabel on a disk.
3. Only an MBR/GPT on a disk.
4. Both MBR/GPT table and disklabel, where any foreign filesystem partition aligns with its disklabel partition.
5. Both MBR/GPT table and a disklabel, where there is mismatch between the MBR/GPT table and the disklabel.

State 1: When the leading blocks of a drive have been overwritten, the drive is only addressable by OpenBSD through its synthetic "c" disklabel partition, the partition that maps to the entire physical drive. It's easy to create State 1. You can wipe all types of partitioning, just by writing over the first sectors. For example, a # dd if=/dev/zero of=/dev/sd7c count=100 will put zeros on the first 100 sectors of a drive, overlaying partitioning schema and boot blocks.

State 2: If a drive only has disklabel partitions, but no MBR or GPT, it cannot be booted on an MBR/GPT architecture computer, and cannot be shared with a foreign OS. But it can readily be used for OpenBSD FFS/FFS2 filesystems and for swapping. A foreign filesystem can be added, but it cannot be shared with a foreign OS.

State 3: A disk drive with MBR or GPT partitioning can be used for foreign filesystems without ever having a disklabel physically written to it. Each time the drive is attached or the OS is booted, the MBR/GPT is read and recognized filesystems assigned starting with "i". The virtual disklabel remains in kernel memory, and is only written to disk if you write it with disklabel(8).

State 4: Transitioning from State 3 to State 4 requires only writing the virtual disklabel to disk. Once this is done, the MBR/GPT will not be read again to search for foreign filesystems. It will be read by disklabel(8) to look for an OpenBSD MBR/GPT partition in order to establish a sector range to add OpenBSD FFS/FFS2 filesystems and swap space. Disk drives in State 4 should only be needed if the drive stores "native" OpenBSD filesystems (FFS/FFS2) or swap space, along with any foreign filesystems.

State 5: It is easy to make this error, by starting from State 4 and then changing foreign filesystems with disklabel only. These changes are stored only in the disklabel, and are not reflected back on the MBR or GPT partition table. Because the MBR/GPT table is not scanned again on OpenBSD, this means the changed partition will only function as intended while running OpenBSD. An OS that scans the MBR/GPT table for filesystems may fail to mount the underlying filesystem if the change has an operational impact. You can prevent this by staying in State 3, with only a virtual disklabel, or keeping the disklabel and MBR/GPT changes in sync.

[ibara]

Oh, it looks like you did the formatting. In the future, you could just stick to using ntfs-3g and not bother reformatting.

Anyhow, in disklabel, you can use sizes postfixed with M or G. So you could have just used 1863G.

EDIT: Or just do what jggimi suggested, though I would still stick with using ntfs-3g, seeing as

# pkg_add ntfs-3g

is way shorter than doing anything else.

[gustaf]

Hi jggimi and ibara,

Thanks for your quick responses. I'll be back later (probably tomorrow) after I work out what all this means. It's a lot more than the simple yes or no answer that I was expecting.

I'm sure I'll get it eventually, but it might take some time.

[gustaf]

Thanks, jggimi, for your clear and focused explanation.

I think I'm beginning to understand the process of formatting a disk in OpenBSD.

First, some useful information I gleaned from reading the Wikipedia articles on Disk_partitioning and the Master_boot_record:

• Hard drives must be partitioned before they can be used. After they are partitioned, they are formatted with a specific file system or systems, one per partition.
  
• Each partition is treated as a separate logical disk on the physical disk.
  
• A partitioned hard drive needs a master boot record, even if the device is not bootable.
  
• The MBR is located at physical offset 0, in the first sector of the drive, before the first partition. It contains the partition table (in addition to the boot loader, if one exists). The partition table contains information about the location, sizes, and organization of the partitions and the file systems they contain.
  
• There are four partition table entries in the MBR partition table.
  
• Without logical block addressing, storage space is limited to 8GB on MBR-partitioned disks with 512-byte sectors. (This probably explains why the Mac truncated this supposedly 1,863GB drive to 7GB. It looks like my trusty old Mac is reaching the limits of its ability to help me. -- sigh).

Where I think I went wrong:

When a physical disk does not have an OpenBSD disklabel on it, the MBR/GPT partition table is scanned looking for recognized foreign filesystems, and OpenBSD assigns them virtually to disklabel partitions starting with partition "i"...

That part I figured out from reading disklabel(8) and one of jggimi's earlier posts.

... If that virtual table is written to disk, the disklabel will be used for partitioning information, without scanning the MBR or GPT table.

This I did not know, but it makes sense in a BSD context.

I mistakenly thought that $ newfs -t msdos was what I needed to reformat the drive. This command uses the defaults in the disklabel to build the file system, so I changed the disklabel from 7.0GB to 1.8TB, then wrote the edited disklabel to the drive.

The output of $ df -h (7.0GB) led me to believe that the only renmaining step was to change the size of the file system by rebuilding it with $ newfs -t msdos.

I didn't realize that I had to account for the MBR as well.

$ newfs -N -t msdos sd1i shows that the formatted drive will have a 1.8 TB, FAT32 file system, based on the values in the disklabel: (512 bytes/sector * 3,903,217,360 sectors = 1,998,447,288,320 bytes / 1024^4 bytes/terabyte = 1.8TB).

I found the file system parameters reported by $ newfs -N -t msdos in the newfs_msdos(8) man page, NOTES section.

After running $ newfs -t msdos sd1, disklabel(8) reports that partition "i" is 1.8TB. That's what I want, but that's not necessarily what I'm going to get because fdisk(8) reports that the MBR size is 7.0GB: (14,714,848 sectors * 512 bytes/sector / 1024^3 bytes/gigabyte).

disklabel(8) and the MBR are now out of sync. This disk is currently in State 5.

Transitioning from State 3 [only an MBR/GPT on a disk] to State 4 [both MBR/GPT and disklabel, where any foreign filesystem partition aligns with its disklabel partition] requires only writing the virtual disklabel to disk.

State 5: [If foreign file systems are changed with disklabel only, the] changes are stored only in the disklabel, and are not reflected back on the MBR or GPT partition table...

If I had not edited the disklabel, the two partitioning systems would still be in sync (State 4), but I'd have a disk with a huge amount of wasted space. By editing the disklabel before writing it to the drive, I went directly from State 3 to State 5.

...Because the MBR/GPT table is not scanned again on OpenBSD, this means the changed partition will only function as intended while running OpenBSD. An OS that scans the MBR/GPT table for filesystems may fail to mount the underlying filesystem if the change has an operational impact...

This improperly-formatted disk crashed my Mac.

newfs(8) is the wrong tool because it requires a disklabel that is written to the disk.


How do I fix this disk?

I want the drive to be in State 3. This is the state of the flash drives I use to pass files back and forth between different operating systems. This system has worked well for many years. The challenge is how to scale this system for backups.

I could try to fix the MBR with fdisk(8), but that looks too risky. I don't have enough knowledge or experience to be comfortable messing around in the guts of a filesystem.

Sometimes it's easier (and safer) to just start over.

1. I plugged in the disk and ran $ sysctl hw.disknames to make sure I was working with the correct disk.

2. $ hexdump -C -n 51200 /dev/sd1c to see the data on the first 100 sectors of the drive:

00000000  33 c0 8e d0 bc 00 7c fb  50 07 50 1f fc be 1b 7c  |3.....|.P.P....||
00000010  bf 1b 06 50 57 b9 e5 01  f3 a4 cb be be 07 b1 04  |...PW...........|
00000020  38 2c 7c 09 75 15 83 c6  10 e2 f5 cd 18 8b 14 8b  |8,|.u...........|
00000030  ee 83 c6 10 49 74 16 38  2c 74 f6 be 10 07 4e ac  |....It.8,t....N.|
00000040  3c 00 74 fa bb 07 00 b4  0e cd 10 eb f2 89 46 25  |<.t...........F%|
00000050  96 8a 46 04 b4 06 3c 0e  74 11 b4 0b 3c 0c 74 05  |..F...<.t...<.t.|
00000060  3a c4 75 2b 40 c6 46 25  06 75 24 bb aa 55 50 b4  |:.u+@.F%.u$..UP.|
00000070  41 cd 13 58 72 16 81 fb  55 aa 75 10 f6 c1 01 74  |A..Xr...U.u....t|
00000080  0b 8a e0 88 56 24 c7 06  a1 06 eb 1e 88 66 04 bf  |....V$.......f..|
00000090  0a 00 b8 01 02 8b dc 33  c9 83 ff 05 7f 03 8b 4e  |.......3.......N|
000000a0  25 03 4e 02 cd 13 72 29  be 46 07 81 3e fe 7d 55  |%.N...r).F..>.}U|
000000b0  aa 74 5a 83 ef 05 7f da  85 f6 75 83 be 27 07 eb  |.tZ.......u..'..|
000000c0  8a 98 91 52 99 03 46 08  13 56 0a e8 12 00 5a eb  |...R..F..V....Z.|
000000d0  d5 4f 74 e4 33 c0 cd 13  eb b8 00 00 00 00 00 00  |.Ot.3...........|
000000e0  56 33 f6 56 56 52 50 06  53 51 be 10 00 56 8b f4  |V3.VVRP.SQ...V..|
000000f0  50 52 b8 00 42 8a 56 24  cd 13 5a 58 8d 64 10 72  |PR..B.V$..ZX.d.r|
00000100  0a 40 75 01 42 80 c7 02  e2 f7 f8 5e c3 eb 74 49  |.@u.B......^..tI|
00000110  6e 76 61 6c 69 64 20 70  61 72 74 69 74 69 6f 6e  |nvalid partition|
00000120  20 74 61 62 6c 65 00 45  72 72 6f 72 20 6c 6f 61  | table.Error loa|
00000130  64 69 6e 67 20 6f 70 65  72 61 74 69 6e 67 20 73  |ding operating s|
00000140  79 73 74 65 6d 00 4d 69  73 73 69 6e 67 20 6f 70  |ystem.Missing op|
00000150  65 72 61 74 69 6e 67 20  73 79 73 74 65 6d 00 00  |erating system..|
00000160  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000180  00 00 00 8b fc 1e 57 8b  f5 cb 00 00 00 00 00 00  |......W.........|
00000190  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000001b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 80 01  |................|
000001c0  01 00 0c 3f 20 10 20 00  00 00 e0 87 e0 00 00 00  |...? . .........|
000001d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa  |..............U.|
00000200  57 45 56 82 04 00 00 00  53 43 53 49 20 64 69 73  |WEV.....SCSI dis|
00000210  6b 00 00 00 00 00 00 00  42 55 50 20 53 6c 69 6d  |k.......BUP Slim|
00000220  20 42 4b 20 20 20 20 20  00 02 00 00 3f 00 00 00  | BK     ....?...|
00000230  ff 00 00 00 01 b6 03 00  c1 3e 00 00 af 88 e0 e8  |.........>......|
00000240  af b5 dd b4 ec a1 df 71  00 00 00 00 00 00 00 00  |.......q........|
00000250  00 00 00 00 af 88 e0 e8  00 00 00 00 00 00 00 00  |................|
00000260  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000270  00 00 01 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000280  00 00 00 00 57 45 56 82  3e 50 10 00 00 20 00 00  |....WEV.>P... ..|
00000290  00 20 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |. ..............|
000002a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000002b0  00 00 00 00 af 88 e0 e8  00 00 00 00 00 00 00 00  |................|
000002c0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000310  00 00 00 00 8f 88 e0 e8  20 00 00 00 00 00 00 00  |........ .......|
00000320  08 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000330  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00004000  eb 58 90 42 53 44 20 20  34 2e 34 00 02 10 20 00  |.X.BSD  4.4... .|
00004010  02 00 00 00 00 f8 00 00  3f 00 ff 00 20 00 00 00  |........?... ...|
00004020  8f 88 e0 e8 cc 14 1d 00  00 00 00 00 02 00 00 00  |................|
00004030  01 00 02 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00004040  00 01 29 ec 1c a1 7f 4e  4f 20 4e 41 4d 45 20 20  |..)....NO NAME  |
00004050  20 20 46 41 54 33 32 20  20 20 fa 31 c0 8e d0 bc  |  FAT32   .1....|
00004060  00 7c fb 8e d8 e8 00 00  5e 83 c6 19 bb 07 00 fc  |.|......^.......|
00004070  ac 84 c0 74 06 b4 0e cd  10 eb f5 30 e4 cd 16 cd  |...t.......0....|
00004080  19 0d 0a 4e 6f 6e 2d 73  79 73 74 65 6d 20 64 69  |...Non-system di|
00004090  73 6b 0d 0a 50 72 65 73  73 20 61 6e 79 20 6b 65  |sk..Press any ke|
000040a0  79 20 74 6f 20 72 65 62  6f 6f 74 0d 0a 00 00 00  |y to reboot.....|
000040b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000041f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa  |..............U.|
00004200  52 52 61 41 00 00 00 00  00 00 00 00 00 00 00 00  |RRaA............|
00004210  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000043e0  00 00 00 00 72 72 41 61  ec 65 8a 0e 20 3b cf 08  |....rrAa.e.. ;..|
000043f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa  |..............U.|
00004400  eb 58 90 42 53 44 20 20  34 2e 34 00 02 10 20 00  |.X.BSD  4.4... .|
00004410  02 00 00 00 00 f8 00 00  3f 00 ff 00 20 00 00 00  |........?... ...|
00004420  8f 88 e0 e8 cc 14 1d 00  00 00 00 00 02 00 00 00  |................|
00004430  01 00 02 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00004440  00 00 29 ec 1c a1 7f 4e  4f 20 4e 41 4d 45 20 20  |..)....NO NAME  |
00004450  20 20 46 41 54 33 32 20  20 20 fa 31 c0 8e d0 bc  |  FAT32   .1....|
00004460  00 7c fb 8e d8 e8 00 00  5e 83 c6 19 bb 07 00 fc  |.|......^.......|
00004470  ac 84 c0 74 06 b4 0e cd  10 eb f5 30 e4 cd 16 cd  |...t.......0....|
00004480  19 0d 0a 4e 6f 6e 2d 73  79 73 74 65 6d 20 64 69  |...Non-system di|
00004490  73 6b 0d 0a 50 72 65 73  73 20 61 6e 79 20 6b 65  |sk..Press any ke|
000044a0  79 20 74 6f 20 72 65 62  6f 6f 74 0d 0a 00 00 00  |y to reboot.....|
000044b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000045f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa  |..............U.|
00004600  52 52 61 41 00 00 00 00  00 00 00 00 00 00 00 00  |RRaA............|
00004610  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000047e0  00 00 00 00 72 72 41 61  ff ff ff ff ff ff ff ff  |....rrAa........|
000047f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa  |..............U.|
00004800  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00008000  f8 ff ff 0f ff ff ff 0f  ff ff ff 0f 00 00 00 00  |................|
00008010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
0000c800

Examining all this data to learn its meaning would be an interesting exercise, but I'll leave that for later.

3. $ doas dd if=/dev/zero of=/dev/sd1c count=100 to zero out the first 100 sectors of the drive.

4. $ hexdump -C -n 51200 /dev/sd1c to see the results:

00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
0000c800

5. Tests:

$ disklabel -pg sd1

# /dev/rsd1c:
type: SCSI
disk: SCSI disk
label: BUP Slim BK     
duid: 0000000000000000
.............................................
total sectors: 3907029167 # total bytes: 1863.0G
boundstart: 0
boundend: 3907029167
drivedata: 0 

16 partitions:
#                size           offset  fstype [fsize bsize   cpg]
  c:          1863.0G                0  unused

$ fdisk sd1

Disk: sd1       geometry: 243201/255/63 [3907029167 Sectors]
Offset: 0       Signature: 0x0
            Starting         Ending         LBA Info:
 #: id      C   H   S -      C   H   S [       start:        size ]
-------------------------------------------------------------------------------
 0: 00      0   0   0 -      0   0   0 [           0:           0 ] unused      
 1: 00      0   0   0 -      0   0   0 [           0:           0 ] unused      
 2: 00      0   0   0 -      0   0   0 [           0:           0 ] unused      
 3: 00      0   0   0 -      0   0   0 [           0:           0 ] unused

6. I plugged the disk into my OS 9 Mac. The Mac reported that the disk is unreadable and offered to initialize it as a DOS disk with a size of 1,863GB.

So far, so good. I've zeroed the data on the drive and have a safe way to start over if my new attempts at formatting fail.


How do I properly format a FAT32L drive on OpenBSD?

I tried newfs_msdos(8):

$ newfs_msdos -N sd1c

/dev/rsd1c: 3903217392 sectors in 243951087 FAT32 clusters (8192 bytes/cluster)
bps=512 spc=16 res=32 nft=2 mid=0xf0 spt=63 hds=255 hid=0 bsec=3907029167 bspf=1905868 rdcl=2 infs=1 bkbs=2

$ doas newfs_msdos sd1c

That didn't work. There is a virtual disklabel, but no partition table:

$ disklabel -pg sd1
    
# /dev/rsd1c:
type: SCSI
disk: SCSI disk
label: BUP Slim BK     
duid: 0000000000000000
flags:
bytes/sector: 512
sectors/track: 63
tracks/cylinder: 255
sectors/cylinder: 16065
cylinders: 243201
total sectors: 3907029167 # total bytes: 1863.0G
boundstart: 0
boundend: 3907029167
drivedata: 0 

16 partitions:
#                size           offset  fstype [fsize bsize   cpg]
  c:          1863.0G                0  unused                    
  i:          1863.0G                0   MSDOS

$ fdisk sd1

Disk: sd1       geometry: 243201/255/63 [3907029167 Sectors]
Offset: 0       Signature: 0xAA55
            Starting         Ending         LBA Info:
 #: id      C   H   S -      C   H   S [       start:        size ]
-------------------------------------------------------------------------------
 0: 00      0   0   0 -      0   0   0 [           0:           0 ] unused      
 1: 00      0   0   0 -      0   0   0 [           0:           0 ] unused      
 2: 00      0   0   0 -      0   0   0 [           0:           0 ] unused      
 3: 00      0   0   0 -      0   0   0 [           0:           0 ] unused

There is a brand-new port for working with MSDOS disks:

sysutils/dosfstools (check and label FAT file systems)

Description:
dosfstools consists of the programs mkfs.fat, fsck.fat and fatlabel to create, check and label file systems of the FAT family.

These tools also require a previously-constructed, valid partition table.

It looks like I need to manually partition the disk first, but I can't figure out fdisk(8). It seems like this is the hardest step, although somewhat easier if the disk will not be bootable.

From the fdisk(8) man page:

DESCRIPTION:
Caution is advised when editing these tables, since some platforms use tricks to boot and may rely on specific configurations created at install time.

CAVEATS:
Hand crafted disk layouts are highly error prone. It is common practice, though by no means required, that MBR partitions start on a cylinder boundary (generally head 0, sector 1, but head 1, sector 1 for track 0), and that MBR partitions also end at cylinder boundaries.

It looks like there are two ways to use fdisk(8), by passing options on the command line or with the interactive editor.

How do I determine the values for the offset, ID, C/H/S, LBA, and size. Are there sane defaults and algorithms? Are there other things I need to know about to construct a valid, cross-platform FAT32L partition table for a non-bootable disk? Where can I find the documentation?


Plea for a really good tutorial

After searching the Web (and searching some more...) I have concluded that we need a step-by-step how-to on this topic.

• I found my notes from over a year ago. There is a lot of information on the Web about partitioning and formatting MSDOS disks on OpenBSD. Most of it is confusing. Some of it is just plain wrong. One poster admitted to guessing.
  
• There are no ready-made tools for OpenBSD, and the tools that do exist are cryptic and baffling. Of all the things there are to love about OpenBSD, formatting disks is not among them.
  
• I am at least the third poster since 2009 to have received jggimi's MBR/GPT vs. disklabel explanation:
  
  1. https://web.archive.org/web/20090710...ry/004236.html (January 2009)
  2. https://daemonforums.org/showthread.php?p=39924 (August 2011)
  3. https://daemonforums.org/showthread.php?p=62205 (August 2017)

I would like to help with writing a how-to guide on formatting MSDOS disks on OpenBSD, but I don't have the technical expertise to do so on my own. I would be willing to test drive a guide and provide feedback.

Many thanks to anyone who can provide assistance.

[jggimi]

Assumed:

• You are starting with a zeroed 2TB drive.
• You want to have a 2TB FAT32 partition on this disk drive, for sharing files between OpenBSD and your Mac.

FAT32 partitions have two key size limitations, which you should be aware of:

1. The total capacity of a FAT32 filesystem is 2TB on a drive with 512-byte sectors, or 16TB on a drive with 4096-byte sectors.
2. Individual files are limited to one byte under 4GB in size, regardless of the size of the filesystem or the drive's sector size.

If the individual file size limit is not a factor, you may proceed with this relatively simple solution: Use OpenBSD's fdisk(8) to create an MBR partition table containing a single 2TB FAT32 partition, and only ever use a virtual disklabel partition to adddress it.

The fdisk(8) and newfs_msdos(8) man pages will be helpful.

You will begin by using fdisk(8) to create an MBR with a single OpenBSD partition, then edit this MBR partition type so that it is defined as FAT32. You will complete the process by formatting the partition as FAT32 using newfs_msdos(8).

1. Start the fdisk(8) program with the -e option to start its interactive editor. If the 2TB drive is attached as sd1 as noted above, the command is:
   
   # fdisk -e sd1
2. Instruct fdisk to create a new MBR, with a partition table containing a single OpenBSD MBR partition in the last partition table entry extending the length of the entire drive excepting a leading set of sectors for boot blocks:
   
   fdisk: 1> reinit mbr
   
   You will see the output of the newly created partition table. The last entry will be an OpenBSD partition of type A6.
3. Modify the OpenBSD partition (number 3, as the first empty partition is number 0) with:
   
   fdisk:*1> e 3
4. You will be prompted for a partition type. Select type 0C, which is the MBR partition type used for FAT32 partitions with long file names:
   
   Partition id ('0' to disable) [01 - FF : [A6] (? for help) c
5. You will be asked if you wish to edit in CHS mode. Press ENTER to take the default response of "n" for "No."
6. You will be asked if you want to revise the partition's starting offset. Press ENTER to leave this unchanged.
7. You will be asked if you want to revise the partition's size. Press ENTER to leave this unchanged.
8. Swap this newly revised partition 3 with partition number 0, so that the MBR partition is in the first position, rather than the last. Some operating systems may only recognize the MBR partition if it is in the first postion.
   
   fdisk:*1> swap 0 3
9. Confirm your newly edited partition is in the first position:
   
   fdisk:*1> p
10. Write the MBR and the partition table to the disk:
    
    fdisk:*1> w
11. Quit the fdisk(8) program:
    
    fdisk: 1> q
12. OpenBSD will scan the MBR partition table and assign virtual disklabel partition "i" to this newly created partition. You may now format it with newfs_msdos(8) as FAT32:
    
    # newfs_msdos -F 32 sd1i

[gustaf]

Assumed:

• You are starting with a zeroed 2TB drive.
• You want to have a 2TB FAT32 partition on this disk drive, for sharing files between OpenBSD and your Mac.

That assumption is mostly correct. This drive will be for my OpenBSD backups. My OS 9 Mac isn't up to the task of formatting or reading a 2TB MSDOS drive. I'm looking for maximum cross-platform compatibility. If something happens to my OpenBSD machine or I decide to switch operating systems (highly unlikely, but possible), I want to be able to access my files. That means this drive must be at least readable by Linux or Windows. I'm assuming that if I partition and format this drive according to your instructions, it will also be fully readable by these other operating systems. Is that assumption correct?

FAT32 partitions have two key size limitations...

The size limits are not a factor. The total capacity of the drive is 2TB, and 4GB is an incomprehensibly huge file (as least by my standards).

...create an MBR partition table containing a single 2TB FAT32 partition, and only ever use a virtual disklabel partition to adddress it.

That's exactly what I want.

I have zeroed the drive, reviewed the man pages, and carefully read your instructions.

I was missing steps 3-11. I didn't realize that creating an MSDOS partition is actually a two-stage process, starting with an OpenBSD partition.

I'm taking a deep breath and a break before I begin partitioning. I'll let you know how it goes.

[jggimi]

Originally Posted by gustaf

My OS 9 Mac isn't up to the task of formatting or reading a 2TB MSDOS drive.

I cannot help you there, as I'm unfamiliar with what FAT filesystems OS 9 was capable of using, and what size limitations it may have had.

I'm looking for maximum cross-platform compatibility. If something happens to my OpenBSD machine or I decide to switch operating systems (highly unlikely, but possible), I want to be able to access my files. That means this drive must be at least readable by Linux or Windows.

Then FAT may be your best choice, even though it is the lowest common denominator. Note that along with file size limitations, FAT does not have file access controls (such as the Unix-like user/group/world read/write/execute).

I'm assuming that if I partition and format this drive according to your instructions, it will also be fully readable by these other operating systems. Is that assumption correct?

By Linux and other Unix-like operating systems, and by Windows, yes. But if you want to retain file access controls, you'll need to use an archival method that retains them, such as tar(1) or pax(1).

...I didn't realize that creating an MSDOS partition is actually a two-stage process, starting with an OpenBSD partition....

That isn't quite correct, so perhaps I wasn't clear. I will state both what an MBR is:

• The MBR (Master Boot Record) can be used for booting and/or for partitioning. If used for booting, a boot program is read from the MBR by the hardware, loaded into memory, and executed.
• The MBR partition table, when used, consists of from 1 to 4 primary partitions.
• The MBR partitioning structure permits more than 4 partitions via a schema of chained partition tables, using extended and logical partitions.
• OpenBSD uses the MBR for 1) booting, 2) defining the range of sectors where OpenBSD's own partitions defined in a disklabel are allowed to reside, and 3) locating foreign file systems when the drive has no disklabel installed.

And, what I proposed:

• The MBR partition table will store a single entry, defined as a FAT filesystem with long file names.
• There will be no OpenBSD disklabel.
• Each time the drive is attached (at boot, or when plugged in) the kernel will inspect the drive. It will find no disklabel, then read the MBR partition table to discover the single foreign file system in the MBR partition table.

The only reason I am suggesting using fdisk(8)'s "reinit" is because it simplifies MBR partition table creation for you. And, as the "reinit" assumes this reinitialized MBR table will not have any foreign file systems, it assigns the entire drive (less blocks at the beginning) to an MBR partition table entry reserved for OpenBSD. But that is very temporary. A moment later, you will edit this MBR partition table entry and revise it to be for a FAT filesystem.

Again, the only purpose is simplification of creating the MBR and its partition table.

[gustaf]

Your clear and easy-to-follow instructions are exactly what I have been looking for. I think this disk is finally formatted properly:

$ disklabel -h sd1

# /dev/rsd1c:
type: SCSI
disk: SCSI disk
label: BUP Slim BK     
duid: 0000000000000000
flags:
bytes/sector: 512
sectors/track: 63
tracks/cylinder: 255
sectors/cylinder: 16065
cylinders: 243201
total sectors: 3907029167 # total bytes: 1.8T
boundstart: 0
boundend: 3907029167
drivedata: 0 

16 partitions:
#                size           offset  fstype [fsize bsize   cpg]
  c:             1.8T                0  unused                    
  i:             1.8T               64   MSDOS

$ fdisk sd1
Disk: sd1       geometry: 243201/255/63 [3907029167 Sectors]
Offset: 0       Signature: 0xAA55
            Starting         Ending         LBA Info:
 #: id      C   H   S -      C   H   S [       start:        size ]
-------------------------------------------------------------------------------
*0: 0C      0   1   2 - 243200 254  63 [          64:  3907024001 ] FAT32L      
 1: 00      0   0   0 -      0   0   0 [           0:           0 ] unused      
 2: 00      0   0   0 -      0   0   0 [           0:           0 ] unused      
 3: 00      0   0   0 -      0   0   0 [           0:           0 ] unused

I have a few final questions:

Will this disk be readable by any machine running Linux or Windows? This will be a backup drive. If something happens to my OpenBSD machine, I want to be able to read and possibly restore my files to another location.

I noticed that the offset of partition "i" and LBA start values have changed from 32 to 64. Does that make a difference?

I'm also wondering how you learned the process for formatting disks. I will probably have to format additional drives on OpenBSD at some point in the future, and I'd really like to be confident that I have the knowledge to do so correctly.

[jggimi]

Originally Posted by gustaf

Will this disk be readable by any machine running Linux or Windows?

I recommend testing it with your Linux and Windows OSes.

I noticed that the offset of partition "i" and LBA start values have changed from 32 to 64. Does that make a difference?

No. Sectors are held in reserve for extended MBR partition tables, boot programs, and similar purposes. The reserved sector count may vary by disk device and the interpretation of the disk partitioning program -- in this case, fdisk(8).

I'm also wondering how you learned the process for formatting disks.

I don't have a simple answer, other than "osmosis" or "experience" but those are trite, and not very helpful. I've been using OpenBSD for about 15 years, and managing hard drives for about 40.

[gustaf]

I tested the disk on Linux; it appears to work correctly. The reported size is 1.8TB. I can see, read, and copy my files.

I'm also wondering how you learned the process for formatting disks.

I don't have a simple answer, other than "osmosis" or "experience" but those are trite, and not very helpful. I've been using OpenBSD for about 15 years, and managing hard drives for about 40.

That's a pretty helpful answer, actually. It tells me that disk formats are way more complicated than I thought, and that the safest way to format them for use with other OSes is to choose MSDOS and follow the guidance in this thread.

Many thanks for generously sharing your knowledge. I would have never figured this out.