Thanks, jggimi, for your clear and focused explanation.
I think I'm beginning to understand the process of formatting a disk in OpenBSD.
First, some useful information I gleaned from reading the Wikipedia articles on
Disk_partitioning and the
Master_boot_record:
- Hard drives must be partitioned before they can be used. After they are partitioned, they are formatted with a specific file system or systems, one per partition.
- Each partition is treated as a separate logical disk on the physical disk.
- A partitioned hard drive needs a master boot record, even if the device is not bootable.
- The MBR is located at physical offset 0, in the first sector of the drive, before the first partition. It contains the partition table (in addition to the boot loader, if one exists). The partition table contains information about the location, sizes, and organization of the partitions and the file systems they contain.
- There are four partition table entries in the MBR partition table.
- Without logical block addressing, storage space is limited to 8GB on MBR-partitioned disks with 512-byte sectors. (This probably explains why the Mac truncated this supposedly 1,863GB drive to 7GB. It looks like my trusty old Mac is reaching the limits of its ability to help me. -- sigh).
Where I think I went wrong:
Quote:
|
When a physical disk does not have an OpenBSD disklabel on it, the MBR/GPT partition table is scanned looking for recognized foreign filesystems, and OpenBSD assigns them virtually to disklabel partitions starting with partition "i"...
|
That part I figured out from reading
disklabel(8) and
one of jggimi's earlier posts.
Quote:
|
... If that virtual table is written to disk, the disklabel will be used for partitioning information, without scanning the MBR or GPT table.
|
This I did not know, but it makes sense in a BSD context.
I mistakenly thought that
$ newfs -t msdos was what I needed to reformat the drive. This command uses the defaults in the disklabel to build the file system, so I changed the disklabel from 7.0GB to 1.8TB, then wrote the edited disklabel to the drive.
The output of
$ df -h (7.0GB) led me to believe that the only renmaining step was to change the size of the file system by rebuilding it with
$ newfs -t msdos.
I didn't realize that I had to account for the MBR as well.
$ newfs -N -t msdos sd1i shows that the formatted drive will have a 1.8 TB, FAT32 file system, based on the values in the disklabel: (512 bytes/sector * 3,903,217,360 sectors = 1,998,447,288,320 bytes / 1024^4 bytes/terabyte = 1.8TB).
I found the file system parameters reported by
$ newfs -N -t msdos in the
newfs_msdos(8) man page, NOTES section.
After running
$ newfs -t msdos sd1,
disklabel(8) reports that partition "i" is 1.8TB. That's what I want, but that's not necessarily what I'm going to get because
fdisk(8) reports that the MBR size is 7.0GB: (14,714,848 sectors * 512 bytes/sector / 1024^3 bytes/gigabyte).
disklabel(8) and the MBR are now out of sync. This disk is currently in State 5.
Quote:
|
Transitioning from State 3 [only an MBR/GPT on a disk] to State 4 [both MBR/GPT and disklabel, where any foreign filesystem partition aligns with its disklabel partition] requires only writing the virtual disklabel to disk.
|
Quote:
|
State 5: [If foreign file systems are changed with disklabel only, the] changes are stored only in the disklabel, and are not reflected back on the MBR or GPT partition table...
|
If I had not edited the disklabel, the two partitioning systems would still be in sync (State 4), but I'd have a disk with a huge amount of wasted space. By editing the disklabel
before writing it to the drive, I went directly from State 3 to State 5.
Quote:
|
...Because the MBR/GPT table is not scanned again on OpenBSD, this means the changed partition will only function as intended while running OpenBSD. An OS that scans the MBR/GPT table for filesystems may fail to mount the underlying filesystem if the change has an operational impact...
|
This improperly-formatted disk crashed my Mac.
newfs(8) is the wrong tool because it requires a disklabel that is written to the disk.
How do I fix this disk?
I want the drive to be in State 3. This is the state of the flash drives I use to pass files back and forth between different operating systems. This system has worked well for many years. The challenge is how to scale this system for backups.
I could try to fix the MBR with fdisk(8), but that looks too risky. I don't have enough knowledge or experience to be comfortable messing around in the guts of a filesystem.
Sometimes it's easier (and safer) to just start over.
1. I plugged in the disk and ran
$ sysctl hw.disknames to make sure I was working with the correct disk.
2.
$ hexdump -C -n 51200 /dev/sd1c to see the data on the first 100 sectors of the drive:
Code:
00000000 33 c0 8e d0 bc 00 7c fb 50 07 50 1f fc be 1b 7c |3.....|.P.P....||
00000010 bf 1b 06 50 57 b9 e5 01 f3 a4 cb be be 07 b1 04 |...PW...........|
00000020 38 2c 7c 09 75 15 83 c6 10 e2 f5 cd 18 8b 14 8b |8,|.u...........|
00000030 ee 83 c6 10 49 74 16 38 2c 74 f6 be 10 07 4e ac |....It.8,t....N.|
00000040 3c 00 74 fa bb 07 00 b4 0e cd 10 eb f2 89 46 25 |<.t...........F%|
00000050 96 8a 46 04 b4 06 3c 0e 74 11 b4 0b 3c 0c 74 05 |..F...<.t...<.t.|
00000060 3a c4 75 2b 40 c6 46 25 06 75 24 bb aa 55 50 b4 |:.u+@.F%.u$..UP.|
00000070 41 cd 13 58 72 16 81 fb 55 aa 75 10 f6 c1 01 74 |A..Xr...U.u....t|
00000080 0b 8a e0 88 56 24 c7 06 a1 06 eb 1e 88 66 04 bf |....V$.......f..|
00000090 0a 00 b8 01 02 8b dc 33 c9 83 ff 05 7f 03 8b 4e |.......3.......N|
000000a0 25 03 4e 02 cd 13 72 29 be 46 07 81 3e fe 7d 55 |%.N...r).F..>.}U|
000000b0 aa 74 5a 83 ef 05 7f da 85 f6 75 83 be 27 07 eb |.tZ.......u..'..|
000000c0 8a 98 91 52 99 03 46 08 13 56 0a e8 12 00 5a eb |...R..F..V....Z.|
000000d0 d5 4f 74 e4 33 c0 cd 13 eb b8 00 00 00 00 00 00 |.Ot.3...........|
000000e0 56 33 f6 56 56 52 50 06 53 51 be 10 00 56 8b f4 |V3.VVRP.SQ...V..|
000000f0 50 52 b8 00 42 8a 56 24 cd 13 5a 58 8d 64 10 72 |PR..B.V$..ZX.d.r|
00000100 0a 40 75 01 42 80 c7 02 e2 f7 f8 5e c3 eb 74 49 |.@u.B......^..tI|
00000110 6e 76 61 6c 69 64 20 70 61 72 74 69 74 69 6f 6e |nvalid partition|
00000120 20 74 61 62 6c 65 00 45 72 72 6f 72 20 6c 6f 61 | table.Error loa|
00000130 64 69 6e 67 20 6f 70 65 72 61 74 69 6e 67 20 73 |ding operating s|
00000140 79 73 74 65 6d 00 4d 69 73 73 69 6e 67 20 6f 70 |ystem.Missing op|
00000150 65 72 61 74 69 6e 67 20 73 79 73 74 65 6d 00 00 |erating system..|
00000160 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000180 00 00 00 8b fc 1e 57 8b f5 cb 00 00 00 00 00 00 |......W.........|
00000190 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
000001b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 80 01 |................|
000001c0 01 00 0c 3f 20 10 20 00 00 00 e0 87 e0 00 00 00 |...? . .........|
000001d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
000001f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa |..............U.|
00000200 57 45 56 82 04 00 00 00 53 43 53 49 20 64 69 73 |WEV.....SCSI dis|
00000210 6b 00 00 00 00 00 00 00 42 55 50 20 53 6c 69 6d |k.......BUP Slim|
00000220 20 42 4b 20 20 20 20 20 00 02 00 00 3f 00 00 00 | BK ....?...|
00000230 ff 00 00 00 01 b6 03 00 c1 3e 00 00 af 88 e0 e8 |.........>......|
00000240 af b5 dd b4 ec a1 df 71 00 00 00 00 00 00 00 00 |.......q........|
00000250 00 00 00 00 af 88 e0 e8 00 00 00 00 00 00 00 00 |................|
00000260 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000270 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000280 00 00 00 00 57 45 56 82 3e 50 10 00 00 20 00 00 |....WEV.>P... ..|
00000290 00 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |. ..............|
000002a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000002b0 00 00 00 00 af 88 e0 e8 00 00 00 00 00 00 00 00 |................|
000002c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000310 00 00 00 00 8f 88 e0 e8 20 00 00 00 00 00 00 00 |........ .......|
00000320 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000330 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00004000 eb 58 90 42 53 44 20 20 34 2e 34 00 02 10 20 00 |.X.BSD 4.4... .|
00004010 02 00 00 00 00 f8 00 00 3f 00 ff 00 20 00 00 00 |........?... ...|
00004020 8f 88 e0 e8 cc 14 1d 00 00 00 00 00 02 00 00 00 |................|
00004030 01 00 02 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00004040 00 01 29 ec 1c a1 7f 4e 4f 20 4e 41 4d 45 20 20 |..)....NO NAME |
00004050 20 20 46 41 54 33 32 20 20 20 fa 31 c0 8e d0 bc | FAT32 .1....|
00004060 00 7c fb 8e d8 e8 00 00 5e 83 c6 19 bb 07 00 fc |.|......^.......|
00004070 ac 84 c0 74 06 b4 0e cd 10 eb f5 30 e4 cd 16 cd |...t.......0....|
00004080 19 0d 0a 4e 6f 6e 2d 73 79 73 74 65 6d 20 64 69 |...Non-system di|
00004090 73 6b 0d 0a 50 72 65 73 73 20 61 6e 79 20 6b 65 |sk..Press any ke|
000040a0 79 20 74 6f 20 72 65 62 6f 6f 74 0d 0a 00 00 00 |y to reboot.....|
000040b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
000041f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa |..............U.|
00004200 52 52 61 41 00 00 00 00 00 00 00 00 00 00 00 00 |RRaA............|
00004210 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
000043e0 00 00 00 00 72 72 41 61 ec 65 8a 0e 20 3b cf 08 |....rrAa.e.. ;..|
000043f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa |..............U.|
00004400 eb 58 90 42 53 44 20 20 34 2e 34 00 02 10 20 00 |.X.BSD 4.4... .|
00004410 02 00 00 00 00 f8 00 00 3f 00 ff 00 20 00 00 00 |........?... ...|
00004420 8f 88 e0 e8 cc 14 1d 00 00 00 00 00 02 00 00 00 |................|
00004430 01 00 02 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00004440 00 00 29 ec 1c a1 7f 4e 4f 20 4e 41 4d 45 20 20 |..)....NO NAME |
00004450 20 20 46 41 54 33 32 20 20 20 fa 31 c0 8e d0 bc | FAT32 .1....|
00004460 00 7c fb 8e d8 e8 00 00 5e 83 c6 19 bb 07 00 fc |.|......^.......|
00004470 ac 84 c0 74 06 b4 0e cd 10 eb f5 30 e4 cd 16 cd |...t.......0....|
00004480 19 0d 0a 4e 6f 6e 2d 73 79 73 74 65 6d 20 64 69 |...Non-system di|
00004490 73 6b 0d 0a 50 72 65 73 73 20 61 6e 79 20 6b 65 |sk..Press any ke|
000044a0 79 20 74 6f 20 72 65 62 6f 6f 74 0d 0a 00 00 00 |y to reboot.....|
000044b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
000045f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa |..............U.|
00004600 52 52 61 41 00 00 00 00 00 00 00 00 00 00 00 00 |RRaA............|
00004610 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
000047e0 00 00 00 00 72 72 41 61 ff ff ff ff ff ff ff ff |....rrAa........|
000047f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa |..............U.|
00004800 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00008000 f8 ff ff 0f ff ff ff 0f ff ff ff 0f 00 00 00 00 |................|
00008010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
0000c800
Examining all this data to learn its meaning would be an interesting exercise, but I'll leave that for later.
3.
$ doas dd if=/dev/zero of=/dev/sd1c count=100 to zero out the first 100 sectors of the drive.
4.
$ hexdump -C -n 51200 /dev/sd1c to see the results:
Code:
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
0000c800
5. Tests:
Code:
$ disklabel -pg sd1
# /dev/rsd1c:
type: SCSI
disk: SCSI disk
label: BUP Slim BK
duid: 0000000000000000
.............................................
total sectors: 3907029167 # total bytes: 1863.0G
boundstart: 0
boundend: 3907029167
drivedata: 0
16 partitions:
# size offset fstype [fsize bsize cpg]
c: 1863.0G 0 unused
Code:
$ fdisk sd1
Disk: sd1 geometry: 243201/255/63 [3907029167 Sectors]
Offset: 0 Signature: 0x0
Starting Ending LBA Info:
#: id C H S - C H S [ start: size ]
-------------------------------------------------------------------------------
0: 00 0 0 0 - 0 0 0 [ 0: 0 ] unused
1: 00 0 0 0 - 0 0 0 [ 0: 0 ] unused
2: 00 0 0 0 - 0 0 0 [ 0: 0 ] unused
3: 00 0 0 0 - 0 0 0 [ 0: 0 ] unused
6. I plugged the disk into my OS 9 Mac. The Mac reported that the disk is unreadable and offered to initialize it as a DOS disk with a size of 1,863GB.
So far, so good. I've zeroed the data on the drive and have a safe way to start over if my new attempts at formatting fail.
How do I properly format a FAT32L drive on OpenBSD?
I tried
newfs_msdos(8):
Code:
$ newfs_msdos -N sd1c
/dev/rsd1c: 3903217392 sectors in 243951087 FAT32 clusters (8192 bytes/cluster)
bps=512 spc=16 res=32 nft=2 mid=0xf0 spt=63 hds=255 hid=0 bsec=3907029167 bspf=1905868 rdcl=2 infs=1 bkbs=2
$ doas newfs_msdos sd1c
That didn't work. There is a virtual disklabel, but no partition table:
Code:
$ disklabel -pg sd1
# /dev/rsd1c:
type: SCSI
disk: SCSI disk
label: BUP Slim BK
duid: 0000000000000000
flags:
bytes/sector: 512
sectors/track: 63
tracks/cylinder: 255
sectors/cylinder: 16065
cylinders: 243201
total sectors: 3907029167 # total bytes: 1863.0G
boundstart: 0
boundend: 3907029167
drivedata: 0
16 partitions:
# size offset fstype [fsize bsize cpg]
c: 1863.0G 0 unused
i: 1863.0G 0 MSDOS
Code:
$ fdisk sd1
Disk: sd1 geometry: 243201/255/63 [3907029167 Sectors]
Offset: 0 Signature: 0xAA55
Starting Ending LBA Info:
#: id C H S - C H S [ start: size ]
-------------------------------------------------------------------------------
0: 00 0 0 0 - 0 0 0 [ 0: 0 ] unused
1: 00 0 0 0 - 0 0 0 [ 0: 0 ] unused
2: 00 0 0 0 - 0 0 0 [ 0: 0 ] unused
3: 00 0 0 0 - 0 0 0 [ 0: 0 ] unused
There is a brand-new port for working with MSDOS disks:
sysutils/dosfstools (check and label FAT file systems)
Quote:
Description:
dosfstools consists of the programs mkfs.fat, fsck.fat and fatlabel to create, check and label file systems of the FAT family.
|
These tools also require a previously-constructed, valid partition table.
It looks like I need to manually partition the disk first, but I can't figure out fdisk(8). It seems like this is the hardest step, although somewhat easier if the disk will not be bootable.
From the
fdisk(8) man page:
Quote:
DESCRIPTION:
Caution is advised when editing these tables, since some platforms use tricks to boot and may rely on specific configurations created at install time.
CAVEATS:
Hand crafted disk layouts are highly error prone. It is common practice, though by no means required, that MBR partitions start on a cylinder boundary (generally head 0, sector 1, but head 1, sector 1 for track 0), and that MBR partitions also end at cylinder boundaries.
|
It looks like there are two ways to use fdisk(8), by passing options on the command line or with the interactive editor.
How do I determine the values for the offset, ID, C/H/S, LBA, and size. Are there sane defaults and algorithms? Are there other things I need to know about to construct a valid, cross-platform FAT32L partition table for a non-bootable disk? Where can I find the documentation?
Plea for a really good tutorial
After searching the Web (and searching some more...) I have concluded that we need a step-by-step how-to on this topic.
I would like to help with writing a how-to guide on formatting MSDOS disks on OpenBSD, but I don't have the technical expertise to do so on my own. I would be willing to test drive a guide and provide feedback.
Many thanks to anyone who can provide assistance.