How should I use ext2 in OpenBSD?

[hanzer]

Originally Posted by jggimi

Yes. "a" is not valid for a foreign filesystem that is is imported into a virtual disklabel via MBR. Try # disklabel sd1 and you should see the foreign filesystem as disklabel partition "i".

I've semi-consciously wondered [for years] why USB drives used the i partition. I created the disklabel partition a on sd1; OpenBSD seemed to be OK with it, at least superficially. Linux couldn't mount the drive (for reasons unknown (the a partition thing(?))). I'll try again with the disklabel partition i and an MBR partion id 83 rather than 85. (GParted used type 83 (I choose 85 arbitrarily (it had "Linux" and "ext" in the name (I should stop programming in Lisp (causes brain damage))))).

Attempt 2 looks like this:

Disk: sd1       geometry: 121601/255/63 [1953525164 Sectors]
Offset: 0       Signature: 0xAA55
            Starting         Ending         LBA Info:
 #: id      C   H   S -      C   H   S [       start:        size ]
-------------------------------------------------------------------------------
 0: 83      0  32  33 - 121601  57  56 [        2048:  1953521664 ] Linux files*
 1: 00      0   0   0 -      0   0   0 [           0:           0 ] unused      
 2: 00      0   0   0 -      0   0   0 [           0:           0 ] unused      
 3: 00      0   0   0 -      0   0   0 [           0:           0 ] unused

16 partitions:
#                size           offset  fstype [fsize bsize  cpg]
  c:       1953525164                0  unused                   
  i:       1953525164                0  ext2fs

/etc/sysctl.conf

hw.allowpowerdown=1
machdep.lidsuspend=0
kern.usermount=1

$ ls -l /dev/sd1i

brw-rw----  1 root  operator    4,  24 Apr 19 15:33 /dev/sd1i

$ userinfo hanzer

login   hanzer
passwd  *
uid     1000
groups  hanzer wheel operator wsrc
change  NEVER
class   staff
gecos   Adam Jensen
dir     /home/hanzer
shell   /bin/ksh
expire  NEVER

$ doas newfs_ext2fs -O1 -m1 /dev/rsd1i

...and waiting for newfs. To be continued...

[ocicat]

Originally Posted by hanzer

I've semi-consciously wondered [for years] why USB drives used the i partition.

The issue isn't specifically USB filesystems as it is filesystems other than FFS.

The heuristic used is to choose the midpoint between partition "a" & "p" -- which is partition "i" for foreign filesystems, or the first unused partition letter if "i" is already in use for something else.

[jggimi]

When there is no disklabel on a drive, OpenBSD will inspect the MBR (or GPT) and look for any known MBR partitions and assign them to a virtual disklabel, starting with "i". It does so in the even you planned to add an OpenBSD MBR partition and native disklabel partitions. Partition "a" through "h" are left unused, excepting "c", the partition that maps to the physical drive.

The admin can write that disklabel to the drive, if he or she chooses. But note that when there is already a disklabel on the drive, the foreign MBR partitions will be ignored, so changes in MBR partitions and disklabel partitions need to be manually synchronized.

You could test with a smaller partition, you know. You need not build a 1TB filesystem.

[hanzer]

Originally Posted by jggimi

[...] You could test with a smaller partition, you know. You need not build a 1TB filesystem.

Pfft, and miss out on this excruciating hour of thumb-twiddling? (I'm told that such things build character).

Actually, I thought this would be a fairly straightforward process. I really thought it was just going to work and, after exploring some of the nuance issues and peculiarities, I could move on with my backups and system reconfigurations. That, and I was rather eager to see if e2fsck from sysutils/e2fsprogs worked any better than the native fsck_ext2fs.

Anyway, so yeah, a new strategy might be in order for the next attempt.

[hanzer]

The story so far...
$ ls -l

drwxr-xr-x   2 hanzer  hanzer  512 Apr 21 18:02 mnt/

$ mount_ext2fs -o rw,nosuid,nodev /dev/sd1i $HOME/mnt

$ ls -l

drwxr-xr-x   3 root    wheel   4096 Apr 21 17:51 mnt/

I didn't expect those permissions.

Temporary fix: $ doas chown -R hanzer:hanzer ./mnt

Carried the USB drive to an Ubuntu machine and it can't mount it. <sigh>. GParted couldn't even recognize a valid partition. Deleted the existing partition with GParted and created a new partition formatted as FAT32 (as a test). Unplugged then plugged, Linux auto-mounted; I copied a file, all good.

On the OpenBSD machine:

$ mount_msdos -o rw,nosuid,nodev /dev/sd1i /home/hanzer/mnt

mount_msdos: /dev/sd1i on /home/hanzer/mnt: Inappropriate file type or format

Trying this: $ mount /dev/sd1i /home/hanzer/mnt
no complaints but:
$ ls mnt

ls: mnt: Bad file descriptor

<sigh> I feel like a monkey trying to get a banana out of one of those puzzle boxes.

I use USB thumb-drives with both Linux and OpenBSD; I know the tech works. What am I doing wrong?

Regrouping...

$ doas fdisk sd1

Disk: sd1       geometry: 121601/255/63 [1953525164 Sectors]
Offset: 0       Signature: 0xAA55
            Starting         Ending         LBA Info:
 #: id      C   H   S -      C   H   S [       start:        size ]
-------------------------------------------------------------------------------
 0: 0B      0  32  33 - 121601  57  56 [        2048:  1953521664 ] Win95 FAT-32
 1: 00      0   0   0 -      0   0   0 [           0:           0 ] unused      
 2: 00      0   0   0 -      0   0   0 [           0:           0 ] unused      
 3: 00      0   0   0 -      0   0   0 [           0:           0 ] unused

$ doas disklabel sd1

16 partitions:
#                size           offset  fstype [fsize bsize  cpg]
  c:       1953525164                0  unused                   
  i:       1953525164                0  ext2fs

Hmm, why is that still there? Trying $ doas disklabel -E sd1 and deleteing the i partition.

Unplug, re-plug, then:
$ mount_msdos -o rw,nosuid,nodev /dev/sd1i /home/hanzer/mnt

mount_msdos: /dev/sd1i on /home/hanzer/mnt: Device not configured

$ mount -o rw,nosuid,nodev /dev/sd1i /home/hanzer/mnt

mount_ffs: /dev/sd1i on /home/hanzer/mnt: Device not configured

Today sucks.

[jggimi]

Pardon me, but part of your problem is you are confusing yourself between MBR and disklabel. The drive has both. Up above, I told you that if you have both, the MBR will not be used. When there is a disklabel on the drive, you don't get a virtual disklabel. Just the real one. Which does not match the MBR you created later.

---

Disklabel locations are architecture dependent. Back before GPT support, on i386/amd64 I recall they were written to the second sector of the drive, if there was no OpenBSD MBR partition (or no MBR), or they were written to the second sector of the OpenBSD partition, if one exists. I'm unsure if they are still written to the same location on GPT disks.

A simple way to ensure a drive is clean is to fill the opening sectors with zeros, such as # dd if=/dev/zero of=/dev/rsd9c count=64

---

How did a disklabel get written to the disk? With disklabel(8).

---

You can force a re-read of the MBR partition, and remapping of foreign partitions if you want. See the D command in disklabel(8)

[hanzer]

Originally Posted by jggimi

A simple way to ensure a drive is clean is to fill the opening sectors with zeros, such as # dd if=/dev/zero of=/dev/rsd9c count=64

Yeah, after I got over my tantrum I let $ doas dd if=/dev/zero of=/dev/rsd1c run for a minute then started over.

Latest update: Ubuntu Linux surprisingly doesn't respond well to ext2; it tries to mount the drive as ext4 and fails. Manually mounting works but it isn't an option and neither is tweaking the default behavior. So it looks like FAT32 is the way to go after all. (Ubuntu default auto-mounts the FAT32 drive in an expected and usable way).

This is what the drive looks like now:
fdisk sd1

-------------------------------------------------------------------------------
 0: 0B      0  32  33 - 121601  57  56 [        2048:  1953521664 ] Win95 FAT-32
 1: 00      0   0   0 -      0   0   0 [           0:           0 ] unused      
 2: 00      0   0   0 -      0   0   0 [           0:           0 ] unused      
 3: 00      0   0   0 -      0   0   0 [           0:           0 ] unused

disklabel sd1

16 partitions:
#                size           offset  fstype [fsize bsize  cpg]
  c:       1953525164                0  unused                   
  i:       1953521664             2048   MSDOS

and $ mount_msdos -o rw,nosuid,nodev /dev/sd1i /home/hanzer/mnt works and permissions are fine.

------
BTW- when the drive was ext2, I got these results from the two fsck implementations:
$ fsck_ext2fs /dev/sd1i

** /dev/rsd1i (NO WRITE)
BAD SUPER BLOCK: VALUES IN SUPER BLOCK DISAGREE WITH THOSE IN FIRST ALTERNATE
/dev/rsd1i: BLOCK SIZE DETERMINED TO BE ZERO

$ e2fsck /dev/sd1i

e2fsck 1.42.12 (29-Aug-2014)
TOSHIBA-USB: clean, 12/61054976 files, 3856106/244190208 blocks
Signal (10) SIGBUS si_code=BUS_OBJERR fault addr=0x71ca16c167b

[jggimi]

• EXT, if it works, provides the ability to carry POSIX gid/uid and file modes semantics across directly. That's not possible with FAT, of course.
• Native NTFS has always been read/only. It's no longer experimental, but it isn't possible to use it for bidirectional transfers. And like FAT, the filesystem does not have POSIX semantics.
• The ntfs-3g package allows writing to NTFS filesystems, has filesystem management tools, and can map Windows access controls to POSIX semantics, but the mapping requires planning and careful provisioning. I tested ntfs-3g when the port was first in development in 2013. Performance was very slow, as it depends on the FUSE library (Filesystem in User Space), and if there is adequate network bandwidth NFS may be faster, easier, and more convenient for transferring files between Linux and OpenBSD.

[jggimi]

I have just created a port of "udfclient" -- it can be used to format, read, and write UDF filesystems. But it is not a filesystem implementation, such as ntfs-3g or sshfs. It is an application with an ftp-ish syntax.

If you set the blocksize to 2048 bytes, you can mount a filesystem you've created and written to previously, with mount_udf(8), as OpenBSD supports UDF filesystems read only, with that blocksize.

The application is confusing to use, there is no documentation except for lists of options that appear when you execute one of the tools without options. I will need to write a README to explain how to use it.

I'll commit it to the openbsd-wip project on Github, if anyone is interested in playing with it, just as soon as I have a README written, perhaps as early as tomorrow. The port was developed on -current, but it should build with 5.9 or 5.8 without any changes needed.

[e1-531g]

Originally Posted by jggimi

• EXT, if it works, provides the ability to carry POSIX gid/uid and file modes semantics across directly. That's not possible with FAT, of course.

You can always use tar/pax to wrap files inside "*.tar" file(s) and carry these information.*

You have two clear options:
1. Wrap all files inside one, big tar file. The easiest, but have some limitations:
it is more prone to damage, it is slowest, it creates the biggest files.
2. Wrap every file inside separate tar file. It needs some scripting, but one liner should be enough.

And you have continuum between these two options, but this requires more scripting, to write something more intelligent scripts, or writing standalone programs.

*BTW When somebody installs package inside Linux and this package contains script creating new user account (i.e. for new daemon), it does not usually specify UID/GID, so every installation od Gnu/Linux system can have different UID/GID for the same daemon.
I don't know if this is true to daemons in OpenBSD, but if you create users by hand you should manually specify appropriate UID/GID, because without that that UID/GID inside tar files and POSIX compliant filesystem is going to map to different or non existent user account.

[e1-531g]

@hanzer
I see that you specified offset of filesystem to 0. I would not do that.
I usually create MBR and specify offset to 1024 and if I specify disklabel manually, I would also specify this as offset. Then I would create new filesystem.
I think offset 1024 is non standard and I also thin I have seen 64 as offset, but I specify 1024 and it works for FAT well. 1024 is probably non optimal but working value.

[jggimi]

Originally Posted by jggimi

I'll commit it to the openbsd-wip project on Github, if anyone is interested in playing with it, just as soon as I have a README written.

Done. The README file is here. There are three optical device utilities in the port I am currently unable to test, so if anyone is interested in testing those, I'd be delighted to receive feedback.

Originally Posted by e1-531g

You can always use tar/pax to wrap files inside "*.tar" file(s) and carry these information.*

And other file packagers which retain this, such as dump(8). Of course, FAT has individual file size limits that one needs to be aware of, the largest is 4GB less one byte under FAT32.

[hanzer]

Originally Posted by e1-531g

Y[...]
2. Wrap every file inside separate tar file. It needs some scripting, but one liner should be enough.

And you have continuum between these two options, but this requires more scripting, to write something more intelligent scripts, or writing standalone programs.
[...]

Ahh! That's clever. I might make use of that method some day

Originally Posted by e1-531g

@hanzer
I see that you specified offset of filesystem to 0. I would not do that.
I usually create MBR and specify offset to 1024 and if I specify disklabel manually, I would also specify this as offset. Then I would create new filesystem.
I think offset 1024 is non standard and I also thin I have seen 64 as offset, but I specify 1024 and it works for FAT well. 1024 is probably non optimal but working value.

Hmm, I didn't really question the defaults for those values. What are the issues?

[jggimi]

Originally Posted by hanzer

What are the issues?

The filesystem might overlay the MBR in sector 0. And if there are extended partitions, the MBR is larger than a single sector. Best practice for decades has been to reserve the first "track" of 63 sectors for MBR sectors and boot blocks. Even though CHS disk geometries have been meaningless, fabricated values nearly as long, needed only by older BIOS implementations.

These days, sometimes 127 sectors or more may be reserved.

[hanzer]

The drive is in use at the moment as a FAT32 workhorse [or mule(?)] and there isn't any forensic data other than the notes posted in this thread.

Speculatively: I suppose the reported offset of 0 could have been the result of a problem with:

1. The preset and/or calculated defaults suggested by fdisk and/or disklabel.
2. The values reported by fdisk/disklabel might not be accurate or might be misleading [or might be prone to misinterpretation(?)].
3. The MBR/disklabel/first-(64+bytes)-on-the-disk in a `wonky state

Analytically: I could probably find an old thumb-drive and with that, the man-pages, the FAQ, and this forum, and then ...well, learn something

It's very possible that I just bumbled something. I haven't really considered and don't really understand [why?] that PC/DOS-esche, close-to-the-disk layer of stuff {sufficiently(?)}.

[jggimi]

It's not that difficult to wrap one's head around. BIOSes for years have had limited compute power, and very limited data sizes, which is why CHS (cylinder/head/sector) addressing has remained a "thing". For BIOS I/O. As an example, that 63-sectors per track number has remained in play so long because the sectors per track were stored in 6-bit fields. 6-bits can count from 1 to 63.

The MBR contains a small boot program and a partition table that holds 4 "primary" partitions. Any additional partitions are called "extended", and they are mapped into other extended MBR sectors near the front of the disk.

[jggimi]

I have been testing UDF for file transfer between OpenBSD and Win7, and revised the README document referenced above as I've learned more.

UDF is the follow on to ISO 9660, and is used on DVDs and other optical media. But it was designed as an inter-OS interchange format, and most operating systems can deal with it in some fashion.

1. OpenBSD's mount_udf(8) can only mount read-only UDF filesystems formatted with a 2048-byte block size, which limits it to optical media or filesystems expressly set to that blocksize. It is not useful for transfers between operating systems. But if you do happen to format a device with a 2048-byte blocksize filesystem, the kernel will recognize it and create a virtual disklabel with disklabel partition "a", as it does for CD9660 filesystems on optical devices, or FAT filesystems on diskette.
2. UDF filesystems on disk drives do not use MBRs. The UDF filesystem is written starting at sector 0. This will please hanzer.
3. Windows can format USB sticks as UDF filesystems, but not from the GUI. A format command must be issued from a command line: C:\> format <drive letter:> /fs:UDF /q

[IdOp]

Originally Posted by jggimi

Any additional partitions are called "extended", and they are mapped into other extended MBR sectors near the front of the disk.

jggimi, I think what happens is within the primary MBR (sector 0) there can be at most 1 reference to an extended partition. Then, in the disk area defined by the extended partition, there can be "logical partitions". Each of these has their own little MBR at the start, which defines how big the logical partition is in one of the slots, and has a pointer to the next logical partition in another slot. The other two slots are 0's. So it's like a chain of extended partitions. In the last link of the chain, the pointer is 0's too. So all of these "extra MBRs" live out in the extended partition somewhere, not at the start of the disk.

[jggimi]

Yep, I sit corrected. On my couch. The link-listed chain of logical MBRs live on in the extended partition, and bootloaders may live in the reserved area of the first track.

Thanks for the correction!

[IdOp]

You're welcome, it's easy to forget these things, as they were designed long ago for a faded OS.

I recall having to deal with that chain in the early 90's. I"d just gotten a brand new whopping big 350 Megabyte hard disk, and was trying to fill it full of 32 MB or smaller partitions. (I didn't like big partitions for fear of losing too much data if the filesystem went bad.) So I was using DOS' fdisk command, and somewhere around partition L: or so, the thing crashed and went into a loop. Apparently a bug in the program, thanks BIll. What to do? I used one of the Norton programs, something like partition editor(?), to step through the chain and see how they worked. Then I was able to edit the last partition where fdisk had crashed, and go on to create the rest of the ones I wanted. This was a harrowing experience for a newb, and probably why I still remember today.