DaemonForums  

Go Back   DaemonForums > OpenBSD > OpenBSD General

OpenBSD General Other questions regarding OpenBSD which do not fit in any of the categories below.

Reply
 
Thread Tools Display Modes
Old 21st April 2016
hanzer's Avatar
hanzer hanzer is offline
Real Name: Adam Jensen
just passing through
 
Join Date: Oct 2013
Location: EST USA
Posts: 307
Default A short time ago on a computer not so far away...

Quote:
Originally Posted by jggimi View Post
Yes. "a" is not valid for a foreign filesystem that is is imported into a virtual disklabel via MBR. Try # disklabel sd1 and you should see the foreign filesystem as disklabel partition "i".
I've semi-consciously wondered [for years] why USB drives used the i partition. I created the disklabel partition a on sd1; OpenBSD seemed to be OK with it, at least superficially. Linux couldn't mount the drive (for reasons unknown (the a partition thing(?))). I'll try again with the disklabel partition i and an MBR partion id 83 rather than 85. (GParted used type 83 (I choose 85 arbitrarily (it had "Linux" and "ext" in the name (I should stop programming in Lisp (causes brain damage))))).

Attempt 2 looks like this:
Code:
Disk: sd1       geometry: 121601/255/63 [1953525164 Sectors]
Offset: 0       Signature: 0xAA55
            Starting         Ending         LBA Info:
 #: id      C   H   S -      C   H   S [       start:        size ]
-------------------------------------------------------------------------------
 0: 83      0  32  33 - 121601  57  56 [        2048:  1953521664 ] Linux files*
 1: 00      0   0   0 -      0   0   0 [           0:           0 ] unused      
 2: 00      0   0   0 -      0   0   0 [           0:           0 ] unused      
 3: 00      0   0   0 -      0   0   0 [           0:           0 ] unused
Code:
16 partitions:
#                size           offset  fstype [fsize bsize  cpg]
  c:       1953525164                0  unused                   
  i:       1953525164                0  ext2fs
/etc/sysctl.conf
Code:
hw.allowpowerdown=1
machdep.lidsuspend=0
kern.usermount=1
$ ls -l /dev/sd1i
Code:
brw-rw----  1 root  operator    4,  24 Apr 19 15:33 /dev/sd1i
$ userinfo hanzer
Code:
login   hanzer
passwd  *
uid     1000
groups  hanzer wheel operator wsrc
change  NEVER
class   staff
gecos   Adam Jensen
dir     /home/hanzer
shell   /bin/ksh
expire  NEVER
$ doas newfs_ext2fs -O1 -m1 /dev/rsd1i

...and waiting for newfs. To be continued...
Reply With Quote
Old 21st April 2016
ocicat ocicat is offline
Administrator
 
Join Date: Apr 2008
Posts: 3,297
Default

Quote:
Originally Posted by hanzer View Post
I've semi-consciously wondered [for years] why USB drives used the i partition.
The issue isn't specifically USB filesystems as it is filesystems other than FFS.

The heuristic used is to choose the midpoint between partition "a" & "p" -- which is partition "i" for foreign filesystems, or the first unused partition letter if "i" is already in use for something else.
Reply With Quote
Old 21st April 2016
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 6,377
Default

When there is no disklabel on a drive, OpenBSD will inspect the MBR (or GPT) and look for any known MBR partitions and assign them to a virtual disklabel, starting with "i". It does so in the even you planned to add an OpenBSD MBR partition and native disklabel partitions. Partition "a" through "h" are left unused, excepting "c", the partition that maps to the physical drive.

The admin can write that disklabel to the drive, if he or she chooses. But note that when there is already a disklabel on the drive, the foreign MBR partitions will be ignored, so changes in MBR partitions and disklabel partitions need to be manually synchronized.

You could test with a smaller partition, you know. You need not build a 1TB filesystem.

Last edited by jggimi; 21st April 2016 at 10:35 PM. Reason: clarity
Reply With Quote
Old 21st April 2016
hanzer's Avatar
hanzer hanzer is offline
Real Name: Adam Jensen
just passing through
 
Join Date: Oct 2013
Location: EST USA
Posts: 307
Default

Quote:
Originally Posted by jggimi View Post
[...] You could test with a smaller partition, you know. You need not build a 1TB filesystem.
Pfft, and miss out on this excruciating hour of thumb-twiddling? (I'm told that such things build character).

Actually, I thought this would be a fairly straightforward process. I really thought it was just going to work and, after exploring some of the nuance issues and peculiarities, I could move on with my backups and system reconfigurations. That, and I was rather eager to see if e2fsck from sysutils/e2fsprogs worked any better than the native fsck_ext2fs.

Anyway, so yeah, a new strategy might be in order for the next attempt.
Reply With Quote
Old 22nd April 2016
hanzer's Avatar
hanzer hanzer is offline
Real Name: Adam Jensen
just passing through
 
Join Date: Oct 2013
Location: EST USA
Posts: 307
Default

The story so far...
$ ls -l
Code:
drwxr-xr-x   2 hanzer  hanzer  512 Apr 21 18:02 mnt/
$ mount_ext2fs -o rw,nosuid,nodev /dev/sd1i $HOME/mnt

$ ls -l
Code:
drwxr-xr-x   3 root    wheel   4096 Apr 21 17:51 mnt/
I didn't expect those permissions.

Temporary fix: $ doas chown -R hanzer:hanzer ./mnt

Carried the USB drive to an Ubuntu machine and it can't mount it. <sigh>. GParted couldn't even recognize a valid partition. Deleted the existing partition with GParted and created a new partition formatted as FAT32 (as a test). Unplugged then plugged, Linux auto-mounted; I copied a file, all good.

On the OpenBSD machine:

$ mount_msdos -o rw,nosuid,nodev /dev/sd1i /home/hanzer/mnt
Code:
mount_msdos: /dev/sd1i on /home/hanzer/mnt: Inappropriate file type or format
Trying this: $ mount /dev/sd1i /home/hanzer/mnt
no complaints but:
$ ls mnt
Code:
ls: mnt: Bad file descriptor
<sigh> I feel like a monkey trying to get a banana out of one of those puzzle boxes.

I use USB thumb-drives with both Linux and OpenBSD; I know the tech works. What am I doing wrong?

Regrouping...

$ doas fdisk sd1
Code:
Disk: sd1       geometry: 121601/255/63 [1953525164 Sectors]
Offset: 0       Signature: 0xAA55
            Starting         Ending         LBA Info:
 #: id      C   H   S -      C   H   S [       start:        size ]
-------------------------------------------------------------------------------
 0: 0B      0  32  33 - 121601  57  56 [        2048:  1953521664 ] Win95 FAT-32
 1: 00      0   0   0 -      0   0   0 [           0:           0 ] unused      
 2: 00      0   0   0 -      0   0   0 [           0:           0 ] unused      
 3: 00      0   0   0 -      0   0   0 [           0:           0 ] unused
$ doas disklabel sd1
Code:
16 partitions:
#                size           offset  fstype [fsize bsize  cpg]
  c:       1953525164                0  unused                   
  i:       1953525164                0  ext2fs
Hmm, why is that still there? Trying $ doas disklabel -E sd1 and deleteing the i partition.

Unplug, re-plug, then:
$ mount_msdos -o rw,nosuid,nodev /dev/sd1i /home/hanzer/mnt
Code:
mount_msdos: /dev/sd1i on /home/hanzer/mnt: Device not configured
$ mount -o rw,nosuid,nodev /dev/sd1i /home/hanzer/mnt
Code:
mount_ffs: /dev/sd1i on /home/hanzer/mnt: Device not configured
Today sucks.
Reply With Quote
Old 22nd April 2016
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 6,377
Default

Pardon me, but part of your problem is you are confusing yourself between MBR and disklabel. The drive has both. Up above, I told you that if you have both, the MBR will not be used. When there is a disklabel on the drive, you don't get a virtual disklabel. Just the real one. Which does not match the MBR you created later.

---

Disklabel locations are architecture dependent. Back before GPT support, on i386/amd64 I recall they were written to the second sector of the drive, if there was no OpenBSD MBR partition (or no MBR), or they were written to the second sector of the OpenBSD partition, if one exists. I'm unsure if they are still written to the same location on GPT disks.

A simple way to ensure a drive is clean is to fill the opening sectors with zeros, such as # dd if=/dev/zero of=/dev/rsd9c count=64

---

How did a disklabel get written to the disk? With disklabel(8).

---

You can force a re-read of the MBR partition, and remapping of foreign partitions if you want. See the D command in disklabel(8)
Reply With Quote
Old 22nd April 2016
hanzer's Avatar
hanzer hanzer is offline
Real Name: Adam Jensen
just passing through
 
Join Date: Oct 2013
Location: EST USA
Posts: 307
Default

Quote:
Originally Posted by jggimi View Post
A simple way to ensure a drive is clean is to fill the opening sectors with zeros, such as # dd if=/dev/zero of=/dev/rsd9c count=64
Yeah, after I got over my tantrum I let $ doas dd if=/dev/zero of=/dev/rsd1c run for a minute then started over.

Latest update: Ubuntu Linux surprisingly doesn't respond well to ext2; it tries to mount the drive as ext4 and fails. Manually mounting works but it isn't an option and neither is tweaking the default behavior. So it looks like FAT32 is the way to go after all. (Ubuntu default auto-mounts the FAT32 drive in an expected and usable way).

This is what the drive looks like now:
fdisk sd1
Code:
-------------------------------------------------------------------------------
 0: 0B      0  32  33 - 121601  57  56 [        2048:  1953521664 ] Win95 FAT-32
 1: 00      0   0   0 -      0   0   0 [           0:           0 ] unused      
 2: 00      0   0   0 -      0   0   0 [           0:           0 ] unused      
 3: 00      0   0   0 -      0   0   0 [           0:           0 ] unused
disklabel sd1
Code:
16 partitions:
#                size           offset  fstype [fsize bsize  cpg]
  c:       1953525164                0  unused                   
  i:       1953521664             2048   MSDOS
and $ mount_msdos -o rw,nosuid,nodev /dev/sd1i /home/hanzer/mnt works and permissions are fine.

------
BTW- when the drive was ext2, I got these results from the two fsck implementations:
$ fsck_ext2fs /dev/sd1i
Code:
** /dev/rsd1i (NO WRITE)
BAD SUPER BLOCK: VALUES IN SUPER BLOCK DISAGREE WITH THOSE IN FIRST ALTERNATE
/dev/rsd1i: BLOCK SIZE DETERMINED TO BE ZERO
$ e2fsck /dev/sd1i
Code:
e2fsck 1.42.12 (29-Aug-2014)
TOSHIBA-USB: clean, 12/61054976 files, 3856106/244190208 blocks
Signal (10) SIGBUS si_code=BUS_OBJERR fault addr=0x71ca16c167b
Reply With Quote
Old 22nd April 2016
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 6,377
Default terrible terabyte transfers test tolerance

  • EXT, if it works, provides the ability to carry POSIX gid/uid and file modes semantics across directly. That's not possible with FAT, of course.
  • Native NTFS has always been read/only. It's no longer experimental, but it isn't possible to use it for bidirectional transfers. And like FAT, the filesystem does not have POSIX semantics.
  • The ntfs-3g package allows writing to NTFS filesystems, has filesystem management tools, and can map Windows access controls to POSIX semantics, but the mapping requires planning and careful provisioning. I tested ntfs-3g when the port was first in development in 2013. Performance was very slow, as it depends on the FUSE library (Filesystem in User Space), and if there is adequate network bandwidth NFS may be faster, easier, and more convenient for transferring files between Linux and OpenBSD.
Reply With Quote
Old 22nd April 2016
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 6,377
Default

I have just created a port of "udfclient" -- it can be used to format, read, and write UDF filesystems. But it is not a filesystem implementation, such as ntfs-3g or sshfs. It is an application with an ftp-ish syntax.

If you set the blocksize to 2048 bytes, you can mount a filesystem you've created and written to previously, with mount_udf(8), as OpenBSD supports UDF filesystems read only, with that blocksize.

The application is confusing to use, there is no documentation except for lists of options that appear when you execute one of the tools without options. I will need to write a README to explain how to use it.

I'll commit it to the openbsd-wip project on Github, if anyone is interested in playing with it, just as soon as I have a README written, perhaps as early as tomorrow. The port was developed on -current, but it should build with 5.9 or 5.8 without any changes needed.

Last edited by jggimi; 22nd April 2016 at 04:17 AM. Reason: clarity
Reply With Quote
Old 22nd April 2016
e1-531g e1-531g is offline
VPN Cryptographer
 
Join Date: Mar 2014
Posts: 448
Default

Quote:
Originally Posted by jggimi View Post
  • EXT, if it works, provides the ability to carry POSIX gid/uid and file modes semantics across directly. That's not possible with FAT, of course.
You can always use tar/pax to wrap files inside "*.tar" file(s) and carry these information.*

You have two clear options:
1. Wrap all files inside one, big tar file. The easiest, but have some limitations:
it is more prone to damage, it is slowest, it creates the biggest files.
2. Wrap every file inside separate tar file. It needs some scripting, but one liner should be enough.

And you have continuum between these two options, but this requires more scripting, to write something more intelligent scripts, or writing standalone programs.

*BTW When somebody installs package inside Linux and this package contains script creating new user account (i.e. for new daemon), it does not usually specify UID/GID, so every installation od Gnu/Linux system can have different UID/GID for the same daemon.
I don't know if this is true to daemons in OpenBSD, but if you create users by hand you should manually specify appropriate UID/GID, because without that that UID/GID inside tar files and POSIX compliant filesystem is going to map to different or non existent user account.

Last edited by e1-531g; 22nd April 2016 at 07:04 AM.
Reply With Quote
Old 22nd April 2016
e1-531g e1-531g is offline
VPN Cryptographer
 
Join Date: Mar 2014
Posts: 448
Default

@hanzer
I see that you specified offset of filesystem to 0. I would not do that.
I usually create MBR and specify offset to 1024 and if I specify disklabel manually, I would also specify this as offset. Then I would create new filesystem.
I think offset 1024 is non standard and I also thin I have seen 64 as offset, but I specify 1024 and it works for FAT well. 1024 is probably non optimal but working value.
Reply With Quote
Old 22nd April 2016
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 6,377
Default

Quote:
Originally Posted by jggimi View Post
I'll commit it to the openbsd-wip project on Github, if anyone is interested in playing with it, just as soon as I have a README written.
Done. The README file is here. There are three optical device utilities in the port I am currently unable to test, so if anyone is interested in testing those, I'd be delighted to receive feedback.
Quote:
Originally Posted by e1-531g View Post
You can always use tar/pax to wrap files inside "*.tar" file(s) and carry these information.*
And other file packagers which retain this, such as dump(8). Of course, FAT has individual file size limits that one needs to be aware of, the largest is 4GB less one byte under FAT32.

Last edited by jggimi; 22nd April 2016 at 01:58 PM. Reason: revised link
Reply With Quote
Old 22nd April 2016
hanzer's Avatar
hanzer hanzer is offline
Real Name: Adam Jensen
just passing through
 
Join Date: Oct 2013
Location: EST USA
Posts: 307
Default

Quote:
Originally Posted by e1-531g View Post
Y[...]
2. Wrap every file inside separate tar file. It needs some scripting, but one liner should be enough.

And you have continuum between these two options, but this requires more scripting, to write something more intelligent scripts, or writing standalone programs.
[...]
Ahh! That's clever. I might make use of that method some day

Quote:
Originally Posted by e1-531g View Post
@hanzer
I see that you specified offset of filesystem to 0. I would not do that.
I usually create MBR and specify offset to 1024 and if I specify disklabel manually, I would also specify this as offset. Then I would create new filesystem.
I think offset 1024 is non standard and I also thin I have seen 64 as offset, but I specify 1024 and it works for FAT well. 1024 is probably non optimal but working value.
Hmm, I didn't really question the defaults for those values. What are the issues?
Reply With Quote
Old 22nd April 2016
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 6,377
Default

Quote:
Originally Posted by hanzer View Post
What are the issues?
The filesystem might overlay the MBR in sector 0. And if there are extended partitions, the MBR is larger than a single sector. Best practice for decades has been to reserve the first "track" of 63 sectors for MBR sectors and boot blocks. Even though CHS disk geometries have been meaningless, fabricated values nearly as long, needed only by older BIOS implementations.

These days, sometimes 127 sectors or more may be reserved.

Last edited by jggimi; 22nd April 2016 at 05:12 PM. Reason: typos, clarity
Reply With Quote
Old 22nd April 2016
hanzer's Avatar
hanzer hanzer is offline
Real Name: Adam Jensen
just passing through
 
Join Date: Oct 2013
Location: EST USA
Posts: 307
Default High!

The drive is in use at the moment as a FAT32 workhorse [or mule(?)] and there isn't any forensic data other than the notes posted in this thread.

Speculatively: I suppose the reported offset of 0 could have been the result of a problem with:
  1. The preset and/or calculated defaults suggested by fdisk and/or disklabel.
  2. The values reported by fdisk/disklabel might not be accurate or might be misleading [or might be prone to misinterpretation(?)].
  3. The MBR/disklabel/first-(64+bytes)-on-the-disk in a `wonky state

Analytically: I could probably find an old thumb-drive and with that, the man-pages, the FAQ, and this forum, and then ...well, learn something

It's very possible that I just bumbled something. I haven't really considered and don't really understand [why?] that PC/DOS-esche, close-to-the-disk layer of stuff {sufficiently(?)}.
Reply With Quote
Old 22nd April 2016
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 6,377
Default

It's not that difficult to wrap one's head around. BIOSes for years have had limited compute power, and very limited data sizes, which is why CHS (cylinder/head/sector) addressing has remained a "thing". For BIOS I/O. As an example, that 63-sectors per track number has remained in play so long because the sectors per track were stored in 6-bit fields. 6-bits can count from 1 to 63.

The MBR contains a small boot program and a partition table that holds 4 "primary" partitions. Any additional partitions are called "extended", and they are mapped into other extended MBR sectors near the front of the disk.

Last edited by jggimi; 22nd April 2016 at 07:17 PM. Reason: clarity
Reply With Quote
Old 22nd April 2016
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 6,377
Default

I have been testing UDF for file transfer between OpenBSD and Win7, and revised the README document referenced above as I've learned more.

UDF is the follow on to ISO 9660, and is used on DVDs and other optical media. But it was designed as an inter-OS interchange format, and most operating systems can deal with it in some fashion.
  1. OpenBSD's mount_udf(8) can only mount read-only UDF filesystems formatted with a 2048-byte block size, which limits it to optical media or filesystems expressly set to that blocksize. It is not useful for transfers between operating systems. But if you do happen to format a device with a 2048-byte blocksize filesystem, the kernel will recognize it and create a virtual disklabel with disklabel partition "a", as it does for CD9660 filesystems on optical devices, or FAT filesystems on diskette.
  2. UDF filesystems on disk drives do not use MBRs. The UDF filesystem is written starting at sector 0. This will please hanzer.
  3. Windows can format USB sticks as UDF filesystems, but not from the GUI. A format command must be issued from a command line: C:\> format <drive letter:> /fs:UDF /q

Last edited by jggimi; 23rd April 2016 at 12:09 AM. Reason: typos
Reply With Quote
Old 23rd April 2016
IdOp's Avatar
IdOp IdOp is offline
Too dumb for a smartphone
 
Join Date: May 2008
Location: twisting on the daemon's fork(2)
Posts: 828
Default

Quote:
Originally Posted by jggimi View Post
Any additional partitions are called "extended", and they are mapped into other extended MBR sectors near the front of the disk.
jggimi, I think what happens is within the primary MBR (sector 0) there can be at most 1 reference to an extended partition. Then, in the disk area defined by the extended partition, there can be "logical partitions". Each of these has their own little MBR at the start, which defines how big the logical partition is in one of the slots, and has a pointer to the next logical partition in another slot. The other two slots are 0's. So it's like a chain of extended partitions. In the last link of the chain, the pointer is 0's too. So all of these "extra MBRs" live out in the extended partition somewhere, not at the start of the disk.
Reply With Quote
Old 23rd April 2016
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 6,377
Default

Yep, I sit corrected. On my couch. The link-listed chain of logical MBRs live on in the extended partition, and bootloaders may live in the reserved area of the first track.

Thanks for the correction!
Reply With Quote
Old 23rd April 2016
IdOp's Avatar
IdOp IdOp is offline
Too dumb for a smartphone
 
Join Date: May 2008
Location: twisting on the daemon's fork(2)
Posts: 828
Default

You're welcome, it's easy to forget these things, as they were designed long ago for a faded OS.

I recall having to deal with that chain in the early 90's. I"d just gotten a brand new whopping big 350 Megabyte hard disk, and was trying to fill it full of 32 MB or smaller partitions. (I didn't like big partitions for fear of losing too much data if the filesystem went bad.) So I was using DOS' fdisk command, and somewhere around partition L: or so, the thing crashed and went into a loop. Apparently a bug in the program, thanks BIll. What to do? I used one of the Norton programs, something like partition editor(?), to step through the chain and see how they worked. Then I was able to edit the last partition where fdisk had crashed, and go on to create the rest of the ones I wanted. This was a harrowing experience for a newb, and probably why I still remember today.
Reply With Quote
Reply

Tags
/etc/fstab, ext2, fstab, openbsd

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Cannot mount Linux ext2 partitions notooth NetBSD General 4 25th October 2015 02:54 PM
inode size support for ext2/3 guitarscn OpenBSD General 2 30th October 2009 03:46 PM
ext2 lost+found gosha General Hardware 0 21st June 2009 01:32 PM
Mounting ext2 partitions seems to fail Sunsawe FreeBSD Installation and Upgrading 2 17th June 2009 01:38 PM
Mounting ext2 in fstab latorion FreeBSD General 3 6th August 2008 04:56 PM


All times are GMT. The time now is 04:57 AM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2018, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick