Whole Disk Encryption and USB Drive

[staus]

I'm trying to switch over to a BSD from Slackware Linux. So far OpenBSD has been almost perfect. I have installed 5.9 with whole disk encryption and it seems to work OK. The glitch is that I can no longer mount any USB devices. dmesg will show that a plugged in flash disk is recognized, but it cannot be mounted. Error message says device is busy and the flash disk led flashes constantly. I'm using an SSD in my laptop. Could this be causing a conflict since softraid is also being used? Any help would be appreciated.

[e1-531g]

Can you provide more details, please?

#
#seconds after pluging of USB
dmesg | tail -n 15
sysctl hw.disknames

and disklabel(8) on USB device.

[staus]

Sorry it took so long to reply.

Here is the dmesg information:

scsibus4 detached
umass0 detached
umass0 at uhub0 port 7 configuration 1 interface 0 "PNY Technologies USB 2.0 FD" rev 2.00/11.00 addr 3
umass0: using SCSI over Bulk-Only
scsibus4 at umass0: 2 targets, initiator 0
sd1 at scsibus4 targ 1 lun 0: <PNY, USB 2.0 FD, 1100> SCSI2 0/direct removable serial.154b007a08YE11003234
sd1: 7788MB, 512 bytes/sector, 15950592 sectors
sd1 detached
scsibus4 detached
umass0 detached
umass0 at uhub0 port 7 configuration 1 interface 0 "TOSHIBA TransMemory" rev 2.00/1.00 addr 3
umass0: using SCSI over Bulk-Only
scsibus4 at umass0: 2 targets, initiator 0
sd1 at scsibus4 targ 1 lun 0: <TOSHIBA, TransMemory, 1.00> SCSI2 0/direct removable serial.09306544CF614007589F
sd1: 29587MB, 512 bytes/sector, 60594432 sectors

Here is the sysctl information:

hw.disknames=wd0:07909548bdb1d885,cd0:,sd0:40143ecea7dff7cb,sd1:

The flashdrive is a Toshiba 32 GB standard vfat.

Thanks again, if this will help solve the problem.

[NaWi]

Wait some time till the led stopps flashing then try again to mount it. As bigger as the disk is, as longer it needs. Experienced the same with a 32 / 64 GB USB stick and a external 1 TB harddisk connected using USB.

[jggimi]

When you insert the USB device, the kernel attaches the umass(4) driver and assigns a pseudo sd(4) driver so that it will operate as a disk drive. This looks correct:

.
.
.
umass0 at uhub0 port 7 configuration 1 interface 0 "PNY Technologies USB 2.0 FD" rev 2.00/11.00 addr 3
umass0: using SCSI over Bulk-Only
scsibus4 at umass0: 2 targets, initiator 0
sd1 at scsibus4 targ 1 lun 0: <PNY, USB 2.0 FD, 1100> SCSI2 0/direct removable serial.154b007a08YE11003234
sd1: 7788MB, 512 bytes/sector, 15950592 sectors
.
.
.

At that point, the device is attached to USB and as pseudo-SCSI drive sd1. The disklabel(8) command can be used to display the OS's understanding of the drive:

# disklabel sd1

If there is no OpenBSD disklabel on the drive -- and for a foreign device, there shouldn't be one -- OpenBSD will read the drive's MBR partition table, and assign virtual disklabel partitions to any recognized MBR partition types, starting with disklabel partition "i".

So, if you insert a typical USB stick with a FAT filesystem, such as those used with Windows, there will be two disklabel partitions shown in output to the disklabel(8) command. "c", which is reserved to address the entire physical drive, and "i", which would be the FAT filesystem defined in the USB stick's MBR partition table. Because there is no OpenBSD disklabel on the drive, this is termed a virtual disklabel.

If this were the case, you should be able to mount that FAT filesystem:

# mount /dev/sd1i /mnt

However, you haven't provided any disklabel command output. Instead, I only see:

sd1 detached
scsibus4 detached
umass0 detached
.
.
.

Which to me is an indication that the device was successfully unplugged.

[staus]

Disklabel seems to correctly identify the flash drive as sd1i. But if I try mounting it, the message error is that it is busy. The led indicator on the flash drive continually flashes. I finally removed the flash after 2 minutes. I tried another Toshiba flash drive. it gave the same error. Led keeps flashing.

[jggimi]

Busy? According to the mount(2) syscall man page, EBUSY will be returned if the mount point directory is held by another process. Your problem is not the device, it is the mount point you are attempting to use.

For example, if your shell's working directory is currently /mnt, you will not be able to successfully use /mnt as your mount point, because .... it is in use by your shell.

[staus]

My working directory is my home directory. And I'm attempting to mount the device under a subdirectory called /usb. As soon as I plug in the flash device, the led begins flashing, even as root. I think I remember testing out mounting flash drives before I did a whole disk encryption, and I think it worked as it should then. Still a mystery why the drive flashes immediately at plugin and never stops.

[jggimi]

The fstat(1) tool will tell you why /usb is busy. Perhaps you have have configured a daemon which takes action on change of state, such as amd(8) or hotplugd(8).

[staus]

I ran fstat and got a lot of info, none of which stands out. I have OpenBSD on a Fujitsu laptop and have specified apm, hotplug, and dbus to be run. Could there be a conflict with these? I kept all defaults during the installation. Didn't change any configurations.

[jggimi]

• amd(8) is the automatic mounting daemon.

• hotplugd(8) runs scripts when devices are added or removed.

One of these two daemons is the reason your LED flashes for more than the few seconds it takes for a kernel probe and device attachment, and for the reading of MBR and disklabel.

If $ fstat /usb shows more than just one line of column headers, the directory is in use. You must choose a different mount point, or terminate the processes.

[staus]

I forgot to say I also have toadd enabled. When I try fstat /usb. I get the error no such file.
If I fstat -f ~/usb, I get a long list, the same as when I fstat -f, only FD MOUNT at wd /home. It does not list /usb as being mounted.
If I try mount /dev/sd1i /mnt, I still get a device busy message.

[jggimi]

Originally Posted by staus

I forgot to say I also have toadd enabled.

You have automation you have provisioned which read disklabels every 8 seconds and attempt to mount for you, and you wonder why your LED is blinking and your device is busy?

Try stopping *all* of your automated mounting services.

# rcctl stop toadd
# rcctl stop hotplugd
# rcctl stop amd

Then insert your USB device, and see if you can mount and not get "busy" messages.

When I try fstat /usb. I get the error no such file.

Yesterday, you mentioned you were attempting to mount to /usb. I misunderstood that you meant ~/usb. My apologies, but I only know about your environment from what you provide.

If I fstat -f ~/usb, I get a long list, the same as when I fstat -f, only FD MOUNT at wd /home.

The -f <path> option asks for all open files in the same file system as the path. If ~/usb happens to be /home/staus/usb, and /home is the filesystem, $ fstat -f ~/usb will list every open file and directory in /home.

If you want to see if /home/staus/usb is busy, the command is $ fstat /home/staus/usb

If I try mount /dev/sd1i /mnt, I still get a device busy message.

I'll take the wild guess that your automation has already mounted your USB filesystem for you. The command $ df -h will show you all mounted filesystems, their mount points, their capacity, and their current storage usage. Perhaps /dev/sd1i is already in that list, mounted by toadd somewhere under /run/media/.

[staus]

Thanks for the help. I stopped toadd, hotplugd, and amd. hotplugd was the only daemon that killed the flashing when the drive is plugged in. I also did an fstat | less (after re-anabling all three daemons) to see all info, and did not find any reference to /usb or /media with or without the flash drive plugged in.
I guess I will need to compare the two lists to see what might be different. Work keeps interfering with solving this problem, so I cannot devote more time like I want.
Thanks again.

[staus]

I did a comparison of fstat with the flash memory uplugged versus plugged. The only difference was the pid for fstat. I'm beginning to think the problem is the softraid setup for whole dick encryption on an ssd.
I really like OpenBSD. It has almost all the functions I need for daily laptop use, except for usb access to flash and external harddrives.

[jggimi]

Softraid and USB insertion should not interfere with each other.

As I don't have a clue about your situation except for what you've posted in this thread, I have a suggestion for diagnosis.

Setup

1. Boot into single-user mode. At the boot> prompt, type "-s" and press the Enter key.
2. Select the default shell as the single user, which is /bin/sh.
3. At the first # prompt, mount all default filesystems with # mount -a.
4. Log console output from this point forward with # script. This will store console keystrokes and output in a file called /typescript. It won't store blue/white kernel messages, however.
5. Issue # df -h. This will list all mounted filesystems, mount points, capacities, and current usage. If /mnt is already in use, please make special note of this, because it may cause problems in step 4 below.

Examination

1. Insert your USB stick. Note the sd(4) device number assigned by the kernel. It should be sd1, per what's previously been described by you in this thread. It might not be. I'm going to use sd1 in the remaining guidance here, only because that is the device number you have stated previously. If it is a different device number, please adapt accordingly.
2. Issue # fdisk sd1. Note the output. If this is a standard "Windows-compatible" USB stick, there will be an MBR partition table with a single MBR filesystem of type 0x0B or a similar FAT filesystem partition type.
3. Issue # disklabel sd1. If there was a single MBR partition table of a FAT type, you should see two disklabel partitions. "c" is the physical drive, and "i" is the foreign (FAT) filesystem.
4. Issue # mount /dev/sd1i /mnt. This of course assumes that /mnt is not already mounted.

Diagnosis

1. Close the script console log, with # exit. This will close the /typescript file.
2. If you do not need the log, you may delete it with # rm /typescript.
3. If you want to review or share the log, remove backspace and other control keystrokes with # col -b < /typescript > /my.console.log
4. After creating /my.console.log, you may delete the /typescript file as in step 2 above.
5. If you wish me to look at any or all of the file, post it here.

[staus]

Thanks for all the help. I ran and saved the log file just as you wrote. The led on the flash drive flashed continually after I successfully mounted it, and a listing proved it really did mount on /mnt.
Here is my log file attached, I hope. I cannot tell where it is on my post.

[jggimi]

Thanks for taking the time to run this test. You have been able to confirm a lot of things the problem is not.

It is:

• Not the computer
• Not the OS kernel
• Not softraid
• Not the USB device
• Not the data structures stored on the USB device
• Not your mount command

We have also learned, apparently, that the flashing LED on that USB device does not indicate active I/O, since you were in single-user mode. Once the mount completed, no process was running except the shell, sitting idle waiting for interactive input with a # prompt.

What are we left with, then, as a possible problem cause? The processes you are running when in multi-user mode.

Is it one of your fleet of automounting daemons? I thought so before, and your test seems to have confirmed it.

To test this, try disabling them and rebooting, instead of stopping them. Stopping them didn't seem to eliminate the problem previously.

# rcctl disable amd
# rcctl disable toadd
# rcctl disable hotplugd

If, after reboot, you are able to successfully mount the file system, you will know your root cause is how one of these daemons is configured.

[denta]

For what it's worth, I've used whole disk encryption on an SSD for years without having any issues at all, including mounting USB drives.

But I think I destroyed a 32 GB SanDisk USB drive once though, dd:ing the entire thing it with /dev/zero. It wasn't even working in windoze after that

[staus]

I finally got the usb flash to reliably mount after disabling amd, toadd, and hotplugd. Lots of reboots later, I found an error in the /etc/rc.conf.local for apmd_flags. Thanks for all the help and suggestions. Now on to further testing and use.