OpenBSD to load kernel from USB stick then continue boot from NVMe drive

[mole]

I upgraded my main disk to an NVMe drive which the motherboard can't boot directly from. It's the last AMD processor series before PSP (AMD's version of IME) so I don't want to upgrade.

I dual boot Linux and OpenBSD, and have GRUB loading the Linux kernel from an old USB stick, then the boot continues with root filesystem of the NVMe drive.

GRUB chainloaded OpenBSD fine with the old SSD, but I can't get OpenBSD to boot with the USB stick/NVME setup.

With Linux I initially --set=root --hint-efi=hd0,gpt2 (the USB stick) then on the kernel /vmlinuz line have root=/dev/nvme0n1p4 which then continues to boot from the NVMe.

Is there a way to do this with chainloading, or a better way to boot OpenBSD without chainloading? At the moment I just have an "error:invalid signature" message

[Head_on_a_Stick]

Please share the full configuration used for OpenBSD.

That "invalid signature" error suggests that the wrong partition has been read (or that SecureBoot is enabled, which is unlikely for such an old system).

Is this for a UEFI or non-UEFI system? A GUID partition table would suggest UEFI, unless you've added a BIOS boot partition to hold GRUB's core.img.

Did you transfer the old OpenBSD system to the NVMe drive? If so please share the exact command(s) used to achieve that.

GRUB can boot OpenBSD directly using the kopenbsd command: https://www.gnu.org/software/grub/ma...oader-commands

Note that using "hd0,gpt2" as an identifier is probably not a good idea because the block device assignment can change from one boot to the next. It's better to use something like

search --fs-uuid --set=root $uuid

And replace $uuid with the actual filesystem UUID of the intended partition.

[mole]

It's a UEFI system, and yes, the the NVMe partitions aren't being found when I try and boot a kernel from the NVMe disk - presumably the disk can't be accessed before the kernel is loaded).

This is the BSD-related part of my grub.cfg file (I have copied /EFI/BSD-74/bootx64.efi and /bsd.rd to the NVME partition 6. As a reboot is needed every time the file is updated I've put most options in there and clutched at a few straws, even adding the PARTUUIDs just in case:

menuentry 'kopenBSD' {
        kopenbsd -r sd0a (ahci,openbsd1)/bsd
        boot
}
menuentry 'kopenBSD starts install' {
        kopenbsd /bsd.rd
        boot
}
menuentry 'OpenBSD 7.4' {
        insmod part_gpt
        insmod part_bsd
        insmod search_fs_uuid
        insmod chain
        root
#       Copy the bootx64.efi file from the OpenBSD install USB EFI directory
        chainloader (hd0,gpt2)/EFI/BSD-74/bootx64.efi
        }
menuentry 'OpenBSD USB stick UUID' {
        insmod part_gpt
        insmod part_bsd
        insmod search_fs_uuid
        insmod chain
        # UUID USB stick
        search --fs-uuid --no-floppy --set=root 4ce43809-90ca-4394-aa9c-021bb8b898d2
        chainloader ($root)/EFI/BSD-74/bootx64.efi
        }
menuentry 'OpenBSD USB stick Part UUID bootx64' {
        insmod part_gpt
        insmod part_bsd
        insmod search_fs_uuid
        insmod chain
        # Part UUID NVMe p6
        search --fs-uuid --no-floppy --set=root 7b46cba8-47a5-7042-abf3-a30d3c0eafac
        chainloader ($root)/EFI/BSD-74/bootx64.efi
        }
menuentry 'OpenBSD Part UUID bootx64' {
        insmod part_gpt
        insmod part_bsd
        insmod search_fs_uuid
        insmod chain
        # Part UUID NVMe p6
        search --fs-uuid --no-floppy --set=root 52ef289c-8087-4b79-a015-a1ec0d3aa55e
        chainloader ($root)/EFI/BSD-74/bootx64.efi
        }
menuentry 'OpenBSD UUID bootx64' {
        insmod part_gpt
        insmod part_bsd
        insmod search_fs_uuid
        insmod chain
        # UUID NVMe p6
        search --fs-uuid --no-floppy --set=root 65972b5b27490a9c
        chainloader ($root)/EFI/BSD-74/bootx64.efi
        }
menuentry 'KopenBSD UUID p6 bsd.rd' {
        insmod part_gpt
        insmod part_bsd
        insmod search_fs_uuid
        insmod chain
        # UUID NVMe p6
        search --fs-uuid --no-floppy --set=root 65972b5b27490a9c
        kopenbsd ($root)/bsd.rd
        }
menuentry 'KopenBSD Part UUID p6 bsd.rd' {
        insmod part_gpt
        insmod part_bsd
        insmod search_fs_uuid
        insmod chain
        # PART UUID NVMe p6
        search --fs-uuid --no-floppy --set=root 52ef289c-8087-4b79-a015-a1ec0d3aa55e
        kopenbsd ($root)/bsd.rd
        }
menuentry 'OpenBSD UUID p6 bsd' {
        insmod part_gpt
        insmod part_bsd
        insmod search_fs_uuid
        insmod chain
        # UUID NVMe p6
        search --fs-uuid --no-floppy --set=root 65972b5b27490a9c
        kopenbsd ($root)/bsd
        }
menuentry 'OpenBSD Part UUID p6 bsd' {
        insmod part_gpt
        insmod part_bsd
        insmod search_fs_uuid
        insmod chain
        # Part UUID NVMe p6
        search --fs-uuid --no-floppy --set=root 52ef289c-8087-4b79-a015-a1ec0d3aa55e
        kopenbsd ($root)/bsd
        }

The NVMe UUIDs give a "no such device error"; using the USB UUID or using (hd0,gpt2) the device is found but the error is "invalid signature"

The "kopenbsd /bsd.rd boot" entry boots into the installer every time.

The Linux config is:

menuentry 'Gentoo GNU/Linux' --class gentoo --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-0b4c17d0-1efd-43b1-88eb-993230274>
        load_video
        insmod gzio
        insmod part_gpt
        insmod ext2
        set root='hd0,gpt2'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint-bios=hd0,gpt2 --hint-efi=hd0,gpt2 --hint-baremetal=ahci0,gpt2  4ce43809-90ca-4394-aa9c-021bb8b898d2
        else
          search --no-floppy --fs-uuid --set=root 4ce43809-90ca-4394-aa9c-021bb8b898d2
        fi
        echo    'Loading Linux 6.5.5-gentoo ...'
        linux   /vmlinuz-6.5.5-gentoo root=/dev/nvme0n1p4 ro radeon.dpm=0 radeon.hard_reset=1
}

nvme0n1p6 has a fresh 7.4 install. I just don't know how to load a kernel (either with kopenbsd or chainloading) from the USB stick which then boots with nvme0n1p6 as the root directory.

[Head_on_a_Stick]

You could try copying the PBR of the OpenBSD system to the USB stick and chainload that.

So from the OpenBSD installer shell ( rsd0a might not be correct, edit for your system):

dd if=/dev/rsd0a of=openbsd.pbr bs=512 count=1

Then copy openbsd.pbr to the USB stick and add a stanza to load it, something like

menuentry 'OpenBSD PBR' {
   search --fs-uuid --set=root 4ce43809-90ca-4394-aa9c-021bb8b898d2
   chainloader /path/to/openbsd.pbr
}

Not sure if that will work for a UEFI system though.

[mole]

Thanks but no luck - from the install I get a "device busy" error trying to dd; I can dd the MBR from the install USB with a laptop running OpenBSD, but trying to then boot from it hangs with "ERR M"

As bsd.rd loads OK, but always wants a fresh install as it doesn't find the root filesystem, I think the issue comes down to how command line arguments can be passed to bsd.rd with GRUB. Or how a custom bootx64.efi file could be made.... both of these are possible with Linux but I think I'll just boot from an SSD rather than the USB, with OpenBSD running on the SSD and Linux on the NVMe. For now, anyway

[Head_on_a_Stick]

Originally Posted by mole

Thanks but no luck - from the install I get a "device busy" error trying to dd

Where did you direct the output? The installer USB stick probably won't accept it.

AFAIK bsd.rd does not accept command line arguments. EDIT: i'm wrong, see below (thanks jggimi).

[jggimi]

The ramdisk kernel (bsd.rd) takes the complete list of -acds arguments.

[mole]

For reference - as this thread was frequently being shown by searches - in case anyone else is having this issue:

After spending a lot of time trying rEFInd and learning the UEFI boot process in detail, the solution was to avoid UEFI and chainloading altogether and use kopenbsd, with grub compiled for grub_platforms_pc support

The GRUB entry in the end was:

menuentry 'BSD 7.5' {
        kopenbsd -r sd0a (hd0,gpt2)/bsd
}

with the first partition as 4MB type EF02
the second partition was type EF00 formatted as ext2 as GRUB rescue seemed to recognise that better than vfat, though vfat is standard.

The nvme drive isn't recognised by the BIOS so GRUB sees the USB stick as (hd0), while OpenBSD always allocates sd0 to the nvme