boot: Segmentation fault message

[shep]

In amd64_current, I notice a boot message:

....
savecore :/bsd : _dumpdev not in name list
checking quotas: done
Segmentation fault
clearing /tmp
....

dmesg shows no reference to the boot messages.

My [FILE]/var/log/messages{/FILE] show this additional error.

sendsyslog: dropped 2 messages. error 57, pid 30456

I am able to login.

I did not see this in 6.6 release. Is there additional info needed before I post to tech@openbsd.org and if so, how best to generate?

[jggimi]

You can see the logic in /etc/rc. After mounting filesystems:

1. savecore(8) is run to save a kernel core dump if one exists.
2. acpidump(8) is run in the event you need to use sendbug(1)
3. quotacheck(8) is run
4. kvm_mkdb(8) and dev_mkdb(8) are run
5. chmod(1) and chown(8) are run on /dev/tty devices
6. If /etc/ptmp exists, logger(1) gets run to warn the admin
7. echo(1) is used to state that /tmp is being cleared.

So you know that something between the quotacheck() and the echo() has segfaulted. I'll guess either one of the two mkdb programs. Are either /var/db/kvm_bsd.db or /var/run/dev.db missing?

[shep]

I have a
/var/db/kwm_bsd.tmp
but no
/var/db/kvm_bsd.db

After my packages load, I'll add the missing file, check the owner/group/user permissions from 6.6 stable, reboot and report back.

[jggimi]

Look in your root directory for a .core file. If one exists, you have something you can use for diagnostics. If not, rerun kvm_mkdb() manually to obtain one.

[shep]

No *.core in /

# kvm_mkdb -v
kvm_mkdb: rebuilding /var/db/kvm_bsd.db
Segmentation fault

[jggimi]

The core(2) man page refers to the KERN_NOSUIDCOREDUMP sysctl(2). That man page says:

KERN_NOSUIDCOREDUMP (kern.nosuidcoredump)
Whether a process may dump core after changing user or group ID:
value	condition	dump core to
0	euid == 0	current directory
1	never	
2	always	        /var/crash
3	depends        	/var/crash/$programname/

And of course, the default is 1 -- never. Change this to get a .core file you can use with gdb(1).

[shep]

Kanga# gdb -c kvm_mkdb.core                                                    
GNU gdb 6.3
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-unknown-openbsd6.6".
Core was generated by `kvm_mkdb'.
Program terminated with signal 11, Segmentation fault.
#0  0x000011e59fb260bb in ?? ()
(gdb)

[jggimi]

You may find building the program from source with debug symbols helpful to determining a probable cause with gdb(). A backtrace will show named entry points, you can step through the program line-by-line, etc. If you're interested and have the time.


The last CVS commit to the program was in June - a mass update to many programs for syscall error handling changes.

[shep]

The system with the Segmentation fault is dual booting with Crux Linux/Grub2. My workstation Dual Boots Debian 10 and 6.6stable and does not have the segmentation fault. With that info, I suspect that the error arose sometime after the 6.6 code freeze/unfreeze.

I'm wondering where the kvm_bsd.tmp file came from? I do not recall any errors when the installer generated a unique kernel.

[jggimi]

The program itself has not changed since June 28, as I noted. But there have been plenty of library changes between 6.6 and today.

This has nothing to do with the kernel, or dual booting. Instead, it has everything to do with whatever the program kvm_mkdb(8) does. To learn a little bit more about your problem, run a backtrace on the core file, eg: # gdb /usr/sbin/kvm_mkdb /path/to/kvm_mkdb.core then (gdb) bt. Libraries are not stripped of their symbols, so you will see those symbols in the backtrace, if the stack has not been trashed. But to learn a whole lot more about the problem, you'll need to learn a little debugging, as I mentioned above: source code, debugging symbols, etc.

[ripe]

Yes. And it seems to me that something is trying to write in already allocated memory address.

[jggimi]

You may be correct, but:

#0  0x000011e59fb260bb in ?? ()

is not enough information to determine the problem source and a corrective action.

[ripe]

Yes!

[shep]

Yes. And it seems to me that something is trying to write in already allocated memory address.

I'm wondering if the address access has been pledge/unveil'd or has something to do with the new upgrade mechanism? Another thought on a Dual boot system is that BOOTX64,EFI is on /boot/efi linux file and is not accessible once /bsd starts booting. Not sure if BOOTX64,EFI is now specific for a unique /bsd and if kvm_mkdb needs to access BOOTX64.EFI.

But to learn a whole lot more about the problem, you'll need to learn a little debugging,

I don't have a problem leaning to debug code but I have a sliver of C background - I read about 1/3 of a C coding book. I'm not sure I would recognize a coding syntax error. I also tend to think in terms of process and jggimi /etc/rc reference was helpful.

I'll run the gdb code on a fresh core dump and document the results. Will also ask the community if anyone has done a single-boot, post 6.6 release -current install and gets the error. May try a dedicated install using the same install66.fs.

[jggimi]

This doesn't appear to be related to pledge(2) or unveil(2). Not directly. Violation of a pledge(2) restriction terminates the process with SIGABRT. Violations of unveil(2) result in EACCES errors.

[jggimi]

As I stated above, this shouldn't be related to dual-booting. Procedurally:

1. EFI/BIOS (directly, or chain loading from Linux/Windows) loads the *OpenBSD* bootloader from disk into RAM.
2. EFI/BIOS turns control over to the bootloader.
3. The bootloader reads the disklabel, finds the root partition "a", locates /etc/boot.conf, provisions if found, prompts the admin if permitted, locates the kernel on disk, and loads it into RAM.
4. The bootloader turns over control to the kernel.
5. The kernel initializes the system, scans hardware, produces several pages of dmesg(8) data, then starts init(8) which runs rc(8) in a shell.
6. rc(8) eventually runs kvm_mkdb(8) ... and here we are.

[shep]

I was able to get the error to go away although I was not rigidly systematic.
I originally was using Fedora whose boot loader writes this to the EFI System Partition.

[EFI System Partition]
`-- EFI
    `-- BOOT
        |-- BOOTX64.EFI
        `-- BOOTIA32.EFI

Reference:
https://ackstorm.de/posts/uefi-openb...dual-boot.html

This is identical where to OpenBSD installs its BOOTX64.EFI/BOOTIA32.EFI. After installing OpenBSD (after Fedora/Crux), the system will initially boot directly into OpenBSD without showing the Grub Boot Menu. You can get grub back by booting the Linux install media as rescue and reinstalling grub2.

With Crux linux, Grub2 booting Linux is via EFI System Partition/EFI/Grub/grub.efi so there is no conflict.

[EFI System Partition]
`-- EFI
    `-- Grub
        |-- grub.efi

I made a EFI System Partition/openbsd/ directory and populated it with OpenBSD BOOTX64.EFI. It appears as /boot/efi/openbsd/bootx64.efi.

Then a /etc/grub.d/40-custom

#!/bin/sh
exec tail -n +3 $0
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
menuentry 'OpenBSD' {
         insmod part_gpt
         insmod search_fs_uuid
         insmod chain
         chainloader (hd0,gpt1)/openbsd/bootx64.efi
}

Ran grub-mkconfig -o /boot/grub/grub.cfg
and can select Crux and OpenBSD from the Grub boot menu. OpenBSD boots without segfault messages.

I also used a newer install66.fs although I checked the changlog and did not see anything that I thought would affect OpenBSD bootloader uefi install. Have to see if it survives an OpenBSD update.

[shep]

More Follow Up. On 3rd machine that originally had the segmentation fault, I ran susupgrade and duplicated the Grub2 boot after reinstalling Grub. It still had a Segmentation Fault. I reinstalled OpenBSD and the kernel failed to re-link and reboot generated the same Segmentation Fault. At this point I'm clueless but I think it is a unique issue and not likely to affect common use. The only difference I can see between the second and third machine is the bios and amd vs intel. I have a system to work on ports with and I do not think additional effort will be helpful to the community.

[jggimi]

If you consistently get this fault on the third machine, and you still believe the root cause is dual booting, then you can test single booting on that specific hardware from an external drive, such as a USB stick.

[shep]

I did exactly that, a single boot with the same install media on the same machine. No Segmentation fault on a single boot.