Fatal trap 12: page fault while in kernel mode

[sixshot]

My Freebsd 6.2 server has crashed twice in the past month requireing a hard reboot eache time to reset. Below is a copy of whats on the screen when this happens for both times. Could it be spam assasin, I have added more hosting accounts in the past couple of months and there is a lot more spam filtering going on. I notice that the perl based spam assasin uses a ton of processing to filter. I am not currently using block lists. I got this server brand new last year and have never had a problem until a month ago. I am going to do a memory scan on the ram tonight.

Any ideas?



June 17 08
----------------------------------------------

kernel trap 12 with interrupts disabled

Fatal trap 12: page fault while in kernel mode

cpuid = 0; apic id = 00
fault virtual address = 0x104
fault code = supervisor read, page not present
instruction pointer = 0x20:0xc066c731
stack pointer = 0x28:8xe34f0c90
frame pointer = 0x28:0xe34f0c9c
code segment = base 0x0, limit 0xfffff, type 0xlb
= DPL 0, pres 1, def32 1, gran 1
processor eflags = resume, IOPL = 0
current process = 5 (thread taskq)
trap number = 12
panic: page fault
cpuid = 0
Uptime: 151d22h34m56s
Cannot dump. No dump device defined.
Automatic reboot in 15 seconds - press a key on the console to abort
Rebooting...
cpu_reset: Stopping other CPUs



7-16-08
--------------------------------------
kernel trap 12 with interrupts disabled

Fatal trap 12: page fault while in kernel mode

cpuid = 0; apic id = 00
fault virtual address = 0x104
fault code = supervisor read, page not present
instruction pointer = 0x20:0xc066c731
stack pointer = 0x28:8xe34f0c90
frame pointer = 0x28:0xe34f0c9c
code segment = base 0x0, limit 0xfffff, type 0xlb
= DPL 0, pres 1, def32 1, gran 1
processor eflags = resume, IOPL = 0
current process = 5 (thread taskq)
trap number = 12
panic: page fault
cpuid = 1

[robbak]

You might want to configure your system with some swap space (at least equal to the memory size), and ensure /var/crash contains free space at least equal to the system memory.

If this is true ( and you have not disabled core dumping in rc.conf), FreeBSD will save a core dump when it panics, and you will be able to get a backtrace, like this:

# kgdb /boot/kernel/kernel /var/crash/vmcore.0
> bt

Although fully understanding backtraces is a fairly advanced science, they can often tell even a novice where the problems are.

[sixshot]

Right now var is a 10 gig partition with 20% used. in var/crash there is a minfree file set to 2048 bu there is no vmcore.0 or other files that minfree in there. I will look into setting up swap space.

[sixshot]

last pid: 7054; load averages: 0.00, 0.01, 0.00 up 0+07:39:38 18:22:58
78 processes: 1 running, 77 sleeping
CPU states: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
Mem: 135M Active, 275M Inact, 123M Wired, 32K Cache, 110M Buf, 456M Free
Swap: 3072M Total, 3072M Free


I am not sure but this top statment says something about swap space

[TerryP]

Terry@dixie$ swapctl -lh                                                   4:24
Device:       1048576-blocks      Used:
/dev/ad0s1b          860          0
Terry@dixie$                                                               4:24

[sixshot]

Ya

#/>swapctl -lh
Device: 1048576-blocks Used:
/dev/ad4s1b 3072 0

[ephemera]

Originally Posted by sixshot

Right now var is a 10 gig partition with 20% used. in var/crash there is a minfree file set to 2048 bu there is no vmcore.0 or other files that minfree in there. I will look into setting up swap space.

you will need to run savecore(8) either manually or specify it in rc.conf.

[robbak]

Let's not forget that the problem is a sig12(kernel page fault), not the lack of a coredump.

The usual suspects:
1. memory faults
2. Binary kernel modules (nvidia/ltmdm etc)
3. Bugs in kernel code (very unlikely)

Getting a dump will help track down 2 & 3, and that memory test _should_ rule out #1.

If you do have any kernel modules from ports, make sure you rebuild them every time you upgrade the kernel.

RE: the dumps:
I was of the impression that, if swap and /var/crash were big enough, dumpon and savecore would be run automagically by their respective rc.d scripts (/etc/rc.d/dumpon and /etc/rc.d/savecore), but this might not be the case in 6.x.
Check your dmesg (or /var/log/messages) for comments on savecore and dumpon, and check /etc/defaults/rc.conf for variables controlling them (put any changes in /etc/rc.conf, of course (sorry if you didn't need that reminder!)).

[sixshot]

I tested the memory and it was fine. The system had 1 gig. I went ahead and added another gig to bring it up to 2 gig. I have not as yet upgraded the kernel. Since I know the memory is ok maybe the problem is a Binary kernel module but I am not sure of the best way to track it down. I am going to subscribe to an rbl list like spamhaus to cut down on spam assasins work. I hope with that and the extra gig of ram the problem goes away. But if anyone has any further ideas im all ears.

[J65nko]

Bad memory is usually the cause of signal 11 errors.

Please follow the procedure outlined in the FreeBSD FAQ about trap 12 errors: http://www.freebsd.org/doc/en_US.ISO...#TRAP-12-PANIC

[sysop1]

I'm having same issue as you, sixshot, and I may have found what is causing the problem (at least in my case). Some differences though - I'm running FreeBSD 7.0-RELEASE - but the I have same issue with recurring "Fatal trap 12".

I've tested all my RAM-modules separately (memtest86+) and they are OK.

The system seems to crash when writing big/many files over the network with smb protocol (Samba) to any of the filesystems (ZFS) on any of the drives. It seems that the harddrive (the one writing data, in this case /dev/ad16) gets disconnected just before the kernel dumps.

Here is my core dump message:

Unread portion of the kernel message buffer:
ad16: FAILURE - device detached
subdisk16: detached
ad16: detached

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0x2c
fault code              = supervisor write, page not present
instruction pointer     = 0x20:0x807489f5
stack pointer           = 0x28:0xd45ffc5c
frame pointer           = 0x28:0xd45ffc70
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 3 (g_up)
trap number             = 12
panic: page fault
cpuid = 0
Uptime: 7m42s
Physical memory: 1267 MB
Dumping 115 MB: 100 84 68 52 36 20 4

#0  doadump () at pcpu.h:195
195     pcpu.h: No such file or directory.
        in pcpu.h

I'm suspecting that it's my PSU (350W cheap one) that can't provide enough power for my rig. I have pretty much stuff and eight harddrives in it.
I'm trying to monitor the voltage, but don't know if results reported by mbmon/chm are valid. If they are I have some serious issues with too small PSU:

chm: 
+5V  = 4.14
+12V = 12.06
-12V = -6
-5V = -2.14
mbmon: 
+5V  = 4.27
+12V = 11.73
-12V = -5.34 
-5V = -1.97

Am I walking on thin ice with my guess?

Or could it be something connected to the numerous problems reported with the ZFS implementation (labeled Experimental) in FreeBSD 7? Have gotten a share load of those errors too, but they use to report different trap numbers and error messages.

[robbak]

the -5 and -12 voltages are rarely used these days. Sound cards used to use them to for their power amplifiers back when they had them, and some modems used them similarly.

It is interesting that the smaller power-supply specs omit -5V, as it is used only on old ISA cards.

Anyway, power supplies are a possible cause of instability, as are failed capacitors (remember the capacitor plague of a few years ago?).