Approaching the limit on PV entries, consider increasing either the vm.pmap.shpgperp
I have a FreeBSD 7.0-STABLE amd64 box which gives this message with apache 2.2 very often. Previously the contents of the box was on 6.3-STABLE x86 and I had no such problems. This started right away when we moved to 7, 64bit.
FreeBSD web.XXXXX.com 7.0-STABLE FreeBSD 7.0-STABLE #0: Tue Apr 22 02:13:30 UTC 2008 yurtesen@web.XXXXX.com:/usr/obj/usr/src/sys/WEB amd64
Approaching the limit on PV entries, consider increasing either the vm.pmap.shpgperproc or the vm.pmap.pv_entry_max sysctl.
Approaching the limit on PV entries, consider increasing either the vm.pmap.shpgperproc or the vm.pmap.pv_entry_max sysctl.
I have increased vm.pmap.shpgperproc to 2000 and this seemed to stop complaints for a little while but they occur again...
ipcs -a return nothing
T ID KEY MODE OWNER GROUP CREATOR CGROUP CBYTES QNUM QBYTES LSPID LRPID STIME RTIME CTIME
T ID KEY MODE OWNER GROUP CREATOR CGROUP NATTCH SEGSZ CPID LPID ATIME DTIME CTIME
T ID KEY MODE OWNER GROUP CREATOR CGROUP NSEMS OTIME CTIME
Here are the active sysctl values:
The box is lightly loaded, it is an 8 core box with load average about 0.2-0.4
Any ideas about what to check/do next? I only could find a post which suggests using:
But I already set it and it has no effect...box has 4gb of memory:
CPU: 0.1% user, 0.0% nice, 0.0% system, 0.0% interrupt, 99.9% idle
Mem: 180M Active, 1584M Inact, 467M Wired, 131M Cache, 214M Buf, 1578M Free
Swap: 8192M Total, 8548K Used, 8184M Free
I have the following in make.conf
CFLAGS= -O2 -fno-strict-aliasing -pipe
COPTFLAGS= -O -pipe
below is the kernel config file:
options SCHED_ULE # ULE scheduler
options PREEMPTION # Enable kernel thread preemption
options INET # InterNETworking
#options INET6 # IPv6 communications protocols
#options SCTP # Stream Control Transmission Protocol
options FFS # Berkeley Fast Filesystem
options SOFTUPDATES # Enable FFS soft updates support
options UFS_DIRHASH # Improve performance on big directories
options UFS_GJOURNAL # Enable gjournal-based UFS journaling
options CD9660 # ISO 9660 Filesystem
options PROCFS # Process filesystem (requires PSEUDOFS)
options PSEUDOFS # Pseudo-filesystem framework
options GEOM_PART_GPT # GUID Partition Tables.
options GEOM_LABEL # Provides labelization
options COMPAT_43TTY # BSD 4.3 TTY compat [KEEP THIS!]
options COMPAT_IA32 # Compatible with i386 binaries
options COMPAT_FREEBSD4 # Compatible with FreeBSD4
options COMPAT_FREEBSD5 # Compatible with FreeBSD5
options COMPAT_FREEBSD6 # Compatible with FreeBSD6
options SCSI_DELAY=5000 # Delay (in ms) before probing SCSI
options KTRACE # ktrace(1) support
options STACK # stack(9) support
options SYSVSHM # SYSV-style shared memory
options SYSVMSG # SYSV-style message queues
options SYSVSEM # SYSV-style semaphores
options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions
options KBD_INSTALL_CDEV # install a CDEV entry in /dev
options ADAPTIVE_GIANT # Giant mutex is adaptive.
options STOP_NMI # Stop CPUS using NMI instead of IPI
#options AUDIT # Security event auditing
# Make an SMP-capable kernel by default
options SMP # Symmetric MultiProcessor Kernel
# CPU frequency control
# Bus support.
# Floppy drives
# ATA and ATAPI devices
device atadisk # ATA disk drives
device atapicd # ATAPI CDROM drives
options ATA_STATIC_ID # Static device numbering
# SCSI peripherals
device scbus # SCSI bus (required for SCSI)
device da # Direct Access (disks)
device cd # CD
device pass # Passthrough device (direct SCSI access)
#device ses # SCSI Environmental Services (and SAF-TE)
# RAID controllers interfaced to the SCSI subsystem
device twa # 3ware 9000 series PATA/SATA RAID
# atkbdc0 controls both the keyboard and the PS/2 mouse
device atkbdc # AT keyboard controller
device atkbd # AT keyboard
device psm # PS/2 mouse
device kbdmux # keyboard multiplexer
device vga # VGA video card driver
device splash # Splash screen and screen saver support
# syscons is the default console driver, resembling an SCO console
device agp # support several AGP chipsets
# PCI Ethernet NICs.
device em # Intel PRO/1000 adapter Gigabit Ethernet Card
device ixgb # Intel PRO/10GbE Ethernet Card
# PCI Ethernet NICs that use the common MII bus controller code.
# NOTE: Be sure to keep the 'device miibus' line in order to use these NICs!
device miibus # MII bus support
device dc # DEC/Intel 21143 and various workalikes
device fxp # Intel EtherExpress PRO/100B (82557, 82558)
device re # RealTek 8139C+/8169/8169S/8110S
device rl # RealTek 8129/8139
device sk # SysKonnect SK-984x & SK-982x gigabit Ethernet
device xl # 3Com 3c90x (``Boomerang'', ``Cyclone'')
# Pseudo devices.
device loop # Network loopback
device random # Entropy device
device ether # Ethernet support
device pty # Pseudo-ttys (telnet etc)
device firmware # firmware assist module
# The `bpf' device enables the Berkeley Packet Filter.
# Be aware of the administrative consequences of enabling this!
# Note that 'bpf' is required for DHCP.
device bpf # Berkeley packet filter
# USB support
device uhci # UHCI PCI->USB interface
device ohci # OHCI PCI->USB interface
device ehci # EHCI PCI->USB interface (USB 2.0)
device usb # USB Bus (required)
device ugen # Generic
device uhid # "Human Interface Devices"
device ukbd # Keyboard
device umass # Disks/Mass storage - Requires scbus and da
device ums # Mouse
# My Additions
# Statically Link in accept filters
# Other Devices & Options
device snp #Snoop device - to look at pty/vty/etc..
Any ideas on what might be going on?
I already read that, but I already have the setting at 2000 and there are less than 50 httpd processes (about 20-30 average) which should be considered as very low load conditions and this problem usually occurs when the load is higher on the boxes according to posts I found from google. There says only that one can increase shpgperproc without much worry. There is no information about how can one see the pv entries used by a single process or how to get rid of this problem altogether. As I mentioned with almost same configuration the sites on this box was working without this error message on 6.3-stable i386. There must be something wrong/changed with FreeBSD 7.0 which is causing this and there of course must be a solution.
I think I already read this on one of freebsd mailing lists.
Read this and others in same thread.
tuning(7) is must read.
I dont see the point using amd64 on machine <= 4GB of RAM. And having 8 GB for SWAP is also strange. If you plan to use 8GB of swap at any time, buy instead more RAM.
Last edited by richardpl; 15th May 2008 at 09:12 PM.
Well, I read all those posts, and tuning, yet my questions are still unanswered...Believe me I have searched everything. Is there anybody who has an answer to how to monitor pv entries used by processes?
While not related, you are mistaken in usage of FreeBSD amd64 for <=4gb ram machines. For one thing, when you use amd64 version, you automatically have access to 64bit registers and as well as double the amount of SSE registers. This simple fact does double the speed of certain programs. Also there are further advantages of using amd64 version as you have access to parts of the processor and functionality which you cant use with i386 version. Thus, even with 1GB of memory, using amd64 version of FreeBSD would be beneficial. As well as this machine has 4GB of memory anyway and with i386 version most machines are only able to use 3.2GB of 4GB memory.
I have asked to freebsd-stable list as well. I am waiting if somebody replies or not.
I am sorry but where do you get this information about one should use more than 4GB ram with 64bit OSes? I do not want to upset anybody here but you might want to check on your resources twice.
The registers work independent from the main memory. A register is the fastest piece of memory available in the computer and processor can only operate on data which is in a register. If you have more registers for usage, you do not have to move data so frequently between slow main memory and registers (which can take several clock cycles) and you can save tremendous amount of processing time if you have a program which operates on same variables very often as you will have more spare registers to keep the data in. You will have advantages whether you use 1gb of memory or 3gb of memory or more.
As examples to irrelevance of the memory amount in a system and 64bit os usage, there have been many 64bit ultrasparc computers with as low as 128mbyte memory as well as a lot of 64bit processors routers use 128mb-256mb memory. The efficiency of using 64bit processor is independent of the memory amount installed on the system.
The fact that 32bit systems can not directly address memory over 4gb means that you must use a 64bit OS to effectively use memory in systems which has more than 4GB of memory. But there is no reason to use a 32bit OS because you have less than 4GB memory.
There are more advantages of using 64bit OS other than the speed gains, a machine owner with a machine with 2gb of ram all of a sudden might decide to upgrade to 8gb of ram. Currently there is no way to upgrade FreeBSD 32bit to FreeBSD 64bit.
As well as if one is using prebuilt packages on an i386 box then they are not using all the advantages that the current processors provide like SSE functions or some other internal processor functions. This is because prebuilt packages should be compatible with a wide range of processors as low as 486 processors, so you are effectively not using any functions added to processor since 486 processors. However if you are using a 64bit OS, as this architecture is very new, all 64bit CPUs (supporting amd64 codeset) for sure support many current functions where the compilers already know that these processors for sure support these functions thus use these advanced functions while compiling a program for 64bit platform.
Long story short, it is always advantageous to use a 64bit OS if the processor is supporting 64bit independent of the amount of memory installed.
amd64 != ultrasparc
I never claimed that i386 is better in any way than amd64 or any other 64 bit or higer technology. My point was that amd64 OS is more useful with more RAM, and having only 4GB is just very minimalistic ... and that of course should not(I hope so) have anything with your current problem.
Having 64bit OS implies that size of int are doubled, and that doubles usage of memory.
Actually, that information is not correct unless I am mistaken. Int variables have a 32bit size and it doesnt matter if you compile your program on a 32bit or 64bit machine. One must use long int if 64bit int is wanted. 32bit registers do exist in 64bit machines too so there is no reason for the compiler to allocate twice the amount of space. As a repeatable example look at this tiny program compiled on 64bit OS.
Also as you can see, the registers, rax,rdx (64bit registers) and ecx,esi (32bit registers) are separetely used depending on the variable size printed.
If you were right, and int variables occupied 64bits on 64bit architectures then the machine couldnt use 32bit registers to store them for printing, also the memory space used would be 8bytes instead of 4bytes.
The only problem which I can see is that the memory alignment can force some small memory gaps however the compiler already does optimize for this this situation.
Honestly, I am quite tired of this discussion. If you think that using 32bit OS you are saving few bytes from 1gb-4gb memory then please go ahead. I just hope that you wouldnt suggest people who has <=4gb memory to use 32bit OS. Because that is a bad suggestion.
Well it is pretty clear to me. When you use AMD64 port of FreeBSD the OS has access to both 32bit parts of the processor and in addition to 64bit parts of the processor. Having a 64bit processor basicly means the OS has access to more resources. The programs might take advantage or not is another question of course. However compilers do optimizations. Agreed, some programs might use slightly more memory but nowadays even laptops ship with 2gb of memory, having a process which use 300mb memory use 400mb instead is negligible downside.
Here is some test results of various applications (not on FreeBSD but...)
According to these results most applications give 20-30% speed increase and some even 100% and only one of them is mentioned to be using noticeably more amount of memory.
So, if one considers pros and cons of this, if you have a 64bit processor you would be better of using 64bit capable binaries, not only for better speed, but also for being able to utilize full capabilities of the processor. Otherwise paying for 64bit processor but use it as 32bit doesnt make much sense.
I already put a note for this in my post with the code. If you compile the program with -pg flag then the source code lines are visible in the objdump -S output. (so I didnt enter them myself, and actually I am not sure if -g flag is also necessary at the moment but I used -pg and -g at the same time). For using -pg flag, you need to have profiled libraries installed in your system.
This also allows gprof to profile the code line-by-line so you can see exactly in which line code your program is spending time when you run gprof. Normally gprof only shows statistics in a function level which is not very useful.
I am writing my master's thesis on code optimization so I have been experimenting quite much with these stuff lately. You would be amazed how much performance increase one can get when you change one variable to use a cpu register actually in a certain program which is they run on supercomputing clusters, with kind of change(and few other changes) we have achieved 2 times speed increase where the program running 1 month could finish in 2 weeks only.
If you are interested in these stuff you might find intel vtune and amd codeanalyst quite interesting as well.
thanks, i found the objdump -S (profiling info is not required for this) feature very useful.
regarding 64 bit performance the ubuntu benchmarks looks overoptimistic.
this paper looks more credible: http://lixom.net/~olof/64bit-perf.pdf
Last edited by ephemera; 17th May 2008 at 09:51 PM.
The results can vary greatly depending on the compiler version and flags used as well as the program used for tests.
For example it seems like indeed OpenSSL test for DES gives slower results in 64bit however I recently wrote a program using Perl with Crypt-DES, and it definetely works magnitudes of faster on 64bit. I didnt take time but on 32bit it was visibly few times slower. (the program was suppose to crack a des encrypted file and it could do it way faster on 64bit)
So few programs maybe get screwed 1%-2% on the speed (especially synthetic benchmarks) but most can and do greatly benefit from 64bit. If you can find some benchmarks which show that the results from those real life programs from the benchmark results I posted are working slower on 64bit, then one of the benchmarks would be wrong of course but we are comparing apples with oranges here. 2 different set of results from few different programs does not really show the overall performance of a real life situation.
If in doubt, do your own tests. You cant really trust a 3rd person's tests. You cant imagine how many times I made some tests which were flawed for one or another reason and realized it later that I made some mistakes.
|Thread||Thread Starter||Forum||Replies||Last Post|
|Limit Bandwidth (not throughput)||plexter||OpenBSD Security||5||9th October 2008 05:10 PM|
|limit use memory by Apache||mfaridi||FreeBSD Security||4||8th July 2008 05:59 PM|
|Approaching the limit on PV entries||ccc||FreeBSD General||6||14th June 2008 06:58 PM|
|Apache data transfer limit||cajunman4life||General software and network||5||7th June 2008 05:13 PM|