DaemonForums  

Go Back   DaemonForums > FreeBSD > FreeBSD General

FreeBSD General Other questions regarding FreeBSD which do not fit in any of the categories below.

Reply
 
Thread Tools Display Modes
  #1   (View Single Post)  
Old 15th May 2008
yurtesen yurtesen is offline
Port Guard
 
Join Date: May 2008
Posts: 18
Default Approaching the limit on PV entries, consider increasing either the vm.pmap.shpgperp

Hello,

I have a FreeBSD 7.0-STABLE amd64 box which gives this message with apache 2.2 very often. Previously the contents of the box was on 6.3-STABLE x86 and I had no such problems. This started right away when we moved to 7, 64bit.

FreeBSD web.XXXXX.com 7.0-STABLE FreeBSD 7.0-STABLE #0: Tue Apr 22 02:13:30 UTC 2008 yurtesen@web.XXXXX.com:/usr/obj/usr/src/sys/WEB amd64

Approaching the limit on PV entries, consider increasing either the vm.pmap.shpgperproc or the vm.pmap.pv_entry_max sysctl.
Approaching the limit on PV entries, consider increasing either the vm.pmap.shpgperproc or the vm.pmap.pv_entry_max sysctl.

I have increased vm.pmap.shpgperproc to 2000 and this seemed to stop complaints for a little while but they occur again...

ipcs -a return nothing

web:/root#ipcs -a
Message Queues:
T ID KEY MODE OWNER GROUP CREATOR CGROUP CBYTES QNUM QBYTES LSPID LRPID STIME RTIME CTIME

Shared Memory:
T ID KEY MODE OWNER GROUP CREATOR CGROUP NATTCH SEGSZ CPID LPID ATIME DTIME CTIME

Semaphores:
T ID KEY MODE OWNER GROUP CREATOR CGROUP NSEMS OTIME CTIME

web:/root#

Here are the active sysctl values:

vm.pmap.pmap_collect_active: 0
vm.pmap.pmap_collect_inactive: 0
vm.pmap.pv_entry_spare: 7366
vm.pmap.pv_entry_allocs: 93953399357
vm.pmap.pv_entry_frees: 93953160939
vm.pmap.pc_chunk_tryfail: 0
vm.pmap.pc_chunk_frees: 559631374
vm.pmap.pc_chunk_allocs: 559632837
vm.pmap.pc_chunk_count: 1463
vm.pmap.pv_entry_count: 238418
vm.pmap.shpgperproc: 2000
vm.pmap.pv_entry_max: 13338058

The box is lightly loaded, it is an 8 core box with load average about 0.2-0.4

Any ideas about what to check/do next? I only could find a post which suggests using:

kern.ipc.shm_use_phys: 1

But I already set it and it has no effect...box has 4gb of memory:

CPU: 0.1% user, 0.0% nice, 0.0% system, 0.0% interrupt, 99.9% idle
Mem: 180M Active, 1584M Inact, 467M Wired, 131M Cache, 214M Buf, 1578M Free
Swap: 8192M Total, 8548K Used, 8184M Free


I have the following in make.conf

CPUTYPE?=core2

CFLAGS= -O2 -fno-strict-aliasing -pipe
CXXFLAGS+= -fconserve-space
COPTFLAGS= -O -pipe

NO_GAMES=true
NO_PROFILE=true

WITHOUT_X11=true

below is the kernel config file:

cpu HAMMER
ident WEB

options SCHED_ULE # ULE scheduler
options PREEMPTION # Enable kernel thread preemption
options INET # InterNETworking
#options INET6 # IPv6 communications protocols
#options SCTP # Stream Control Transmission Protocol
options FFS # Berkeley Fast Filesystem
options SOFTUPDATES # Enable FFS soft updates support
options UFS_DIRHASH # Improve performance on big directories
options UFS_GJOURNAL # Enable gjournal-based UFS journaling
options CD9660 # ISO 9660 Filesystem
options PROCFS # Process filesystem (requires PSEUDOFS)
options PSEUDOFS # Pseudo-filesystem framework
options GEOM_PART_GPT # GUID Partition Tables.
options GEOM_LABEL # Provides labelization
options COMPAT_43TTY # BSD 4.3 TTY compat [KEEP THIS!]
options COMPAT_IA32 # Compatible with i386 binaries
options COMPAT_FREEBSD4 # Compatible with FreeBSD4
options COMPAT_FREEBSD5 # Compatible with FreeBSD5
options COMPAT_FREEBSD6 # Compatible with FreeBSD6
options SCSI_DELAY=5000 # Delay (in ms) before probing SCSI
options KTRACE # ktrace(1) support
options STACK # stack(9) support
options SYSVSHM # SYSV-style shared memory
options SYSVMSG # SYSV-style message queues
options SYSVSEM # SYSV-style semaphores
options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions
options KBD_INSTALL_CDEV # install a CDEV entry in /dev
options ADAPTIVE_GIANT # Giant mutex is adaptive.
options STOP_NMI # Stop CPUS using NMI instead of IPI
#options AUDIT # Security event auditing

# Make an SMP-capable kernel by default
options SMP # Symmetric MultiProcessor Kernel

# CPU frequency control
device cpufreq

# Bus support.
device acpi
device pci

# Floppy drives
device fdc

# ATA and ATAPI devices
device ata
device atadisk # ATA disk drives
device atapicd # ATAPI CDROM drives
options ATA_STATIC_ID # Static device numbering

# SCSI peripherals
device scbus # SCSI bus (required for SCSI)
device da # Direct Access (disks)
device cd # CD
device pass # Passthrough device (direct SCSI access)
#device ses # SCSI Environmental Services (and SAF-TE)

# RAID controllers interfaced to the SCSI subsystem
device twa # 3ware 9000 series PATA/SATA RAID

# atkbdc0 controls both the keyboard and the PS/2 mouse
device atkbdc # AT keyboard controller
device atkbd # AT keyboard
device psm # PS/2 mouse

device kbdmux # keyboard multiplexer

device vga # VGA video card driver

device splash # Splash screen and screen saver support

# syscons is the default console driver, resembling an SCO console
device sc

device agp # support several AGP chipsets

# PCI Ethernet NICs.
device em # Intel PRO/1000 adapter Gigabit Ethernet Card
device ixgb # Intel PRO/10GbE Ethernet Card

# PCI Ethernet NICs that use the common MII bus controller code.
# NOTE: Be sure to keep the 'device miibus' line in order to use these NICs!
device miibus # MII bus support
device dc # DEC/Intel 21143 and various workalikes
device fxp # Intel EtherExpress PRO/100B (82557, 82558)
device re # RealTek 8139C+/8169/8169S/8110S
device rl # RealTek 8129/8139
device sk # SysKonnect SK-984x & SK-982x gigabit Ethernet
device xl # 3Com 3c90x (``Boomerang'', ``Cyclone'')

# Pseudo devices.
device loop # Network loopback
device random # Entropy device
device ether # Ethernet support
device pty # Pseudo-ttys (telnet etc)
device firmware # firmware assist module

# The `bpf' device enables the Berkeley Packet Filter.
# Be aware of the administrative consequences of enabling this!
# Note that 'bpf' is required for DHCP.
device bpf # Berkeley packet filter

# USB support
device uhci # UHCI PCI->USB interface
device ohci # OHCI PCI->USB interface
device ehci # EHCI PCI->USB interface (USB 2.0)
device usb # USB Bus (required)
device ugen # Generic
device uhid # "Human Interface Devices"
device ukbd # Keyboard
device umass # Disks/Mass storage - Requires scbus and da
device ums # Mouse

# My Additions

# Statically Link in accept filters
options ACCEPT_FILTER_DATA
options ACCEPT_FILTER_HTTP

# Other Devices & Options
device snp #Snoop device - to look at pty/vty/etc..



Any ideas on what might be going on?


Thanks,
Evren
Reply With Quote
  #2   (View Single Post)  
Old 15th May 2008
anomie's Avatar
anomie anomie is offline
Local
 
Join Date: Apr 2008
Location: Texas
Posts: 445
Default

Read this email thread in its entirety (i.e. the original and the reply).
__________________
Kill your t.v.
Reply With Quote
  #3   (View Single Post)  
Old 15th May 2008
yurtesen yurtesen is offline
Port Guard
 
Join Date: May 2008
Posts: 18
Default

I already read that, but I already have the setting at 2000 and there are less than 50 httpd processes (about 20-30 average) which should be considered as very low load conditions and this problem usually occurs when the load is higher on the boxes according to posts I found from google. There says only that one can increase shpgperproc without much worry. There is no information about how can one see the pv entries used by a single process or how to get rid of this problem altogether. As I mentioned with almost same configuration the sites on this box was working without this error message on 6.3-stable i386. There must be something wrong/changed with FreeBSD 7.0 which is causing this and there of course must be a solution.
Reply With Quote
  #4   (View Single Post)  
Old 15th May 2008
richardpl richardpl is offline
Spam Deminer
 
Join Date: May 2008
Location: Croatia
Posts: 284
Default

I think I already read this on one of freebsd mailing lists.

Read this and others in same thread.
http://docs.freebsd.org/cgi/getmsg.c...freebsd-stable

tuning(7) is must read.

I dont see the point using amd64 on machine <= 4GB of RAM. And having 8 GB for SWAP is also strange. If you plan to use 8GB of swap at any time, buy instead more RAM.

Last edited by richardpl; 15th May 2008 at 09:12 PM.
Reply With Quote
  #5   (View Single Post)  
Old 15th May 2008
yurtesen yurtesen is offline
Port Guard
 
Join Date: May 2008
Posts: 18
Default

Well, I read all those posts, and tuning, yet my questions are still unanswered...Believe me I have searched everything. Is there anybody who has an answer to how to monitor pv entries used by processes?

While not related, you are mistaken in usage of FreeBSD amd64 for <=4gb ram machines. For one thing, when you use amd64 version, you automatically have access to 64bit registers and as well as double the amount of SSE registers. This simple fact does double the speed of certain programs. Also there are further advantages of using amd64 version as you have access to parts of the processor and functionality which you cant use with i386 version. Thus, even with 1GB of memory, using amd64 version of FreeBSD would be beneficial. As well as this machine has 4GB of memory anyway and with i386 version most machines are only able to use 3.2GB of 4GB memory.
Reply With Quote
  #6   (View Single Post)  
Old 16th May 2008
richardpl richardpl is offline
Spam Deminer
 
Join Date: May 2008
Location: Croatia
Posts: 284
Default

Ask developers, try freebsd-hackers mailing lists.

Just because of doubled registers you should use more than 4GB of RAM ....
Reply With Quote
  #7   (View Single Post)  
Old 16th May 2008
yurtesen yurtesen is offline
Port Guard
 
Join Date: May 2008
Posts: 18
Default

I have asked to freebsd-stable list as well. I am waiting if somebody replies or not.

I am sorry but where do you get this information about one should use more than 4GB ram with 64bit OSes? I do not want to upset anybody here but you might want to check on your resources twice.

The registers work independent from the main memory. A register is the fastest piece of memory available in the computer and processor can only operate on data which is in a register. If you have more registers for usage, you do not have to move data so frequently between slow main memory and registers (which can take several clock cycles) and you can save tremendous amount of processing time if you have a program which operates on same variables very often as you will have more spare registers to keep the data in. You will have advantages whether you use 1gb of memory or 3gb of memory or more.

As examples to irrelevance of the memory amount in a system and 64bit os usage, there have been many 64bit ultrasparc computers with as low as 128mbyte memory as well as a lot of 64bit processors routers use 128mb-256mb memory. The efficiency of using 64bit processor is independent of the memory amount installed on the system.

The fact that 32bit systems can not directly address memory over 4gb means that you must use a 64bit OS to effectively use memory in systems which has more than 4GB of memory. But there is no reason to use a 32bit OS because you have less than 4GB memory.

There are more advantages of using 64bit OS other than the speed gains, a machine owner with a machine with 2gb of ram all of a sudden might decide to upgrade to 8gb of ram. Currently there is no way to upgrade FreeBSD 32bit to FreeBSD 64bit.

As well as if one is using prebuilt packages on an i386 box then they are not using all the advantages that the current processors provide like SSE functions or some other internal processor functions. This is because prebuilt packages should be compatible with a wide range of processors as low as 486 processors, so you are effectively not using any functions added to processor since 486 processors. However if you are using a 64bit OS, as this architecture is very new, all 64bit CPUs (supporting amd64 codeset) for sure support many current functions where the compilers already know that these processors for sure support these functions thus use these advanced functions while compiling a program for 64bit platform.

Long story short, it is always advantageous to use a 64bit OS if the processor is supporting 64bit independent of the amount of memory installed.

Thanks,
Evren
Reply With Quote
  #8   (View Single Post)  
Old 16th May 2008
richardpl richardpl is offline
Spam Deminer
 
Join Date: May 2008
Location: Croatia
Posts: 284
Default

amd64 != ultrasparc

I never claimed that i386 is better in any way than amd64 or any other 64 bit or higer technology. My point was that amd64 OS is more useful with more RAM, and having only 4GB is just very minimalistic ... and that of course should not(I hope so) have anything with your current problem.

Having 64bit OS implies that size of int are doubled, and that doubles usage of memory.
Reply With Quote
  #9   (View Single Post)  
Old 16th May 2008
yurtesen yurtesen is offline
Port Guard
 
Join Date: May 2008
Posts: 18
Default

Actually, that information is not correct unless I am mistaken. Int variables have a 32bit size and it doesnt matter if you compile your program on a 32bit or 64bit machine. One must use long int if 64bit int is wanted. 32bit registers do exist in 64bit machines too so there is no reason for the compiler to allocate twice the amount of space. As a repeatable example look at this tiny program compiled on 64bit OS.

Quote:
#include <stdio.h>

main() {
int a,c;
long int b,d;

a=2147483649;
c=32;
b=2147483649;
d=64;

printf("%d %ld %d %ld \n",a,b,c,d);

}
If you look at the object code (the source lines are visible if you use -pg -g switches while compiling and you will need profiled libraries in the system to be able to use -pg)

Quote:
main() {
400618: 55 push %rbp
400619: 48 89 e5 mov %rsp,%rbp
40061c: 48 83 ec 20 sub $0x20,%rsp
400620: e8 b3 fe ff ff callq 4004d8 <mcount@plt>
int a,c;
long int b,d;

a=2147483649;
400625: c7 45 e8 01 00 00 80 movl $0x80000001,0xffffffffffffffe8(%rbp)
c=32;
40062c: c7 45 ec 20 00 00 00 movl $0x20,0xffffffffffffffec(%rbp)
b=2147483649;
400633: c7 45 f0 01 00 00 80 movl $0x80000001,0xfffffffffffffff0(%rbp)
40063a: c7 45 f4 00 00 00 00 movl $0x0,0xfffffffffffffff4(%rbp)
d=64;
400641: 48 c7 45 f8 40 00 00 movq $0x40,0xfffffffffffffff8(%rbp)
400648: 00

printf("%d %ld %d %ld \n",a,b,c,d);
400649: 48 8b 45 f8 mov 0xfffffffffffffff8(%rbp),%rax
40064d: 8b 4d ec mov 0xffffffffffffffec(%rbp),%ecx
400650: 48 8b 55 f0 mov 0xfffffffffffffff0(%rbp),%rdx
400654: 8b 75 e8 mov 0xffffffffffffffe8(%rbp),%esi
400657: 49 89 c0 mov %rax,%r8
40065a: bf 88 07 40 00 mov $0x400788,%edi
40065f: b8 00 00 00 00 mov $0x0,%eax
400664: e8 2f fe ff ff callq 400498 <printf@plt>

}
I have made bold the first memory locations for int variables, e8 and ec, if you do a ec - e8 = 4 you can see that there is 4bytes difference when int variables are used meaning int variable occupies 4bytes in the memory. Whereas long int (64bit integer) variables f8 - f0 = 8 meaning they occupy 8bytes of memory space.

Also as you can see, the registers, rax,rdx (64bit registers) and ecx,esi (32bit registers) are separetely used depending on the variable size printed.

If you were right, and int variables occupied 64bits on 64bit architectures then the machine couldnt use 32bit registers to store them for printing, also the memory space used would be 8bytes instead of 4bytes.

Output:

Quote:
-2147483647 2147483649 32 64
As you can see the, 32bit int variable have overflowed and changed into a negative value. Because we have exceeded 32bit storage limit.

The only problem which I can see is that the memory alignment can force some small memory gaps however the compiler already does optimize for this this situation.

Honestly, I am quite tired of this discussion. If you think that using 32bit OS you are saving few bytes from 1gb-4gb memory then please go ahead. I just hope that you wouldnt suggest people who has <=4gb memory to use 32bit OS. Because that is a bad suggestion.

Thanks,
Evren
Reply With Quote
Old 16th May 2008
ephemera's Avatar
ephemera ephemera is offline
Knuth's homeboy
 
Join Date: Apr 2008
Posts: 537
Default

yes, fbsd on x64 uses LP64 data model ie. int=32 bits pointers and long=64 bits.

64 bit software has its pros and cons. the performance advantages of 64bit over 32bit software is not as clear-cut as you claim.
Reply With Quote
Old 17th May 2008
yurtesen yurtesen is offline
Port Guard
 
Join Date: May 2008
Posts: 18
Default

Well it is pretty clear to me. When you use AMD64 port of FreeBSD the OS has access to both 32bit parts of the processor and in addition to 64bit parts of the processor. Having a 64bit processor basicly means the OS has access to more resources. The programs might take advantage or not is another question of course. However compilers do optimizations. Agreed, some programs might use slightly more memory but nowadays even laptops ship with 2gb of memory, having a process which use 300mb memory use 400mb instead is negligible downside.

Here is some test results of various applications (not on FreeBSD but...)
http://www.scribd.com/doc/363677/Ben...it-mode-Ubuntu
According to these results most applications give 20-30% speed increase and some even 100% and only one of them is mentioned to be using noticeably more amount of memory.

So, if one considers pros and cons of this, if you have a 64bit processor you would be better of using 64bit capable binaries, not only for better speed, but also for being able to utilize full capabilities of the processor. Otherwise paying for 64bit processor but use it as 32bit doesnt make much sense.
Reply With Quote
Old 17th May 2008
ephemera's Avatar
ephemera ephemera is offline
Knuth's homeboy
 
Join Date: Apr 2008
Posts: 537
Default

<digressing> how did you get the assembly o/p with the C code?
can gcc do that or did you just insert them yourself?
Reply With Quote
Old 17th May 2008
yurtesen yurtesen is offline
Port Guard
 
Join Date: May 2008
Posts: 18
Default

I already put a note for this in my post with the code. If you compile the program with -pg flag then the source code lines are visible in the objdump -S output. (so I didnt enter them myself, and actually I am not sure if -g flag is also necessary at the moment but I used -pg and -g at the same time). For using -pg flag, you need to have profiled libraries installed in your system.

This also allows gprof to profile the code line-by-line so you can see exactly in which line code your program is spending time when you run gprof. Normally gprof only shows statistics in a function level which is not very useful.

I am writing my master's thesis on code optimization so I have been experimenting quite much with these stuff lately. You would be amazed how much performance increase one can get when you change one variable to use a cpu register actually in a certain program which is they run on supercomputing clusters, with kind of change(and few other changes) we have achieved 2 times speed increase where the program running 1 month could finish in 2 weeks only.

If you are interested in these stuff you might find intel vtune and amd codeanalyst quite interesting as well.
Reply With Quote
Old 17th May 2008
ephemera's Avatar
ephemera ephemera is offline
Knuth's homeboy
 
Join Date: Apr 2008
Posts: 537
Default

thanks, i found the objdump -S (profiling info is not required for this) feature very useful.

regarding 64 bit performance the ubuntu benchmarks looks overoptimistic.

this paper looks more credible: http://lixom.net/~olof/64bit-perf.pdf

Last edited by ephemera; 17th May 2008 at 09:51 PM.
Reply With Quote
Old 18th May 2008
yurtesen yurtesen is offline
Port Guard
 
Join Date: May 2008
Posts: 18
Default

The results can vary greatly depending on the compiler version and flags used as well as the program used for tests.

For example it seems like indeed OpenSSL test for DES gives slower results in 64bit however I recently wrote a program using Perl with Crypt-DES, and it definetely works magnitudes of faster on 64bit. I didnt take time but on 32bit it was visibly few times slower. (the program was suppose to crack a des encrypted file and it could do it way faster on 64bit)

So few programs maybe get screwed 1%-2% on the speed (especially synthetic benchmarks) but most can and do greatly benefit from 64bit. If you can find some benchmarks which show that the results from those real life programs from the benchmark results I posted are working slower on 64bit, then one of the benchmarks would be wrong of course but we are comparing apples with oranges here. 2 different set of results from few different programs does not really show the overall performance of a real life situation.

If in doubt, do your own tests. You cant really trust a 3rd person's tests. You cant imagine how many times I made some tests which were flawed for one or another reason and realized it later that I made some mistakes.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Limit Bandwidth (not throughput) plexter OpenBSD Security 5 9th October 2008 05:10 PM
limit use memory by Apache mfaridi FreeBSD Security 4 8th July 2008 05:59 PM
Approaching the limit on PV entries ccc FreeBSD General 6 14th June 2008 06:58 PM
Apache data transfer limit cajunman4life General software and network 5 7th June 2008 05:13 PM


All times are GMT. The time now is 08:03 AM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick