DaemonForums  

Go Back   DaemonForums > OpenBSD > OpenBSD Packages and Ports

OpenBSD Packages and Ports Installation and upgrading of packages and ports on OpenBSD.

Reply
 
Thread Tools Display Modes
  #1   (View Single Post)  
Old 3rd August 2013
daemonfowl daemonfowl is offline
bsdstudent
 
Join Date: Jan 2012
Location: DaemonLand
Posts: 834
Thanked 0 Times in 0 Posts
Default wget: memory exhausted ..

Hi all !
I'm using OpenBSD current
Code:
kern.version=OpenBSD 5.4 (GENERIC.MP) #44: Tue Jul 30 12:13:32 MDT 2013
    deraadt@i386.openbsd.org:/usr/src/sys/arch/i386/compile/GENERIC.MP
I was wgetting -k -m .. sudently it stopped with an error message:
Code:
wget: memory exhausted
I have 2giga of ram + 4giga swap + I was on fvwm running only wget.

Is there any way to prevent this from happening -other than upgrading ram- ?
Regards.
Reply With Quote
  #2   (View Single Post)  
Old 3rd August 2013
daemonfowl daemonfowl is offline
bsdstudent
 
Join Date: Jan 2012
Location: DaemonLand
Posts: 834
Thanked 0 Times in 0 Posts
Default

That's exactly what I've issued :
Code:
wget --no-proxy -r -m -k --no-parent
When target size reached 1.6 g I got that message .. file conversion hadn't taken place yet.
Reply With Quote
  #3   (View Single Post)  
Old 3rd August 2013
ocicat ocicat is offline
Administrator
 
Join Date: Apr 2008
Posts: 2,902
Thanked 190 Times in 160 Posts
Default

Quote:
Originally Posted by daemonfowl View Post
Code:
wget: memory exhausted
I have 2giga of ram + 4giga swap + I was on fvwm running only wget.

Is there any way to prevent this from happening -other than upgrading ram- ?
The amount of RAM you currently have installed is not the problem. What you need to be focusing on is the amount of memory allocated to the active user. This is controlled at login by login.conf(5). You can see what settings are currently in place by issuing the following command:

$ user info xyzzy

...where the active username is "xyzzy". Your username will most likely be different.

Note that the login.conf(5) manpage will indicate that modifications can be done to class "staff". If your active user is not in class "staff" (which can be determined by the user(8) command given above...), use chpass(1) or vipw(8) as root to do so.

Last edited by ocicat; 3rd August 2013 at 05:07 AM. Reason: clarity
Reply With Quote
  #4   (View Single Post)  
Old 3rd August 2013
daemonfowl daemonfowl is offline
bsdstudent
 
Join Date: Jan 2012
Location: DaemonLand
Posts: 834
Thanked 0 Times in 0 Posts
Default

Thanks ocicat
'user info' doesn't show relevent info
I don't have a ~/login.conf and /etc/login.conf is still virgin .. should I set memoryuse capability in ~/login.conf or /etc/login.conf per user ??
Reply With Quote
  #5   (View Single Post)  
Old 3rd August 2013
daemonfowl daemonfowl is offline
bsdstudent
 
Join Date: Jan 2012
Location: DaemonLand
Posts: 834
Thanked 0 Times in 0 Posts
Default

user info me
Code:
login	me
passwd	*
uid	1000
groups	me wheel
change	NEVER
class	staff
gecos	me
dir	/home/me
shell	/bin/sh
expire	NEVER
Reply With Quote
  #6   (View Single Post)  
Old 4th August 2013
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 3,797
Thanked 214 Times in 189 Posts
Default

Quote:
Originally Posted by daemonfowl View Post
I don't have a ~/login.conf and /etc/login.conf is still virgin .. should I set memoryuse capability in ~/login.conf or /etc/login.conf per user ??
It appears you misunderstand the purpose of login.conf. It is used to govern authentication methods and resource allowance, and therefore is only centrally managed. The OS does not give individual users the right to determine their own authentication methods and resource access.

Authentication and resource are assigned by CLASS. Classes are defined in /etc/login.conf, and assigned in /etc/master.passwd. The login.conf delivered with OpenBSD defines these classes: default, daemon, staff, authpf, and bgpd. You may revise these classes or define your own.

--

On OpenBSD/i386, memory mapping has a limited address space per process. As ocicat pointed out, your physical memory (RAM) does not have anything to do with process memory mapping -- process memory allocation is virtual.

On i386, the staff class resources are defined as follows:
Code:
staff:\
    :datasize-cur=512M:\
    :datasize-max=infinity:\
    :maxproc-max=512:\
    :maxproc-cur=128:\
In this case, "infinity" is not infinite. It is 2GB, if I recall correctly. Larger memory mapping is only possible in other architectures, such as sparc64 and amd64.

You may check your limits, or alter them (up to the appropriate class values in login.conf) via an internal shell command. The internal shell commands are different, depending on your shell. To check limits on Bourne shells, use $ ulimit -a and on C shells use % limit -- you will find instructions on how to use the limit and ulimit commands in the appropriate shell man page.

Last edited by jggimi; 4th August 2013 at 03:06 AM. Reason: clarity
Reply With Quote
  #7   (View Single Post)  
Old 4th August 2013
ocicat ocicat is offline
Administrator
 
Join Date: Apr 2008
Posts: 2,902
Thanked 190 Times in 160 Posts
Default

Two recent threads on ports@ & openbsd-newbies@ may be of interest to readers of this thread:In particular, both discuss crashes in Firefox, but discussion is applicable to any user application which attempts to consume more memory than what is allocated to a login class.
Reply With Quote
  #8   (View Single Post)  
Old 8th August 2013
daemonfowl daemonfowl is offline
bsdstudent
 
Join Date: Jan 2012
Location: DaemonLand
Posts: 834
Thanked 0 Times in 0 Posts
Default

Thanks.
I have ulimit -c 0 in ~/.profile .. could it be the cause ?
Reply With Quote
  #9   (View Single Post)  
Old 8th August 2013
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 3,797
Thanked 214 Times in 189 Posts
Default

No.
Reply With Quote
Old 8th August 2013
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 3,797
Thanked 214 Times in 189 Posts
Default

I have time to expand on my "No", above.

Previously, you asked how to eliminate the production of core(5) files, as you had no interest in doing your own debugging of applications. Alphalutra1 gave you an example ulimit shell command, which you dutifully copied and pasted into your $HOME/.profile script.

Whether or not a .core file is produced after an application fails has nothing to do with the application's use of addressable virtual memory. As I stated previously, each process in OpenBSD/i386 has a 4GB address space, with 2GB of that available to the process for application and data. That's all any individual process gets to use.

Once you have taken the time to confirm you are using that 2GB maximum datasize, rather than the 512M default starting value for the "staff" class, if you still have application failures you have three choices:
  1. Use an architecture with a larger process address space.
  2. Find and use another application or OS to meet your needs, that uses address space(s) differently.
  3. Lobby the Gnu Project and convince them to redesign the application to meet your needs.

Last edited by jggimi; 8th August 2013 at 05:46 PM. Reason: typo, clarity, and correction of 4GB (32 bit) address space with 2GB datasize.
Reply With Quote
Old 9th August 2013
daemonfowl daemonfowl is offline
bsdstudent
 
Join Date: Jan 2012
Location: DaemonLand
Posts: 834
Thanked 0 Times in 0 Posts
Default

ulimit -a
Code:
time(cpu-seconds)    unlimited
file(blocks)         unlimited
coredump(blocks)     0
data(kbytes)         524288
stack(kbytes)        4096
lockedmem(kbytes)    682913
memory(kbytes)       2045756
nofiles(descriptors) 512
processes            128
I still have application failures (mplayer , firefox , wget , httrack ...) , nevertheless I'd never consider option 2 -as I am deeply in love with Puffy- nor can I afford option 3

Thank you Teacher jggimi !
Reply With Quote
Old 9th August 2013
IdOp's Avatar
IdOp IdOp is offline
Too dumb for a smartphone
 
Join Date: May 2008
Location: twisting on the daemon's fork(2)
Posts: 572
Thanked 14 Times in 13 Posts
Default

Quote:
Originally Posted by daemonfowl View Post
I still have application failures (mplayer , firefox , wget , httrack ...)
Similar problem but on a different OS: I was having trouble with emacs on NetBSD, and after some googling found that setting the stack ( -s flag ) to 8192 (from 2048) did the trick. Unfortunately firefox and seamonkey weren't happy with that. Anyway, you could try playing with -s and see if it helps. It's easy to do, but not guaranteed to work.
Reply With Quote
Old 9th August 2013
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 3,797
Thanked 214 Times in 189 Posts
Default

Let me try again, and I will try to be as clear as I can.

I showed you an excerpt from /etc/login.conf for i386. There are two lines that mention values for datasize. Please note, one says "cur" for "current", the other says "max" for "maximum":
Quote:
Originally Posted by jggimi View Post
Code:
staff:\
    :datasize-cur=512M:\
    :datasize-max=infinity:\
I also wrote:
Quote:
Originally Posted by jggimi View Post
...confirm you are using that 2GB maximum datasize, rather than the 512M default starting value for the "staff" class...
Your datasize is still only 512M, because you did not raise it to the 2GB maximum:
Quote:
Originally Posted by daemonfowl View Post
ulimit -a
Code:
time(cpu-seconds)    unlimited
file(blocks)         unlimited
coredump(blocks)     0
data(kbytes)         524288
stack(kbytes)        4096
lockedmem(kbytes)    682913
memory(kbytes)       2045756
nofiles(descriptors) 512
processes            128
Here is a "howto" to raise your datasize to the 2GB maximum.

I have a user in the staff class using the ksh(1) shell, and I set the datasize using ulimit -d to 2GB. I then try to add one more MB and receive a failure notice.
Quote:
$ ulimit -a
time(cpu-seconds) unlimited
file(blocks) unlimited
coredump(blocks) unlimited
data(kbytes) 524288
stack(kbytes) 4096
lockedmem(kbytes) 336781
memory(kbytes) 998184
nofiles(descriptors) 1024
processes 128
$ ulimit -d 2048*1024
$ ulimit -d 2049*1024
/bin/ksh: ulimit: exceeds allowable limit
$ ulimit -a
time(cpu-seconds) unlimited
file(blocks) unlimited
coredump(blocks) unlimited
data(kbytes) 2097152
stack(kbytes) 4096
lockedmem(kbytes) 336781
memory(kbytes) 998184
nofiles(descriptors) 1024
processes 128
$

Last edited by jggimi; 9th August 2013 at 12:14 PM. Reason: typo. Later, added highlighting to the "howto"
Reply With Quote
Old 9th August 2013
daemonfowl daemonfowl is offline
bsdstudent
 
Join Date: Jan 2012
Location: DaemonLand
Posts: 834
Thanked 0 Times in 0 Posts
Default

Thank you very much jggimi ! now it's better .. I've set ulimit in my .profile .. what if I want to set it for all users ? some way other than editing their respective .profiles ?

Hi dop ! Thank you for the tip .. I'll remember to use it when on NetBSD.
Reply With Quote
Old 9th August 2013
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 3,797
Thanked 214 Times in 189 Posts
Default

Quote:
Originally Posted by daemonfowl View Post
.. what if I want to set it for all users ? some way other than editing their respective .profiles ?
You can edit /etc/login.conf, as I mentioned above, where I wrote:
Quote:
Originally Posted by jggimi View Post
...The login.conf delivered with OpenBSD defines these classes: default, daemon, staff, authpf, and bgpd. You may revise these classes or define your own.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
why do some websites refuse wget -r -m -k --no-parent ? daemonfowl General software and network 0 10th June 2013 05:18 PM
hello, how BSD reboot but keep memory misssir FreeBSD General 8 11th October 2011 02:43 PM
Memory limit jaax NetBSD General 2 19th February 2011 11:21 AM
World shrugs as IPv4 addresses finally exhausted J65nko News 0 2nd February 2011 10:23 PM
wget Nk2Network OpenBSD Packages and Ports 3 27th February 2009 01:40 PM


All times are GMT. The time now is 06:06 PM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick