DaemonForums  

Go Back   DaemonForums > FreeBSD > FreeBSD Ports and Packages

FreeBSD Ports and Packages Installation and upgrading of ports and packages on FreeBSD.

Reply
 
Thread Tools Display Modes
  #1   (View Single Post)  
Old 8th October 2008
MetalHead MetalHead is offline
Port Guard
 
Join Date: Sep 2008
Posts: 24
Default Compile ports in parallel?

Hi!

If running dual or quad CPU's - is it possible to compile ports in parallel?

TIA
Reply With Quote
  #2   (View Single Post)  
Old 8th October 2008
vermaden's Avatar
vermaden vermaden is offline
Administrator
 
Join Date: Apr 2008
Location: pl_PL.lodz
Posts: 1,056
Default

There are some patches for that but I do not remember where, google it out, wait for other members speak, or search http://forums.bsdnexus.com for answer, I remember it has also been talked there mate.
__________________
religions, worst damnation of mankind
"If 386BSD had been available when I started on Linux, Linux would probably never had happened." Linus Torvalds

Linux is not UNIX! Face it! It is not an insult. It is fact: GNU is a recursive acronym for “GNU's Not UNIX”.
vermaden's: links resources deviantart spreadbsd
Reply With Quote
  #3   (View Single Post)  
Old 8th October 2008
graudeejs's Avatar
graudeejs graudeejs is offline
Real Name: Aldis Berjoza
ISO Quartermaster
 
Join Date: Jul 2008
Location: Riga, Latvia
Posts: 589
Default

compiling ports in parallel can result in failure because of dependencies.... (well at least for me it did fail)

i used -j witch
http://www2.lv.freebsd.org/doc/en_US...tml#Q24.4.14.5.

While it did work for rebuilding world, it failed on ports
(I did it for purpose of experiment)
Reply With Quote
  #4   (View Single Post)  
Old 9th October 2008
mdh's Avatar
mdh mdh is offline
Real Name: Matt D. Harris
FreeBSD 2.2.6 User
 
Join Date: Oct 2008
Location: West Virginia
Posts: 139
Default

iirc, make doesn't pass the -j option down to sub-make processes, so it wouldn't really be useful for ports compilation (since the actual port Makefile doesn't do any compiling itself, it unarchives the distfile into work/ and then runs a sub-make in the dist directory to actually do the build.)

One of these days, I'd rather like to see a make.conf option or somesuch to have a default number of compiler instances so that -j wouldn't even be needed, as make would be parsing make.conf each time it runs regardless.
Reply With Quote
  #5   (View Single Post)  
Old 9th October 2008
MetalHead MetalHead is offline
Port Guard
 
Join Date: Sep 2008
Posts: 24
Default

What is the point of ccache?
Anyone had any luck with this program?

ccache is a compiler cache. It acts as a caching pre-processor to C/C++
compilers, using the -E compiler switch and a hash to detect when a
compilation can be satisfied from cache. This often results in a 5 to 10
times speedup in common compilations.



To use ccache add the following to /etc/make.conf

.if (!empty(.CURDIR:M/usr/src*) || !empty(.CURDIR:M/usr/obj*)) && !defined(NOCCACHE)
CC=/usr/local/libexec/ccache/world-cc
CXX=/usr/local/libexec/ccache/world-c++
.endif

For Korn/Bourne shells Add the following to /etc/profile:
export PATH=/usr/local/libexec/ccache:$PATH
export CCACHE_PATH=/usr/bin:/usr/local/bin

For csh/tcsh Add the following to /etc/csh.cshrc:
setenv PATH /usr/local/libexec/ccache:$PATH
setenv CCACHE_PATH /usr/bin:/usr/local/bin

For icc users:
Add /usr/local/intel_cc_80/bin to CCACHE_PATH

--

To use distcc:

For Korn/Bourne shells Add the following to /etc/profile:
export CCACHE_PREFIX=distcc
export DISTCC_HOSTS="localhost host1 host2"

For csh/tcsh Add the following to /etc/csh.cshrc:
setenv CCACHE_PREFIX distcc
setenv DISTCC_HOSTS "localhost host1 host2"

--

If you have a problem building world
define NOCCACHE and try again.

If you have a problem building a port
reset PATH=$CCACHE_PATH and try again.

--

Ccache has installed links for the following compilers
cc c++ gcc g++ gcc32 gcc33 gcc34 gcc40 gcc41 gcc42 gcc43 gcc44 gcc-ooo g++32 g++33 g++34 g++40 g++41 g++42 g++43 g++44 g++-ooo
in /usr/local/libexec/ccache

!!! WARNING !!!
Please make sure you have the compiler installed before you do this or your
build will fail because it will not be able to find the compiler.

Any time you change CC/CXX you need to reinstall devel/libtool15 or you
will run in to problems.

--

Ccache's default cache size is 1GB and this is larger than the default size of
the root partition (/) on FreeBSD. To over come this you may specify a new
cache location on a different partition with the following examples.

For Korn/Bourne shells Add the following to /etc/profile:
export CCACHE_DIR=/usr/.ccache

For csh/tcsh Add the following to /etc/csh.cshrc
setenv CCACHE_DIR "/usr/.ccache"
Reply With Quote
  #6   (View Single Post)  
Old 9th October 2008
MetalHead MetalHead is offline
Port Guard
 
Join Date: Sep 2008
Posts: 24
Default

Quote:
Originally Posted by mdh View Post
iirc, make doesn't pass the -j option down to sub-make processes, so it wouldn't really be useful for ports compilation (since the actual port Makefile doesn't do any compiling itself, it unarchives the distfile into work/ and then runs a sub-make in the dist directory to actually do the build.)

One of these days, I'd rather like to see a make.conf option or somesuch to have a default number of compiler instances so that -j wouldn't even be needed, as make would be parsing make.conf each time it runs regardless.
Exactly.
I'm coming from Gentoo to FreeBSD. I like the fact that on Gentoo I
could set -j x the number of CPU's and have (X) numbers of
CPU's compile source programs.
If you have 2 or 4 cpu's than put them to use. I don't like the fact
that only ONE cpu is being used when I have three others that are
setting idle when compiling ports.

I have no problems building world using make -j12 buildworld etc. Just
think it would be nice if we could put those cpu's to use when
compiling software from ports.
Reply With Quote
  #7   (View Single Post)  
Old 9th October 2008
vermaden's Avatar
vermaden vermaden is offline
Administrator
 
Join Date: Apr 2008
Location: pl_PL.lodz
Posts: 1,056
Default

@MetalHead;15533

I created a ccache HOWTO long time ago here
http://daemonforums.org/showthread.php?t=124

Add your part of distcc as the comment to that thread.
__________________
religions, worst damnation of mankind
"If 386BSD had been available when I started on Linux, Linux would probably never had happened." Linus Torvalds

Linux is not UNIX! Face it! It is not an insult. It is fact: GNU is a recursive acronym for “GNU's Not UNIX”.
vermaden's: links resources deviantart spreadbsd
Reply With Quote
  #8   (View Single Post)  
Old 9th October 2008
MetalHead MetalHead is offline
Port Guard
 
Join Date: Sep 2008
Posts: 24
Default

Quote:
Originally Posted by vermaden View Post
@MetalHead;15533

I created a ccache HOWTO long time ago here
http://daemonforums.org/showthread.php?t=124

Add your part of distcc as the comment to that thread.
Nice..Thanks for putting that info up...
Reply With Quote
  #9   (View Single Post)  
Old 9th October 2008
mousaka mousaka is offline
New User
 
Join Date: May 2008
Location: Switzerland
Posts: 4
Default

Have a look at buildflags from bsdadminscripts. This lets you more easily define which ports are compile with multiple threads (-j), distcc and/or ccache. Some ports compile with this options where others don't.

There was a SoC-project this year:Allowing for parallel builds in the FreeBSD Ports Collection.
Hopefully we can soon benefit from this work.

mousaka
Reply With Quote
Old 9th October 2008
robbak's Avatar
robbak robbak is offline
Real Name: Robert Backhaus
VPN Cryptographer
 
Join Date: May 2008
Location: North Queensland, Australia
Posts: 366
Default

Many ports code will not build if -j options are used. The code written by the third parties who wrote the software is not -j safe: It might assume that parts of the code that may be built in parallel are finished before others, so it may bomb out. Finding which ports are -j safe is not easy - not the sort of thing that a single build can work out - so it has ot been done.
A better answer is to build two or more non-dependant ports at once - that is what a SOC project will be looking at.
__________________
The only dumb question is a question not asked.
The only dumb answer is an answer not given.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Broken ports in /usr/ports/x11 DNAeon FreeBSD Ports and Packages 3 5th July 2009 08:20 PM
Can't compile autoconf262 Sunsawe FreeBSD Ports and Packages 2 26th August 2008 07:53 PM
Patch and compile bind94 on FreeBSD ports collection dylan912 FreeBSD Ports and Packages 2 4th July 2008 12:23 PM
Parallel to USB cable on FreeBSD? + virtualisation? deadeyes FreeBSD General 4 2nd July 2008 04:58 PM
How come gnome2 won't compile? Damien787 FreeBSD Ports and Packages 10 16th June 2008 05:20 PM


All times are GMT. The time now is 09:36 AM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick