DaemonForums  

Go Back   DaemonForums > Miscellaneous > Programming

Programming C, bash, Python, Perl, PHP, Java, you name it.

Reply
 
Thread Tools Display Modes
  #1   (View Single Post)  
Old 17th December 2016
Funkygoby Funkygoby is offline
Fdisk Soldier
 
Join Date: Aug 2015
Posts: 57
Default Simple build system

Hello,
I have lilypond files organised as such transciptions/artist/album/[tune_name].ly or compositions/band/[compo_name].ly. I would like a build system inspired on the port system, it has already been discussed here: http://daemonforums.org/showthread.php?t=9861
Based on this thread, make seems to be the best solution.
Currently, I have a Makefiles in every folder. In Vim, I can run make and build the resulting pdf in the same folder.

This worked as long as you have few scores but today I have hundreds of .pdf, .midi hanging around in those folders built from differents lilypond versions (so not visually consistent).

Time for a more convenient system. I would to like be able to:
-build a whole artist discography
Code:
cd artist && make
-build a specific tune
Code:
cd artist/album && make
-have the resulting files stored in a pdfs folder either with a hierarchy or with names such as artist - album - tune.pdf (tune being the original .ly filename)

I have a little experience with Makefiles for C and Java project. My guess is:
-I need a Makefile in every places I want to be able to run make
-I need to have a "mother-Makefile" for generic rules that is included by all other Makefiles

My issues are that I never really understood recursivity with make. In the case of
Code:
cd artist/ && make
how to tell make to enter every subfolder and read the Makefile.
How to handle the resulting files? My Makefiles are always dirty with this because I move files around from the folder where make is executing which, in my case, happens to be the sub-sub-folder.
Do you have any recommendations? Hint?

Extra question:
While writing a thread, how to highlight command names? or ports? I read the forum "README" but couldn't find a thing about this.
Reply With Quote
  #2   (View Single Post)  
Old 17th December 2016
J65nko J65nko is offline
Administrator
 
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 4,125
Default

I have used make(1) for simple web pages and XML/XSLT transformations, and creating post-install shell command files from small shell min-blocks. I never did a recursive Makefile, but I think something like this should do it ....

Code:
cd subdirA && make
cd subdirB && make
cd subdirC && make
Or a loop using a make .for loop:
Code:
.for

.endfor
You can find some examples of this the bsd.mk directory that is included in most OpenBSD makefiles. (I am on a Linux system right now, so I am not sure that it the right directory name ...)

Although the FreeBSD and OpenBSD versions of make(1) have some differences I learned a lot from PMake — A Tutorial by Adam de Boor. His counsel "Variables are your friend" helped me a very much to do what I wanted to achieve.

For your complicated "making" you also could use make or the shell to generate a customized Makefile and then have it run.

For your target files in one place you could use:

Quote:
.OBJDIR
Path to the directory where targets are built. At startup, make searches for an alternate directory to place target files. make tries to chdir(2) into MAKEOBJDIR (or obj if MAKEOBJDIR is not defined), and sets .OBJDIR accordingly. Should that fail, .OBJDIR is set to .CURDIR.
as described in make(1)
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump
Reply With Quote
  #3   (View Single Post)  
Old 18th December 2016
Funkygoby Funkygoby is offline
Fdisk Soldier
 
Join Date: Aug 2015
Posts: 57
Default

Hello J65nko,
mentioning the subdirs explicitly is what I did with C projects. With lilypond scores, I ld rather avoird this. When I create a new artist/album, I want to reuse existing Makefiles without bothering with the differents subdirs.
I will try the for loop (didn't know make had loops).

Reading the ports Makefile should have been my first steps... I will start here + man make and come back with the result.
Reply With Quote
  #4   (View Single Post)  
Old 19th December 2016
J65nko J65nko is offline
Administrator
 
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 4,125
Default

Although this is FreeBSD make, an example of a .for ... .endfor loop can be found at http://daemonforums.org/showthread.php?t=8032#post48207

If you create a Transformation rule your Makefiles will even become more simpler.

Good luck
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump
Reply With Quote
  #5   (View Single Post)  
Old 19th December 2016
ocicat ocicat is offline
Administrator
 
Join Date: Apr 2008
Posts: 3,318
Default

Quote:
Originally Posted by Funkygoby View Post
I would to like be able to:
-build a whole artist discography
...
-build a specific tune
You may want to play with creating different targets within your Makefile's -- target discography create a discography with the command:

$ make discography

...while target tunes performs different functionality via:

$ make tunes
Quote:
When I create a new artist/album, I want to reuse existing Makefiles without bothering with the differents subdirs.
My suggestion is to carefully think through implementing one artist, & then scale up to multiple artists.
Quote:
Reading the ports Makefile should have been my first steps... I will start here + man make and come back with the result.
Looking through some examples of tiered OpenBSD ports may be useful:
  • /usr/ports/devel/jdk
  • /usr/ports/lang/python
...but again, the ports system addresses a more complex set of requirements than what you describe. My suggestion is to start simple & iterate. You will likely not come up with the perfect solution on the first try. Iterating again & again is what programming is all about. Really.
Quote:
While writing a thread, how to highlight command names? or ports? I read the forum "README" but couldn't find a thing about this.
Discussion in the following sticky thread may help:

http://daemonforums.org/showthread.php?t=51

Last edited by ocicat; 20th December 2016 at 01:21 AM. Reason: add clarity
Reply With Quote
  #6   (View Single Post)  
Old 14th March 2017
Funkygoby Funkygoby is offline
Fdisk Soldier
 
Join Date: Aug 2015
Posts: 57
Default

Thank you all for you advices. I have finally put some work into it. I'll show some code (few lines so far) but first, for every one with the patience, here are 2 things that occurred to me since last time.

1. I want to either run make recursively and build every .ly it encouters or target a specific .ly file.
I was naively considering two approaches:
$ make artist/[album/[tune.ly]]
or
$ cd artist/[album] && make [tune.ly]
With the 1st approach I was hoping to have a single centralized makefile at the root folder but this solution seems unrealistic. There is the "-C" switch but it's the same behavior as the 2nd approach ($ cd /folder/ && make)
So I took the 2nd approach (more readable) which then requires a Makefile in every folder.

So far I have a makefile that goes with the .ly files that will build "all" .ly or a specific file. Keep in mind that I am a musician. I love C programming but shell script and makefiles will never make sense to me. They seem so aliens to me.
Code:
all:
	lilypond *.ly

%: %.ly
	lilypond $<
Then I have another makefile that goes in every parent folders (artist and album folders) that runs make in every sub-folders sub-folders:
Code:
SUBDIRS = $(shell ls -d */)

.PHONY: subdirs $(SUBDIRS)

subdirs: $(SUBDIRS)

$(SUBDIRS):
	make -C $@ (or cd $@ && make && cd ..)
This works with my minimalist test tree. I have one concern though:
- Every makefiles will be a clone of one of those 2 "stem makefiles". Should I use hardlinks instead of copies? symlinks break (of course!) if I change folders architecture.
As ocicat mentionned, I want an artist/album template that I can easily copy, adapt (i.e: artist/album/arrangement/) and maintain (evolve the build process)! So far (7 years):
$ $ ls -R | grep .ly | wc
286 388 3610


2. Have a single .OBJDIR (thanks J65nko). My problem is to have every makefiles aware of the .OBJDIR path. I was thinking:
- export the path during login ($ export LILYOUT=/home/funkygoby/somewhere/). That's dirty and easily forgettable while migrating between system)
- hardcode the path is the makefiles. Remember that my folder architecture have heterogeneous depths so a hardcoded relative path (../../output/) will breaks. An absolute path looks dirty too but maybe that is the solution?
- My first idea was to include a Makefile.inc containing some variables ($MAKE, .OBJDIR, etc...) but then the problem is only moved: how do you pass the path to every makefiles?
- Then I dug into the ports tree. In a the archivers/bzip2 Makefile, I found .include <bsd.port.mk>. What's does this means? Why is it .include instead of include? bsd.port.mk is not in the same folder as the Makefile. How can make find it?

Summary:
In older to naively solve my problems, I attend to use 2 "stem makefiles". Every others makefiles are hardlinks to those 2. In order to specify an .OBJDIR, I am planning to hardcode the path in one of the "stem makefiles".
Any inputs?

Last edited by Funkygoby; 14th March 2017 at 05:53 PM. Reason: tags, formatting kind of...
Reply With Quote
  #7   (View Single Post)  
Old 5th July 2017
Funkygoby Funkygoby is offline
Fdisk Soldier
 
Join Date: Aug 2015
Posts: 57
Default

Hello,
After improving, almost finalizing my build system, I hit a brick wall and gave up makefiles. I now use a Python script.

Here is my story:

1. The build system (take 1)
When I started working again on my makefiles (see the old version above), it was to add the capability to have a centralized include folder (for code shared between a lot of sources files: layout description) and output folder (where the compiled pdfs and midi would go).
As mentionned previously, I wanted to avoid copy/pasting the makefiles everywhere and decided to use symlinks instead that would need to be created automatically for convenience. Speaking of automation, that would be the time for cleaning the mess in the source tree.

2. Cleaning
Since Shell script are still alien to me after all those years, I went with Python and managed to craft a script that spotted and removed 7 years of pdfs, midi, ps, hidden vim files, ... accumulated everywhere. It also reorganized the tree, created all the symlinks in the right places and edited the sources files (I know I shouldn't do that) to adapt the include to the new system.
The cleanning process and build tests worked fine on my test tree. Time to try this on the real thing

3. Sadness
The cleaning script worked fine with only minor manual intervention for corner cases but trying to build the whole tree crashed as soon as it tried to build a source file with spaces in name.
Obviously since make(1) uses spaces to separate between target, he will see esa mujer.ly as two different targets esa and mujer.ly. There are workaround but that won't work with wildcard and since I used that a lot (my tree is very dynamic, shape-shifting), I was screwed.

4. Reinvent the wheel
Every steps of the goal I was trying to achieve with make(1) were painful and everything that make makefiles great were useless to me. Wrong tool for the job.
I took a quick look to other build automation systems and finally decided to keep going with scripts and python but to craft everything myself instead.
Two evenings later, I had a working build.py sitting in ~/bin. It can be called from everywhere in the tree, can take several arguments if several file/folder need to be built, it doesn't require to put symlinks everywhere and I understand every line of it. It needs improvements and cleaner comments before sharing it. Anyone interested?

In the end, the problem I was trying to solve seems to be niche enough to require a custom made solution.

Thanks everyone.
Reply With Quote
  #8   (View Single Post)  
Old 25th March 2018
Funkygoby Funkygoby is offline
Fdisk Soldier
 
Join Date: Aug 2015
Posts: 57
Default

Few months later, I decided to learn a bit more about the command find(1). The recursivity aspect of my build system could be solved with it through a "simple" shell. By using
the -exec construct, one could execute a single command on any occurence. Using a simple shell script with some arguments parsing, colored output, input validation was an appealing idea. And no dependencies on python3!

So I translated my python script into shell.
It is my first real use of the shell language (still quite arcane to me) with a lot of comments so that it also serves as a personnal unix shell textbook.
It is almost as long as the python script and it smells like dirty coding.

By editing the hardcoded pathes and build variables, one can reuse this with another compiler like pandoc.
Anyway:
Code:
#!/bin/sh

#TODO why are the variable use as "${VAR}" or "{$VAR}" can't remember
#TODO make a sanitize function that check the Paths (like the python version). Also, if we are at least in the root folder or any subfolder (not anywhere else that would put ../../ in paths)

#Paths
ROOT="/home/funkygoby/documents/musique/partitions"
OUTPUT="$ROOT/output"
INCLUDE="$ROOT/include"
EXCLUDE="$ROOT/exclude"

#Build variables, --output is set at runtime for each file processed because we can't modify OUTPUT from subshell
LILYPOND="lilypond"
OPT="--include=$INCLUDE"
LILYPOND_EXT="ly"

#Scope: toutes les variables sont globales sauf si déclarées local ainsi que $@ $1 etc qui sont propres à la fonction appelée. Quelle bordel.
#Ici $1 désigne le premier argument passé à la fonction (mais pas au script)
#La notation ksh function maFonction () {..} met les variables en scope local automatique mais ça n'a pas l'air très portable.
#Ok mais si la fonction est appelée dans un subshell alors elle hérite de l'env mais ne peut pas modifier les variables globales
buildFile()
{
	echo "\033[01;33mProcessing File\033[01;37m $1\033[00;38m"
	#Extract the interesting part of the path
	tmp="$(dirname "${1#$ROOT}")"
	#Pack it wit OUTPUT (double '/' but it is not important)
	outputfinal="$OUTPUT/$tmp"
	mkdir -p "$outputfinal"
	#Extract the basename, change ly->pdf and add the outputfinal path so that we can compare modification date and find out if the source file is older than the pdf
	pdf="$(basename "$1")"
	pdf="${pdf%."$LILYPOND_EXT"}"
	pdf="$outputfinal/$pdf.pdf"
	#if $1 is newerthan $pdf
	if [ "$1" -nt "$pdf" ]
	then
		$LILYPOND "$OPT" --output="$outputfinal" "$1"
	else
		echo "\033[01;32mAlready up to date\033[01;37m $1\033[00;38m"
	fi
	#$? is the return_status from the last command executed. We need to store it before executing anything else
	local status=$?

	if [ $status -eq 0 ]
	then
		echo "\033[01;32mDone File\033[01;37m $1\033[00;38m"
	else
		echo "\033[01;31mFailed File\033[01;37m $1\033[00;38m"
	fi
	return $status
}

buildDir()
{
#If there is \n in filename, we are screwed but I have never seen one in real life
#Another way on SO:
#export -f buildFile
#find "$1" -name "*.ly" -exec sh -c 'buidFile "$0"' {} \;
#Another way: I could do recursivity, cd into directoy until there isn't subdir anymore then fire buildFile at everything.

#The while read spawn a new shell. It inherit the env but can't modify it. So we add an exit and made sure that it is in the subshell with ()
	echo "\033[01;33mProcessing Dir\033[01;37m $1\033[00;38m"
	status=0
	find "$1" -name "*$LILYPOND_EXT" |
	( while read file
	do
		buildFile "$file"
		#Increment by the exit status of buildFile
		#status scope is bound to this while loop
		status=$(($status+$?))
	done
	exit $status )

	status=$(($status+$?))
	if [ $status -eq 0 ]
	then
		echo "\033[01;32mDone Dir\033[01;37m $1\033[00;38m"
	else
		echo "\033[01;31mFailed Dir $status times\033[01;37m $1\033[00;38m"
	fi
	return $status
}

#Parse arguments. $@ is all arguments but already separated
if [ $# -eq 0 ]
then
	buildDir "$PWD"
fi
for arg in "$@"
do
	#readlink expand link but also ./ ~ and stuff that would mess up outputfinal
	#eval isn't a solution because it would interpret rm and such
	arg="$(readlink -f "$arg")"
	if [ -f "$arg" ]
	then
		buildFile "$arg"
	elif [ -d "$arg" ]
	then
		buildDir "$arg"
	else
		echo "Invalid argument $arg"
	fi
done
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Ninja, a replacement for make, in Chrome's build system J65nko News 7 14th February 2011 04:14 AM
Create MS/XP file system so it will be recognized on a XP system. FBSD Guides 0 1st May 2010 06:49 AM
Simple pf ruleset jhp FreeBSD General 2 30th March 2010 02:05 PM
A simple question Mr-Biscuit Off-Topic 1 16th April 2009 04:26 PM
Which file system use to share data on Bsd system? aleunix Other BSD and UNIX/UNIX-like 2 1st June 2008 04:14 PM


All times are GMT. The time now is 10:13 AM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick