|
Programming C, bash, Python, Perl, PHP, Java, you name it. |
|
Thread Tools | Display Modes |
|
|||
Update httpd.conf IPs from DNS zones.
Hi guys,
Here's what I'm trying to accomplish: I have VirtualHost entries on my httpd.conf that looks similar to the following: Code:
<VirtualHost 301.163.169.211:80> ServerName accounttest.com ServerAlias www.accounttest.com ServerAdmin webmaster@accounttest.com DocumentRoot /home/accountt/public_html <IfModule mod_suphp.c> suPHP_UserGroup accountt accountt </IfModule> <IfModule !mod_disable_suexec.c> User accountt Group accountt </IfModule> BytesLog /usr/local/apache/domlogs/accounttest.com-bytes_log CustomLog /usr/local/apache/domlogs/accounttest.com combined ScriptAlias /cgi-bin/ /home/accountt/public_html/cgi-bin/ # To customize this VirtualHost use an include file at the following locatio n # Include "/usr/local/apache/conf/userdata/accountt/accounttest.com/*.conf" </VirtualHost> Code:
; cPanel 11.24.0-CURRENT_30944 ; Zone file for accounttest.com $TTL 14400 @ 86400 IN SOA ns5.domain.com. cpm.domain.com. ( 2008111802 ; serial, todays date+todays 86400 ; refresh, seconds 7200 ; retry, seconds 3600000 ; expire, seconds 86400 ) ; minimum, seconds accounttest.com. 86400 IN NS ns5.domain.com. accounttest.com. 86400 IN NS ns6.domain.com. accounttest.com. IN A 35.45.68.21 localhost.accounttest.com. IN A 127.0.0.1 accounttest.com. IN MX 0 accounttest.com. mail IN CNAME accounttest.com. www IN CNAME accounttest.com. ftp IN A 35.45.68.21 1) How can I grab the IP from this line: accounttest.com. IN A 35.45.68.21 In the above line, the IP would be 35.45.68.21 (so the $4 variable of the line I guess) 2) Then how can I use the variable $4 from the line above and replace the IP that's being used by the VirtualHost entry of accounttest.com? So the VirtualHost entry would look like this: <VirtualHost 35.45.68.21> ServerName accounttest.com ... ... I'm sorry if this sounds confusing. I kinda have an idea of what needs to be done, but I don't know where to start. Any help will be very appreciated. |
|
|||
To get you started
Code:
$ cat source1 accounttest.com. IN A 35.45.68.21 $ cat source2 gorilla.org. IN A 11.22.33.44 $ cat zone2vhost #!/bin/sh FILE="$1" IP=$(awk '/^[^ ]+ +IN +A +/ { print $4 }' <${FILE} ) name=$(awk '/^[^ ]+ +IN +A +/ { print $1 }' <${FILE} ) NAME=$( echo ${name} | sed -e 's!\.$!!' ) #echo $IP #echo $NAME cat <<END <VirtualHost ${IP} > ServerName ${NAME} ServerAdmin webmaster@${NAME} DocumentRoot /www/docs/${NAME} ErrorLog logs/error_${NAME}.log </VirtualHost> END $ for FILE in source1 source2 ; do zone2vhost ${FILE} ; done <VirtualHost 35.45.68.21 > ServerName accounttest.com ServerAdmin webmaster@accounttest.com DocumentRoot /www/docs/accounttest.com ErrorLog logs/error_accounttest.com.log </VirtualHost> <VirtualHost 11.22.33.44 > ServerName gorilla.org ServerAdmin webmaster@gorilla.org DocumentRoot /www/docs/gorilla.org ErrorLog logs/error_gorilla.org.log </VirtualHost> Actually it would be better to first extract all IN A records into a separate file. After the file has been checked for sanity, then process it.
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump Last edited by J65nko; 18th November 2008 at 09:30 PM. Reason: Hint at problems and possible solution |
|
|||
Quote:
I'm a not an experienced programmer, so here's my question: How can I extract the A record lines from the zone files and add them in to a separate file? This is the line I would like to extract from the zone files: someDomain.com. IN A someIP The first delimiter would be the domain name and the fourth would be whatever IP the domain is using. Then based on that, the script would search httpd.conf for: ServerName\ $1. ($1 would be someDomain.com in this case). Once it finds that, it looks for the line above ServerName which is: VirtualHost 0.0.0.0. Then it would replace 0.0.0.0 with $4 which would be someIP from the dns zone file. |
|
|||
Code:
$ cat extract #!/bin/sh # match IN A record from DNS zone file egrep "^[^[:blank:]]+[[:blank:]]+IN[[:blank:]]+A[[:blank:]]+\ [0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}" $1 # ^ : beginning of line # [ : start of character class # ^[:blank:] : not (space or horizontal tab) # ] : end character class # + : one or more of preeceding atom # One or more non-blanks at beginning # of line = domain name # [ : start of character class # [:blank:]+ : one or more spaces or horiz tabs # ] : end character class # IN : followed by I and an N # [[:blank:]]+ : one or more spaces or horiz tabs # A : A # [[:blank:]]+ : one or more spaces or horiz tabs # # Matching the IP number # # [0-9] : a digit # {1,3} : of the preceding atom at least one, at most 3 # \. : a period '.' # because '.' in a regex pattern stands for 'any character' # we have to use a "\" to tell the regex machine that we # want to match a literal "." # # [0-9]{1,3}\. : second octet of IP number followed by a period # [0-9]{1,3}\. : third octet of IP number followed by a period # [0-9]{1,3} : fourth and last octet of IP number Code:
$ ./extract zone-example accounttest.com. IN A 35.45.68.21 localhost.accounttest.com. IN A 127.0.0.1 ftp IN A 35.45.68.21 Do you keep those zone files in a directory you can do something like what I did in my first post: use the files in a loop calling the extracting script.
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump |
|
|||
Code:
So we already have a slight problem here Houston Do you keep those zone files in a directory you can do something like what I did in my first post: use the files in a loop calling the extracting script. To answer your question: Yes, all of the zone files are kept in a directory called /var/named/ . And all of the zone files has the file extension .db . So how could I run your second script so that it pretty much looks for all the zone files inside /var/named/ (or any other directory for that matter), and replace any VirtualHost IP in httpd.conf, that does not match the ip with the zone file? We're getting close! (Well, YOU are getting close would be the correct term. ) |
|
|||
To produce output to stdout
Code:
for FILE in /var/named/*db ; do ./extract $FILE ; done Code:
for FILE in /var/named/*db ; do ./extract $FILE ; done >all-records.txt Code:
./extract zone-example | egrep -v '(^localhost|^ftp)' accounttest.com. IN A 35.45.68.21
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump |
|
|||
Separate the virtual hosts sections from the main httpd.conf. Put all of them in a single file. Use the "Include" directive to include this virtualhost-only configuration file into the main httpd.conf.
There are two possibilities:
BTW a possible simpler way would be to scan for the virtual host names and use dig to resolve the name to the IP address[/b]. A couple of ServerName entries to test: Code:
$ grep ServerName virtual-sample ServerName accounttest.com ServerName gorilla.org ServerName monkey.org Code:
#!/bin/sh FILE=virtual-sample ALL=$( sed -ne 's/^[[:blank:]]*ServerName *//p' ${FILE} ) for Host in ${ALL} ; do echo "${Host}=$(dig +short -t A ${Host})" done Code:
$ sh dig-them accounttest.com= gorilla.org=128.121.215.221 monkey.org=152.160.49.201
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump |
|
|||
Sorry for the late reply J65nko.
Once again, I thank you for the help you've been giving me. To answer you question: No, I can't just generate the VirtualHost. I need to scan httpd.conf and replace the IP. Like you said, I'll prob have to find someone to help me out with that part. |
|
|||
I actually started working on the script in Perl, although my Perl skills have become a little bit rusty
I already can read in VirtualHost section and store it in an array, ready to be processed.
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump |
|
|||
Quote:
If you're able to come up with such script that would save me a lot of time hehehehe. I understand that you've doing this totally out of your freewill so I'm thankful for the help you have given me. |
|
|||
I have a working script. A snippet from the current output
Code:
Storing: <VirtualHost 11.22.33.44:80> Storing: ServerName accounttest.com Storing: ServerAlias www.accounttest.com Storing: </VirtualHost> Processing: ServerName accounttest.com | (Extracted domain name = [accounttest.com] ) No IP address known for accounttest.com Following section rejected --<VirtualHost 11.22.33.44:80> -- ServerName accounttest.com -- ServerAlias www.accounttest.com --</VirtualHost> === ===# a remark Storing: <VirtualHost 111.222.333.444:80> Storing: ServerName gorilla.org Storing: ServerAlias www.gorilla.org Storing: </VirtualHost> Processing: ServerName gorilla.org | (Extracted domain name = [gorilla.org] ) Processing: <VirtualHost 111.222.333.444:80> ( Extracted IP number: 111.222.333.444 ) Port number = :80 Changing 111.222.333.444 -> 128.121.215.221 Result: <VirtualHost 128.121.215.221:80>
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump |
|
|||
Nice!
I can't wait to test it and see how it works out. |
|
|||
This is what I have now. Not completely as I want it, but didn't want to have you wait any longer
This is the sample virtual hosts configuration file I used: Code:
$cat virt_orig.txt # start of virtual hosts <VirtualHost 11.11.11.11:80> ServerName kpn.com ServerAlias www.kpn.com </VirtualHost> # a remark preceded by an empty line <VirtualHost 22.22.22.22:80> # ServerName fish.org ServerName gorilla.org ServerAlias www.gorilla.org </VirtualHost> # another remark followed by an empty line <VirtualHost 33.33.33.33:8080> # --- home of the monkeys ServerName monkey.org ServerAlias www.monkey.org ServerAdmin webmaster@monkey.org </VirtualHost> <VirtualHost 4.4.4.4:8080> # # Our forum # ServerName daemonforums.org ServerAlias www.daemonforum.org ServerAdmin webmaster@monkey.org </VirtualHost> Code:
$ ./dig-them virt_orig.txt kpn.com=145.7.192.133 gorilla.org=128.121.215.221 monkey.org=152.160.49.201 daemonforums.org=67.205.67.40 The perl script, I called it 'virt-update', will read a file called 'domain-addr.txt' and expects this format. To create this file Code:
$ ./dig-them virt_orig.txt >domain-addr.txt Now you can run the following perl script: Code:
#!/usr/bin/perl -w # $Id: virt-update,v 1.4 2008/11/24 01:47:17 j65nko Exp $ # Copyright (c) 2008 J65nko <Administrator daemonforums.org> # # Permission to use, copy, modify, and/or distribute this software for any # purpose with or without fee is hereby granted, provided that the above # copyright notice and this permission notice appear in all copies. # # THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES # WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF # MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR # ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES # WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN # ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF # OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. # use strict ; my @section ; # <VirtualHost> ... </VirtualHost> section my %address ; # hash with domain -> IP mappings my $debug ; sub parse ; sub process_section ; sub show ; sub get_ServerName ; sub write_section ; sub write_rejects ; sub parse { my $i = -1 ; # first element of array is array[0], and we use ++$i while (<>) { if ( /^\s*<VirtualHost/ .. /^\s*<\/VirtualHost>/ ) { $section[++$i] = $_ ; # store line in array @section #print "Storing: $_" ; if ( /^\s*<\/VirtualHost>/ ) { # complete section stored? process_section ; $#section = 0 ; # empty @section array $i = -1 ; # init index } next ; } #print "\nPassing through unprocessed: $_" ; print RESULT ; # simply pass through non-section lines } # end while } # end sub sub process_section { my $domain ; my $ip ; my $port ; $domain = get_ServerName ; # scan @section for Servername if ( $domain eq "" ) { die 'Panic: No ServerName directive' ; } # we didn't die so there is life after a Servername ... print " (Extracted domain name = [$domain] )" ; if ( not exists $address{$domain} ) { # domain -> addr mapping? print "\nNo IP address known for $domain\n" . "Following section rejected" ; show ; unshift @section, "# NOT UPDATED because of unresolvable domain name\n" ; write_rejects ; # write section to rejected file write_section ; # but still write to output file return ; } # $section[0] is first element of opening '<Virtualhost 1.2.3.4:80>' if ( $section[0] =~ m/^\s*<VirtualHost\s+(\d+\.\d+\.\d+.\d+)(:\d+)?\s*>/ ) { $ip = $1 ; $port = $2 ; if ( $debug == 1 ) { print "\nProcessing: $section[0]" ; print " ( Extracted IP number: $ip )" ; print " Port number = $port " ; } if ( $ip eq $address{$domain} ) { if ( $debug == 1 ) { print "\n======= No change for $domain at $ip ======" ; #show ; #print "\n======= end of unchanged section ==========" ; } write_section ; return ; } else { print "\nChanging $ip -> $address{$domain}" ; $section[0] =~ s/\d+\.\d+\.\d+\.\d+(:\d+)?/$address{$domain}$port/ ; print "\nResult: $section[0]" ; write_section ; } } else { die "Panic: Aborting processing ... No <VirtualHost> entry found!" ; } } # end sub process_section # scan @section array for ServerName directive sub get_ServerName { if ( $debug == 1 ) { print "\nScanning section for ServerName\n" ; } for (@section) { print "$_" ; if ( /^\s*ServerName\s+(\S+)$/ ) { if ($debug == 1 ) { print "Processing : $_" ; } return $1 ; } } } # --- read "domain -> IPaddr" mappings file and store in hash table %address sub get_mapping { my ($domain, $ip ) ; my $file = 'domain-addr.txt' ; open ( FILE, "< $file" ) or die "Cannot open $file" ; while ( <FILE> ) { chomp ; # get rid of LF ($domain, $ip) = split /=/ ; #print "\nDomain: [$domain] IP: [$ip]" ; if ( $ip eq "" ) { print "\nDomain $domain has no IP address. Skipping $domain ..." ; next ; } $address{$domain} = $ip ; } } sub show { for (@section) { print "\n--$_" ; } } sub write_section { for (@section) { print RESULT ; # print implied $_ } } sub write_rejects { for (@section) { print REJECT ; # print implied $_ } } # --- main get_mapping ; # read file 'domain-addr.txt' into %address hash open ( RESULT, "> virt_updated.txt" ) or die "Cannot open updated.txt!" ; open ( REJECT, "> virt_rejects.txt" ) or die "Cannot open updated.txt!" ; # output (only to see synchronized debugging messages) $debug = 1 ; if ( $debug == 1 ) { $| = 1 ; # enable autoflush for synced debugging stuff } parse ; print <<EOF ; ---- Message Updated file : virt_updated.txt A copy of unmodified sections because of unknown IP address : virt_rejects.txt To see the differences run diff -u name_of_original_file virt_updated.txt --- End of message EOF END { close RESULT ; close REJECT ; } The domains for which the 'domain-addr.txt' has found no IP address, the virtual hosts sections will be copied to the file "virt-updated.txt'. An extra copy will be written to the file 'virt_rejects.txt'. So here we go: Code:
$ ./virt-update virt_orig.txt Code:
Scanning section for ServerName <VirtualHost 4.4.4.4:8080> # # Our forum # ServerName daemonforums.org Processing : ServerName daemonforums.org (Extracted domain name = [daemonforums.org] ) Processing: <VirtualHost 4.4.4.4:8080> ( Extracted IP number: 4.4.4.4 ) Port number = :8080 Changing 4.4.4.4 -> 67.205.67.40 Result: <VirtualHost 67.205.67.40:8080> Code:
$ diff -u virt_orig.txt virt_updated.txt --- virt_orig.txt Mon Nov 24 02:41:22 2008 +++ virt_updated.txt Mon Nov 24 03:19:52 2008 @@ -1,25 +1,25 @@ # start of virtual hosts -<VirtualHost 11.11.11.11:80> +<VirtualHost 145.7.192.133:80> ServerName kpn.com ServerAlias www.kpn.com </VirtualHost> # a remark preceded by an empty line -<VirtualHost 22.22.22.22:80> +<VirtualHost 128.121.215.221:80> # ServerName fish.org ServerName gorilla.org ServerAlias www.gorilla.org </VirtualHost> # another remark followed by an empty line -<VirtualHost 33.33.33.33:8080> +<VirtualHost 152.160.49.201:8080> # --- home of the monkeys ServerName monkey.org ServerAlias www.monkey.org ServerAdmin webmaster@monkey.org </VirtualHost> -<VirtualHost 4.4.4.4:8080> +<VirtualHost 67.205.67.40:8080> # # Our forum # Code:
$ ls -l virt*txt -rw-r--r-- 1 j65nko j65nko 680 Nov 24 02:41 virt_orig.txt -rw-r--r-- 1 j65nko j65nko 0 Nov 24 03:19 virt_rejects.txt -rw-r--r-- 1 j65nko j65nko 694 Nov 24 03:19 virt_updated.txt Code:
$ tar xvzf virt-updater.tgz
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump Last edited by J65nko; 24th November 2008 at 02:32 AM. Reason: Attach file |
|
|||
Hi J65nko,
I really appreciate all the help you've given me. Sorry for the late reply, I've been really busy with alot of stuff. Anyways, I'm gonna test your script and I'll let you know how it goes. |
|
|||
As an alternative for the "dig-then" sh script, I wrote a Perl script that does the same thing.
Code:
$cat select-domains #!/usr/bin/perl -w # $Id: select-domains,v 1.3 2008/11/27 01:35:49 j65nko Exp $ # Copyright (c) 2008 J65nko <Administrator daemonforums.org> # # Permission to use, copy, modify, and/or distribute this software for any # purpose with or without fee is hereby granted, provided that the above # copyright notice and this permission notice appear in all copies. # # THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES # WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF # MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR # ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES # WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN # ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF # OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. # use strict ; my $filename = "domain-addr.txt" ; my @domains ; my $IP ; # scan VirtualHost input file for ServerName entries, extract the domain name # and store in array '@domains' print "\n$0: Scanning domain names ....\n" ; while (<>) { if ( /^\s*<VirtualHost/ .. /^\s*<\/VirtualHost>/ ) { if ( /^\s*ServerName\s+(\S+)$/ ) { print "\nDomain: $1" ; push @domains,$1 } } } open ( FILE, "> $filename" ) or die "Cannot open $filename !" ; print "\n\n$0: Starting to resolve domains....\n" ; # for every domain in array, use 'dig' to lookup IP address for ( @domains ) { print "\nResolving: $_ " ; $IP = '' ; $IP = `dig +short -t A $_` ; # Use 'dig' to resolve domain print FILE "$_=" ; if ( $IP =~ m/^(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}).*/ ) { $IP = $1 ; chomp $IP ; print " IP: $IP" ; print FILE "$IP\n" ; } else { print " IP: Not resolved" ; print FILE "\n" ; } } print "\n\n$0: Finished!\n" ; END { close (FILE) or print "Couldn't close file: $filename" ; } # --- end of program A sample run, where again 'virt_orig.txt" is the VirtualHost configuration file Code:
$ select-domains virt_orig.txt | tee logfile ./select-domains: Scanning domain names .... Domain: google.com Domain: ixcdgorilla.org Domain: monkey.org Domain: daemonforums.org ./select-domains: Starting to resolve domains.... Resolving: google.com IP: 209.85.171.100 Resolving: ixcdgorilla.org IP: Not resolved Resolving: monkey.org IP: 152.160.49.201 Resolving: daemonforums.org IP: 67.205.67.40 ./select-domains: Finished! The actual results are in the file "domain-addr.txt": Code:
$ cat domain-addr.txt google.com=209.85.171.100 ixcdgorilla.org= monkey.org=152.160.49.201 daemonforums.org=67.205.67.40
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump |
|
|||
WOW!!
Wow J65nko!!
You have done more than if I was paying programmer to create that script. LOL Thank you once again for help. P.S. Once I get my programming skills up, I plan on helping the community just like you have helped me. |
|
|||
Thanks for the kind words.
I found the problem interesting, and thought it was a nice opportunity to polish up my rusty Perl scripting skills
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump |
Thread Tools | |
Display Modes | |
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
httpd problem or something else | c0mrade | Other BSD and UNIX/UNIX-like | 6 | 15th January 2009 09:19 PM |
Apache : httpd could not be started | lalebarde | General software and network | 13 | 13th November 2008 11:51 PM |
difference between rc.conf and loader.conf | disappearedng | FreeBSD General | 5 | 3rd September 2008 05:54 AM |
httpd -DNOHTTPACCEPT | starbuck | FreeBSD General | 9 | 23rd August 2008 12:14 PM |
httpd.conf | Snoop1990 | General software and network | 5 | 29th July 2008 04:30 AM |