Maintenance free webserver

[Yuka]

Hi all,

I'm looking for tips into building a worry-free FreeBSD webserver that runs with zero maintenance. The goal here is to have it run if I go out of the country for several years, become unavailable, get hit by the proverbial bus, etc.

Hardware fault tolerance isn't an issue because it will be hosted on a quite large and professionally monitored/maintained VMWare ESX cluster. While the group that maintains it will be unable to maintain the FreeBSD box. Furthermore, I don't have to worry about things like money coming in as the box's web content pays for itself to run automatically. So I could potentially die and it should run for at least a good bit of time before they axe my account for not paying and assuming I can make the OS take care of itself indefinitely, it'll run until the website stops making money, the webserver fails irreparably, the datacenter learns that I have died, or the datacenter goes out of business.

My big problem is going to be hard drive space running out through things like log files, etc. I'd prefer to set logging to a minimum and I plan to do so. Or perhaps have it send only critical log entries to a gmail address instead of writing it to disk. Are there any good log file managers out there that can do such a task?

The webserver's content itself won't expand much, maybe a couple hundred megs a year during an extremely busy year. The server currently has 200 GB of storage and it's using about 15 GB.

Things that it will be running are:
apache 2.x
MySQL 5.x
PHP 5.x
postfix (not sure on the version yet)
proftpd

I'm the only administrator of this box. There are a handful of people that use the server for hosting that I provide but they only have access to their fileshare via FTP and their database via phpmyadmin. I'll have to set up a web portal that will allow them to change/reset passwords for both of those accounts. Any recommendations would be great, but coding it myself shouldn't be considerably hard.

Running this box has become my hobby and honestly, I don't intend for it to run after my death as I probably won't care much that it does in such a case, but I forsee myself travelling to places that I won't be able to get in touch with the box to fix problems and such journeys may be for months or even years at a time. I'd like to see how well and stable I can make this work in an effort to take my hobby to the next level.

So really, in doing my research, I'm open to suggestions from anyone that might have built a bulletproof machine in the past. What things should I look for that I might overlook while doing this? The issue of hard drive space consumption seems trivial compared to security threats and hackers that might cause system instability. Any suggestion is a good one.

Right now my list of things to do is as follows:
1. Find a solution to the log file issue. (Maybe even as simple as doing minimal logging and sending the log files to a gmail account daily and clearing the local log files.)
2. Security - A broad subject to cover but still a very important subject.
3. Self-Service for FTP and MySQL.
4. Self-check on the box itself. (Check for processes running and attentive and start if stopped, reboot if fails to start.) I think this will be a huge part to make it repair itself. I can go further by detecting a failure to start then checking validity of critical files to make it run. If invalid, it takes an action depending on which file is invalid.
5. Anything else that can be recommended.

[mdh]

Achieving actual 0 maintenance is not easy, but basically I suppose you'll want to start by looking into some really good shell/perl scripts that run out of crontab to do certain things. Such things might include updating ports for security fixes, or even updating the base system. Still, doing it without any admin interaction will likely, at some point break, and you'll need to come back from vacation to fix it. It may last a month, it may last 10 years, but eventually something big will change to the point that you just need to do something manually.

Logging is easy, and the tools exist to deal with log files (newsyslog, etc).

Security in this sense is going to mean Doing It Right at the beginning, and applying security fixes as they become available. This means automating upgrades, and that's the most likely thing to eventually break down on you, ie if a port name changes, or something about config locations changes in the port, or whatnot.

Self-service stuff can be achieved by using any number of free or commercial products out there, though I'm not personally familiar with any of them.

Self-checking is likewise not a terribly difficult task - nagios on a monitoring server can accomplish this with some scripting, as can cron jobs. Again, coding will likely be required, even if just something simple like Perl.

That was the long answer. It's peppered with stuff like "you'll eventually need to DO something." That leads into the short answer, which is basically that you cannot do what you want without sacrificing security. Programs have bugs. Updating them will eventually require human interaction. Not updating them will lead to sacrificing security.