Apache Randomly Stops Working

[plexter]

Hello,

I've been running OpenBSD 4.4-stable for a little while now acting as a Apache webserver.

As far as I know it's been running smoothly for a good 40-50 days since it's last reboot. Today that changed.

I don't have a clue what is going on. I was about to make a change on my site by sftp'ing into the box. Which no longer seems to work at all. Doesn't matter if I have PF enabled or not.

When I ssh it takes quite a while after entering my username. Eeeeventually the password prompt will appear (probably after 5 minutes) and I can login. SSH seems fine after that.

I notice when I did reboot (thinking at the time this would fix it) that a) apache took an abnormally long time to start and b) I originally canceled (CTRL+Z) apache during boot since it took so long and tried starting it manually which also took longer than I was willing to wait.

Apache does start after a good 5-10 minutes during the bootup.

So, does anyone have any idea what could be going on? Anyone have any diagnostic stuff for me to try?

Note: My system has close to a gig of free memory.
HD Used space is minimal

Your help is appreciated. Thanks!

[plexter]

Okay ssh/sftp is no longer an issue. Turns out that was an independent issue.

Apache on the other hand does seem to take a bit to start... but not really that big a deal and thoughts are nice though.

[jggimi]

I assume your ssh/sftp issue was caused by your reverse DNS config (FAQ 8.22).

Which Apache has a performance problem? The built-in audited secured improved Apache 1.3, or the ported, unaudited Apache 2 package?

Anyone have any diagnostic stuff for me to try?

Have you looked in the webserver's error and activity logs? (Hint: for Apache 1.3, look in /var/www/logs)

Have you tried starting it manually after boot completes? (Hint: httpd(8) can be executed by the superuser.)

Have you looked to see what processes were forked after starting it? (Hint: top(1), ps(1), systat(1) are all good choices for examining a running system)

[plexter]

Hi jggimi,

Sorry for the late reply. Thanks for your reply.

I am running Apache 1.3 which came with OpenBSD.

As for logs. I see a lot of, under error log.

[Fri Apr 17 01:47:21 2009] [error] [client 69.196.133.154] File does not exist: "path to some file" 

[Fri Apr 17 01:45:46 2009] [error] ALERT - ASCII-NUL chars not allowed within request variables - dropped variable 'returnpath' (attacker '69.196.133.154', file ' path to some file ')

That kind of bothers me. I'd prefer not to be attacked. :P

Maybe is just me being impatient with the start up. Everything does seem to be running just felt it was a bit slow starting Apache. I'm only initially starting like 3 childs for it.

As far as the "attackers" ; would you have any idea what I could do to help prevent that? Are there any updated lists of "known" bad or proxied ip's. That I could download and install as a table. I have not found anything like that which is remotely current.

Ideally I would like to be blocking (on my firewall) any known proxies...etc and known bad hosts. (yes I know this is not really related sorry)

Oh is there anyway of outputting these logs to my syslog server?

Your thoughts are appreciated. Thanks!

[jggimi]

Welcome to the Internet.

I use block lists for e-mail servers. They are part of a layered "defense in depth" for them. I'm very judicious in my use of them. You may realize, of course, that there are two main problems with "block lists":

1. They are an opinion.

There's an infinte number of block lists, because each is the expression of what the particular list creator/maintainer wants it to be. You'll need to find a list where the opinions of the list maintainers match your requirements. Some may fit, most won't. And each will have limited coverage, so you may need several to fill gaps.

2. They need to be maintained.

Are there ways for addresses to be dropped from the list, or, are they there forever? How do new addresses get added?

And, of course, you need a way to integrate a block list with your web server. I'm sure there's a way to integrate DNS-based ones with a web server, but I've never investigated it.

----

There are four self-described webserver scanners in the security section of the ports tree: security/arirang, security/cgichk, security/nikto, and security/whisker.

Under the www section, there are more security-related tools, such as: www/mod_security, www/ratproxy, www/calamaris, and www/transproxy.

I've never used any of these, so can make no recommendations.

----

I used to use net/snort. Snort is a very popular network analysis tool, and is often used as the main ingredient in an Intrusion Detection System (IDS). It can be integrated with PF, by having it add IPs to blocking tables and then killing state table entries, see the "flexresp" flavor.

[plexter]

Hi jggimi,

I've used mod_security before. Seems pretty decent and figure I will probably implement SNORT sometime in the future.

Would you have any sources that I could download (updated) blacklists from which would be usable in PF? I'm not picky and probably would prefer over blocking than under blocking. I, in theory, could add substitutions if the lists have unwanted stuff. Anything would be better than nothing any how. :P

[jggimi]

When I first started playing with blacklists and whitelists, I set up static tables for PF. But I found, over time, that the DNS-based block lists were much easier to use, as I didn't have to maintain them; someone else had that chore.

But the lists I use today are strictly for SPAM, and would not be applicable to website abuse or attacks.

I have no advice in this area, other than to recommend you Google for web abuse block lists, and research a few that seem to meet your needs.

[BSDfan666]

The problem here is that blacklists are never 100% complete, new malicious systems pop up on the Internet faster then any static list can be updated.

Infected zombie systems are a fact of Internet life, there are workstations with technically ignorant users at the helm.. and servers that were setup eons ago that are no longer maintained.

Instead of trying to block all these evil people, make sure the software you're utilizing is secure and up to date.

That is all, certainly others here will disagree.

[plexter]

Hi BSDfan666,

Yea I know blacklists are not 100% accurate but they would still have a high chance of preventing common stuff from bothering the servers. Secure or not.

Secure is a broad term though. At what point does one deem something secure?


Anyway I have not had much luck with finding the right kind of lists or there from like 2006.

[ocicat]

Originally Posted by plexter

At what point does one deem something secure?

It is a subjective decision based on the perceived worth of the data being protected (How much effort would be required to replace/regenerate the data, or is someone going to come after me legally if said data becomes available to others?), & the cost of all measures used to protect it. It all comes down to risk management.

[BSDfan666]

One other problem in this increasingly scarce IPv4 world, many of the IP's in said blacklist could be in an ISP's dynamic range.. you might be blocking potential users/clients or customers.

That itself may put your job at risk.. some food for thought.

Good luck..

[plexter]

That's true however if the list is well maintained than updating it would/should account for IP changes, but yeah it never would be totally accurate. :P

I do have another thought speaking of Apache Security that I wouldn't mind verifying. If I'm running the Apache that came pre-compiled with OpenBSD. Does the OpenBSD patches account for security vuln...etc that may turn up in the future? Does Apache get updated to other builds? 1.3.1 - 1.3.2 as an example.

OR do I still need to manually maintain this past the standard OBSD patches?

(is there a good "auto update" type solution similar to Mac, Linux, and even Windows?)

Anyway thanks for all your input.

[BSDfan666]

The local version of Apache is a forked version of 1.3.29.. security vulnerabilities are backported from newer versions and are made available in the -STABLE branches, and on the errata pages respectfully.

It is a fork because the OpenBSD developers have preformed their own audits and changes to the code, it is not a vanilla copy of Apache 1.3.29.

As had been reiterated countless times on this forum, OpenBSD requires a certain level of user maintenance.. it is your responsibly to monitor the errata page for bug fixes and to update your copy of the source manually, there is no automatic update functionality, nor should there be one either.

Hope that helps.

[ocicat]

Originally Posted by plexter

OR do I still need to manually maintain this past the standard OBSD patches?

The modified version of Apache 1.3 which ships with OpenBSD is a fork maintained by the OpenBSD developers. Thinking that one can go to the apache.org Website & apply their patches to OpenBSD's hybrid version is a shipwreck waiting to happen. I would not advocate such an approach.

[plexter]

Okay so as long as I follow "http://openbsd.org/errata44.html" I will get any security patches for Apache (that came with OpenBSD).

"there is no automatic update functionality, nor should there be one either" why not?

[BSDfan666]

As long as you follow the relevant errata page for the release that you use, but remember that only the last 2 are supported.

An automatic update feature implies that you want this system to sit a cold dark corner without giving it any attention at all.. this is woefully inadequate, an administrator should carefully monitor the errata page and apply the updates themselves, such updates will require at times recompilation of the entire userland/libraries or the kernel.. actions that require super user privileges, they should not be scripted.

This is my opinion on the mater, and now you have it.. for whatever it's worth.

[marcolino]

Originally Posted by BSDfan666

it is your responsibly to monitor the errata page for bug fixes and to update your copy of the source manually, there is no automatic update functionality, nor should there be one either.

Slightly off-topic, but I'd like to add that this is the philosophy I follow for all systems for which I am responsible. Know what it is you are patching, why it needs to be patched, and what are the ramifications of either patching or leaving alone.

I am refering to Windows systems here, primarily. Even the so-called Critical Updates are not recommended for all systems, and may even cause problems if applied to a working system. However, I'd advise one to do the same for any of their systems.

Bottom line: Know what you are doing before you do it.

[plexter]

Hello all,

Thanks for your input.

Let me re phrase what I was meaning. I don't mean having an application running in the background automatically downloading AND installing patches without prompting...etc. I mean something that will alert you of a new patch and download if requested automatically. Installation of the patch should be done at the discretion of the administrator...etc and never without interaction of some kind or another. I agree.

I find it useful to not have to manually investigate if patches are available, and rather be notified when they come out.

Thoughts?
Thanks!

[jggimi]

From www.openbsd.org/mail.html

announce
Important announcements. This low volume list is excellent for people who just want occasional news about the project.

security-announce
Security announcements. This low volume list receives OpenBSD security advisories and pointers to security patches as they become available.

[BSDfan666]

Those lists are always updated, nor are they very active.. monitoring the errata page is the recommended method, but some people have created unofficial RSS feeds that you could utilize.

The most reliable one would likely be the undeadly.org feed, I cannot however recommend any additional software that can alert you of new erratas.. you're on your own.

It is currently targeting the 4.5 release now, not 4.4.. perhaps someone else knows of a 4.4 feed.