5.7 system freeze - reproducible

[hanzer]

I came across something curious. On this machine, httpd hosts two Tor (/usr/ports/net/tor) "hidden service" web sites - both are simple, single page HTML text; one has an animated gif of a printer printing a test page...

Here's the interesting part, when running Torshammer (A Slow POST Denial Of Service Testing Tool) against the server, pointed at the [hidden] site hosting the simple page with the gif image, the computer freezes about a minute into it. This happens every time. None of the individual pieces seem to have a problem - they all work fine separately.

Assuming the problem has to do with the DoS attack on httpd, it's *very* surprising that it causes the system to freeze (or, as they say on the TV series Silicon Valley - the computer "shit the bed" (a company "cakes its pants" (pop culture is fascinating))).

[jggimi]

There is a recent thread on misc@ regarding httpd buffer exhaustion, which your DoS testing may be incurring. It affects both -stable and -current. From the thread, it has appeared on amd64, but i386 may also be susceptible.

http://marc.info/?t=143614704800002&r=1&w=2

Even if yours is not the same problem, -dvvv may tell you what is occurring.

[jggimi]

Additionally, and perhaps more importantly, net/tor received a -stable update on 9 April 2015, which you might not be using. See revision 1.75.2.2 in the CVS web portal log for ports/net/tor/Makefile.

According to the Tor Project link found in the -stable update log entry, this update affects reliability and availability for hidden services.

If you have not deployed this -stable port, either build the port from a -stable ports branch, or use the -stable package update services offered by mtier.org.

[hanzer]

Thanks, jggimi! You're like a well organized army of on-line technical support. (And you have the chomps to speculate like a engineer (or a consultant to developers)).

On the enthusiastic explorer side of things, I would have guessed/hoped that there would have been some process [*resource* & access] isolation on such a server {httpd}.

Would systrace possibly have the capabilities to enforce resource access and restrict a process (like httpd) in such a way that it can't render the system into a state of guru meditation? ( systrace takes a security perspective ).

[jggimi]

Systrace will kill a process that violates its ruleset, unless it is run interactively. But those rules are by syscall, not by resource consumption limits. That's what login.conf and various ulimits are for. I think it would be the wrong tool. As you like analogies, lets say I believe it may be the wrong hammer for this particular window glazing task.

If you believe httpd is at fault, try a different webserver. We've got lots in the ports tree.

If you believe the fault is net/tor, try the -stable package, if you haven't already.

It could just be your stress test is too stressful for your i386 machine.

[hanzer]

Originally Posted by jggimi

Systrace will kill a process that violates its ruleset

Linux cgroups and FreeBSD sysjails are interesting to think about and that leads to [architectural] thoughts of ChromeOS, coreOS, Docker, and Kubernetes (fun stuff to speculate about), but I think some kind of containerization at the process level might be nifty on OpenBSD - process containment that could communicate to the running app/proc the current resource usage and other events so the process could throttle itself rather than trample the system or crash the system or compromise the system. Of course, some recalcitrant processes will need to be shot in the head.

Originally Posted by jggimi

It could just be your stress test is too stressful for your i386 machine.

I don't think is was a stress issue in the sense of resource exhaustion. CPU and Memory were both reasonably low - far less load than compiling software, e.g., from /usr/ports.

[hanzer]

Originally Posted by jggimi

Additionally, and perhaps more importantly, net/tor received a -stable update on 9 April 2015, which you might not be using. See revision 1.75.2.2 in the CVS web portal log for ports/net/tor/Makefile.

It's a fresh 5.7 installation and update[-stable] (last couple of days).

hanzer[/usr/ports/net/tor] $ cat distinfo                                                                                                       
SHA256 (tor-0.2.5.12.tar.gz) = VQ/a//60weMDW7jMQubknVrxeteVY70RivIsEQf3K0k=
SIZE (tor-0.2.5.12.tar.gz) = 3311423
hanzer[/usr/ports/net/tor] $ tor --version
Tor version 0.2.5.12 (git-99d0579ff5e0349f).

Originally Posted by jggimi

According to the Tor Project link found in the -stable update log entry, this update affects reliability and availability for hidden services.

If you have not deployed this -stable port, either build the port from a -stable ports branch, or use the -stable package update services offered by mtier.org.

I'm having a look at Tor on OpenBSD (and Erlang on OpenBSD).

[jggimi]

OK, then you have the -stable package deployed. If the problem continues, you can try httpd(8) with -dvvv as mentioned in my first reply above, to see if you receive debug messages indicating that you to are having the resource issues mentioned on misc@, or, you can try an alternate webserver if you believe httpd(8) is the root cause of the problem. Or both.

[hanzer]

Originally Posted by jggimi

OK, then you have the -stable package deployed. If the problem continues, you can try httpd(8) with -dvvv as mentioned in my first reply above, to see if you receive debug messages indicating that you to are having the resource issues mentioned on misc@, or, you can try an alternate webserver if you believe httpd(8) is the root cause of the problem. Or both.

It would be interesting to locate the point of failure and I'll probably pursue that to some extent (somewhat casually). As part of the Erlang research & exploration, I might set up Yaws and stress test it - to see an Erlang system under load (on OpenBSD). Might as well hook up Yaws to Tor and throw the Slow POST DoS attack at it.

[hanzer]

Forked the crash problem to the thread: "Tor-0.2.5.12 can crash OpenBSD-5.7-stable"