Tor-0.2.5.12 can crash OpenBSD-5.7-stable

[hanzer]

Continued from thread: "5.7 system freeze - reproducible"

Running Torshammer against a Tor hidden service hosted locally by httpd crashes OpenBSD-5.7-stable (machine).

Collecting information is tricky because it really crashes - the machine has to be shutdown by disconnecting power and the filesystems are left in a messy state. So, difficult to experiment...

The good news is that it doesn't seem to be a problem with httpd. Running the slow post attack directly against httpd without going through the Tor network does not result in a crash. The system load increases but it chugs along well enough.

However, when the layer 7 slow post attack is routed through the Tor network at a web site set up as a Tor hidden service, the OS freezes. (all on the same box, load is not very high - python (Torshammer), httpd, Tor, all together don't use more than 50% CPU or memory - the machine is not thrashing)

I am tempted to tweak /etc/login.conf and create addition login classes for _tor and www but I'm not sure if this would help, i.e., I'm not sure how Tor is crashing the system. Actually, I'm surprised it's even possible.

The daemon login class is:

daemon:\
        :ignorenologin:\
        :datasize=infinity:\
        :maxproc=infinity:\
        :openfiles-cur=128:\
        :stacksize-cur=8M:\
        :localcipher=blowfish,9:\
        :tc=default:

Memory isn't being exhausted (as far as I can tell) and I don't see more than one _tor process.

Any ideas on how to proceed?

[jggimi]

A kernel hang requires the use of a suite of debugging tools. Get set for a long learning curve.

I'd start with ddb(4). If the kernel is still responding to interrupts at the time the OS stops functioning, your system under test should be able to enter ddb as described in that man page. It's a debugger that includes trace tools, can set breakpoints, etc.

(This assumes, of course, that your system under test is not running an X server. You shouldn't normally need to, when conducting this sort of testing, and X would limit your ability to debug a kernel issue, if you have one.)

If the kernel does not respond, you may be able to run gdb(1) traces on the failing system as it fails, via serial connection to a second platform. Instructions can be found in crash(8) for live system monitoring with gdb(1) and then in info(1) gdb documentation for remote debugging.

Luckily (for me), I've never had to do this. Unlucky for you, because I don't have any guidance. I don't even have a clue how to go about it, because while I've muddled through with info(1) once or twice on an emergency basis, I've never invested the three hours I would need to learn to use the GNU info(1) command properly. So I can't even read GNU docs to provide any suggestions, or help with any problems that arise.

Kernel services (syscalls) requested by userland processes can be monitored with ktrace(1), and reports produced with kdump(1).

There are the system monitoring tools that may indicate resource consumption trends: top(1), vmstat(1), systat(1), iostat(1). While the system may hang, having these running in tmux(1) windows on the console may permit you to monitor the failure from different reporting points of view, and perhaps, learn what resource has been exhausted, if that is indeed what is happening. Each of these can be set to report at frequent intervals. If I recall correctly one second is the minimum.

[hanzer]

How well could the file-system be prepared to [both] survive crashes and get data onto the disk right up to the crash point (or near enough (hopefully))?

The system could potentially be instrumented and, while recording, the crash could be induced (it's *very* reproducible, like an OpenBSD remote kill button (bzzzt - dead)).

[jggimi]

Disable soft updates (mount option softdep, a.k.a soft dependencies), if you are using that option -- soft dependencies defer write operations.

Refer to FAQ 14.6 and the Ganger/Patt paper referenced within.

[jggimi]

And, while removing softdep as an option -- which makes metadata updates synchronous, add the sync option. This will force writes of file data to be performed synchronously, also.

Yes, disk performance will then be very slow. But this should result in fewer sectors pending in RAM at the point when the system ceases to function.

[hanzer]

Same behavior with Tor version 0.2.6.10 - a layer 7 slow post attack results in a kernel hang within a minute or two of the beginning of the attack. Has anyone tried (successfully or otherwise) to duplicate this behavior?

I could be off here but it seems like there might be people in the world whose lifestyle, livelihood, or life might depend on this (Tor) software. Since OpenBSD is tauted as a security centric OS, some people might select it for their privacy critical servers. If there isn't something peculiar about my particular setup, if Tor+OpenBSD is vulnerable to this attack then I guess the alarms should sound so people can protect themselves.

I'll probably keep tinkering with it but my setup has limitations.

If anyone out there makes an attempt to test Tor+OpenBSD under a Torshammer attack, I would really like to hear about it!

[ibara]

You might have more luck on the tor-bsd list: tor-bsd@lists.nycbug.org
At least there I know there are people that do use tor heavily on OpenBSD.

[hanzer]

Good news, I don't think the kernel hang is due to Tor (or httpd).

I borrowed a laptop from a friend and booted Tails from a USB flash drive then ran the Torshammer attack against the Tor hidden service running on the OpenBSD machine. No crash! System load stayed below 10% and it even continued to serve pages, albeit, with some lag.

My current guess is the wpi firmware, under the load of the Tor/hidden-service + Tor/spDoS traffic is what caused the kernel to hang.

[jggimi]

Don't guess. Do a root cause analysis.

Are you able to enter ddb? If so, you can backtrace to determine what the kernel is busy doing.

If not ... did you turn off softdep, turn on sync, set up a console with a slew of tmux windows logging (see the tmux pipe-pane option, there's an example in the portslogger(8) man page) and recreate the problem?

One common cause for a "hung system" is running out of swap space. If a userland process cannot allocate memory, the process will fail. But, if the kernel cannot allocate memory, the OS will not continue to function.

[hanzer]

Originally Posted by jggimi

Don't guess. Do a root cause analysis.

Are you able to enter ddb? If so, you can backtrace to determine what the kernel is busy doing.

If not ... did you turn off softdep, turn on sync, set up a console with a slew of tmux windows logging (see the tmux pipe-pane option, there's an example in the portslogger(8) man page) and recreate the problem?

One common cause for a "hung system" is running out of swap space. If a userland process cannot allocate memory, the process will fail. But, if the kernel cannot allocate memory, the OS will not continue to function.

I'm in the process of rebuilding the machine at the moment. It couldn't boot after a crash earlier today. The filesystems all had 'sync, noatime' added to the default mount settings (it had been running that way for a day or so, since post #5).

I had a go at Tmux Logging and the Tmux Plugin Manager in an attempt to generate nice data from top and systat reports, but the log files were too ugly to deal with (not simple ascii). So, to move forward with the data collection process I just used vmstat and iostat and piped their outputs to files; launched Torshammer; the kernel hang occurred as expected; then power cycled the machine but it wouldn't get past:

acpi0: sleep states S0 S3 S4 S5

Since this is my Internet gateway machine, I am becoming a bit gun-shy about the 'let it crash' method of experimentation.

[jggimi]

In your other thread, I'd recommended this misc@ discussion regarding httpd hangs. The group testing and debugging included a user who proposed a patch to TLS handling in httpd(8), which was accepted with minor differences. That patch was committed to -current yesterday, though the problem reported affected -stable also.

I don't know if your system under test includes HTTPS, which uses TLS. If it does, this fix may produce different results for you.

There was also a -stable commit for improper handling of 0-byte TCP packets on the 13th. You'd posted a dmesg of a -stable kernel built on the 10th. If your system under test need to manage 0-byte TCP packets, this fix may produce different results for you.

---

I don't know if either of these would have any impact. As with most problems, there's more about them I don't know than I do.

[hanzer]

Scenario 1 - kernel hangs


Scenario 2 - system chugs along


Scenario 3 - system chugs along


My current guess is the kernel hang is due to issues in the wpi driver and/or firmware when under heavy load of this type. A simple way to test that guess is to recreate Scenario 1 on a machine with a different NIC. Unfortunately, I'm not suitably equipped (hardware) to run that test but I would happily talk someone through it...