URL logging

[beavers]

I'm looking to set up some kind of mechanism to log all of the URLs that go over my home web connection (and probably do some blocking as well). I had initially considered using a squid proxy, but I recently discovered that relayd can do this! Is this the best route to take, or is there something better to be using?

[jggimi]

It comes as part of OpenBSD, so there is nothing to install. But note that relayd(8) uses syslog(3) for logging. If you want to record those logs, you will likely want to provision syslog.conf(5) to isolate the messages. See http://openbsd-archive.7691.n7.nabbl...e-td76656.html for a provisioning discussion.

[e1-531g]

Do you need all URL or domain is enough?

[beavers]

Originally Posted by jggimi

It comes as part of OpenBSD, so there is nothing to install. But note that relayd(8) uses syslog(3) for logging. If you want to record those logs, you will likely want to provision syslog.conf(5) to isolate the messages. See http://openbsd-archive.7691.n7.nabbl...e-td76656.html for a provisioning discussion.

Nice, I'll take a look at changing the logging destination for relayd. (This will also be nice to help set a separate place to stash the noise from dhcpd. )

Originally Posted by e1-531g

Do you need all URL or domain is enough?

I'll probably end up settling for just the domains, but I want to be sure to capture those for all http and https traffic.

[e1-531g]

One problem with DNS is that browsers have DNS prefetch. But some browsers also have HTTP(S) prefetch as well.
When you log DNS queries you also end up logging all DNS queries. Not only HTTP and HTTPS.
You can also log IP addresses of TCP connection destination.

[beavers]

Now that I think about it, monitoring full URLs might be more what I'm after. I did manage to get relayd working on both http and https, although the browsers are now complaining about cert mismatch, and I'm not sure how to work around that.

My ultimate goal is to filter out all the baddies (ads, tracking, etc) at the gateway level. To that end I've set up a big blacklist on unbound(8), and force all traffic on 53 to my own server via pf. So now I'm interested in setting up some monitoring to see what else might be getting by. I figured a transparent proxy would be a good way to go about that. Or, is there a better option I've not thought of?

[jggimi]

...cert mismatch...

TronDD posted in a misc@ mailing list discussion about relayd certificates (source):

'ca key' and 'ca cert' is for MITM roll your own certs on the fly.

For server certs, like a web server would have, you don't specify them.
relayd looks for address:port.key and address:port.crt as per the 'listen
on' description in relayd.conf(5)

[beavers]

Yep, that's how I've set it up.

certs:

/etc/ssl/ca.crt
/etc/ssl/127.0.0.1:8443.crt
/etc/ssl/private/ca.key
/etc/ssl/private/127.0.0.1:8443.key

/etc/relayd.conf:

http protocol httpfilter {
    return error
    match request label "URL filtered!"
    block request quick url "example.com/" value "*"
}

http protocol tlsfilter {
    return error
    match request label "URL filtered!"
    block request quick url "example.com/" value "*"
    tls ca key "/etc/ssl/private/ca.key" password "password"
    tls ca cert "/etc/ssl/ca.crt"
}

relay httpproxy {
    listen on 127.0.0.1 port 8080
    protocol httpfilter
    forward to destination
}

relay tlsproxy {
    listen on 127.0.0.1 port 8443 tls
    protocol tlsfilter
    forward with tls to destination
}

relevant portion of /etc/pf.conf:

pass in quick log on $int_ifs inet proto { tcp udp } from $wired_if:network to port 53 rdr-to $wired_if:0
pass in log on $int_ifs inet proto tcp from $wired_if:network to port www   divert-to localhost port 8080
pass in log on $int_ifs inet proto tcp from $wired_if:network to port https divert-to localhost port 8443

With this setup, Chromium complains that the cert for https://duckduckgo.com/ doesn't match (since the name on the cert is 127.0.0.1).

[beavers]

Is there some trick I'm missing to get things rewritten so that the cert appears to a browser to be valid?

[jggimi]

My guess? (Yes, its only a guess.) The cert you provide to the browser need to be authenticated with a CA the browser accepts.

[beavers]

Yeah, that's definitely what's happening. How would I get a cert that's CA-approved for a host on a private IP? Hoping to avoid having to import the cert manually on each and every browser that crosses my network.

[jggimi]

One way might be via letsencrypt.org, a free certificate service operated by the ISRG. It requires you to have an on-the-Internet web server to obtain the certificate and key, and is designed to renew automatically. See the acme-client(1) man page.

Once you have the cert and key acquired at your public web server, you may be able to copy the pair to your private server, replacing it every with every 60-90 day renewal. However I expect you would also require a split-horizon DNS, so that the private server resolves correctly for those browsers using it.

[jggimi]

While I'm not (currently) a relayd user, I happened to test yesterday's scenario -- a private server using a public server's key pair -- because I am considering putting relayd in front of a wordpress server to block access to /wp-admin and /wp-login paths. The browser's connection to the private server works fine, as long as the FQDN resolves to the private IP address.

Testing with relayd still pends. My plan is to use use the web server's key pair for TLS inspection.

At the moment, those two paths are protected with httpd.conf(5) authenticate, and then with some wordpress security plugins. Not good enough for me.

[jggimi]

Originally Posted by beavers

Hoping to avoid having to import the cert manually on each and every browser that crosses my network.

Well, in reading up on relayd this evening in preparation for initial testing, I have learned that this is what most organizations that use TLS inspection tools actually do. They distribute their private CA certificates to all clients.

[jggimi]

I've tested TLS acceleration. Not much to accelerate, as http(8) was listening on a loopback address on the same test system. TLS inspection works fine. However password prompts from the back end web server pop up a cleartext warning, as the back end server was using cleartext. I may be able to remove this with header modification, which would mean more testing.

I'm able to move my key pair from system to system, as long as the IP address resolves at the calling client system (such as with an /etc/hosts entry or a non-authoritative DNS server). I can uses my key pair with relayd. My CA will confirm OCSP queries issued by the browser, because the certificate is valid.

[jggimi]

Originally Posted by jggimi

I may be able to remove this with header modification...

No, it seems all I needed was to configure the back end server to use TLS also.