Who needs browsers? An opinionated look at the 4 HTTP clients in the base install ;^)

[fvgit]

To shorten the time while we wait for 6.8 to hit the dance floor I'd like to entertain you with a tounge-in-cheek introduction to the four(!) HTTP-clients that come with OpenBSD.


#1: If you're lazy, you use ftp(1).

• redirect to stdout:

  ftp -o - https://www.openbsd.org/faq/faq4.html

• save as file:

  ftp -o fosdem2020_unwind.pdf https://www.openbsd.org/papers/fosdem2020_unwind.pdf

If required, a User-Agent can be provided with the -U option. Every OpenBSD user should know this command. Period.

By the way, you can clean up most of the HTML-tags from the output with an sed statement like this one:

(...) | sed -e :a -e 's/<[^>]*>//g;/</N;//ba'

(lifted from the sed1liners.txt file floating about the interwebs and attributed to an S.G. Ravenhall).


#2: If you're 1337, you use nc(1).

The equivalent of impressing your misguided Emacs-using friends with your cool ed(1) skills.

• redirect to stdout:

  printf "GET /plus.html HTTP/1.0\r\n\r\n" | nc -c www.openbsd.org 443

• save as file:

  printf "GET /images/poster36.jpg HTTP/1.0\r\n\r\n" | nc -c www.openbsd.org 443 > poster36.jpg

  You can shave off HTTP headers by adding an sed process to the pipe at the end:
  

  (...) | sed "/HTTP/{N; N; N; N; N; s/.*//g; }" > poster36.jpg

  or just
  

  (...) ] sed "1,6d" > poster36.jpg

  Note that the number of HTTP headers differs between HTTP/1.0 and HTTP/1.1.
  
  If you need to supply a User-Agent you can switch to HTTP/1.1 by expanding the printf statement in the following way:
  

  printf "GET /plus.html HTTP/1.1\r\n\
  Host: https://www.openbsd.org/\r\n\
  Accept-Language: en\r\n\
  User-Agent: curl/7.\r\n\
  Connection: close\r\n\r\n" (...)

#3: If you're paranoid, you use perl(1) with HTTP::Tiny(3p), OpenBSD::Pledge(3p), and OpenBSD::Unveil(3p).

• redirect to stdout (no file system access whatsoever):

  #!/usr/bin/perl
  
  use strict;
  use warnings;
  use OpenBSD::Unveil;
  use OpenBSD::Pledge;
  use HTTP::Tiny;
  
  unveil() || die "Unable to lock unveil: $!";
  pledge( qw( inet dns ) ) || die "Unable to pledge: $!";
  my $http = HTTP::Tiny->new( agent => "Can't touch this.v3" );
  my $response = $http->get($ARGV[0]);
  die "Failed!\n" unless $response->{success};
  print $response->{content} if length $response->{content};

  Usage:

  perl script.pl http://www.openbsd.org/goals.html

• save as file (highly restricted file system access only):

  #!/usr/bin/perl
  
  use strict;
  use warnings;
  use OpenBSD::Unveil;
  use OpenBSD::Pledge;
  use Env qw(HOME);
  use HTTP::Tiny;
  
  my $dir = "$HOME/groundzero/";
  my $dir2 = "/usr/libdata/perl5/Time/Local.pm";
  unveil( $dir, "rwc" ) || die "Unable to unveil: $!";
  unveil( $dir2, "r" ) || die "Unable to unveil: $!";
  unveil() || die "Unable to lock unveil: $!";
  pledge( qw( rpath wpath cpath inet fattr dns ) ) || die "Unable to pledge: $!";
  
  my $http = HTTP::Tiny->new( agent => 'Too sexy for my shirt/v2.1 LLCF' );
  my $response = $http->mirror($ARGV[0], $ARGV[1]);
  die "Failed!\n" unless $response->{success};

  Usage:

  perl script.pl http://www.openbsd.org/goals.html ~/groundzero/goals.html

  for example.
  
  The fattr pledge and the Time/Local.pm unveil aren't strictly necessary to successfully retrieve the file, but if they're not present perl will dump core afterwards because HTTP::Tiny will attempt to set file access/modification times after finishing the download.
  
  NOTE: This solution (#3) is at least in part an exception to the rule because the two perl modules required for SSL support are not included in the base install. While you can stay in the base install by using the nc-trick outlined below at the end of section #4, the cleaner solution would be to simply do a

  pkg_add p5-IO-Socket-SSL

  . This will pull in just one dependency (p5-Net-SSLeay). Both are necessary as per the manpage. As usual these perl module packages are very tiny and it's a small price to pay in order to use the HTTP::Tiny code in the proper way. After package installation the scripts need to be told to use the IO::Socket::SSL pragma:
  

  use IO::Socket::SSL;

  And in case of the first script (output to stdout) you'll need to either add 'rpath' or 'error' to the pledge list otherwise it'll dump core. This is interesting. With the rpath pledge added the script will work without aborting. The unveil() command still prevents access to the file system. With the error pledge instead it'll also work. That would mean the rpath pledge isn't necessary. Something somewhere in the perl interpreter or the modules tries a restricted operation which triggers the SIGABRT (leading to the core file). The 'error' pledge prevents killing the process but shows no ENOSYS (error) output and the script finishes normally. Not sure if this is a bug, a rare edge case, or simply an insufficient grasp of how pledge operates in general on my side. Wouldn't surprise me if it were the latter.
  
  Read on anyway, to find out how to bolt on SSL with base system tools.

#4: If you're out of your mind, you use telnet(1).

• redirect to stdout:

  telnet www.openbsd.org 80

  then type at the telnet prompt:

  GET /plus68.html HTTP/1.0

  and hit Return twice(!).
• save as file:

  telnet www.openbsd.org 80 | sed '1,10d' > plus68.html

  NOTE: This time you won't see the telnet prompt, only the blinking cursor. Just type

  GET /plus68.html HTTP/1.0

  and hit Return twice(!) and you'll see the connection close and the document saved as plus68.html. The sed statement will remove the HTTP headers as well as the telnet status information which would otherwise be included at the beginning of the output file.
  
  
  I can already hear you complain: "Oi mate, that's all fine and dandy, but what if the webserver only supports HTTPS?" No problem, nc will take care of that.
  
  First we prepare the tunnel:

  nc -l localhost 8080 | nc -c www.openbsd.org 443 | sed '1,6d' > poster36.jpg &

  Now we initiate the download:

  telnet localhost 8080

  At the telnet prompt we do the usual:

  GET /images/poster36.jpg HTTP/1.0

  followed by typing Return twice(!), and then quit to exit telnet.
  
  What exactly happens here? The HTTP GET command we just typed at the telnet prompt is sent to the first nc process listening on localhost port 8080, which in turn pipes it to the second nc process which opens an SSL tunnel to the webserver (here openbsd.org). The output returned from that is then piped to sed which shaves of the first six lines of it (the HTTP headers) and the remainder of the data is finally redirected to the file poster36.jpg.

That's it. I hope you enjoyed my little HTTP romp. And seriously, who needs browsers?!


PS: Feel free to point out any errors if you see them. Thanks.

[IdOp]

Thanks for the nice write-up! The idea of using nc to do the TLS is interesting. I have been using stunnel for a similar thing sometimes. But nc looks like it may be cleaner (and for OpenBSD in the base). Will have to look into that.

Although not central to the purpose of your write-up, sometimes a GET is not enough. A "Host: " or "Referer: " field may be required in the HTTP header. The Host: will have to refer to the remote server, not the localhost.

[fvgit]

Thanks for the kind words. To use nc with HTTPS remember to set the -c flag, it's easy to overlook. I speak from experience. I actually didn't even look closely at the HTTP protocol. There are numerous ways in which this can fail. That being said, for what it is, it's actually pretty neat. I have a follow-up piece coming up regarding some of the popular image sites. And maybe a few other ideas. And there's probably some potential just waiting to be unlocked when it comes to HTTP::Tiny. After all it supports cookies, redirects, POST requests and whatnot.

In the meantime how about some quick'n'dirty gopher fun?

Using technique #2 to access a gopher server (commands in green, follow-up target in red)

$ printf "/fun/xkcd \r\n" | nc gopher.floodgap.com 70
1Floodgap Gopher Fun and Games  /fun    gopher.floodgap.com     70
i               error.host      1
i            88 A Webcomic of Romance,    88            error.host      1
i            88 Sarcasm, Math & Language. 88            error.host      1
i            88                           88            error.host      1
i8b,     ,d8 88   ,d8  ,adPPYba,  ,adPPYb,88            error.host      1
i `Y8, ,8P'  88 ,a8"  a8"     "" a8"    `Y88            error.host      1
i   )888(    8888[    8b         8b       88            error.host      1
i ,d8" "8b,  88`"Yba, "8a,   ,aa "8a,   ,d88            error.host      1
i8P'     `Y8 88   `Y8a `"Ybbd8"'  `"8bbdP"Y8            error.host      1
i               error.host      1
iCurrent comic:         error.host      1
1xkcd - A Webcomic - Masks (2367)       /fun/xkcd/comics/2360/2367      gopher.floodgap.com     70
i               error.host      1
1Browse all xkcd comics /fun/xkcd/comics        gopher.floodgap.com     70
i               error.host      1
iWarning: this comic occasionally contains strong language (which               error.host      1
imay be unsuitable for children), unusual humor (which may be           error.host      1
iunsuitable for adults), and advanced mathematics (which may be         error.host      1
iunsuitable for liberal-arts majors).           error.host      1
iUpdates usually Monday/Wednesday/Friday.               error.host      1
iSome older Gopher clients may not understand PNG itemtype.             error.host      1
hAbout xkcd (WWW)       URL:http://www.xkcd.com/about/  gopher.floodgap.com     70
i               error.host      1
iComics reproduced from www.xkcd.com under license.             error.host      1
hxkcd License (WWW)     URL:http://www.xkcd.com/license.html    gopher.floodgap.com     70
hxkcd Web Site (WWW)    URL:http://www.xkcd.com/        gopher.floodgap.com     70

.

Now the subpage:

 $ printf "/fun/xkcd/comics/2360/2367 \r\n" | nc gopher.floodgap.com 70
1xkcd - A Webcomic of Romance, Sarcasm, Math and Language.      /fun/xkcd       gopher.floodgap.com     70
1Floodgap Gopher Fun and Games  /fun    gopher.floodgap.com     70
i               error.host      1
ixkcd - A Webcomic - Masks (2367)               error.host      1
i               error.host      1
iImage:         error.host      1
pmasks.png      /fun/xkcd/comics/2360/2367/masks.png    gopher.floodgap.com     70
i               error.host      1
iHaunted Halloween masks from a mysterious costume shop that turn               error.host      1
iyou evil and grow into your skin score a surprisingly high 80%         error.host      1
ifiltration efficiency in R. L. Stine-sponsored NIOSH tests.            error.host      1
i               error.host      1
iComics reproduced from www.xkcd.com under license.             error.host      1
hxkcd License (WWW)     URL:https://www.xkcd.com/license.html   gopher.floodgap.com     70
hxkcd Web Site (WWW)    URL:https://www.xkcd.com/       gopher.floodgap.com     70
i               error.host      1
iPrevious:              error.host      1
1xkcd - A Webcomic - Amelia's Farm Fresh Cookies (2366) /fun/xkcd/comics/2360/2366      gopher.floodgap.com     70
.

And retrieving the image:

printf "/fun/xkcd/comics/2360/2367/masks.png \r\n" | nc gopher.floodgap.com 70 > masks.png

To access the root directory just do a:

printf "/ \r\n" | nc gopher.floodgap.com 70

or

telnet gopher.floodgap.com 70

and hit Enter.

[fvgit]

@the pic!

I haven't played with Tor, yet. But it's definitely on my long-term list of things to fool around with. There are howtos out there explaining how to 'torify' applications other than firefox, but it seems it's not recommended. Apparently the Firefox bloat is part of the camouflage when it comes to Tor. Not because of its bloat per se, but because everyone is using the same browser. As soon as you change that you're sticking out from the general traffic on the Tor network! And that's the first step to being tracked and/or de-anonymized.

But that's definitely a discussion warranting a separate topic.

Anyway, I have a few more neat little tricks coming up over the coming weeks. Stay tuned.

[bradley]

I have a neat trick up my sleeve too. In case someone is using the terminal and needs a map, just use:

telnet mapscii.me

h,j,k,l are for navigation, a and z for zoom, q for quit.

Regarding Tor there are some hints here.

[IdOp]

Originally Posted by bradley

I have a neat trick up my sleeve too. In case someone is using the terminal and needs a map, just use:

telnet mapscii.me

h,j,k,l are for navigation, a and z for zoom, q for quit.

LOL. Google is doomed!

[fvgit]

Originally Posted by bradley

I have a neat trick up my sleeve too. In case someone is using the terminal and needs a map, just use:

telnet mapscii.me

h,j,k,l are for navigation, a and z for zoom, q for quit.

Love it! Thanks for posting.