DaemonForums  

Go Back   DaemonForums > OpenBSD > OpenBSD General

OpenBSD General Other questions regarding OpenBSD which do not fit in any of the categories below.

 
 
Thread Tools Display Modes
Prev Previous Post   Next Post Next
  #1   (View Single Post)  
Old 4th October 2020
fvgit's Avatar
fvgit fvgit is offline
Spikes in tights
 
Join Date: May 2016
Location: perl -MMIME::Base64 -le 'print decode_base64("U2hlcndvb2QgRm9yZXN0")'
Posts: 314
Default Who needs browsers? An opinionated look at the 4 HTTP clients in the base install ;^)

To shorten the time while we wait for 6.8 to hit the dance floor I'd like to entertain you with a tounge-in-cheek introduction to the four(!) HTTP-clients that come with OpenBSD.


#1: If you're lazy, you use ftp(1).
  • redirect to stdout:
    Code:
    ftp -o - https://www.openbsd.org/faq/faq4.html
  • save as file:
    Code:
    ftp -o fosdem2020_unwind.pdf https://www.openbsd.org/papers/fosdem2020_unwind.pdf
If required, a User-Agent can be provided with the -U option. Every OpenBSD user should know this command. Period.

By the way, you can clean up most of the HTML-tags from the output with an sed statement like this one:
Code:
(...) | sed -e :a -e 's/<[^>]*>//g;/</N;//ba'
(lifted from the sed1liners.txt file floating about the interwebs and attributed to an S.G. Ravenhall).


#2: If you're 1337, you use nc(1).

The equivalent of impressing your misguided Emacs-using friends with your cool ed(1) skills.
  • redirect to stdout:
    Code:
    printf "GET /plus.html HTTP/1.0\r\n\r\n" | nc -c www.openbsd.org 443
  • save as file:
    Code:
    printf "GET /images/poster36.jpg HTTP/1.0\r\n\r\n" | nc -c www.openbsd.org 443 > poster36.jpg
    You can shave off HTTP headers by adding an sed process to the pipe at the end:
    Code:
    (...) | sed "/HTTP/{N; N; N; N; N; s/.*//g; }" > poster36.jpg
    or just
    Code:
    (...) ] sed "1,6d" > poster36.jpg
    Note that the number of HTTP headers differs between HTTP/1.0 and HTTP/1.1.

    If you need to supply a User-Agent you can switch to HTTP/1.1 by expanding the printf statement in the following way:
    Code:
    printf "GET /plus.html HTTP/1.1\r\n\
    Host: https://www.openbsd.org/\r\n\
    Accept-Language: en\r\n\
    User-Agent: curl/7.\r\n\
    Connection: close\r\n\r\n" (...)

#3: If you're paranoid, you use perl(1) with HTTP::Tiny(3p), OpenBSD::Pledge(3p), and OpenBSD::Unveil(3p).
  • redirect to stdout (no file system access whatsoever):
    Code:
    #!/usr/bin/perl
    
    use strict;
    use warnings;
    use OpenBSD::Unveil;
    use OpenBSD::Pledge;
    use HTTP::Tiny;
    
    unveil() || die "Unable to lock unveil: $!";
    pledge( qw( inet dns ) ) || die "Unable to pledge: $!";
    my $http = HTTP::Tiny->new( agent => "Can't touch this.v3" );
    my $response = $http->get($ARGV[0]);
    die "Failed!\n" unless $response->{success};
    print $response->{content} if length $response->{content};
    Usage:
    Code:
    perl script.pl http://www.openbsd.org/goals.html
  • save as file (highly restricted file system access only):
    Code:
    #!/usr/bin/perl
    
    use strict;
    use warnings;
    use OpenBSD::Unveil;
    use OpenBSD::Pledge;
    use Env qw(HOME);
    use HTTP::Tiny;
    
    my $dir = "$HOME/groundzero/";
    my $dir2 = "/usr/libdata/perl5/Time/Local.pm";
    unveil( $dir, "rwc" ) || die "Unable to unveil: $!";
    unveil( $dir2, "r" ) || die "Unable to unveil: $!";
    unveil() || die "Unable to lock unveil: $!";
    pledge( qw( rpath wpath cpath inet fattr dns ) ) || die "Unable to pledge: $!";
    
    my $http = HTTP::Tiny->new( agent => 'Too sexy for my shirt/v2.1 LLCF' );
    my $response = $http->mirror($ARGV[0], $ARGV[1]);
    die "Failed!\n" unless $response->{success};
    Usage:
    Code:
    perl script.pl http://www.openbsd.org/goals.html ~/groundzero/goals.html
    for example.

    The fattr pledge and the Time/Local.pm unveil aren't strictly necessary to successfully retrieve the file, but if they're not present perl will dump core afterwards because HTTP::Tiny will attempt to set file access/modification times after finishing the download.

    NOTE: This solution (#3) is at least in part an exception to the rule because the two perl modules required for SSL support are not included in the base install. While you can stay in the base install by using the nc-trick outlined below at the end of section #4, the cleaner solution would be to simply do a
    Code:
    pkg_add p5-IO-Socket-SSL
    . This will pull in just one dependency (p5-Net-SSLeay). Both are necessary as per the manpage. As usual these perl module packages are very tiny and it's a small price to pay in order to use the HTTP::Tiny code in the proper way. After package installation the scripts need to be told to use the IO::Socket::SSL pragma:
    Code:
    use IO::Socket::SSL;
    And in case of the first script (output to stdout) you'll need to either add 'rpath' or 'error' to the pledge list otherwise it'll dump core. This is interesting. With the rpath pledge added the script will work without aborting. The unveil() command still prevents access to the file system. With the error pledge instead it'll also work. That would mean the rpath pledge isn't necessary. Something somewhere in the perl interpreter or the modules tries a restricted operation which triggers the SIGABRT (leading to the core file). The 'error' pledge prevents killing the process but shows no ENOSYS (error) output and the script finishes normally. Not sure if this is a bug, a rare edge case, or simply an insufficient grasp of how pledge operates in general on my side. Wouldn't surprise me if it were the latter.

    Read on anyway, to find out how to bolt on SSL with base system tools.

#4: If you're out of your mind, you use telnet(1).
  • redirect to stdout:
    Code:
    telnet www.openbsd.org 80
    then type at the telnet prompt:
    Code:
    GET /plus68.html HTTP/1.0
    and hit Return twice(!).
  • save as file:
    Code:
    telnet www.openbsd.org 80 | sed '1,10d' > plus68.html
    NOTE: This time you won't see the telnet prompt, only the blinking cursor. Just type
    Code:
    GET /plus68.html HTTP/1.0
    and hit Return twice(!) and you'll see the connection close and the document saved as plus68.html. The sed statement will remove the HTTP headers as well as the telnet status information which would otherwise be included at the beginning of the output file.


    I can already hear you complain: "Oi mate, that's all fine and dandy, but what if the webserver only supports HTTPS?" No problem, nc will take care of that.

    First we prepare the tunnel:
    Code:
    nc -l localhost 8080 | nc -c www.openbsd.org 443 | sed '1,6d' > poster36.jpg &
    Now we initiate the download:
    Code:
    telnet localhost 8080
    At the telnet prompt we do the usual:
    Code:
    GET /images/poster36.jpg HTTP/1.0
    followed by typing Return twice(!), and then quit to exit telnet.

    What exactly happens here? The HTTP GET command we just typed at the telnet prompt is sent to the first nc process listening on localhost port 8080, which in turn pipes it to the second nc process which opens an SSL tunnel to the webserver (here openbsd.org). The output returned from that is then piped to sed which shaves of the first six lines of it (the HTTP headers) and the remainder of the data is finally redirected to the file poster36.jpg.

That's it. I hope you enjoyed my little HTTP romp. And seriously, who needs browsers?!


PS: Feel free to point out any errors if you see them. Thanks.
Reply With Quote
 

Tags
ftp, nc, openbsd, perl, telnet

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Delay in keystroke with web browsers roddierod OpenBSD General 5 27th November 2019 03:39 PM
Xorg and OpenBSD security record on the Base Install. shep News 3 27th October 2018 08:42 AM
removing unused system daemons from the base install (for security) puffyborg OpenBSD Security 2 24th August 2018 08:15 PM
EFF: More than 80% of browsers have trackable signatures J65nko News 0 18th May 2010 05:57 PM
Limitations of Console Web Browsers JMJ_coder General software and network 17 8th September 2008 06:06 PM


All times are GMT. The time now is 12:14 PM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick