DaemonForums  

Go Back   DaemonForums > OpenBSD > OpenBSD General

OpenBSD General Other questions regarding OpenBSD which do not fit in any of the categories below.

Reply
 
Thread Tools Display Modes
  #1   (View Single Post)  
Old 4th October 2020
fvgit's Avatar
fvgit fvgit is offline
Spikes in tights
 
Join Date: May 2016
Location: perl -MMIME::Base64 -le 'print decode_base64("U2hlcndvb2QgRm9yZXN0")'
Posts: 304
Default Who needs browsers? An opinionated look at the 4 HTTP clients in the base install ;^)

To shorten the time while we wait for 6.8 to hit the dance floor I'd like to entertain you with a tounge-in-cheek introduction to the four(!) HTTP-clients that come with OpenBSD.


#1: If you're lazy, you use ftp(1).
  • redirect to stdout:
    Code:
    ftp -o - https://www.openbsd.org/faq/faq4.html
  • save as file:
    Code:
    ftp -o fosdem2020_unwind.pdf https://www.openbsd.org/papers/fosdem2020_unwind.pdf
If required, a User-Agent can be provided with the -U option. Every OpenBSD user should know this command. Period.

By the way, you can clean up most of the HTML-tags from the output with an sed statement like this one:
Code:
(...) | sed -e :a -e 's/<[^>]*>//g;/</N;//ba'
(lifted from the sed1liners.txt file floating about the interwebs and attributed to an S.G. Ravenhall).


#2: If you're 1337, you use nc(1).

The equivalent of impressing your misguided Emacs-using friends with your cool ed(1) skills.
  • redirect to stdout:
    Code:
    printf "GET /plus.html HTTP/1.0\r\n\r\n" | nc -c www.openbsd.org 443
  • save as file:
    Code:
    printf "GET /images/poster36.jpg HTTP/1.0\r\n\r\n" | nc -c www.openbsd.org 443 > poster36.jpg
    You can shave off HTTP headers by adding an sed process to the pipe at the end:
    Code:
    (...) | sed "/HTTP/{N; N; N; N; N; s/.*//g; }" > poster36.jpg
    or just
    Code:
    (...) ] sed "1,6d" > poster36.jpg
    Note that the number of HTTP headers differs between HTTP/1.0 and HTTP/1.1.

    If you need to supply a User-Agent you can switch to HTTP/1.1 by expanding the printf statement in the following way:
    Code:
    printf "GET /plus.html HTTP/1.1\r\n\
    Host: https://www.openbsd.org/\r\n\
    Accept-Language: en\r\n\
    User-Agent: curl/7.\r\n\
    Connection: close\r\n\r\n" (...)

#3: If you're paranoid, you use perl(1) with HTTP::Tiny(3p), OpenBSD::Pledge(3p), and OpenBSD::Unveil(3p).
  • redirect to stdout (no file system access whatsoever):
    Code:
    #!/usr/bin/perl
    
    use strict;
    use warnings;
    use OpenBSD::Unveil;
    use OpenBSD::Pledge;
    use HTTP::Tiny;
    
    unveil() || die "Unable to lock unveil: $!";
    pledge( qw( inet dns ) ) || die "Unable to pledge: $!";
    my $http = HTTP::Tiny->new( agent => "Can't touch this.v3" );
    my $response = $http->get($ARGV[0]);
    die "Failed!\n" unless $response->{success};
    print $response->{content} if length $response->{content};
    Usage:
    Code:
    perl script.pl http://www.openbsd.org/goals.html
  • save as file (highly restricted file system access only):
    Code:
    #!/usr/bin/perl
    
    use strict;
    use warnings;
    use OpenBSD::Unveil;
    use OpenBSD::Pledge;
    use Env qw(HOME);
    use HTTP::Tiny;
    
    my $dir = "$HOME/groundzero/";
    my $dir2 = "/usr/libdata/perl5/Time/Local.pm";
    unveil( $dir, "rwc" ) || die "Unable to unveil: $!";
    unveil( $dir2, "r" ) || die "Unable to unveil: $!";
    unveil() || die "Unable to lock unveil: $!";
    pledge( qw( rpath wpath cpath inet fattr dns ) ) || die "Unable to pledge: $!";
    
    my $http = HTTP::Tiny->new( agent => 'Too sexy for my shirt/v2.1 LLCF' );
    my $response = $http->mirror($ARGV[0], $ARGV[1]);
    die "Failed!\n" unless $response->{success};
    Usage:
    Code:
    perl script.pl http://www.openbsd.org/goals.html ~/groundzero/goals.html
    for example.

    The fattr pledge and the Time/Local.pm unveil aren't strictly necessary to successfully retrieve the file, but if they're not present perl will dump core afterwards because HTTP::Tiny will attempt to set file access/modification times after finishing the download.

    NOTE: This solution (#3) is at least in part an exception to the rule because the two perl modules required for SSL support are not included in the base install. While you can stay in the base install by using the nc-trick outlined below at the end of section #4, the cleaner solution would be to simply do a
    Code:
    pkg_add p5-IO-Socket-SSL
    . This will pull in just one dependency (p5-Net-SSLeay). Both are necessary as per the manpage. As usual these perl module packages are very tiny and it's a small price to pay in order to use the HTTP::Tiny code in the proper way. After package installation the scripts need to be told to use the IO::Socket::SSL pragma:
    Code:
    use IO::Socket::SSL;
    And in case of the first script (output to stdout) you'll need to either add 'rpath' or 'error' to the pledge list otherwise it'll dump core. This is interesting. With the rpath pledge added the script will work without aborting. The unveil() command still prevents access to the file system. With the error pledge instead it'll also work. That would mean the rpath pledge isn't necessary. Something somewhere in the perl interpreter or the modules tries a restricted operation which triggers the SIGABRT (leading to the core file). The 'error' pledge prevents killing the process but shows no ENOSYS (error) output and the script finishes normally. Not sure if this is a bug, a rare edge case, or simply an insufficient grasp of how pledge operates in general on my side. Wouldn't surprise me if it were the latter.

    Read on anyway, to find out how to bolt on SSL with base system tools.

#4: If you're out of your mind, you use telnet(1).
  • redirect to stdout:
    Code:
    telnet www.openbsd.org 80
    then type at the telnet prompt:
    Code:
    GET /plus68.html HTTP/1.0
    and hit Return twice(!).
  • save as file:
    Code:
    telnet www.openbsd.org 80 | sed '1,10d' > plus68.html
    NOTE: This time you won't see the telnet prompt, only the blinking cursor. Just type
    Code:
    GET /plus68.html HTTP/1.0
    and hit Return twice(!) and you'll see the connection close and the document saved as plus68.html. The sed statement will remove the HTTP headers as well as the telnet status information which would otherwise be included at the beginning of the output file.


    I can already hear you complain: "Oi mate, that's all fine and dandy, but what if the webserver only supports HTTPS?" No problem, nc will take care of that.

    First we prepare the tunnel:
    Code:
    nc -l localhost 8080 | nc -c www.openbsd.org 443 | sed '1,6d' > poster36.jpg &
    Now we initiate the download:
    Code:
    telnet localhost 8080
    At the telnet prompt we do the usual:
    Code:
    GET /images/poster36.jpg HTTP/1.0
    followed by typing Return twice(!), and then quit to exit telnet.

    What exactly happens here? The HTTP GET command we just typed at the telnet prompt is sent to the first nc process listening on localhost port 8080, which in turn pipes it to the second nc process which opens an SSL tunnel to the webserver (here openbsd.org). The output returned from that is then piped to sed which shaves of the first six lines of it (the HTTP headers) and the remainder of the data is finally redirected to the file poster36.jpg.

That's it. I hope you enjoyed my little HTTP romp. And seriously, who needs browsers?!


PS: Feel free to point out any errors if you see them. Thanks.
Reply With Quote
  #2   (View Single Post)  
Old 5th October 2020
IdOp's Avatar
IdOp IdOp is offline
Too dumb for a smartphone
 
Join Date: May 2008
Location: twisting on the daemon's fork(2)
Posts: 999
Default

Thanks for the nice write-up! The idea of using nc to do the TLS is interesting. I have been using stunnel for a similar thing sometimes. But nc looks like it may be cleaner (and for OpenBSD in the base). Will have to look into that.

Although not central to the purpose of your write-up, sometimes a GET is not enough. A "Host: " or "Referer: " field may be required in the HTTP header. The Host: will have to refer to the remote server, not the localhost.
Reply With Quote
  #3   (View Single Post)  
Old 5th October 2020
fvgit's Avatar
fvgit fvgit is offline
Spikes in tights
 
Join Date: May 2016
Location: perl -MMIME::Base64 -le 'print decode_base64("U2hlcndvb2QgRm9yZXN0")'
Posts: 304
Default

Thanks for the kind words. To use nc with HTTPS remember to set the -c flag, it's easy to overlook. I speak from experience. I actually didn't even look closely at the HTTP protocol. There are numerous ways in which this can fail. That being said, for what it is, it's actually pretty neat. I have a follow-up piece coming up regarding some of the popular image sites. And maybe a few other ideas. And there's probably some potential just waiting to be unlocked when it comes to HTTP::Tiny. After all it supports cookies, redirects, POST requests and whatnot.

In the meantime how about some quick'n'dirty gopher fun?

Using technique #2 to access a gopher server (commands in green, follow-up target in red)
Code:
$ printf "/fun/xkcd \r\n" | nc gopher.floodgap.com 70
1Floodgap Gopher Fun and Games  /fun    gopher.floodgap.com     70
i               error.host      1
i            88 A Webcomic of Romance,    88            error.host      1
i            88 Sarcasm, Math & Language. 88            error.host      1
i            88                           88            error.host      1
i8b,     ,d8 88   ,d8  ,adPPYba,  ,adPPYb,88            error.host      1
i `Y8, ,8P'  88 ,a8"  a8"     "" a8"    `Y88            error.host      1
i   )888(    8888[    8b         8b       88            error.host      1
i ,d8" "8b,  88`"Yba, "8a,   ,aa "8a,   ,d88            error.host      1
i8P'     `Y8 88   `Y8a `"Ybbd8"'  `"8bbdP"Y8            error.host      1
i               error.host      1
iCurrent comic:         error.host      1
1xkcd - A Webcomic - Masks (2367)       /fun/xkcd/comics/2360/2367      gopher.floodgap.com     70
i               error.host      1
1Browse all xkcd comics /fun/xkcd/comics        gopher.floodgap.com     70
i               error.host      1
iWarning: this comic occasionally contains strong language (which               error.host      1
imay be unsuitable for children), unusual humor (which may be           error.host      1
iunsuitable for adults), and advanced mathematics (which may be         error.host      1
iunsuitable for liberal-arts majors).           error.host      1
iUpdates usually Monday/Wednesday/Friday.               error.host      1
iSome older Gopher clients may not understand PNG itemtype.             error.host      1
hAbout xkcd (WWW)       URL:http://www.xkcd.com/about/  gopher.floodgap.com     70
i               error.host      1
iComics reproduced from www.xkcd.com under license.             error.host      1
hxkcd License (WWW)     URL:http://www.xkcd.com/license.html    gopher.floodgap.com     70
hxkcd Web Site (WWW)    URL:http://www.xkcd.com/        gopher.floodgap.com     70

.
Now the subpage:
Code:
 $ printf "/fun/xkcd/comics/2360/2367 \r\n" | nc gopher.floodgap.com 70
1xkcd - A Webcomic of Romance, Sarcasm, Math and Language.      /fun/xkcd       gopher.floodgap.com     70
1Floodgap Gopher Fun and Games  /fun    gopher.floodgap.com     70
i               error.host      1
ixkcd - A Webcomic - Masks (2367)               error.host      1
i               error.host      1
iImage:         error.host      1
pmasks.png      /fun/xkcd/comics/2360/2367/masks.png    gopher.floodgap.com     70
i               error.host      1
iHaunted Halloween masks from a mysterious costume shop that turn               error.host      1
iyou evil and grow into your skin score a surprisingly high 80%         error.host      1
ifiltration efficiency in R. L. Stine-sponsored NIOSH tests.            error.host      1
i               error.host      1
iComics reproduced from www.xkcd.com under license.             error.host      1
hxkcd License (WWW)     URL:https://www.xkcd.com/license.html   gopher.floodgap.com     70
hxkcd Web Site (WWW)    URL:https://www.xkcd.com/       gopher.floodgap.com     70
i               error.host      1
iPrevious:              error.host      1
1xkcd - A Webcomic - Amelia's Farm Fresh Cookies (2366) /fun/xkcd/comics/2360/2366      gopher.floodgap.com     70
.
And retrieving the image:
Code:
printf "/fun/xkcd/comics/2360/2367/masks.png \r\n" | nc gopher.floodgap.com 70 > masks.png
To access the root directory just do a:
Code:
printf "/ \r\n" | nc gopher.floodgap.com 70
or
Code:
telnet gopher.floodgap.com 70
and hit Enter.
Reply With Quote
  #4   (View Single Post)  
Old 27th October 2020
Prevet Prevet is offline
Package Pilot
 
Join Date: Oct 2017
Posts: 196
Default

This is pretty cool.

https://i.kym-cdn.com/entries/icons/...mWg2Hxd4ro.jpg

Since you mentioned ed, this could be integrated into it using simple scripts.

The only other thing I would want is to be able to direct all traffic through the Tor network. Then I could delete Tor Browser which is currently eating up 1GB of RAM. I made some scripts to do this that I once used with a VPN provider. Perhaps they could be modified to work with Tor.

Last edited by Prevet; 27th October 2020 at 08:27 AM.
Reply With Quote
  #5   (View Single Post)  
Old 30th October 2020
fvgit's Avatar
fvgit fvgit is offline
Spikes in tights
 
Join Date: May 2016
Location: perl -MMIME::Base64 -le 'print decode_base64("U2hlcndvb2QgRm9yZXN0")'
Posts: 304
Default

@the pic!

I haven't played with Tor, yet. But it's definitely on my long-term list of things to fool around with. There are howtos out there explaining how to 'torify' applications other than firefox, but it seems it's not recommended. Apparently the Firefox bloat is part of the camouflage when it comes to Tor. Not because of its bloat per se, but because everyone is using the same browser. As soon as you change that you're sticking out from the general traffic on the Tor network! And that's the first step to being tracked and/or de-anonymized.

But that's definitely a discussion warranting a separate topic.

Anyway, I have a few more neat little tricks coming up over the coming weeks. Stay tuned.
Reply With Quote
  #6   (View Single Post)  
Old 16th November 2020
bradley bradley is offline
Fdisk Soldier
 
Join Date: Jul 2020
Posts: 50
Default

I have a neat trick up my sleeve too. In case someone is using the terminal and needs a map, just use:

Code:
telnet mapscii.me
h,j,k,l are for navigation, a and z for zoom, q for quit.

Regarding Tor there are some hints here.

Last edited by bradley; 16th November 2020 at 02:52 AM.
Reply With Quote
  #7   (View Single Post)  
Old 16th November 2020
IdOp's Avatar
IdOp IdOp is offline
Too dumb for a smartphone
 
Join Date: May 2008
Location: twisting on the daemon's fork(2)
Posts: 999
Default

Quote:
Originally Posted by bradley View Post
I have a neat trick up my sleeve too. In case someone is using the terminal and needs a map, just use:

Code:
telnet mapscii.me
h,j,k,l are for navigation, a and z for zoom, q for quit.
LOL. Google is doomed!
Reply With Quote
  #8   (View Single Post)  
Old 18th November 2020
fvgit's Avatar
fvgit fvgit is offline
Spikes in tights
 
Join Date: May 2016
Location: perl -MMIME::Base64 -le 'print decode_base64("U2hlcndvb2QgRm9yZXN0")'
Posts: 304
Default

Quote:
Originally Posted by bradley View Post
I have a neat trick up my sleeve too. In case someone is using the terminal and needs a map, just use:

Code:
telnet mapscii.me
h,j,k,l are for navigation, a and z for zoom, q for quit.
Love it! Thanks for posting.
Reply With Quote
Reply

Tags
ftp, nc, openbsd, perl, telnet

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Delay in keystroke with web browsers roddierod OpenBSD General 5 27th November 2019 03:39 PM
Xorg and OpenBSD security record on the Base Install. shep News 3 27th October 2018 08:42 AM
removing unused system daemons from the base install (for security) puffyborg OpenBSD Security 2 24th August 2018 08:15 PM
EFF: More than 80% of browsers have trackable signatures J65nko News 0 18th May 2010 05:57 PM
Limitations of Console Web Browsers JMJ_coder General software and network 17 8th September 2008 06:06 PM


All times are GMT. The time now is 09:02 PM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick