DaemonForums  

Go Back   DaemonForums > OpenBSD > OpenBSD Packages and Ports

OpenBSD Packages and Ports Installation and upgrading of packages and ports on OpenBSD.

Reply
 
Thread Tools Display Modes
  #1   (View Single Post)  
Old 10th June 2017
lenlynch lenlynch is offline
New User
 
Join Date: Jan 2016
Location: Midwest North America
Posts: 3
Default python traceback captured when run from cron

Looking to improve my understanding of why the following situation occurs.

I'm running OpenBSD 6.1 -stable amd64.
Binary ports package of urlwatch installed using pkg_add, v2.6 (thanks @sthen !)
I used to run i386, which can be a different set and versions of ports.
I did move my urlwatch version 1 data set from a different machine (running OpenBSD 6.0 i386), read the upgrade notes on github README.md, and followed them.

I run urlwatch interactively as a standard user and as a daily cron job as my standard user.

When urlwatch reports changes under cron execution, I have it configured for email notification.
This works as expected, to OpenSMTPD for local delivery.

After the change event report, the following traceback is captured as part of the cron job's execution.

Code:
Traceback (most recent call last):
  File "/usr/local/bin/urlwatch", line 101, in <module>
    urlwatch_command.run()
  File "/usr/local/lib/python3.6/site-packages/urlwatch/command.py", line 167, in run
    self.urlwatcher.close()
  File "/usr/local/lib/python3.6/site-packages/urlwatch/main.py", line 96, in close
    self.report.finish()
  File "/usr/local/lib/python3.6/site-packages/urlwatch/handler.py", line 125, in finish
    ReporterBase.submit_all(self, self.job_states, duration)
  File "/usr/local/lib/python3.6/site-packages/urlwatch/reporters.py", line 92, in submit_all
    subclass(report, cfg, job_states, duration).submit()
  File "/usr/local/lib/python3.6/site-packages/urlwatch/reporters.py", line 315, in submit
    print(self._red(line))
UnicodeEncodeError: 'ascii' codec can't encode character '\xdc' in position 2: ordinal not in range(128)
This traceback appears to be complaining about unicode characters being returned instead of ascii characters for the checked item? Have I got that right?

I would not have noticed these messages if they weren't captured by the cron itself.
I have run urlwatch interactively many times in the past, and I don't ever recall seeing traceback output while doing that.
I get that I can silence the traceback details (using shell redirection) but I'm not sure that this is what I want to do as yet.

Questions that this raised:

Was I just unlucky enough to miss observing a traceback when I ran it interactively in the past, based on the changed item data returned?

If tracebacks like this are expected/normal in the output, is that due to python, urlwatch or one of its dependent packages?

Is cron's execution of programs fundamentally different from an interactive user in important ways? If so, what are they?

Anything else I should be looking at on my system configuration that could be attributing to this issue?


Thanks in advance for any insight or pointers to more details!
__________________
Summary: Solutions Architect, Networking, IaaS, DataSec, Un*x, Windows
Realizes: “Culture eats technology for breakfast…” -Russ White

Last edited by lenlynch; 10th June 2017 at 08:40 AM.
Reply With Quote
  #2   (View Single Post)  
Old 10th June 2017
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 5,797
Default

Hello, and welcome!
Quote:
Originally Posted by lenlynch View Post
This traceback appears to be complaining about unicode characters being returned instead of ascii characters for the checked item? Have I got that right?
That's my assessment also. Perhaps your interactive use is with a Unicode-capable terminal such as xterm(1), and cron(1) runs in a standard shell which may not have LC_CTYPE set.
Quote:
If tracebacks like this are expected/normal in the output, is that due to python, urlwatch or one of its dependent packages?
Python Unicode handling. See https://stackoverflow.com/questions/...ion-20/9942822
Reply With Quote
  #3   (View Single Post)  
Old 10th June 2017
lenlynch lenlynch is offline
New User
 
Join Date: Jan 2016
Location: Midwest North America
Posts: 3
Default

Thank you jggimi!

Quote:
Originally Posted by jggimi View Post
Perhaps your interactive use is with a Unicode-capable terminal such as xterm(1), and cron(1) runs in a standard shell which may not have LC_CTYPE set.
I do SSH using a client that is configured for UTF-8 support, so that is indeed something to follow-up on. Great point.

I do see this with regards to X/xenocara and xterm, which provides some outline to the problem of there being a protocol specific issue of negotiating character encodings between different systems...

http://undeadly.org/cgi?action=artic...20160308204011

It also points to the larger concern of data handling safety, which can not be guaranteed between non-OpenBSD systems interacting.

LOCALE(1) manpage hints at parts of the system are not UTF-8 friendly:

Code:
     A locale is a set of environment variables telling programs which
     character encoding, language and cultural conventions the user prefers.
     Programs in the OpenBSD base system ignore the locale except for the
     character encoding, and it is not recommended to use any of these
     variables except that the following non-default setting is supported as
     an option:

           export LC_CTYPE=en_US.UTF-8
...

     The OpenBSD base system supports two locales: the default of LC_CTYPE=C
     selects the US-ASCII character set and encoding, treating the bytes 0x80
     to 0xff as non-printable characters of application-specific meaning.
     LC_CTYPE=POSIX is an alias for LC_CTYPE=C.  The alternative of
     LC_CTYPE=en_US.UTF-8 selects the UTF-8 encoding of the Unicode character
     set, which is supported by many parts of the system, but not yet fully
     supported by all parts.

Updated this post, there was an error in my conclusion, which was removed, sorry in advance.

Has anyone heard-of/experienced security issues with cron and UTF-8 ?

Thanks in advance.
__________________
Summary: Solutions Architect, Networking, IaaS, DataSec, Un*x, Windows
Realizes: “Culture eats technology for breakfast…” -Russ White

Last edited by lenlynch; 10th June 2017 at 07:05 AM.
Reply With Quote
  #4   (View Single Post)  
Old 10th June 2017
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 5,797
Default

It appears cron() has locale support. Jan Stary posted this in tech@ last November recommending its removal - it did not receive any follow up posts and locale is still hooked into cron.c:

http://openbsd-archive.7691.n7.nabbl...7742.html#none

I've never set locale values with cron jobs, I don't normally use locale settings on terminals, I don't use urlwatch, and I'm not a Python programmer. So I'll step back and let others weigh in on a possible circumvention or solution.
Reply With Quote
  #5   (View Single Post)  
Old 11th June 2017
lenlynch lenlynch is offline
New User
 
Join Date: Jan 2016
Location: Midwest North America
Posts: 3
Default

Thank you for attention, feedback and pointers. Much appreciated.
__________________
Summary: Solutions Architect, Networking, IaaS, DataSec, Un*x, Windows
Realizes: “Culture eats technology for breakfast…” -Russ White
Reply With Quote
  #6   (View Single Post)  
Old 18th June 2017
Carpetsmoker's Avatar
Carpetsmoker Carpetsmoker is offline
Real Name: Martin
Old man from scene 24
 
Join Date: Apr 2008
Location: Dutchman living in the UK
Posts: 2,195
Default

Quote:
Originally Posted by jggimi View Post
Hello, and welcome!That's my assessment also. Perhaps your interactive use is with a Unicode-capable terminal such as xterm(1), and cron(1) runs in a standard shell which may not have LC_CTYPE set.Python Unicode handling. See https://stackoverflow.com/questions/...ion-20/9942822
That Stack Overflow question is pretty out of date; most of the top answers deal with Python 2 only.
Unicode handling was significantly changed in Python 3, which the OP seems to be using (in fact, it was the entire reason for starting Python 3 in the first place).

At any rate, back to the problem, try setting the PYTHONIOENCODING" environment variable to "utf-8" .
__________________
UNIX was not designed to stop you from doing stupid things, because that would also stop you from doing clever things.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
The Woman Who Captured Snowden J65nko News 8 8th February 2015 07:57 PM
cron newsyslog message starbuck FreeBSD General 2 6th August 2008 07:33 PM
Cron running but not working stukov Other BSD and UNIX/UNIX-like 26 24th July 2008 01:17 PM
Please help me automate getmail with cron cssgalactic FreeBSD General 2 9th July 2008 10:13 PM
Using sendmail in a cron job erehwon OpenBSD General 6 15th May 2008 09:03 PM


All times are GMT. The time now is 08:51 PM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick