python traceback captured when run from cron

[lenlynch]

Looking to improve my understanding of why the following situation occurs.

I'm running OpenBSD 6.1 -stable amd64.
Binary ports package of urlwatch installed using pkg_add, v2.6 (thanks @sthen !)
I used to run i386, which can be a different set and versions of ports.
I did move my urlwatch version 1 data set from a different machine (running OpenBSD 6.0 i386), read the upgrade notes on github README.md, and followed them.

I run urlwatch interactively as a standard user and as a daily cron job as my standard user.

When urlwatch reports changes under cron execution, I have it configured for email notification.
This works as expected, to OpenSMTPD for local delivery.

After the change event report, the following traceback is captured as part of the cron job's execution.

Traceback (most recent call last):
  File "/usr/local/bin/urlwatch", line 101, in <module>
    urlwatch_command.run()
  File "/usr/local/lib/python3.6/site-packages/urlwatch/command.py", line 167, in run
    self.urlwatcher.close()
  File "/usr/local/lib/python3.6/site-packages/urlwatch/main.py", line 96, in close
    self.report.finish()
  File "/usr/local/lib/python3.6/site-packages/urlwatch/handler.py", line 125, in finish
    ReporterBase.submit_all(self, self.job_states, duration)
  File "/usr/local/lib/python3.6/site-packages/urlwatch/reporters.py", line 92, in submit_all
    subclass(report, cfg, job_states, duration).submit()
  File "/usr/local/lib/python3.6/site-packages/urlwatch/reporters.py", line 315, in submit
    print(self._red(line))
UnicodeEncodeError: 'ascii' codec can't encode character '\xdc' in position 2: ordinal not in range(128)

This traceback appears to be complaining about unicode characters being returned instead of ascii characters for the checked item? Have I got that right?

I would not have noticed these messages if they weren't captured by the cron itself.
I have run urlwatch interactively many times in the past, and I don't ever recall seeing traceback output while doing that.
I get that I can silence the traceback details (using shell redirection) but I'm not sure that this is what I want to do as yet.

Questions that this raised:

Was I just unlucky enough to miss observing a traceback when I ran it interactively in the past, based on the changed item data returned?

If tracebacks like this are expected/normal in the output, is that due to python, urlwatch or one of its dependent packages?

Is cron's execution of programs fundamentally different from an interactive user in important ways? If so, what are they?

Anything else I should be looking at on my system configuration that could be attributing to this issue?


Thanks in advance for any insight or pointers to more details!

[jggimi]

Hello, and welcome!

Originally Posted by lenlynch

This traceback appears to be complaining about unicode characters being returned instead of ascii characters for the checked item? Have I got that right?

That's my assessment also. Perhaps your interactive use is with a Unicode-capable terminal such as xterm(1), and cron(1) runs in a standard shell which may not have LC_CTYPE set.

If tracebacks like this are expected/normal in the output, is that due to python, urlwatch or one of its dependent packages?

Python Unicode handling. See https://stackoverflow.com/questions/...ion-20/9942822

[lenlynch]

Thank you jggimi!

Originally Posted by jggimi

Perhaps your interactive use is with a Unicode-capable terminal such as xterm(1), and cron(1) runs in a standard shell which may not have LC_CTYPE set.

I do SSH using a client that is configured for UTF-8 support, so that is indeed something to follow-up on. Great point.

I do see this with regards to X/xenocara and xterm, which provides some outline to the problem of there being a protocol specific issue of negotiating character encodings between different systems...

http://undeadly.org/cgi?action=artic...20160308204011

It also points to the larger concern of data handling safety, which can not be guaranteed between non-OpenBSD systems interacting.

LOCALE(1) manpage hints at parts of the system are not UTF-8 friendly:

     A locale is a set of environment variables telling programs which
     character encoding, language and cultural conventions the user prefers.
     Programs in the OpenBSD base system ignore the locale except for the
     character encoding, and it is not recommended to use any of these
     variables except that the following non-default setting is supported as
     an option:

           export LC_CTYPE=en_US.UTF-8
...

     The OpenBSD base system supports two locales: the default of LC_CTYPE=C
     selects the US-ASCII character set and encoding, treating the bytes 0x80
     to 0xff as non-printable characters of application-specific meaning.
     LC_CTYPE=POSIX is an alias for LC_CTYPE=C.  The alternative of
     LC_CTYPE=en_US.UTF-8 selects the UTF-8 encoding of the Unicode character
     set, which is supported by many parts of the system, but not yet fully
     supported by all parts.

Updated this post, there was an error in my conclusion, which was removed, sorry in advance.

Has anyone heard-of/experienced security issues with cron and UTF-8 ?

Thanks in advance.

[jggimi]

It appears cron() has locale support. Jan Stary posted this in tech@ last November recommending its removal - it did not receive any follow up posts and locale is still hooked into cron.c:

http://openbsd-archive.7691.n7.nabbl...7742.html#none

I've never set locale values with cron jobs, I don't normally use locale settings on terminals, I don't use urlwatch, and I'm not a Python programmer. So I'll step back and let others weigh in on a possible circumvention or solution.

[lenlynch]

Thank you for attention, feedback and pointers. Much appreciated.

[Carpetsmoker]

Originally Posted by jggimi

Hello, and welcome!That's my assessment also. Perhaps your interactive use is with a Unicode-capable terminal such as xterm(1), and cron(1) runs in a standard shell which may not have LC_CTYPE set.Python Unicode handling. See https://stackoverflow.com/questions/...ion-20/9942822

That Stack Overflow question is pretty out of date; most of the top answers deal with Python 2 only.
Unicode handling was significantly changed in Python 3, which the OP seems to be using (in fact, it was the entire reason for starting Python 3 in the first place).

At any rate, back to the problem, try setting the PYTHONIOENCODING" environment variable to "utf-8" .