DaemonForums  

Go Back   DaemonForums > Miscellaneous > General software and network

General software and network General OS-independent software and network questions, X11, MTA, routing, etc.

Reply
 
Thread Tools Display Modes
  #1   (View Single Post)  
Old 8th June 2008
gosha gosha is offline
Spam Deminer
 
Join Date: Jun 2008
Location: China
Posts: 256
Thanked 0 Times in 0 Posts
Default xkb multibyte characters input problem

I just posted in bsdforum, but found out most people moved here.

> Hello folks,
> this time before writing I took my very long time to study and read and try hard to solve it myself, but I'm still stuck.
> Please help
> I really want get rid of Windows, but before I can do it I need all the multilanguage functions it has, it is a must for me.
> I don't care about a full locale, but I need to get multi-bytes carachers input and display.
> At the same time I don't want to have a Windows-like gui, so, no KDE, no GNOME (I guess a lot of my problems would be already solved with them).
> I'm running OpenBSD 4.3 on a laptop (HP Omnibook 900b). I'm sticking to OpenBSD because in my very (VERY) numerous installs to try out what's best for me, the only one that went really smooth and where everything (hardware-wise) works is OpenBSD.
> I'm running: Ion3 as a window manager, Mlterm as terminal, Mined as an editor for multi-language scripts, w3m as a browser, Pidgin for chats, Alpine for imap emails, Postgresql for database (MS Access has screwed up my tiny database, one of the reasons why I'm in a rush to switch. And for my little problem with postgres, see my other thread).
> So, what's the problem?
> First problem:
> I managed to get chinese characters display and input also in terminal, but filenames in chinese are not displayed even after using "convmv" to get them all to utf-8(the encoding I'm using mlterm with), that is, they get displayed by convmv but then disappear again if I do "ls".
> But, strangely enough, if I know the filename I can type it in and get it recognized and opened. Actually, this is not the biggest problem, I'd be ready to change all my filenames to latin letters.
> Second problem:
> I'm using xkb to switch keyboard layouts, but once switched the characters don't work in terminal (that is, after switching keyboard, typing results in no response for russian, and in funny garbage for accented characters), But they work in Firefox in the Google search. So it is probably a misconfiguration of mine, but I've tryed them all without success. The same problem applies to w3m and alpine, where I can input chinese but not cyrillic or accented characters.
>
> Here is the sections of xorg.conf that might help ("wqy" is wenquanyi, chinese bitmap font, very nice):
>
> Section "Files"
> RgbPath "/usr/X11R6/share/X11/rgb"
> ModulePath "/usr/X11R6/lib/modules"
> FontPath "/usr/X11R6/lib/X11/fonts/misc/"
> FontPath "/usr/X11R6/lib/X11/fonts/wqy/"
> FontPath "/usr/X11R6/lib/X11/fonts/cyrillic/"
> FontPath "/usr/X11R6/lib/X11/fonts/TTF/"
> FontPath "/usr/X11R6/lib/X11/fonts/OTF"
> FontPath "/usr/X11R6/lib/X11/fonts/Type1/"
> FontPath "/usr/X11R6/lib/X11/fonts/100dpi/:unscaled"
> FontPath "/usr/X11R6/lib/X11/fonts/75dpi/:unscaled"
> FontPath "/usr/local/lib/X11/fonts/ghostscript/"
> FontPath "/usr/local/lib/X11/fonts/mscorefonts/"
> EndSection
>
> Section "Module"
> Load "GLcore"
> Load "dbe"
> Load "extmod"
> Load "glx"
> Load "record"
> Load "xtrap"
> Load "freetype"
> Load "type1"
> EndSection
>
> Section "InputDevice"
> Identifier "Keyboard0"
> Driver "kbd"
> Option "XkbRules" "xorg"
> Option "XkbModel" "pc101"
> Option "XkbLayout" "us,it,ru(winkeys)"
> Option "XkbOptions" "grp:lwin_toggle"
> Option "XkbVariant" "nodeadkeys"
> EndSection
>
> Here is Mlterm configuration files:
> main:
> ENCODING = UTF-8
> fontsize=12
> input_method = uim
> scrollbar_view_name = simple
> line_space = 1
>
> font:
>ISO10646_UCS4_1=-schumacher-clean-medium-r-normal-*-12-120-75-75-*-*-iso10646-1;
> ISO10646_UCS4_1_BIWIDTH=-wenquanyi-wenquanyi bitmap song-medium-r-normal-*-12-120-75-75-p-*-iso10646-1:120;
>
> .inputrc:
> set convert-meta off
> set input-meta on
> set output-meta on
>
> .profile:
> # $OpenBSD: dot.profile,v 1.4 2005/02/16 06:56:57 matthieu Exp $
> #
> # sh/ksh initialization
>
> PATH=$HOME/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/X11R6/bin:/usr/local/bin:/usr/local/sbin:/usr/games:.
> export PATH HOME TERM
> export PKG_PATH=ftp://ftp.kaist.ac.kr/pub/OpenBSD/4.3/packages/i386/:ftp://ftp.btradianz.se/pub/OpenBSD/.../packages/i386/
> export GDK_USE_XFT=0
> set +o emacs-usemeta
>
> I hope I did not forget anything
> Thank you in advance.
Reply With Quote
  #2   (View Single Post)  
Old 8th June 2008
lvlamb's Avatar
lvlamb lvlamb is offline
Real Name: Louis V. Lambrecht
Spam Deminer
 
Join Date: May 2008
Location: .be
Posts: 221
Thanked 25 Times in 24 Posts
Default

The operating system locale is different from the X locale.
So, if you chose utf-8 in X, and if your applications support utf-8, you should have no problem re-reading the the filenames correctly with another application supporting utf-8.

On xterm or on the CLI, when you ls, you will read the names of the filesystem in a 8-bit raw format.
This is ASCII-7bit, which is a subset of utf-8, and and 8th bit containing extended characters. These depend on the OS set locale (.profile, .login, ... etc).

From stock, X (and xenocara) now include luit (man luit) which act as a filter between two character sets. Could help you, even on the CLI.

Also, xterm can be called as uxterm for unicode environments.
Many other graphical terminals are Unicode and|or UTF-8 aware.

Basically, if you stay under a window manager with utf-8 aware applications, your display should be correct.

Here are some links to utf-8 patches (OS and file system)
http://web.archive.org/web/200406041...tch-src_citrus
http://sigsegv.s25.xrea.com/distfiles/citrus/OpenBSD/

If you use GNOME
The xkb definitions now are under /etx/X11/xkb but gnome seems to be searching under
/usr/X11R6/lib/X11/xkb

you might need to
ln -s /etc/X11/xkb /usr/X11R6/lib/X11/xkb
the get the keyboard indicator applet

Just rememner that OS and file system locales are different from Xorg and xlocales.
__________________
da more I know I know I know nuttin'
Reply With Quote
  #3   (View Single Post)  
Old 8th June 2008
gosha gosha is offline
Spam Deminer
 
Join Date: Jun 2008
Location: China
Posts: 256
Thanked 0 Times in 0 Posts
Default

Thanks for you long answer.
I really did not know the system locale is different from the one in X.
And where should I set it? In .xinitrc?
I knew about citrus, I tried it many times, but something always went wrong. This machine is slow and it takes a whole day to rebuild, so I'm not going to try that again.

To tell the truth, I still did not understand what I should do exactly.

And actually in the first post I did forget something. I can only input filenames in chinese in bash, and not in ksh. But no russian there also.

Last edited by gosha; 8th June 2008 at 10:40 PM.
Reply With Quote
  #4   (View Single Post)  
Old 9th June 2008
lvlamb's Avatar
lvlamb lvlamb is offline
Real Name: Louis V. Lambrecht
Spam Deminer
 
Join Date: May 2008
Location: .be
Posts: 221
Thanked 25 Times in 24 Posts
Default

System locale
usually set system wide in /etc/login.conf
for users, in their ~/.profile
Note for ksh
FAQ 8.16 - ksh(1) does not appear to read my .profile!
Code:
ls -d /usr/share/locale/zh*
X locale
Normally in ~/.Xdefaults
Code:
ls -d /usr/X11R6/share/X11/locale/zh*
Terminal applications:
man luit
This said, I have no way to check if a /usr/share/locale/zh_TW.eucTW
can be filtered into a /usr/X11R6/share/X11/locale/zh_TW.UTF-8


X applications:
as you did for mlterm
for each X application, most of the time the application itself can set the encodings (thunderbird, firefox, gnumeric, gedit, ...).
Other times, the application has a dot file or directory.
See
http://eyegene.ophthy.med.umich.edu/unicode/
for ideas.
I like the small wrapper scripts they give as example.

Not tested but .xinitrc being a list of shell commands, it should be possible to setenv or env to a locale.

I assume you would get better answers on Chinese OpenBSD forums
__________________
da more I know I know I know nuttin'
Reply With Quote
  #5   (View Single Post)  
Old 9th June 2008
gosha gosha is offline
Spam Deminer
 
Join Date: Jun 2008
Location: China
Posts: 256
Thanked 0 Times in 0 Posts
Default

Thanks again, I'll give it a try now.
Actually, the main problem for me is not Chinese, because I can still make an alias for ls and get my filenames: (ls | iconv -f gb2312 -t utf-8 *). And by the way, I tried to download pfm, and it displays correctly the filenames in chinese converted to utf-8, so either I'm misconfiguring mlterm, either there's something wrong with it.
My biggest problem, the one I cannot resolve, is my cyrillic input through xkb. Where's the problem in my configuration? I need to input cyrillic and accented characters in terminal (basically, for Alpine and psql).
Reply With Quote
  #6   (View Single Post)  
Old 9th June 2008
lvlamb's Avatar
lvlamb lvlamb is offline
Real Name: Louis V. Lambrecht
Spam Deminer
 
Join Date: May 2008
Location: .be
Posts: 221
Thanked 25 Times in 24 Posts
Default

GNOME has the keyboard switcher applet,
Xfce has one, but does not switches: use the keymap of the xorg.conf,
KDE? never tried but almost sure there is a setting somewhere.

These WM applets should take precedence over other choices.
In any case, I would use, as an extra, the ~/.Xdefaults tip from the FAQ.

----
edit
Have been reading
man keyboard
about the layout lists in
Option "XkbLayout"
found no answer but
Quote:
Some other XKB-related options are available, but they are
incompatible with the ones listed above and are not recom-
mended, so they are not documented here.
So, I'm rather at the end of my ideas.
__________________
da more I know I know I know nuttin'

Last edited by lvlamb; 9th June 2008 at 02:28 AM.
Reply With Quote
  #7   (View Single Post)  
Old 9th June 2008
gosha gosha is offline
Spam Deminer
 
Join Date: Jun 2008
Location: China
Posts: 256
Thanked 0 Times in 0 Posts
Smile

As I said in my firs post, I'm not willing to run a whole desktop like Gnome or Kde. Actually I tried, but this machine is far too old for it. And I don't really want to switch from a Windows gui to another gui.
Also, the problem is not in switching the keyboard layout, it does switch, but once switched I cannot input the extra characters. That is, under russian keyboard if I press the keys for "." or "," and the likes which are commom, I get them printed in terminal, but if I press the letters nothing comes out. The same for accented characters, if I press "è" I get nothing, or funny garbage. But as you see, here in firefox (which eats up over 100 M of my 190 M ram...) it works, but in terminal-based applications it does not.
This is the problem I'm stuck with.


In case it should be of any help, this is what xev reports when I press "a" under russian keyboard layout:

KeyPress event, serial 25, synthetic NO, window 0x1400001,
root 0x46, subw 0x0, time 1804070102, (60,2), root576,318),
state 0x4000, keycode 38 (keysym 0x6c6, Cyrillic_ef), same_screen YES,
XLookupString gives 0 bytes:
XmbLookupString gives 0 bytes:
XFilterEvent returns: False

KeyRelease event, serial 25, synthetic NO, window 0x1400001,
root 0x46, subw 0x0, time 1804070229, (60,2), root576,318),
state 0x4000, keycode 38 (keysym 0x6c6, Cyrillic_ef), same_screen YES,
XLookupString gives 0 bytes:
XFilterEvent returns: False

Thank you anyway for your help.

Last edited by gosha; 9th June 2008 at 04:22 AM.
Reply With Quote
  #8   (View Single Post)  
Old 9th June 2008
gosha gosha is offline
Spam Deminer
 
Join Date: Jun 2008
Location: China
Posts: 256
Thanked 0 Times in 0 Posts
Cool solved in part

We are getting there.
Apparently the .Xdefaults line for ksh did the trick. Now I can input Russian in uxterm (not in xterm). But now I cannot display correctly Chinese in uxterm (and still no russian in mlterm).
I'll keep on tricking around, I'm sure I'll get there somehow. At least I have now a partial solution using two different terminals.
Thanks a lot.
Reply With Quote
  #9   (View Single Post)  
Old 13th June 2008
gosha gosha is offline
Spam Deminer
 
Join Date: Jun 2008
Location: China
Posts: 256
Thanked 0 Times in 0 Posts
Default still stuck

I think I've tried them all. But the situation does not change.
Now I can input chinese in mlterm but not in xterm or uxterm, although uim is started in .xinitrc. At the same time in mlterm still cannot input either russian or different characters (but I can display them, for example, if I go to a russian website with w3m it is displayed, but I cannot make a search using cyrillic).
I tried to install mlterm's termcap and terminfo from source. I've tried to make a similar line for ksh in .Xdefaults for mlterm. I've tried to put in ~/.mlterm/main this line: use_login_shell = true. I've tried to change fonts, or encodings.
Nothing works.
If I use xemacs from mlterm, it can input russian, but with the old keyboard layout, not with winkeys like I put in my xorg.conf, so I guess emacs is using its own input? As I said, the only application so far that behaves as expected is firefox where I can input chinese, cyrillic or accented characters all according to my settings.
It must be a misconfiguration of mine, but I already have no idea where to look for an aswer.
Any help will be greatly appreciated.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Input foreign characters under X11 Beastie General software and network 5 30th August 2009 11:51 AM
sorting special characters gosha Programming 15 9th April 2009 02:29 AM
Username longer than 16 characters _hmp_ FreeBSD General 5 13th January 2009 10:01 AM
C and file input/output 18Googol2 Programming 3 20th August 2008 04:02 PM
shell script with input c0mrade Programming 5 13th July 2008 04:33 AM


All times are GMT. The time now is 10:43 PM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick