encoding issues

kevin_Thackray · September 8, 2004, 3:20pm

hi everyone,

I went throught a tricky encoding problem with the maskededit component:

I have a string with some iso-8859-1 encoding (value , type 'str') which i apply the method SetValue(str) , and i raise this error :

  File "/home/kevin/cvs/pylaf/pylaf/client/wx/form/ctrl.py", line 141, in setValue
    self._ctrl.SetValue(str(value).decode('iso-8859-15'))
  File "/home/francois/tmp/pkg/usr/lib/python2.3/site-packages/wx/lib/masked/textctrl.py", line 200, in SetValue
  File "/usr/lib/python2.3/site-packages/wx/lib/masked/maskededit.py", line 5568, in _Paste
    valid_paste, replacement_text, replace_to = self._validatePaste(paste_text, sel_start, sel_to, raise_on_invalid)
  File "/usr/lib/python2.3/site-packages/wx/lib/masked/maskededit.py", line 5424, in _validatePaste
    if not self._isTemplateChar(replace_to) and self._isCharAllowed( char, replace_to, allowAutoSelect=False, ignoreInsertRight=True):
  File "/usr/lib/python2.3/site-packages/wx/lib/masked/maskededit.py", line 4385, in _isCharAllowed
    approved = char in okChars
UnicodeDecodeError: 'ascii' codec can't decode byte 0xaa in position 52: ordinal not in range(128)

I've tried lots of things, and the only working solution i came with, is patching the maskededit.py (line 4385):
* former version of maskededit.py :
> approved = char in okChars
* Patched version :
>if type(char) is types.UnicodeType:
> char = char.encode("iso-8859-1")
>approved = char in okChars

My working version of wx is : 2.5.2.7
As I am not a wx expert, is a patch the right solution (never the less, not mine as i butaly hard-code the encoding), or do i miss something??

regards,

Kevin Thackray

John_A_Meinel1 · September 8, 2004, 3:21pm

kevin Thackray wrote:

hi everyone,

I went throught a tricky encoding problem with the maskededit component:

I have a string with some iso-8859-1 encoding (value , type 'str') which i apply the method SetValue(str) , and i raise this error :

I've tried lots of things, and the only working solution i came with, is patching the maskededit.py (line 4385):
* former version of maskededit.py :
> approved = char in okChars
* Patched version :
>if type(char) is types.UnicodeType:
> char = char.encode("iso-8859-1")
>approved = char in okChars

My working version of wx is : 2.5.2.7
As I am not a wx expert, is a patch the right solution (never the less, not mine as i butaly hard-code the encoding), or do i miss something??

regards,

Kevin Thackray

What you really want is:

SetValue(str.decode('iso-8859-1'))

str.decode yields a unicode string, decoded from some other encoding. str.encode takes a unicode string and converts it *into* the given encoding.

John
=:->

kevin_Thackray1 · September 8, 2004, 3:50pm

hi john and everyone,

Thanks you for your quick answer, but, if I a understood your answer it is not my solution.
To get that strait, when you say 'str' you meant a string and not the str built-in function??!!
SetValue(str.decode('iso-8859-1'))

Here is my backtrace :

File "/home/kevin/cvs/pylaf/pylaf/client/wx/form/ctrl.py", line 142, in
setValue

self._ctrl.SetValue(value.decode('iso-8859-15')) <======= (former SetValue(str(value).decode('iso-8859-1')))

  File"/home/francois/tmp/pkg/usr/lib/python2.3/site-packages/wx/lib/masked/textctrl.py", line 200, in SetValue
  File "/usr/lib/python2.3/site-packages/wx/lib/masked/maskededit.py",line 5568, in _Paste valid_paste, replacement_text, replace_to = self._validatePaste(paste_text, sel_start, sel_to, raise_on_invalid)
  File "/usr/lib/python2.3/site-packages/wx/lib/masked/maskededit.py", line 5424, in _validatePaste
    if not self._isTemplateChar(replace_to) and self._isCharAllowed(
char, replace_to, allowAutoSelect=False, ignoreInsertRight=True):
  File "/usr/lib/python2.3/site-packages/wx/lib/masked/maskededit.py",
line 4385, in _isCharAllowed
    approved = char in okChars
UnicodeDecodeError: 'ascii' codec can't decode byte 0xaa in position 52:
ordinal not in range(128)

still diging... but my answer might be in the 'okChars', but i thought i
figure that out with :

locale.setlocale(locale.LC_ALL, 'fr_FR')

and my textCtrl mask factory instanciate the object like this :
masked.TextCtrl( parent, wId, '', mask = 'X', formatcodes = formatcodes)

ARRRRGG i am really getting mad with encoding

regards,

kevin thackray

John_A_Meinel1 · September 8, 2004, 3:56pm

kevin Thackray wrote:

hi john and everyone,

Thanks you for your quick answer, but, if I a understood your answer it is not my solution.
To get that strait, when you say 'str' you meant a string and not the str built-in function??!!
SetValue(str.decode('iso-8859-1'))

Here is my backtrace :

File "/home/kevin/cvs/pylaf/pylaf/client/wx/form/ctrl.py", line 142, in
setValue

self._ctrl.SetValue(value.decode('iso-8859-15')) <======= (former SetValue(str(value).decode('iso-8859-1')))

[...]

still diging... but my answer might be in the 'okChars', but i thought i
figure that out with :

locale.setlocale(locale.LC_ALL, 'fr_FR')

and my textCtrl mask factory instanciate the object like this :
masked.TextCtrl( parent, wId, '', mask = 'X', formatcodes = formatcodes)

ARRRRGG i am really getting mad with encoding

regards,

kevin thackray

Okay, I guess I need to figure out the steps that you are doing.

You have a string called 'value', and you are trying to put that value into the maskededit component.

First, what platform are you on, and are you using the Unicode version of wxPython?

I can't say too much about maskedit, I haven't used it. I *have* had some encoding issues, though, so I thought I could help there.

You shouldn't need to do str(value).decode, if value is already a 'str'. value.decode should work fine.

I realize I didn't look quite closely enough, and now I see that you were in fact decoding it already.

I'm guessing that maskedit is expecting certain characters and shouldn't be. Sorry I can't help you further. Hopefully someone else knows more about maskedit.py

John
=:->

Robin · September 8, 2004, 4:12pm

John Meinel wrote:

I'm guessing that maskedit is expecting certain characters and shouldn't be. Sorry I can't help you further. Hopefully someone else knows more about maskedit.py

We've been discussing this on the wxPython-dev list, but more input is always welcome. The fundamental issue is that the MaskedEdit control assumes strings, but in a unicode build anything going to or from the underlying textctrl is a unicode object and so you run into encoding issues as soon as you try to use something that is not in the default ascii encoding.

My proposal is to make the MaskedEdit control do everything internally with unicode objects when on a unicode build of wxPython and push the responsibility of encoding/decoding out to the programmer to deal with. I don't know how feasible implementing that will be but I think Will is looking into it. If anybody else has any ideas now would be the time to bring them up.

···

--
Robin Dunn
Software Craftsman
http://wxPython.org Java give you jitters? Relax with wxPython!

Werner_F_Bruhin1 · September 8, 2004, 4:25pm

Hi Kevin,

I am using maskedit all over the place and use just the following "locale" in my main module.

locale.setlocale(locale.LC_ALL, '')

I have no problems writing things like "éçöü" etc etc into the controls either using the keyboard or using validators to get things from the screen to the db or from the db to the controls.

BUT, I am on wxPython 2.4.x, haven't moved to 2.5 yet as I wait for Boa to completely support it, and as I frankly didn't have time yet.

What about putting a small sample together showing the problem. I could try it out under 2.4.

See you
Werner

kevin Thackray wrote:

···

hi john and everyone,

Thanks you for your quick answer, but, if I a understood your answer it is not my solution.
To get that strait, when you say 'str' you meant a string and not the str built-in function??!!
SetValue(str.decode('iso-8859-1'))

Here is my backtrace :

File "/home/kevin/cvs/pylaf/pylaf/client/wx/form/ctrl.py", line 142, in
setValue

   self._ctrl.SetValue(value.decode('iso-8859-15')) <======= (former SetValue(str(value).decode('iso-8859-1')))

File"/home/francois/tmp/pkg/usr/lib/python2.3/site-packages/wx/lib/masked/textctrl.py", line 200, in SetValue
File "/usr/lib/python2.3/site-packages/wx/lib/masked/maskededit.py",line 5568, in _Paste valid_paste, replacement_text, replace_to = self._validatePaste(paste_text, sel_start, sel_to, raise_on_invalid)
File "/usr/lib/python2.3/site-packages/wx/lib/masked/maskededit.py", line 5424, in _validatePaste
   if not self._isTemplateChar(replace_to) and self._isCharAllowed(
char, replace_to, allowAutoSelect=False, ignoreInsertRight=True):
File "/usr/lib/python2.3/site-packages/wx/lib/masked/maskededit.py",
line 4385, in _isCharAllowed
   approved = char in okChars
UnicodeDecodeError: 'ascii' codec can't decode byte 0xaa in position 52:
ordinal not in range(128)

still diging... but my answer might be in the 'okChars', but i thought i
figure that out with :

locale.setlocale(locale.LC_ALL, 'fr_FR')

and my textCtrl mask factory instanciate the object like this :
masked.TextCtrl( parent, wId, '', mask = 'X', formatcodes = formatcodes)

ARRRRGG i am really getting mad with encoding

regards,

kevin thackray

---------------------------------------------------------------------
To unsubscribe, e-mail: wxPython-users-unsubscribe@lists.wxwidgets.org
For additional commands, e-mail: wxPython-users-help@lists.wxwidgets.org

Robin · September 8, 2004, 9:48pm

Werner F. Bruhin wrote:

Hi Kevin,

I am using maskedit all over the place and use just the following "locale" in my main module.

locale.setlocale(locale.LC_ALL, '')

I have no problems writing things like "éçöü" etc etc into the controls either using the keyboard or using validators to get things from the screen to the db or from the db to the controls.

Unicode or ansi build of wxPython? (I'm guessing the latter, which is why you don't run into the unicode problem...)

···

--
Robin Dunn
Software Craftsman
http://wxPython.org Java give you jitters? Relax with wxPython!

kevin_Thackray · September 9, 2004, 8:16am

hi everyone,

Hi Kevin,

I am using maskedit all over the place and use just the following "locale" in my main module.

locale.setlocale(locale.LC_ALL, '')

I have no problems writing things like "éçöü" etc etc into the controls either using the keyboard or using validators to get things from the screen to the db or from the db to the controls.

Unicode or ansi build of wxPython? (I'm guessing the latter, which is why you don't run into the unicode problem...)

As I said before, i am using wxPython 2.5.2.7 with unicode supports. My example is very simple , let my get iot straight :

locale.setlocale(locale.LC_ALL, 'fr_FR')
....
ctr = masked.TextCtrl(....)
....
value = "éééééààààùùùù"
ctrl.SetValue(value.decode('iso-8859-15'))

and that raise me this exception :
UnicodeDecodeError: 'ascii' codec can't decode byte 0xaa in position 52: ordinal not in range(128) (cf. other mail for complet trace back)

Question for Rodin dunn : what do you think about my patch, even if, it is just a work-around to try to solve this problem. Do you thing that the deepest layer (maskededit.py) of wx should handle "str" object or unicode object??

Regards,

Kevin Thackray

Werner_F_Bruhin1 · September 9, 2004, 8:27am

Hi Robin,

Robin Dunn wrote:

Werner F. Bruhin wrote:

Hi Kevin,

I am using maskedit all over the place and use just the following "locale" in my main module.

locale.setlocale(locale.LC_ALL, '')

I have no problems writing things like "éçöü" etc etc into the controls either using the keyboard or using validators to get things from the screen to the db or from the db to the controls.

Unicode or ansi build of wxPython? (I'm guessing the latter, which is why you don't run into the unicode problem...)

You are right, I am on ansi build, didn't catch on that Kevin is using Unicode and saw your message too late.

See you
Werner

Robin · September 10, 2004, 12:44am

kevin Thackray wrote:

Question for Rodin dunn : what do you think about my patch, even if, it is just a work-around to try to solve this problem.

The problem with using a specific encoding within the masked modules is that somebody is eventually going to want to use a different one.

Do you thing that the deepest layer (maskededit.py) of wx should handle "str" object or unicode object??

My thought is that when running on a unicode build of wxPython then it should use all unicode objects interally, but it's up to Will.

···

--
Robin Dunn
Software Craftsman
http://wxPython.org Java give you jitters? Relax with wxPython!

Ricardo_Pedroso · September 20, 2004, 2:07am

By the begining of this month there was a thread about encoding issues in
maskedControls.
Finally, I think, I came up with a solution (thanks to boa-constructor
documentation).
What I did was to put the,
  import locale
  locale.setlocale(locale.LC_ALL, '')
stuff like the maskededit documention says and create a sitecustomize.py
file side-by-side with my program's main file with,
  import sys
  sys.setdefaultencoding('iso-8859-15')

This sitecustomize.py file could be put on python path, if it should be
used in all python modules.

My question:
Are there any (other) issues with this approach???

PS: wxpython 2.4.2.4 (unicode) on a winXP.

Ricardo