Unicode troubles with wxPropertyGrid

I'm using wxPropertyGrid through wxPython for a program I'm working on and
have been having heaps of trouble getting it to work with unicode.

After a lot of head scratching and learning more about unicode than I ever
wanted to know I think I've managed to work out what's happening though I'm
clueless as to why! When I create a PGProperty and set its value, passing it
a unicode object, it seems to be encoding it to bytes using the utf-8 codec,
and then decoding it to unicode using latin-1 or something similar... This
doesn't cause problems for code points below 127 but anything higher gets
mangled. If my understanding is correct then this would be roughly
appropriate behaviour (except for the choice of 'latin-1' codec) if I were
passing it a string not a unicode object, but it is definitely unicode!

At this point I would love to provide you with minimal sample code
demonstrating this behaviour, but the code below which is meant to do just
that seems to work fine! If I just copy the same string into my program it
behaves as described above though!

I'm using Python 2.7.2 and wxPython 2.9.2.4 on OSX. I’m running it through
PyDev on eclipse and the default encoding is set to utf-8. I’ve also set
wx.SetDefaultPyEncoding and wx.Font.SetDefaultEncoding to utf-8 to no avail.

I appreciate that the information I’ve provided probably isn’t massively
helpful but if anyone has any thoughts about possible causes I’d be grateful
for any help you can give me. In the meantime I'll keep banging my head
against it in the hope that something useful falls out!

Cheers,

John

Sample code, requires python default encoding to be utf-8:

# coding=utf-8
import wx
import wx.propgrid as pg

app = wx.App(False)
wx.SetDefaultPyEncoding('utf-8')
dlg = wx.Dialog(None)

grid = pg.PropertyGrid(dlg)

value = u'§˚¨†ßƒ∂^'
# This should print the correct value
print value

sprop = pg.StringProperty('String Property', 'sprop', value=value)
# This should be wrong; specifically it should be '§˚¨†ßƒ∂^'
print sprop.GetValue()

# But doing this should get you back to the original string
# value = bytes(value)
# value = value.encode('latin-1')
# value = value.decode('utf-8')
# print value # gives -> '§˚¨†ßƒ∂^'

def handler(event):
    property = grid.GetSelection()
    # This should be correct
    print property.GetValue()

grid.Bind(pg.EVT_PG_CHANGED, handler)
grid.Append(sprop)
dlg.ShowModal()

···

--
View this message in context: http://wxpython-users.1045709.n5.nabble.com/Unicode-troubles-with-wxPropertyGrid-tp5713345.html
Sent from the wxPython-users mailing list archive at Nabble.com.

after further investigating I've found that it's doing the same thing when I
apply a unicode value to any label or field.

···

--
View this message in context: http://wxpython-users.1045709.n5.nabble.com/Unicode-troubles-with-wxPropertyGrid-tp5713345p5713351.html
Sent from the wxPython-users mailing list archive at Nabble.com.

Right I've finally managed to reproduce it outside of my program, and it's
weird. All the problems start after showing a file dialog for some
reason... Hopefully someone else will be able to reproduce this and confirm
that I'm not losing my mind!

Run the following under wxMac Carbon 2.9.2.4:

# coding=utf-8
import wx
import wx.propgrid as pg

app = wx.App(False)

def show_dlg():
    dlg = wx.Dialog(None)
    
    grid = pg.PropertyGrid(dlg)
    grid.SetSize((400,400))
    
    value = u'§˚¨†ßƒ∂^'
    sprop = pg.StringProperty('String Property', 'sprop', value=value)
    
    def handler(event):
        property = grid.GetSelection()
        print property.GetValue()
    
    grid.Bind(pg.EVT_PG_CHANGED, handler)
    
    grid.Append(sprop)
    
    dlg.ShowModal()
    
# This call shows the unicode characters correctly...
show_dlg()

# Show this file dialog and select cancel or OK
with wx.FileDialog(
                           None, message="New Project",
                           defaultDir='',
                           defaultFile="untitled.imtproj",
                           style=wx.SAVE|wx.FD_OVERWRITE_PROMPT
                           ) as dlg1:
       
    dlg1.ShowModal()

# This should be wrong; specifically it should be '§˚¨†ßƒ∂^'
show_dlg()

···

--
View this message in context: http://wxpython-users.1045709.n5.nabble.com/Unicode-troubles-with-wxPropertyGrid-tp5713345p5713353.html
Sent from the wxPython-users mailing list archive at Nabble.com.

Does it also misbehave if you show the value in a UI element (such as wx.MessageBox) instead of printing it to the terminal?

         wx.MessageBox(property.GetValue())

The encoding that the Terminal is using is not something that we can control, and I suppose that it's possible that showing the file dialog is triggering some bug in the Terminal application.

The other side of the equation is the encoding used by Python when it auto-converts the unicode value to a string value for printing. Again that codec is not normally under our control at runtime (although there are tricks) and it can vary from system to system so it should not be relied upon to be any specific codec. Instead the better practice is to convert unicode values yourself when they are transitioning to string values (like for printing or writing to a file.) For example::

         print property.GetValue().encode('utf-8')

···

On 5/29/12 9:13 AM, John Evans wrote:

Right I've finally managed to reproduce it outside of my program, and it's
weird. All the problems start after showing a file dialog for some
reason... Hopefully someone else will be able to reproduce this and confirm
that I'm not losing my mind!

--
Robin Dunn
Software Craftsman

Robin Dunn wrote

Right I've finally managed to reproduce it outside of my program, and
it's
weird. All the problems start after showing a file dialog for some
reason... Hopefully someone else will be able to reproduce this and
confirm
that I'm not losing my mind!

Does it also misbehave if you show the value in a UI element (such as
wx.MessageBox) instead of printing it to the terminal?

         wx.MessageBox(property.GetValue())

Showing the value in a wx.MessageBox does indeed misbehave, shows the same
value as when setting the StringProperty, so clearly further reaching than
just PropertyGrid...

The encoding that the Terminal is using is not something that we can
control, and I suppose that it's possible that showing the file dialog
is triggering some bug in the Terminal application.

The other side of the equation is the encoding used by Python when it
auto-converts the unicode value to a string value for printing. Again
that codec is not normally under our control at runtime (although there
are tricks) and it can vary from system to system so it should not be
relied upon to be any specific codec. Instead the better practice is to
convert unicode values yourself when they are transitioning to string
values (like for printing or writing to a file.) For example::

         print property.GetValue().encode('utf-8')

--
Robin Dunn
Software Craftsman
http://wxPython.org

--
To unsubscribe, send email to wxPython-users+unsubscribe@
or visit http://groups.google.com/group/wxPython-users?hl=en

Yeah I think I'm going to have to go through checking any transitions to
make sure it's encoding everything explicitly, but given that it shows the
erroneous value in the UI element as well as the terminal wouldn't this
eliminate the possibility of it being a problem with the terminal encoding?
I'm not sure if I made this clear but if I enter unicode values into the UI
element it returns the correct value on a call to GetValue and prints it
correctly to the terminal, it's only when setting the value programmatically
that it's doing this weird conversion.

I'm stumped as to how I can set the UI elements to the correct values though
given that it seems determined to mess with the encodings.

I've been considering moving over to the cocoa build anyway so I'll see if
it behaves any better.

···

On 5/29/12 9:13 AM, John Evans wrote:

--
View this message in context: http://wxpython-users.1045709.n5.nabble.com/Unicode-troubles-with-wxPropertyGrid-tp5713345p5713381.html
Sent from the wxPython-users mailing list archive at Nabble.com.

I've been considering moving over to the cocoa build anyway so I'll see if
it behaves any better.

It does not

···

--
View this message in context: http://wxpython-users.1045709.n5.nabble.com/Unicode-troubles-with-wxPropertyGrid-tp5713345p5713383.html
Sent from the wxPython-users mailing list archive at Nabble.com.

Try with the 2.9.4 preview build, your sample works correctly for me with that version (at least with the cocoa build.)

  https://groups.google.com/d/topic/wxpython-dev/E_JbCS2a1gU/discussion

···

On 6/1/12 7:04 AM, John Evans wrote:

I've been considering moving over to the cocoa build anyway so I'll see if
it behaves any better.

It does not

--
Robin Dunn
Software Craftsman

Robin Dunn wrote

I've been considering moving over to the cocoa build anyway so I'll see
if
it behaves any better.

It does not

Try with the 2.9.4 preview build, your sample works correctly for me
with that version (at least with the cocoa build.)

  https://groups.google.com/d/topic/wxpython-dev/E_JbCS2a1gU/discussion

--
Robin Dunn
Software Craftsman
http://wxPython.org

--
To unsubscribe, send email to wxPython-users+unsubscribe@
or visit http://groups.google.com/group/wxPython-users?hl=en

It works! Thanks muchly for your assistance Robin.

···

On 6/1/12 7:04 AM, John Evans wrote:

--
View this message in context: http://wxpython-users.1045709.n5.nabble.com/Unicode-troubles-with-wxPropertyGrid-tp5713345p5713467.html
Sent from the wxPython-users mailing list archive at Nabble.com.

Hello,

The wxPython list is so resourceful that I dare, once again, asking a non-directly wx-related question. Note that I tried to ask on the cx-freeze list and got no answer.

I got a bug in the compiled version of my code, but not when running the python script. So:
- 'python myCode.py' works well.
- 'myCode.exe' has the bug (workflow process hanging on a progress bar)
I create 'myCode.exe' using cx_Freeze.

This is on Windows 7. I start 'myCode.exe' from a 'cmd' window. My problem is the debugging: when running 'myCode.exe', no error is shown, no Traceback, no printout.. So, it is pretty difficult (or unpossible) to find out what goes wrong. I'm pretty sure the error info goes somewhere (but where ?), or there is some kind of flag to turn off/on to make it happen. Note that I use 'redirect=True' in my wx.App().

I use cx_Freeze 4.2

Any help would be highly appreciated.

Raphael

Hi,

Hello,

The wxPython list is so resourceful that I dare, once again, asking a
non-directly wx-related question. Note that I tried to ask on the cx-freeze
list and got no answer.

I got a bug in the compiled version of my code, but not when running the
python script. So:
- 'python myCode.py' works well.
- 'myCode.exe' has the bug (workflow process hanging on a progress bar)
I create 'myCode.exe' using cx_Freeze.

This is on Windows 7. I start 'myCode.exe' from a 'cmd' window. My problem
is the debugging: when running 'myCode.exe', no error is shown, no
Traceback, no printout.. So, it is pretty difficult (or unpossible) to find
out what goes wrong. I'm pretty sure the error info goes somewhere (but
where ?), or there is some kind of flag to turn off/on to make it happen.
Note that I use 'redirect=True' in my wx.App().

I use cx_Freeze 4.2

Any help would be highly appreciated.

Have you tried using out "executable-builders" to see if the error
persists? Try and give py2exe or PyInstaller a go.

Also, are you using the multiprocessing module? If the answer is
"yes", did you include the support for freezing in your app? I.e.:

from multiprocessing import freeze_support

if __name__ == '__main__':
    freeze_support()
    MyApp()

Andrea.

"Imagination Is The Only Weapon In The War Against Reality."
http://xoomer.alice.it/infinity77/

···

On 6 June 2012 14:02, Raphael Mayoraz wrote:

Thanks Andrea,

I was missplacing 'freeze_support()'. So, this seems to fix my bug.

Now, to get the Traceback and printouts, what would you recommend: py2exe (seems that the latest version, 0.6.9 is 4 years old ???) or PyInstaller (don't they say they are some issues with Windows 7 ?).

Raphael

···

On 6/6/2012 2:23 PM, Andrea Gavana wrote:

Hi,

On 6 June 2012 14:02, Raphael Mayoraz wrote:

Hello,

The wxPython list is so resourceful that I dare, once again, asking a
non-directly wx-related question. Note that I tried to ask on the cx-freeze
list and got no answer.

I got a bug in the compiled version of my code, but not when running the
python script. So:
- 'python myCode.py' works well.
- 'myCode.exe' has the bug (workflow process hanging on a progress bar)
I create 'myCode.exe' using cx_Freeze.

This is on Windows 7. I start 'myCode.exe' from a 'cmd' window. My problem
is the debugging: when running 'myCode.exe', no error is shown, no
Traceback, no printout.. So, it is pretty difficult (or unpossible) to find
out what goes wrong. I'm pretty sure the error info goes somewhere (but
where ?), or there is some kind of flag to turn off/on to make it happen.
Note that I use 'redirect=True' in my wx.App().

I use cx_Freeze 4.2

Any help would be highly appreciated.

Have you tried using out "executable-builders" to see if the error
persists? Try and give py2exe or PyInstaller a go.

Also, are you using the multiprocessing module? If the answer is
"yes", did you include the support for freezing in your app? I.e.:

from multiprocessing import freeze_support

if __name__ == '__main__':
     freeze_support()
     MyApp()

Andrea.

"Imagination Is The Only Weapon In The War Against Reality."
http://xoomer.alice.it/infinity77/

Hi,

Hi,

Hello,

The wxPython list is so resourceful that I dare, once again, asking a
non-directly wx-related question. Note that I tried to ask on the
cx-freeze
list and got no answer.

I got a bug in the compiled version of my code, but not when running the
python script. So:
- 'python myCode.py' works well.
- 'myCode.exe' has the bug (workflow process hanging on a progress bar)
I create 'myCode.exe' using cx_Freeze.

This is on Windows 7. I start 'myCode.exe' from a 'cmd' window. My
problem
is the debugging: when running 'myCode.exe', no error is shown, no
Traceback, no printout.. So, it is pretty difficult (or unpossible) to
find
out what goes wrong. I'm pretty sure the error info goes somewhere (but
where ?), or there is some kind of flag to turn off/on to make it happen.
Note that I use 'redirect=True' in my wx.App().

I use cx_Freeze 4.2

Any help would be highly appreciated.

Have you tried using out "executable-builders" to see if the error
persists? Try and give py2exe or PyInstaller a go.

Also, are you using the multiprocessing module? If the answer is
"yes", did you include the support for freezing in your app? I.e.:

from multiprocessing import freeze_support

if __name__ == '__main__':
freeze_support()
MyApp()

Andrea.

"Imagination Is The Only Weapon In The War Against Reality."
http://xoomer.alice.it/infinity77/

Thanks Andrea,

I was missplacing 'freeze_support()'. So, this seems to fix my bug.

Now, to get the Traceback and printouts, what would you recommend: py2exe
(seems that the latest version, 0.6.9 is 4 years old ???) or PyInstaller
(don't they say they are some issues with Windows 7 ?).

I am personally happy with either one of them, so if I were you I
would just pick the one you feel more comfortable with. If and when
you stumble upon a barrier or some obscure error, you may try and give
the other tool a try.

Just remember that the single-file option for py2exe doesn't work very
well anymore, but I usually don't care about that because I repackage
my executable using InnoSetup to build a single-file installer.
PyInstaller may seem a better alternative for single-file executable,
but be careful that for projects containing many data-files, or
complex executables, or relying heavily on multiprocessing PyInstaller
may bog down your system once it unpacks the single-file executable in
a local folder and/or in memory. I have been bitten by this a couple
of times already.

Andrea.

"Imagination Is The Only Weapon In The War Against Reality."
http://xoomer.alice.it/infinity77/

···

On 7 June 2012 10:23, Raphael Mayoraz wrote:

On 6/6/2012 2:23 PM, Andrea Gavana wrote:

On 6 June 2012 14:02, Raphael Mayoraz wrote: