What is wxPython doing to the locale to makes pandas crash?

zobal · May 8, 2020, 6:32pm

When I try to import pandas into my wxPython application a “ValueError: unknown locale: en-US” exception is thrown. I can avoid this by importing pandas before the application starts, but that seems like a silly requirement.

This is easily repeatable in a simple wx app:

import wx


class TestApp(wx.App):
    def __init__(self, redirect=False, filename=None):
        wx.App.__init__(self, redirect, filename)
        import pandas

        self.frame = wx.Frame(None, size=wx.Size(670,670), title='Test App')
        self.sizer = wx.BoxSizer(wx.VERTICAL)

        self.frame.SetSizer(self.sizer)

if __name__ == '__main__':
    app = TestApp()
    app.MainLoop()

When I run this app from my virtual environment I get this stack trace:

(test) PS env_test> python .\test2.py                                                       Traceback (most recent call last):
  File ".\test2.py", line 16, in <module>
    app = TestApp()
  File ".\test2.py", line 8, in __init__
    import pandas
  File "E:\Envs\test\lib\site-packages\pandas\__init__.py", line 32, in <module>
    from pandas._libs import hashtable as _hashtable, lib as _lib, tslib as _tslib
  File "E:\Envs\test\lib\site-packages\pandas\_libs\__init__.py", line 3, in <module>
    from .tslibs import (
  File "E:\Envs\test\lib\site-packages\pandas\_libs\tslibs\__init__.py", line 3, in <module>
    from .conversion import localize_pydatetime, normalize_date
  File "pandas\_libs\tslibs\c_timestamp.pxd", line 7, in init pandas._libs.tslibs.conversion
  File "pandas\_libs\tslibs\c_timestamp.pyx", line 1, in init pandas._libs.tslibs.c_timestamp
  File "pandas\_libs\tslibs\tzconversion.pyx", line 1, in init pandas._libs.tslibs.tzconversion
  File "pandas\_libs\tslibs\timedeltas.pyx", line 1, in init pandas._libs.tslibs.timedeltas
  File "pandas\_libs\tslibs\offsets.pyx", line 1, in init pandas._libs.tslibs.offsets
  File "pandas\_libs\tslibs\ccalendar.pyx", line 13, in init pandas._libs.tslibs.ccalendar
  File "pandas\_libs\tslibs\strptime.pyx", line 625, in init pandas._libs.tslibs.strptime
  File "pandas\_libs\tslibs\strptime.pyx", line 530, in pandas._libs.tslibs.strptime.TimeRE.__init__
  File "pandas\_libs\tslibs\strptime.pyx", line 405, in pandas._libs.tslibs.strptime.LocaleTime.__init__
  File "pandas\_libs\tslibs\strptime.pyx", line 362, in pandas._libs.tslibs.strptime._getlang
  File "c:\python37\lib\locale.py", line 587, in getlocale
    return _parse_localename(localename)
  File "c:\python37\lib\locale.py", line 495, in _parse_localename
    raise ValueError('unknown locale: %s' % localename)
ValueError: unknown locale: en-US

I’m running a virtual environment on Windows 10 using python3.7.4 with minimal packages installed:

Package         Version
--------------- -------
numpy           1.18.4
pandas          1.0.3
Pillow          7.1.2
pip             20.1
python-dateutil 2.8.1
pytz            2020.1
setuptools      46.1.3
six             1.14.0
wheel           0.34.2
wxPython        4.1.0

I can import pandas outside of the app just fine:

Python 3.7.4 (tags/v3.7.4:e09359112e, Jul  8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
>>> df = pandas.DataFrame()
>>>

I’m flummoxed, and I’ve been unable to find answers elsewhere. Other pandas “unknown locale” errors seem to be related to not having the locale exported on OSX.

Robin · May 8, 2020, 9:00pm

Is this with Python 3.8 on Windows? If so there was a change in Python where it sets the locale during startup instead of just ignoring it and doing nothing as it did in prior versions. Since that caused wx’s locale to be out of sync there would be mismatch errors raised from wx.

I added some code to try and bring the two back in sync, but maybe something more needs to be done, or something done differently. You can see the code in wx/core.py in the
App.InitLocale method if you (or anybody) feel like experimenting and finding a solution.

ChuckM · May 8, 2020, 9:23pm

Where are you located? I’m in the US building a pandas/wxPython app on a Mac (and testing under Windows) and I have not seen that pop up. The only thing I’m seeing in my environment that is remotely locale-based is LC_CTYPE=en_US.UTF-8.

My system is MacOS 10.15.4, Python 3.8 (from python.org) and I installed pandas and wxPython via pip in a Python3 virtual environment.

zobal · May 11, 2020, 2:24pm

I’m using Python 3.74. on Windows 10 in a virtual environment. I’m in Vancouver. I think what’s happening is that the locale is missing the encoding portion of the locale string (maybe?). Chuck’s LC_CTYPE is “en_US.UTF-8”, where mine is just “en_US”. I’ll have a look at App.InitLocale. I wonder if it’s just a case problem? Locale.py maps “en_us” to ''en_US.ISO8859-1", in a locale_alias dict. It’s this dict lookup failing on “en_US” and then concluding that the locale is unknown.

J_Grimmtooth · May 11, 2020, 6:37pm

I have seem something similar pop up using datetime to parse date strings in a wx app. I kludged around it but this looks exactly what I was seeing (no pandas in this case). Using wx 4.0 was fine, wx 4.1, not so much.

This is with Python 3.8 on Windows as well, and Robin’s description makes perfect sense in that context since the datetime is Python native - it does act like Python and wx are out of sync.

No idea how to address that cleanly yet.

zobal · May 11, 2020, 6:51pm

I suspect that any module that parses dates/time will have a similar issue. I experienced it importing datetime as well.

ChuckM · May 11, 2020, 11:22pm

When I tried your sample app, I got the same error. After I moved the import pandas statement out of the class and just under the import wx statement the problem seemed to go away. I’m not saying that’s the issue, but it might fix the problem for you.

stuarthsmith · May 26, 2020, 5:26pm

Robin, I’m having this issue with locale mismatch on Windows 10, Python 3.8, and wxPython 4.1.0. Whenever I run a strptime() function I get an error “unknown locale: en-US”
What is the best way to work around this so that I can use the strptime function in my program? I can’t seem to find any info on this anywhere except in your mention here. Any ideas would be greatly appreciated.

Karsten_Hilbert · May 26, 2020, 9:21pm

This really should be en_US.

Windows has got some strange ideas with regard to locale
names. Somewhere in the PostgreSQL mailing list archives
there should be in-depth discussions of that.

Eryk Syn on the python mailing list may know more, too.

Karsten

stuarthsmith · May 26, 2020, 10:11pm

I tried adding some entries to alias and the windows locale dictionaries in locale.py, but that didn’t help. I don’t understand enough about how “locale” is used and by which pieces of code, so if anyone can suggest a workaround I would really appreciate it. As it stands right now, I cannot make a “strptime” call in wxPython, so I can’t pass a string into a datetime field. Kind of stuck.

Robin · May 27, 2020, 4:28am

It looks like the hyphenated “en-US” is coming from the Windows API GetLocaleInfo function. Leave it to Microsoft to do their own thing in an incompatible manner…

I haven’t been able to duplicate whatever it was that prompted me to setup the default locale settings when the wx.App is initialized. So try this: derive a new class from wx.App and override the InitLocale method, like this:

    class MyApp(wx.App):
        def InitLocale(self):
            self.ResetLocale()

Let me know if you run into any locale- or translations-related problems. I have some ideas for an alternate approach that will avoid the ‘en-US’ problem, but if it is no longer needed I can just rip out that chunk of code entirely.

For those who are interested, in Python 3.7 and earlier, on my Windows machine, locale.getlocale() returns (None, None), so it seems that the process’ locale is left in its default state. But in Python 3.8 on Windows it returns ('English_United States', '1252') indicating that Python or something is now setting the locale to the system default. If I remember correctly, the problem I was trying to workaround was that by default wxWidgets would assume that the locale is set to the default “C” locale, and there was something that failed due to that difference. But now I don’t remember what it was that caused the problem back then…

stuarthsmith · May 27, 2020, 1:19pm

Thanks, Robin. Unfortunately I tried your code snippet in a variety of places in my code and it never seemed to make a difference. I eventually found a workaround…right after the wx.App is instantiated, I added this to nail me down to “en_US” and my strptime functions work now.

import locale
locale.setlocale(locale.LC_ALL, 'en_US')

I don’t know if that is some sort of taboo to hard set this, but after a few hours of reading and trying to understand what is going on here, it was the best I could do to get my code working.

Karsten_Hilbert · May 27, 2020, 3:20pm

Preferably:

import locale
hyphenated_locale = wx.WhateverGivesTheStrangeLocale()
locale.setlocale(locale.LC_ALL, hyphenated_locale.replace('-', '_'))

or some such.

Best,
Karsten

Zylyco · May 27, 2020, 3:34pm

Just a note about this :
I’m facing this issue on a Windows 8.1 platform.
It’s not happening on Windows 10 exclusively.

Does it make any difference ?

Karsten_Hilbert · May 27, 2020, 4:29pm

Other than that, it sorta defeats the whole purpose of I18N

But maybe that was symbolic code only.

Karsten

stuarthsmith · May 27, 2020, 5:01pm

Thanks for that. If I’m reading the documentation correctly (https://wxpython.org/Phoenix/docs/html/wx.Locale.html), it looks like “wx.Locale.GetName()” might be the function to get the short name. Will test that tonight.

Robin · May 27, 2020, 7:07pm

Okay. Please try this wx.App implementation:

class MyApp(wx.App):
    def InitLocale(self):
        self.ResetLocale()
        import locale
        lang, enc = locale.getdefaultlocale()
        self._initial_locale = wx.Locale(lang, lang[:2], lang)
        locale.setlocale(locale.LC_ALL, lang)

Karsten_Hilbert · May 27, 2020, 8:04pm

From the looks of it this should work (barring typos etc)
and goes to the core of the issue rather than kluding around
the symptom.

It also fixes Stuart’s issue of “None does not have
replace()” by unpacking the tuple (gotten from
getdefaultlocale() in this case).

Except that it is not recommended by the Python manual to
use

locale.setlocale(locale.LC_ALL, lang)

in libraries:

It is generally a bad idea to call setlocale() in some
library routine, since as a side effect it affects the
entire program. Saving and restoring it is almost as bad:
it is expensive and affects other threads that happen to
run before the settings have been restored.

It might be safer like so:

 class MyApp(wx.App):
     def InitLocale(self):
         self.ResetLocale()
         import locale
         lang, enc = locale.getdefaultlocale()
         self._initial_locale = wx.Locale(lang, lang[:2], lang)
         curr_lang, curr_enc = locale.getlocale()
         if curr_lang is None:
             try:
                 locale.setlocale(locale.LC_ALL, lang)
             except locale.Error:
                 whatever_log_is_available('cannot set LC_ALL to %' % lang)

meaning that

- it only tries to set the locale if it hasn't been set before
- or the locale cannot currently be determined
- and wxPython GUI apps always initialize the locale system
- which makes sense for GUI apps

Best,
Karsten

Robin · May 27, 2020, 8:30pm

Unfortunately doing a locale.getlocale() after wx.Locale(...) will trigger the exception due to the ‘en-US’ being set as the system locale. It happens when the wx.Locale is created, in some MSW-specific code. (It was something like, check the requested language/locale is valid, get the info with GetLocaleInfo API, and then set the locale using the name returned from GetLocalInfo, which is the ‘en-US’ name.)

I still need to check with wx-dev if what is going on currently is actually correct, and check if they have any ideas for Python’s locale restrictions. But in the meantime, it looks like we will need to call locale.setlocale to make the wx and Python locale settings to be equivalent. I will likely add a conditional so the code is only run on Windows however.

As to the warning about setlocale in a library, I would argue that wxPython is more of an application framework than a library, and so the warning doesn’t apply, (at least not as strongly) as it would for something like numpy, some database adapter, or whatever. The warning implies that it is up to the application to worry about setting the locale, and that is exactly what wx.App is. If the developer wants to do something differently they are free to override InitLocale themselves.

Karsten_Hilbert · May 27, 2020, 9:14pm

Unfortunately doing a locale.getlocale() after
wx.Locale(...) will trigger the exception due to the
‘en-US’ being set as the system locale. It happens when the
wx.Locale is created, in some MSW-specific code. (It was
something like, check the requested language/locale is valid,
get the info with GetLocaleInfo API, and then set the
locale using the name returned from GetLocalInfo, which is
the ‘en-US’ name.)

I don’t fully parse (but then even the Python manual warns
that locale implementations tend to be strange and wondrous
beasts) but I smell the problem.

I still need to check with wx-dev if what is going on
currently is actually correct, and check if they have any
ideas for Python’s locale restrictions. But in the meantime,
likely add a conditional so the code is only run on Windows
however.

Sounds good.

As to the warning about setlocale in a library, I would
argue that wxPython is more of an application framework than
a library, and so the warning doesn’t apply, (at least not as
strongly) as it would for something like numpy, some database
adapter, or whatever.

Agree.

The warning implies that it is up to
the application to worry about setting the locale, and that
is exactly what wx.App is. If the developer wants to do
something differently they are free to override InitLocale
themselves.

There’s this issue, however, unless I am mistaken:

python application starts up and sets up environment,
including activating a desired, but non-default, locale
(say, user selection from a config file)
application the proceeds to import wxPython and generates
the GUI
wx.App() will re-set the locale to whatever the default
locale for the user is

This means applications need to be adapted to re-activate the
desired locale init()ing wx.App, or overwrite
App.InitLocale() as you say. Both ways warrant a note in the
4.0/4.1 gotcha notes.

Also, would re-activating the desired locale require using wx
facilities rather than stdlib locale.* ones ?

On yet another level, will this entanglement prevent
decoupling string translation from GUI toolkit use ?

(currently, one can use gettext.gettext() without ever
touching wx.Translation() ?

Regards,
Karsten