Having Problem extending wxPython

Robin · January 13, 2012, 5:50am

With an environment like that I wouldn't be surprised[1] if something somewhere is only setting sys.stdout, sys.stderr to the console's queues when it is expecting for there to be possible output from some Python code being executed by it, and then restoring it back to the previous values when that code returns. When the wx event loop calls event handlers or other callbacks those are happening at unexpected times (unexpected by the console anyway) and so the sys.stdout and sys.stderr have not been set to where you are expecting them to be at that point in time.

Try saving a reference to sys.stdout and sys.stderr in some Python code that is executed from the console, and then set sys.stdout and sys.stderr to those saved streams at the beginning of your event handlers or callback code.

[1] Because I've written code like that myself in similar situations.

···

On 1/11/12 11:57 AM, Emmanuel Bacry wrote:

Emmanuel Bacry wrote:

The "only" problem I get, is that when I use (within the callbacks) PyErr_.... functions, for instance

PyErr_SetString("There is an error"); \\ Generate an error
PyErr_Print()); \\ prints it

I am very surprised because I just don't understand what error stream it uses. It is NOT using the python stream sys.stderr (as my soft in wxPython does)

PyErr_Print imports sys and writes to sys.stderr. What leads you to
believe that's not happening?

I am using the qtconsole of ipython with the wx loop. I don't know if you are familiar with that, but it lets you control the ipython kernel (within a wx loop) from a console.
The connection between the console and the kernel is made using TCP using the library zmq.
It works VERY well.

So the sys.stderr is set to a TCP client that I can see in the qtconsole

Now, within the C AND within an event callback execution of wxwidgets, I can check that the object sys.stderr is still the same.
Moreover if I do PyRun_SimpleString("print>> sys.stderr 'kjkj'") it works fine, it prints in the qt console
but when I do a PyErr_Print() call it prints the message on the standard stderr (i.e., in the unix terminal)

Let me point out that this seems to happen ONLY when these calls are made within a wx call back (not if I make a "direct" call from the qtconsole)

--
Robin Dunn
Software Craftsman

Emmanuel_Bacry · January 13, 2012, 5:34pm

Emmanuel Bacry wrote:

The "only" problem I get, is that when I use (within the callbacks) PyErr_.... functions, for instance

PyErr_SetString("There is an error"); \\ Generate an error
PyErr_Print()); \\ prints it

I am very surprised because I just don't understand what error stream it uses. It is NOT using the python stream sys.stderr (as my soft in wxPython does)

PyErr_Print imports sys and writes to sys.stderr. What leads you to
believe that's not happening?

I am using the qtconsole of ipython with the wx loop. I don't know if you are familiar with that, but it lets you control the ipython kernel (within a wx loop) from a console.
The connection between the console and the kernel is made using TCP using the library zmq.
It works VERY well.

So the sys.stderr is set to a TCP client that I can see in the qtconsole

Now, within the C AND within an event callback execution of wxwidgets, I can check that the object sys.stderr is still the same.
Moreover if I do PyRun_SimpleString("print>> sys.stderr 'kjkj'") it works fine, it prints in the qt console
but when I do a PyErr_Print() call it prints the message on the standard stderr (i.e., in the unix terminal)

Let me point out that this seems to happen ONLY when these calls are made within a wx call back (not if I make a "direct" call from the qtconsole)

With an environment like that I wouldn't be surprised[1] if something somewhere is only setting sys.stdout, sys.stderr to the console's queues when it is expecting for there to be possible output from some Python code being executed by it, and then restoring it back to the previous values when that code returns. When the wx event loop calls event handlers or other callbacks those are happening at unexpected times (unexpected by the console anyway) and so the sys.stdout and sys.stderr have not been set to where you are expecting them to be at that point in time.

Try saving a reference to sys.stdout and sys.stderr in some Python code that is executed from the console, and then set sys.stdout and sys.stderr to those saved streams at the beginning of your event handlers or callback code.

[1] Because I've written code like that myself in similar situations.

Thank you very much for your advice ! I am going to try and let you know.

I am just surprised that at the very same place of the C code if I do

PyErr_SetString("There is an error"); \\ Generate an error
PyErr_Print()); \\ prints it

it prints on stderr

and if I do

PyRun_SimpleString("print>> sys.stderr 'kjkj'")

it does print on the terminal

Emmanuel

···

Le 13 janv. 2012 à 06:50, Robin Dunn a écrit :

On 1/11/12 11:57 AM, Emmanuel Bacry wrote:

Emmanuel_Bacry · January 20, 2013, 9:02pm

Robin Dunn <robin <at> alldunn.com> writes:

> Yes !
>
> It looks like I found a solution :
>
> Each C++ callback starts with
>
> PyGILState_STATE gstate;
> gstate = PyGILState_Ensure();
>
> and ends with
>
> PyGILState_Release(gstate);
>
> ......
>
> It does not segmentation fault any more...
> Of course, from what I understand it forbids

running threads in my call backs... but I don't really

care.

Only while running in the Python part of your callbacks. If they
subsequently call out to C++ code (adding another layer to the sandwich)
then the GIL may be released and reacquired again in that C++ code and
other Python threads could be run at that time. This should be the same
as any other transition to/from C++/wxPython code too.

>
> Does it look safe to you ?

Yes.

Hello,

I have been working a lot lately on my project... everything was going fine ...
till now, something is
going weired and I feel it might be related to your answer you
made to me last year !!

So I have some more questions, I hope you will have time to answer...

Let me refresh you :

1- I use python with wxPython

2- I write some C++ callbacks of wxWidgets. Each call back
starts with
PyGILState_STATE gstate;
gstate = PyGILState_Ensure()

and ends with

PyGILState_Release(gstate);

3- The C++ callback sometimes make some calls in python (using C-API)

4- The python code that is executed from these calls never create any thread.
It is pretty simple.
HOWEVER it can call back C++ extensions again (no new thread too).
==> creating several layers in the "sandwich" to use your expressions !

5- In the whole process of the call back I do not use any thread.
So EVERYTHING, i.e., all the layers are executed within my initial
PyGILState_Ensure()
PyGILState_Release(gstate);

6- It looks like that at some point some objects are freed unexpectedly !?

I am not sure to understand your answer below.
Do I have to release/reacquired the GIL again during
the process of my call backs ?

Thank you very much for any help !

Emmanuel

···

On 1/1/12 8:30 AM, Emmanuel Bacry wrote:

Robin · January 23, 2013, 4:06am

Emmanuel Bacry wrote:

Robin Dunn<robin<at> alldunn.com> writes:

Yes !

It looks like I found a solution :

Each C++ callback starts with

PyGILState_STATE gstate;
gstate = PyGILState_Ensure();

and ends with

PyGILState_Release(gstate);

......

It does not segmentation fault any more...
Of course, from what I understand it forbids
running threads in my call backs... but I don't really

care.

Only while running in the Python part of your callbacks. If they
subsequently call out to C++ code (adding another layer to the sandwich)
then the GIL may be released and reacquired again in that C++ code and
other Python threads could be run at that time. This should be the same
as any other transition to/from C++/wxPython code too.

Does it look safe to you ?

Yes.

Hello,

I have been working a lot lately on my project... everything was going fine ...
till now, something is
going weired and I feel it might be related to your answer you
made to me last year !!

So I have some more questions, I hope you will have time to answer...

Let me refresh you :

1- I use python with wxPython

2- I write some C++ callbacks of wxWidgets. Each call back
starts with
PyGILState_STATE gstate;
gstate = PyGILState_Ensure()

and ends with

PyGILState_Release(gstate);

3- The C++ callback sometimes make some calls in python (using C-API)

4- The python code that is executed from these calls never create any thread.
It is pretty simple.
HOWEVER it can call back C++ extensions again (no new thread too).
==> creating several layers in the "sandwich" to use your expressions !

5- In the whole process of the call back I do not use any thread.
So EVERYTHING, i.e., all the layers are executed within my initial
PyGILState_Ensure()
PyGILState_Release(gstate);

6- It looks like that at some point some objects are freed unexpectedly !?

I am not sure to understand your answer below.
Do I have to release/reacquired the GIL again during
the process of my call backs ?

Thank you very much for any help !

You should read the docs about the thread state and GIL at Initialization, Finalization, and Threads — Python 2.7.18 documentation.

In a nutshell:

1. Any time you have Python code call C/C++ code (or at least any c/C++ code that could block or could result in a callback to Python) then it should release the GIL so other Python threads can run while the C++ code in the current thread is being run. When that block of C++ code is finished then it should reacquire the GIL before returning to Python.

2. The other side of the coin is that whenever C++ code manipulates Python objects or calls Python code then it should restore a valid thread state and acquire the GIL first, effectively blocking other Python threads from running (until the interpreter allows them to run anyway) and then release it again after that python code is done.

There are 4 functions/macros in the Python API that help with this, 2 for each of the situations above. For situation #1 it looks like this:

     Python code calling C
         Py_BEGIN_ALLOW_THREADS
             C/C++ code
         Py_END_ALLOW_THREADS
     Return to Python

And for #2 it looks like this:

     C/C++ code about to use Python
         state = PyGILState_Ensure()
             call Python code or manipulate PyObjects
         PyGILState_Release(state)
     Return to ordinary C/C++ code

In the wxPython API I have some wrappers around the Python APIs mentioned above so I can do a little bit of extra housekeeping in some cases. They are:

     // For Python --> C++
     PyThreadState* wxPyBeginAllowThreads();
     void wxPyEndAllowThreads(PyThreadState* state);

     // For C++ --> Python
     wxPyBlock_t wxPyBeginBlockThreads();
     void wxPyEndBlockThreads(wxPyBlock_t blocked);

And in 2.9 and Phoenix there is also the wxPyThreadBlocker class that is another way to take care of situation #2 that is a bit easier. It does the Ensure when an instance of the class is created and the Release when that instance is deleted.

···

On 1/1/12 8:30 AM, Emmanuel Bacry wrote:

--
Robin Dunn
Software Craftsman

Emmanuel_Bacry · April 25, 2013, 4:06pm

Hello,

I wrote an extension of wxWidgets that is a python module that works well with wxPython.
For that purpose of course I need to link my extension against the same wxWidget dynamic libraries as the ones used by wxPython.

On MacOS X and Unix it works well.

On Windows platform (using mingw) I have a problem.
This is what I have done

1- Downloaded the binaries of wxPython : wxPython2.9-win32-py27
on the web page : http://www.wxpython.org/download.php#unstable

2- Downloaded the corresponding development files : 32-bit binaries for MSVC 9
(same web page)

Now I have to link my extension (C++) using mingw with the dynamic libraries in downloaded 1-
(compilation works fine)

I cannot understand what's the proper command for linking with these dynamic libraries : I cannot find the right options !?!
I always get the "undefined reference" error on all the wxWidgets symbols ….. though clearly the .dll's I link to contain the corresponding symbols

Could any one help me and tell me what are the right options ??

Thank you very much

Emmanuel

Robin · April 25, 2013, 5:28pm

Emmanuel Bacry wrote:

Hello,

I wrote an extension of wxWidgets that is a python module that works well with wxPython.
For that purpose of course I need to link my extension against the same wxWidget dynamic libraries as the ones used by wxPython.

On MacOS X and Unix it works well.

On Windows platform (using mingw) I have a problem.
This is what I have done

1- Downloaded the binaries of wxPython : wxPython2.9-win32-py27
on the web page : Redirecting...

2- Downloaded the corresponding development files : 32-bit binaries for MSVC 9
(same web page)

Now I have to link my extension (C++) using mingw with the dynamic libraries in downloaded 1-
(compilation works fine)

I cannot understand what's the proper command for linking with these dynamic libraries : I cannot find the right options !?!
I always get the "undefined reference" error on all the wxWidgets symbols ï¿½.. though clearly the .dll's I link to contain the corresponding symbols

Could any one help me and tell me what are the right options ??

I'm not sure about mingw32 but for MSVC you actually link with import libraries (*.lib files) that define the symbols found in the DLL and other info, not with the DLL files directly like you would on other platforms with shared libraries. I seem to recall reading something about a tool that would generate an appropriate lib*.a file from a given DLL that could be used with mingw32 similarly to how MSVC uses the *.lib files.

However, you are going to have a bigger problem. Since wxWidgets and the wxPython wrappers are C++ code, and since different C++ compilers do the "C++ name mangling" differently than other compilers, then you will not be able to link C++ libs/DLLs made with MSVC with your code compiled with mingw32. From http://en.wikipedia.org/wiki/Name_mangling:

"""
There isn't a standard scheme by which even trivial C++ identifiers are mangled, and consequently different compiler vendors (or even different versions of the same compiler, or the same compiler on different platforms) mangle public symbols in radically different (and thus totally incompatible) ways.
"""

Try using Visual C++ 2008 Express.

···

--
Robin Dunn
Software Craftsman

Emmanuel_Bacry · April 28, 2013, 8:57pm

Thank you very very much

your answer helped me a lot.

… I am currently trying …

Emmanuel

···

Emmanuel Bacry wrote:

Hello,

I wrote an extension of wxWidgets that is a python module that works well with wxPython.
For that purpose of course I need to link my extension against the same wxWidget dynamic libraries as the ones used by wxPython.

On MacOS X and Unix it works well.

On Windows platform (using mingw) I have a problem.
This is what I have done

1- Downloaded the binaries of wxPython : wxPython2.9-win32-py27
on the web page : Redirecting...

2- Downloaded the corresponding development files : 32-bit binaries for MSVC 9
(same web page)

Now I have to link my extension (C++) using mingw with the dynamic libraries in downloaded 1-
(compilation works fine)

I cannot understand what's the proper command for linking with these dynamic libraries : I cannot find the right options !?!
I always get the "undefined reference" error on all the wxWidgets symbols ….. though clearly the .dll's I link to contain the corresponding symbols

Could any one help me and tell me what are the right options ??

I'm not sure about mingw32 but for MSVC you actually link with import libraries (*.lib files) that define the symbols found in the DLL and other info, not with the DLL files directly like you would on other platforms with shared libraries. I seem to recall reading something about a tool that would generate an appropriate lib*.a file from a given DLL that could be used with mingw32 similarly to how MSVC uses the *.lib files.

However, you are going to have a bigger problem. Since wxWidgets and the wxPython wrappers are C++ code, and since different C++ compilers do the "C++ name mangling" differently than other compilers, then you will not be able to link C++ libs/DLLs made with MSVC with your code compiled with mingw32. From http://en.wikipedia.org/wiki/Name_mangling:

"""
There isn't a standard scheme by which even trivial C++ identifiers are mangled, and consequently different compiler vendors (or even different versions of the same compiler, or the same compiler on different platforms) mangle public symbols in radically different (and thus totally incompatible) ways.
"""

Try using Visual C++ 2008 Express.

--
Robin Dunn
Software Craftsman
http://wxPython.org

--
You received this message because you are subscribed to the Google Groups "wxPython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wxpython-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.