wxPython 4/wx.Yield: C++ assert failure

Hi Robin and others,

In one of the projects I’m involved in (NonVisual Desktop Access), numerous calls to wx.Yield function are made from various modules. After porting NVDA to wxPython 4 alpha, when wx.Yield is called from some modules, I get a traceback that ends with the following:

wxAssertionError: C++ assertion “Assert failure” failed at …\src\common\evtloopcmn.cpp(110) in wxEventLoopBase::Yield(): wxYield called recursively

Relevant GitHub issue for NVDA can be found at:

https://github.com/nvaccess/nvda/issues/7077

The source code for a version of NVDA that uses wxPython 4 can be found at:

https://github.com/josephsl/nvda

Branch is “wxPy4”. I and other NVDA developers involved in porting NVDA to wxPython 4 consider this a showstopper, with a lead developer commenting that this might be due to reentrancy issue. Once this is resolved, developers plan to use wxPython 4 version of NVDA for alpha testing purposes for a while (at least two weeks) in hopes of catching new issues and writing workarounds.

Thanks. Comments are appreciated (if you do have GitHub account, please do comment on the issue linked above).

Cheers,

Joseph

Hi Robin and others,

In one of the projects I’m involved in (NonVisual Desktop Access), numerous calls to wx.Yield function are made from various modules. After porting NVDA to wxPython 4 alpha, when wx.Yield is called from some modules, I get a traceback that ends with the following:

wxAssertionError:
C++ assertion “Assert failure” failed at …\src\common\evtloopcmn.cpp(110) in wxEventLoopBase::Yield(): wxYield called recursively

Relevant GitHub issue for NVDA can be found at:

https://github.com/nvaccess/nvda/issues/7077

The source code for a version of NVDA that uses wxPython 4 can be found at:

https://github.com/josephsl/nvda

Branch is “wxPy4”. I and other NVDA developers involved in porting NVDA to wxPython 4 consider this a showstopper, with a lead developer commenting
that this might be due to reentrancy issue.

Hi Robin,

At least wx.YieldIfNeeded suppresses the assertions. However, according to a lead NVDA developer, this does not really resolve this problem, as NVDA needs to handle other events external to wxWidgets such as accessibility events. The lead developer (not me) believes that the root problem might have to do with timers and certain things that are internally different between wxPython classic and phoenix.

Can you try out this example and see if we can trace this:

https://www.dropbox.com/s/p8uuf52hzlmyd3x/wxReentrance.py?dl=1

Note that the example requires Python 2.7 with wxPython 4.0.0A1. Thanks.

Cheers,

Joseph

···

From: wxpython-dev@googlegroups.com [mailto:wxpython-dev@googlegroups.com] On Behalf Of Robin Dunn
Sent: Thursday, April 27, 2017 12:17 AM
To: wxPython-dev@googlegroups.com
Subject: Re: [wxPython-dev] wxPython 4/wx.Yield: C++ assert failure

Joseph Lee wrote:

Hi Robin and others,

In one of the projects I’m involved in (NonVisual Desktop Access), numerous calls to wx.Yield function are made from various modules. After porting NVDA to wxPython 4 alpha, when wx.Yield is called from some modules, I get a traceback that ends with the following:

wxAssertionError: C++ assertion “Assert failure” failed at …..\src\common\evtloopcmn.cpp(110) in wxEventLoopBase::Yield(): wxYield called recursively

Relevant GitHub issue for NVDA can be found at:

https://github.com/nvaccess/nvda/issues/7077

The source code for a version of NVDA that uses wxPython 4 can be found at:

https://github.com/josephsl/nvda

Branch is “wxPy4”. I and other NVDA developers involved in porting NVDA to wxPython 4 consider this a showstopper, with a lead developer commenting that this might be due to reentrancy issue.

Yes. Basically some event(s) happens while within a wx.Yield call which then results in some other code being called that uses wx.Yield.

The best fix would be to find ways to avoid needing to use yield. Otherwise you could switch to wx.YieldIfNeeded() (which is just a convenience wrapper for wx.GetApp().Yield(True)…) That will just return when the recursion is detected instead of asserting about it. The wx.EventLoopBase.Yield documentation is applicable to this as all the incarnations of Yield eventually get to that method.


Robin Dunn
Software Craftsman
http://wxPython.org


You received this message because you are subscribed to the Google Groups “wxPython-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an email to wxPython-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Hi Joseph,

The NVDA developer is likely correct, but your example shows expected behavior, not a bug. You’re adding an infinite number of timer events, then calling wx.Yield in them. This will cause infinite recursion, as wx.Yield will process the next timer event, which will call wx.Yield, which will process the next timer event, which will call wx.Yield, and so on. I don’t think a change in wx.Yield is at fault here, it’s just alerting you to the recursion instead of causing it. To get to the bottom of this, you’ll need to find out what is triggering the recursion in NVDA.

Taking a quick look at the NVDA code’s use of wx.Yield, I think the first step would probably be to find out if api.py’s processPendingEvents is somehow calling itself multiple times. If you can identify the specific set of steps that trigger recursion, the NVDA developers will probably be able to come up with some ideas for workarounds or fixes.

Regards,

Kevin

···

From: wxpython-dev@googlegroups.com on behalf of Joseph Lee joseph.lee22590@gmail.com
Reply-To: wxPython-dev@googlegroups.com
Date: Thursday, April 27, 2017 at 9:36 AM
To: wxPython-dev@googlegroups.com
Subject: RE: [wxPython-dev] wxPython 4/wx.Yield: C++ assert failure

Hi Robin,

At least wx.YieldIfNeeded suppresses the assertions. However, according to a lead NVDA developer, this does not really resolve this problem, as NVDA needs to handle other events external to wxWidgets such as accessibility events. The lead developer (not me) believes that the root problem might have to do with timers and certain things that are internally different between wxPython classic and phoenix.

Can you try out this example and see if we can trace this:

https://www.dropbox.com/s/p8uuf52hzlmyd3x/wxReentrance.py?dl=1

Note that the example requires Python 2.7 with wxPython 4.0.0A1. Thanks.

Cheers,

Joseph

From: wxpython-dev@googlegroups.com [mailto:wxpython-dev@googlegroups.com] On Behalf Of Robin Dunn
Sent: Thursday, April 27, 2017 12:17 AM
To: wxPython-dev@googlegroups.com
Subject: Re: [wxPython-dev] wxPython 4/wx.Yield: C++ assert failure

Joseph Lee wrote:

Hi Robin and others,

In one of the projects I’m involved in (NonVisual Desktop Access), numerous calls to wx.Yield function are made from various modules. After porting NVDA to wxPython 4 alpha, when wx.Yield is called from some modules, I get a traceback that ends with the following:

wxAssertionError: C++ assertion “Assert failure” failed at …..\src\common\evtloopcmn.cpp(110) in wxEventLoopBase::Yield(): wxYield called recursively

Relevant GitHub issue for NVDA can be found at:

https://github.com/nvaccess/nvda/issues/7077

The source code for a version of NVDA that uses wxPython 4 can be found at:

https://github.com/josephsl/nvda

Branch is “wxPy4”. I and other NVDA developers involved in porting NVDA to wxPython 4 consider this a showstopper, with a lead developer commenting that this might be due to reentrancy issue.

Yes. Basically some event(s) happens while within a wx.Yield call which then results in some other code being called that uses wx.Yield.

The best fix would be to find ways to avoid needing to use yield. Otherwise you could switch to wx.YieldIfNeeded() (which is just a convenience wrapper for wx.GetApp().Yield(True)…) That will just return when the recursion is detected instead of asserting about it. The wx.EventLoopBase.Yield documentation is applicable to this as all the incarnations of Yield eventually get to that method.


Robin Dunn
Software Craftsman
http://wxPython.org


You received this message because you are subscribed to the Google Groups “wxPython-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an email to wxPython-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google Groups “wxPython-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an email to wxPython-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Hi.

I’m the NVDA developer Joseph was discussing this with. First, my thanks to all who have helped with this so far.

Even though I totally understand that wx.Yield can cause re-entrance (this is clearly documented), this wasn’t previously the case when starting the same timer within that timer’s callback. That is, another timer or a wx event might get fired within wx.Yield, but the same timer never fired. Instead, the inner call to the timer was dropped. That is what this test case demonstrates. In wxPython 3, you never see the re-entrance message. In wxPython 4, you do.

The question is whether this change is intentional/expected and whether the previous behaviour will be restored. It’s reasonable to argue that we were relying on undefined behaviour (or maybe even a bug), and if that is the case and this won’t be reverted, we’ll find another way. However, I wanted to confirm this before putting a huge amount of work into “fixing” this.

The reason we use wx.Yield here is that we need to trigger a Windows message pump when sending key presses and checking for pending accessibility events, since keyboard events and accessibility events depend on the Windows message queue. I guess we could just do our own message pump, but I’m concerned this would end up processing wx events anyway.

Thanks!

Jamie

···

On Friday, April 28, 2017 at 4:30:44 AM UTC+10, kevin...@theolliviers.com wrote:

Hi Joseph,

The NVDA developer is likely correct, but your example shows expected behavior, not a bug. You’re adding an infinite number of timer events, then calling wx.Yield in them. This will cause infinite recursion, as wx.Yield will process the next timer event, which will call wx.Yield, which will process the next timer event, which will call wx.Yield, and so on. I don’t think a change in wx.Yield is at fault here, it’s just alerting you to the recursion instead of causing it. To get to the bottom of this, you’ll need to find out what is triggering the recursion in NVDA.

Taking a quick look at the NVDA code’s use of wx.Yield, I think the first step would probably be to find out if api.py’s processPendingEvents is somehow calling itself multiple times. If you can identify the specific set of steps that trigger recursion, the NVDA developers will probably be able to come up with some ideas for workarounds or fixes.

Regards,

Kevin

From: wxpyth...@googlegroups.com on behalf of Joseph Lee joseph....@gmail.com
Reply-To: wxPyth...@googlegroups.com
Date: Thursday, April 27, 2017 at 9:36 AM
To: wxPyth...@googlegroups.com
Subject: RE: [wxPython-dev] wxPython 4/wx.Yield: C++ assert failure

Hi Robin,

At least wx.YieldIfNeeded suppresses the assertions. However, according to a lead NVDA developer, this does not really resolve this problem, as NVDA needs to handle other events external to wxWidgets such as accessibility events. The lead developer (not me) believes that the root problem might have to do with timers and certain things that are internally different between wxPython classic and phoenix.

Can you try out this example and see if we can trace this:

https://www.dropbox.com/s/p8uuf52hzlmyd3x/wxReentrance.py?dl=1

Note that the example requires Python 2.7 with wxPython 4.0.0A1. Thanks.

Cheers,

Joseph

From: wxpyth...@googlegroups.com [mailto:wxpyth...@googlegroups.com] On Behalf Of Robin Dunn
Sent: Thursday, April 27, 2017 12:17 AM
To: wxPyth...@googlegroups.com
Subject: Re: [wxPython-dev] wxPython 4/wx.Yield: C++ assert failure

Joseph Lee wrote:

Hi Robin and others,

In one of the projects I’m involved in (NonVisual Desktop Access), numerous calls to wx.Yield function are made from various modules. After porting NVDA to wxPython 4 alpha, when wx.Yield is called from some modules, I get a traceback that ends with the following:

wxAssertionError: C++ assertion “Assert failure” failed at …..\src\common\evtloopcmn.cpp(110) in wxEventLoopBase::Yield(): wxYield called recursively

Relevant GitHub issue for NVDA can be found at:

https://github.com/nvaccess/nvda/issues/7077

The source code for a version of NVDA that uses wxPython 4 can be found at:

https://github.com/josephsl/nvda

Branch is “wxPy4”. I and other NVDA developers involved in porting NVDA to wxPython 4 consider this a showstopper, with a lead developer commenting that this might be due to reentrancy issue.

Yes. Basically some event(s) happens while within a wx.Yield call which then results in some other code being called that uses wx.Yield.

The best fix would be to find ways to avoid needing to use yield. Otherwise you could switch to wx.YieldIfNeeded() (which is just a convenience wrapper for wx.GetApp().Yield(True)…) That will just return when the recursion is detected instead of asserting about it. The wx.EventLoopBase.Yield documentation is applicable to this as all the incarnations of Yield eventually get to that method.


Robin Dunn
Software Craftsman
http://wxPython.org


You received this message because you are subscribed to the Google Groups “wxPython-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an email to wxPython-dev...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google Groups “wxPython-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an email to wxPython-dev...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Hi Jamie,

On Mac, the sample code does trigger the re-entrance warning on wxPython 3. I suspect the same test on Linux would provide similar results. The reason it doesn’t on Windows is that the time.sleep(1) call appears to be causing the timer event not to even be started, or at least fired. When I remove that time.sleep call, I get the re-entrance warning on Windows too.

That time.sleep(1) would stop a wx.PyTimer from being started or fired is definitely not expected or defined behavior, and is almost certainly some sort of bug, possibly with Classic’s SWIG bindings on Windows.

So is the need to use wx.Yield because you have to process those keyboard and accessibility events right at that moment in the code? i.e. you can’t wait until the normal event processing happens?

Regards,

Kevin

···

From: wxpython-dev@googlegroups.com on behalf of James Teh jamie@nvaccess.org
Reply-To: wxPython-dev@googlegroups.com
Date: Thursday, April 27, 2017 at 11:28 PM
To: wxPython-dev wxPython-dev@googlegroups.com
Subject: Re: [wxPython-dev] wxPython 4/wx.Yield: C++ assert failure

Hi.

I’m the NVDA developer Joseph was discussing this with. First, my thanks to all who have helped with this so far.

Even though I totally understand that wx.Yield can cause re-entrance (this is clearly documented), this wasn’t previously the case when starting the same timer within that timer’s callback. That is, another timer or a wx event might get fired within wx.Yield, but the same timer never fired. Instead, the inner call to the timer was dropped. That is what this test case demonstrates. In wxPython 3, you never see the re-entrance message. In wxPython 4, you do.

The question is whether this change is intentional/expected and whether the previous behaviour will be restored. It’s reasonable to argue that we were relying on undefined behaviour (or maybe even a bug), and if that is the case and this won’t be reverted, we’ll find another way. However, I wanted to confirm this before putting a huge amount of work into “fixing” this.

The reason we use wx.Yield here is that we need to trigger a Windows message pump when sending key presses and checking for pending accessibility events, since keyboard events and accessibility events depend on the Windows message queue. I guess we could just do our own message pump, but I’m concerned this would end up processing wx events anyway.

Thanks!

Jamie

On Friday, April 28, 2017 at 4:30:44 AM UTC+10, kevin...@theolliviers.com wrote:

Hi Joseph,

The NVDA developer is likely correct, but your example shows expected behavior, not a bug. You’re adding an infinite number of timer events, then calling wx.Yield in them. This will cause infinite recursion, as wx.Yield will process the next timer event, which will call wx.Yield, which will process the next timer event, which will call wx.Yield, and so on. I don’t think a change in wx.Yield is at fault here, it’s just alerting you to the recursion instead of causing it. To get to the bottom of this, you’ll need to find out what is triggering the recursion in NVDA.

Taking a quick look at the NVDA code’s use of wx.Yield, I think the first step would probably be to find out if api.py’s processPendingEvents is somehow calling itself multiple times. If you can identify the specific set of steps that trigger recursion, the NVDA developers will probably be able to come up with some ideas for workarounds or fixes.

Regards,

Kevin

From: wxpyth...@googlegroups.com on behalf of Joseph Lee joseph....@gmail.com
Reply-To: wxPyth...@googlegroups.com
Date: Thursday, April 27, 2017 at 9:36 AM
To: wxPyth...@googlegroups.com
Subject: RE: [wxPython-dev] wxPython 4/wx.Yield: C++ assert failure

Hi Robin,

At least wx.YieldIfNeeded suppresses the assertions. However, according to a lead NVDA developer, this does not really resolve this problem, as NVDA needs to handle other events external to wxWidgets such as accessibility events. The lead developer (not me) believes that the root problem might have to do with timers and certain things that are internally different between wxPython classic and phoenix.

Can you try out this example and see if we can trace this:

https://www.dropbox.com/s/p8uuf52hzlmyd3x/wxReentrance.py?dl=1

Note that the example requires Python 2.7 with wxPython 4.0.0A1. Thanks.

Cheers,

Joseph

From: wxpyth...@googlegroups.com [mailto:wxpyth...@googlegroups.com] On Behalf Of Robin Dunn
Sent: Thursday, April 27, 2017 12:17 AM
To: wxPyth...@googlegroups.com
Subject: Re: [wxPython-dev] wxPython 4/wx.Yield: C++ assert failure

Joseph Lee wrote:

Hi Robin and others,

In one of the projects I’m involved in (NonVisual Desktop Access), numerous calls to wx.Yield function are made from various modules. After porting NVDA to wxPython 4 alpha, when wx.Yield is called from some modules, I get a traceback that ends with the following:

wxAssertionError: C++ assertion “Assert failure” failed at …..\src\common\evtloopcmn.cpp(110) in wxEventLoopBase::Yield(): wxYield called recursively

Relevant GitHub issue for NVDA can be found at:

https://github.com/nvaccess/nvda/issues/7077

The source code for a version of NVDA that uses wxPython 4 can be found at:

https://github.com/josephsl/nvda

Branch is “wxPy4”. I and other NVDA developers involved in porting NVDA to wxPython 4 consider this a showstopper, with a lead developer commenting that this might be due to reentrancy issue.

Yes. Basically some event(s) happens while within a wx.Yield call which then results in some other code being called that uses wx.Yield.

The best fix would be to find ways to avoid needing to use yield. Otherwise you could switch to wx.YieldIfNeeded() (which is just a convenience wrapper for wx.GetApp().Yield(True)…) That will just return when the recursion is detected instead of asserting about it. The wx.EventLoopBase.Yield documentation is applicable to this as all the incarnations of Yield eventually get to that method.


Robin Dunn
Software Craftsman
http://wxPython.org


You received this message because you are subscribed to the Google Groups “wxPython-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an email to wxPython-dev...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google Groups “wxPython-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an email to wxPython-dev...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google Groups “wxPython-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an email to wxPython-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Hi Jamie,

James Teh wrote:

Hi.

I’m the NVDA developer Joseph was discussing
this with. First, my thanks to all who have helped with this so far.

Even
though I totally understand that wx.Yield can cause re-entrance (this is clearly documented), this wasn’t previously the case when starting the same timer within that timer’s callback. That is, another timer or
a wx event might get fired within wx.Yield, but the same timer never fired. Instead, the inner call to the timer was dropped. That is what this test case demonstrates. In wxPython 3, you never see the re-entrance message. In wxPython 4, you do.

The question is whether this change is intentional/expected and whether the previous behaviour will be restored. It’s reasonable to argue that we were relying on undefined behaviour (or maybe even a bug), and if that is the
case and this won’t be reverted, we’ll find another way. However, I wanted to confirm this before putting a huge amount of work into “fixing” this.

The wx.PyTimer class is implemented differently in Phoenix than it was in Classic, but I don’t see how that difference could cause this difference in behavior. I’m not aware of changes in the C++ wxTimer that
might lead to this, but that doesn’t mean there aren’t any. (There were
no changes in the MSW implementation of wxTimer, but it could be that one of the changes in how the events are dispatched or something could have had some impact.)

Either way, the way it is working now is how I’ve always expected timers
and yielding to work, so I think that this probably is a case of relying on some undefined behavior.

The reason we use wx.Yield here is that we need to trigger a Windows message pump when sending key presses and checking for
pending accessibility events, since keyboard events and accessibility events depend on the Windows message queue. I guess we could just do our
own message pump, but I’m concerned this would end up processing wx events anyway.

If I understand correctly what you are doing I think you are correct, unless you manage your message queue on a different thread. It’s been a long time since I worked at that level of the Windows API so I may be forgetting something however.

There may be some alternatives though. I’ve not used this before so I don’t know if it will have the effect you are looking for, but I see that the wx.EventLoopBase class has a WakeUp method that sounds interesting. You can call it something like this:

wx.GetApp().GetMainLoop().WakeUp()

Or you can make your own temporary wx event loop with code like this:

evtLoop = wx.GetApp().GetTraits().CreateEventLoop()

with wx.EventLoopActivator(evtLoop):

    # automatically restores the old one upon exit from the context manager

Once you have the event loop you can selectively yield for categories of
events with the YieldFor method. Messages received that are not in the given category will be put back on the queue to be processed later. I’m not sure if this will provide enough control for your needs, but it is probably worth investigating.

You can also derive a new class from wx.EventLoopBase and use that instead of the default returned from CreateEventLoop. With a derived class you can implement your own MainLoop and implement your message processing there.

HTH,

Robin

···


Robin Dunn

Software Craftsman

http://wxPython.org