Windows fatal exception: access violation

[Andrea_Gavana]

Hello wxPython Users,

I am almost done in porting a very large, legacy Python 2 application to Python 3. Everything works all right except I am getting access violation error messages that bring down the whole Python process. I can trace it back to wx.ProgressDialog.Pulse() using the Python faulthandler environment variable.

I have a very simple subclass of wx.ProgressDialog, which does not do anything different from the standard wx.ProgressDialog beside setting an icon for it in the __init__ method. This is my overloaded Pulse method:

def Pulse(self, *args):

    print('I am in thread:', threading.current_thread())
    print('I am:', self)
    print('Deleted:', self.IsBeingDeleted(), 'Nonzero:', self.__nonzero__())
    print()
    if not self or self.IsBeingDeleted():
        return

    wx.ProgressDialog.Pulse(self, *args)
    wx.SafeYield()

The Pulse() method is called from another thread but always using wx.CallAfter. Just so we are clear, there is no instance in which a separate thread accesses GUI elements.

The presence or absence of wx.SafeYield() makes no difference to the outcome.

I get the following after a few seconds:

I am in thread: <_MainThread(MainThread, started 22296)>
I am: <mdp_widgets.PhaserProgressDialog object at 0x000001FA9D227670>
Deleted: False Nonzero: True

... more prints, another wx.ProgressDialog is created ...

I am in thread: <_MainThread(MainThread, started 22296)>
I am: <mdp_widgets.PhaserProgressDialog object at 0x000001FABE3A0550>
Deleted: False Nonzero: True

Windows fatal exception: access violation

Current thread 0x00005718 (most recent call first):
  File "C:\Users\J0514162\MyProjects\Phaser\refactoring_github\py3_working\mdp_widgets.py", line 11344 in Pulse
  File "C:\Users\J0514162\WinPython39\WPy64-39100\python-3.9.10.amd64\lib\site-packages\wx\core.py", line 3427 in <lambda>
  File "C:\Users\J0514162\MyProjects\Phaser\refactoring_github\py3_working\mdp_widgets.py", line 11344 in Pulse
  File "C:\Users\J0514162\WinPython39\WPy64-39100\python-3.9.10.amd64\lib\site-packages\wx\core.py", line 3427 in <lambda>
  File "C:\Users\J0514162\MyProjects\Phaser\refactoring_github\py3_working\mdp_widgets.py", line 11344 in Pulse
  File "C:\Users\J0514162\WinPython39\WPy64-39100\python-3.9.10.amd64\lib\site-packages\wx\core.py", line 3427 in <lambda>
  File "C:\Users\J0514162\WinPython39\WPy64-39100\python-3.9.10.amd64\lib\site-packages\wx\core.py", line 2262 in MainLoop
  File "C:\Users\J0514162\MyProjects\Phaser\refactoring_github\py3_working\Phaser.py", line 57 in phaser_main
  File "C:\Users\J0514162\MyProjects\Phaser\refactoring_github\py3_working\Phaser.py", line 69 in <module>

You may notice that line 3427 in core.py (at least on wxPython 4.2) is the implementation of wx.CallAfter:

This is so fantastically annoying that I have no words to describe the frustration. It has taken me longer to diagnose this issue and trying to find a workaround (which I cannot not find) than it took me to port 150,000 lines of code from Python 2 to Python 3, and from Classic to Phoenix.

Any suggestion on how to address this issue - workarounds, hacks, anything - would be most welcome.

Thank you in advance.

Andrea.

[AndersMunch]

You are issuing CallAfter’s faster than they can be executed, causing you to have multiple pending Pulse calls queued.

wx.SafeYield causes CallAfter events to be processed, which means your Pulse method is called recursively, and then wx.SafeYield is called recursively. I don’t know if it SafeYield is supposed to be able to handle that – apparently it doesn’t, hence your crash.

There are several ways to rewrite this and avoid the recursion, but what I’m wondering is, why is there a SafeYield call there in the first place? The code returns to the wx event loop immediately after, so why is there a need to process events here? Perhaps Pulse is also called from blocking code and in this case it is needed, but when called using CallAfter, I don’t believe it is.

[Andrea_Gavana]

Hi,

as I mentioned specifically in my message:

The presence or absence of wx.SafeYield() makes no difference to the outcome.

I get an access violation even if I remove it. And I have 2 or 3 wx.CallAfter in the process, not 30. Not to mention that all of this did not crash in wxPython Classic.

I may understand a wx.wxAssertionError or RuntimeError, what is inexplicable is the whole Python process coming down because of this.

[swt2c]

Do you have a SCCCE that reproduces the problem?

[AndersMunch]

@Andrea_Gavana

The presence or absence of wx.SafeYield() makes no difference to the outcome.

Missed that. Long post.

Your traceback does show recursion, though, and that’s a classic for triggering messy program-crashing C/C++ bugs. I’d get rid of that before anything else.

@Andrea_Gavana

I have 2 or 3 wx.CallAfter in the process, not 30.

Your traceback shows core.py:34127 and Pulse thrice. The explanation that comes most readily to mind is that you have multiple CallAfter events pending, and somehow you ended up processing the next CallAfter recursively. You may have another explanation.

[Andrea_Gavana]

Hi,

I can try and put together one later. Hopefully I’ll be able to reproduce the issue.

As a side note, I have tried to put a guard inside the overridden Pulse method, like “just return and don’t do anything if less than 0.5 seconds have passed since your last Pulse()”.

While this helps a bit with the problem by delaying it, after a certain number of simulations (higher than without the guard) it still crashes with the same access violation error.

[AndersMunch]

I think wx.ProgressDialog.Pulse is the culprit.

src\generic\progdlgg.cpp:

bool wxGenericProgressDialog::Pulse(const wxString& newmsg, bool *skip)
{
    if ( !DoBeforeUpdate(skip) )
        return false;

   ...
}

bool wxGenericProgressDialog::DoBeforeUpdate(bool *skip)
{
    // we have to yield because not only we want to update the display but
    // also to process the clicks on the cancel and skip buttons
    // NOTE: using YieldFor() this call shouldn't give re-entrancy problems
    //       for event handlers not interested to UI/user-input events.
    wxEventLoopBase::GetActive()->YieldFor(wxEVT_CATEGORY_UI|wxEVT_CATEGORY_USER_INPUT);
   ...
}

I bet the YieldFor is implicated in your problem. It’s there to collect keypresses/clicks when in the middle of a long-running event handler, but if you’re not actually using the progress dialog in a long-running event handler, then perhaps you should go around it.

If you replace the wx.ProgressDialog with a wx.Gauge on a regular frame, then you can just call Pulse on the gauge directly, without incurring any weird event-loopery.

[Andrea_Gavana]

Thank you for your answer. I seem to remember though that by default the Windows wx.ProgressDialog is not based on the generic version but it uses the native widget.

I do not know if that piece of code is shared between the implementations - usually the shared sources live in the common/ directory - but I’ll take a look.

[da-dada]

I just wonder how the CallAfter gets in here? or am I on the wrong planet

from threading import Thread
from time import sleep
import wx

class Gui(wx.Frame):

    def __init__(self, parent):
        print(f'{self.__class__} __init__')
        super().__init__(parent, title='ProgressDialog')

        wx.Button(self, label='left click me, please')
        self.Bind(wx.EVT_BUTTON, lambda _: Action().start())
        self.Centre()
        self.Show()
        Action().start()

class Action(Thread):

    def __init__(self):
        super().__init__()
        self.dlg = wx.ProgressDialog(
                'from thread', 'start',
                style=wx.PD_CAN_ABORT|wx.PD_AUTO_HIDE)
                # add 'wx.PD_APP_MODAL' for modal
        self.dlg.Show()

    def run(self):
        pls = 1
        while not self.dlg.WasCancelled():
            sleep(2)
            self.dlg.Pulse(newmsg=f'pulse {str(pls)}')
            pls += 1

app = wx.App()
Gui(None)
app.MainLoop()

[Andrea_Gavana]

My understanding is that all UI interactions should happen in the main thread - I.e. the one that creates the application top window and the one where the app lives.

I am not sure how and why your example does not bomb out altogether - although I haven’t tried it myself - but we have always been told that calling UI things from other threads was a no-no to start with.

[Thom]

@Andrea_Gavana

I am almost done in porting a very large, legacy Python 2 application to Python 3

So am I, Andrea, and successfully.

During my porting Python 2 to 3 and Classic to Phoenix up to 4.2 (and lower versions), I encountered many wx related bugs. Not a fatal access violation, though.

So FWIW, I’m currently running a very stable combination, for me at least:

• Python 3.8.12 (heads/master:bd5e0bb, Aug 31 2021, 10:21:45) [MSC v.1929 64 bit (AMD64)] on win32
• 4.0.7.post2 msw (phoenix) wxWidgets 3.0.5

I wonder:

1. what version combination do you currently use?
2. have you tried other version combinations?

Cheers, Thom

[RichardT]

I also believe that you should only update the UI from the main thread. However, instead of using using wx.CallAfter() I just send events using AddPendingEvent(). I have used this technique in several applications, in some of which the child thread can quickly generate a lot of events containing progress messages. The UI typically displays the messages in an STC and I haven’t yet had any problems with it keeping up with the child thread. Note: I have only tested this on Linux.

The code below shows a much simplified example which uses two custom events and an operation class derived from Thread. The class’s run() method sends the appropriate events. The operation class also contains a Queue object which provides a thread safe mechanism to send messages from the main thread to the child thread. I use the Queue object as a way to abort the child thread if the Cancel button on the ProgressDialog is pressed.

from threading import Thread
from queue import Queue, Empty
from time import sleep
import wx

ABORT = "abort"

evt_op_progress_type = wx.NewEventType()
EVT_OP_PROGRESS = wx.PyEventBinder(evt_op_progress_type, 1)

class OpProgressEvent(wx.PyCommandEvent):
    def __init__(self, count):
        wx.PyCommandEvent.__init__(self, evt_op_progress_type, 1)
        self.count = count

evt_op_ended_type = wx.NewEventType()
EVT_OP_ENDED = wx.PyEventBinder(evt_op_ended_type, 1)

class OpEndedEvent(wx.PyCommandEvent):
    def __init__(self, status):
        wx.PyCommandEvent.__init__(self, evt_op_ended_type, 1)
        self.status = status

class MyOperation(Thread):
    def __init__(self, limit):
        Thread.__init__(self)
        self.wx_app = wx.GetApp()
        self.queue  = Queue()
        self.limit = limit

    def checkQueue(self):
        try:
            item = self.queue.get_nowait()
        except Empty:
            item = ""
        return item

    def run(self):
        count = 0
        while True:
            if self.checkQueue() == ABORT:
                evt = OpEndedEvent("aborted")
                self.wx_app.AddPendingEvent(evt)
                return
            count += 1
            evt = OpProgressEvent(count)
            self.wx_app.AddPendingEvent(evt)
            if count > self.limit:
                evt = OpEndedEvent("completed")
                self.wx_app.AddPendingEvent(evt)
                return
            sleep(.01)


class MyFrame(wx.Frame):
    def __init__(self, parent):
        wx.Frame.__init__(self, parent, title='Test ProgressDialog')
        sizer = wx.BoxSizer(wx.VERTICAL)
        self.start_button = wx.Button(self, label='Start')
        sizer.Add(self.start_button)
        self.Bind(wx.EVT_BUTTON, self.OnStart, self.start_button)
        self.wx_app = wx.GetApp()
        self.wx_app.Bind(EVT_OP_PROGRESS, self.OnProgress)
        self.wx_app.Bind(EVT_OP_ENDED, self.OnOpEnded)
        self.SetSizer(sizer)
        self.Layout()
        self.Center()
        self.Show()


    def OnOpEnded(self, evt):
        self.thread.join()
        self.dlg.Destroy()
        print("Status = %s" % evt.status)
        self.start_button.Enable()


    def OnProgress(self, evt):
        if self.dlg.WasCancelled():
            self.thread.queue.put(ABORT)
        msg = "Count = %d" % evt.count
        self.dlg.Pulse(msg)


    def OnStart(self, _evt):
        self.start_button.Disable()
        limit = 1000
        title = "Count to %d" % limit
        self.dlg = wx.ProgressDialog(title, 'Count = 0',
                                     style=wx.PD_CAN_ABORT|wx.PD_AUTO_HIDE)
        self.dlg.Show()

        self.thread = MyOperation(limit)
        self.thread.start()


app = wx.App()
MyFrame(None)
app.MainLoop()

[Andrea_Gavana]

Hi Thom,

Thank you for your answer. I’m currently on a standard Windows 10 64 bit laptop, rather ok per se (64 GB RAM, 8 cores).

I have installed Python 3.9.10 64 bit via WinPython (which gives me all the optimised scientific stack for free) and wxPython 4.2.0, from PyPI. Plus a gazillion of other libraries always from PyPI.

I haven’t tried any other combination, although I guess it might be worth a try downgrading wxPython to see if anything changes… you never know.

What I know for certain now is that this specific part of the code needed zero changes to work on Python 3. And going from a (potential) wx.PyAssertionError (the old wx.wxAssertionError) to a hard crash of the whole application is a large punch in the face. You would think that porting dozens of custom C/Fortran extensions from a 10 year old code to modern Cython/f2py plus numerically heavy Python codes may be one of the tallest orders, but no, the suffering is in a wx.ProgressDialog…

It is proving very difficult to reproduce the issue in a simple app, but I’ll do my best.

[Andrea_Gavana]

Hi Richard,

Thank you for the detailed example. I was actually on the same road as well, trying to replace my various wx.CallAfters with AddPendingEvent.

It was actually out of desperation, but seeing that there’s others using the same mechanism gives me some hope.

On a marginally-related note: on the exact same version of Windows, the appearance and behaviour of wx.ProgressDialog are different. I’m Python 2.7 + wxPython 2.9.4, the gauge pulses back and forth in any case (even though the text on the dialog may not have time to update), while on Python 3.9 + wxPython 4.2.0 it does not. The visuals are also different. I’ll post a picture later of what I see, just for posterity…

[da-dada]

well, this ‘dialog’ works even without main loop , so in that case posting events may not be a good idea (I think )

from threading import Thread
from time import sleep
import wx

class Action(Thread):

    def __init__(self):
        super().__init__()
        self.dlg = wx.ProgressDialog(
                'from thread', 'start',
                style=wx.PD_CAN_ABORT|wx.PD_AUTO_HIDE)
                # add 'wx.PD_APP_MODAL' for modal
        self.dlg.Show()
        self.start()

    def run(self):
        pls = 1
        while not self.dlg.WasCancelled():
            sleep(2)
            self.dlg.Pulse(newmsg=f'pulse {str(pls)}')
            pls += 1
        self.dlg.Destroy()

app = wx.App()
Action()

[AndersMunch]

@Andrea_Gavana

I seem to remember though that by default the Windows wx.ProgressDialog is not based on the generic version but it uses the native widget.

You’re right, but the structure is similar enough that it doesn’t make a difference. You just arrive at wxEventLoop::GetActive()->YieldFor via a different path.

[swt2c]

@Andrea_Gavana

Hi Richard,

Thank you for the detailed example. I was actually on the same road as well, trying to replace my various wx.CallAfters with AddPendingEvent.

It was actually out of desperation, but seeing that there’s others using the same mechanism gives me some hope.

On a marginally-related note: on the exact same version of Windows, the appearance and behaviour of wx.ProgressDialog are different. I’m Python 2.7 + wxPython 2.9.4, the gauge pulses back and forth in any case (even though the text on the dialog may not have time to update), while on Python 3.9 + wxPython 4.2.0 it does not. The visuals are also different. I’ll post a picture later of what I see, just for posterity…

wxPython 2.9.4 (released over 10 years ago!) -> wxPython 4.2.0 is a massive change. Specifically, there have been a large number of changes to wxProgressDialog on MSW:

History for src/msw/progdlg.cpp - wxWidgets/wxWidgets

[Andrea_Gavana]

Yes yes, no doubt many things have changed in 10 years, it was just a bit surprising (to me) how different they look and behave. But it’s an irrelevant detail.

For the record, I think I found where the issue was, and in preliminary tests I don’t see the access violation anymore - but it’s way too early to celebrate.

Many years ago, a younger (and smartass) me had decided to put this line buried somewhere in the numerically-heavy section of the code:

sys.setcheckinterval(8192)

My older self (less smartass but still smartass enough) simply googled the replacement for that function (since in Python 3.9 it does not exist anymore) and substituted it like this:

sys.setswitchinterval(8192)

Without checking that the behaviour was the same. It definitely was not, by a mile or more.

From the docs (emphasis mine):

sys.setcheckinterval(interval )

Set the interpreter’s “check interval”. This integer value determines how often the interpreter checks for periodic things such as thread switches and signal handlers. The default is 100, meaning the check is performed every 100 Python virtual instructions.

sys.setswitchinterval(interval )

Set the interpreter’s thread switch interval (in seconds)

Both functions attempt to play with thread scheduling timings for the interpreter, but thei inputs and behaviour are completely different.

I will soon see if I’m out of the woods or if there’s yet another bomb ticking behind a yet-undiscovered subtle change.

[da-dada]

well, I don’t think CallAfter or Python’s Threads cause any problems with this ‘dialog’

from threading import Thread, Lock
from time import sleep
import wx

class Gui(wx.Frame):

    def __init__(self, parent):
        super().__init__(parent, title='ProgressDialog')

        wx.Button(self, label='left click me, please')
        self.Bind(wx.EVT_BUTTON, lambda _: self.action())
        self.Bind(wx.EVT_CLOSE, self.evt_close)
        self.Centre()
        self.Show()
        self.thrs = 0
        self.lock = Lock()
        self.action()

    def action(self):
        Action(self.thrs, self.thrs_ref)

    def thrs_ref(self, plus):
        with self.lock:
            if plus:
                self.thrs += 1
            else:
                self.thrs -= 1

    def evt_close(self, _):
        if self.thrs > 0:
            wx.Bell()
        else:
            self.Destroy()

class Action(Thread):

    def __init__(self, thrs, thrs_ref):
        super().__init__()
        thrs_ref(True)
        self.thrs_ref = thrs_ref
        self.dlg = wx.ProgressDialog(
                f'from thread {thrs}', 'start',
                style=wx.PD_CAN_ABORT|wx.PD_AUTO_HIDE)
        self.dlg.Show()
        self.start()

    def run(self):
        pls = 1
        while not self.dlg.WasCancelled():
            sleep(0.01)
            wx.CallAfter(self.pulse, pls)
            if pls > 500:
                break
            pls += 1
        self.thrs_ref(False)

    def pulse(self, pls):
        self.dlg.Pulse(newmsg=f'pulse {str(pls)}')

app = wx.App()
Gui(None)
app.MainLoop()