wx.CallLater being very late

Hey everyone,

In my wxPython app I have an EVT_IDLE handler that calls some function
that has to be called once every 150 milliseconds or so. After calling
the function, the handler calls:
wx.CallLater(150,self._clear_idle_block_and_do)

That `_clear_idle_block_and_do` function basically posts another
EVT_IDLE event, continuing the cycle.

Now I'm noticing that when other widgets in the GUI are doing hard
work, the EVT_IDLE event handler hardly gets called! Sometime it is
not called for 4 seconds, which is way too much.

Is this because wx.CallLater is not performing well? Is there anything
I could do?

Hello,

Hey everyone,

In my wxPython app I have an EVT_IDLE handler that calls some function
that has to be called once every 150 milliseconds or so. After calling
the function, the handler calls:
wx.CallLater(150,self._clear_idle_block_and_do)

That `_clear_idle_block_and_do` function basically posts another
EVT_IDLE event, continuing the cycle.

Now I'm noticing that when other widgets in the GUI are doing hard
work, the EVT_IDLE event handler hardly gets called! Sometime it is
not called for 4 seconds, which is way too much.

Well this is to be expected because IDLE events are only sent when the
app is idle ;). CallLater uses a Timer and basic timer events on most
platforms are a low priority event, meaning that they are only
processed when the event queue is empty (i.e the app is idle).

Is this because wx.CallLater is not performing well? Is there anything
I could do?

If the stuff being done in your other widgets is blocking the main
thread then no other gui code can run while this is going on. You can
try to move some work into a separate thread or break the work up into
smaller pieces and give the app a chance to process the event queue in
between intervals. Hard to advise more without a better idea of what
your application is doing.

Cody

···

On Fri, Aug 14, 2009 at 8:51 AM, cool-RR<ram.rachum@gmail.com> wrote:

Okay Cody, this is interesting. Let's break this down: You said IDLE
events are called only when the app is idle. But I said I'm calling
the idle event *manually*. Does wxPython still give it low priority
over other events or something?
If the answer is yes, then I will need a way to modify this priority.
If the answer is no, then I'm stumped and I'll probably have to give
you a more complete description of my app.

Ram.

···

On Aug 14, 4:03 pm, Cody Precord <codyprec...@gmail.com> wrote:

Hello,

On Fri, Aug 14, 2009 at 8:51 AM, cool-RR<ram.rac...@gmail.com> wrote:

> Hey everyone,

> In my wxPython app I have an EVT_IDLE handler that calls some function
> that has to be called once every 150 milliseconds or so. After calling
> the function, the handler calls:
> wx.CallLater(150,self._clear_idle_block_and_do)

> That `_clear_idle_block_and_do` function basically posts another
> EVT_IDLE event, continuing the cycle.

> Now I'm noticing that when other widgets in the GUI are doing hard
> work, the EVT_IDLE event handler hardly gets called! Sometime it is
> not called for 4 seconds, which is way too much.

Well this is to be expected because IDLE events are only sent when the
app is idle ;). CallLater uses a Timer and basic timer events on most
platforms are a low priority event, meaning that they are only
processed when the event queue is empty (i.e the app is idle).

> Is this because wx.CallLater is not performing well? Is there anything
> I could do?

If the stuff being done in your other widgets is blocking the main
thread then no other gui code can run while this is going on. You can
try to move some work into a separate thread or break the work up into
smaller pieces and give the app a chance to process the event queue in
between intervals. Hard to advise more without a better idea of what
your application is doing.

Cody

Hello,

Okay Cody, this is interesting. Let's break this down: You said IDLE
events are called only when the app is idle. But I said I'm calling
the idle event *manually*. Does wxPython still give it low priority
over other events or something?
If the answer is yes, then I will need a way to modify this priority.
If the answer is no, then I'm stumped and I'll probably have to give
you a more complete description of my app.

What I interpret from above is that you have an IDLE handler that in
turn calls CallLater that in turn will invoke a function that posts
another idle event. Please correct me if I am miss-reading that.

So when the app gets blocked up it will not be generating its own IDLE
events and the TIMER event used by CallLater will also not be
processed until the event queue is emptied out or when the other
currently running code releases the processor and the app has a chance
to work through the queue.

I am not sure how the idle events are handled in the wx layer (where
your manually posted ones would exist), but I would guess that it
mimics the underlying system library in that they are handled after
events of other types have been dispatched. It probably doesn't make
much sense for you to post IDLE events, it would probably be better
use a custom event type or simply call your idle handler directly.

Though this does not really sound like the main issue, the main issue
sounds like you have some other code that is blocking the main thread,
so no matter what type of event you send it will still not be
processed until the blocking code has completed and released its hold
on the cpu. Does your app become non responsive when the 'other
widgets' are busy?

Cody

···

On Fri, Aug 14, 2009 at 9:07 AM, cool-RR<ram.rachum@gmail.com> wrote:

Hello,

> Okay Cody, this is interesting. Let's break this down: You said IDLE
> events are called only when the app is idle. But I said I'm calling
> the idle event *manually*. Does wxPython still give it low priority
> over other events or something?
> If the answer is yes, then I will need a way to modify this priority.
> If the answer is no, then I'm stumped and I'll probably have to give
> you a more complete description of my app.

What I interpret from above is that you have an IDLE handler that in
turn calls CallLater that in turn will invoke a function that posts
another idle event. Please correct me if I am miss-reading that.

You got it right.

So when the app gets blocked up it will not be generating its own IDLE
events and the TIMER event used by CallLater will also not be
processed until the event queue is emptied out or when the other
currently running code releases the processor and the app has a chance
to work through the queue.

I see. You are saying that the function I specify with CallLater will
not actually be called until all events are cleared out, even though
the time I specified has long since passed.

I am not sure how the idle events are handled in the wx layer (where
your manually posted ones would exist), but I would guess that it
mimics the underlying system library in that they are handled after
events of other types have been dispatched. It probably doesn't make
much sense for you to post IDLE events, it would probably be better
use a custom event type or simply call your idle handler directly.

I just create a custom event type to replace EVT_IDLE - Didn't help.

I don't think that calling the function directly, as opposed to
calling a function that posts an event will help - I checked and even
that event-calling function doesn't get called in time.

Though this does not really sound like the main issue, the main issue
sounds like you have some other code that is blocking the main thread,
so no matter what type of event you send it will still not be
processed until the blocking code has completed and released its hold
on the cpu. Does your app become non responsive when the 'other
widgets' are busy?

No, it does not. There are three busy widgets, all getting refreshed
continuously. I understand that each of them takes CPU, but it's all
done in little units, and I think it should be possible for wxPython
to do my background process in between these units.

Any more ideas?

···

On Aug 14, 4:28 pm, Cody Precord <codyprec...@gmail.com> wrote:

On Fri, Aug 14, 2009 at 9:07 AM, cool-RR<ram.rac...@gmail.com> wrote:

Hello,

So when the app gets blocked up it will not be generating its own IDLE
events and the TIMER event used by CallLater will also not be
processed until the event queue is emptied out or when the other
currently running code releases the processor and the app has a chance
to work through the queue.

I see. You are saying that the function I specify with CallLater will
not actually be called until all events are cleared out, even though
the time I specified has long since passed.

Yes, because CallLater starts a one shot timer that will fire at the
time you specified. When its timer fires it posts a TIMER event to a
temporary handler that it setup, and as mentioned TIMER events are low
priority and are usually only processed in idle time, so that timer
handler will not be called on time if other things are busy.

I just create a custom event type to replace EVT_IDLE - Didn't help.

I don't think that calling the function directly, as opposed to
calling a function that posts an event will help - I checked and even
that event-calling function doesn't get called in time.

Well if this is happening then it means the other running code is
blocking the execution of the code on that thread and that you will
like need to find a more efficient way of doing what your doing.

No, it does not. There are three busy widgets, all getting refreshed
continuously. I understand that each of them takes CPU, but it's all
done in little units, and I think it should be possible for wxPython
to do my background process in between these units.

Any more ideas?

It all depends on how 'busy' the widgets are, if they are constantly
redrawing themselves or updating like that in some manner the event
queue will be flooded with paint and other events.

What are the busy widgets doing? How continuously are they being
refreshed, you might be refreshing them more than you need to or
something.

Cody

···

On Fri, Aug 14, 2009 at 10:35 AM, cool-RR<ram.rachum@gmail.com> wrote:

cool-RR wrote:

What I interpret from above is that you have an IDLE handler that in
turn calls CallLater that in turn will invoke a function that posts
another idle event. Please correct me if I am miss-reading that.

You got it right.

Any more ideas?

yes -- get away from Idle event altogether -- I've never found a use for them.

From what I understand, what you want is for something to happen about 150 milliseconds or so. In that case, what you want is a wx.Timer, set to fire every 150ms. It won't guarantee that it will happen, and can still et blocked by a long-running event handler, but it's likely to be more reliable -- give it a try.

-Chris

···

On Aug 14, 4:28 pm, Cody Precord <codyprec...@gmail.com> wrote:

--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov

Chris, I did get away from the idle event - didn't help. Also, I'm
using wx.CallLater, which I was told runs a wx.Timer.

···

On Aug 14, 6:08 pm, Christopher Barker <Chris.Bar...@noaa.gov> wrote:

cool-RR wrote:
> On Aug 14, 4:28 pm, Cody Precord <codyprec...@gmail.com> wrote:
>> What I interpret from above is that you have an IDLE handler that in
>> turn calls CallLater that in turn will invoke a function that posts
>> another idle event. Please correct me if I am miss-reading that.

> You got it right.
> Any more ideas?

yes -- get away from Idle event altogether -- I've never found a use for
them.

From what I understand, what you want is for something to happen about
150 milliseconds or so. In that case, what you want is a wx.Timer, set
to fire every 150ms. It won't guarantee that it will happen, and can
still et blocked by a long-running event handler, but it's likely to be
more reliable -- give it a try.

-Chris

--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris.Bar...@noaa.gov

Yes, because CallLater starts a one shot timer that will fire at the
time you specified. When its timer fires it posts a TIMER event to a
temporary handler that it setup, and as mentioned TIMER events are low
priority and are usually only processed in idle time, so that timer
handler will not be called on time if other things are busy.

Maybe there is a way to make Timer events have higher priority?

It all depends on how 'busy' the widgets are, if they are constantly
redrawing themselves or updating like that in some manner the event
queue will be flooded with paint and other events.

Yes, they are constantly redrawing themselves. I understand that the
event queue gets flooded, and that's okay: I just want the events that
call my background function to have a high enough priority in the
queue, so they will not be starved by the other events.

What are the busy widgets doing? How continuously are they being
refreshed, you might be refreshing them more than you need to or
something.

My program is a platform for running simulations. It's called
GarlicSim. The three busy widgets are the following:

seek_bar - A seek bar that displays which time point of the simulation
we are watching, and what time segment of the simulation is ready for
watching.

tree_browser - A widgets that shows the "tree" of the simulation. The
tree is usually just a straight line, but if the user wants to explore
alternate scenarios, he will make a split in the tree.

state_showing_window - A widget that shows the state of the simulated
world in the selected point in time.

When playing back the simulation, all three widgets must be constantly
updated.
The "background function" I have to call every 150ms is a function
that collects simulation data from worker threads and integrated them
into the tree.

···

-----

The best idea I have so far is to change the priority of Timer events
somehow. Do you know how? Or do you have a better idea?

Hello,

Yes, because CallLater starts a one shot timer that will fire at the
time you specified. When its timer fires it posts a TIMER event to a
temporary handler that it setup, and as mentioned TIMER events are low
priority and are usually only processed in idle time, so that timer
handler will not be called on time if other things are busy.

Maybe there is a way to make Timer events have higher priority?

No I dont think so, but as previously mentioned this will not fix your
problem, as you said that even with your custom event type it was not
being called on time.

When playing back the simulation, all three widgets must be constantly
updated.

What I asked was how often are they updating themselves (time
interval?). Because you may be updating them far more often than you
need to, or faster than a human eye can even discern.

The "background function" I have to call every 150ms is a function
that collects simulation data from worker threads and integrated them
into the tree.

To simplify this as Chris had mentioned I would get rid of the IDLE
handling and CallLater and just have a single timer that you start and
tell it to fire every 150ms and use an EVT_TIMER handler.

self.timer = wx.Timer(self)
...
self.Bind(wx.EVT_TIMER, self.OnTimer)
...
self.timer.Start(150)
def OnTimer(self,evt):
    do_your background_function()

If its blocking to much for this to be called in a timely manner you
need to figure out how to improve the efficiency of what your doing
elsewhere.

Short story is that you are likely blocking the main thread to much
with your other 'continuous updates' so you need to figure out how to
work around that. You could try Yielding in some places to let the
event queue be dispatched but there is always a danger of recursive
overflow if your not careful with Yield.

Cody

···

On Fri, Aug 14, 2009 at 11:19 AM, cool-RR<ram.rachum@gmail.com> wrote:

> Maybe there is a way to make Timer events have higher priority?

No I dont think so, but as previously mentioned this will not fix your
problem, as you said that even with your custom event type it was not
being called on time.

I think you're mixing things up here - Even with my custom event, I
was still dependent on the timer event to make things work in time. So
I think if we could change the priority of the timer events, it will
solve the problem.

> When playing back the simulation, all three widgets must be constantly
> updated.

What I asked was how often are they updating themselves (time
interval?). Because you may be updating them far more often than you
need to, or faster than a human eye can even discern.

Once every 50ms, which I think is not that frequent.

> The "background function" I have to call every 150ms is a function
> that collects simulation data from worker threads and integrated them
> into the tree.

To simplify this as Chris had mentioned I would get rid of the IDLE
handling and CallLater and just have a single timer that you start and
tell it to fire every 150ms and use an EVT_TIMER handler.

self.timer = wx.Timer(self)
...
self.Bind(wx.EVT_TIMER, self.OnTimer)
...
self.timer.Start(150)
def OnTimer(self,evt):
do_your background_function()

Did that right now - still the same problem.

If its blocking to much for this to be called in a timely manner you
need to figure out how to improve the efficiency of what your doing
elsewhere.

Short story is that you are likely blocking the main thread to much
with your other 'continuous updates' so you need to figure out how to
work around that. You could try Yielding in some places to let the
event queue be dispatched but there is always a danger of recursive
overflow if your not careful with Yield.

Unless you have any idea about how to make the timer events have
higher priority, I'd be interested to read about wx.Yield. I couldn't
find any explanation about it.

Ram.

Hello,

I think you're mixing things up here - Even with my custom event, I
was still dependent on the timer event to make things work in time. So
I think if we could change the priority of the timer events, it will
solve the problem.

> When playing back the simulation, all three widgets must be constantly
> updated.

What I asked was how often are they updating themselves (time
interval?). Because you may be updating them far more often than you
need to, or faster than a human eye can even discern.

Once every 50ms, which I think is not that frequent.

Actually this is probably a bit aggressive I would think.

Unless you have any idea about how to make the timer events have
higher priority, I'd be interested to read about wx.Yield. I couldn't
find any explanation about it.

I still think that the main problem is your blocking the main thread
with your updates but... If you want to try it you can make your own
thread timer class using the threading and time modules and see if
that does anything different.

i.e) use something like this to replace wx.Timer
import threading
import time
class ThreadTimer(threading.Thread):
   def __init__(self, parent)
        self.parent = parent
        self.continue = False
        self.interval = 0

   def run(self):
       while continue:
           time.sleep(self.interval)
           event = MyCustomTimerEvent(...)
           wx.PostEvent(self.parent, event)

   def Start(self, interval):
       self.interval = interval
       self.start()

   def Stop(self):
       self.continue = False

Cody

···

On Fri, Aug 14, 2009 at 11:57 AM, cool-RR<ram.rachum@gmail.com> wrote:

Cody Precord wrote:
[...]

I am not sure how the idle events are handled in the wx layer (where
your manually posted ones would exist), but I would guess that it
mimics the underlying system library in that they are handled after
events of other types have been dispatched.

The real EVT_IDLE events are sent when the system event queue *becomes* empty. A common misconception is that they are sent continuously while the app is idle, but unless you tell it otherwise (by calling event.RequestMore) they are only sent once at the beginning of the idle period. For the manually sent idle events then it depends on how they are sent, ProcessEvent will cause the event to be processed within the scope of the ProcessEvent call, while wx.PostEvent will add it to the app's pending event queue and they will be processed some time later.

On the other hand, just the act of getting a timer event will cause the system event queue to *become* empty once more. That means that an idle event will be coming soon anyway so manually sending one from the timer handler is a bit of overkill.

It probably doesn't make
much sense for you to post IDLE events, it would probably be better
use a custom event type or simply call your idle handler directly.

Agreed.

Though this does not really sound like the main issue, the main issue
sounds like you have some other code that is blocking the main thread,
so no matter what type of event you send it will still not be
processed until the blocking code has completed and released its hold
on the cpu.

Agreed. Cool-RR, please (re)read LongRunningTasks - wxPyWiki and Non-Blocking Gui - wxPyWiki

···

--
Robin Dunn
Software Craftsman

Cody Precord wrote:

Hello,

Yes, because CallLater starts a one shot timer that will fire at the
time you specified. When its timer fires it posts a TIMER event to a
temporary handler that it setup, and as mentioned TIMER events are low
priority and are usually only processed in idle time, so that timer
handler will not be called on time if other things are busy.

Maybe there is a way to make Timer events have higher priority?

No I dont think so,

Correct. At least wx has no way to do it, I don't know about platform-specific APIs. (wx.Timers are built on top of the timers provided by the underlying platform, and EVT_TIMER events are sent directly from whatever notification that the system provides of a timer expiration. IOW, wx does very little work in the timers beyond what the system provides.)

but as previously mentioned this will not fix your
problem, as you said that even with your custom event type it was not
being called on time.

Agreed. If the gui thread is spending too much time outside the MainLoop then delivery time of all types of events will suffer.

When playing back the simulation, all three widgets must be constantly
updated.

What I asked was how often are they updating themselves (time
interval?). Because you may be updating them far more often than you
need to, or faster than a human eye can even discern.

150ms is only about 6-7 frames per second, so that is well below the human eye threshold... Maybe a better question to ask is how long does it take to do the processing and the drawing that is being done every 150ms? If it's taking 151ms or more then it becomes clear that there is more work required than can be done in the time allotted and that the gui thread is too busy to be able to keep up with the timer events and paint events and everything else. I'd guess that even if it only took 100ms that it is still too much. So if cool-RR can determine how much time it takes to do the work as the app is currently organized then that will help determine how to rebalance things so the gui thread will not be overloaded.

As an extreme goal try to move all the heavy lifting to another thread (or even another process) and then make your timer event handler do nothing but fetch the next chunk of processed data and do a Refresh(), and your paint handlers do nothing but convert the fetched data to a drawing usign wx.DC methods. You may not need to go all the way to that extreme as you may find that somewhere between there and where you are now is good enough.

Also, if you are not familiar with it already you should learn about the problems that Python has with multi-threaded apps on a multi-core system when one or more threads are CPU bound. That may drive some of your design decisions too. There was a link posted here a few weeks ago of a video of David Beasley's talk about it.

···

On Fri, Aug 14, 2009 at 11:19 AM, cool-RR<ram.rachum@gmail.com> wrote:

--
Robin Dunn
Software Craftsman

cool-RR wrote:

When playing back the simulation, all three widgets must be constantly
updated.

What I asked was how often are they updating themselves (time
interval?). Because you may be updating them far more often than you
need to, or faster than a human eye can even discern.

Once every 50ms, which I think is not that frequent.

If the data is only being updated every 150ms then why is the drawing being done every 50?

I'd be interested to read about wx.Yield. I couldn't
find any explanation about it.

It essentially makes a nested, temporary event loop so you can cause pending events to be dispatched without needing to return control from whatever is taking a long time back to the MainLoop. Yielding should be used very carefully however as your event handlers will need to either be reentrant or to protect themselves from reentry, and it is an error for wx.Yield to be called from within some other wx.Yield call.

···

--
Robin Dunn
Software Craftsman

Robin Dunn wrote:

As an extreme goal try to move all the heavy lifting to another thread (or even another process) and then make your timer event handler do nothing but fetch the next chunk of processed data and do a Refresh(), and your paint handlers do nothing but convert the fetched data to a drawing usign wx.DC methods.

Or you can do the drawing in the timer handler with the target being a wx.Bitmap in a MemoryDC. The paint events then can be nothing but a DrawBitmap call, then they are super efficient and that will benefit the refreshes that happen, for example, when part of the window is damaged by other windows being dragged over your app.

···

--
Robin Dunn
Software Craftsman

> Once every 50ms, which I think is not that frequent.

Actually this is probably a bit aggressive I would think.

Aggressive? That is only 20 fps.

> Unless you have any idea about how to make the timer events have
> higher priority, I'd be interested to read about wx.Yield. I couldn't
> find any explanation about it.

I still think that the main problem is your blocking the main thread
with your updates but... If you want to try it you can make your own
thread timer class using the threading and time modules and see if
that does anything different.

i.e) use something like this to replace wx.Timer
import threading
import time
class ThreadTimer(threading.Thread):
def __init__(self, parent)
self.parent = parent
self.continue = False
self.interval = 0

def run(self):
while continue:
time.sleep(self.interval)
event = MyCustomTimerEvent(...)
wx.PostEvent(self.parent, event)

def Start(self, interval):
self.interval = interval
self.start()

def Stop(self):
self.continue = False

Cody

Works great! I edited it and now my background function gets called
consistently even when the GUI is very busy.
Here's my edited version:

#threadtimer.py

import threading
import time
import wx

wxEVT_THREAD_TIMER = wx.NewEventType()
EVT_THREAD_TIMER = wx.PyEventBinder(wxEVT_THREAD_TIMER, 1)

class ThreadTimer(object):
   def __init__(self, parent):
        self.parent = parent
        self.thread = Thread()
        self.thread.parent = self
        self.alive = False

   def start(self, interval):
       self.interval = interval
       self.alive = True
       self.thread.start()

   def stop(self):
       self.alive = False

class Thread(threading.Thread):
    def run(self):
       while self.parent.alive:
           time.sleep(self.parent.interval / 1000.0)
           event = wx.PyEvent()
           event.SetEventType(wxEVT_THREAD_TIMER)
           wx.PostEvent(self.parent.parent, event)

Thanks!

Now I have another problem. This one happened before this new
ThreadTimer, and still happens now. When my GUI is busy, and I press
the X button to close the window, it doesn't close it. Only when the
GUI stop working (because the simulation playback has finished) does
the application quit.
Any idea what to do?

Ram.

The update that happens once every 150ms brings in new "frames" of the
simulation, it may be a 100 frames at a time. The 50ms delay is the
delay between displaying one frame and the next.

As I said in my post, the timer problem is solved, thanks to
ThreadTimer. Now my only problem is the non-responsive X button.

···

On Aug 14, 7:23 pm, Robin Dunn <ro...@alldunn.com> wrote:

cool-RR wrote:

>>> When playing back the simulation, all three widgets must be constantly
>>> updated.
>> What I asked was how often are they updating themselves (time
>> interval?). Because you may be updating them far more often than you
>> need to, or faster than a human eye can even discern.

> Once every 50ms, which I think is not that frequent.

If the data is only being updated every 150ms then why is the drawing
being done every 50?

Robin Dunn wrote:

be reentrant or to protect themselves from reentry, and it is an error for wx.Yield to be called from within some other wx.Yield call.

Doesn't SafeYield handle this? It's generally worked for me.

-Chris

···

--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov

cool-RR wrote:

Now I have another problem. This one happened before this new
ThreadTimer, and still happens now. When my GUI is busy, and I press
the X button to close the window, it doesn't close it. Only when the
GUI stop working (because the simulation playback has finished) does
the application quit.
Any idea what to do?

What platform is it? If on Windows and you have paint events that do not create a wx.PaintDC for the window that sent the event then change it so they do. I've seen this problem before if there are paint handlers that don't create a PintDC or do create one but for the wrong window.

Otherwise, it sounds like more of what we've been discussing already, something is preventing the normal processing of events from happening.

···

--
Robin Dunn
Software Craftsman