Timer Module vs. Windows OS Questions

Hello, All.

This post is to get some general advice and answers from the Group to
some questions about Python Timer modules and the Windows Operating
System.

Please note that this isn't just a question of how to write a timer
application in Python code; everyone has been pointing out a number of
examples for me to study in that regard; Thank You for that.

These questions are to try to get a clearer understanding of the Timer
modules themselves - their limitations and characteristics, and what
they are connecting to "behind the scenes" on the PC. I am also trying
to get a basic understanding of timer structuring on the PC in
general, for I believe the selection of one Timer module over another
might make an important difference in my particular application.

So forgive me if my questions are a bit unusual, but hopefully, I have
made my concerns clear enough to understand and answer.

The best way to begin is by explaining what I am ultimately desiring
to do in Python:

I wish to run a GUI program in the Main Python application, which will
collect User Commands, and display Operational Status on a laptop
screen. Meanwhile, in the background, on a regular timed basis, the
program flow will pass to a routine that will output the User Commands
and input new Operational Status by means of a very short packet
exchange on the Ethernet port. The routine will then return to the
Main GUI application, passing on the new parameters to the GUI,
updating it.

This background process will operate continually, at a regular fixed-
timed interval. I am currently targeting a time-slice update rate of
about 33.33 ms. (30 times a second), but this value isn't etched in
stone. I could probably go to 66.66 ms. (or 15 times a second), if
absolutely need be.

What is critical, is that this update rate doesn't vary greatly in
time from update to update (resulting in what is referred to as "Data
Frame Jitter". That is a problem that I must put maximum effort in to
avoid.

And that point brings up my first set of questions in regards to how
PCs can reliably produce this time interval:

For example, "wxPython in Action" makes it clear that in regards to
the wx.Timer module, "Not all operating systems have system clocks
with millisecond precision." So if I can't set the timer time in
milliseconds, what does it get set to - SECONDS? How can I know which
systems work in milliseconds and which don't - I can't put out a
distribution disk with unknowns like that.

As an explanation, I have been told and have read that the Windows
Operating Systems have no real reliable clock sources to hook software
methods to; that they vary from PC-to-PC, OS-to-OS, etc. Because of
this, most Python-based Timer modules are either very coarse in
resolution, or are sloppy in performance from machine-to-machine.

Okay. Second set of questions: Video player software must put out
video frames at a fixed reliable rate. Wave sound files decode at
typical sample rates of 44.1 khz. These rates must be stable and
reliable, or all sorts of distortion occurs to the media. How are they
keeping stable update times there, all across the Windows OS? Does
multimedia modules like PyGame have better Timer modules to access?

I heard there was a problem similar to this in Windows-based "C"
routines; using the standard API Timer function would vary the setting
resolution from 10 millisecond to 50 millisecond steps, depending on
the OS. Many programmers resolved the resolution problem by going to
the Multimedia Timer. Has Python done something similar?

Does the wx.Timer have some known standards around it, where a Windows
OS version such as XP guarantees the millisecond resolution setting?
Is there a wxPython GetData() type method that will let you know what
the fastest reliable rate that can be used for timer intervals?

Are there ways a programmer can make up his own code to get around
this problem, in a more "long-handed" variation to applying the Timer
modules normally?

I can't believe a program language as all-powerful as Python could be
stymied by something as fundamental as this. It's obviously just my
incredible ignorance in this area. Hopefully, a little explanation to
me will help eliminate my confusion.

This is what I meant when I mentioned that my questions were more
fundamental than just requesting examples on implementing the Timer
module itself.

Another issue is how to empirically test that the Timer routines
operate correctly, and at the right interval. I'm not exactly sure how
that is done with software checks, but here's some ideas I wanted to
run by everyone:

One test was this: I took the Timer example in "wxPython in Action",
deactivated the clock frame display, and set up the "OnTimer" method
to accumulate the amount of event calls to that method by the Timer. I
then made a single print of this value to the command prompt window
when the amount should have equaled every 15 seconds in elapsed time.
The 15 second prints were to minimize any extra processor operations
other than timing functions. I then started the Timer with different
millisecond attributes to see how it handled those. I then compared
the printed number on the screen with a stopwatch after 30 minutes
elapsed time.

I found a Windows 98 OS was "slipping" horribly when I used Timer
resolutions of 100, 250, & 500 milliseconds. The same tests on a
Windows XP OS looked to be a lot tighter to the actual time. The
problem was that this was a pretty stupid approach to check this, and
I'm not sure I really measured anything conclusively.

However, as a mostly hardware-based designer, I came up with two
alternative ideas that I believe would take all question out of the
issue:

One idea is to install the PySerial module, and on every Timer event,
toggle a Serial signal pin on the Serial port. I then put an
oscilloscope on it, and measure the resulting square waveform. This
will also show me any Data Frame Jitter, if it is occurring.

Another idea was to send out a brief UDP packet out the Ethernet port,
instead of toggling the Serial bit. I then monitor the packets with
Wireshark, which has a "packet received time" stamped in microseconds
for each received packet. By the way, how are THEY able to get such
accurate time within THEIR program?

These are the main issues I am in the dark about, so I should stop
here. If this post is too long, I apologize. I don't expect any one
person to try to chew on this whole issue themselves. If someone has
an obvious answer to one or two of the above questions, perhaps they
could post their thoughts on just those. That way I can get the
answers over time, without taking too much of anyone's time.

Thanks for your kind attention,

Regards,

Kit

This doesn’t mean that the system clock doesn’t understand milliseconds - I would assume that the wx.Timer function always operates on the level of milliseconds, regardless of internal clock precision. It is simply addressing precision: how accurately the OS actually measures the requested interval. In other words, on some OSes wxPython may really fire off the timer events every 33 milliseconds, but some might vary from (for instance) 31 to 35ms in practice. This isn’t a problem for most apps - if it is for yours, you should avoid whatever operating systems do this. (Not sure which these are, however - for my programs, time precision is unimportant.)

-Nat

···

On Mon, Mar 29, 2010 at 12:54 PM, Kit kit.2000@yahoo.com wrote:

For example, “wxPython in Action” makes it clear that in regards to

the wx.Timer module, "Not all operating systems have system clocks

with millisecond precision." So if I can’t set the timer time in

milliseconds, what does it get set to - SECONDS? How can I know which

systems work in milliseconds and which don’t - I can’t put out a

distribution disk with unknowns like that.

Correct. On Windows the precision is in the neighborhood of 55ms (at least that's the way it was the last time I checked) and on the other platforms it's somewhat better than that. You still specify your desired timeout in ms no matter what, and the system will do it's best to deliver timer events on schedule, but it won't be perfect.

However keep in mind that there is not only the documented precision level of the system clock that can effect this, but it also depends on what else your gui thread is doing at the time. The APIs that wx.Timer are built upon basically just add a WM_TIMER (or equivalent) message on the queue as close to the desired timeout as possible, but this is a low priority event so if there is other stuff going on at the time it can be several (or many) more milliseconds before a timer event is actually delivered and the handler is called.

There are other options if more precision or shorter timeout intervals are needed, but you'll need to use platform-specific code to do it. Although Python probably isn't the best choice for real-time programming in any case.

···

On 3/29/10 1:11 PM, Nathaniel Echols wrote:

On Mon, Mar 29, 2010 at 12:54 PM, Kit <kit.2000@yahoo.com > <mailto:kit.2000@yahoo.com>> wrote:

    For example, "wxPython in Action" makes it clear that in regards to
    the wx.Timer module, "Not all operating systems have system clocks
    with millisecond precision." So if I can't set the timer time in
    milliseconds, what does it get set to - SECONDS? How can I know which
    systems work in milliseconds and which don't - I can't put out a
    distribution disk with unknowns like that.

This doesn't mean that the system clock doesn't understand milliseconds
- I would assume that the wx.Timer function always operates on the level
of milliseconds, regardless of internal clock precision. It is simply
addressing precision: how accurately the OS actually measures the
requested interval. In other words, on some OSes wxPython may really
fire off the timer events every 33 milliseconds, but some might vary
from (for instance) 31 to 35ms in practice.

--
Robin Dunn
Software Craftsman

To Robin & Nat:

I am not at this current time needing to panic over a few milliseconds
variances in the wxTimer call interval. But I have problems if we're
talking +/- 25%+ or something like that. Also, I assume this problem
reduces as you increase the millisecond attribute value.

For example, SetValue=30ms. and OSAccuracy=50ms. equals
Performance=SetValue +/- 66%?

Therefore, SetValue=50ms. and OSAccuracy=50ms. equals
Performance=SetValue +/- 0%?

I'm not saying that this is a valid formula here, but hopefully you
get what I'm asking? That once I get my SetValue => OSAccuracy, the
Performance tracks pretty tight?

Would threading help this overall problem, or simply convolute it? I
would make the Timed update routine the top priority thread, over
GUIs, since I really do not care what the sacrifice in time during the
GUI updates are (so long as it isn't in viewable lags). I only want to
update numerical entries, not play with any graphics.

I've heard of embedding C code in-line with Python. Would making some
C-based multimedia timer service routine, and embedding it into the
overall Python GUI routine work here?

In regards to platform-specific code, as I mentioned in my original
posting, I am trying to make this work over as wide a range of Windows
OS as I can, but if I have to sacrifice this to keep a necessary
feature, so be it. So If I dropped the Windows98 compatibility, and
mandated Windows XP or better (and some minimum processor speed),
would that provide the "fudge room" needed?

Regards,

Kit

···

On Mar 30, 7:57 am, Robin Dunn <ro...@alldunn.com> wrote:

On 3/29/10 1:11 PM, Nathaniel Echols wrote:

> On Mon, Mar 29, 2010 at 12:54 PM, Kit <kit.2...@yahoo.com > > <mailto:kit.2…@yahoo.com>> wrote:

> For example, "wxPython in Action" makes it clear that in regards to
> the wx.Timer module, "Not all operating systems have system clocks
> with millisecond precision." So if I can't set the timer time in
> milliseconds, what does it get set to - SECONDS? How can I know which
> systems work in milliseconds and which don't - I can't put out a
> distribution disk with unknowns like that.

> This doesn't mean that the system clock doesn't understand milliseconds
> - I would assume that the wx.Timer function always operates on the level
> of milliseconds, regardless of internal clock precision. It is simply
> addressing precision: how accurately the OS actually measures the
> requested interval. In other words, on some OSes wxPython may really
> fire off the timer events every 33 milliseconds, but some might vary
> from (for instance) 31 to 35ms in practice.

Correct. On Windows the precision is in the neighborhood of 55ms (at
least that's the way it was the last time I checked) and on the other
platforms it's somewhat better than that. You still specify your
desired timeout in ms no matter what, and the system will do it's best
to deliver timer events on schedule, but it won't be perfect.

However keep in mind that there is not only the documented precision
level of the system clock that can effect this, but it also depends on
what else your gui thread is doing at the time. The APIs that wx.Timer
are built upon basically just add a WM_TIMER (or equivalent) message on
the queue as close to the desired timeout as possible, but this is a low
priority event so if there is other stuff going on at the time it can be
several (or many) more milliseconds before a timer event is actually
delivered and the handler is called.

There are other options if more precision or shorter timeout intervals
are needed, but you'll need to use platform-specific code to do it.
Although Python probably isn't the best choice for real-time programming
in any case.

--
Robin Dunn
Software Craftsmanhttp://wxPython.org

Kit wrote:

I am not at this current time needing to panic over a few milliseconds
variances in the wxTimer call interval. But I have problems if we're
talking +/- 25%+ or something like that. Also, I assume this problem
reduces as you increase the millisecond attribute value.

Well, yes, but if you care about timing much at all, I don't think you want to use wx.Timer. It's tied up in the event loop, and I think can get delayed by any number of things.

Would threading help this overall problem, or simply convolute it?

Threading is not all that robust in Python -- google "GIL" or "Global Interpreter Lock". In short, you can never count on a thread not getting held up for an undetermined period of time.

I've heard of embedding C code in-line with Python. Would making some
C-based multimedia timer service routine, and embedding it into the
overall Python GUI routine work here?

That may be the way to go -- though probably not in-line (weave is the only way I've heard to do that). I'd recommend calling the C routines either with ctypes or Cython.

You still may run into GIL issues. You might try the "subprocess" module in Python 2.6 -- running your timer in another process can avoid the GIL. However, you're not on a real time OS, so I don't know how good it can get, and you still need to call into your main code to get the data.

A couple thoughts:

Do some more googling, and post a question on the main Python list -- if you listen to me, this is no longer a wx question.

Think about how important the timing really is .. if it's OK for it to be on-target most of the time, maybe wx.Timer will do -- you can always try it ans see -- it's the easiest way.

-Chris

···

--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov

To Chris and All,

Thank you for your candid, and to the point replies. It frankly rose
the hairs up on my neck to find one of the Wizards of wxPython saying
that for the mere sake of needing some close monitoring to real-time,
I should consider abandoning Python altogether - especially since
Python and wxPython makes GUIs (which are 95% of my needs) something
attainable in weeks, where I have been stymied by that issue for
YEARS.

I think I would like to try and tackle conquering the time-interval
thing, using whatever tricks I could muster, before giving up on
Python (and wxPython) so easily. And again my question - HOW do they
resolve this with Python-based video and audio players, which face the
same problem? There's a missing link here, and I need to figure why
this is an impossible task for me, but not for them.

One of the main points of this thread was to admittedly determine
whether Timer modules from one toolkit or another was a better choice.
I think I'm reading here from Chris' reply that as far as this Timer
method is concerned, the wx.Timer module is not the best choice, and I
should seek answers in the more generalized Python community.

This is perfectly understandable; the whole purpose of wxPython is to
produce GUIs, not obscure Timer utilities. As far as GUIs are
concerned, I consider wxPython nothing short of **MAGICAL**. Since it
had a Timer module that seemed to do exactly what I needed, I had
hoped it would work as magically as the GUIs. The whole point of this
meandering thread was to see if I was asking the right Forum the right
questions. I respect all who contribute here, and do not want to waste
their valuable time.

I still leave my post open for discussion; ANY thoughts to the problem
will be GREATLY appreciated. I am open to UNLIMITED options to resolve
this.

For example, I have even created programs that read signals injected
into the parallel printer port from an external hardware crystal-based
clock, which I used to interrupt the program to do the update routine
(this was after everyone went to laptops, losing the ability to insert
interface cards of my own design into PC slots). NOTHING is
impossible.

I shall widen my search on solutions to the Timer thing, but I look
forward to any input this Forum has on the subject. I am in love with
wxPython, and plan to use it in my application at least for the GUIs,
as I have never been able to produce results anywhere approaching the
level created with this toolkit.

Thank you for your frank answers, and please post any thoughts you may
have in the future.

Warmest Regards,

Kit

···

On Mar 30, 2:55 pm, Christopher Barker <Chris.Bar...@noaa.gov> wrote:

Kit wrote:
> I am not at this current time needing to panic over a few milliseconds
> variances in the wxTimer call interval. But I have problems if we're
> talking +/- 25%+ or something like that. Also, I assume this problem
> reduces as you increase the millisecond attribute value.

Well, yes, but if you care about timing much at all, I don't think you
want to use wx.Timer. It's tied up in the event loop, and I think can
get delayed by any number of things.

> Would threading help this overall problem, or simply convolute it?

Threading is not all that robust in Python -- google "GIL" or "Global
Interpreter Lock". In short, you can never count on a thread not getting
held up for an undetermined period of time.

> I've heard of embedding C code in-line with Python. Would making some
> C-based multimedia timer service routine, and embedding it into the
> overall Python GUI routine work here?

That may be the way to go -- though probably not in-line (weave is the
only way I've heard to do that). I'd recommend calling the C routines
either with ctypes or Cython.

You still may run into GIL issues. You might try the "subprocess" module
in Python 2.6 -- running your timer in another process can avoid the
GIL. However, you're not on a real time OS, so I don't know how good it
can get, and you still need to call into your main code to get the data.

A couple thoughts:

Do some more googling, and post a question on the main Python list -- if
you listen to me, this is no longer a wx question.

Think about how important the timing really is .. if it's OK for it to
be on-target most of the time, maybe wx.Timer will do -- you can always
try it ans see -- it's the easiest way.

-Chris

--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris.Bar...@noaa.gov

I'm not sure what Python-based A/V software you're thinking of. The
wx.MediaCtrl just wraps native media players (I think). PyMedia is
another major library, but I don't know how that works...fortunately,
the source is there, so you're free to browse it. Mythbuntu is based
on a lot of Python scripts, so it might be worth looking at too,
although I'm pretty sure the GUI is PyGTK. The backend code is
probably interesting though.

···

On Mar 31, 12:37 am, Kit <kit.2...@yahoo.com> wrote:

To Chris and All,

Thank you for your candid, and to the point replies. It frankly rose
the hairs up on my neck to find one of the Wizards of wxPython saying
that for the mere sake of needing some close monitoring to real-time,
I should consider abandoning Python altogether - especially since
Python and wxPython makes GUIs (which are 95% of my needs) something
attainable in weeks, where I have been stymied by that issue for
YEARS.

I think I would like to try and tackle conquering the time-interval
thing, using whatever tricks I could muster, before giving up on
Python (and wxPython) so easily. And again my question - HOW do they
resolve this with Python-based video and audio players, which face the
same problem? There's a missing link here, and I need to figure why
this is an impossible task for me, but not for them.

-------------------
Mike Driscoll

Blog: http://blog.pythonlibrary.org

Haven't followed all of this thread, if this was already suggested just ignore it.

There seems to be quite a bit on the net and multiple persons have tried "real time" with Python or at least looked at it in the past, here a link which looked intesting, but there are others when you google "real time python".

http://www.its.caltech.edu/~astraw/publications/realtime_computing.html

Werner

···

On 31/03/2010 07:37, Kit wrote:

To Chris and All,

Thank you for your candid, and to the point replies. It frankly rose
the hairs up on my neck to find one of the Wizards of wxPython saying
that for the mere sake of needing some close monitoring to real-time,
I should consider abandoning Python altogether - especially since
Python and wxPython makes GUIs (which are 95% of my needs) something
attainable in weeks, where I have been stymied by that issue for
YEARS.

I think I would like to try and tackle conquering the time-interval
thing, using whatever tricks I could muster, before giving up on
Python (and wxPython) so easily. And again my question - HOW do they
resolve this with Python-based video and audio players, which face the
same problem? There's a missing link here, and I need to figure why
this is an impossible task for me, but not for them.

One of the main points of this thread was to admittedly determine
whether Timer modules from one toolkit or another was a better choice.
I think I'm reading here from Chris' reply that as far as this Timer
method is concerned, the wx.Timer module is not the best choice, and I
should seek answers in the more generalized Python community.

This is perfectly understandable; the whole purpose of wxPython is to
produce GUIs, not obscure Timer utilities. As far as GUIs are
concerned, I consider wxPython nothing short of **MAGICAL**. Since it
had a Timer module that seemed to do exactly what I needed, I had
hoped it would work as magically as the GUIs. The whole point of this
meandering thread was to see if I was asking the right Forum the right
questions. I respect all who contribute here, and do not want to waste
their valuable time.

I still leave my post open for discussion; ANY thoughts to the problem
will be GREATLY appreciated. I am open to UNLIMITED options to resolve
this.

For example, I have even created programs that read signals injected
into the parallel printer port from an external hardware crystal-based
clock, which I used to interrupt the program to do the update routine
(this was after everyone went to laptops, losing the ability to insert
interface cards of my own design into PC slots). NOTHING is
impossible.

I shall widen my search on solutions to the Timer thing, but I look
forward to any input this Forum has on the subject. I am in love with
wxPython, and plan to use it in my application at least for the GUIs,
as I have never been able to produce results anywhere approaching the
level created with this toolkit.

Thank you for your frank answers, and please post any thoughts you may
have in the future.

Hi again, Mike:

I should have clarified myself earlier where I was making the
assumptions on the multimedia timing; my apologies.

My first book on learning Python was entitled "Python Programming for
the Absolute Beginner (second edition)" by Michael Dawson.

The book included a disk with Python 2.3.5, along with versions of the
PyGame and Livewires packages. It APPEARED to me at the
time(admittedly a beginning novice) that these packages were providing
methods to run video, bitmap slides, and decode audio files by its own
internal methods, all linked somehow through Python examples in the
book. The methods had video frame rate adjustment attributes, timers
for actions and delays adjustable in milliseconds, etc.

Perhaps I just didn't understand what was really happening there, and
have it completely wrong. But the book SEEMED to be indicating it was
all Python-based operations doing this, not any pre-existent resident
Windows multimedia utilities. Since that book, I have simply assumed
getting a steady updating clock reference should be no big deal in
Python.

I see now that there is obviously more going on there than I thought,
and will take a much closer look at the whole problem. At the very
least, I should check into those packages that came on the disk with
the book.

I then learned about the fantastic wxPython GUI toolkit, and picked up
the book "wxPython in Action". Besides showing all the terrific GUI
modules I wanted to learn and use, the book's last chapter described
the wx.Timer module, which I saw would serve my purpose exactly. I at
that point just assumed that accurate timing was no big deal.

As a matter of fact, the whole purpose for writing this thread was to
see if I had my assumptions correct or not. Thanks to the Forum, I'm
getting a more realistic understanding of the issue.

Regards,

Kit

···

On Mar 31, 6:03 am, Mike Driscoll <kyoso...@gmail.com> wrote:

On Mar 31, 12:37 am, Kit <kit.2...@yahoo.com> wrote:

I'm not sure what Python-based A/V software you're thinking of. The
wx.MediaCtrl just wraps native media players (I think). PyMedia is
another major library, but I don't know how that works...fortunately,
the source is there, so you're free to browse it. Mythbuntu is based
on a lot of Python scripts, so it might be worth looking at too,
although I'm pretty sure the GUI is PyGTK. The backend code is
probably interesting though.

-------------------
Mike Driscoll

Blog: http://blog.pythonlibrary.org

I can't really understand what you're trying to do, but are you really
sure you need the precision you state you need? You said:

"I wish to run a GUI program in the Main Python application, which will
collect User Commands, and display Operational Status on a laptop
screen. Meanwhile, in the background, on a regular timed basis, the
program flow will pass to a routine that will output the User Commands
and input new Operational Status by means of a very short packet
exchange on the Ethernet port. The routine will then return to the
Main GUI application, passing on the new parameters to the GUI,
updating it.

This background process will operate continually, at a regular fixed-
timed interval. I am currently targeting a time-slice update rate of
about 33.33 ms. (30 times a second), but this value isn't etched in
stone. I could probably go to 66.66 ms. (or 15 times a second), if
absolutely need be."

Do you really want to sample user's commands 30x every second? Users
can't possibly attain anything like that rate, so why should you? And
then update the GUI 30x every second? That is physiological trackable
I guess (video is 30 fps), but is it really necessary for what you are
doing? Or am I totally misunderstanding you?

Che

···

On Wed, Mar 31, 2010 at 12:13 PM, Kit <kit.2000@yahoo.com> wrote:

Hi again, Mike:

I should have clarified myself earlier where I was making the
assumptions on the multimedia timing; my apologies.

My first book on learning Python was entitled "Python Programming for
the Absolute Beginner (second edition)" by Michael Dawson.

The book included a disk with Python 2.3.5, along with versions of the
PyGame and Livewires packages. It APPEARED to me at the
time(admittedly a beginning novice) that these packages were providing
methods to run video, bitmap slides, and decode audio files by its own
internal methods, all linked somehow through Python examples in the
book. The methods had video frame rate adjustment attributes, timers
for actions and delays adjustable in milliseconds, etc.

Perhaps I just didn't understand what was really happening there, and
have it completely wrong. But the book SEEMED to be indicating it was
all Python-based operations doing this, not any pre-existent resident
Windows multimedia utilities. Since that book, I have simply assumed
getting a steady updating clock reference should be no big deal in
Python.

I see now that there is obviously more going on there than I thought,
and will take a much closer look at the whole problem. At the very
least, I should check into those packages that came on the disk with
the book.

I then learned about the fantastic wxPython GUI toolkit, and picked up
the book "wxPython in Action". Besides showing all the terrific GUI
modules I wanted to learn and use, the book's last chapter described
the wx.Timer module, which I saw would serve my purpose exactly. I at
that point just assumed that accurate timing was no big deal.

As a matter of fact, the whole purpose for writing this thread was to
see if I had my assumptions correct or not. Thanks to the Forum, I'm
getting a more realistic understanding of the issue.

Kit wrote:

The book included a disk with Python 2.3.5, along with versions of the
PyGame

PyGame is built on the SDL library. SDL is doing all the heavy lifting.

You'll need to do something similar -- either find a library that does what you want, or maybe write something small yourself, in C or Cython, or...

I've never needed anything like this, so I'm not entirely clear on how you can have a semi-real-time component call into Python exactly when you want it to, but I'll bet some googling and asking on c.l.p will help there.

Getting back to your need -- I'm a little confused about what your near-real-time needs are. If you have a process that the user needs to able to set parameters on, etc, with a GUI, then they are going to be doing that on Human Time -- so why does your process need to get updated within milliseconds of when a change happens?

Or maybe I've misunderstood your needs.

In any case, the dynamic nature of Python makes for really nice rapid development. But it does come at some cost -- performance, timing etc. I think the real power of Python is that it can easily be extended to tap into code written in other languages, so those small parts of your program that Python is not suitable for can be written in other ways.

As you've discovered, GUI programming is a lot of code -- doing all that in C, C++, whatever, so that one tiny bit of your program can run faster is crazy!

It looks like Andrew Straws SciPy paper (linked by Werner) may be just what you are looking for, however, it looks like the link is broken. I'll bet you can find a copy somewhere, or send a note to Andrew and ask him for it.

-Chris

···

--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov

They do it in C extension modules, or perhaps directly to external DLLs or shared libraries via ctypes.

···

On 3/30/10 10:37 PM, Kit wrote:

I think I would like to try and tackle conquering the time-interval
thing, using whatever tricks I could muster, before giving up on
Python (and wxPython) so easily. And again my question - HOW do they
resolve this with Python-based video and audio players, which face the
same problem? There's a missing link here, and I need to figure why
this is an impossible task for me, but not for them.

--
Robin Dunn
Software Craftsman

Hello, Werner:

Thank you for the link and the tip about real-time Python discussions.
They look VERY INTERESTING! It looks like others are finding the need
to resolve the same problem that I am, for about the same reasons.

I will check into these topics today.

Many Thanks,

Kit

···

On Mar 31, 6:21 am, werner <wbru...@free.fr> wrote:

On 31/03/2010 07:37, Kit wrote:

Haven't followed all of this thread, if this was already suggested just
ignore it.

There seems to be quite a bit on the net and multiple persons have tried
"real time" with Python or at least looked at it in the past, here a
link which looked intesting, but there are others when you google "real
time python".

http://www.its.caltech.edu/~astraw/publications/realtime_computing.html

Werner

I should have clarified myself earlier where I was making the
assumptions on the multimedia timing; my apologies.
...
The book included a disk with Python 2.3.5, along with versions of the
PyGame and Livewires packages. It APPEARED to me at the
time(admittedly a beginning novice) that these packages were providing
methods to run video, bitmap slides, and decode audio files by its own
internal methods, all linked somehow through Python examples in the
book. The methods had video frame rate adjustment attributes, timers
for actions and delays adjustable in milliseconds, etc.

Perhaps I just didn't understand what was really happening there, and
have it completely wrong. But the book SEEMED to be indicating it was
all Python-based operations doing this, not any pre-existent resident
Windows multimedia utilities.

Nope. That's really impossible. There's just too much overhead
involved in video for that to work. There are Python wrappers around
SDL, Video for Windows, and DirectShow, all of which are C or C++
frameworks that do all of the buffer management and timing.

I then learned about the fantastic wxPython GUI toolkit, and picked up
the book "wxPython in Action". Besides showing all the terrific GUI
modules I wanted to learn and use, the book's last chapter described
the wx.Timer module, which I saw would serve my purpose exactly. I at
that point just assumed that accurate timing was no big deal.

It depends on what you mean. The word "timing" is quite overloaded in
our industry. You can do INTERVAL timing (meaning "how long did this
action take?") in Windows and Linux with a relatively high degree of
precision, using several different methods. Heck, the Pentium cycle
counter is universally available, and provides interval timing down to
the sub-nanosecond level (well, sort of).

However, what you are after is timer EVENTS (meaning, "wake me after X
milliseconds"). That is each in the coarse sense, but much, much
trickier in the fine sense. Neither Windows nor Linux are real-time
operating systems. They make no guarantees about interrupt latencies
and interrupt recall periods. The WM_TIMER message, which is used by
wx.Timer, is extremely coarse. It is only sent during scheduler
intervals, which happen every 10ms, 16ms, 20ms, or 25ms, depending on
which Windows version you have. (55ms was used in DOS and Win9X.)
WM_TIMER merely guarantees that you will not be called more often than
the interval you specify. Further, WM_TIMER is part of the message
loop, and will not be seen by your application if the GUI is busy
doing something else.

There are ways to get finer control. Windows provides "multimedia
timers" that do callbacks instead of messages, and those allow you to
control the timer interrupt frequency. So, you can ask for 1
millisecond accuracy instead of 16 millisecond accuracy. That (not
surprisingly) increases the system overhead substantially. And,
again, there are no guarantees. All it takes is one stupid audio
driver to sit in a high-priority kernel thread for 30 milliseconds,
and all of your numbers go to pot.

···

On Mar 31, 9:13 am, Kit <kit.2...@yahoo.com> wrote:
--
Tim Roberts, timr@probo.com
Providenza & Boekelheide, Inc.

...

It looks like Andrew Straws SciPy paper (linked by Werner) may be just what you are looking for, however, it looks like the link is broken. I'll bet you can find a copy somewhere, or send a note to Andrew and ask him for it.

The link further down on the page might be interesting too:

http://dickinson.caltech.edu/Research/MultiTrack
http://code.astraw.com/projects/motmot/packages.html

On the last one looking at the Packages reference there is even some wxPython stuff:
http://code.astraw.com/projects/motmot/packages.html

Werner

···

On 31/03/2010 18:38, Christopher Barker wrote:

To Che, Chris, Robin & All:

I am sorry if my questions have left some holes in your thinking,
because I've been a little light on the details of my application. I
was trying to keep the subject restricted to Python, and not drift off
to the unusual processes I apply. To be honest, the application is a
bit abstract, and is a highly unusual one.

However, I see now leaving out details is only creating confusion, and
what appears to be conflicting information to you. I'll try to clear
up things a bit in the paragraphs below.

Here is a very brief overview of the process I am using, and what is
required of it (be aware that this IS an existing process, and not a
theory being tried):

The focus of the system is NOT on the GUIs, but on the Updating
routine that passes a number of parameters from the PC to a set of
external devices.

Each device receives its own unique set of parameters (typically about
40 bytes in size), then replies back with about 20 bytes back to the
PC in response. Device quantities can be as little as 1 device to over
32, depending on the real-world application.

These external devices are independent, dedicated embedded-processor
mechanisms, not PCs. They essentially get operational instructions on
how to behave moment-to-moment by receiving their parameters many
times a second. Exactly how many times a second is dictated by the
compexity of the operation these mechanisms are required to perform.
Typical rates are from 12 to 60 updates per second, with 30 updates
per second being nominal.

These parameters are sometimes created in a large file in advance, and
"played" real-time during an operation. At other times, certain non-
real-time-critical parameters are loaded by a User by entering data in
a GUI on the PC. A User can also watch the run-time actions of an
operation on a GUI, based on the parameters returned from the external
device(s) during operation. Under this relationship, the GUI screen
updates as it can, when it can. The reading and writing to the
external devices take precedent in time; as many have already pointed
out, GUIs can be delayed to allow for this, without any real noticable
consequence.

The external devices however, *MUST* be updated with new data by a
reliably repeatable Update rate, or the overall process simply will
not work. This means that if I set up an Update rate for 30 ms., I
must send and receive all the data to and from the devices before that
period runs out, then return back to use the balance of the period to
do the less-critical tasks like updating the screen display. If all is
designed right, the tasks finish comfortably in that amount of time,
continue to cycle through the GUI background stuff, until the real-
time clock signals the next 30 ms. has lapsed. Then the procedure is
repeated ad infinitum.

The easiest way to understand this is to think of the problem applied
to an old DOS C or Pascal program (which was often how I used to do
this in the old days): The GUI routines run continually, until a real-
time interrupt calls the Device Update function, which must be slim
enough to do its job and return before the time runs out. Otherwise,
an Interrupt Overrun would occur, which would be catastrophic. For
this reason, the Update routines are made up of very tight and
efficent code.

There are two reasons the Update rate must be reliably accurate. The
first is the obvious Overrun problem mentioned just above.

The second reason is complicated, and I will keep it as brief and
simplified as possible here, without being cryptic:

The external devices used here are highly-complex motion-control
devices. The parameters they are given per Update, are fed to the
device as elements to a complex math algorithm which is acted upon by
the device dynamically. The result of this algorithm governs the
device's operation from split-second to split-second. It should be
noted that along with the parameters that are written to the device,
the UPDATE RATE of those parameters is also one of the elements
applied to the operational math. This is used for time-slice domain
positional calculations, and is often deliberately varied to
manipulate the math just as varying any other parameter would be done.
For this reason, the Update frequency must be repeatable and reliable.

As a side note, I know I mentioned in an earlier posting that under
some conditions the Update frequency is not so critical. This is
because I have two flavors of this technology. One type is much
simpler in functions, and does not use the Update rate as part of its
processing math. It becomes simply a matter of getting the parameters
to the device in a responsive manner, and not overrun the Update
routine. This device is extremely limited in its applications, though.

The "Rolls-Royce" version of the design DOES use the Update rate in
its equations however, and can therefore be applied to almost any
application imagined. Obviously this is my design target.

The way this process was done in the old days was by a lot of
dedicated hardware and software in PCs, if I didn't just go to
embedded systems of my own design. I later used high-speed serial
processes with extremely limited parameter structures, severely
limiting device operations.

Now I am using a dedicated LAN to these devices by way of the PC's
Ethernet Port. So to get my systems swung over to a newer efficient
design, I need a good GUI toolkit, and a dependable real-time clock to
call the Update method reliably.

I have the Ethernet Update routine working, and I think wxPython can
easily provide the GUIs; that leaves one problem to go.

I'm sorry this post is so long, but I understand your need to get a
clear picture of what I am trying to do. I hope this at least helps
you understand that my requirements for the Update rate are real and
necessary.

Regards,

Kit

···

On Mar 31, 8:31 am, C M <cmpyt...@gmail.com> wrote:

I can't really understand what you're trying to do, but are you really
sure you need the precision you state you need? You said:

"I wish to run a GUI program in the Main Python application, which will
collect User Commands, and display Operational Status on a laptop
screen. Meanwhile, in the background, on a regular timed basis, the
program flow will pass to a routine that will output the User Commands
and input new Operational Status by means of a very short packet
exchange on the Ethernet port. The routine will then return to the
Main GUI application, passing on the new parameters to the GUI,
updating it.

This background process will operate continually, at a regular fixed-
timed interval. I am currently targeting a time-slice update rate of
about 33.33 ms. (30 times a second), but this value isn't etched in
stone. I could probably go to 66.66 ms. (or 15 times a second), if
absolutely need be."

Do you really want to sample user's commands 30x every second? Users
can't possibly attain anything like that rate, so why should you? And
then update the GUI 30x every second? That is physiological trackable
I guess (video is 30 fps), but is it really necessary for what you are
doing? Or am I totally misunderstanding you?

Che

Thanks for the details. Here is my initial impression on how to approach this:

* First, decouple the GUI from the task that is talking to the devices. It may be okay to simply run that in a separate thread, but my guess is that it will do much better as a separate process, and perhaps either partially or fully in C or C++. Although doing it initially all in Python would probably be a good idea, then you can later optimize into C/C++ only those portions that really need it once you've got the whole thing working and you're able to do some profiling.

* Create or use an existing IPC mechanism that will allow the GUI to talk to the background task. There are lots available, although I've done my own sometimes when I couldn't find a good fit for a specific need.

* Write the GUI in such a way that it is not dependent on the timing of everything that is going on in the background task, but instead allow it to periodically poll the background task to get a current snapshot of all of the device states, responses, and whatever else you need to have to be able to display the UI. In other words, don't expect the GUI to be updated for absolutely every update from the devices but rather allow it to check in whenever it is able to and get the current state at that moment. Another way to state this is that the GUI will periodically pull info from the background task, instead of the background task pushing everything to the GUI. It likely doesn't matter much to the user if they are able to see values on screen change every 25ms or every 250ms because depending on how much data is being displayed they probably can't process it that fast anyway. (You may even want to make it possible for the user to adjust the polling frequency.) Any commands that the GUI should be able to send to the background task can be handled by the same IPC mechanism, (in fact getting the snapshot would probably be just another one of the set of commands that are supported.)

By designing it this way the timing critical parts are independent from the GUI and will not be impacted by any of the issues associated with the timers being delayed by other pending events, running in conjunction with a GUI event loop, etc. and it's easy to add more features to the UI later without needing to worry about how it might impact the background task.

···

On 3/31/10 12:58 PM, Kit wrote:

To Che, Chris, Robin& All:

I am sorry if my questions have left some holes in your thinking,
because I've been a little light on the details of my application. I
was trying to keep the subject restricted to Python, and not drift off
to the unusual processes I apply. To be honest, the application is a
bit abstract, and is a highly unusual one.

--
Robin Dunn
Software Craftsman

Robin Dunn wrote:

* Create or use an existing IPC mechanism that will allow the GUI to talk to the background task. There are lots available, although I've done my own sometimes when I couldn't find a good fit for a specific need.

This one looks very promising for this kind of thing:

The ipython folks are working on python bindings for it. See this thread:

http://thread.gmane.org/gmane.comp.python.ipython.devel/4732/focus=4739

By the way, iPython is a fabulous development tool you should know about anyway.

-Chris

Robin & Chris,

Thank you for being stubborn enough to follow me through this
thing. :slight_smile: Since I posted the original thread, I have followed
everyone's advice and researched this problem on the net, and have
found to my dismay, I truly am asking for a bit of atom-splitting
here.

I confess that at this moment I don't know anything about IPCs. But
your description of program flow and priorities indicate that you do
now understand my application, and your recommendations of program
isolation and disconnect seem right, and in line with my thinking.

I shall look into the links Chris shows right away, and pursue the
problem along these lines. As you say, I think the first order of
things is to build up a best-effort test model, and see where that
empirically takes me. I can then refine things accordingly.

Thank you very much for your patience and advice.

Sincerely,

Kit

···

On Mar 31, 1:39 pm, Robin Dunn <ro...@alldunn.com> wrote:

On 3/31/10 12:58 PM, Kit wrote:

Thanks for the details. Here is my initial impression on how to
approach this:

* First, decouple the GUI from the task that is talking to the devices.
  It may be okay to simply run that in a separate thread, but my guess
is that it will do much better as a separate process, and perhaps either
partially or fully in C or C++. Although doing it initially all in
Python would probably be a good idea, then you can later optimize into
C/C++ only those portions that really need it once you've got the whole
thing working and you're able to do some profiling.

* Create or use an existing IPC mechanism that will allow the GUI to
talk to the background task. There are lots available, although I've
done my own sometimes when I couldn't find a good fit for a specific need.

* Write the GUI in such a way that it is not dependent on the timing of
everything that is going on in the background task, but instead allow it
to periodically poll the background task to get a current snapshot of
all of the device states, responses, and whatever else you need to have
to be able to display the UI. In other words, don't expect the GUI to
be updated for absolutely every update from the devices but rather allow
it to check in whenever it is able to and get the current state at that
moment. Another way to state this is that the GUI will periodically
pull info from the background task, instead of the background task
pushing everything to the GUI. It likely doesn't matter much to the
user if they are able to see values on screen change every 25ms or every
250ms because depending on how much data is being displayed they
probably can't process it that fast anyway. (You may even want to make
it possible for the user to adjust the polling frequency.) Any commands
that the GUI should be able to send to the background task can be
handled by the same IPC mechanism, (in fact getting the snapshot would
probably be just another one of the set of commands that are supported.)

By designing it this way the timing critical parts are independent from
the GUI and will not be impacted by any of the issues associated with
the timers being delayed by other pending events, running in conjunction
with a GUI event loop, etc. and it's easy to add more features to the UI
later without needing to worry about how it might impact the background
task.

--
Robin Dunn
Software Craftsmanhttp://wxPython.org

Nice. I may have to play with that just for the fun of it. I've contributed to some iMatix OSS projects in the past, they do real good stuff.

···

On 3/31/10 5:47 PM, Chris Barker wrote:

Robin Dunn wrote:

* Create or use an existing IPC mechanism that will allow the GUI to
talk to the background task. There are lots available, although I've
done my own sometimes when I couldn't find a good fit for a specific
need.

This one looks very promising for this kind of thing:

http://www.zeromq.org/

--
Robin Dunn
Software Craftsman