Dream continuous-testing setup for a wxPython project

wxPython Users
1

Hello folks!

I’ve been working on my wxPython project for about two years. It’s high time for me to do some automated GUI testing.

I already have a nose test suite for the non-GUI parts of my program. It’s pretty comprehensive and it works great. But I must test the GUI as well.

Let me try to describe my ideal continuous-testing setup:

I want real, mouse-keyboard-screen testing for my GUI, not just manually posting events. And the tests should work cross-platform.

Also I want to start using Hudson, and have it run my existing test suite plus the GUI tests. Every time I push a commit of my project to GitHub, I want Hudson to pull it and run all the tests, including the mouse-keyboard-screen GUI tests, on all three platforms, without any human intervention. (Probably using a bunch of slave machines, physical or virtual, that run the different OSes.) Then I want it to post the results on one single page where I can review them.

I may not get all of the above, but I want to get my concept of the ideal system right before I see what actually exists and try to get a system that’s as close as possible to my dream system.

My questions:

  1. Do you have any other additions/suggestions to this dream continuous-testing setup that might make the developer’s life easier?

  2. Do you know if what I want is possible? If so, how?

  3. Do you have your own continuous-testing setup for your wxPython project? If so, what is it like and what tools are you using?

Best Wishes,

Ram.

2

Hello folks!

I've been working on my wxPython project for about two years. It's high
time for me to do some automated GUI testing.

I already have a nose test suite for the non-GUI parts of my program.
It's pretty comprehensive and it works great. But I must test the GUI as
well.

Let me try to describe my ideal continuous-testing setup:

I want real, mouse-keyboard-screen testing for my GUI, not just manually
posting events. And the tests should work cross-platform.

Also I want to start using Hudson, and have it run my existing test
suite plus the GUI tests. Every time I push a commit of my project to
GitHub, I want Hudson to pull it and run all the tests, including the
mouse-keyboard-screen GUI tests, on all three platforms, without any
human intervention. (Probably using a bunch of slave machines, physical
or virtual, that run the different OSes.) Then I want it to post the
results on one single page where I can review them.

I may not get all of the above, but I want to get my concept of the
ideal system right before I see what actually exists and try to get a
system that's as close as possible to my dream system.

My questions:

1. Do you have any other additions/suggestions to this dream
continuous-testing setup that might make the developer's life easier?

In 2.9.2 there will be the ability to generate real mouse and key events for the platform, such that they come to wx the same as the real events, and will then go on to the native widget if wx doesn't consume the event. It's still not complete on all platforms, but it works well where it is.

2. Do you know if what I want is possible? If so, how?

There are some 3rd party tools for recording and scripted playback of mouse and key events, but last I checked none of them are cross platform.

3. Do you have your own continuous-testing setup for your wxPython
project? If so, what is it like and what tools are you using?

Well, first you get a room full of computers, and then a couple dozen monkeys to pound on the keyboards, and then...

···

On 1/20/11 9:15 AM, cool-RR wrote:

--
Robin Dunn
Software Craftsman

3

You clean up feces all day. Sounds like business as usual!

···

On Thu, Jan 20, 2011 at 9:07 PM, Robin Dunn <robin@alldunn.com> wrote:

Well, first you get a room full of computers, and then a couple dozen
monkeys to pound on the keyboards, and then...

4

Hello folks!

I’ve been working on my wxPython project for about two years. It’s high

time for me to do some automated GUI testing.

I already have a nose test suite for the non-GUI parts of my program.

It’s pretty comprehensive and it works great. But I must test the GUI as

well.

Let me try to describe my ideal continuous-testing setup:

I want real, mouse-keyboard-screen testing for my GUI, not just manually

posting events. And the tests should work cross-platform.

Also I want to start using Hudson, and have it run my existing test

suite plus the GUI tests. Every time I push a commit of my project to

GitHub, I want Hudson to pull it and run all the tests, including the

mouse-keyboard-screen GUI tests, on all three platforms, without any

human intervention. (Probably using a bunch of slave machines, physical

or virtual, that run the different OSes.) Then I want it to post the

results on one single page where I can review them.

I may not get all of the above, but I want to get my concept of the

ideal system right before I see what actually exists and try to get a

system that’s as close as possible to my dream system.

My questions:

  1. Do you have any other additions/suggestions to this dream

continuous-testing setup that might make the developer’s life easier?

In 2.9.2 there will be the ability to generate real mouse and key events for the platform, such that they come to wx the same as the real events, and will then go on to the native widget if wx doesn’t consume the event. It’s still not complete on all platforms, but it works well where it is.

I guess that’s an improvement, but I’d prefer something that actually controls the mouse, keyboard and screen, since that’s closer to how a human user interacts with the program.

  1. Do you know if what I want is possible? If so, how?

There are some 3rd party tools for recording and scripted playback of mouse and key events, but last I checked none of them are cross platform.

Sikuli supports Windows, Mac and Linux. I tried Sikuli a few times and it had a really annoying interface, but they’ve made a few upgrades since, so maybe I could get it to work reliably for my dream CI setup.

  1. Do you have your own continuous-testing setup for your wxPython

project? If so, what is it like and what tools are you using?

Well, first you get a room full of computers, and then a couple dozen monkeys to pound on the keyboards, and then…

–

Robin Dunn

Ah, then I think we can do better than monkeys.

It’ll be nice if we could figure out how to build a good CI stack for cross-platform wxPython testing. I’m thinking EC2/Hudson/Nose/Sikuli, but I’ll be happy to hear opinions from other people in the group.

(One caveat is that I think EC2 can’t create Mac instances.)

Ram.

···

On Thu, Jan 20, 2011 at 9:07 PM, Robin Dunn robin@alldunn.com wrote:

On 1/20/11 9:15 AM, cool-RR wrote:

5

Hi Ram,

Hello folks!

I’ve been working on my wxPython project for about two years. It’s high

time for me to do some automated GUI testing.

I already have a nose test suite for the non-GUI parts of my program.

It’s pretty comprehensive and it works great. But I must test the GUI as

well.

Let me try to describe my ideal continuous-testing setup:

I want real, mouse-keyboard-screen testing for my GUI, not just manually

posting events. And the tests should work cross-platform.

Also I want to start using Hudson, and have it run my existing test

suite plus the GUI tests. Every time I push a commit of my project to

GitHub, I want Hudson to pull it and run all the tests, including the

mouse-keyboard-screen GUI tests, on all three platforms, without any

human intervention. (Probably using a bunch of slave machines, physical

or virtual, that run the different OSes.) Then I want it to post the

results on one single page where I can review them.

I may not get all of the above, but I want to get my concept of the

ideal system right before I see what actually exists and try to get a

system that’s as close as possible to my dream system.

My questions:

  1. Do you have any other additions/suggestions to this dream

continuous-testing setup that might make the developer’s life easier?

In 2.9.2 there will be the ability to generate real mouse and key events for the platform, such that they come to wx the same as the real events, and will then go on to the native widget if wx doesn’t consume the event. It’s still not complete on all platforms, but it works well where it is.

I guess that’s an improvement, but I’d prefer something that actually controls the mouse, keyboard and screen, since that’s closer to how a human user interacts with the program.

What he’s talking about is wxUIActionSimulator, and that is exactly what it does. It is a wrapper around native APIs that allow for programmatic control of the keyboard and mouse on the platform. Yes, you’re scripting rather than doing true record and playback, but you could also write a tool that records events and turns them into a series of wxUIActionSimulator calls. Still, I think it’s better to do things like programmatically find the position of a control and then move the mouse to it than to use harcoded coordinates like (245, 345) which will need to be redone each time your layout changes.

Regards,

Kevin

···

On Jan 20, 2011, at 11:40 PM, cool-RR wrote:

On Thu, Jan 20, 2011 at 9:07 PM, Robin Dunn robin@alldunn.com wrote:

On 1/20/11 9:15 AM, cool-RR wrote:

  1. Do you know if what I want is possible? If so, how?

There are some 3rd party tools for recording and scripted playback of mouse and key events, but last I checked none of them are cross platform.

Sikuli supports Windows, Mac and Linux. I tried Sikuli a few times and it had a really annoying interface, but they’ve made a few upgrades since, so maybe I could get it to work reliably for my dream CI setup.

  1. Do you have your own continuous-testing setup for your wxPython

project? If so, what is it like and what tools are you using?

Well, first you get a room full of computers, and then a couple dozen monkeys to pound on the keyboards, and then…

–

Robin Dunn

Ah, then I think we can do better than monkeys.

It’ll be nice if we could figure out how to build a good CI stack for cross-platform wxPython testing. I’m thinking EC2/Hudson/Nose/Sikuli, but I’ll be happy to hear opinions from other people in the group.

(One caveat is that I think EC2 can’t create Mac instances.)

Ram.

–

To unsubscribe, send email to wxPython-users+unsubscribe@googlegroups.com

or visit http://groups.google.com/group/wxPython-users?hl=en

6

Hi Ram,

    In 2.9.2 there will be the ability to generate real mouse and key
    events for the platform, such that they come to wx the same as the
    real events, and will then go on to the native widget if wx
    doesn't consume the event. It's still not complete on all
    platforms, but it works well where it is.

I guess that's an improvement, but I'd prefer something that actually
controls the mouse, keyboard and screen, since that's closer to how a
human user interacts with the program.

What he's talking about is wxUIActionSimulator, and that is exactly what
it does. It is a wrapper around native APIs that allow for programmatic
control of the keyboard and mouse on the platform. Yes, you're scripting
rather than doing true record and playback, but you could also write a
tool that records events and turns them into a series of
wxUIActionSimulator calls.

Which is basically what the sample is doing, although it is just demoing key events currently. To make it more fun I also record the delay between events so if you pause while recording then there will be a pause during playback too, although I cut the delay times in half during playback because I'm impatient.

http://trac.wxwidgets.org/browser/wxPython/trunk/demo/UIActionSimulator.py

Still, I think it's better to do things like
programmatically find the position of a control and then move the mouse
to it than to use harcoded coordinates like (245, 345) which will need
to be redone each time your layout changes.

If you record the positions relative to the widget that receives the event, and then recalc the position using the same widget during event playback, then I think that most of the position related issues will go away.

···

On 1/21/11 10:21 AM, Kevin Ollivier wrote:

On Jan 20, 2011, at 11:40 PM, cool-RR wrote:

On Thu, Jan 20, 2011 at 9:07 PM, Robin Dunn <robin@alldunn.com >> <mailto:robin@alldunn.com>> wrote:

--
Robin Dunn
Software Craftsman

7

Ram, if you wind up getting a nice soup-to-notes approach to testing
wxPython apps (both the non-GUI and GUI parts), I for one would
welcome an entry on the wiki on how you do it, if you are up for
writing it.

Automated testing (as opposed to manually testing) is something I have
entirely avoided because I have just been trying to learn actual
creation and making progress with applications, but if I can get
testing into the mix it would be a great improvement, and a good wiki
entry could help lower the activation energy for trying it.

Thanks,
Che

···

On Thu, Jan 20, 2011 at 12:15 PM, cool-RR <ram.rachum@gmail.com> wrote:

Hello folks!
I've been working on my wxPython project for about two years. It's high time
for me to do some automated GUI testing.
I already have a nose test suite for the non-GUI parts of my program. It's
pretty comprehensive and it works great. But I must test the GUI as well.

8

Ram,

Hello folks!

I've been working on my wxPython project for about two years. It's high time for me to do some automated GUI testing.

I already have a nose test suite for the non-GUI parts of my program. It's pretty comprehensive and it works great. But I must test the GUI as well.

Let me try to describe my ideal continuous-testing setup:

I want real, mouse-keyboard-screen testing for my GUI, not just manually posting events. And the tests should work cross-platform.

Also I want to start using Hudson, and have it run my existing test suite plus the GUI tests. Every time I push a commit of my project to GitHub, I want Hudson to pull it and run all the tests, including the mouse-keyboard-screen GUI tests, on all three platforms, without any human intervention. (Probably using a bunch of slave machines, physical or virtual, that run the different OSes.) Then I want it to post the results on one single page where I can review them.

I may not get all of the above, but I want to get my concept of the ideal system right before I see what actually exists and try to get a system that's as close as possible to my dream system.

My questions:

1. Do you have any other additions/suggestions to this dream continuous-testing setup that might make the developer's life easier?

2. Do you know if what I want is possible? If so, how?

3. Do you have your own continuous-testing setup for your wxPython project? If so, what is it like and what tools are you using?

Best Wishes,
Ram.

Are you aware of:

Version 3.4.1 according to the announcement on python.announce also supports wxPython.

I don't do any auto testing but put this announcement aside as I really should do so - maybe one of these days.

Werner

9

Okay, update.

I’ve been working hard on my CI setup. The non-GUI testing works now, using Hudson and EC2. Now I want to do GUI testing.

I gave Sikuli a test-drive. It’s still not mature enough, in my opinion. I also looked at a few other solutions.

I’m starting to think that wxUIActionSimulator is the way to go. Two questions about it:

  1. Will I be able to use wxUIActionSimulator in a different process than the wxPython program I’m testing? I want my program to be 2.8-based, and to do the GUI testing on a separate process that uses 2.9. Is that possible?
  2. I understand that wxUIActionSimulator will be released with wxPython 2.9.2. Can you give me a crude estimate on when that will be released?

Ram.

···

On Sun, Jan 23, 2011 at 4:55 AM, werner wbruhin@free.fr wrote:

Ram,

Hello folks!

I’ve been working on my wxPython project for about two years. It’s high time for me to do some automated GUI testing.

I already have a nose test suite for the non-GUI parts of my program. It’s pretty comprehensive and it works great. But I must test the GUI as well.

Let me try to describe my ideal continuous-testing setup:

I want real, mouse-keyboard-screen testing for my GUI, not just manually posting events. And the tests should work cross-platform.

Also I want to start using Hudson, and have it run my existing test suite plus the GUI tests. Every time I push a commit of my project to GitHub, I want Hudson to pull it and run all the tests, including the mouse-keyboard-screen GUI tests, on all three platforms, without any human intervention. (Probably using a bunch of slave machines, physical or virtual, that run the different OSes.) Then I want it to post the results on one single page where I can review them.

I may not get all of the above, but I want to get my concept of the ideal system right before I see what actually exists and try to get a system that’s as close as possible to my dream system.

My questions:

  1. Do you have any other additions/suggestions to this dream continuous-testing setup that might make the developer’s life easier?

  2. Do you know if what I want is possible? If so, how?

  3. Do you have your own continuous-testing setup for your wxPython project? If so, what is it like and what tools are you using?

Best Wishes,

Ram.

Are you aware of:

http://sourceforge.net/projects/pyusecase/

Version 3.4.1 according to the announcement on python.announce also supports wxPython.

I don’t do any auto testing but put this announcement aside as I really should do so - maybe one of these days.

Werner

–

To unsubscribe, send email to wxPython-users+unsubscribe@googlegroups.com

or visit http://groups.google.com/group/wxPython-users?hl=en

10

I've been working hard on my CI setup. The non-GUI testing works now,
using Hudson and EC2. Now I want to do GUI testing.

I gave Sikuli a test-drive. It's still not mature enough, in my opinion.
I also looked at a few other solutions.

I'm starting to think that `wxUIActionSimulator` is the way to go. Two
questions about it:

   1. Will I be able to use `wxUIActionSimulator` in a different process
      than the wxPython program I'm testing? I want my program to be
      2.8-based, and to do the GUI testing on a separate process that
      uses 2.9. Is that possible?

No, it only sends the events to itself.

   2. I understand that `wxUIActionSimulator` will be released with
      wxPython 2.9.2. Can you give me a crude estimate on when that will
      be released?

A release hasn't been discussed on wx-dev yet, but the last time I said that the discussion started even before I clicked on the send button. So it could happen any time or it could still be a long ways off. I do plan on starting to do some preview builds soon (a couple weeks or so) so some more testing can be done.

···

On 2/4/11 4:47 AM, cool-RR wrote:

--
Robin Dunn
Software Craftsman