Test results in CI builds

Hi,

I’m not sure if most people in this group subscribe to Phoenix GitHub notifications. Maybe they could be automatically forwarded to this Google Group with a Mail hook: https://help.github.com/articles/managing-notifications-for-pushes-to-a-repository/ ?

But for those who don’t get them, I just submitted a (Work In Progress) pull request which I’d be interested in some feedback on: https://github.com/wxWidgets/Phoenix/pull/248

Basically, I would like to see Phoenix tests running in CI (continuous integration) services like AppVeyor and Travis CI. I couldn’t get the tests from the official branch (@ 4fc5f9ed1047c243a7fe0c2a6489064c7dc78a4b) to run smoothly, which I think is due to race conditions in the tearDown of one test sometimes preventing subsequent tests from running.

After making a few changes in my fork of Phoenix (https://github.com/wettenhj/Phoenix/tree/appveyor), I got the tests running on the AppVeyor (Windows) CI service (just using Python 2.7.13 for now), as shown here: https://ci.appveyor.com/project/wettenhj/phoenix/build/1.0.7

However some of the changes I made to get the tests running probably went a bit too far, e.g. preventing certain timers from running if there is no MainLoop. I still believe that it might be necessary to avoid some timers when running tests, but testing for wx.GetMainLoop() is not the right way to do it, because a timer could be legitimately started in an application’s OnInit method before the MainLoop has started. A suggestion for an alternative approach is here: https://github.com/wxWidgets/Phoenix/pull/248#issuecomment-287644362

I think the priority for improving tests should be to avoid exceptions which can cause the interpreter to crash or hang partway through the tests. (Feel free to correct me if that has never been an issue for other people running “python build.py test”.)

But it’s also nice to avoid ugly tracebacks when enabling stderr (–extra_pytest="–capture=no" or --extra_pytest="-s"). I’ve had multiple attempts at fixing one such issue (tearDown of the UltimateListCtrl tests somehow terminating the App too early, triggering wx.PyNoAppErrors or “No wx.App created yet”) which I still haven’t fully resolved, as you can see in the following lines of the test output: https://ci.appveyor.com/project/wettenhj/phoenix/build/1.0.7#L5691

wx\lib\agw\ultimatelistctrl.py:10876: in __init__

wx.CallAfter(self.Layout)

def CallAfter(callableObj, *args, **kw):

assert callable(callableObj), “callableObj is not callable”

app = wx.GetApp()

> assert app is not None, ‘No wx.App created yet’

E AssertionError: No wx.App created yet

I’ve enabled coverage (some output here: https://ci.appveyor.com/project/wettenhj/phoenix/build/1.0.7#L5346), but keep in mind that this is only one platform. Codecov.io is good at merging test coverage from multiple platforms.

Cheers,

James

James,

No time to review any of this :frowning:

But great work! Having a proper CI would be really nice.

-CHB

···

On Mar 20, 2017, at 12:03 AM, James Wettenhall james.wettenhall@monash.edu wrote:

Hi,

I’m not sure if most people in this group subscribe to Phoenix GitHub notifications. Maybe they could be automatically forwarded to this Google Group with a Mail hook: https://help.github.com/articles/managing-notifications-for-pushes-to-a-repository/ ?

But for those who don’t get them, I just submitted a (Work In Progress) pull request which I’d be interested in some feedback on: https://github.com/wxWidgets/Phoenix/pull/248

Basically, I would like to see Phoenix tests running in CI (continuous integration) services like AppVeyor and Travis CI. I couldn’t get the tests from the official branch (@ 4fc5f9ed1047c243a7fe0c2a6489064c7dc78a4b) to run smoothly, which I think is due to race conditions in the tearDown of one test sometimes preventing subsequent tests from running.

After making a few changes in my fork of Phoenix (https://github.com/wettenhj/Phoenix/tree/appveyor), I got the tests running on the AppVeyor (Windows) CI service (just using Python 2.7.13 for now), as shown here: https://ci.appveyor.com/project/wettenhj/phoenix/build/1.0.7

However some of the changes I made to get the tests running probably went a bit too far, e.g. preventing certain timers from running if there is no MainLoop. I still believe that it might be necessary to avoid some timers when running tests, but testing for wx.GetMainLoop() is not the right way to do it, because a timer could be legitimately started in an application’s OnInit method before the MainLoop has started. A suggestion for an alternative approach is here: https://github.com/wxWidgets/Phoenix/pull/248#issuecomment-287644362

I think the priority for improving tests should be to avoid exceptions which can cause the interpreter to crash or hang partway through the tests. (Feel free to correct me if that has never been an issue for other people running “python build.py test”.)

But it’s also nice to avoid ugly tracebacks when enabling stderr (–extra_pytest=“–capture=no” or --extra_pytest=“-s”). I’ve had multiple attempts at fixing one such issue (tearDown of the UltimateListCtrl tests somehow terminating the App too early, triggering wx.PyNoAppErrors or “No wx.App created yet”) which I still haven’t fully resolved, as you can see in the following lines of the test output: https://ci.appveyor.com/project/wettenhj/phoenix/build/1.0.7#L5691

wx\lib\agw\ultimatelistctrl.py:10876: in __init__

wx.CallAfter(self.Layout)

def CallAfter(callableObj, *args, **kw):

assert callable(callableObj), “callableObj is not callable”

app = wx.GetApp()

> assert app is not None, ‘No wx.App created yet’

E AssertionError: No wx.App created yet

I’ve enabled coverage (some output here: https://ci.appveyor.com/project/wettenhj/phoenix/build/1.0.7#L5346), but keep in mind that this is only one platform. Codecov.io is good at merging test coverage from multiple platforms.

Cheers,

James

You received this message because you are subscribed to the Google Groups “wxPython-dev” group.

To unsubscribe from this group and stop receiving emails from it, send an email to wxPython-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Hi Chris and others,

···

Yes I completely understand the lack of time to do open-source work not specifically required by your employer!

I’ll try to come up with a new more palatable pull request, with

  • a minimal number of commits (easier to review)

  • just focusing on reducing risks of tests hanging partway through (not fixing deprecation warnings etc.)

  • more CI services (multiple OS’s) connected to my fork of Phoenix (to give confidence that I haven’t broken anything)

Cheers,

James