Removing the wxPython call overhead

wxPython Users
1

In snooping around wx, I found that methods were bound like so:

    def GetPrintMode(self, *_args, **_kwargs):
        val = apply(wxc.wxPyApp_GetPrintMode,(self,) + _args, _kwargs)
        return val

This looks extremely inefficient because:
   * apply has to be looked up in the __builtins__ dict
   * a new tuple has to be allocated as a result of the tuple addition
   * wxc has to be looked up in the globals dictionary
   * the func name has to be looked up in the module's dictionary

I tried some different implementations and got the following timings for
invocations of the method:

$ python overhead.py 100000
reps: 100000
2.2.2 (#1, Nov 26 2002, 16:14:37)
[GCC 3.2 (Mandrake Linux 9.0 3.2-1mdk)]
14.2 seconds for FooOld
10.6 seconds for FooNew1
9.8 seconds for FooNew2
3.2 seconds for FooNew3

FooOld is the current approach. FooNew3 is simply this:
    class FooNew3:
        GetPrintMode = wxc.wxPyApp_GetPrintMode

This approach eliminates all those extra steps I outlined above.

The code is here:
http://chuckesterbrook.com/files/python/wx/method-overhead/

I realize that as the function implementations increase in complexity,
the call overhead becomes less of an issue. On the other hand, the C++
native code is so much faster than Python bytecode, that this overhead
may still be significant for a lot of methods.

I think it's worth it to cut out the overhead and make things snappier.
If this can be applied everywhere, then all apps and wxPython users
will benefit!

Robin/Patrick/whoever: Is this an improvement that could be applied
universally? (I can't think why not, but maybe I'm missing
something...) And can it made in the 2.4.x series?

···

--
Chuck
http://ChuckEsterbrook.com

2

Chuck Esterbrook wrote:

In snooping around wx, I found that methods were bound like so:

    def GetPrintMode(self, *_args, **_kwargs):
        val = apply(wxc.wxPyApp_GetPrintMode,(self,) + _args, _kwargs)
        return val

[...]

FooOld is the current approach. FooNew3 is simply this:
    class FooNew3:
        GetPrintMode = wxc.wxPyApp_GetPrintMode

[...]

I think it's worth it to cut out the overhead and make things snappier. If this can be applied everywhere, then all apps and wxPython users will benefit!

Robin/Patrick/whoever: Is this an improvement that could be applied universally? (I can't think why not, but maybe I'm missing something...) And can it made in the 2.4.x series?

The problem is that not all the methods look like your sample above (many have to postprocess after the native call) and so they can't universally be switched to the simple assignment.

OTOH, I already have it on my ToDo to change my custom wxSWIG so it doesn't use the apply() but "foo(self, *_args, **_kwargs)" instead. That will help a bit, especially since apply will be deprecated soon.

Of course switching to SWIG 1.3.x for 2.5/3.0 will change everything around in the proxy classes again and probably will have a new set of issues to wonder about...

···

--
Robin Dunn
Software Craftsman
http://wxPython.org Java give you jitters? Relax with wxPython!

3

Robin Dunn wrote:

OTOH, I already have it on my ToDo to change my custom wxSWIG so it doesn't use the apply() but "foo(self, *_args, **_kwargs)" instead. That will help a bit, especially since apply will be deprecated soon.

Why not foo( *args, **_kw ) and args[0] instead of self?

Niki Spahiev

4

Seems like we could detect which ones those are and pass on them. I
didn't see any btw when I was looking, but I only breezed through...

Down the road, I might crank out a Python script to do this.

···

On Monday 24 March 2003 09:51 pm, Robin Dunn wrote:

The problem is that not all the methods look like your sample above
(many have to postprocess after the native call) and so they can't
universally be switched to the simple assignment.

--
Chuck
http://ChuckEsterbrook.com

5

Chuck Esterbrook <ChuckEsterbrook@yahoo.com> writes:

···

On Monday 24 March 2003 09:51 pm, Robin Dunn wrote:
> The problem is that not all the methods look like your sample above
> (many have to postprocess after the native call) and so they can't
> universally be switched to the simple assignment.

Seems like we could detect which ones those are and pass on them. I
didn't see any btw when I was looking, but I only breezed through...

Down the road, I might crank out a Python script to do this.

I would be very interested in seeing the results of your suggested
changes. It would be great if you could create a script that modified
the swig'd code then ran a few apps, like demo, to see what kind of a
performance impact it has. If the changes could be applied in a
clean, automated fashion that would be an easy win. (Which you know,
of course. Just thinking out loud.)

--
Patrick K. O'Brien
Orbtech http://www.orbtech.com/web/pobrien
-----------------------------------------------
"Your source for Python programming expertise."
-----------------------------------------------

6

Niki Spahiev wrote:

Robin Dunn wrote:

OTOH, I already have it on my ToDo to change my custom wxSWIG so it doesn't use the apply() but "foo(self, *_args, **_kwargs)" instead. That will help a bit, especially since apply will be deprecated soon.

Why not foo( *args, **_kw ) and args[0] instead of self?

Because the proxy methods are currently written as

  def pyfoo(self, *_args, **_kwargs):
    ...

···

--
Robin Dunn
Software Craftsman
http://wxPython.org Java give you jitters? Relax with wxPython!

7

Robin Dunn wrote:

Niki Spahiev wrote:

Why not foo( *args, **_kw ) and args[0] instead of self?

Because the proxy methods are currently written as

    def pyfoo(self, *_args, **_kwargs):
        ...

I volunteer to change that. Where are wxSWIG's sources?

Niki Spahiev

8

Niki Spahiev wrote:

Robin Dunn wrote:

Niki Spahiev wrote:

Why not foo( *args, **_kw ) and args[0] instead of self?

Because the proxy methods are currently written as

    def pyfoo(self, *_args, **_kwargs):
        ...

I volunteer to change that.

Thanks!

Where are wxSWIG's sources?

In the wxWidows CVS at wxWindows/wxPython/wxSWIG. The files in question will be in wxSWIG/Modules/python.cxx and pycpp.cxx. Be sure to use the WX_2_4_BRANCH tag and send me a patch as described on the website at the bottom of http://wxpython.org/codeguidelines.php

···

--
Robin Dunn
Software Craftsman
http://wxPython.org Java give you jitters? Relax with wxPython!

9

Ok here is the patch.

Tested on stock 2.4.0.7 sources, demo and my programs run ok.

in some files e.g. clip_dnd.* some (but not all!) occurences of _args[0].thisown=0 should become _args[1].thisown=0.

I wonder should we patch keyword arguments too?

I can make __repr__ method to show actual derived class in such cases:

     class MyFrane(wxFrame):
         passs

now repr shows wxFrame insted of MyFrame.

Niki Spahiev

zap_apply.diff (2.31 KB)

10

Ok here is the patch.

Tested on stock 2.4.0.7 sources, demo and my programs run ok.

in some files e.g. clip_dnd.* some (but not all!) occurences of
_args[0].thisown=0 should become _args[1].thisown=0.

I wonder should we patch keyword arguments too?

I can make __repr__ method to show actual derived class in such cases:

      class MyFrane(wxFrame):
          passs

now repr shows wxFrame insted of MyFrame.

Niki Spahiev

zap_apply.diff (2.31 KB)

11

Niki Spahiev <niki@vintech.bg> writes:

Ok here is the patch.

Tested on stock 2.4.0.7 sources, demo and my programs run ok.

Are you seeing a performance improvement with this patch? Any idea
how much of an improvement?

···

--
Patrick K. O'Brien
Orbtech http://www.orbtech.com/web/pobrien
-----------------------------------------------
"Your source for Python programming expertise."
-----------------------------------------------

12

Patrick K. O'Brien wrote:

Niki Spahiev <niki@vintech.bg> writes:

Ok here is the patch.

Tested on stock 2.4.0.7 sources, demo and my programs run ok.

Are you seeing a performance improvement with this patch? Any idea
how much of an improvement?

This is preparation for removing python code from most of the methods. So far only apply is removed. So did not measured yet.

Niki Spahiev

13

Niki Spahiev wrote:

Ok here is the patch.

Tested on stock 2.4.0.7 sources, demo and my programs run ok.

Thanks. I'll try this out today.

in some files e.g. clip_dnd.* some (but not all!) occurences of
_args[0].thisown=0 should become _args[1].thisown=0.

I wonder should we patch keyword arguments too?

In what way?

I can make __repr__ method to show actual derived class in such cases:

     class MyFrane(wxFrame):
         passs

now repr shows wxFrame insted of MyFrame.

I think the intent is to show that it is really just a proxy around the named C++ class and the "swigified" pointer to that C++ object, but I don't really care either way. Perhaps the repr could display something like this instead:

<MyFrame instance, Proxy of wxFrame instance at _874f828_wxFrame_p>

···

--
Robin Dunn
Software Craftsman
http://wxPython.org Java give you jitters? Relax with wxPython!

14

Robin Dunn wrote:

Niki Spahiev wrote:

Ok here is the patch.

Tested on stock 2.4.0.7 sources, demo and my programs run ok.

I've made the remove-apply change (with some changes/additions) and also made the default __repr__'s more informative. I have also regnerated all the swigged code and checked it in to CVS. If somebody else could run their eyes over it and test things I would appreciate it.

P.S. There is also a nice little surprise waiting in CVS for those adveturous enough to check out and compile...

···

--
Robin Dunn
Software Craftsman
http://wxPython.org Java give you jitters? Relax with wxPython!

15

Robin Dunn wrote:

Niki Spahiev wrote:

in some files e.g. clip_dnd.* some (but not all!) occurences of
_args[0].thisown=0 should become _args[1].thisown=0.

I wonder should we patch keyword arguments too?

In what way?

if 'somename' in _kwargs:
     _kwargs['somename'].thisown=0

currently only positional arguments loose ownership.

Niki Spahiev