Update on the Py2.6 manifest issue on Windows

Hi All,

I've been holding off on doing the 2.8.10.x release until I could solve or workaround the issue of Python 2.6 not including a reference to the themed version of the comctl32 DLL in the executable's embedded manifest. The last time I worked on it I got so frustrated that I really procrastinated taking another look at it, but I finally did this week and have made a little more progress, but still don't have a real good solution.

A little background info:

* Two versions of the comctl32 (common controls) DLL are distributed on Windows since XP. One that supports XP style themes and one that doesn't. Instead it provides the old unthemable UI that was available on win2k.

* To tell the system to use the new comctl32 DLL an application is supposed to provide a manifest file (or embedded resource) which is a little snippet of XML that specifies which "side by side assemblies" are to be loaded. (SxS assemblies are MS's answer to DLL Hell, but most developers agree that it just made it worse, not better.)

* Prior to Python 2.6 the python executables did not include a manifest (embedded or in an external file) so I was able to install my own manifest file that caused the system to load comctl32 v6 when python[w].exe is run and all was good. Python 2.6 does include an embedded manifest, but it does *not* include a reference to the comctl32 assembly, just the C Runtime. Using the old file manifest trick seems to still work on 32-bit XP but not 64-bit XP or either version of Vista. For some people wxPython apps would just look ugly, for others they would crash.

* I submitted a bug report to Python asking that the embedded resource be updated, but got enough push-back on it that I decided to try and find another solution. http://bugs.python.org/issue5019.

* wxPython 2.8.9.2 is distributed with a little script that can be used to hack the embedded manifest in the python executable, but that is obviously not a very nice or safe thing to do.

* I tried using the windows API to create explicit activation contexts that load a manifest with the reference to the themed version of comctl32. At first it seemed to have absolutely no effect whatsoever and I spent a bunch of time chasing snakes down the wrong rat hole, thinking that by the time the wxPython extension modules are loaded that it was too late to try and force a different DLL to be loaded. After all, the extension modules have their own manifest resources and they didn't make any difference either...

* Eventually we figured out that if the wxPython demo used the splash screen then it would use the old unthemed version of the DLL. If I commented out the use of the splash screen then it would use the themed version. The odd thing is that wxSpashScreen is mostly generic code and does almost nothing differently than any other wxFrame, so it just didn't make any sense that it could have this kind of effect. This is where I practically gave up several weeks ago and started procrastinating and focusing on 2.9 instead.

Ok, so that was more than just a bit of background info...

This week I was able to further narrow down the source of the problem, but still no real cause nor how to work around it. The wxSpashScreen uses a timer and in the wxPython demo we are creating and showing the main frame from the splash's EVT_CLOSE handler and the close is happening from in the timer event. If I instead click on the splash so that it closes itself from the mouse handler then the themed comctl32 is correctly used. At first I thought that it only mattered if the first window was created inside a timer event or not, but I then looked at other timers and found that any UI objects created in timer events are ignoring the explicit activation context that I set and are falling back to the process's default activation context (using the manifest embedded in python.exe.) So in other words, if you create window objects from timers you can easily have a situation where some windows are using themes and others are not. (Please don't ask me what I think about microsoft right now, you would be ashamed of me after hearing the words I would use.)

So far Google hasn't turned up anything relevant to this situation. wxTimer is just using basic win32 APIs for implementing the timer so nothing weird is going on there... If anybody has any ideas, or better Google searches, then please let me know. I do have one solution, but it involves putting what would probably be wxPython-specific code into wxWidgets, so I'd like to avoid that if possible.

P.S. The wxErlang folks ran into the same problem and talked to me about it a couple weeks ago. They eventually just convinced the Erlang project to update their embedded manifest, but since it was just a matter of talking to the guy in the cubicle down the hall it wasn't too difficult to get it approved. :wink:

···

--
Robin Dunn
Software Craftsman
http://wxPython.org Java give you jitters? Relax with wxPython!

Hi Robin,

Thanks for the detailed update and for drudging through this mess! :frowning: I really think you might want to post this update to the Python bug report, especially the point about the timer goofiness, and make one more plea to have Python fix their manifest. It just seems to me like that is where the fix belongs, unless they have some specific reason for not wanting to use the new comctl32 dll, which I didn't see in the bug tracker. (IMHO the new comctl32.dll ought to be the default, but I digress.)

Oh, one longshot idea... (wx)WebKit uncovered an issue on MSW where setting a callback function for a timer leads to any errors that occur in the callback function being eaten. Maybe they're doing some magic around the callback function that is messing things up. To get around the problem, WebKit now uses a hidden window which responds to the timer events. You can check it out here:

http://trac.webkit.org/browser/trunk/WebCore/platform/win/SharedTimerWin.cpp

Again, as I said, a longshot, but there doesn't seem to be any logical explanation as to why timers would affect this, so I thought it might be worth a mention.

Thanks,

Kevin

···

On Apr 22, 2009, at 7:27 PM, Robin Dunn wrote:

Hi All,

I've been holding off on doing the 2.8.10.x release until I could solve or workaround the issue of Python 2.6 not including a reference to the themed version of the comctl32 DLL in the executable's embedded manifest. The last time I worked on it I got so frustrated that I really procrastinated taking another look at it, but I finally did this week and have made a little more progress, but still don't have a real good solution.

A little background info:

* Two versions of the comctl32 (common controls) DLL are distributed on Windows since XP. One that supports XP style themes and one that doesn't. Instead it provides the old unthemable UI that was available on win2k.

* To tell the system to use the new comctl32 DLL an application is supposed to provide a manifest file (or embedded resource) which is a little snippet of XML that specifies which "side by side assemblies" are to be loaded. (SxS assemblies are MS's answer to DLL Hell, but most developers agree that it just made it worse, not better.)

* Prior to Python 2.6 the python executables did not include a manifest (embedded or in an external file) so I was able to install my own manifest file that caused the system to load comctl32 v6 when python[w].exe is run and all was good. Python 2.6 does include an embedded manifest, but it does *not* include a reference to the comctl32 assembly, just the C Runtime. Using the old file manifest trick seems to still work on 32-bit XP but not 64-bit XP or either version of Vista. For some people wxPython apps would just look ugly, for others they would crash.

* I submitted a bug report to Python asking that the embedded resource be updated, but got enough push-back on it that I decided to try and find another solution. Issue 5019: Specifying common controls DLL in manifest - Python tracker.

* wxPython 2.8.9.2 is distributed with a little script that can be used to hack the embedded manifest in the python executable, but that is obviously not a very nice or safe thing to do.

* I tried using the windows API to create explicit activation contexts that load a manifest with the reference to the themed version of comctl32. At first it seemed to have absolutely no effect whatsoever and I spent a bunch of time chasing snakes down the wrong rat hole, thinking that by the time the wxPython extension modules are loaded that it was too late to try and force a different DLL to be loaded. After all, the extension modules have their own manifest resources and they didn't make any difference either...

* Eventually we figured out that if the wxPython demo used the splash screen then it would use the old unthemed version of the DLL. If I commented out the use of the splash screen then it would use the themed version. The odd thing is that wxSpashScreen is mostly generic code and does almost nothing differently than any other wxFrame, so it just didn't make any sense that it could have this kind of effect. This is where I practically gave up several weeks ago and started procrastinating and focusing on 2.9 instead.

Ok, so that was more than just a bit of background info...

This week I was able to further narrow down the source of the problem, but still no real cause nor how to work around it. The wxSpashScreen uses a timer and in the wxPython demo we are creating and showing the main frame from the splash's EVT_CLOSE handler and the close is happening from in the timer event. If I instead click on the splash so that it closes itself from the mouse handler then the themed comctl32 is correctly used. At first I thought that it only mattered if the first window was created inside a timer event or not, but I then looked at other timers and found that any UI objects created in timer events are ignoring the explicit activation context that I set and are falling back to the process's default activation context (using the manifest embedded in python.exe.) So in other words, if you create window objects from timers you can easily have a situation where some windows are using themes and others are not. (Please don't ask me what I think about microsoft right now, you would be ashamed of me after hearing the words I would use.)

So far Google hasn't turned up anything relevant to this situation. wxTimer is just using basic win32 APIs for implementing the timer so nothing weird is going on there... If anybody has any ideas, or better Google searches, then please let me know. I do have one solution, but it involves putting what would probably be wxPython-specific code into wxWidgets, so I'd like to avoid that if possible.

P.S. The wxErlang folks ran into the same problem and talked to me about it a couple weeks ago. They eventually just convinced the Erlang project to update their embedded manifest, but since it was just a matter of talking to the guy in the cubicle down the hall it wasn't too difficult to get it approved. :wink:

--
Robin Dunn
Software Craftsman
http://wxPython.org Java give you jitters? Relax with wxPython!

_______________________________________________
wxpython-dev mailing list
wxpython-dev@lists.wxwidgets.org
http://lists.wxwidgets.org/mailman/listinfo/wxpython-dev

Robin Dunn <robin <at> alldunn.com> writes:

...

Thanks for the info.
The problem seems to be so accute and there are so many people suffering from
this, I'm just wondering if "Python" should not plead guilty in that case.

Jean-Michel Fauth, Switzerland

Hi Robin,

Kevin Ollivier wrote:

Hi Robin,

Thanks for the detailed update and for drudging through this mess! :frowning: I really think you might want to post this update to the Python bug report, especially the point about the timer goofiness, and make one more plea to have Python fix their manifest. It just seems to me like that is where the fix belongs, unless they have some specific reason for not wanting to use the new comctl32 dll, which I didn't see in the bug tracker. (IMHO the new comctl32.dll ought to be the default, but I digress.)

Wouldn't "tk" have the same issue? Maybe that might be an additional point to make when requesting that the new comctl32.dll be defined in the default manifest.

Werner

Kevin Ollivier wrote:

Hi Robin,

Thanks for the detailed update and for drudging through this mess! :frowning: I really think you might want to post this update to the Python bug report, especially the point about the timer goofiness, and make one more plea to have Python fix their manifest. It just seems to me like that is where the fix belongs, unless they have some specific reason for not wanting to use the new comctl32 dll, which I didn't see in the bug tracker. (IMHO the new comctl32.dll ought to be the default, but I digress.)

Maybe TK is a bad pick, is it even affected, as I haven't seen any nice GUI done with it.

Maybe PythonWin would be a better example, especially as Mark Hammond is very up to speed on Win stuff.

Werner

Robin Dunn wrote:

in python.exe.) So in other words, if you create window objects from timers you can easily have a situation where some windows are using themes and others are not.

Wild idea: can timer code reactivate the special activation context?

HTH

Niki Spahiev

Kevin Ollivier wrote:

Hi Robin,

Thanks for the detailed update and for drudging through this mess! :frowning: I really think you might want to post this update to the Python bug report, especially the point about the timer goofiness, and make one more plea to have Python fix their manifest. It just seems to me like that is where the fix belongs, unless they have some specific reason for not wanting to use the new comctl32 dll, which I didn't see in the bug tracker. (IMHO the new comctl32.dll ought to be the default, but I digress.)

Maybe a good candidate for the Python bug day for Saturday April 25, 2009 :slight_smile:
Werner

Hi, I almost forgot about this ridiculous issue (this is honestly a stupid problem to have to deal with). Have you committed any of the changes that you have been working on? I have a few ideas that I would like to try but it might be a good idea to make sure they aren't futile first.

Incidentally, do you guys have a page explaining what I need to do first in order to build .pyds? Or is it a fairly standard build process?

···

--------------------------------------------------
From: "Robin Dunn" <robin@alldunn.com>
Sent: Wednesday, April 22, 2009 10:27 PM
To: <wxPython-dev@lists.wxwidgets.org>; "Vadim Zeitlin" <vadim@wxwidgets.org>
Subject: [wxpython-dev] Update on the Py2.6 manifest issue on Windows

Hi All,

I've been holding off on doing the 2.8.10.x release until I could solve or workaround the issue of Python 2.6 not including a reference to the themed version of the comctl32 DLL in the executable's embedded manifest. The last time I worked on it I got so frustrated that I really procrastinated taking another look at it, but I finally did this week and have made a little more progress, but still don't have a real good solution.

A little background info:

* Two versions of the comctl32 (common controls) DLL are distributed on Windows since XP. One that supports XP style themes and one that doesn't. Instead it provides the old unthemable UI that was available on win2k.

* To tell the system to use the new comctl32 DLL an application is supposed to provide a manifest file (or embedded resource) which is a little snippet of XML that specifies which "side by side assemblies" are to be loaded. (SxS assemblies are MS's answer to DLL Hell, but most developers agree that it just made it worse, not better.)

* Prior to Python 2.6 the python executables did not include a manifest (embedded or in an external file) so I was able to install my own manifest file that caused the system to load comctl32 v6 when python[w].exe is run and all was good. Python 2.6 does include an embedded manifest, but it does *not* include a reference to the comctl32 assembly, just the C Runtime. Using the old file manifest trick seems to still work on 32-bit XP but not 64-bit XP or either version of Vista. For some people wxPython apps would just look ugly, for others they would crash.

* I submitted a bug report to Python asking that the embedded resource be updated, but got enough push-back on it that I decided to try and find another solution. Issue 5019: Specifying common controls DLL in manifest - Python tracker.

* wxPython 2.8.9.2 is distributed with a little script that can be used to hack the embedded manifest in the python executable, but that is obviously not a very nice or safe thing to do.

* I tried using the windows API to create explicit activation contexts that load a manifest with the reference to the themed version of comctl32. At first it seemed to have absolutely no effect whatsoever and I spent a bunch of time chasing snakes down the wrong rat hole, thinking that by the time the wxPython extension modules are loaded that it was too late to try and force a different DLL to be loaded. After all, the extension modules have their own manifest resources and they didn't make any difference either...

* Eventually we figured out that if the wxPython demo used the splash screen then it would use the old unthemed version of the DLL. If I commented out the use of the splash screen then it would use the themed version. The odd thing is that wxSpashScreen is mostly generic code and does almost nothing differently than any other wxFrame, so it just didn't make any sense that it could have this kind of effect. This is where I practically gave up several weeks ago and started procrastinating and focusing on 2.9 instead.

Ok, so that was more than just a bit of background info...

This week I was able to further narrow down the source of the problem, but still no real cause nor how to work around it. The wxSpashScreen uses a timer and in the wxPython demo we are creating and showing the main frame from the splash's EVT_CLOSE handler and the close is happening from in the timer event. If I instead click on the splash so that it closes itself from the mouse handler then the themed comctl32 is correctly used. At first I thought that it only mattered if the first window was created inside a timer event or not, but I then looked at other timers and found that any UI objects created in timer events are ignoring the explicit activation context that I set and are falling back to the process's default activation context (using the manifest embedded in python.exe.) So in other words, if you create window objects from timers you can easily have a situation where some windows are using themes and others are not. (Please don't ask me what I think about microsoft right now, you would be ashamed of me after hearing the words I would use.)

So far Google hasn't turned up anything relevant to this situation. wxTimer is just using basic win32 APIs for implementing the timer so nothing weird is going on there... If anybody has any ideas, or better Google searches, then please let me know. I do have one solution, but it involves putting what would probably be wxPython-specific code into wxWidgets, so I'd like to avoid that if possible.

P.S. The wxErlang folks ran into the same problem and talked to me about it a couple weeks ago. They eventually just convinced the Erlang project to update their embedded manifest, but since it was just a matter of talking to the guy in the cubicle down the hall it wasn't too difficult to get it approved. :wink:

--
Robin Dunn
Software Craftsman
http://wxPython.org Java give you jitters? Relax with wxPython!

_______________________________________________
wxpython-dev mailing list
wxpython-dev@lists.wxwidgets.org
http://lists.wxwidgets.org/mailman/listinfo/wxpython-dev

Kevin Ollivier wrote:

Oh, one longshot idea... (wx)WebKit uncovered an issue on MSW where setting a callback function for a timer leads to any errors that occur in the callback function being eaten. Maybe they're doing some magic around the callback function that is messing things up. To get around the problem, WebKit now uses a hidden window which responds to the timer events. You can check it out here:

http://trac.webkit.org/browser/trunk/WebCore/platform/win/SharedTimerWin.cpp

Again, as I said, a longshot, but there doesn't seem to be any logical explanation as to why timers would affect this, so I thought it might be worth a mention.

YES!!!

I played around with having a hidden window on Monday or Tuesday and it didn't seem to make any difference, but based on the webkit code I tried it again a little differently and it looks like it is working! YAY!

···

--
Robin Dunn
Software Craftsman
http://wxPython.org Java give you jitters? Relax with wxPython!

niki wrote:

Robin Dunn wrote:

in python.exe.) So in other words, if you create window objects from timers you can easily have a situation where some windows are using themes and others are not.

Wild idea: can timer code reactivate the special activation context?

Yes. That's the option I mentioned that required some wxPython specific code in wxWidgets. (I essentially wrapped the call to timer.Notify inside code that resets the activation context and then releases it again.) Looks like I won't need to use that fallback option however.

···

--
Robin Dunn
Software Craftsman
http://wxPython.org Java give you jitters? Relax with wxPython!

Te-jé Rodgers wrote:

Hi, I almost forgot about this ridiculous issue (this is honestly a stupid problem to have to deal with). Have you committed any of the changes that you have been working on? I have a few ideas that I would like to try but it might be a good idea to make sure they aren't futile first.

I haven't committed anything yet, but I will soon.

Incidentally, do you guys have a page explaining what I need to do first in order to build .pyds? Or is it a fairly standard build process?

wxPython's pyd's or in general? The former is at http://wxpython.org/builddoc.php

···

--
Robin Dunn
Software Craftsman
http://wxPython.org Java give you jitters? Relax with wxPython!

Werner F. Bruhin wrote:

Maybe TK is a bad pick, is it even affected, as I haven't seen any nice GUI done with it.

TK (at least used to) draw its own widgets, so it's not an issue there.

-Chris

···

--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov

Hi Robin,

Kevin Ollivier wrote:

Oh, one longshot idea... (wx)WebKit uncovered an issue on MSW where setting a callback function for a timer leads to any errors that occur in the callback function being eaten. Maybe they're doing some magic around the callback function that is messing things up. To get around the problem, WebKit now uses a hidden window which responds to the timer events. You can check it out here:
http://trac.webkit.org/browser/trunk/WebCore/platform/win/SharedTimerWin.cpp Again, as I said, a longshot, but there doesn't seem to be any logical explanation as to why timers would affect this, so I thought it might be worth a mention.

YES!!!

I played around with having a hidden window on Monday or Tuesday and it didn't seem to make any difference, but based on the webkit code I tried it again a little differently and it looks like it is working! YAY!

Woah, awesome, thanks for checking this out!! Even I doubted it would work when I suggested it... :wink: Since wxWebKit will only work with Python 2.6, having this solved is a big help. :slight_smile:

Regards,

Kevin

···

On Apr 23, 2009, at 9:59 AM, Robin Dunn wrote:

--
Robin Dunn
Software Craftsman
http://wxPython.org Java give you jitters? Relax with wxPython!

_______________________________________________
wxpython-dev mailing list
wxpython-dev@lists.wxwidgets.org
http://lists.wxwidgets.org/mailman/listinfo/wxpython-dev

I've tried a deb package of wxWebKit for Ubuntu from an earlier thread
and it seemed to work with Python 2.5. Will support for Python 2.5 be
removed?

···

On Fri, Apr 24, 2009 at 5:09 AM, Kevin Ollivier <kevino@theolliviers.com> wrote:

Hi Robin,

On Apr 23, 2009, at 9:59 AM, Robin Dunn wrote:

Kevin Ollivier wrote:

Oh, one longshot idea... (wx)WebKit uncovered an issue on MSW where
setting a callback function for a timer leads to any errors that occur in
the callback function being eaten. Maybe they're doing some magic around the
callback function that is messing things up. To get around the problem,
WebKit now uses a hidden window which responds to the timer events. You can
check it out here:

http://trac.webkit.org/browser/trunk/WebCore/platform/win/SharedTimerWin.cpp Again,
as I said, a longshot, but there doesn't seem to be any logical explanation
as to why timers would affect this, so I thought it might be worth a
mention.

YES!!!

I played around with having a hidden window on Monday or Tuesday and it
didn't seem to make any difference, but based on the webkit code I tried it
again a little differently and it looks like it is working! YAY!

Woah, awesome, thanks for checking this out!! Even I doubted it would work
when I suggested it... :wink: Since wxWebKit will only work with Python 2.6,
having this solved is a big help. :slight_smile:

--
Alexei Vinidiktov

Hi Robin & All,

niki wrote:

Robin Dunn wrote:

in python.exe.) So in other words, if you create window objects from
timers you can easily have a situation where some windows are using themes
and others are not.

Wild idea: can timer code reactivate the special activation context?

Yes. That's the option I mentioned that required some wxPython specific
code in wxWidgets. (I essentially wrapped the call to timer.Notify inside
code that resets the activation context and then releases it again.) Looks
like I won't need to use that fallback option however.

I am happy you were able to sort this thing out.

<rant>
It's a shame you spent so much time on a stupid issue like this, while
it might have taken a couple of minutes for a Python-dev to modify the
Python source to simplify the life of wxPython (and other packages)
maintainers. This is the second time I feel like blaming Python-devs.
The first was for Python 3 itself.

</rant>

Andrea.

"Imagination Is The Only Weapon In The War Against Reality."
http://xoomer.alice.it/infinity77/

···

On Thu, Apr 23, 2009 at 6:00 PM, Robin Dunn wrote: