lib.iewin hangs py2exe build

Mike Conley wrote:

Robin got me on the right track here.
  comtypes\gen\_3050F1C5_98B5_11CF_BB82_00AA00BDCE0B_0_4_0.py
is a 60,000+ file. the .pyc is over 1.4Mb

trying to run a one liner that just imports that file through py2exe creates the problem. I suspect that letting it run long enough would eventually go to conclusion. Unfortunately we do our build on a relatively low power machine and probably can't change that any time soon (startups don't have much money).

Does anyone know if there is a technique to tell py2exe to "put this in your output, but don't bother to analyze it"? If so, I can manually include this file and any relevant dependencies.

  Here Is an email I just sent the maintainer of py2exe Jimmy Retzlaff

  Adding "skip_scan" option to py2exe

···

===================================

  Refer To: py2exe version 0.6.9

  Introduction
  =============
  I'm using py2exe for exe-ing a wxPython application. Recently I had to add
  the embedded Internet Explorer module "wx.lib.iewin" to my application. This
  module uses "comtypes" which is generating *very* big module that need to be
  included in the distribution executable. Scanning this big module takes
  about 15 minutes on my computer (Duo Core, 2G RAM). The idea is to tell
  py2exe not to scan this module and add any required module manually through
  the "includes" option.

  Implementation
  ===============
  I added a "skip_scan" option to py2exe, which is a list of modules names
  (like: includes, excludes)

  Changes in the code:

  build_exe.py:
    After line 162 in py2exe, insert:
    ("skip-scan=", None, "comma-separated list of modules not to scan for imported modules"),

    Change line 342 in py2exe.create_modulefinder to:
    return ModuleFinder(excludes=self.excludes, skip_scan=self.skip_scan)

  mf.py:
    Change line 87 definition of ModuleFinder.__init__ to:
    def __init__(self, path=None, debug=0, excludes=, replace_paths=,
  skip_scan=):

    After line 96 in ModuleFinder.__init__ insert:
    self.skip_scan = skip_scan

    After line 404 in ModuleFinder.scan_code insert:
    if m.__name__ in self.skip_scan: return

  Usage
  ======
  A strip down example:
    setup (
      windows = my_win_app,
      options = {
          'py2exe': {
            'includes': ['my_module', <any module "my_module" needs>],
            'skip_scan': ['my_module']
          }
      }
  "my_module" will be included in the executable, but will not be scanned for
  modules imported from it.

  Risks
  ======
  of course I introduced the risk of missing modules in the executable. The
  developer will have to test the application, and make sure all the required
  modules exists. In my case, it is a risk worth taking as it saves almost 15
  minutes from the build process.

  Thank you for developing py2exe, it is an excellent tool!

  Erez Bibi
  Consultant for blackwoodproductions.com
  erezbibi AT frontiernet DOT net

Erez Bibi wrote:

Does anyone know if there is a technique to tell py2exe to "put this in your output, but don't bother to analyze it"? If so, I can manually include this file and any relevant dependencies.

Here Is an email I just sent the maintainer of py2exe Jimmy Retzlaff

great! thanks, that's been a pain for me, too. I hope your patch gets accepted.

-Chris

···

--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov

Almost time to add another option to GUI2Exe then :smiley:

Andrea.

"Imagination Is The Only Weapon In The War Against Reality."
http://xoomer.alice.it/infinity77/

···

On Mon, Jun 1, 2009 at 9:39 PM, Christopher Barker wrote:

Erez Bibi wrote:

Does anyone know if there is a technique to tell py2exe to "put this in
your output, but don't bother to analyze it"? If so, I can manually include
this file and any relevant dependencies.

Here Is an email I just sent the maintainer of py2exe Jimmy Retzlaff

great! thanks, that's been a pain for me, too. I hope your patch gets
accepted.

Erez Bibi schrieb:

Mike Conley wrote:

Robin got me on the right track here.
  comtypes\gen\_3050F1C5_98B5_11CF_BB82_00AA00BDCE0B_0_4_0.py
is a 60,000+ file. the .pyc is over 1.4Mb

trying to run a one liner that just imports that file through py2exe
creates the problem. I suspect that letting it run long enough would
eventually go to conclusion. Unfortunately we do our build on a
relatively low power machine and probably can't change that any time
soon (startups don't have much money).

Does anyone know if there is a technique to tell py2exe to "put this
in your output, but don't bother to analyze it"? If so, I can manually
include this file and any relevant dependencies.

  Here Is an email I just sent the maintainer of py2exe Jimmy Retzlaff

Have you received an answer from Jimmy? He told me (some time ago, in private
email) that he wants to step back from maintaining py2exe. So, you should at
least post your 'patch' to the py2exe-users list, and submit it to the
py2exe bug tracker. No guarantee that this will lead to inclusion of your
patch, but at least it won't get lost so easily.

  Adding "skip_scan" option to py2exe
  ===================================

  Refer To: py2exe version 0.6.9

  Introduction
  =============
  I'm using py2exe for exe-ing a wxPython application. Recently I had to add
  the embedded Internet Explorer module "wx.lib.iewin" to my application. This
  module uses "comtypes" which is generating *very* big module that need to be
  included in the distribution executable. Scanning this big module takes
  about 15 minutes on my computer (Duo Core, 2G RAM). The idea is to tell
  py2exe not to scan this module and add any required module manually through
  the "includes" option.

  Implementation
  ===============
  I added a "skip_scan" option to py2exe, which is a list of modules names
  (like: includes, excludes)

[...]

Have you tried to profile and speed up modulefinder, so that it gets faster?
I think that this would also be worth to try.

···

--
Thanks,
Thomas

OT, but...

Thomas Heller wrote:

Have you tried to profile and speed up modulefinder, so that it gets faster?
I think that this would also be worth to try.

or port py2exe to modulegraph, used by py2app, and now by bbfreeze. I think it's faster and maybe more robust, but that may be a function of what machines I'm running on...

It would be really nice to have a grand unification of GUI builders. It strikes me that there are three components to any GUI builder:

1) The API presented to the user: how you specify what you want

2) The module finding code: how the system figure out what to include?

3) The actual app builder -- how everything gets packaged up.
    - this is necessarily different for each platform, and appears to be one of the key differences between different systems on the same OS.

Anyway, ti would be nice if there was more shared code between the GUI builders -- at least (2) would help a lot. aside from coding efficiencies, It would be nice to be more platform independent. For instance, py2app's API is modeled after py2exe. However, they use totally different module finding code, so I find I need to give totally different extra directive for any but trivial packages -- that's a real pain.

oh well, I'm not offering to write anything...

-Chris

···

--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov

Christopher Barker schrieb:

OT, but...

Thomas Heller wrote:

Have you tried to profile and speed up modulefinder, so that it gets faster?
I think that this would also be worth to try.

or port py2exe to modulegraph, used by py2app, and now by bbfreeze. I
think it's faster and maybe more robust, but that may be a function of
what machines I'm running on...

It would be really nice to have a grand unification of GUI builders. It
strikes me that there are three components to any GUI builder:

GUI builders? You mean APP builders, I assume.

1) The API presented to the user: how you specify what you want

2) The module finding code: how the system figure out what to include?

3) The actual app builder -- how everything gets packaged up.
    - this is necessarily different for each platform, and appears to be
one of the key differences between different systems on the same OS.

Anyway, ti would be nice if there was more shared code between the GUI
builders -- at least (2) would help a lot. aside from coding
efficiencies, It would be nice to be more platform independent. For
instance, py2app's API is modeled after py2exe. However, they use
totally different module finding code, so I find I need to give totally
different extra directive for any but trivial packages -- that's a real
pain.

Sure, plus a hooks mechanism like PyInstaller has it...

oh well, I'm not offering to write anything...

And that is the real problem. Given the popularity of py2exe and friends
I really wonder why there isn't more activity in development.

···

--
Thomas

Thomas Heller <theller <at> ctypes.org> writes:

> oh well, I'm not offering to write anything...

And that is the real problem. Given the popularity of py2exe and friends
I really wonder why there isn't more activity in development.

- Because it is a complex task. It requires a very good knowlege
of Python *and* MiscroSoft stuff.
- It does not require only Python, also the MS tools.
- Idealy, a potential developer shoud have more than one Windows version.
(today, XP, Vista, 7).
- It's true, there are plenty of users. Most of them do not understand
how py2exe works.
- ...
- Practically , a "full time job".

You know this better than me.

Contrary to Chris' opinion, I think such a packager tool should be
platform specific. It's complicate enough [*] and the philosophy
"one for all" leads to "half backed" applications.

[*] I spent my evening yesterday toying with embedding icons in a
test exe for Vista...

py2exe is a great tool.

jmf

Thomas Heller-2 wrote:

Have you received an answer from Jimmy? He told me (some time ago, in
private
email) that he wants to step back from maintaining py2exe. So, you should
at
least post your 'patch' to the py2exe-users list, and submit it to the
py2exe bug tracker. No guarantee that this will lead to inclusion of your
patch, but at least it won't get lost so easily.

I have submitted a py2exe patch and to the users list (thanks for the
advice)

Thomas Heller-2 wrote:

Have you tried to profile and speed up modulefinder, so that it gets
faster?
I think that this would also be worth to try.

Like Chris wrote "oh well, I'm not offering to write anything..."

Well me too (at least for now), I just needed this feature.

···

--
View this message in context: http://www.nabble.com/lib.iewin-hangs-py2exe-build-tp22973879p23877937.html
Sent from the wxPython-users mailing list archive at Nabble.com.

j wrote:

- Practically , a "full time job".

so true.

Contrary to Chris' opinion, I think such a packager tool should be
platform specific. It's complicate enough [*] and the philosophy
"one for all" leads to "half backed" applications.

I DO think that they need to be platform specific, but the entire thing does not -- figuring out what modules need to be included is exactly the same on all platforms -- why have multiple *half-baked* implementations of that?

I also don't see any reason they couldn't have the same API.

Actually bundling up the stuff certainly has to be platform dependent.

-Chris

···

--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov

Christopher Barker schrieb:

OT, but...

Thomas Heller wrote:

Have you tried to profile and speed up modulefinder, so that it gets faster?
I think that this would also be worth to try.

or port py2exe to modulegraph, used by py2app, and now by bbfreeze. I
think it's faster and maybe more robust, but that may be a function of
what machines I'm running on...

Well, I finally tried out parsing the comtypes\gen\MSHTML.py module (which
pulls in this giant comtypes\gen\_3050F1C5_98B5_11CF_BB82_00AA00BDCE0B_0_4_0.py
module), and there were no diffence in the time needed to parse this code.

Both took 3.9 seconds on my machine.

···

--
Thanks,
Thomas

Hello,

sorry to pull out this old topic again, but I ran into exactly the same
issue today (although build time was closer to 5 minutes than 15 minutes).

The patch below basically works, but it is missing these two bits:

build_exe.py:
       after line 200 ("self.dll_excludes = None") put:
           self.skip_scan = None

       after line 234 ("self.dll_excludes = ...") put:
           self.skip_scan = fancy_split(self.skip_scan)

After these two changes I got it working. I used this as the specific
modulename for wx.lib.iewin:

      'skip_scan' : [
'comtypes.gen._3050F1C5_98B5_11CF_BB82_00AA00BDCE0B_0_4_0' ]

I've also checked SVN at

       py2exe download | SourceForge.net

and it doesn't seem this patch made it through so far. If I prepare a
patch is there anybody who will commit it?

-Matthias

···

Am 01.06.2009, 21:59 Uhr, schrieb Erez Bibi <erezbibi@frontiernet.net>:

  Here Is an email I just sent the maintainer of py2exe Jimmy Retzlaff

  Adding "skip_scan" option to py2exe
  ===================================

  Refer To: py2exe version 0.6.9

  Introduction
  =============
  I'm using py2exe for exe-ing a wxPython application. Recently I had to add
  the embedded Internet Explorer module "wx.lib.iewin" to my application. This
  module uses "comtypes" which is generating *very* big module that need to be
  included in the distribution executable. Scanning this big module takes
  about 15 minutes on my computer (Duo Core, 2G RAM). The idea is to tell
  py2exe not to scan this module and add any required module manually through
  the "includes" option.

  Implementation
  ===============
  I added a "skip_scan" option to py2exe, which is a list of modules names
  (like: includes, excludes)

  Changes in the code:

  build_exe.py:
    After line 162 in py2exe, insert:
    ("skip-scan=", None, "comma-separated list of modules not to scan for imported modules"),

    Change line 342 in py2exe.create_modulefinder to:
    return ModuleFinder(excludes=self.excludes, skip_scan=self.skip_scan)

  mf.py:
    Change line 87 definition of ModuleFinder.__init__ to:
    def __init__(self, path=None, debug=0, excludes=, replace_paths=,
  skip_scan=):

    After line 96 in ModuleFinder.__init__ insert:
    self.skip_scan = skip_scan

    After line 404 in ModuleFinder.scan_code insert:
    if m.__name__ in self.skip_scan: return

  Usage
  ======
  A strip down example:
    setup (
      windows = my_win_app,
      options = {
          'py2exe': {
            'includes': ['my_module', <any module "my_module" needs>],
            'skip_scan': ['my_module']
          }
      }
  "my_module" will be included in the executable, but will not be scanned for
  modules imported from it.

  Risks
  ======
  of course I introduced the risk of missing modules in the executable. The
  developer will have to test the application, and make sure all the required
  modules exists. In my case, it is a risk worth taking as it saves almost 15
  minutes from the build process.

  Thank you for developing py2exe, it is an excellent tool!

  Erez Bibi
  Consultant for blackwoodproductions.com
  erezbibi AT frontiernet DOT net
_______________________________________________
wxpython-users mailing list
wxpython-users@lists.wxwidgets.org
http://lists.wxwidgets.org/mailman/listinfo/wxpython-users