use zipfile and wxImageFromStream instead of img2py

Since Cliff Wells brought this up last night in an off-list email and Mike
Fletcher just posted info about his ResourcePackage code I have to ask the
same question that I posed to Cliff last night. Now that we have
wxImageFromStream what exactly is the point of using img2py? Also, there
seems to be little benefit of storing wxWindows style image related objects
since it complicates editing and changing the underlying bitmaps and icons.

http://www.python.org/doc/current/lib/module-zipfile.html

My solution is to just use zip files if you want to bundle up a bunch of
files for an app. If you're using py2exe to make a standalone, the zip file
just becomes one of the data_files in the py2exe setup.py script.

I stuck a couple of images into a doodle.zip zipfile and then did this in
the shell with the PythonCard doodle sample to show a simple inspection of a
zip file and then extracting the image into a wxImage with
wxImageFromStream.

import zipfile
f = zipfile.ZipFile('doodle.zip')
f.filelist

[<zipfile.ZipInfo instance at 0x011B0398>, <zipfile.ZipInfo instance at
0x011B02D0>]

m = f.filelist[0]
m.filename

'background.jpg'

f.namelist()

['background.jpg', 'edit.gif']

data = f.read('background.jpg')
len(data)

30379

from cStringIO import StringIO
img = wx.wxImageFromStream(StringIO(data))
comp.bufOff.DrawBitmap(img)

ka

I considered making my package generate zip files, but here's the problems with them:

    You need to find them at run-time (which can be a pain, it requires knowing whether you're py2exe'd or not (you need to be able to find the root directory of the install)). That's solvable, after all, there's only two pieces of code, one for "py2exe'd", one for "not". You could include that code in a module and just use it when needed. f = zipfile.ZipFile('doodle.zip') isn't realistic in my (admittedly limited) experience, as you need to know in which directory 'doodle.zip' has been stored.

    You need to refresh the zipfile when you change a file (most graphics packages don't write into zipfiles directly), so you wind up needing scanning code similar to resourcepackage's.

    Requires the latest wxPython (note, the current version of resourcepackage's wxgenerators also does, but that's configurable simply by subclassing for older versions).

and here's the problems I see with img2py and resourcepackage:

    Code bloat (python files are very large compared to the binary versions).

    Manual generation w/ img2py.

That said, we could eliminate the first two problems for zipfiles by having resourcepackage refresh into a zipfile, and by having a module to do the "find the zipfile regardless of which type of install it is" code.

Shrug,
Mike

Kevin Altis wrote:

Since Cliff Wells brought this up last night in an off-list email and Mike
Fletcher just posted info about his ResourcePackage code I have to ask the
same question that I posed to Cliff last night. Now that we have
wxImageFromStream what exactly is the point of using img2py? Also, there
seems to be little benefit of storing wxWindows style image related objects
since it complicates editing and changing the underlying bitmaps and icons.

http://www.python.org/doc/current/lib/module-zipfile.html

My solution is to just use zip files if you want to bundle up a bunch of
files for an app. If you're using py2exe to make a standalone, the zip file
just becomes one of the data_files in the py2exe setup.py script.

...

···

_______________________________________
  Mike C. Fletcher
  Designer, VR Plumber, Coder
  http://members.rogers.com/mcfletch/

You probably just want sys.argv[0], in which case you can store it away
early in your main module.

import os, sys
print os.path.abspath(sys.argv[0])

If you need something that is relative to the __file__ attribute of a
module, then again, early in your main module, change sys.path[0] to be
absolute and then all the __file__ attributes will end up absolute. This was
a recent topic of discussion on python-dev. It has been patched for Python
2.3, but the code below will essentially have the same effect.

import sys
sys.path[0] = os.path.abspath(sys.path[0])

Does that solve the problem?

The scanning and packaging code for creating and/or updating a zip would
still be useful and that could automatically be called as part of a py2exe
setup.py script. Gordon McMillan's Installer would be a bit trickier since
it uses a .spec file.

Either the zipfile itself could be used as the manifest for the code that
updates all files in the zip or a separate manifest could be used, but using
the zip seems to make the most sense. It just means you have to create the
initial zip with the hierarchy and filenames you want and after that a
simple Python script reads the zip and updates any files that have changed.

ka

···

-----Original Message-----
From: Mike C. Fletcher [mailto:mcfletch@rogers.com]
Sent: Saturday, January 11, 2003 9:33 AM
To: wxPython-users@lists.wxwindows.org
Subject: Re: [wxPython-users] use zipfile and wxImageFromStream instead
of img2py

I considered making my package generate zip files, but here's the
problems with them:

    You need to find them at run-time (which can be a pain, it requires
knowing whether you're py2exe'd or not (you need to be able to find the
root directory of the install)). That's solvable, after all, there's
only two pieces of code, one for "py2exe'd", one for "not". You could
include that code in a module and just use it when needed. f =
zipfile.ZipFile('doodle.zip') isn't realistic in my (admittedly limited)
experience, as you need to know in which directory 'doodle.zip' has been
stored.

    You need to refresh the zipfile when you change a file (most
graphics packages don't write into zipfiles directly), so you wind up
needing scanning code similar to resourcepackage's.

    Requires the latest wxPython (note, the current version of
resourcepackage's wxgenerators also does, but that's configurable simply
by subclassing for older versions).

and here's the problems I see with img2py and resourcepackage:

    Code bloat (python files are very large compared to the binary
versions).

    Manual generation w/ img2py.

That said, we could eliminate the first two problems for zipfiles by
having resourcepackage refresh into a zipfile, and by having a module to
do the "find the zipfile regardless of which type of install it is" code.

Shrug,
Mike

Kevin Altis wrote:

>Since Cliff Wells brought this up last night in an off-list
email and Mike
>Fletcher just posted info about his ResourcePackage code I have
to ask the
>same question that I posed to Cliff last night. Now that we have
>wxImageFromStream what exactly is the point of using img2py? Also, there
>seems to be little benefit of storing wxWindows style image
related objects
>since it complicates editing and changing the underlying bitmaps
and icons.
>
>http://www.python.org/doc/current/lib/module-zipfile.html
>
>My solution is to just use zip files if you want to bundle up a bunch of
>files for an app. If you're using py2exe to make a standalone,
the zip file
>just becomes one of the data_files in the py2exe setup.py script.
>
>
...

_______________________________________
  Mike C. Fletcher
  Designer, VR Plumber, Coder
  http://members.rogers.com/mcfletch/

---------------------------------------------------------------------
To unsubscribe, e-mail: wxPython-users-unsubscribe@lists.wxwindows.org
For additional commands, e-mail: wxPython-users-help@lists.wxwindows.org

Kevin Altis wrote:

You probably just want sys.argv[0], in which case you can store it away
early in your main module.

import os, sys
print os.path.abspath(sys.argv[0])

If you need something that is relative to the __file__ attribute of a
module, then again, early in your main module, change sys.path[0] to be
absolute and then all the __file__ attributes will end up absolute. This was
a recent topic of discussion on python-dev. It has been patched for Python
2.3, but the code below will essentially have the same effect.

import sys
sys.path[0] = os.path.abspath(sys.path[0])

Does that solve the problem?

Not entirely (well, it doesn't feel particularly "clean" to me), as it requires that the resource-set be application-specific (every application needs to copy the .zip from every library it uses to it's root (or whatever) directory). Would be really nice to have that inclusion and updating automated. Also doesn't deal with needing the root all over the app/package (i.e. you'd want a module somewhere to do the loading of the .zip).

Example:

wxoo has, at present, 10 icon resources. Those are used all over wxoo and wxprop. Each point in wxoo and wxprop that uses a zipfile-based system needs access to the "root" of the application directory. It then computes the filename root + "wxoo.zip", and your py2exe command then needs to include that .zip in its data-files set.

ConflictSolver (which uses wxoo) has 21 icon resources (at the moment), plus a number of non-icon resources. Again, these resources are used all over the application, and every point a resource is used needs to know how to get to the resource archive for ConflictSolver.

ConflictSolver is normally (during development) run with conflictsolver/ui/application.py . Py2exe makes the result conflictsolver/conflictsolver.exe, so os.path.abspath(sys.path[0]) (root directory) is changing from conflictsolver/ to conflictsolver/ui depending on whether it's the py2exe'd version or not, so you'd want to be able to fix that to not need multiple .zip files.

With resourcepackage, I create the two directories, copy the __init__.py file into them, and can then import the resource modules. When I compose multiple packages, they are all able to access their resources w/out any work in the application that's using them (e.g. don't need to copy the .zip to the application directory). The run-time code has no dependencies on the scanning __init__ mechanism or resourcepackage, it's just modules with raw Python code (or wxPython code, depending on the generators chosen).

The scanning and packaging code for creating and/or updating a zip would
still be useful and that could automatically be called as part of a py2exe
setup.py script. Gordon McMillan's Installer would be a bit trickier since
it uses a .spec file.

Either the zipfile itself could be used as the manifest for the code that
updates all files in the zip or a separate manifest could be used, but using
the zip seems to make the most sense. It just means you have to create the
initial zip with the hierarchy and filenames you want and after that a
simple Python script reads the zip and updates any files that have changed.

...

The problems are all solvable, no question, but as of yet I think I'll stick with python-module-embedding via resourcepackage (I just finished switching wxpypropdist and conflictsolver to it). I'll be happy to switch to zip-mediated versions when it seems more convenient :slight_smile: . I'd even be willing to help with it if the approach sounds like it'd be more convenient/reliable than what I've got now.

BTW, if the zip packaging is only called during py2exe, are you including code somewhere to load directly from disk if running w/out py2exe-ing? I'd suggest it's _much_ nicer to have the same mechanism used all the time.

Enjoy,
Mike

I think you are making it more complicated than it needs to be. Yes, it may
require a refactor of your current resource handling strategy, but it seems
like that is worth doing to improve usability.

One, any module can get sys.argv[0] whenever it needs it. But it is probably
just simpler in your specific case to still use a module that knows how to
get the resources and every other module just imports and uses that module
normally. That also means that you won't be keeping multiple copies of the
same wxBitmap, wxIcon or whatever, just using one reference from a
dictionary or list in your resource handling module. For example, module x
tries to use bitmapY from the resource module and if it isn't already
avaiable, the resource module automatically loads it from the zip.

The main point of all this is to simply get away from sticking images, text,
html, xml, or any other support data your application needs in Python simply
for the sake of being able to use Python imports.

Where the zip goes can be module relative or application relative, it is up
to you, it will work either way regardless of whether you are using a
standalone or in development.

ka

···

-----Original Message-----
From: Mike C. Fletcher [mailto:mcfletch@rogers.com]
Sent: Saturday, January 11, 2003 11:00 AM
To: wxPython-users@lists.wxwindows.org
Subject: Re: [wxPython-users] use zipfile and wxImageFromStream instead
of img2py

Kevin Altis wrote:

>You probably just want sys.argv[0], in which case you can store it away
>early in your main module.
>
>import os, sys
>print os.path.abspath(sys.argv[0])
>
>If you need something that is relative to the __file__ attribute of a
>module, then again, early in your main module, change sys.path[0] to be
>absolute and then all the __file__ attributes will end up
absolute. This was
>a recent topic of discussion on python-dev. It has been patched
for Python
>2.3, but the code below will essentially have the same effect.
>
>import sys
>sys.path[0] = os.path.abspath(sys.path[0])
>
>Does that solve the problem?
>

Not entirely (well, it doesn't feel particularly "clean" to me), as it
requires that the resource-set be application-specific (every
application needs to copy the .zip from every library it uses to it's
root (or whatever) directory). Would be really nice to have that
inclusion and updating automated. Also doesn't deal with needing the
root all over the app/package (i.e. you'd want a module somewhere to do
the loading of the .zip).

Example:

wxoo has, at present, 10 icon resources. Those are used all over wxoo
and wxprop. Each point in wxoo and wxprop that uses a zipfile-based
system needs access to the "root" of the application directory. It then
computes the filename root + "wxoo.zip", and your py2exe command then
needs to include that .zip in its data-files set.

ConflictSolver (which uses wxoo) has 21 icon resources (at the moment),
plus a number of non-icon resources. Again, these resources are used all
over the application, and every point a resource is used needs to know
how to get to the resource archive for ConflictSolver.

ConflictSolver is normally (during development) run with
conflictsolver/ui/application.py . Py2exe makes the result
conflictsolver/conflictsolver.exe, so os.path.abspath(sys.path[0]) (root
directory) is changing from conflictsolver/ to conflictsolver/ui
depending on whether it's the py2exe'd version or not, so you'd want to
be able to fix that to not need multiple .zip files.

With resourcepackage, I create the two directories, copy the __init__.py
file into them, and can then import the resource modules. When I
compose multiple packages, they are all able to access their resources
w/out any work in the application that's using them (e.g. don't need to
copy the .zip to the application directory). The run-time code has no
dependencies on the scanning __init__ mechanism or resourcepackage, it's
just modules with raw Python code (or wxPython code, depending on the
generators chosen).

>The scanning and packaging code for creating and/or updating a zip would
>still be useful and that could automatically be called as part
of a py2exe
>setup.py script. Gordon McMillan's Installer would be a bit
trickier since
>it uses a .spec file.
>
>Either the zipfile itself could be used as the manifest for the code that
>updates all files in the zip or a separate manifest could be
used, but using
>the zip seems to make the most sense. It just means you have to
create the
>initial zip with the hierarchy and filenames you want and after that a
>simple Python script reads the zip and updates any files that
have changed.
>
>
...

The problems are all solvable, no question, but as of yet I think I'll
stick with python-module-embedding via resourcepackage (I just finished
switching wxpypropdist and conflictsolver to it). I'll be happy to
switch to zip-mediated versions when it seems more convenient :slight_smile: . I'd
even be willing to help with it if the approach sounds like it'd be more
convenient/reliable than what I've got now.

BTW, if the zip packaging is only called during py2exe, are you
including code somewhere to load directly from disk if running w/out
py2exe-ing? I'd suggest it's _much_ nicer to have the same mechanism
used all the time.

Enjoy,
Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: wxPython-users-unsubscribe@lists.wxwindows.org
For additional commands, e-mail: wxPython-users-help@lists.wxwindows.org

Kevin Altis wrote:

I think you are making it more complicated than it needs to be. Yes, it may
require a refactor of your current resource handling strategy, but it seems
like that is worth doing to improve usability.

...

How does your approach improve usability? And for whom (certainly not me; I'm at the point where I do almost 0 total work for a new resource, just save it to the resources directory and then import it in my code, and total work to create a resourcepackage is copying a single file to a new directory)? Do you mean that the end-user is going to notice this stuff? Is it because you want them to be able to replace the files (which is a valid argument, just not a huge concern for me yet, (though ideologically I like the idea of providing this ability)).

The main point of all this is to simply get away from sticking images, text,
html, xml, or any other support data your application needs in Python simply
for the sake of being able to use Python imports.

Where the zip goes can be module relative or application relative, it is up
to you, it will work either way regardless of whether you are using a
standalone or in development.

ka

...
You haven't explained _why_ you want to move away from embedding data. There are some reasons:

    * less bulky in the packages (biggest problem I see)
    * naming problems (e.g. you can only use proper Python names for
      files in resourcepackage, and in the current code, there's no
      differentiation between icon.txt, icon.gif and icon.jpg (it will
      raise an error if you create both in a package). That's easily
      fixed, but it requires a slightly different naming scheme that
      does something like: "from wxoo.resources import up_jpg,
      down_png", which I consider rather ugly compared to "from
      wxoo.resources import up, down").
    * you want users to be able to replace files at run-time in py2exe'd
      configurations (not easily done with resourcepackage without
      package-specific mucking-about to exclude a top-level package from
      the py2exe code archive). I may just add code to mitigate this
      problem for resource-package, it'd be a fairly trivial
      enhancement, basically a py2exe-specific __init__.py that
      interacts with a modified extract.py so that it scans a run-time
      directory or .zip-file for updates (and then we would both be
      using basically the same code base :wink: ).

but in return you've got to either:

    * manually manage the .zip files _or_
    * write a package that will automate the process of bundling all
      included resources from all projects o each client project has to invoke your bundling process
            explicitly, and all packages must use the same process to
            ahieve automation (which strongly argues that this needs to
            be a (standard) distributable package)
          o this package is now a run-time dependency of your
            application (not a big deal, but another required install
            for Python-only installations)
          o write code to coordinate the .zip namespace (so that two
            packages don't both choose "resources.zip" as the name of
            their resources and wind up with one resource-set missing),
            or use a single .zip and use file prefixes or the like to
            keep package's resources seperate.

BTW, you (we?) might want to consider embedding the resource .zip file directly into the py2exe archive in some cases (which is what py2exe is doing anyway with the Python-code .zip).

I'm not yet convinced to switch to your approach (seems like lots more work), and you're not convinced of mine, so we can both continue to be happy with our own approaches :slight_smile: . Anyway, I may wind up adding support for something like your approach just to support run-time replacement of files :slight_smile: .

I can likely create a mechanism to support automated .zip management on top of resourcepackage which would work fine _as long as everyone needing resource handling of this type were using it_ (which is somewhat onerous :frowning: , it's not something that's required by resourcepackage at the moment ). We'll see how I feel in the morning :slight_smile: .

Enjoy,
Mike

Kevin Altis wrote:

You probably just want sys.argv[0], in which case you can store it away
early in your main module.

import os, sys
print os.path.abspath(sys.argv[0])

This is not reliable on *nix systems. sys.argv[0] could be a soft link,
and not the saem place as averything else. I've seen a number of
discussions about this, and it appears to be next to impossible to an
application to know where it's executable is stored. The way this is
handled in apps for *nix that need to know where their whole install
hierarchy is a start-up script that sets an enviroment variable tha the
app can access.

On the whole: what is the downside of storing resources in a *.py (or
*.pyc) file? is file size the only downside? that is very rarely an
issue for me, and doing it with python code seems particularly elegant
and nifty to me.

By the way, I'm glad to see folks working on this. I've had it in mind
to write some kind of resource storage and access module and never
gotten around to it.

-Chris

···

--
Christopher Barker, Ph.D.
Oceanographer
                                        
NOAA/OR&R/HAZMAT (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov