Headers for wxPython 2.6

Hi,

Does anyone have any comments to make on the following,
with a view that I'll probably commit this change if
no one can find anything wrong with it.

thanks!
Ron

···

On Thu, Oct 13, 2005 at 04:53:23PM -0500, Ken McIvor wrote:

Ron,

I believe I have identified the source of the problem with the headers
being installed in the source directory under Python 2.6, and have attached
a patch which ought to fix things. This is an upstream bug, so I will also
file that half of the patch at sourceforge and post to wxpython-dev.

As I suspected, the problem was caused by some cleverness in the Python
build scripts, specifically that `wxPython/config.py' provides a special
implementation of setup.py's install_headers command. This version
attempts to do the Right Thing by installing the headers off of the prefix
returned by
"wx-config --prefix". If you were building wxPython after having installed
wxWidgets, the prefix would be `/usr' and the headers would be installed
correctly.

However, setup.py uses the build version of "wx-config" when building the
Debian package. In this case, the prefix is the location of the source
tree, which is why the headers get installed there. This approach would
also break if you were trying to do a home directory installation of
wxPython, something that setup.py explicitly supports. Please let me know
if this doesn't make any sense, and I'll send an example which walks
through the code and demonstrates the problem.

The big problem with this approach is that there is no way to turn off the
cleverness: it ignores the --install-headers option, which is supposed to
let you specify a directory to put the headers in. I changed the
implementation of install_headers to respect this option again, and updated
`debian/rules' to use it.

I also added the "--no-compile" option to the invocations of "setup.py
install", which turns off automatically byte-compiling the Python files.
This makes more sense to me because lets you avoid byte-compiling the files
twice (when running "setup.py install" and in the postinst). However, this
is not an important part of the patch.

Ken

diff -ur wxwidgets2.6-2.6.1.2.orig/debian/rules wxwidgets2.6-2.6.1.2/debian/rules
--- wxwidgets2.6-2.6.1.2.orig/debian/rules 2005-08-25 07:54:17.000000000 -0500
+++ wxwidgets2.6-2.6.1.2/debian/rules 2005-10-13 16:08:16.000000000 -0500
@@ -94,6 +94,10 @@
     $(objdir_msw_shared) $(objdir_msw_static) $(objdir_msw_dbg) \
     $(objdir_msw_install)

+# setup.py's --prefix option
+pyprefix := `pwd`/../debian/$(package_gtk_py_lib)/usr
+pydprefix := `pwd`/../debian/$(package_gtk_dbg_py)/usr
+
# note that the i18n package is actually arch indep (once built)
# but must be built (and installed) during the arch any phase as
# it's pulled out of the wxGTK shared lib package.
@@ -517,7 +521,9 @@
   dh_installdirs usr/lib/wx/python
   cd wxPython \
     && $(python_ver) ./setup.py install \
- --prefix=`pwd`/../debian/$(package_gtk_py_lib)/usr \
+ --prefix=$(pyprefix) \
+ --install-headers=$(pyprefix)/include/wx-$(release)/wx \
+ --no-compile \
             WX_CONFIG='$(wxconfig)' \
             SYS_WX_CONFIG='$(py_wxconfig)' \
             WXPORT=$(pytoolkit) \
@@ -625,7 +631,9 @@
   dh_installdirs usr/lib/wx/python
   cd wxPython \
     && $(python_ver) ./setup.py install \
- --prefix=`pwd`/../debian/$(package_gtk_dbg_py)/usr \
+ --prefix=$(pydprefix) \
+ --install-headers=$(pydprefix)/include/wx-$(release)/wx \
+ --no-compile \
             WX_CONFIG='$(wxconfig-dbg)' \
             SYS_WX_CONFIG='$(pyd_wxconfig)' \
             WXPORT=$(pytoolkit) \
diff -ur wxwidgets2.6-2.6.1.2.orig/wxPython/config.py wxwidgets2.6-2.6.1.2/wxPython/config.py
--- wxwidgets2.6-2.6.1.2.orig/wxPython/config.py 2005-08-25 07:53:28.000000000 -0500
+++ wxwidgets2.6-2.6.1.2/wxPython/config.py 2005-10-13 16:05:43.000000000 -0500
@@ -32,6 +32,7 @@
import distutils.command.install_headers
import distutils.command.clean

+
#----------------------------------------------------------------------
# flags and values that affect this script
#----------------------------------------------------------------------
@@ -487,8 +488,9 @@

class wx_install_headers(distutils.command.install_headers.install_headers):
     """
- Install the header files to the WXPREFIX, with an extra dir per
- filename too
+ Installs the header files off of WXPREFIX by default, but allow users to
+ specify an alternative path using the distutils command line options. Also
+ adds support for an extra directory per filename.
     """
     def initialize_options(self):
         self.root = None
@@ -506,18 +508,40 @@
             return

         root = self.root
- if root is None or WXPREFIX.startswith(root):
- root = ''
+ if not self.is_default_headers_install_dir(self.install_dir):
+ prefix = self.install_dir
+ else:
+ if root is None or WXPREFIX.startswith(root):
+ prefix = WXPREFIX
+ else:
+ prefix = root + WXPREFIX
+ prefix += '/include/wx-%d.%d/wx' % (VER_MAJOR, VER_MINOR)
+
         for header, location in headers:
- install_dir = os.path.normpath(root +
- WXPREFIX +
- '/include/wx-%d.%d/wx' % (VER_MAJOR, VER_MINOR) +
- location)
+ install_dir = os.path.normpath(prefix + location)
             self.mkpath(install_dir)
             (out, _) = self.copy_file(header, install_dir)
             self.outfiles.append(out)

+ def is_default_headers_install_dir(self, path):
+ """
+ Tests if a path is the default header install directory under unix.
+ """
+ from distutils.util import subst_vars
+ from distutils.command.install import INSTALL_SCHEMES
+ install_cmd = self.get_finalized_command('install')
+ def check(scheme):
+ return (os.path.abspath(path) ==
+ os.path.abspath(
+ os.path.expanduser(
+ subst_vars(
+ INSTALL_SCHEMES[scheme][ 'headers'],
+ install_cmd.config_vars))))

+ if install_cmd.home:
+ return check('unix_home')
+ else:
+ return check('unix_prefix')

def build_locale_dir(destdir, verbose=1):

Hi,

Ken, now that Robin is back, I'm going to punt the decision
about the best way to handle this to him :wink: I would like
to get this fix in (somehow) for 2.6.3 though, and backport
the relevant changes to 2.4 also once what needs to be done
is clear.

Robin, if you need more context, please ask. Ken seems
more versed in the details of this than I am still.

Thanks,
Ron

···

On Fri, Oct 14, 2005 at 10:52:59AM +0930, Ron wrote:

Hi,

Does anyone have any comments to make on the following,
with a view that I'll probably commit this change if
no one can find anything wrong with it.

thanks!
Ron

On Thu, Oct 13, 2005 at 04:53:23PM -0500, Ken McIvor wrote:
> Ron,
>
> I believe I have identified the source of the problem with the headers
> being installed in the source directory under Python 2.6, and have attached
> a patch which ought to fix things. This is an upstream bug, so I will also
> file that half of the patch at sourceforge and post to wxpython-dev.
>
> As I suspected, the problem was caused by some cleverness in the Python
> build scripts, specifically that `wxPython/config.py' provides a special
> implementation of setup.py's install_headers command. This version
> attempts to do the Right Thing by installing the headers off of the prefix
> returned by
> "wx-config --prefix". If you were building wxPython after having installed
> wxWidgets, the prefix would be `/usr' and the headers would be installed
> correctly.
>
> However, setup.py uses the build version of "wx-config" when building the
> Debian package. In this case, the prefix is the location of the source
> tree, which is why the headers get installed there. This approach would
> also break if you were trying to do a home directory installation of
> wxPython, something that setup.py explicitly supports. Please let me know
> if this doesn't make any sense, and I'll send an example which walks
> through the code and demonstrates the problem.
>
> The big problem with this approach is that there is no way to turn off the
> cleverness: it ignores the --install-headers option, which is supposed to
> let you specify a directory to put the headers in. I changed the
> implementation of install_headers to respect this option again, and updated
> `debian/rules' to use it.
>
> I also added the "--no-compile" option to the invocations of "setup.py
> install", which turns off automatically byte-compiling the Python files.
> This makes more sense to me because lets you avoid byte-compiling the files
> twice (when running "setup.py install" and in the postinst). However, this
> is not an important part of the patch.
>
> Ken

> diff -ur wxwidgets2.6-2.6.1.2.orig/debian/rules wxwidgets2.6-2.6.1.2/debian/rules
> --- wxwidgets2.6-2.6.1.2.orig/debian/rules 2005-08-25 07:54:17.000000000 -0500
> +++ wxwidgets2.6-2.6.1.2/debian/rules 2005-10-13 16:08:16.000000000 -0500
> @@ -94,6 +94,10 @@
> $(objdir_msw_shared) $(objdir_msw_static) $(objdir_msw_dbg) \
> $(objdir_msw_install)
>
> +# setup.py's --prefix option
> +pyprefix := `pwd`/../debian/$(package_gtk_py_lib)/usr
> +pydprefix := `pwd`/../debian/$(package_gtk_dbg_py)/usr
> +
> # note that the i18n package is actually arch indep (once built)
> # but must be built (and installed) during the arch any phase as
> # it's pulled out of the wxGTK shared lib package.
> @@ -517,7 +521,9 @@
> dh_installdirs usr/lib/wx/python
> cd wxPython \
> && $(python_ver) ./setup.py install \
> - --prefix=`pwd`/../debian/$(package_gtk_py_lib)/usr \
> + --prefix=$(pyprefix) \
> + --install-headers=$(pyprefix)/include/wx-$(release)/wx \
> + --no-compile \
> WX_CONFIG='$(wxconfig)' \
> SYS_WX_CONFIG='$(py_wxconfig)' \
> WXPORT=$(pytoolkit) \
> @@ -625,7 +631,9 @@
> dh_installdirs usr/lib/wx/python
> cd wxPython \
> && $(python_ver) ./setup.py install \
> - --prefix=`pwd`/../debian/$(package_gtk_dbg_py)/usr \
> + --prefix=$(pydprefix) \
> + --install-headers=$(pydprefix)/include/wx-$(release)/wx \
> + --no-compile \
> WX_CONFIG='$(wxconfig-dbg)' \
> SYS_WX_CONFIG='$(pyd_wxconfig)' \
> WXPORT=$(pytoolkit) \
> diff -ur wxwidgets2.6-2.6.1.2.orig/wxPython/config.py wxwidgets2.6-2.6.1.2/wxPython/config.py
> --- wxwidgets2.6-2.6.1.2.orig/wxPython/config.py 2005-08-25 07:53:28.000000000 -0500
> +++ wxwidgets2.6-2.6.1.2/wxPython/config.py 2005-10-13 16:05:43.000000000 -0500
> @@ -32,6 +32,7 @@
> import distutils.command.install_headers
> import distutils.command.clean
>
> +
> #----------------------------------------------------------------------
> # flags and values that affect this script
> #----------------------------------------------------------------------
> @@ -487,8 +488,9 @@
>
> class wx_install_headers(distutils.command.install_headers.install_headers):
> """
> - Install the header files to the WXPREFIX, with an extra dir per
> - filename too
> + Installs the header files off of WXPREFIX by default, but allow users to
> + specify an alternative path using the distutils command line options. Also
> + adds support for an extra directory per filename.
> """
> def initialize_options(self):
> self.root = None
> @@ -506,18 +508,40 @@
> return
>
> root = self.root
> - if root is None or WXPREFIX.startswith(root):
> - root = ''
> + if not self.is_default_headers_install_dir(self.install_dir):
> + prefix = self.install_dir
> + else:
> + if root is None or WXPREFIX.startswith(root):
> + prefix = WXPREFIX
> + else:
> + prefix = root + WXPREFIX
> + prefix += '/include/wx-%d.%d/wx' % (VER_MAJOR, VER_MINOR)
> +
> for header, location in headers:
> - install_dir = os.path.normpath(root +
> - WXPREFIX +
> - '/include/wx-%d.%d/wx' % (VER_MAJOR, VER_MINOR) +
> - location)
> + install_dir = os.path.normpath(prefix + location)
> self.mkpath(install_dir)
> (out, _) = self.copy_file(header, install_dir)
> self.outfiles.append(out)
>
> + def is_default_headers_install_dir(self, path):
> + """
> + Tests if a path is the default header install directory under unix.
> + """
> + from distutils.util import subst_vars
> + from distutils.command.install import INSTALL_SCHEMES
> + install_cmd = self.get_finalized_command('install')
> + def check(scheme):
> + return (os.path.abspath(path) ==
> + os.path.abspath(
> + os.path.expanduser(
> + subst_vars(
> + INSTALL_SCHEMES[scheme][ 'headers'],
> + install_cmd.config_vars))))
>
> + if install_cmd.home:
> + return check('unix_home')
> + else:
> + return check('unix_prefix')
>
>
> def build_locale_dir(destdir, verbose=1):

---------------------------------------------------------------------
To unsubscribe, e-mail: wxPython-dev-unsubscribe@lists.wxwidgets.org
For additional commands, e-mail: wxPython-dev-help@lists.wxwidgets.org

Please see my comment at

https://sourceforge.net/tracker/index.php?func=detail&aid=1326241&group_id=10718&atid=310718

Ron wrote:

···

Hi,

Ken, now that Robin is back, I'm going to punt the decision
about the best way to handle this to him :wink: I would like
to get this fix in (somehow) for 2.6.3 though, and backport
the relevant changes to 2.4 also once what needs to be done
is clear.

Robin, if you need more context, please ask. Ken seems
more versed in the details of this than I am still.

Thanks,
Ron

On Fri, Oct 14, 2005 at 10:52:59AM +0930, Ron wrote:

Hi,

Does anyone have any comments to make on the following,
with a view that I'll probably commit this change if
no one can find anything wrong with it.

thanks!
Ron

On Thu, Oct 13, 2005 at 04:53:23PM -0500, Ken McIvor wrote:

Ron,

I believe I have identified the source of the problem with the headers being installed in the source directory under Python 2.6, and have attached a patch which ought to fix things. This is an upstream bug, so I will also file that half of the patch at sourceforge and post to wxpython-dev.

As I suspected, the problem was caused by some cleverness in the Python build scripts, specifically that `wxPython/config.py' provides a special implementation of setup.py's install_headers command. This version attempts to do the Right Thing by installing the headers off of the prefix returned by
"wx-config --prefix". If you were building wxPython after having installed wxWidgets, the prefix would be `/usr' and the headers would be installed correctly.

However, setup.py uses the build version of "wx-config" when building the Debian package. In this case, the prefix is the location of the source tree, which is why the headers get installed there. This approach would also break if you were trying to do a home directory installation of wxPython, something that setup.py explicitly supports. Please let me know if this doesn't make any sense, and I'll send an example which walks through the code and demonstrates the problem.

The big problem with this approach is that there is no way to turn off the cleverness: it ignores the --install-headers option, which is supposed to let you specify a directory to put the headers in. I changed the implementation of install_headers to respect this option again, and updated `debian/rules' to use it.

I also added the "--no-compile" option to the invocations of "setup.py install", which turns off automatically byte-compiling the Python files. This makes more sense to me because lets you avoid byte-compiling the files twice (when running "setup.py install" and in the postinst). However, this is not an important part of the patch.

Ken

diff -ur wxwidgets2.6-2.6.1.2.orig/debian/rules wxwidgets2.6-2.6.1.2/debian/rules
--- wxwidgets2.6-2.6.1.2.orig/debian/rules 2005-08-25 07:54:17.000000000 -0500
+++ wxwidgets2.6-2.6.1.2/debian/rules 2005-10-13 16:08:16.000000000 -0500
@@ -94,6 +94,10 @@
    $(objdir_msw_shared) $(objdir_msw_static) $(objdir_msw_dbg) \
    $(objdir_msw_install)

+# setup.py's --prefix option
+pyprefix := `pwd`/../debian/$(package_gtk_py_lib)/usr
+pydprefix := `pwd`/../debian/$(package_gtk_dbg_py)/usr
+
# note that the i18n package is actually arch indep (once built)
# but must be built (and installed) during the arch any phase as
# it's pulled out of the wxGTK shared lib package.
@@ -517,7 +521,9 @@
  dh_installdirs usr/lib/wx/python
  cd wxPython \
    && $(python_ver) ./setup.py install \
- --prefix=`pwd`/../debian/$(package_gtk_py_lib)/usr \
+ --prefix=$(pyprefix) \
+ --install-headers=$(pyprefix)/include/wx-$(release)/wx \
+ --no-compile \
            WX_CONFIG='$(wxconfig)' \
            SYS_WX_CONFIG='$(py_wxconfig)' \
            WXPORT=$(pytoolkit) \
@@ -625,7 +631,9 @@
  dh_installdirs usr/lib/wx/python
  cd wxPython \
    && $(python_ver) ./setup.py install \
- --prefix=`pwd`/../debian/$(package_gtk_dbg_py)/usr \
+ --prefix=$(pydprefix) \
+ --install-headers=$(pydprefix)/include/wx-$(release)/wx \
+ --no-compile \
            WX_CONFIG='$(wxconfig-dbg)' \
            SYS_WX_CONFIG='$(pyd_wxconfig)' \
            WXPORT=$(pytoolkit) \
diff -ur wxwidgets2.6-2.6.1.2.orig/wxPython/config.py wxwidgets2.6-2.6.1.2/wxPython/config.py
--- wxwidgets2.6-2.6.1.2.orig/wxPython/config.py 2005-08-25 07:53:28.000000000 -0500
+++ wxwidgets2.6-2.6.1.2/wxPython/config.py 2005-10-13 16:05:43.000000000 -0500
@@ -32,6 +32,7 @@
import distutils.command.install_headers
import distutils.command.clean

+
#----------------------------------------------------------------------
# flags and values that affect this script
#----------------------------------------------------------------------
@@ -487,8 +488,9 @@

class wx_install_headers(distutils.command.install_headers.install_headers):
    """
- Install the header files to the WXPREFIX, with an extra dir per
- filename too
+ Installs the header files off of WXPREFIX by default, but allow users to
+ specify an alternative path using the distutils command line options. Also
+ adds support for an extra directory per filename.
    """
    def initialize_options(self):
        self.root = None
@@ -506,18 +508,40 @@
            return

        root = self.root
- if root is None or WXPREFIX.startswith(root):
- root = ''
+ if not self.is_default_headers_install_dir(self.install_dir):
+ prefix = self.install_dir
+ else:
+ if root is None or WXPREFIX.startswith(root):
+ prefix = WXPREFIX
+ else:
+ prefix = root + WXPREFIX
+ prefix += '/include/wx-%d.%d/wx' % (VER_MAJOR, VER_MINOR)
+
        for header, location in headers:
- install_dir = os.path.normpath(root +
- WXPREFIX +
- '/include/wx-%d.%d/wx' % (VER_MAJOR, VER_MINOR) +
- location)
+ install_dir = os.path.normpath(prefix + location)
            self.mkpath(install_dir)
            (out, _) = self.copy_file(header, install_dir)
            self.outfiles.append(out)

+ def is_default_headers_install_dir(self, path):
+ """
+ Tests if a path is the default header install directory under unix.
+ """
+ from distutils.util import subst_vars
+ from distutils.command.install import INSTALL_SCHEMES
+ install_cmd = self.get_finalized_command('install')
+ def check(scheme):
+ return (os.path.abspath(path) ==
+ os.path.abspath(
+ os.path.expanduser(
+ subst_vars(
+ INSTALL_SCHEMES[scheme][ 'headers'],
+ install_cmd.config_vars))))

+ if install_cmd.home:
+ return check('unix_home')
+ else:
+ return check('unix_prefix')

def build_locale_dir(destdir, verbose=1):

---------------------------------------------------------------------
To unsubscribe, e-mail: wxPython-dev-unsubscribe@lists.wxwidgets.org
For additional commands, e-mail: wxPython-dev-help@lists.wxwidgets.org

---------------------------------------------------------------------
To unsubscribe, e-mail: wxPython-dev-unsubscribe@lists.wxwidgets.org
For additional commands, e-mail: wxPython-dev-help@lists.wxwidgets.org

Ok. I guess we should start from the top then...

The initial problem began with someone noticing that the
.debs have never supplied the bits required to build wxPy
extension modules. It appears that we somehow should do
this.

On closer inspection, a couple of weird things were seen.
Some or all of what is required for this is copied to the
root of the source tree during the build process, and none
of it is subsequently installed to the (temporary) 'system'
directories, along with the rest of wxPython, when the
`setup.py install` script is run.

I'm open to advice on whether these files should be distributed
with the main wxpython package, or in a separate package for
only those who really need them -- but I'm not really clear
on what the best way to get setup.py to install them to the
correct place really is in the first place.

As mentioned elsewhere previously, I added SYS_WX_CONFIG to
avoid build_options being created using the wrong WXCONFIG,
but I don't know if something like that is what is needed here.

I need to be able to build wxPython using 'uninstalled' wx libs
(for which I provide a suitable wx-config) and then install both
of those to a temporary tree, configured in such a way that they
will work as expected when later installed to the vendor tree of
the users os.

That works fine for most things, but I'm not sure what crack
these extra bits fall into, or the best way to get them out.
Or even what the existing logic is for where and when to install
them. I was blissfully unaware of them myself until recently,
so I can only really act on advice at this stage.

cheers,
Ron

  (who believes there is an ubuntu release looming that it
   might be nice to catch if it is not already too late)

Please see my comment at

https://sourceforge.net/tracker/index.php?func=detail&aid=1326241&group_id=10718&atid=310718

Which, just to keep it all together, was:

Unfortunatly this change breaks my installs for the RPMs.
The is_default_headers_install_dir function will not return
True if --root is used and so the WXPREFIX version is not used.

···

On Tue, Dec 20, 2005 at 03:36:12PM -0800, Robin Dunn wrote:

Ron wrote:
>Hi,
>
>Ken, now that Robin is back, I'm going to punt the decision
>about the best way to handle this to him :wink: I would like
>to get this fix in (somehow) for 2.6.3 though, and backport
>the relevant changes to 2.4 also once what needs to be done
>is clear.
>
>Robin, if you need more context, please ask. Ken seems
>more versed in the details of this than I am still.
>
>Thanks,
>Ron

Recap: I originally suggested that you change `debian/rules' to explicitly copy the wxPython headers into the correct place in the build directory. At your request, I filed a patch upstream to try to fix the problem within setup.py itself. Robin has indicated that my patch breaks his RPM builds because it fails to correctly account for the --root argument to setup.py.

I think that changing `debian/rules' is probably the most robust solution at this point. That being said, I will be happy to work on another patch if that's what you and Robin think is best.

Ken

···

On Dec 20, 2005, at 8:46 PM, Ron wrote:

That works fine for most things, but I'm not sure what crack
these extra bits fall into, or the best way to get them out.
Or even what the existing logic is for where and when to install
them. I was blissfully unaware of them myself until recently,
so I can only really act on advice at this stage.

>That works fine for most things, but I'm not sure what crack
>these extra bits fall into, or the best way to get them out.
>Or even what the existing logic is for where and when to install
>them. I was blissfully unaware of them myself until recently,
>so I can only really act on advice at this stage.

Recap: I originally suggested that you change `debian/rules' to
explicitly copy the wxPython headers into the correct place in the
build directory. At your request, I filed a patch upstream to try to
fix the problem within setup.py itself. Robin has indicated that my
patch breaks his RPM builds because it fails to correctly account for
the --root argument to setup.py.

I don't recall how much of the early discussion was forwarded here,
the originally salient bits may have been just a private exchange...

I think that changing `debian/rules' is probably the most robust
solution at this point. That being said, I will be happy to work on
another patch if that's what you and Robin think is best.

I think we'll have to. There is no legitimate tweak to /rules that
we can make, and implementing what should be a function of setup.py
there will be anything but robust. setup.py _is_ installing the
files, but to the wrong place -- that is a bug and needs to be fixed
somehow. When that is fixed, the rest of this will 'just work' as
expected.

I'd rather wait until someone who really needs this figures out
what is currently wrong with it, than hide the bugs under a mat
of brittle hacks where no-one will find them again until it is
too late... I don't know what the rpms are doing or why this
is incompatible with them, but we _clearly_ need to find something
that does works as expected everywhere, as there is an assumption
somewhere in setup.py that is going pear-shaped here.

I'm not really the best person to say exactly which and why, but
I do know that manually copying what `setup.py install` should
handle is not the right answer by a fairly long shot.

Someone more savvy with distutils than me is going to have to
haggle with Robin over the finer details though...

cheers,
Ron

(oh, and I get wxpy-dev, no need to cc' me on these)

···

On Thu, Dec 29, 2005 at 02:47:56PM -0600, Ken McIvor wrote:

On Dec 20, 2005, at 8:46 PM, Ron wrote: