Linux wheels with manylinux2010

wxPython Dev
#1

Sorry, I know Linux wheels has come up a few times, and I know that using manylinux had failed in the past (although I don’t think I’ve seen discussion of what exactly caused the issue).

Just wanted to flag that the manylinux has now been updated to be based on a newer version of Centos https://discuss.python.org/t/manylinux2010-docker-image-now-available

That has made it possible to build some packages that weren’t previously possible and maybe it’s worth trying again to build a phoneix wheel with that?

#2

Yep, I’m looking into it.

The main problem with manylinux1 was that the spec and the reference platform did not include, or include new enough, required libraries that are dependencies of wxWidgets. I was able to hack around things a bit, breaking some compatibility with the reference platform, and also include some extra libs in the wheel not included in the spec, which spec does allow, but doing that with gtk and gtk dependents resulted in them not being able to use things like system themes and probably other non-library parts of GTK. The result did run on newer linuxes, but was severely limited, was gtk2-only, looked butt-ugly without access to themes, almost all of the optional features had to be turned off, and the size of the wheel was huge because of the need to include so many 3rd party libs. I expect that there would be license issues too because of including those libs.

I felt that there would be much less complaints about the need to build it yourself than there would be from releasing a linux wheel with those problems. Hopefully manylinx2010 will be better.

1 Like
#3

But it being based on CentOS 6 makes me think that it may not be better. CentOS 7 is probably the minimum needed.

#4

OK, well I’ll be interested to hear if you make any progress there. I was working on a project that needed XInput2 and the step to Centos6 was enough to get that one over the line, so I was hoping it would help wx too

#5

There is now a manylinux2014 built on Centos7 and it looks like they’re adding newer ones based on debian versions:

It would sure be great to get linux wheels fixed for wxPython. It’s still always a pain point for our (PsychoPy’s) ubuntu users

#6

Revisiting this and have made some progress. Since it was last discussed:

  • there is now a manylinux2014 (based on Centos 7)
  • there is now a cibuildwheel available pretty easily on GitHub Actions that makes it pretty easy to use manylinux without handling docker environments directly

With the new tools I’ve got quite far (I think). So far I’m just trying this for Py3.8 but the system should extend up to any versions that are supported, as well as handling win/mac builds all still using cibuildwheel. The c-compiling steps all now seem to run, but the building the wheel isn’t working due to python packaging.

You can see logs of the builds here
Actions · peircej/wxpython_buildwheels · GitHub
I’m still tweaking the dependency configs - it’s currently failing because sip/siplib license isn’t where waf expects it to be :cry: but I hope this can be made to work and anyone with time to help debug the build config would be welcome to join in!

#7

Ah, the current issue, with failing to find sip/siplib/license.txt is that that file was deleted in this commit:
Update to use sip 6.6.x · wxWidgets/Phoenix@90171ba · GitHub

We either need to add sip/siplib/license.txt back in or we need to remove the line in updateLicenseFiles that requires it

#8

Yep, looks like sip-module isn’t including the license files in the sdist it generates. I reported that to upstream sip.

#9

Cool, in the meantime I’m testing from a separate copy of Phoenix with tweaked updateLicenseFiles to see what the next barrier will be! :laughing:

#10

It seems quite a bit of the build architecture is expecting sip and siplib to be within this repo so when that got removed the build system broke. When I fix the license issue above, we get hit with errors about missing sip/siplib.sip.h but I can see in wscript (which is called during build) there are lots of references to the siplib files, all hard-coded to this location

The conversation around the commit suggests that builds have successfully completed on other systems though so I’m puzzled how that was possible! :thinking:

#11

@peircej, yeah the sip/siplib files are removed from the repository, but they are now being automatically generated by this step:

BTW, the issue with the license files being missing seems to be an issue with older versions of setuptools.

There was also an issue recently with some of the sip files not being included in the snapshots, see: Fix inclusion of siplib and sip.h in sdist with sip 6.6.2 · wxWidgets/Phoenix@3a20dd4 · GitHub

#12

I took a quick look at your log files. Please note this warning:

    running bdist_wheel
    running build
    WARNING: Building this way assumes that all generated files have been
    generated already.  If that is not the case then use build.py directly
    to generate the source and perform the build stage.  You can use
    --skip-build with the bdist_* or install commands to avoid this
    message and the wxWidgets and Phoenix build steps in the future.

Basically, you can’t use bdist_wheel without generating all of the generated files first. You might want to take a look at how the project’s CI builds its wheels.

#13

So (taking this from buildbot/master.cfg) there need to be the following commands preceding pip build bdist_wheel?

python build.py build_wx --gtk3
python build.py dox etg sip --gtk3
python build.py build_py --gtk3

Is that what creates the files you mean? Or is this more/less than needed?

#14

Apologies for the slightly dumb questions - I’ve never worked with a 2000-line multi-step build script before! :laughing:

#15

Yes. That should get you what you need. Note that you don’t need --gtk3
anymore as that’s the default.

You may wish to refer to the Azure CI configuration also:

#16

You should be able to skip the build_py command, as bdist_wheel will do that step too. (Although it’s been a while since I tested it that way…) The build.py script can set up and launch the bdist_weel command too, so if desired you can do it all on one command-line:

python build.py etg sip build_wx bdist_wheel

You should probably also toss in a --no_doc to save some time by skipping the generation of the Sphinx docs. And while develooping/testing, once you’ve generated the code one time, you don’t need to run the etg or sip steps again unless you clean the build folders.

#17

Thanks both. I’ve made some progress.

Using:

python build.py build_wx
python build.py dox etg --nodoc sip
python build.py build_py

the build now completes but using the wheel in a separate test environment fails to import wx with a segfault. You can see the full logs for that fail here:
Move all build scripts into Phoenix project · peircej/wxpython_buildwheels@9e9c785 · GitHub

@Robin this is on a GitHub VM so it’s a clean environment every time. I’m sure there would be ways to cache intermediate files but I haven’t had time to look into that. FYI the build process is taking around 40 mins (to build the Py3.8 wheel for manylinux flavours)

Note that cibuildwheel, which I used here to provide the support for manylinux, does also run on Azure pipelines, so you wouldn’t need to change much of your current workflow to get the many linux flavours in one go. I just used Github Actions to test the principle because I know it better from my own projects

#18

Give this a try:

It’s not based on cibuildwheel but I thought I would give this approach a try when I was playing around with it last night. It came together much easier than the last time I attempted manylinux.

1 Like
#19

Sounds great @Robin

I was in no way wedded to using cibuildwheel - i just saw that as the easiest way to get it moving. If you’ve got a build working using an even newer variant of manylinux that’s great. If you want me to tyre-kick the resulting wheel then let me know! :slight_smile:

#20

I’ve put a set of tires to be kicked here. Have fun.

1 Like