building from source with PyPy

wxPython Dev
#1

Hi. I am a PyPy dev. Today I tried building wxPython Phoenix using PyPy, and would like to share my experience .

It took me a while to find the github repo, the wiki and web site do not seem to mention it AFAICT. In fact, the buildbot instructions here https://wiki.wxpython.org/ProjectPhoenix/BuildbotSlave tricked me into thinking that SVN was still the preferred method of source control for the project. But once I found the github repo https://github.com/wxWidgets/Phoenix everything became easier.

Since I am using a dev version of PyPy (not packaged by the Debian maintainer, rather the nightly HEAD), I needed to roll my own pypy-config based on the python27-config script.

Of course, the build immediately crashed since our C-API compatibility is not yet good enough. We seem to have some badly declared function signatures that do not match CPython, I will be fixing these over the next few weeks, and may need to submit patches or fixes for too heavy-duty use of c pointers that I cannot fix on the PyPy side of things.

I would love to get the wxPython toolkit working under PyPY, so I hope there are not too many patches needed.

FWIW, we did try to use CFFI instead of the C-API a few years back during a Google Summer of Code, the results are languishing on https://bitbucket.org/waedt/wxpython_cffi.

So thanks for wxPython, and stay tuned, hopefully I can make some progress here

Matti

PyPy and wxPython
#2

Hi, could you repost the Google Summer of Code code. The above link no longer works.

#3

I resuscitated the repo from the internet archives and uploaded it to GitHub - pypy/wxpython-cffi: Experiment in getting SIP to emit cffi code

Please let me know if it helps. The original GSOC was 2013, the code has not been updated since 2015. At that time much of the code generation worked, I have no idea what the current situation is. There is probably work needed to make it run on python3.

#4

Thank you very much for this. Should I use the GitHub ticketing system to make requests, or should I do them here?

#5

If it is convenient for you I think github issues/PRs would be the most efficient. Feel free to ping me (@mattip) for any assistance needed.

#6

But once I found the github repo…

It has been there for a quite a few years now, and wxPython (Phoenix) is fully pip-installable (although not yet wheel-friendly on linux, see here). Also, you might want to take a look to the latest snapshot builds to get the most updated dev version.

I may need to submit patches or fixes for too heavy-duty use of c pointers

I probably don’t understand what you mean here, so you may find my answer too naive… You know that wxPython is (mostly) a wrapper around wxWidgets, which is built automatically with SIP… so I don’t think we do c pointers directly… The source of your troubles may turn up to be something in SIP or wxWidgets itself.

On top of that, I’m afraid Robin may be on hiatus again… it might take a little longer to get a response from the landlord here… but we’ll be glad to assist you for the very little we can!

#7

@ricpol my post was made 5 years ago in 2016. Since then PyPy has become much more stable and supports most of the C-API out of the box, although it might be much slower since PyPy needs to emulate the CPython object layout.

#8

ops! sorry, did’t check the date… good to know!

#9

Have you received any notifications for the issues made so far on Github yet?

#10

I have been studying the wxpython-cffi GitHub project for a couple of weeks now. It is very complex. First you need to build a bunch of xml files using Doxygen. Then you have to build wxWidgets. Then you have to use something called etg scripts to build SIP code. Then CFFI code is generated. Oh boy. I understand why progress has been so slow with this project. I think the best thing to do is to find a way to simplify things. Since CFFI requires making C code to call wxWidgets’ C++ code I think we should replace it with cppyy. cppyy is made to work with C++ code. What do you think?

#11

From PyPy’s perspective, I would not want a separate “PyPy wxPython version” that would require additional maintenance. While cppyy is certainly attractive, it would make the most sense to convince the wxPython project to use one solution. I don’t know how much work it would be to create a cppyy version of wxPython and how disruptive it would be in terms of user-facing interface changes.

Note that over the years wxPython has grown its own classes and features beyond wxWidgets. Ideally none of that code would need refactoring.

#12

Lets forget about cppyy. I spent 5+ hours waiting for cppyy to finish building. My patients ran out and I gave up on it.

While I was waiting for cppyy to build I researched all these options for accessing C++ code from Python:

  • ctypes
  • CFFI
  • pybind11
  • cython
  • pybindgen
  • Boost.python
  • Shiboken
  • SWIG
  • Scapix

After all my research I came to the conclusion that pybindgen is the best option. It comes with a tool that can read header files and auomatically make binds from it. So I change my vote to pybindgen. Here is its website: PyBindGen Tutorial — PyBindGen 0.17.0 documentation. It looks pretty easy to use.

As for the wxPython only classes, since they are written in python, they should work just fine in pypy.

#13

I understand that wxPython currently uses SIP. If keeping only one C++ to Python middleware is important then would it be possible for SIP to be ported over to PyPy?

#14

The advantage of CFFI and cppyy is that they are fast for PyPy and not slow for CPython. In the years since the wxPython-CFFI fork was started, PyPy has grown a capable C-API emulation layer. The other options should all JustWork with PyPy, including SIP, but will be slow. Does PyPy compile the standard wxPython code as-is or are there missing C-API functions?

#15

Well I haven’t had much luck with making wxPython work with PyPy so far. For the wxPython-CFFI project I could not use wxWidgets 2.9 that the project was made for so I had to go with wxWigets 3.15. wxWidgets does build but at the “build.py etg --generator=cffi --nodoc” step is when building wxPython-cffi fails. I have been trying to go around problems by commenting out problem code but I don’t think this is a winning strategy. I think it is time to try something else. From what you say about the C-API layer we should be able to run wxPython in PyPy now. I just have to figure how how to do this.

#16

Have you tried following the instructions for CPython https://github.com/wxWidgets/Phoenix/blob/64e5d863f7833f10df6a0fbcf3221a730562224b/README.rst, just using the PyPy interpreter instead?

#17

I tried building wxPython using this command “./build.py build --python /opt/local/bin/pypy”. The result was this error message:

Running command: build_py
Checking for wxPython-4.1.1/bin/waf-2.0.19…
“/opt/local/bin/pypy” /wxPython-4.1.1/bin/waf-2.0.19 --wx_config=/wxPython-4.1.1/build/wxbld/wx-config --python="/opt/local/bin/pypy" --out=build/waf/2.7 configure build
Waf: The wscript in ‘/wxPython-4.1.1’ is unreadable
Traceback (most recent call last):
File “/wxPython-4.1.1/bin/.waf-2.0.19-1f3c580272b15a03d2566843c5fe872a/waflib/Scripting.py”, line 102, in waf_entry_point
set_main_module(wscript)
File “/wxPython-4.1.1/bin/.waf-2.0.19-1f3c580272b15a03d2566843c5fe872a/waflib/Scripting.py”, line 142, in set_main_module
Context.g_module=Context.load_module(file_path)
File “/wxPython-4.1.1/bin/.waf-2.0.19-1f3c580272b15a03d2566843c5fe872a/waflib/Context.py”, line 362, in load_module
exec(compile(code,path,‘exec’),module.dict)
File “/wxPython-4.1.1/wscript”, line 12, in
import setuptools
ImportError: No module named setuptools

#18

After doing some work I was able to install setuptools and go past this error. The next errors looked like this:

Erroneous order constraint after=‘siplib’ on bld(source=‘siplib.pypy-73.so’, target=‘pkg.siplib.pypy-73’, meths=[‘check_err_features’, ‘check_err_order’, ‘process_rule’, ‘process_source’], features=[], path=/wxPython-4.1.1, idx=2, tg_idx_count=2, rule=<function copyFileToPkg at 0x00007fd2e05675b0>, after=‘siplib’, _name=‘pkg.siplib.pypy-73’, posted=True) in /wxPython-4.1.1 (no such class)

I’m not sure what this means yet. My theory is because PyPy hasn’t been ported over to ARM based Mac OS yet I have to use the x86_64 version instead. Maybe this is an incompatibility with ARM CPUs.