waf could be configured to compile cython modules, yes.
waf is a meta build system. It’s not actually bound to any language/compiler. You could configure
waf to invoke and use PyInstaller as a compiler if you wanted.
RE Documentation. https://sphinx-doc.org is by far the way to go. I use that for all of my personal projects, and it’s used on PyInstaller, wxPython, and Python itself for documentation generation - and most *.readthedocs.io sites are sphinx-generated. reST is recommended. Sphinx supports LaTeX PDF, html, and epub output.
So as I understand you have a project. The GUI - and maybe more - is in python. High-speed stuff is written in C, with Cython binding it into python extensions. You need to compile it all with a one-command setup. I’d use
waf or write a python/shell/batch/powershell script(s) which call each of the seperate compilers and pull all of the output together. (We have 4 seperate bootloader builds for PyInstaller. We only need run
python ./waf all to build them all because
waf can do that.
And finally installers. When it comes to Linux/macOS I must admit to be as stumped as you - or more. I personally use InnoSetup for windows, which is kind of open source. It’s a great solution that generates a single
.exe file which starts a Wizard to install everything. I’ve also heard NSIS is good, but never used it myself.
PS There is a solution for distribution on Linux/macOS which is easy and I know of. Unfortunately it’s commercial and costs - a lot.