Sphinx docs generator issue

wxPython Dev
1

Hi Andrea (or anybody else who is interested),

http://buildbot.wxpython.org:8010/builders/build-docs/builds/190/steps/shell/logs/stdio

As can be seen by that build log the sphinx_generator is having an issue with the doxygen docs for the html2.WebView class recently added to the build. It looks like the problem is with paragraphs like this in the source document:

"""
     @par wxWEB_VIEW_BACKEND_IE (MSW)

     The IE backend uses Microsoft's Trident rendering engine, specifically the
     ...
"""

The XML generated for that looks like this:

"""
<para><simplesect kind="par"><title>wxWEB_VIEW_BACKEND_IE (MSW)</title><para></para></simplesect>
The IE backend uses Microsoft&apos;s Trident rendering engine, specifically the
...
"""

The Doxygen @par command is supposed to start a new paragraph with an optional title, which is rendered in the html like a subsection header.

http://docs.wxwidgets.org/trunk/classwx_web_view.html

What would be the best way to deal with this in sphinx_generator.py?

···

--
Robin Dunn
Software Craftsman
http://wxPython.org

2

Hi Robin,

Hi Andrea (or anybody else who is interested),

http://buildbot.wxpython.org:8010/builders/build-docs/builds/190/steps/shell/logs/stdio

As can be seen by that build log the sphinx_generator is having an issue
with the doxygen docs for the html2.WebView class recently added to the
build. It looks like the problem is with paragraphs like this in the source
document:

"""
    @par wxWEB_VIEW_BACKEND_IE (MSW)

    The IE backend uses Microsoft's Trident rendering engine, specifically
the
    ...
"""

The XML generated for that looks like this:

"""
<para><simplesect kind="par"><title>wxWEB_VIEW_BACKEND_IE
(MSW)</title><para></para></simplesect>
The IE backend uses Microsoft&apos;s Trident rendering engine, specifically
the
...
"""

The Doxygen @par command is supposed to start a new paragraph with an
optional title, which is rendered in the html like a subsection header.

wxWidgets: wxWebView Class Reference

What would be the best way to deal with this in sphinx_generator.py?

I'll take a look at it tomorrow at work, workload permitting. I don't
think it would be much of a hassle, as I believe it would simply be a
matter of handling the "@par" tag as a docutils "spacer" (i.e., a
simple "|" character) and then instruct sphinx_generator.py to
generate a sub-section title out of that.

By the way, now that I am back on track after the South America
adventure, I was thinking if it may be possible (or would be
nice-to-have) to actually be able to run the
extractors/tweakers/generators stuff in parallel. I don't know all the
details of what the extractors do, but in principle it should be
possible to harness all the idle power of the modern-day computers to
run the whole wxPython dox/sip/sphinx stuff with multiprocessing.

Andrea.

"Imagination Is The Only Weapon In The War Against Reality."

# ------------------------------------------------------------- #
def ask_mailing_list_support(email):

    if mention_platform_and_version() and include_sample_app():
        send_message(email)
    else:
        install_malware()
        erase_hard_drives()
# ------------------------------------------------------------- #

···

On 28 November 2012 03:18, Robin Dunn wrote:

3

Yes, I've been thinking about that too. I think that all the various steps should not be started until the prior step is fully complete, but certainly within the individual command functions using a multiprocessing pool would make a lot of sense.

There is already a --jobs option (which defaults to number of CPUs) that is used for make and waf, and could also be used to set the max number of processes for the pool. Some of the command functions could probably be made multiprocess capable fairly easily. In the etg function in build.py, for example it should just be a matter of sending the runcmd function call on the last line to the pool for execution. And then outside of the for loop it should close and join the pool so it will wait for all the jobs to be done.

···

On 11/28/12 11:16 AM, Andrea Gavana wrote:

By the way, now that I am back on track after the South America
adventure, I was thinking if it may be possible (or would be
nice-to-have) to actually be able to run the
extractors/tweakers/generators stuff in parallel. I don't know all the
details of what the extractors do, but in principle it should be
possible to harness all the idle power of the modern-day computers to
run the whole wxPython dox/sip/sphinx stuff with multiprocessing.

--
Robin Dunn
Software Craftsman

4

Hi Robin,

Hi Andrea (or anybody else who is interested),

http://buildbot.wxpython.org:8010/builders/build-docs/builds/190/steps/shell/logs/stdio

As can be seen by that build log the sphinx_generator is having an issue
with the doxygen docs for the html2.WebView class recently added to the
build. It looks like the problem is with paragraphs like this in the source
document:

"""
    @par wxWEB_VIEW_BACKEND_IE (MSW)

    The IE backend uses Microsoft's Trident rendering engine, specifically
the
    ...
"""

The XML generated for that looks like this:

"""
<para><simplesect kind="par"><title>wxWEB_VIEW_BACKEND_IE
(MSW)</title><para></para></simplesect>
The IE backend uses Microsoft&apos;s Trident rendering engine, specifically
the
...
"""

The Doxygen @par command is supposed to start a new paragraph with an
optional title, which is rendered in the html like a subsection header.

wxWidgets: wxWebView Class Reference

What would be the best way to deal with this in sphinx_generator.py?

I have managed to fix this - more or less - in my local copy of the
sources. I wrote "more or less" because, although it seems to be
working, the way the tags are declared has messed up the processing.
See the following XML snippet:

- <para>
- <simplesect kind="par">
     <title>wxWEB_VIEW_WEBKIT (OSX)</title>
     <para />
  </simplesect>
  The OSX WebKit backend uses Apple's
  <ulink url="http://developer.apple.com/library/mac/#documentation/Cocoa/Reference/WebKit/Classes/WebView_Class/Reference/Reference.html\#//apple\_ref/doc/uid/20001903&quot;&gt;WebView&lt;/ulink&gt;
  class. This backend has full support for custom schemes and virtual
file systems.
</para>

The </para> closing tag inside the <simplesect> section really,
really, really should not be there.

But this is less of a problem regarding another one I found, which is
very bad. If you look at this page:

http://wxpython.org/Phoenix/docs/html/ComboBox.html#combobox

You will see that there are 2 sections called "Events Emitted by this
Class", and this happens pretty much everywhere else (in almost all
other widgets, I mean). This is because the headings for the Events in
the wxWidgets docs are now screwed up (in terms of the XML elements
hierarchy).

If you look at the class_wx_web_view.xml file, you will see that the
"Events emitted by this class" heading sits below the "Virtual File
Systems and Custom Schemes" section id, which doesn't make any sense
(and it screws up completely and absolutely the output of
sphinx_generator.py). A similar thing can be seen in the file
classwx_combo_box.xml, where the "Events emitted by this class"
heading sits below the paragraph dedicated to the window styles.

I am not sure what changed in the wxWidgets source code to change the
XML layout so immensely, or what (yet-another) bug the Doxygen people
have introduced. As it stands, I seem to be unable to fix this problem
in the sphinx_generator.py. It is a very bad issue as almost all the
widgets documentation are affected. Hopefully the wx-devs will have
some suggestion about this...

Andrea.

"Imagination Is The Only Weapon In The War Against Reality."

# ------------------------------------------------------------- #
def ask_mailing_list_support(email):

    if mention_platform_and_version() and include_sample_app():
        send_message(email)
    else:
        install_malware()
        erase_hard_drives()
# ------------------------------------------------------------- #

···

On 28 November 2012 03:18, Robin Dunn wrote:

5

Hi Robin,

Hi Andrea (or anybody else who is interested),

http://buildbot.wxpython.org:8010/builders/build-docs/builds/190/steps/shell/logs/stdio

As can be seen by that build log the sphinx_generator is having an issue
with the doxygen docs for the html2.WebView class recently added to the
build. It looks like the problem is with paragraphs like this in the source
document:

"""
     @par wxWEB_VIEW_BACKEND_IE (MSW)

     The IE backend uses Microsoft's Trident rendering engine, specifically
the
     ...
"""

The XML generated for that looks like this:

"""
<para><simplesect kind="par"><title>wxWEB_VIEW_BACKEND_IE
(MSW)</title><para></para></simplesect>
The IE backend uses Microsoft&apos;s Trident rendering engine, specifically
the
...
"""

The Doxygen @par command is supposed to start a new paragraph with an
optional title, which is rendered in the html like a subsection header.

wxWidgets: wxWebView Class Reference

What would be the best way to deal with this in sphinx_generator.py?

I have managed to fix this - more or less - in my local copy of the
sources. I wrote "more or less" because, although it seems to be
working, the way the tags are declared has messed up the processing.
See the following XML snippet:

- <para>
- <simplesect kind="par">
      <title>wxWEB_VIEW_WEBKIT (OSX)</title>
      <para />
   </simplesect>
   The OSX WebKit backend uses Apple's
   <ulink url="http://developer.apple.com/library/mac/#documentation/Cocoa/Reference/WebKit/Classes/WebView_Class/Reference/Reference.html\#//apple\_ref/doc/uid/20001903&quot;&gt;WebView&lt;/ulink&gt;
   class. This backend has full support for custom schemes and virtual
file systems.
</para>

The </para> closing tag inside the <simplesect> section really,
really, really should not be there.

Since it is <para /> that means that it is both the beginning and ending of a para, like <para></para> which I would interpret as an empty paragraph inside that section, which could probably be ignored. Is that not correct? Or am I not understanding what the problem is?

But this is less of a problem regarding another one I found, which is
very bad. If you look at this page:

http://wxpython.org/Phoenix/docs/html/ComboBox.html#combobox

You will see that there are 2 sections called "Events Emitted by this
Class", and this happens pretty much everywhere else (in almost all
other widgets, I mean). This is because the headings for the Events in
the wxWidgets docs are now screwed up (in terms of the XML elements
hierarchy).

If you look at the class_wx_web_view.xml file, you will see that the
"Events emitted by this class" heading sits below the "Virtual File
Systems and Custom Schemes" section id, which doesn't make any sense
(and it screws up completely and absolutely the output of
sphinx_generator.py). A similar thing can be seen in the file
classwx_combo_box.xml, where the "Events emitted by this class"
heading sits below the paragraph dedicated to the window styles.

Why does the order of the sections matter? Oh, I see. Is is because the <header> tag for the events section is appearing before the closing </para> tag belonging to the itemizedlist group in the previous section? Is there a way for your code to ignore the para tags when an itemizizedlist tag is being processed?

I am not sure what changed in the wxWidgets source code to change the
XML layout so immensely, or what (yet-another) bug the Doxygen people
have introduced. As it stands, I seem to be unable to fix this problem
in the sphinx_generator.py. It is a very bad issue as almost all the
widgets documentation are affected. Hopefully the wx-devs will have
some suggestion about this...

We recently upgraded to Doxygen 1.8.2 from 1.7.4, so the bulk of the changes you are seeing are probably due to that. There was also some changes made to how doxygen is run and for the wx html docs output, but I doubt that had much impact on the generation of the xml.

I've CC'd Bryan Petty on this email in case he knows what might be happening here. (Bryan, you'll probably have to join wxPython-dev to reply.)

···

On 11/29/12 3:59 AM, Andrea Gavana wrote:

On 28 November 2012 03:18, Robin Dunn wrote:

--
Robin Dunn
Software Craftsman

6

Yeah, Doxygen is notorious for doing weird stuff like this between versions.

For what it's worth though, the "events emitted" sections are mostly
controlled through alias commands, and from within there, we should be
able to specifically control most of the XML structure it uses there
using the "\xmlonly" command.

And as far as wxWebView goes, that could just use something other than
@par, and we can get that changed in the source docs.

Can you just briefly outline which ones are still a problem here and
what the ideal structure might be? (I had a little trouble following
the thread)

Regards,
Bryan Petty

···

On Thu, Nov 29, 2012 at 3:12 PM, Robin Dunn <robin@alldunn.com> wrote:

On 11/29/12 3:59 AM, Andrea Gavana wrote:

I am not sure what changed in the wxWidgets source code to change the
XML layout so immensely, or what (yet-another) bug the Doxygen people
have introduced. As it stands, I seem to be unable to fix this problem
in the sphinx_generator.py. It is a very bad issue as almost all the
widgets documentation are affected. Hopefully the wx-devs will have
some suggestion about this...

We recently upgraded to Doxygen 1.8.2 from 1.7.4, so the bulk of the changes
you are seeing are probably due to that. There was also some changes made
to how doxygen is run and for the wx html docs output, but I doubt that had
much impact on the generation of the xml.

7

Hi Robin, Bryan,

I am not sure what changed in the wxWidgets source code to change the
XML layout so immensely, or what (yet-another) bug the Doxygen people
have introduced. As it stands, I seem to be unable to fix this problem
in the sphinx_generator.py. It is a very bad issue as almost all the
widgets documentation are affected. Hopefully the wx-devs will have
some suggestion about this...

We recently upgraded to Doxygen 1.8.2 from 1.7.4, so the bulk of the changes
you are seeing are probably due to that. There was also some changes made
to how doxygen is run and for the wx html docs output, but I doubt that had
much impact on the generation of the xml.

Yeah, Doxygen is notorious for doing weird stuff like this between versions.

For what it's worth though, the "events emitted" sections are mostly
controlled through alias commands, and from within there, we should be
able to specifically control most of the XML structure it uses there
using the "\xmlonly" command.

And as far as wxWebView goes, that could just use something other than
@par, and we can get that changed in the source docs.

Can you just briefly outline which ones are still a problem here and
what the ideal structure might be? (I had a little trouble following
the thread)

I have managed to fix this thing from the Python side, with a slightly
shaky text processing routine, but it seems to work. As a reference,
the ideal structure would be that a new heading (like "Events emitted
by this class") should not sit below a section dedicated to "Virtual
File Systems and Custom Schemes" (or any other un-related section).
Also, the heading and the event macros description should be in the
same hierarchy, not split across different sections.

Anyway, it should be fixed now.

@Robin:

Just for the sake of it:

Since it is <para /> that means that it is both the beginning and ending of a para,
like <para></para> which I would interpret as an empty paragraph inside that
section, which could probably be ignored. Is that not correct? Or am I not
understanding what the problem is?

The section specified by the "@par" doxygen tag is not inside a
<para>, is inside a <simplesect> section:

<simplesect kind="par">
  <title>wxWEB_VIEW_WEBKIT (GTK)</title>
  <para />
</simplesect>

There is no opening <para> tag, and in fact the opening <para> and
closing </para> tags are unbalanced (there are more closing than
opening in the XML file). But then again, this is mostly redundant
now, it appears everything is working smoothly. I'll commit the
updated Python code tonight.

Andrea.

"Imagination Is The Only Weapon In The War Against Reality."

# ------------------------------------------------------------- #
def ask_mailing_list_support(email):

    if mention_platform_and_version() and include_sample_app():
        send_message(email)
    else:
        install_malware()
        erase_hard_drives()
# ------------------------------------------------------------- #

···

On 30 November 2012 02:11, Bryan Petty wrote:

On Thu, Nov 29, 2012 at 3:12 PM, Robin Dunn <robin@alldunn.com> wrote:

On 11/29/12 3:59 AM, Andrea Gavana wrote: