Hi all,
Is there a way to display several thousand nodes in a HyperTreeList without performance loss ?
Cheers,
Thierry BRIZZI.
Thierry Brizzi wrote:
Hi all,
Is there a way to display several thousand nodes in a HyperTreeList
without performance loss ?
Normally this is done in wx.TreeCtrl (and I think that the HTL follows the same pattern) is to only populate the tree with what is needed to display the currently expanded nodes. For example, you could add the root node and add its child nodes, and for each of the child nodes which have children of their own just call SetItemHasChildren instead of actually adding them. Then when that node is expanded you can add the child nodes in the EVT_TREE_ITEM_EXPANDING event. If you want to remove child items when a node is collapsed then you can do that too in the EVT_TREE_ITEM_COLLAPSED event handler.
···
--
Robin Dunn
Software Craftsman
Thank you Robin,
In fact, it is not the time to populate the tree which is critical for my widget. Navigating with the scroll bar a node containing several thousand children is more critical because it lags.
It seems the HTL paints all items contained in the tree rather than only visible items. I’ll investigate if it is possible to overload the way items are displayed.
···
Le mercredi 25 septembre 2013 21:28:01 UTC+2, Robin Dunn a écrit :
Thierry Brizzi wrote:
Hi all,
Is there a way to display several thousand nodes in a HyperTreeList
without performance loss ?
Normally this is done in wx.TreeCtrl (and I think that the HTL follows
the same pattern) is to only populate the tree with what is needed to
display the currently expanded nodes. For example, you could add the
root node and add its child nodes, and for each of the child nodes which
have children of their own just call SetItemHasChildren instead of
actually adding them. Then when that node is expanded you can add the
child nodes in the EVT_TREE_ITEM_EXPANDING event. If you want to remove
child items when a node is collapsed then you can do that too in the
EVT_TREE_ITEM_COLLAPSED event handler.–
Robin DunnSoftware Craftsman
Hi,
I’ve a look inside the OnPaint method of the HyperTreeList. It calls the method PaintLevel which draws in a recursive manner all the items contained in a tree. I don’t know if I’ve correctly understood the way it works but it seems that to draw an item it requires to know the position of the last drawn item. Consequently, all the tree is redrawn each time a node is expanded or the scrollbar moved. Moreover, it is impossible to draw only visible items because the visible items need the positions of their predecessors to be drawn.
I don’t know if the HyperTreeList is under active developement but it needs some major display performance improvements. It would be great if the author of this widget gives me some advices to modify the PaintLevel method to consider only visible items.
···
Le mardi 1 octobre 2013 14:20:47 UTC+2, Thierry Brizzi a écrit :
Thank you Robin,
In fact, it is not the time to populate the tree which is critical for my widget. Navigating with the scroll bar a node containing several thousand children is more critical because it lags.
It seems the HTL paints all items contained in the tree rather than only visible items. I’ll investigate if it is possible to overload the way items are displayed.
Le mercredi 25 septembre 2013 21:28:01 UTC+2, Robin Dunn a écrit :
Thierry Brizzi wrote:
Hi all,
Is there a way to display several thousand nodes in a HyperTreeList
without performance loss ?
Normally this is done in wx.TreeCtrl (and I think that the HTL follows
the same pattern) is to only populate the tree with what is needed to
display the currently expanded nodes. For example, you could add the
root node and add its child nodes, and for each of the child nodes which
have children of their own just call SetItemHasChildren instead of
actually adding them. Then when that node is expanded you can add the
child nodes in the EVT_TREE_ITEM_EXPANDING event. If you want to remove
child items when a node is collapsed then you can do that too in the
EVT_TREE_ITEM_COLLAPSED event handler.–
Robin DunnSoftware Craftsman
Hi,
I’ve finally decided to open a ticket in the WxWidgets bug tracker : http://trac.wxwidgets.org/ticket/15554
Hope this improvement request will be confirmed soon.
···
Le mercredi 2 octobre 2013 11:34:47 UTC+2, Thierry Brizzi a écrit :
Hi,
I’ve a look inside the OnPaint method of the HyperTreeList. It calls the method PaintLevel which draws in a recursive manner all the items contained in a tree. I don’t know if I’ve correctly understood the way it works but it seems that to draw an item it requires to know the position of the last drawn item. Consequently, all the tree is redrawn each time a node is expanded or the scrollbar moved. Moreover, it is impossible to draw only visible items because the visible items need the positions of their predecessors to be drawn.
I don’t know if the HyperTreeList is under active developement but it needs some major display performance improvements. It would be great if the author of this widget gives me some advices to modify the PaintLevel method to consider only visible items.
Le mardi 1 octobre 2013 14:20:47 UTC+2, Thierry Brizzi a écrit :
Thank you Robin,
In fact, it is not the time to populate the tree which is critical for my widget. Navigating with the scroll bar a node containing several thousand children is more critical because it lags.
It seems the HTL paints all items contained in the tree rather than only visible items. I’ll investigate if it is possible to overload the way items are displayed.
Le mercredi 25 septembre 2013 21:28:01 UTC+2, Robin Dunn a écrit :
Thierry Brizzi wrote:
Hi all,
Is there a way to display several thousand nodes in a HyperTreeList
without performance loss ?
Normally this is done in wx.TreeCtrl (and I think that the HTL follows
the same pattern) is to only populate the tree with what is needed to
display the currently expanded nodes. For example, you could add the
root node and add its child nodes, and for each of the child nodes which
have children of their own just call SetItemHasChildren instead of
actually adding them. Then when that node is expanded you can add the
child nodes in the EVT_TREE_ITEM_EXPANDING event. If you want to remove
child items when a node is collapsed then you can do that too in the
EVT_TREE_ITEM_COLLAPSED event handler.–
Robin DunnSoftware Craftsman
Thierry Brizzi wrote:
Hi,
I've finally decided to open a ticket in the WxWidgets bug tracker :
wxTrac has been migrated to GitHub Issues - wxWidgetsHope this improvement request will be confirmed soon.
Thanks. If you have ideas for some way to implement the painting in a more optimal way then I'm sure Andrea would welcome a patch.
Also, have you tried the newest AGW code instead of what comes with 2.8.12.1? It's always a good idea to try the latest AGW because it often will have fixes already in place that haven't made it into a wxPython release yet.
Finally, you may want to look at switching to 2.9.5 as it has the DataViewCtrl. It is a new C++ widget (native on Linux and Mac, generic on Windows) that can be used as a flat list view, hierarchical tree view, or a combined treelist view. It is not as flexible as the HyperTreeList but if you don't need any of the extra stuff that the HTL provides then it will likely solve your performance issues with having thousands of items in an expanded node.
···
--
Robin Dunn
Software Craftsman
Hi,
No, I haven’t make the update because the critical part of the HTL hasn’t changed since years. But yes, I’ll try to move to wxPython 2.9.5. Is there a change log listing backward incompatibilities ?
Cheers.
···
Le vendredi 4 octobre 2013 19:57:27 UTC+2, Robin Dunn a écrit :
Thierry Brizzi wrote:
Hi,
I’ve finally decided to open a ticket in the WxWidgets bug tracker :
Hope this improvement request will be confirmed soon.
Thanks. If you have ideas for some way to implement the painting in a
more optimal way then I’m sure Andrea would welcome a patch.Also, have you tried the newest AGW code instead of what comes with
2.8.12.1? It’s always a good idea to try the latest AGW because it
often will have fixes already in place that haven’t made it into a
wxPython release yet.Finally, you may want to look at switching to 2.9.5 as it has the
DataViewCtrl. It is a new C++ widget (native on Linux and Mac, generic
on Windows) that can be used as a flat list view, hierarchical tree
view, or a combined treelist view. It is not as flexible as the
HyperTreeList but if you don’t need any of the extra stuff that the HTL
provides then it will likely solve your performance issues with having
thousands of items in an expanded node.–
Robin DunnSoftware Craftsman
Concerning replacing the HTL with another widget, I think it is not possible as I insert some text fields and checkboxes in the view. I don’t know exactly how works a DataViewCtrl, but it seems it hasn’t got this kind of features.
The HTL is a beautiful widget because it is highly flexible. It’s really easy to overload its default behavior and make it respond to many sorts of usecases.
···
Le lundi 7 octobre 2013 11:25:13 UTC+2, Thierry Brizzi a écrit :
Hi,
No, I haven’t make the update because the critical part of the HTL hasn’t changed since years. But yes, I’ll try to move to wxPython 2.9.5. Is there a change log listing backward incompatibilities ?
Cheers.
Le vendredi 4 octobre 2013 19:57:27 UTC+2, Robin Dunn a écrit :
Thierry Brizzi wrote:
Hi,
I’ve finally decided to open a ticket in the WxWidgets bug tracker :
Hope this improvement request will be confirmed soon.
Thanks. If you have ideas for some way to implement the painting in a
more optimal way then I’m sure Andrea would welcome a patch.Also, have you tried the newest AGW code instead of what comes with
2.8.12.1? It’s always a good idea to try the latest AGW because it
often will have fixes already in place that haven’t made it into a
wxPython release yet.Finally, you may want to look at switching to 2.9.5 as it has the
DataViewCtrl. It is a new C++ widget (native on Linux and Mac, generic
on Windows) that can be used as a flat list view, hierarchical tree
view, or a combined treelist view. It is not as flexible as the
HyperTreeList but if you don’t need any of the extra stuff that the HTL
provides then it will likely solve your performance issues with having
thousands of items in an expanded node.–
Robin DunnSoftware Craftsman
Hi, besided the DVC control (check out the 2.9.5 demo to see them in action) you might want to check out the ObjectListView.
http://objectlistview.sourceforge.net/python/index.html
A patched version which I use with 2.9.5 is available in Mike's MediaLocker repo:
https://bitbucket.org/driscollis/medialocker/src/4771c5a1f5b6448a81bb3beee6863d98ebb0df5c/mlsrc/ObjectListView?at=default
Werner
···
On 07/10/2013 11:52, Thierry Brizzi wrote:
Concerning replacing the HTL with another widget, I think it is not possible as I insert some text fields and checkboxes in the view. I don't know exactly how works a DataViewCtrl, but it seems it hasn't got this kind of features.
The HTL is a beautiful widget because it is highly flexible. It's really easy to overload its default behavior and make it respond to many sorts of usecases.
http://wxpython.org/recentchanges.php
Werner
···
On 07/10/2013 11:25, Thierry Brizzi wrote:
Hi,
No, I haven't make the update because the critical part of the HTL hasn't changed since years. But yes, I'll try to move to wxPython 2.9.5. Is there a change log listing backward incompatibilities ?
Hi,
I’ve begun to implement a subclass of the TreeListMainWindow used by the HyperTreeList widget as a proxy. I’ve found a way to improve display performances by (1) reducing the number of items iterated during the OnPaint method and (2) storing horizontal and vertical connection lines in two separate lists in order to use the DrawLineList at the end of the OnPaint method to draw all connection lines as fast as possible.
But now, I’m facing a bottleneck which I cannot figure out : drawing the horizontal connection lines with a wx.Pen having the wx.STIPPLE (or wx.USER_DASH) style is much more faster than drawing the vertical ones with the same wx.Pen. The tree becomes ugly, but all is fine when the wx.DOT style is used. Is this a known limitation of wxPython ?
Cheers.
···
Le lundi 7 octobre 2013 12:56:18 UTC+2, werner a écrit :
On 07/10/2013 11:25, Thierry Brizzi wrote:
Hi,
No, I haven’t make the update because the critical part of the HTL
hasn’t changed since years. But yes, I’ll try to move to wxPython
2.9.5. Is there a change log listing backward incompatibilities ?http://wxpython.org/recentchanges.php
Werner
I’ve finally written a program subclassing the way the HTL is displayed.
The tree contains 2 nodes, each of them containing 20000 sub-nodes. Using wxPython 2.8.12.1, the program prints the time for displaying horizontal and vertical connectors. When the scrollbar is moved with the mouse to the end of the tree, it shows that drawing horizontal connectors is “instantaneous” whereas drawing vertical connectors is longer and longer.
subclassing_htl.py (14.3 KB)
···
Le mardi 8 octobre 2013 17:15:32 UTC+2, Thierry Brizzi a écrit :
Hi,
I’ve begun to implement a subclass of the TreeListMainWindow used by the HyperTreeList widget as a proxy. I’ve found a way to improve display performances by (1) reducing the number of items iterated during the OnPaint method and (2) storing horizontal and vertical connection lines in two separate lists in order to use the DrawLineList at the end of the OnPaint method to draw all connection lines as fast as possible.
But now, I’m facing a bottleneck which I cannot figure out : drawing the horizontal connection lines with a wx.Pen having the wx.STIPPLE (or wx.USER_DASH) style is much more faster than drawing the vertical ones with the same wx.Pen. The tree becomes ugly, but all is fine when the wx.DOT style is used. Is this a known limitation of wxPython ?
Cheers.
Le lundi 7 octobre 2013 12:56:18 UTC+2, werner a écrit :
On 07/10/2013 11:25, Thierry Brizzi wrote:
Hi,
No, I haven’t make the update because the critical part of the HTL
hasn’t changed since years. But yes, I’ll try to move to wxPython
2.9.5. Is there a change log listing backward incompatibilities ?http://wxpython.org/recentchanges.php
Werner
Thierry Brizzi wrote:
I've finally written a program subclassing the way the HTL is displayed.
The tree contains 2 nodes, each of them containing 20000 sub-nodes.
Using wxPython 2.8.12.1, the program prints the time for displaying
horizontal and vertical connectors. When the scrollbar is moved with the
mouse to the end of the tree, it shows that drawing horizontal
connectors is "instantaneous" whereas drawing vertical connectors is
longer and longer.Le mardi 8 octobre 2013 17:15:32 UTC+2, Thierry Brizzi a �crit :
Hi,
I've begun to implement a subclass of the TreeListMainWindow used by
the HyperTreeList widget as a proxy. I've found a way to improve
display performances by (1) reducing the number of items iterated
during the OnPaint method and (2) storing horizontal and vertical
connection lines in two separate lists in order to use the
DrawLineList at the end of the OnPaint method to draw all connection
lines as fast as possible.But now, I'm facing a bottleneck which I cannot figure out : drawing
the horizontal connection lines with a wx.Pen having the wx.STIPPLE
(or wx.USER_DASH) style is much more faster than drawing the
vertical ones with the same wx.Pen. The tree becomes ugly, but all
is fine when the wx.DOT style is used. Is this a known limitation of
wxPython ?
I expect that it is an issue related to the huge vertical size of the scrolled window's virtual size. If you look at the contents of v_lines you'll see that the y values are getting up to 400k or more. I'm not exactly sure how that would effect drawing times, but considering that pixel positions > 32k or 64k would simply fail on Windows as recently as Windows XP it's not too surprising.
Unfortunately the only other approach that comes to mind is probably much more work than it's worth. (Reimplementing all the virtual size aspects of the scrolled window such that only physical client area coordinates are ever passed to the native APIs.)
···
--
Robin Dunn
Software Craftsman
I found the bottleneck : in fact some vertical connectors are drawn from the start of the tree to the end of the tree. As some lines are more than 300k pixels long, it explains why the DrawLineList hangs for vertical connectors.
···
Le mercredi 9 octobre 2013 03:22:48 UTC+2, Robin Dunn a écrit :
Thierry Brizzi wrote:
I’ve finally written a program subclassing the way the HTL is displayed.
The tree contains 2 nodes, each of them containing 20000 sub-nodes.
Using wxPython 2.8.12.1, the program prints the time for displaying
horizontal and vertical connectors. When the scrollbar is moved with the
mouse to the end of the tree, it shows that drawing horizontal
connectors is “instantaneous” whereas drawing vertical connectors is
longer and longer.
Le mardi 8 octobre 2013 17:15:32 UTC+2, Thierry Brizzi a �crit :
Hi,I've begun to implement a subclass of the TreeListMainWindow used bythe HyperTreeList widget as a proxy. I've found a way to improvedisplay performances by (1) reducing the number of items iteratedduring the OnPaint method and (2) storing horizontal and verticalconnection lines in two separate lists in order to use theDrawLineList at the end of the OnPaint method to draw all connectionlines as fast as possible.But now, I'm facing a bottleneck which I cannot figure out : drawingthe horizontal connection lines with a wx.Pen having the wx.STIPPLE(or wx.USER_DASH) style is much more faster than drawing thevertical ones with the same wx.Pen. The tree becomes ugly, but allis fine when the wx.DOT style is used. Is this a known limitation ofwxPython ?I expect that it is an issue related to the huge vertical size of the
scrolled window’s virtual size. If you look at the contents of v_lines
you’ll see that the y values are getting up to 400k or more. I’m not
exactly sure how that would effect drawing times, but considering that
pixel positions > 32k or 64k would simply fail on Windows as recently as
Windows XP it’s not too surprising.Unfortunately the only other approach that comes to mind is probably
much more work than it’s worth. (Reimplementing all the virtual size
aspects of the scrolled window such that only physical client area
coordinates are ever passed to the native APIs.)–
Robin DunnSoftware Craftsman
Hopefully, low level functions of wxpython weren’t involved. So, I’ve updated the original program to draw vertical connectors in a faster way and prevent the OnPaint method to compute every times item positions because it is already done in the OnInternalIdle method of CustomTreeCtrl. In my mind, even if drawing thousand of nodes hangs, the OnPaint method has reached acceptable performaces regarding the needs of my application.
So now, how can these improvements be integrated in AGW ?
Cheers.
subclassing_htl.py (14.4 KB)
···
Le mercredi 9 octobre 2013 10:04:24 UTC+2, Thierry Brizzi a écrit :
I found the bottleneck : in fact some vertical connectors are drawn from the start of the tree to the end of the tree. As some lines are more than 300k pixels long, it explains why the DrawLineList hangs for vertical connectors.
Le mercredi 9 octobre 2013 03:22:48 UTC+2, Robin Dunn a écrit :
Thierry Brizzi wrote:
I’ve finally written a program subclassing the way the HTL is displayed.
The tree contains 2 nodes, each of them containing 20000 sub-nodes.
Using wxPython 2.8.12.1, the program prints the time for displaying
horizontal and vertical connectors. When the scrollbar is moved with the
mouse to the end of the tree, it shows that drawing horizontal
connectors is “instantaneous” whereas drawing vertical connectors is
longer and longer.
Le mardi 8 octobre 2013 17:15:32 UTC+2, Thierry Brizzi a �crit :
Hi,I've begun to implement a subclass of the TreeListMainWindow used bythe HyperTreeList widget as a proxy. I've found a way to improvedisplay performances by (1) reducing the number of items iteratedduring the OnPaint method and (2) storing horizontal and verticalconnection lines in two separate lists in order to use theDrawLineList at the end of the OnPaint method to draw all connectionlines as fast as possible.But now, I'm facing a bottleneck which I cannot figure out : drawingthe horizontal connection lines with a wx.Pen having the wx.STIPPLE(or wx.USER_DASH) style is much more faster than drawing thevertical ones with the same wx.Pen. The tree becomes ugly, but allis fine when the wx.DOT style is used. Is this a known limitation ofwxPython ?I expect that it is an issue related to the huge vertical size of the
scrolled window’s virtual size. If you look at the contents of v_lines
you’ll see that the y values are getting up to 400k or more. I’m not
exactly sure how that would effect drawing times, but considering that
pixel positions > 32k or 64k would simply fail on Windows as recently as
Windows XP it’s not too surprising.Unfortunately the only other approach that comes to mind is probably
much more work than it’s worth. (Reimplementing all the virtual size
aspects of the scrolled window such that only physical client area
coordinates are ever passed to the native APIs.)–
Robin DunnSoftware Craftsman
Hi Thierry,
···
On 09/10/2013 10:55, Thierry Brizzi wrote:
Hopefully, low level functions of wxpython weren't involved. So, I've updated the original program to draw vertical connectors in a faster way and prevent the OnPaint method to compute every times item positions because it is already done in the OnInternalIdle method of CustomTreeCtrl. In my mind, even if drawing thousand of nodes hangs, the OnPaint method has reached acceptable performaces regarding the needs of my application.
So now, how can these improvements be integrated in AGW ?
A patch against latest SVN to the dev mail list (wxpython-dev@googlegroups.com).
I think, but I am not sure, a separate pull request (Commits · RobinD42/Phoenix · GitHub) to ensure this also gets into Phoenix.
Werner
Thierry Brizzi wrote:
I found the bottleneck : in fact some vertical connectors are drawn from
the start of the tree to the end of the tree. As some lines are more
than 300k pixels long, it explains why the DrawLineList hangs for
vertical connectors.
Yep, that would do it. I should have looked closer at the line start and stop coords than I did.
···
--
Robin Dunn
Software Craftsman
werner wrote:
Hi Thierry,
Hopefully, low level functions of wxpython weren't involved. So, I've
updated the original program to draw vertical connectors in a faster
way and prevent the OnPaint method to compute every times item
positions because it is already done in the OnInternalIdle method of
CustomTreeCtrl. In my mind, even if drawing thousand of nodes hangs,
the OnPaint method has reached acceptable performaces regarding the
needs of my application.So now, how can these improvements be integrated in AGW ?
A patch against latest SVN to the dev mail list
(wxpython-dev@googlegroups.com).
Better would probably be submitting it to Trac (trac.wxwidgets.org) with a follow-up note to wxPython-dev. That way if it gets ignored on the mail list for too long it will be easy to find later in Trac when somebody is able to work on it.
I think, but I am not sure, a separate pull request
(Commits · RobinD42/Phoenix · GitHub) to ensure this also
gets into Phoenix.
Yes, that would be good. The code is diverging a little between classic and Phoenix branches for things like Py3 compatibility and other migration issues, so it would be good to create a Phoenix specific version of the patch too.
···
On 09/10/2013 10:55, Thierry Brizzi wrote:
--
Robin Dunn
Software Craftsman
Hi,
The previous source code contains display bugs. Vertical connectors are not correctly displayed. You can find attached to this email a new version of “subclassing_htl.py” that should correct the problem. Hope this piece of code will help wxpython developers to improve the base code of the HyperTreeList class.
Cheers,
Thierry BRIZZI.
subclassing_htl.py (13.3 KB)
···
Le jeudi 10 octobre 2013 01:56:58 UTC+2, Robin Dunn a écrit :
Thierry Brizzi wrote:
I found the bottleneck : in fact some vertical connectors are drawn from
the start of the tree to the end of the tree. As some lines are more
than 300k pixels long, it explains why the DrawLineList hangs for
vertical connectors.
Yep, that would do it. I should have looked closer at the line start
and stop coords than I did.–
Robin DunnSoftware Craftsman
Was this ever integrated into the wxpython distribution package/download?
···
On Thursday, November 7, 2013 3:05:21 AM UTC-8, Thierry Brizzi wrote:
Hi,
The previous source code contains display bugs. Vertical connectors are not correctly displayed. You can find attached to this email a new version of “subclassing_htl.py” that should correct the problem. Hope this piece of code will help wxpython developers to improve the base code of the HyperTreeList class.
Cheers,
Thierry BRIZZI.
Le jeudi 10 octobre 2013 01:56:58 UTC+2, Robin Dunn a écrit :
Thierry Brizzi wrote:
I found the bottleneck : in fact some vertical connectors are drawn from
the start of the tree to the end of the tree. As some lines are more
than 300k pixels long, it explains why the DrawLineList hangs for
vertical connectors.
Yep, that would do it. I should have looked closer at the line start
and stop coords than I did.–
Robin DunnSoftware Craftsman