DataView and virtual trees

Hi All,

     in the latest application I am writing, I am currently using
CustomTreeCtrl to display a series of keywords coming from text files
rendered with StyledTextCtrl. However, some files may contain many,
many keywords and the realtime update of a lot of tree items in
CustomTreeCtrl becomes expensive. So, as much as it pains me to leave
CustomTreeCtrl, I have begun investigating the DataView stuff.

Now, I know nothing of DataView things, but I still have a couple of questions:

1) Is there a way to implement virtual tree controls with the various
DataView classes?
2) (opportunistic-question) If it is possible, does anyone have a
sample on how to actually go for it (or some suggestions/guidances on
how to implement it)?

Thank you in advance.

Andrea.

"Imagination Is The Only Weapon In The War Against Reality."
http://xoomer.alice.it/infinity77/

Take a look at the DVC_DataViewModel sample in the demo. It implements a tree model, the key point for the hierarchy being the implementation of the GetChildren and GetParent methods.

···

On 4/23/12 3:01 AM, Andrea Gavana wrote:

Hi All,

      in the latest application I am writing, I am currently using
CustomTreeCtrl to display a series of keywords coming from text files
rendered with StyledTextCtrl. However, some files may contain many,
many keywords and the realtime update of a lot of tree items in
CustomTreeCtrl becomes expensive. So, as much as it pains me to leave
CustomTreeCtrl, I have begun investigating the DataView stuff.

Now, I know nothing of DataView things, but I still have a couple of questions:

1) Is there a way to implement virtual tree controls with the various
DataView classes?
2) (opportunistic-question) If it is possible, does anyone have a
sample on how to actually go for it (or some suggestions/guidances on
how to implement it)?

--
Robin Dunn
Software Craftsman

Thanks Robin, I had somehow missed this demo. I got to a relatively
good point, but I am now facing a very weird issue.

The text editor I built is a tabbed notebook, and every time the user
switches tab the dataview tree gets repopulated. Now, after a while of
this back and forth switching, I get the following:

Traceback (most recent call last):
  File "C:\MyProjects\DeckEd\src\sidepages.py", line 144, in GetValue
    node = self.ItemToObject(item)
  File "C:\Python27\lib\site-packages\wx-2.9.3-msw\wx\dataview.py",
line 773, in ItemToObject
    return self.objmapper.ItemToObject(item)
  File "C:\Python27\lib\site-packages\wx-2.9.3-msw\wx\dataview.py",
line 721, in ItemToObject
    oid = item.GetID()
  File "C:\Python27\lib\site-packages\wx-2.9.3-msw\wx\dataview.py",
line 110, in GetID
    return _dataview.DataViewItem_GetID(*args, **kwargs)
PyAssertionError: C++ assertion "Assert failure" failed at
..\..\src\msw\dcmemory.cpp(131) in wxMemoryDCImpl::DoSelect():
Couldn't select a bitmap into wxMemoryDC

And many other errors after this, seemingly related to saturations of
GDI objects (after 4 or 5 tab switches the GDI count reaches 10,000).
There are a couple of issues I am not sure about:

1) I am using DataViewIconText to render images close to the text in
the tree (BTW, why the use of wx.Icon??? It was so convenient to have
wx.Bitmap everywhere...), but if I do the following in my
PyDataViewModel:

def GetValue(self, item, col):

    node = self.ItemToObject(item)
    return dv.DataViewIconText(node.name, node.icon)

SWIG complains as follow:

swig/python detected a memory leak of type 'wxDataViewIconText *', no
destructor found.

The only way to make this message go away is to store the
DataViewIconText as an attribute of my node class, i.e.:

self.iconText = dv.DataViewIconText(name, icon)

and then in PyDataViewModel:

def GetValue(self, item, col):

    node = self.ItemToObject(item)
    return node.iconText

2) It appears that the GDI count increase without bounds every time I
call AssociateModel() to DataViewCtrl. The problem is, my model
changes completely from one tab to another, so I can't simply use
ItemsChanged() and/or ItemsDeleted() or whatever. I need to rebuild
the model from scratch and re-associate it with DVC.

If someone sees something wrong in what I am doing, please do offer
your suggestion. In the meanwhile I'll try to put together a sample
demonstrating the problem.

···

On 23 April 2012 18:36, Robin Dunn wrote:

On 4/23/12 3:01 AM, Andrea Gavana wrote:

Hi All,

 in the latest application I am writing, I am currently using

CustomTreeCtrl to display a series of keywords coming from text files
rendered with StyledTextCtrl. However, some files may contain many,
many keywords and the realtime update of a lot of tree items in
CustomTreeCtrl becomes expensive. So, as much as it pains me to leave
CustomTreeCtrl, I have begun investigating the DataView stuff.

Now, I know nothing of DataView things, but I still have a couple of
questions:

1) Is there a way to implement virtual tree controls with the various
DataView classes?
2) (opportunistic-question) If it is possible, does anyone have a
sample on how to actually go for it (or some suggestions/guidances on
how to implement it)?

Take a look at the DVC_DataViewModel sample in the demo. It implements a
tree model, the key point for the hierarchy being the implementation of the
GetChildren and GetParent methods.

--
Andrea.

"Imagination Is The Only Weapon In The War Against Reality."
http://xoomer.alice.it/infinity77/

Hi All,

 in the latest application I am writing, I am currently using

CustomTreeCtrl to display a series of keywords coming from text files
rendered with StyledTextCtrl. However, some files may contain many,
many keywords and the realtime update of a lot of tree items in
CustomTreeCtrl becomes expensive. So, as much as it pains me to leave
CustomTreeCtrl, I have begun investigating the DataView stuff.

Now, I know nothing of DataView things, but I still have a couple of
questions:

1) Is there a way to implement virtual tree controls with the various
DataView classes?
2) (opportunistic-question) If it is possible, does anyone have a
sample on how to actually go for it (or some suggestions/guidances on
how to implement it)?

Take a look at the DVC_DataViewModel sample in the demo. It implements a
tree model, the key point for the hierarchy being the implementation of the
GetChildren and GetParent methods.

Thanks Robin, I had somehow missed this demo. I got to a relatively
good point, but I am now facing a very weird issue.

The text editor I built is a tabbed notebook, and every time the user
switches tab the dataview tree gets repopulated. Now, after a while of
this back and forth switching, I get the following:

Traceback (most recent call last):
File "C:\MyProjects\DeckEd\src\sidepages.py", line 144, in GetValue
node = self.ItemToObject(item)
File "C:\Python27\lib\site-packages\wx-2.9.3-msw\wx\dataview.py",
line 773, in ItemToObject
return self.objmapper.ItemToObject(item)
File "C:\Python27\lib\site-packages\wx-2.9.3-msw\wx\dataview.py",
line 721, in ItemToObject
oid = item.GetID()
File "C:\Python27\lib\site-packages\wx-2.9.3-msw\wx\dataview.py",
line 110, in GetID
return _dataview.DataViewItem_GetID(*args, **kwargs)
PyAssertionError: C++ assertion "Assert failure" failed at
..\..\src\msw\dcmemory.cpp(131) in wxMemoryDCImpl::DoSelect():
Couldn't select a bitmap into wxMemoryDC

<snip>

If someone sees something wrong in what I am doing, please do offer
your suggestion. In the meanwhile I'll try to put together a sample
demonstrating the problem.

OK, I managed to put together a sample demonstrating the problem. On
my machine (Windows Vista, Python 2.7.2 64 bit, wxPython 2.9.3.1
Classic), switching back and forth from a notebook tab to another adds
between 50 and 100 GDI objects *per switch*. Which means that it gets
relatively quickly to 10,000, more so in my real app where I use many
more icons and fancy stuff.

Any suggestion would be very appreciated...

Thank you!

Andrea.

"Imagination Is The Only Weapon In The War Against Reality."
http://xoomer.alice.it/infinity77/

dv_test2.py (5.94 KB)

···

On 24 April 2012 12:25, Andrea Gavana wrote:

On 23 April 2012 18:36, Robin Dunn wrote:

On 4/23/12 3:01 AM, Andrea Gavana wrote:

[snip]

Any suggestion would be very appreciated...

How about having a separate data view control for each page in the notebook?

You could show/hide/layout the relevant controls on a page change event, or maybe even integrate the data views in the pages.

This would allow you to change/recreate the models only when the data is actually changed, not when the user navigates the tabs.

Toni

···

On Tue, 24 Apr 2012 13:05:35 +0200, Andrea Gavana <andrea.gavana@gmail.com> wrote:

That's a possibility, but then when the user has 20+ tabs opened I am
not sure how heavy the application will become... If all else fails,
I'll probably go down that road.

Thank you for your suggestion.

Andrea.

"Imagination Is The Only Weapon In The War Against Reality."
http://xoomer.alice.it/infinity77/

···

On 24 April 2012 15:01, Toni Ruža wrote:

On Tue, 24 Apr 2012 13:05:35 +0200, Andrea Gavana <andrea.gavana@gmail.com> > wrote:

[snip]

Any suggestion would be very appreciated...

How about having a separate data view control for each page in the notebook?

You could show/hide/layout the relevant controls on a page change event, or
maybe even integrate the data views in the pages.

This would allow you to change/recreate the models only when the data is
actually changed, not when the user navigates the tabs.

To further complicate things you could make a fixed amount of data views and distribute the workload of however many tabs the user has open. But yeah, these kinds of hacks really shouldn't be required for something like this.

Toni

···

On Tue, 24 Apr 2012 15:32:29 +0200, Andrea Gavana <andrea.gavana@gmail.com> wrote:

On 24 April 2012 15:01, Toni Ruža wrote:

On Tue, 24 Apr 2012 13:05:35 +0200, Andrea Gavana <andrea.gavana@gmail.com> >> wrote:

[snip]

Any suggestion would be very appreciated...

How about having a separate data view control for each page in the notebook?

You could show/hide/layout the relevant controls on a page change event, or
maybe even integrate the data views in the pages.

This would allow you to change/recreate the models only when the data is
actually changed, not when the user navigates the tabs.

That's a possibility, but then when the user has 20+ tabs opened I am
not sure how heavy the application will become... If all else fails,
I'll probably go down that road.

I've got a fix for the swig/python memory leak warning, but it will need a new wxPython build.

It appears that the old model objects are not being fully cleaned up, or at least some of the data structures on the Python side of things. I'll need to spend some more time to fully diagnose it. In the meantime you can greatly reduce or eliminate the GDI growth by reusing icons instead of creating new instances for each item in the tree.

···

On 4/24/12 4:05 AM, Andrea Gavana wrote:

On 24 April 2012 12:25, Andrea Gavana wrote:

On 23 April 2012 18:36, Robin Dunn wrote:

On 4/23/12 3:01 AM, Andrea Gavana wrote:

Hi All,

      in the latest application I am writing, I am currently using
CustomTreeCtrl to display a series of keywords coming from text files
rendered with StyledTextCtrl. However, some files may contain many,
many keywords and the realtime update of a lot of tree items in
CustomTreeCtrl becomes expensive. So, as much as it pains me to leave
CustomTreeCtrl, I have begun investigating the DataView stuff.

Now, I know nothing of DataView things, but I still have a couple of
questions:

1) Is there a way to implement virtual tree controls with the various
DataView classes?
2) (opportunistic-question) If it is possible, does anyone have a
sample on how to actually go for it (or some suggestions/guidances on
how to implement it)?

Take a look at the DVC_DataViewModel sample in the demo. It implements a
tree model, the key point for the hierarchy being the implementation of the
GetChildren and GetParent methods.

Thanks Robin, I had somehow missed this demo. I got to a relatively
good point, but I am now facing a very weird issue.

The text editor I built is a tabbed notebook, and every time the user
switches tab the dataview tree gets repopulated. Now, after a while of
this back and forth switching, I get the following:

Traceback (most recent call last):
  File "C:\MyProjects\DeckEd\src\sidepages.py", line 144, in GetValue
    node = self.ItemToObject(item)
  File "C:\Python27\lib\site-packages\wx-2.9.3-msw\wx\dataview.py",
line 773, in ItemToObject
    return self.objmapper.ItemToObject(item)
  File "C:\Python27\lib\site-packages\wx-2.9.3-msw\wx\dataview.py",
line 721, in ItemToObject
    oid = item.GetID()
  File "C:\Python27\lib\site-packages\wx-2.9.3-msw\wx\dataview.py",
line 110, in GetID
    return _dataview.DataViewItem_GetID(*args, **kwargs)
PyAssertionError: C++ assertion "Assert failure" failed at
..\..\src\msw\dcmemory.cpp(131) in wxMemoryDCImpl::DoSelect():
Couldn't select a bitmap into wxMemoryDC

<snip>

If someone sees something wrong in what I am doing, please do offer
your suggestion. In the meanwhile I'll try to put together a sample
demonstrating the problem.

OK, I managed to put together a sample demonstrating the problem. On
my machine (Windows Vista, Python 2.7.2 64 bit, wxPython 2.9.3.1
Classic), switching back and forth from a notebook tab to another adds
between 50 and 100 GDI objects *per switch*. Which means that it gets
relatively quickly to 10,000, more so in my real app where I use many
more icons and fancy stuff.

--
Robin Dunn
Software Craftsman

Here is some more info: Since the PyDataViewModel holds a reference to itself (so it can reflect calls to virtual methods to their C++ counterparts) then the normal cleanup is not going to happen when self.model is replaced with a new value. So we need to help it a bit. Since the C++ wxDataViewModel is using ref counting then we can't just call self.model.Destroy(), but will need to call DecRef instead. I ended up doing something like this in your sample:

         newModel = TreeModel(data)
         self.AssociateModel(newModel)
         if self.model is not None:
             self.model.thisown = False
             self.model.DecRef()
         self.model = newModel

Setting thisown is a SWIG thing and is needed so it won't try to call DefRef one more time when the Python part of self.model is gc'd.

Also I noticed that there is sometimes a call to GetValue with a bad item after the model has been changed. I expect it might be the selection or something else that was set internally with the old model. So you should wrap your call to ItemToObject there with a KeyError exception handler just in case.

Another approach to consider is instead of replacing the model each time just clear and reinitialize its .data attribute. IIRC if you call the model's Cleared() method then that should inform all the associated views that the data model has been reset and that they should reinitialize themselves with the new contents of the model.

···

On 4/24/12 1:33 PM, Robin Dunn wrote:

On 4/24/12 4:05 AM, Andrea Gavana wrote:

On 24 April 2012 12:25, Andrea Gavana wrote:

On 23 April 2012 18:36, Robin Dunn wrote:

On 4/23/12 3:01 AM, Andrea Gavana wrote:

Hi All,

in the latest application I am writing, I am currently using
CustomTreeCtrl to display a series of keywords coming from text files
rendered with StyledTextCtrl. However, some files may contain many,
many keywords and the realtime update of a lot of tree items in
CustomTreeCtrl becomes expensive. So, as much as it pains me to leave
CustomTreeCtrl, I have begun investigating the DataView stuff.

Now, I know nothing of DataView things, but I still have a couple of
questions:

1) Is there a way to implement virtual tree controls with the various
DataView classes?
2) (opportunistic-question) If it is possible, does anyone have a
sample on how to actually go for it (or some suggestions/guidances on
how to implement it)?

Take a look at the DVC_DataViewModel sample in the demo. It
implements a
tree model, the key point for the hierarchy being the implementation
of the
GetChildren and GetParent methods.

Thanks Robin, I had somehow missed this demo. I got to a relatively
good point, but I am now facing a very weird issue.

The text editor I built is a tabbed notebook, and every time the user
switches tab the dataview tree gets repopulated. Now, after a while of
this back and forth switching, I get the following:

Traceback (most recent call last):
File "C:\MyProjects\DeckEd\src\sidepages.py", line 144, in GetValue
node = self.ItemToObject(item)
File "C:\Python27\lib\site-packages\wx-2.9.3-msw\wx\dataview.py",
line 773, in ItemToObject
return self.objmapper.ItemToObject(item)
File "C:\Python27\lib\site-packages\wx-2.9.3-msw\wx\dataview.py",
line 721, in ItemToObject
oid = item.GetID()
File "C:\Python27\lib\site-packages\wx-2.9.3-msw\wx\dataview.py",
line 110, in GetID
return _dataview.DataViewItem_GetID(*args, **kwargs)
PyAssertionError: C++ assertion "Assert failure" failed at
..\..\src\msw\dcmemory.cpp(131) in wxMemoryDCImpl::DoSelect():
Couldn't select a bitmap into wxMemoryDC

<snip>

If someone sees something wrong in what I am doing, please do offer
your suggestion. In the meanwhile I'll try to put together a sample
demonstrating the problem.

OK, I managed to put together a sample demonstrating the problem. On
my machine (Windows Vista, Python 2.7.2 64 bit, wxPython 2.9.3.1
Classic), switching back and forth from a notebook tab to another adds
between 50 and 100 GDI objects *per switch*. Which means that it gets
relatively quickly to 10,000, more so in my real app where I use many
more icons and fancy stuff.

I've got a fix for the swig/python memory leak warning, but it will need
a new wxPython build.

It appears that the old model objects are not being fully cleaned up, or
at least some of the data structures on the Python side of things. I'll
need to spend some more time to fully diagnose it. In the meantime you
can greatly reduce or eliminate the GDI growth by reusing icons instead
of creating new instances for each item in the tree.

--
Robin Dunn
Software Craftsman

Thanks Robin.

I have tried both approaches, although the second one does not
re-display the new model correctly, i.e. using model.Cleared() and
re-setting the model data most of the time generates KeyErrors (which
I handle as you suggested) or black (empty) icons or empty text.

The first approach eliminates the GDI object increase but I always get
a gazillion of:

swig/python detected a memory leak of type 'wxDataViewIconText *', no
destructor found.

I don't mind having these leaks (for the moment), but as Python prints
out these messages to stdio/stderr/whatever, it slows down the entire
interface until it has finished to print them out. I suppose there is
no way to silence these messages...

I think I am almost there, although one thing DVC is still missing is
a modification of TreeMixin.ExpansionState to store and set item
expansion states :slight_smile: . I guess I'll try and write one, although DVC is
not exactly compatible with wx.TreeCtrl and CustomTreeCtrl.

Andrea.

"Imagination Is The Only Weapon In The War Against Reality."
http://xoomer.alice.it/infinity77/

···

On 25 April 2012 03:06, Robin Dunn wrote:

On 4/24/12 1:33 PM, Robin Dunn wrote:

On 4/24/12 4:05 AM, Andrea Gavana wrote:

On 24 April 2012 12:25, Andrea Gavana wrote:

On 23 April 2012 18:36, Robin Dunn wrote:

On 4/23/12 3:01 AM, Andrea Gavana wrote:

Hi All,

in the latest application I am writing, I am currently using
CustomTreeCtrl to display a series of keywords coming from text files
rendered with StyledTextCtrl. However, some files may contain many,
many keywords and the realtime update of a lot of tree items in
CustomTreeCtrl becomes expensive. So, as much as it pains me to leave
CustomTreeCtrl, I have begun investigating the DataView stuff.

Now, I know nothing of DataView things, but I still have a couple of
questions:

1) Is there a way to implement virtual tree controls with the various
DataView classes?
2) (opportunistic-question) If it is possible, does anyone have a
sample on how to actually go for it (or some suggestions/guidances on
how to implement it)?

Take a look at the DVC_DataViewModel sample in the demo. It
implements a
tree model, the key point for the hierarchy being the implementation
of the
GetChildren and GetParent methods.

Thanks Robin, I had somehow missed this demo. I got to a relatively
good point, but I am now facing a very weird issue.

The text editor I built is a tabbed notebook, and every time the user
switches tab the dataview tree gets repopulated. Now, after a while of
this back and forth switching, I get the following:

Traceback (most recent call last):
File "C:\MyProjects\DeckEd\src\sidepages.py", line 144, in GetValue
node = self.ItemToObject(item)
File "C:\Python27\lib\site-packages\wx-2.9.3-msw\wx\dataview.py",
line 773, in ItemToObject
return self.objmapper.ItemToObject(item)
File "C:\Python27\lib\site-packages\wx-2.9.3-msw\wx\dataview.py",
line 721, in ItemToObject
oid = item.GetID()
File "C:\Python27\lib\site-packages\wx-2.9.3-msw\wx\dataview.py",
line 110, in GetID
return _dataview.DataViewItem_GetID(*args, **kwargs)
PyAssertionError: C++ assertion "Assert failure" failed at
..\..\src\msw\dcmemory.cpp(131) in wxMemoryDCImpl::DoSelect():
Couldn't select a bitmap into wxMemoryDC

<snip>

If someone sees something wrong in what I am doing, please do offer
your suggestion. In the meanwhile I'll try to put together a sample
demonstrating the problem.

OK, I managed to put together a sample demonstrating the problem. On
my machine (Windows Vista, Python 2.7.2 64 bit, wxPython 2.9.3.1
Classic), switching back and forth from a notebook tab to another adds
between 50 and 100 GDI objects *per switch*. Which means that it gets
relatively quickly to 10,000, more so in my real app where I use many
more icons and fancy stuff.

I've got a fix for the swig/python memory leak warning, but it will need
a new wxPython build.

It appears that the old model objects are not being fully cleaned up, or
at least some of the data structures on the Python side of things. I'll
need to spend some more time to fully diagnose it. In the meantime you
can greatly reduce or eliminate the GDI growth by reusing icons instead
of creating new instances for each item in the tree.

Here is some more info: Since the PyDataViewModel holds a reference to
itself (so it can reflect calls to virtual methods to their C++
counterparts) then the normal cleanup is not going to happen when self.model
is replaced with a new value. So we need to help it a bit. Since the C++
wxDataViewModel is using ref counting then we can't just call
self.model.Destroy(), but will need to call DecRef instead. I ended up
doing something like this in your sample:

   newModel = TreeModel\(data\)
   self\.AssociateModel\(newModel\)
   if self\.model is not None:
       self\.model\.thisown = False
       self\.model\.DecRef\(\)
   self\.model = newModel

Setting thisown is a SWIG thing and is needed so it won't try to call DefRef
one more time when the Python part of self.model is gc'd.

Also I noticed that there is sometimes a call to GetValue with a bad item
after the model has been changed. I expect it might be the selection or
something else that was set internally with the old model. So you should
wrap your call to ItemToObject there with a KeyError exception handler just
in case.

Another approach to consider is instead of replacing the model each time
just clear and reinitialize its .data attribute. IIRC if you call the
model's Cleared() method then that should inform all the associated views
that the data model has been reset and that they should reinitialize
themselves with the new contents of the model.

They are being printed to the C stdout so you could probably speed things up by redirecting to a file using > on the command line. Python's sys.stdout is initially set to the C stdout, but you can reset that to something else if you need to see log messages or something.

···

On 4/25/12 5:24 AM, Andrea Gavana wrote:

Thanks Robin.

I have tried both approaches, although the second one does not
re-display the new model correctly, i.e. using model.Cleared() and
re-setting the model data most of the time generates KeyErrors (which
I handle as you suggested) or black (empty) icons or empty text.

The first approach eliminates the GDI object increase but I always get
a gazillion of:

swig/python detected a memory leak of type 'wxDataViewIconText *', no
destructor found.

I don't mind having these leaks (for the moment), but as Python prints
out these messages to stdio/stderr/whatever, it slows down the entire
interface until it has finished to print them out. I suppose there is
no way to silence these messages...

--
Robin Dunn
Software Craftsman