trouble with wx.ListCtrl.DeleteAllItems()

Karsten_Hilbert · January 24, 2013, 9:52am

Hi all,

from time to time -- under not yet fully understood
circumstances -- the following code snippet actually
succeeds in throwing an exception in our application
(www.gnumed.de) running under wxPython 2.8.12.1 on
Debian (self is a wx.ListCtrl child):

def set_string_items(self, items = None):
"""All item members must be unicode()able or None."""

    self.DeleteAllItems()
    if self.ItemCount != 0:
      raise ValueError('.ItemCount not 0 after .DeleteAllItems()')

There's basically three potential reasons:

1) DeleteAllItems() is broken

  This is unlikely because a) it works as expected most of
  the time and b) it seems quite a prominent method in the
  wxPython body of code such that this kind of breakage
  would have been fixed by now.

2) DeleteAllItems() only schedules deletion of items which
then happens in a delayed fashion

I have not been able to confirm this on the web anywhere.

3) two threads get in the way of each other trying to
use set_string_items()

  A race *could* happen between
    self.DeleteAllItems()
  and
    self.ItemCount

  Yes, we do have threads but, no, we are taking care to
  wrap them in wx.CallAfter() as needed (might have missed
  something somewhere).

I am currently investigating 3). Are there any further
hints, or maybe knowledge on 1) or 2) to be had ?

Thanks,
Karsten

···

--
GPG key ID E4071346 @ gpg-keyserver.de
E167 67FD A291 2BEA 73BD 4537 78B9 A9F9 E407 1346

Karsten_Hilbert · January 24, 2013, 11:35am

Or maybe I should not trust self.ItemCount "too soon" (the
web talks about it being updated only "later").

As to why this self.ItemCount check is there in the first place:

We've initially had an exception where we ended up with
double the list items than we expected (despite calling
DeleteAllItems() each time before setting items).

Is there a way other than .ItemCount which tells me the
current number of items ?

Should I

while self.ItemCount > 0:
wx.SafeYield()

?

Or should I forego DeleteAllItems() and manually
DeleteItem()s one by one (which would require knowledge of
the highest index in order to use the web-suggested approach
of starting from the bottom of the list) ?

Karsten

···

On Thu, Jan 24, 2013 at 10:52:30AM +0100, Karsten Hilbert wrote:

from time to time -- under not yet fully understood
circumstances -- the following code snippet actually
succeeds in throwing an exception in our application
(www.gnumed.de) running under wxPython 2.8.12.1 on
Debian (self is a wx.ListCtrl child):

  def set_string_items(self, items = None):
    """All item members must be unicode()able or None."""

    self.DeleteAllItems()
    if self.ItemCount != 0:
      raise ValueError('.ItemCount not 0 after .DeleteAllItems()')

--
GPG key ID E4071346 @ gpg-keyserver.de
E167 67FD A291 2BEA 73BD 4537 78B9 A9F9 E407 1346

Gadget_Steve · January 24, 2013, 2:32pm

Why not replace the whole DeleteAllItems with self.ClearAll() it
would seem simpler?

···

On 24/01/13 11:35, Karsten Hilbert
wrote:

On Thu, Jan 24, 2013 at 10:52:30AM +0100, Karsten Hilbert wrote:


	from time to time -- under not yet fully understood
circumstances -- the following code snippet actually
succeeds in throwing an exception in our application
() running under wxPython 2.8.12.1 on
Debian (self is a wx.ListCtrl child):
def set_string_items(self, items = None):
"""All item members must be unicode()able or None."""
self.DeleteAllItems()
if self.ItemCount != 0:
raise ValueError('.ItemCount not 0 after .DeleteAllItems()')


Or maybe I should not trust self.ItemCount "too soon" (the
web talks about it being updated only "later").
As to why this self.ItemCount check is there in the first place:
We've initially had an exception where we ended up with
double the list items than we expected (despite calling
DeleteAllItems() each time before setting items).
Is there a way other than .ItemCount which tells me the
current number of items ?
Should I
while self.ItemCount > 0:
wx.SafeYield()
?
Or should I forego DeleteAllItems() and manually
DeleteItem()s one by one (which would require knowledge of
the highest index in order to use the web-suggested approach
of starting from the bottom of the list) ?
Karsten

–
Steve Gadget Barnes

www.gnumed.de

Karsten_Hilbert · January 24, 2013, 2:36pm

Because I don't want to delete columns.

Also,

self.DeleteAllItems()

is not any simpler than

self.ClearAll()

Karsten

···

On Thu, Jan 24, 2013 at 02:32:59PM +0000, Steve Barnes wrote:

>Or should I forego DeleteAllItems() and manually
>DeleteItem()s one by one (which would require knowledge of
>the highest index in order to use the web-suggested approach
>of starting from the bottom of the list) ?
>
>Karsten
Why not replace the whole DeleteAllItems with self.ClearAll()

--
GPG key ID E4071346 @ gpg-keyserver.de
E167 67FD A291 2BEA 73BD 4537 78B9 A9F9 E407 1346

Karsten_Hilbert · January 24, 2013, 2:39pm

The other way round, that is.

Karsten

···

On Thu, Jan 24, 2013 at 03:36:43PM +0100, Karsten Hilbert wrote:

Also,

self.DeleteAllItems()

is not any simpler than

self.ClearAll()

--
GPG key ID E4071346 @ gpg-keyserver.de
E167 67FD A291 2BEA 73BD 4537 78B9 A9F9 E407 1346

Werner1 · January 24, 2013, 2:49pm

Maybe also check the return value of DeleteAllItems, if False something went wrong during delete, to help narrow down the problem.

Werner

···

On 24/01/2013 10:52, Karsten Hilbert wrote:

Hi all,

from time to time -- under not yet fully understood
circumstances -- the following code snippet actually
succeeds in throwing an exception in our application
(www.gnumed.de) running under wxPython 2.8.12.1 on
Debian (self is a wx.ListCtrl child):

  def set_string_items(self, items = None):
    """All item members must be unicode()able or None."""

    self.DeleteAllItems()
    if self.ItemCount != 0:
      raise ValueError('.ItemCount not 0 after .DeleteAllItems()')

Tim_Roberts · January 24, 2013, 6:07pm

Karsten Hilbert wrote:

3) two threads get in the way of each other trying to
   use set_string_items()

  A race *could* happen between
    self.DeleteAllItems()
  and
    self.ItemCount

  Yes, we do have threads but, no, we are taking care to
  wrap them in wx.CallAfter() as needed (might have missed
  something somewhere).

If you are using CallAfter, then you aren't really using multiple
threads, and that's how it must be. Anything that tickles the user
interface state MUST happen in the UI thread.

···

--
Tim Roberts, timr@probo.com
Providenza & Boekelheide, Inc.

Karsten_Hilbert · January 24, 2013, 6:56pm

I know. That's why we are using wx.CallAfter() in the first
place. Nonetheless, one can forget to wrap a call site or
two. Which is why I am looking into thread IDs, currently.

Karsten

···

On Thu, Jan 24, 2013 at 10:07:19AM -0800, Tim Roberts wrote:

> 3) two threads get in the way of each other trying to
> use set_string_items()
>
> A race *could* happen between
> self.DeleteAllItems()
> and
> self.ItemCount
>
> Yes, we do have threads but, no, we are taking care to
> wrap them in wx.CallAfter() as needed (might have missed
> something somewhere).

If you are using CallAfter, then you aren't really using multiple
threads, and that's how it must be. Anything that tickles the user
interface state MUST happen in the UI thread.

--
GPG key ID E4071346 @ gpg-keyserver.de
E167 67FD A291 2BEA 73BD 4537 78B9 A9F9 E407 1346

Robin · January 24, 2013, 7:17pm

Karsten Hilbert wrote:

from time to time -- under not yet fully understood
circumstances -- the following code snippet actually
succeeds in throwing an exception in our application
(www.gnumed.de) running under wxPython 2.8.12.1 on
Debian (self is a wx.ListCtrl child):

  def set_string_items(self, items = None):
    """All item members must be unicode()able or None."""

    self.DeleteAllItems()
    if self.ItemCount != 0:
      raise ValueError('.ItemCount not 0 after .DeleteAllItems()')

Or maybe I should not trust self.ItemCount "too soon" (the
web talks about it being updated only "later").

I don't see anything in the code for the generic listctrl (which is the one used on wxGTK) that is delaying any part of the delete-all to idle time or anything like that. It boils down to basically just sending a notification and clearing the list of list items. The GetItemCount simply returns the length of that list.

If you can consistently duplicate this problem in a small sample then I would be happy to be proven wrong, but in the meantime I'm thinking thats your speculation about a rogue thread interaction is probably the problem.

···

On Thu, Jan 24, 2013 at 10:52:30AM +0100, Karsten Hilbert wrote:

--
Robin Dunn
Software Craftsman

Karsten_Hilbert · February 3, 2013, 5:47pm

I think I have proven myself wrong:

Tracking thread.get_ident() when the exception occurs always
shows one and the same thread, the main (GUI/wxpython)
thread.

And, no, I haven't been able to create a small sample - not
even been able to consistently duplicate the exception. It
seems to depend on complex interactions of

  - scheduling data reload when certain signals are
    received from a PostgreSQL database (sets a flag
    checked at EVT_PAINT time)

- yes, the database listener runs in a different
thread but it does not touch the GUI AFAICT

  - reloading data from the database on EVT_PAINT
    -> this always runs in the main thread because
       it is triggered by wxPython itself
    -> should I just wx.CallAfter() reloading from there ?

- also reloading data on EVT_NOTEBOOK_PAGE_CHANGED

  - both effects (EVT_NOTEBOOK_PAGE_CHANGED and EVT_PAINT)
    appearing to get on top of each other DESPITE
    proven to be running in the same thread !?!

  - the user effecting scheduling of data reload
    from elsewhere in the app by changing data while
    the list in question is not in view and also
    rapidly switching notebook pages while at it

  - a potential timing issue may be confounded by
    the fact that there are typically at least
    50 items to be read from the database (which
    does run over UNIX domain sockets on the local
    machine, however), those items are, of course,
    retrieved in one SQL go, not one by one

The corresponding snippet from above now reads:

def set_string_items(self, items=None):
"""All item members must be unicode()able or None."""

    if self.debug is not None:
      _log.debug('GetItemCount() before DeleteAllItems(): %s (%s, thread [%s])', self.GetItemCount(), self.debug, thread.get_ident())
    if not self.DeleteAllItems():
      _log.debug('DeleteAllItems() failed (%s)', self.debug)
    item_count = self.GetItemCount()
    if self.debug is not None:
      _log.debug('GetItemCount() after DeleteAllItems(): %s (%s)', item_count, self.debug)
    if item_count != 0:
      _log.debug('GetItemCount() not 0 after DeleteAllItems() (%s)', self.debug)
    ...

(where self.debug let's me pinpoint individual listctrls as
needed)

This does keep logging things like

"GetItemCount() not 0 after DeleteAllItems() (provider inbox)"

from time to time which will then lead to a second set of
list items gladly being added onto the listctrl while the
corresponding set of list item data (which my listctrl is
tracking itself) is properly set to only the first set of
items. Which later on leads to a follow-on exception when a
lower-down list item is activated whereby its associated
data is attempted to be retrieved but does not exist.

If anyone has got more ideas for what to look at I'd be
indebted. Meanwhile I'm not reloading on the
EVT_NOTEBOOK_PAGE_CHANGED and logging problems in search for
more clues.

Karsten

···

On Thu, Jan 24, 2013 at 11:17:31AM -0800, Robin Dunn wrote:

>On Thu, Jan 24, 2013 at 10:52:30AM +0100, Karsten Hilbert wrote:
>
>>from time to time -- under not yet fully understood
>>circumstances -- the following code snippet actually
>>succeeds in throwing an exception in our application
>>(www.gnumed.de) running under wxPython 2.8.12.1 on
>>Debian (self is a wx.ListCtrl child):
>>
>> def set_string_items(self, items = None):
>> """All item members must be unicode()able or None."""
>>
>> self.DeleteAllItems()
>> if self.ItemCount != 0:
>> raise ValueError('.ItemCount not 0 after .DeleteAllItems()')
>
>Or maybe I should not trust self.ItemCount "too soon" (the
>web talks about it being updated only "later").

I don't see anything in the code for the generic listctrl (which is
the one used on wxGTK) that is delaying any part of the delete-all to
idle time or anything like that. It boils down to basically just
sending a notification and clearing the list of list items. The
GetItemCount simply returns the length of that list.

If you can consistently duplicate this problem in a small sample then
I would be happy to be proven wrong, but in the meantime I'm thinking
thats your speculation about a rogue thread interaction is probably
the problem.

--
GPG key ID E4071346 @ gpg-keyserver.de
E167 67FD A291 2BEA 73BD 4537 78B9 A9F9 E407 1346