Efficient large file processing displaying

wxPython Users
#1

Hello,

Just to give a little background.
The purpose of the GUI that was built is to process large text files and extract certain lines from those files. Those saved lines are then presented through a Panel / TextCtrl to the user for viewing / further manipulation if needed.

The extracted lines can amount to a large lists.
When I then try to display those many lines in the TxtCtrl this actually takes comparatively a lot of time (>20s). I.e. doing SetValue on the TxtCtrl is what takes the most time due to large amount of text.

How would you approach to efficiently / quickly display large amount of text?

I was thinking perhaps splitting up the large lists of text into smaller chunks, and then displaying those, but I’m not 100% sure on the best way to implement this approach either.

Your help is greatly appreciated.

Thank you

#2

Copied from another thread:

[…]have a look at Grid and ListCtrl.
They can be created as virtual grid / list such that they will request the required data on demand.

For the ListCtrl, hold out for the style wx.LC_VIRTUAL . For the grid, you need to implement your own DataTable and tell the grid to use it with the SetTable method.

See the demos: Grid_MegaExample.py, GridHugeTable.py, ListCtrl_virtual.py

1 Like
#3

Building on @DietmarSchwertberger’s answer a little, the general idea is to not put all that data into the widget all at the same time. Rather, using the ListCtrl in wx.LC_VIRTUAL mode approach, you tell the listctrl how many items are in your data list, and then it will ask you, via calling overloaded methods, when it needs some of those data items for updating the display on the list. So instead of needing to preload thousands of lines of data in the with widget, it will ask you for a few at a time (depending on how many rows are visible.)

The UI/UX won’t be exactly like a TextCtrl, and editing in place would not be easy to do, but when viewing and editing large amounts of data it’s not unreasonable to view things as a list and require something like pressing enter or double-clicking an item in order to open a dialog or something to edit that line.

1 Like