I have a very large sqlite database that I need to map to a wxgrid. I’ll be using a gridtable to help me, as I can’t load the entire database in memory at one time. I also need to write a middleman between the database and the gridtable that can cache x number of tuples from the database so that I don’t query the database for every cell’s GetValue().
I need to know how many rows I should cache from the database. Is there a way to find out the “page size” of a gridtable ?
Btw, by “page size” I mean how many rows are loaded in the grid by the gridtable at each scroll step. I will probably decide to cache the page row count multiplied by a factor that I’m going to discover through further testing.
If anyone has extra input on how to would solve such a problem, please post it.
Well, that was simple, I printed to row indexes to console and it seems that the table grid only requests as many rows as the current size of the grid can display.
···
On Tuesday, 19 November 2013 14:23:38 UTC+2, Alexandru Popa wrote:
I have a very large sqlite database that I need to map to a wxgrid. I’ll be using a gridtable to help me, as I can’t load the entire database in memory at one time. I also need to write a middleman between the database and the gridtable that can cache x number of tuples from the database so that I don’t query the database for every cell’s GetValue().
I need to know how many rows I should cache from the database. Is there a way to find out the “page size” of a gridtable ?
Btw, by “page size” I mean how many rows are loaded in the grid by the gridtable at each scroll step. I will probably decide to cache the page row count multiplied by a factor that I’m going to discover through further testing.
If anyone has extra input on how to would solve such a problem, please post it.
Take a look at the virtual grid methods in the demo - you will find
that it requests a start row, end row range, etc., from your
accessor(s).
Gadget/Steve
···
On 19/11/13 12:23, Alexandru Popa
wrote:
I have a very large sqlite database that I need to
map to a wxgrid. I’ll be using a gridtable
to help me, as I can’t load the entire database in memory at one
time. I also need to write a middleman between the database and
the gridtable that can cache x number of tuples from the
database so that I don’t query the database for every cell’s
GetValue().
I need to know how many rows I should cache from the database.
Is there a way to find out the “page size” of a gridtable ?
Btw, by "page size" I mean how many rows are loaded in the grid
by the gridtable at each scroll step. I will probably decide to
cache the page row count multiplied by a factor that I’m going
to discover through further testing.
If anyone has extra input on how to would solve such a problem,
please post it.
–
You received this message because you are subscribed to the Google
Groups “wxPython-users” group.
To unsubscribe from this group and stop receiving emails from it,
send an email to .
For more options, visit .
wxpython-users+unsubscribe@googlegroups.com
https://groups.google.com/groups/opt_out
I’ve hit a snag. I managed to fetch data from the database, but scrolling is slow, as whenever I need to supply a row from outside the cached buffer I do a database query.
I need some pointers on how to make scrolling smooth. I’m thinking of dispatching the database query to a separate thread that watches over a variable representing the latest requested row, or maybe activate the thread by an event instead of always checking the current row. Has anyone tried something similar?
···
On Tuesday, 19 November 2013 14:23:38 UTC+2, Alexandru Popa wrote:
I have a very large sqlite database that I need to map to a wxgrid. I’ll be using a gridtable to help me, as I can’t load the entire database in memory at one time. I also need to write a middleman between the database and the gridtable that can cache x number of tuples from the database so that I don’t query the database for every cell’s GetValue().
I need to know how many rows I should cache from the database. Is there a way to find out the “page size” of a gridtable ?
Btw, by “page size” I mean how many rows are loaded in the grid by the gridtable at each scroll step. I will probably decide to cache the page row count multiplied by a factor that I’m going to discover through further testing.
If anyone has extra input on how to would solve such a problem, please post it.
I've hit a snag. I managed to fetch data from the database, but scrolling
is slow, as whenever I need to supply a row from outside the cached buffer
I do a database query.
I need some pointers on how to make scrolling smooth. I'm thinking of
dispatching the database query to a separate thread that watches over a
variable representing the latest requested row, or maybe activate the
thread by an event instead of always checking the current row. Has anyone
tried something similar?
I haven't yet displayed large datasets in a grid, but I am about to... so here's my take on it:
I wonder if scrolling if the right approach.
The user wanting to see rows from the middle of the data would have to scroll down a lot.
For my project, I'm planning on chunking the data into multiple tables (by year in my case)
and have buttons for calling up a grid per year.
So the user doesn't have to scroll 1000s of rows to get to see his data,
and from a usability point a loading delay on opening a new screen is acceptable
(at least more acceptable than delayed scrolling).
Regards,
Michael
···
On Fri, 22 Nov 2013 11:59:19 +0100, Alexandru Popa <a.ephesos@gmail.com> wrote:
On Tuesday, 19 November 2013 14:23:38 UTC+2, Alexandru Popa wrote:
I have a very large sqlite database that I need to map to a wxgrid. I'll
be using a gridtable<Weiterleitungshinweis help me, as I can't load the entire database in memory at one time. I
also need to write a middleman between the database and the gridtable that
can cache x number of tuples from the database so that I don't query the
database for every cell's GetValue().
I need to know how many rows I should cache from the database. Is there a
way to find out the "page size" of a gridtable ?
Btw, by "page size" I mean how many rows are loaded in the grid by the
gridtable at each scroll step. I will probably decide to cache the page row
count multiplied by a factor that I'm going to discover through further
testing.
If anyone has extra input on how to would solve such a problem, please
post it.
--
Erstellt mit Operas E-Mail-Modul: Opera Web Browser | Faster, Safer, Smarter | Opera
Just have a thread for updating the cache and set reasonable buffer size and fetch points, e.g. if your current view is records from 100..130 you should have records 40..190 in memory and if the user scrolls up to 130..160 your background task should be fetching 190..220, etc., some playing with the numbers should get it smooth.
···
On 22/11/13 10:59, Alexandru Popa wrote:
I've hit a snag. I managed to fetch data from the database, but scrolling is slow, as whenever I need to supply a row from outside the cached buffer I do a database query.
I need some pointers on how to make scrolling smooth. I'm thinking of dispatching the database query to a separate thread that watches over a variable representing the latest requested row, or maybe activate the thread by an event instead of always checking the current row. Has anyone tried something similar?
This is a fine idea for when user may be actually reading, or at last
scanning, as they scroll.
but i the user wants to quickly jump from record 0 to line 1 million, you
want a way for them to do that without fetching all the records in between.
If you use the scrollbar as the sole navigation, you'll want to see what
happens if they drag the scroll bar from near the top to tner the bottom
(or similar). If you get a bunch of scroll events, you'll need to do
something smart -- maybe a short timeout, and only do the query if there
isn't another call for anew batch right away.
Just something to look out for.
-Chris
···
On Fri, Nov 22, 2013 at 12:17 PM, Steve Barnes <gadgetsteve@live.co.uk>wrote:
Just have a thread for updating the cache and set reasonable buffer size
and fetch points, e.g. if your current view is records from 100..130 you
should have records 40..190 in memory and if the user scrolls up to
130..160 your background task should be fetching 190..220, etc., some
playing with the numbers should get it smooth.
--
Christopher Barker, PhD
Python Language Consulting
- Teaching
- Scientific Software Development
- Desktop GUI and Web Development
- wxPython, numpy, scipy, Cython