Threading, database operation and no-blocking gui

hi to all,

I have some doubts on the management of the threaths.

I have created a simple application in which I make to see as I use these when I must recover recordset from a database to populate a listbox.

Of intention I have used a listctrl instead that a virtual to slow down everything.

In practice when we launch the application a database and a table is created and this is populated with 100000 records

The class WorkerThread call the init of DBMS class that create the db and populate the table, then through always this class (DBMS)

execute a query on the table just created and with wx.CallAfter method it returns recordset to the listbox.

Now the questions are:

this is the correct way to use the WorkerThread class?

there is difference among to use run or start methods of WorkerThread class?

why,during the database operation, if I click the OK button I must wait for the end of the operations to see the tied up message to this.

the sequence of the events is this

OnOpen => OnClick => wx.ID_REFRESH =>self.acquire_data(call WorkerThread class that call on init method of DBMS class,create, populate db and retrive recordset)=>wx.CallAfter=>show_data(populate the listctrl)

regards

beppe

simpleapp.py (5.96 KB)

beppe wrote:

In practice when we launch the application a database and a table is
created and this is populated with 100000 records

The class WorkerThread call the init of DBMS class that create the db
and populate the table, then through always this class (DBMS)
execute a query on the table just created and with wx.CallAfter method
it returns recordset to the listbox.

Now the questions are:
this is the correct way to use the WorkerThread class?

No, at least not to achieve the results you want.

there is difference among to use run or start methods of WorkerThread
class?

They have different purposes. "run" is a function that YOU implement,
which does the real work of the thread. "start" is function provided by
the Thread object that actually creates a thread and then calls the
"run" function. If you call the "run" function yourself, it will do the
work, but it will not be done in a separate thread. To get threading
involved, you have to call "start".

And that's a part of your problem. You are doing all of your long
database work in the __init__ function. That function will get called
when you create the object:
        task = WorkerThread(self, args[0], args[1], args[2])
BUT it's going to run in the same thread -- not in a different thread.
That statement will not finish until the database is initialized. Only
then do you call task.start, which is what starts the thread running,
but the only thing you are doing in another thread is the CallAfter
function. Clearly, that's not what you wanted.

To have the database creation done in its own thread, you need to move
the call to self.make_db() out of __init__ so that it gets called during
run(). That's going to mean a little bit of re-organizing on your part,
because you won't be able to run on_query during __init__ either. For
example, you might have WorkerThread.__init__ remember its arguments in
a list, then use that list in run. For example (and this is only a
crude example):

    class DBMS(object):
        self.open_connection()
        # DO NOT call self.make_db here!
        ...

    class WorkerThread(threading.Thread,DBMS):
        def __init__(self, window, which, sql, args):
            threading.Thread.__init__(self)
            #create a db and a table with 100000 records!!!!!am'I quite
blocker?
            DBMS.__init__(self)

            self.window = window
            #which widgets is calling?
            self.which = which
            # Remember what we need to do.
            self.task = (sql, args)
       
    def run(self):
        # Go create the database.
        self.make_db()
        # Now go do a query on that data.
        sql,args = self.task
        rs = self.on_query(True, sql, args)
        wx.CallAfter(self.window.show_data, self.which, rs)

There are a lot of ways this could be extended, depending on what you
need to do. You could, for example, have "self.task" be a list of
queries that need to be made. Then, you could add a "submitTask" method
to add something to that list of tasks, and have the "run" method sit in
a loop and wait for a job to do. The Python Queue object is a better
choice for something like that,

why,during the database operation, if I click the OK button I must
wait for the end of the operations to see the tied up message to this.

Because the "acquire_data" function will not finish until the database
has been created, and that creating is happening in the same thread.

the sequence of the events is this

OnOpen => OnClick => wx.ID_REFRESH =>self.acquire_data(call
WorkerThread class that call on init method of DBMS class,create,
populate db and retrive recordset)=>wx.CallAfter=>show_data(populate
the listctrl)

That's sort of correct, but you have other issues here. You are calling
OnOpen before you even call MainLoop. That means your windows won't
actually exist yet. Remember that these operating systems (Windows and
X) are all event driven. When you create a window, all that does is
send a message to the window saying "go create yourself". That message
sits in a message queue until someone pulls it out and sends it off to a
window handler. In wxPython, the "someone" that does that is
app.MainLoop.

So, here's exactly what happens:
    MyForm.OnOpen --> queues up a "click" message on ID_REFRESH
    app.MainLoop --> starts handling messages, eventually sees the click
    MyForm.OnClick
        MyForm.acquire_data
            WorkerThread.__init__
                DBMS.__init__
                    DBMS.open_connection
                    DBMS.make_db
                    ... long wait ...
                DBMS.on_query
            # Note that EVERYTHING up to here has been done on the main
thread
            WorkerThread.start --> here's where it creates another thread
                WorkerThread.run --> running in a new thread
                    CallAfter --> but it only makes one call!
    app.MainLoop
        MyForm.show_data --> triggered by the CallAfter

I think you can see this is not what you wanted.

···

--
Tim Roberts, timr@probo.com
Providenza & Boekelheide, Inc.

hi TIm,

I have thought over a lot on what you have written me.

I have made some changes to the script even if the management of the thread still appears me dark…

I have moved the creation of the database in an external function to try to simplify the example

the thread should now deal only with to recover the records from the database,

but if I use

task.run

it works even if the gui blocks itself on on_show() function

and if i use

task.start

I receive the error

<class ‘sqlite3.ProgrammingError’>

SQLite objects created in a thread can only be used in that same thread.The object was created in thread id 140724934530816 and this is thread id 140724715284224

<traceback object at 0x3359518>

I am a little confused

where I can deepen this matter? because for it misses me to me some piece…

simpleapp.py (6.45 KB)

···

Il giorno lunedì 22 aprile 2013 18:44:08 UTC+2, beppe ha scritto:

hi to all,

I have some doubts on the management of the threaths.

I have created a simple application in which I make to see as I use these when I must recover recordset from a database to populate a listbox.

Of intention I have used a listctrl instead that a virtual to slow down everything.

In practice when we launch the application a database and a table is created and this is populated with 100000 records

The class WorkerThread call the init of DBMS class that create the db and populate the table, then through always this class (DBMS)

execute a query on the table just created and with wx.CallAfter method it returns recordset to the listbox.

Now the questions are:

this is the correct way to use the WorkerThread class?

there is difference among to use run or start methods of WorkerThread class?

why,during the database operation, if I click the OK button I must wait for the end of the operations to see the tied up message to this.

the sequence of the events is this

OnOpen => OnClick => wx.ID_REFRESH =>self.acquire_data(call WorkerThread class that call on init method of DBMS class,create, populate db and retrive recordset)=>wx.CallAfter=>show_data(populate the listctrl)

regards

beppe

beppe wrote:
>

  > ...the thread should now deal only with to recover the

records from the database,

  > but if I use

  >     task.run

  > it works  even if the gui blocks itself on on_show() function

  >

  > and if i use

  >     task.start

  > I receive the error

  > <class 'sqlite3.ProgrammingError'>

  > SQLite objects created in a thread can only be used in that

same thread.The object was created in thread id 140724934530816
and this is thread id 140724715284224

  >

  > I am  a little confused

  > where I can deepen this matter? because for it misses me to

me some piece…

The error message is pretty clear.  SQLite is not thread-safe.  You

cannot open an SQLite database in one thread and use it in another
thread. SQLite is not intended to be a server-class product. It is
lightweight and limited.

So, you have a few choices.

1. You could forget about using threads.  If your database is small,

the performance won’t be noticeable. If your database is large,
then SQLite may not the right choice to begin with.
2. You could write your own SQLite database server thread inside
your app. Have a single thread that does all the database
operations, and use something like the Python Queue module to send
commands to it from your frame.
3. You could switch to a different database. Postgres and MySQL
both have excellent support in Python. Depending on what you are
storing, perhaps you could even use one of the new non-SQL
databases, like MongoDB.

···

timr@probo.com

  1. You could write your own SQLite database server thread inside your app. Have a single
    thread that does all the database operations, and use something like the Python Queue

module to send commands to it from your frame.

OMG this is a fantastic idea. Thank you.

In fact, thanks to everyone on this list-- I’m lurker, but the ideas offered here are fantastic. I hope I’m smart enough one day to participate.

···

On Tue, Apr 23, 2013 at 11:56 AM, Tim Roberts timr@probo.com wrote:

beppe wrote:

>

  > ...the thread should now deal only with to recover the

records from the database,

but if I use

  >     task.run

  > it works  even if the gui blocks itself on on_show() function

  >

  > and if i use

  >     task.start

  > I receive the error

  > <class 'sqlite3.ProgrammingError'>

  > SQLite objects created in a thread can only be used in that

same thread.The object was created in thread id 140724934530816
and this is thread id 140724715284224

  >

I am a little confused

  > where I can deepen this matter? because for it misses me to

me some piece…

The error message is pretty clear.  SQLite is not thread-safe.  You

cannot open an SQLite database in one thread and use it in another
thread. SQLite is not intended to be a server-class product. It is
lightweight and limited.

So, you have a few choices.



1. You could forget about using threads.  If your database is small,

the performance won’t be noticeable. If your database is large,
then SQLite may not the right choice to begin with.

2. You could write your own SQLite database server thread inside

your app. Have a single thread that does all the database
operations, and use something like the Python Queue module to send
commands to it from your frame.

3. You could switch to a different database.  Postgres and MySQL

both have excellent support in Python. Depending on what you are
storing, perhaps you could even use one of the new non-SQL
databases, like MongoDB.

--

Tim Roberts, timr@probo.com

Providenza & Boekelheide, Inc.

You received this message because you are subscribed to the Google Groups “wxPython-users” group.

To unsubscribe from this group and stop receiving emails from it, send an email to wxpython-users+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

beppe wrote:

>

  > ...the thread should now deal only with to recover the

records from the database,

but if I use

  >     task.run

  > it works  even if the gui blocks itself on on_show() function

  >

  > and if i use

  >     task.start

  > I receive the error

  > <class 'sqlite3.ProgrammingError'>

  > SQLite objects created in a thread can only be used in that

same thread.The object was created in thread id 140724934530816
and this is thread id 140724715284224

  >

I am a little confused

  > where I can deepen this matter? because for it misses me to

me some piece…

The error message is pretty clear.  SQLite is not thread-safe.  You

cannot open an SQLite database in one thread and use it in another
thread. SQLite is not intended to be a server-class product. It is
lightweight and limited.

So, you have a few choices.



1. You could forget about using threads.  If your database is small,

the performance won’t be noticeable. If your database is large,
then SQLite may not the right choice to begin with.

2. You could write your own SQLite database server thread inside

your app. Have a single thread that does all the database
operations, and use something like the Python Queue module to send
commands to it from your frame.

3. You could switch to a different database.  Postgres and MySQL

both have excellent support in Python. Depending on what you are
storing, perhaps you could even use one of the new non-SQL
databases, like MongoDB.

I use SQLAlchemy to access sqlite database from multiple different threads, which works brilliantly, as a forth option.

···

On 23 April 2013 17:56, Tim Roberts timr@probo.com wrote:

Tim Roberts, timr@probo.com

Providenza & Boekelheide, Inc.

You received this message because you are subscribed to the Google Groups “wxPython-users” group.

To unsubscribe from this group and stop receiving emails from it, send an email to wxpython-users+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

You ARE participating, thank you very much!
And the answers you get here are helping others, too!
Rufus

···

On 4/23/2013 1:32 PM, edoc wrote:

     2. You

could write your own SQLite database server thread inside your
app. Have a single
>thread
that does all the database operations, and use something
like the Python Queue

        >module

to send commands to it from your frame.

        OMG this

is a fantastic idea. Thank you.

        In fact,

thanks to everyone on this list-- I’m lurker, but the ideas
offered here are fantastic. I hope I’m smart enough one day
to participate.