Reduce loading time

wxPython Users
1

Hello all,
The attached script shows a process that too long to load and respond. This program adds all pictures in the Pictures folder to wx.BoxSizer. Here, the path is set to - /home/user/Pictures. This will work on all GNU/Linux distros. If you are not using a GNU/Linux distro, change the path and test the script (Ln : 14, Col : 31).

Lets, come to the point. I need to reduce the time of this process. I tried with threading, multiprocess modules and with our wx.Timer. But, I failed to run it successfully. Please help. Thanks in advance.

process.py (701 Bytes)

2

Hello all,
The attached script shows a process that too long to load and respond. This program adds all pictures in the Pictures folder to wx.BoxSizer. Here, the path is set to - /home/user/Pictures. This will work on all GNU/Linux distros. If you are not using a GNU/Linux distro, change the path and test the script (Ln : 14, Col : 31).

Lets, come to the point. I need to reduce the time of this process. I tried with threading, multiprocess modules and with our wx.Timer. But, I failed to run it successfully. Please help. Thanks in advance.

Loading a lot of images like that will cause memory overload if you don’t have enough memory.
Are you running 64 bit. Do you have 64GB+ RAM?
Sorry most people dont.

Look into building the list of images, then loading them 1-3 at a time. Not all at once.

As far as optimizing, use generators, or generator functions.
Also make everything such variables preferably local, but if not global. They access much faster.

local vars is faster than global vars(wow factor) and global vars is faster than library lookup vars(meh).
Ex: osSep = os.sep
Remember: less dots = better performance

so for example your code

for bmp in os.listdir(‘%s/My Documents/My Pictures/Girls’ % (os.path.expanduser(‘~’))):
vbox.Add(wx.StaticBitmap(self.pane, wx.NewId(), wx.Bitmap(‘%s/Pictures/%s’ % (os.path.expanduser(‘~’), bmp), wx.BITMAP_TYPE_ANY)))

could be optimized something like so…

osListDir = os.listdir
wxBitmap = wx.Bitmap
wxStaticBitmap = wx.StaticBitmap
wxBITMAP_TYPE_ANY = wx.BITMAP_TYPE_ANY
selfPane = self.pane
expandUser = os.path.expanduser(‘~’)

Here is where the loop starts and makes and adds the staticbitmaps to the sizer.

[vbox.Add(wxStaticBitmap(selfPane, -1, wxBitmap(‘%s/Pictures/%s’ % (expandUser, bmp), wxBITMAP_TYPE_ANY))
for bmp in osListDir(‘%s/Pictures’ % expandUser)]

In the above example there is 1 dot(period). Inbetween vbox and Add.

If you absolutely need wx.NewId() calls for some odd reason, this will slow it down considerably more.
If code placed in a def, the vars defined in the def will be garbage collected(when function is done executing) and only used when needed, ex: on first/only call.
Hope this basic example helps.

···

On Saturday, December 14, 2013 9:19:21 AM UTC-6, Navaneeth wrote:

3

Also if all the Girls don’t want to come to the party, then you can be selective with range() or filter certain blondish-types out with a colorrange_filter/wildcard_filter.

Also remember that dictionaries loads faster than lists, tuples. And also accessing a tuple is faster than accessing a list.

If filetype is your motive, then change the BITMAP_TYPE or .ext lookup.

4

I am expecting an answer related to threads or timers. Your answer don’t finds a solution. I tested it. But, it also takes time and becomes more slow. i.e, local vars is not faster than global vars. I need to add almost 75 images to my panel. Expecting a better solution. Thank you for your reply.

5

Well, it seems like doing anything graphic related in a thread usually causes crash, so either try building the list of image paths, and load them incrementally depending on the ones that are supposed to be on the screen(the viewable ones) depending on scroll position,

or

Try looking into wx.lib.agw ThumbnailCtrl. I recall it does threading stuff with the images. There is a option for using PIL also in their somewhere.

If I where you I would look into ThumbnailCtrl first for a solution, then consider other possible routes.

6

I beg to differ there… Can you show a timeit example proving otherwise?
import timeit
Tests: Loaded once and multiple times.

If global is really faster than local, then it would be quicker
if I flew around the world in a hot air balloon in 80 days, when maybe I only needed to walk 10 feet/meters to ask my neighbor a question, right?
Sorry, someone better that I (the python devs/BDFL have proven this point different(let alone they write the language and have provided you options))

Please provide a small sample app as most do. It really helps around here to understand what the end result you want will be.

···

On Sunday, December 15, 2013 12:14:57 AM UTC-6, Navaneeth wrote:

i.e, local vars is not faster than global vars.

7

Navaneeth wrote:

I am expecting an answer related to threads or timers. Your answer
don't finds a solution. I tested it. But, it also takes time and
becomes more slow. i.e, local vars is not faster than global vars. I
need to add almost 75 images to my panel. Expecting a better solution.

You can't use multithreading for this -- all user interface stuff must
be done from the main thread.

Look, there is no magic solution. When you have to load, resize and
render 75 images, that's going to take time. There's no two ways about
it. The best you can do is reduce the PERCEIVED time. What I mean by
that is to let the application come up without the images, so that a
window pops right up. Then, using something like a periodic timer, add
the images a few at a time. The user isn't as likely to notice the delay.

So, you could do your listdir into a list, then create a 100ms timer.
Then, each time through the timer, process a couple of the images from
the list.

···

--
Tim Roberts, timr@probo.com
Providenza & Boekelheide, Inc.

8

Either way, your point is utterly irrelevant to the problem at
hand. His program is slow because it is reading, parsing and
rendering 75 external image files, not because of
micro-optimizations like this.

···

Metallicow wrote:

    On Sunday, December 15, 2013 12:14:57 AM UTC-6, Navaneeth wrote:

        i.e, local vars is not faster than global

vars.

      I beg to differ there... Can you show a timeit example

proving otherwise?

-- Tim Roberts, Providenza & Boekelheide, Inc.

timr@probo.com

9

There may be something to be said for just rethinking the idea of adding 75 images to one panel, particularly all at once. If they are small and compressed thumbnails, OK, maybe, but then their size shouldn’t burden the loading as much. If they are not small thumbnails, why would you possibly want to do that?

···

On Mon, Dec 16, 2013 at 3:43 PM, Tim Roberts timr@probo.com wrote:

Navaneeth wrote:

I am expecting an answer related to threads or timers. Your answer

don’t finds a solution. I tested it. But, it also takes time and

becomes more slow. i.e, local vars is not faster than global vars. I

need to add almost 75 images to my panel. Expecting a better solution.

You can’t use multithreading for this – all user interface stuff must

be done from the main thread.

Look, there is no magic solution. When you have to load, resize and

render 75 images, that’s going to take time. There’s no two ways about

it. The best you can do is reduce the PERCEIVED time. What I mean by

that is to let the application come up without the images, so that a

window pops right up. Then, using something like a periodic timer, add

the images a few at a time. The user isn’t as likely to notice the delay.

So, you could do your listdir into a list, then create a 100ms timer.

Then, each time through the timer, process a couple of the images from

the list.

–

Tim Roberts, timr@probo.com

Providenza & Boekelheide, Inc.

–

You received this message because you are subscribed to the Google Groups “wxPython-users” group.

To unsubscribe from this group and stop receiving emails from it, send an email to wxpython-users+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

10

I agree. The last time someone asked about doing something like this on this group, the recommendation was to do some pre-processing where the program created a set of thumbnails and loaded those. Then the next time you run the program, it would just load the thumbnails it finds and you could use a background thread to check for new additions and create new thumbnails and add them to the display at well.

Adding many full resolution image files is probably a bad idea. I don’t think any of the photo editing software I use does that. They all do thumbnails as well.

Speaking of thumbnails, wxPython has a thumbnail control that you could use.

Mike

···

On Monday, December 16, 2013 2:50:41 PM UTC-6, Che M wrote:

There may be something to be said for just rethinking the idea of adding 75 images to one panel, particularly all at once. If they are small and compressed thumbnails, OK, maybe, but then their size shouldn’t burden the loading as much. If they are not small thumbnails, why would you possibly want to do that?

11

wxPython has a thumbnail control. It’s impressive and extremely fast. But, it’s not much attractive. Please tell me how can I implement that algorithm in my program. I am also showing another script which is so fast. This is not written by me. I need your help to implement it on my panel. I need to launch a huge project on the basis of this.

mpthumb.py (9.71 KB)

12

wxPython has a thumbnail control. It's impressive and extremely fast. But, it's not much attractive.

What is not attractive about it?

Please tell me how can I implement that algorithm in my program. I am also showing another script which is so fast. This is not written by me. I need your help to implement it on my panel. I need to launch a huge project on the basis of this.

Why don't you have a go at it and then post what you come up with. It shouldn't be too difficult to use a wx.Panel instead the wx.Frame which is used in this code, I think you just need to get rid of the CreateStatusBar and the EVT_CLOSE handler.

Werner

···

On 17/12/2013 13:10, Navaneeth wrote:

13

wxPython has a thumbnail control. It’s impressive and extremely fast. But, it’s not much attractive. Please tell me how can I implement that algorithm in my program. I am also showing another script which is so fast. This is not written by me. I need your help to implement it on my panel. I need to launch a huge project on the basis of this.

Here is a visual comp pic of the attached sample and my thumbnailer, which uses agw.ThumbnailCtrl
http://img69.imageshack.us/img69/2108/duwq.png

It seems that the code sample runs pretty close in comparison with the ThumbnailCtrl.
Noted visual differences.
The MP thumbnailer, shows/flashes a short loading, please wait message when scrolling the list, as where the agw thumbnailer doesn’t.
The agw Thumbnailer seemed to take a bit longer to load 1000+ images, but only by about a second.
The agw Thumbnailer, has zooming/scaling effects, which seem to slow down scrolling through the list a bit more than the MP, which apparently uses fixed thumb size. This probably has to do with resizing checks when scrolling.

Overall, If the scrolled list style look is what you are looking for, personally that seems to be a better bet for your project, maybe. It seems to work fine for me for it’s purpose and the code is quite a bit smaller.
But you might still want to look into the thumb algorithm for agw ThumbnailCtrl if you want extra features or for example get rid of the loading image blip while scrolling or add other metadata like the agw tooltips.

Both thumbnailers loaded 1000+ in 1-2 seconds on a dualcore @ 2.2 - 3.0ghz. Is this not fast enough? How many images are you looking into loading as a maximum?

The agw seems to take more memory also.
Haven’t looked at the agw code recently, but it is probably double buffered somewhere also.

As far a being inside a panel, the MP thumbnailer is already in a panel sizer, so moving it around should be fairly easy.

self.vlist_box = vlist_box = Box(panel, queue_tasks)

    sizer.Add(vlist_box, 1, wx.EXPAND, 0)

Also Note that Publisher caused a ImportError on greater than wx2.8, so the Publisher location seems to have been moved inbetween wx versions. You may want to handle that also in the imports sect.

Also, what is the licensing on the posted sample app? Is that what is causing an issue?

···

On Tuesday, December 17, 2013 6:10:18 AM UTC-6, Navaneeth wrote:

14

Anybody please attach a sample script that uses wx.lib.agw.thumbnailctrl. It will be helpful because, I am unable to find a working sample script of that on the internet. All I’ve tried often failed to work properly without bugs on undefined variables. I am not trying to implement the visual style of the second script I’ve attached. I like to implement an application that stands different from others. So, I am thinking to implement a new algorithm with my own visual style. Please help.

15

Anybody please attach a sample script that uses wx.lib.agw.thumbnailctrl.

Take a look at the wxPython Demo. There is a demo of the thumbnailcontrol (you need PIL for it to work).

16

geez… what would people do without SourceCoder…? Or the wxPy Demo for that matter…?

wxapp, Ctrl+Space, Ctrl+K
add thumbnailer, upload.

ThumbnailCtrlSample.py (1.75 KB)

···

On Wednesday, December 18, 2013 1:16:12 AM UTC-6, Navaneeth wrote:

Anybody please attach a sample script that uses wx.lib.agw.thumbnailctrl. It will be helpful because, I am unable to find a working sample script of that on the internet. All I’ve tried often failed to work properly without bugs on undefined variables. I am not trying to implement the visual style of the second script I’ve attached. I like to implement an application that stands different from others. So, I am thinking to implement a new algorithm with my own visual style. Please help.

17

wx.lib.agw.thumbnailctrl.py

Or from wx.lib.agw.thumbnailctrl.doc
(That’s the documentation string in the file itself BTW. Also probably a sample in the agw docs also.)
NOTE: there is a lowercasing of filenames patch in my phoenix fork you will want to look at also, as it hasn’t been backported to Classic.
Basically UPPERCASE extensions stuff.

···

Usage example::

import os

import wx
import wx.lib.agw.thumbnailctrl as TC

class MyFrame(wx.Frame):

    def __init__(self, parent):

        wx.Frame.__init__(self, parent, -1, "ThumbnailCtrl Demo")

        panel = wx.Panel(self)

        sizer = wx.BoxSizer(wx.VERTICAL)

        thumbnail = TC.ThumbnailCtrl(panel, imagehandler=TC.NativeImageHandler)
        sizer.Add(thumbnail, 1, wx.EXPAND | wx.ALL, 10)

        thumbnail.ShowDir(os.getcwd())
        panel.SetSizer(sizer)


# our normal wxApp-derived class, as usual

app = wx.App(0)

frame = MyFrame(None)
app.SetTopWindow(frame)
frame.Show()

app.MainLoop()
18

Hi,

....

Also Note that Publisher caused a ImportError on greater than wx2.8, so the Publisher location seems to have been moved inbetween wx versions. You may want to handle that also in the imports sect.

2.9.4.0 had a change to use the 'kwarg' API of pubsub and IIRC at that point the import changed a bit:

from wx.lib.pubsub import pub

or if you use the non wxPython bundled pubsub then it is:

from pubsub import pub

See also the new documentation on pubsub:
http://pubsub.sourceforge.net/usage/usage_basic.html

Werner

···

On 17/12/2013 17:28, Metallicow wrote:

19

I'm the one that wrote the mpthumb.py, some year ago, as a way to
understand the
integration of multiprocessing into a wxPython application.

As I recall, there is a major implementation difference between mpthumb and
ThumbnailCtrl.
mpthumb doesn't load all the images, it will just load the images requested
by VListBox as you scroll,
so only visible images are loaded.

The thumbnails are generated in different processes so you can scroll up
and down and the gui
doesn't lag.

When a thumbnail is generated, it will be cached in memory.

Ricardo

···

On Tue, Dec 17, 2013 at 4:28 PM, Metallicow <metaliobovinus@gmail.com>wrote:

On Tuesday, December 17, 2013 6:10:18 AM UTC-6, Navaneeth wrote:

wxPython has a thumbnail control. It's impressive and extremely fast.
But, it's not much attractive. Please tell me how can I implement that
algorithm in my program. I am also showing another script which is so fast.
This is not written by me. I need your help to implement it on my panel. I
need to launch a huge project on the basis of this.

Here is a visual comp pic of the attached sample and my thumbnailer, which
uses agw.ThumbnailCtrl
ImageShack - Best place for all of your image hosting and image sharing needs

It seems that the code sample runs pretty close in comparison with the
ThumbnailCtrl.
Noted visual differences.
The MP thumbnailer, shows/flashes a short loading, please wait message
when scrolling the list, as where the agw thumbnailer doesn't.
The agw Thumbnailer seemed to take a bit longer to load 1000+ images, but
only by about a second.
The agw Thumbnailer, has zooming/scaling effects, which seem to slow down
scrolling through the list a bit more than the MP, which apparently uses
fixed thumb size. This probably has to do with resizing checks when
scrolling.

20

Gee, I was thinking the ThumbnailCtrl had some sort of multiprocessing/virtual like attributes. Maybe that performance will change quite a bit inbetween the PIL version implementation and NativeRenderer. I’m using a PILLOW’d version.
Maybe PIL is forcing the grunt of the load on my graphics cards and ram. The GUI didn’t lock up when I did that…
I’ll have too look at the thumbnailctrl more closely to see if I cannot maybe provide an option for something virtual for scrolling performance possibly with large image counts.
It probably gets by loading in chunks in the background if I was to guess ATM.
And to think of it, if It does load all that into memory, if one called the ShowDir() in the apps startup init code that could well cause a bad memory error/overload/crash if calling ginormous amounts of images like I tested with.

The MP is a nice change cause using pubsub/vlistbox is a bit different approach from most thumbnailers I have seen. I recall Cornice had one, tho right off hand, I don’t know if it was virtual either…
The whole images in threads and multiprocessing seems to be rather tricky to get right without crash problems.

BTW, what is the MP Thumbnailers license? wxPython perhaps maybe? Didn’t see one. I think it would be a nice small addition to the wxPy Demo as intro into multiprocessing.

Most of my stress tests lately have been with animations/pygame/sdl/rendering meshes/blender with wx. The Images catagory hasn’t been getting to much attention lately, sorry to say.

···

On Wednesday, December 18, 2013 4:50:58 PM UTC-6, Ricardo Pedroso wrote:

I’m the one that wrote the mpthumb.py, some year ago, as a way to understand the
integration of multiprocessing into a wxPython application.

As I recall, there is a major implementation difference between mpthumb and ThumbnailCtrl.

mpthumb doesn’t load all the images, it will just load the images requested by VListBox as you scroll,

so only visible images are loaded.

The thumbnails are generated in different processes so you can scroll up and down and the gui

doesn’t lag.

When a thumbnail is generated, it will be cached in memory.

Ricardo