Limits on wx.Bitmap size?

I am trying to display a document in
a scrolled window using a buffered bit map that contains the whole
thing so it can be scrolled rapidly from
beginning to end. I also want to be able to zoom in at as high a
magnification as I can get. This is working OK for a range of typical
documents that I have, but I want to guard against trying to request
bit maps bigger than can be handled.

I have done a few tests both with my application and with the
“double-buffered” example that Chris Barker put on the wxPython wiki.
Both produce similar results so my hacked-about version of the wiki
program is attached (with apologies to Chris for vandalising his
program).

Details of the various errors are commented in the code but both XP and
OS X fall over with bitmaps around W x H > 256 * 10**6.

Is this some fundamental limit of wxWidgets? I was initially expecting
it to be related to the amount of free memory, but on a paged-memory
system that’s a bit of a slippery concept - unless there is some
per-process limit. I’m guessing that I will have to put in some
hard-coded maximum bit map size rather than be able to interrogate the
system for it.

BarkerCanvas-scrolled.py (10.6 KB)

···

Regards,

David Hughes

Forestfield Software

It's a bit more work, but consider using tiles of the large bitmap to
overcome this limitation. That way you won't have to have one huge
bitmap in memory and you can load/construct tiles on "just in time" in
a separate thread.

You could try a simple solution for tiling using panels in a GridSizer
that start drawing their bitmap if they are close enough to the
viewport.

David Hughes wrote:

I am trying to display a document in a scrolled window using a buffered bit map that contains the whole thing so it can be scrolled rapidly from beginning to end. I also want to be able to zoom in at as high a magnification as I can get. This is working OK for a range of typical documents that I have, but I want to guard against trying to request bit maps bigger than can be handled.

I have done a few tests both with my application and with the "double-buffered" example that Chris Barker put on the wxPython wiki. Both produce similar results so my hacked-about version of the wiki program is attached (with apologies to Chris for vandalising his program).
Details of the various errors are commented in the code but both XP and OS X fall over with bitmaps around W x H > 256 * 10**6.

Thats 256 000 000 * 3 (24bit) --> 730MB RAM
Sure you have that available on the machine?

BTW, the script works fine for me on ArchLinux (2G RAM).

Manne

Manne Merak wrote:

David Hughes wrote:

I am trying to display a document in a scrolled window using a buffered bit map that contains the whole thing so it can be scrolled rapidly from beginning to end. I also want to be able to zoom in at as high a magnification as I can get. This is working OK for a range of typical documents that I have, but I want to guard against trying to request bit maps bigger than can be handled.
I have done a few tests both with my application and with the "double-buffered" example that Chris Barker put on the wxPython wiki. Both produce similar results so my hacked-about version of the wiki program is attached (with apologies to Chris for vandalising his program).
Details of the various errors are commented in the code but both XP and OS X fall over with bitmaps around W x H > 256 * 10**6.
Thats 256 000 000 * 3 (24bit) --> 730MB RAM
Sure you have that available on the machine?

Yes, I have 4G RAM

BTW, the script works fine for me on ArchLinux (2G RAM).

The default values in the script are 8000 x
8000. What happens as you pass in progressively bigger values as
argv[1] and argv[2] each time you run the script?

···
Manne

Regards,

David Hughes

David Hughes wrote:

Manne Merak wrote:

David Hughes wrote:
  

I am trying to display a document in a scrolled window using a buffered bit map that contains the whole thing so it can be scrolled rapidly from beginning to end. I also want to be able to zoom in at as high a magnification as I can get. This is working OK for a range of typical documents that I have, but I want to guard against trying to request bit maps bigger than can be handled.

I have done a few tests both with my application and with the "double-buffered" example that Chris Barker put on the wxPython wiki. Both produce similar results so my hacked-about version of the wiki program is attached (with apologies to Chris for vandalising his program).
Details of the various errors are commented in the code but both XP and OS X fall over with bitmaps around W x H > 256 * 10**6.
    

Thats 256 000 000 * 3 (24bit) --> 730MB RAM
Sure you have that available on the machine?
  

Yes, I have 4G RAM

BTW, the script works fine for me on ArchLinux (2G RAM).
  

The default values in the script are 8000 x 8000. What happens as you pass in progressively bigger values as argv[1] and argv[2] each time you run the script?

[xxx@xxx ~]$ python BarkerCanvas-scrolled.py 22000 22000
Width input = 22000
Height input = 22000
about to initialize the app. USE_BUFFERED_DC = 0
Bitmap: width 22000 height 22000 depth 24 bits
Buffersize 1384 MBytes

Works, but not usable - very slow. Also freezes my input while it is loading.

[xxx@xxx ~]$ python BarkerCanvas-scrolled.py 25000 25000
Width input = 25000
Height input = 25000
about to initialize the app. USE_BUFFERED_DC = 0
The program 'python' received an X Window System error.
This probably reflects a bug in the program.
The error was 'BadAlloc (insufficient resources for operation)'.
...........................................

Manne

The default values in the script are 8000 x 8000. What happens as you pass in progressively bigger values as argv[1] and argv[2] each time you run the script?

[xxx@xxx ~]$ python BarkerCanvas-scrolled.py 22000 22000
Width input = 22000
Height input = 22000
about to initialize the app. USE_BUFFERED_DC = 0
Bitmap: width 22000 height 22000 depth 24 bits
Buffersize 1384 MBytes
Works, but not usable - very slow. Also freezes my input while it is loading.

[xxx@xxx ~]$ python BarkerCanvas-scrolled.py 25000 25000



Width input = 25000
Height input = 25000
about to initialize the app. USE_BUFFERED_DC = 0
The program 'python' received an X Window System error.
This probably reflects a bug in the program.
The error was 'BadAlloc (insufficient resources for operation)'.
...........................................
Manne

It wasn’t really intended to be usable, just to see how big it got
before it crashed. Thanks for the information - David

raz wrote:

It's a bit more work, but consider using tiles of the large bitmap to
overcome this limitation. That way you won't have to have one huge
bitmap in memory and you can load/construct tiles on "just in time" in
a separate thread.

You could try a simple solution for tiling using panels in a GridSizer
that start drawing their bitmap if they are close enough to the
viewport

That sounds a useful line of approach. Thank you - David

Manne Merak wrote:

David Hughes wrote:
  

The default values in the script are 8000 x 8000. What happens as you pass in progressively bigger values as argv[1] and argv[2] each time you run the script?
    

[xxx@xxx ~]$ python BarkerCanvas-scrolled.py 22000 22000
Width input = 22000
Height input = 22000
about to initialize the app. USE_BUFFERED_DC = 0
Bitmap: width 22000 height 22000 depth 24 bits
Buffersize 1384 MBytes

Works, but not usable - very slow. Also freezes my input while it is loading.

[xxx@xxx ~]$ python BarkerCanvas-scrolled.py 25000 25000
Width input = 25000
Height input = 25000
about to initialize the app. USE_BUFFERED_DC = 0
The program 'python' received an X Window System error.
This probably reflects a bug in the program.
The error was 'BadAlloc (insufficient resources for operation)'.
...........................................

Manne

I tried on my machine, XP SP3, python 2.6, latest wx and it seems the largest values I can use seem to be 16400 for w/h. If I go up to 165000, crash:

C:\Documents and Settings\Steve\My Documents>python BarkerCanvas-scrolled.py 16200 16200
Buffersize 750 MBytes
wx.free 557 MBytes

C:\Documents and Settings\Steve\My Documents>python BarkerCanvas-scrolled.py 16300 16300
Buffersize 760 MBytes
wx.free 547 MBytes

C:\Documents and Settings\Steve\My Documents>python BarkerCanvas-scrolled.py 16400 16400
Buffersize 769 MBytes
wx.free 538 MBytes

C:\Documents and Settings\Steve\My Documents>python BarkerCanvas-scrolled.py 16500 16500
Traceback (most recent call last):
  File "BarkerCanvas-scrolled.py", line 298, in <module>
    app = DemoApp(0)
  .[omitted]

David Hughes wrote:

I have done a few tests both with my application and with the "double-buffered" example that Chris Barker put on the wxPython wiki.

Since you mentioned me...

Details of the various errors are commented in the code but both XP and OS X fall over with bitmaps around W x H > 256 * 10**6.

Way back when I first started looking at this stuff, I tried what you are doing -- making a big buffer and then scrolling around it. It was pretty speedy, which is nice, but as you can see, you are pretty limited as to buffer size. Back then, no one had 4GB of RAM, but even now there are limits, particularly with 32bit systems.

At the time, I came to the conclusion that wasn't the way to go, except perhaps if you were limited to a buffer may twice the screen size.

Anyway, at the end of it all, I addressed my drawing needs be developing FloatCanvas, which is now in wx.lib.floatcanvas, which does double buffer, but with a buffer the size of the Window.

What is it you need to do? Maybe FloatCanvas can help, or something else, but I suspect that the One-Big-Huge-Buffer approach is not the way to go.

-Chris

···

--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov

Christopher Barker wrote:

David Hughes wrote:

I have done a few tests both with my application and with the "double-buffered" example that Chris Barker put on the wxPython wiki.

Since you mentioned me...

Details of the various errors are commented in the code but both XP and OS X fall over with bitmaps around W x H > 256 * 10**6.

Way back when I first started looking at this stuff, I tried what you are doing -- making a big buffer and then scrolling around it. It was pretty speedy, which is nice, but as you can see, you are pretty limited as to buffer size. Back then, no one had 4GB of RAM, but even now there are limits, particularly with 32bit systems.

At the time, I came to the conclusion that wasn't the way to go, except perhaps if you were limited to a buffer may twice the screen size.

Anyway, at the end of it all, I addressed my drawing needs be developing FloatCanvas, which is now in wx.lib.floatcanvas, which does double buffer, but with a buffer the size of the Window.

What is it you need to do? Maybe FloatCanvas can help, or something else, but I suspect that the One-Big-Huge-Buffer approach is not the way to go.

Just as an additional data point for comparison, the guts of wxStyledTextCtrl (scintilla) can handle huge documents but only double buffers at most the visible portion of the window, and depending on the kind of update being done it may only buffer the current line being edited. (IIRC) The key, I think, is to be able to organize your document data in memory in such a way that you can quickly calculate what is the visible portion, and be able to draw that portion without needing to do a lot of data access to or drawing of the non-visible portions.

Another optimization of the scroll refresh (and also the refresh that happens when the window is "damaged" by some other window moving over it) that you can do is you can avoid redrawing the entire visible portion to a double buffer when the window is scrolled and just draw the portion that has just become exposed. This works because the platform will automatically move the visible pixels appropriately when the window is scrolled and then sets the update region to whatever the rest of the visible part of the window is that needs your attention. In other words, instead of maintaining a buffer of the whole visible portion of the window, you just draw the part(s) of it that need to be refreshed in any given paint event, buffering the parts on the fly if needed.

So if your document data is organized such that you can take a given rectangle and quickly calculate and draw the portion of the document that lies in that rectangle, then you can apply this technique to draw it in the paint event handler by getting the current update region and then either iterate through the rectangles in the region or get the rectangle that encloses the entire region and then make a temporary buffer[1] just for the rectangle, draw to that buffer, and blit it to the paint dc. There is an example of using an update region in the ColourDB sample in the demo.

[1] the temporary buffer would only really be needed on Windows, so another optimization for the other platforms would be to just draw the rectangle directly to the paint dc and don't do the temp buffer and blit.

···

--
Robin Dunn
Software Craftsman

Thanks for all the responses, and to
Chris and Robin in particular. I did initially try using a buffer the
size of the visible window but failed miserably to make it work. I have
tried again, knowing now that it should work, and it is looking
promising. Now to try the optimisations Robin suggested…

···

Regards,

David Hughes

Forestfield Software

Robin Dunn wrote:

Christopher Barker wrote:
David Hughes wrote:
I have done a few tests both with my application and with the "double-buffered" example that Chris Barker put on the wxPython wiki.
Since you mentioned me...
Details of the various errors are commented in the code but both XP and OS X fall over with bitmaps around W x H > 256 * 10**6.

Way back when I first started looking at this stuff, I tried what you are doing -- making a big buffer and then scrolling around it. It was pretty speedy, which is nice, but as you can see, you are pretty limited as to buffer size. Back then, no one had 4GB of RAM, but even now there are limits, particularly with 32bit systems.
At the time, I came to the conclusion that wasn't the way to go, except perhaps if you were limited to a buffer may twice the screen size.
Anyway, at the end of it all, I addressed my drawing needs be developing FloatCanvas, which is now in wx.lib.floatcanvas, which does double buffer, but with a buffer the size of the Window.
What is it you need to do? Maybe FloatCanvas can help, or something else, but I suspect that the One-Big-Huge-Buffer approach is not the way to go.

Just as an additional data point for comparison, the guts of wxStyledTextCtrl (scintilla) can handle huge documents but only double buffers at most the visible portion of the window, and depending on the kind of update being done it may only buffer the current line being edited. (IIRC) The key, I think, is to be able to organize your document data in memory in such a way that you can quickly calculate what is the visible portion, and be able to draw that portion without needing to do a lot of data access to or drawing of the non-visible portions.
Another optimization of the scroll refresh (and also the refresh that happens when the window is "damaged" by some other window moving over it) that you can do is you can avoid redrawing the entire visible portion to a double buffer when the window is scrolled and just draw the portion that has just become exposed. This works because the platform will automatically move the visible pixels appropriately when the window is scrolled and then sets the update region to whatever the rest of the visible part of the window is that needs your attention. In other words, instead of maintaining a buffer of the whole visible portion of the window, you just draw the part(s) of it that need to be refreshed in any given paint event, buffering the parts on the fly if needed.
So if your document data is organized such that you can take a given rectangle and quickly calculate and draw the portion of the document that lies in that rectangle, then you can apply this technique to draw it in the paint event handler by getting the current update region and then either iterate through the rectangles in the region or get the rectangle that encloses the entire region and then make a temporary buffer[1] just for the rectangle, draw to that buffer, and blit it to the paint dc. There is an example of using an update region in the ColourDB sample in the demo.
[1] the temporary buffer would only really be needed on Windows, so another optimization for the other platforms would be to just draw the rectangle directly to the paint dc and don't do the temp buffer and blit.

David Hughes wrote:

Thanks for all the responses, and to Chris and Robin in particular. I did initially try using a buffer the size of the visible window but failed miserably to make it work. I have tried again, knowing now that it _should_ work, and it is looking promising. Now to try the optimisations Robin suggested....

What is it you need to display? That could drive how you can best do it.

-Chris

···

--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov