Speed problem with wxMemoryDC

Hi guys,

This is my first message here, but Robbin Dunn, you are a great man,

and many a time searching for answers has lead me to your post,
which has solved answers, searching has not saved me this time,
hence this email.

I'm working on some software that'll replace some closed source bit

of tat that students across the world are forced to buy, However
they (users) actually liked a certain aspect a lot more than I
thought, a 3d drawing of the “world”, I can do this, I’ve basically
created the OpenGL spec in Python, I know what you’re thinking, what
a crap idea, that’ll be really slow, it’s not slow for the reasons
you’d think it would be.

The point of writing it in Python was mainly "because I can", it

means they don’t have to install anything other than Python and
wxPython (A lot of them use Windows), it’s something I want to
explore.

I've implemented Vertex and Fragment shaders, everything actually

works, which took all of 7 hours actually, with plenty of testing,
because that first hello-world Torus outline in the depth buffer was
so cool to look at.

So anyway, here's some profiling results:

<small><small>`Profile summary: Testing stuff``

    ``          Location:          

Target:
Name/Desc: Calls: Total (Abs)
time: Avg time: Std
dev: ``

    ``                              <function draw at

0x2ed02a8>
draw 73 16.18% ~ 7.415316
(44.429733) 0.101580 (0.608626) 0.017291 (0.021072)
``

    ``                              <function set at

0x2ecf5f0>
set 1065780 2.59% ~ 1.185920
(1.185920) 0.000001 (0.000001) 0.000001 (0.000001)
``

    ``                              <function clearDepthBuffer at

0x2ecfcf8> clearDepthBuffer
73 1.18% ~ 0.541950 (0.541950) 0.007424
(0.007424) 0.002129 (0.002129) ``

    ``                              <function __getitem__ at

0x2da0848> getitem
2190000 5.11% ~ 2.340605 (2.340605) 0.000001
(0.000001) 0.000001 (0.000001) ``

    ``                              <function get at

0x2ecf668>
get 597432 1.40% ~ 0.642502
(0.642502) 0.000001 (0.000001) 0.000001 (0.000001)
``

    ``                              <function clone at

0x2ecf758>
clone 219 0.05% ~ 0.022992
(0.029213) 0.000105 (0.000133) 0.000040 (0.000051)
``

    ``                              <function

setVertexShaderAttribute at 0x2ecfed8>
setVertexShaderAttribute 73 0.00% ~ 0.001987
(0.001987) 0.000027 (0.000027) 0.000007 (0.000007)
``

    ``                              <function onDraw at

0x2ed0578>
onDraw 73 0.04% ~ 0.017156
(45.102545) 0.000235 (0.617843) 0.000059 (0.020795)
``

    ``                              <function Draw at

0x2cacc80>
Draw 73 0.01% ~ 0.003555
(45.106995) 0.000049 (0.617904) 0.000015 (0.020795)
``

    ``                              <function UpdateDrawing at

0x7f1f4940c500> UpdateDrawing
73 1.52% ~ 0.698443 (45.821557) 0.009568
(0.627693) 0.002209 (0.021146) ``

    ``                              <function onPrepare at

0x2ed0de8>
onPrepare 73 0.00% ~ 0.001627
(0.139565) 0.000022 (0.001912) 0.000008 (0.000728)
``

    ``                              <function OnPaint at

0x7f1f4947be60>
OnPaint 71 0.04% ~ 0.016845
(0.016845) 0.000237 (0.000237) 0.000072 (0.000072)
``

    ``                              <function OnIdle at

0x7f1f4947b230>
OnIdle 69 0.02% ~ 0.007404
(43.406363) 0.000107 (0.629078) 0.000068 (0.018467)
``

    ``                              <function drawDepthBuffer at

0x2ed0320> drawDepthBuffer
73 15.21% ~ 6.970698 (33.136861) 0.095489
(0.453930) 0.000916 (0.004267) ``

    ``                              <function requestDraw at

0x2ecfb90> requestDraw
73 0.00% ~ 0.000895 (45.103440) 0.000012
(0.617855) 0.000002 (0.020795) ``

    ``                              <function

setVertexShaderConstant at 0x2ecff50>
setVertexShaderConstant 219 0.01% ~ 0.002566
(0.031779) 0.000012 (0.000145) 0.000005 (0.000055)
``

    ``                              <function OnSize at

0x7f1f4947b6e0>
OnSize 4 0.00% ~ 0.000660
(2.423258) 0.000165 (0.605815) 0.000030 (0.048200)
``

    ``                              <function run at

0x2ed0e60>
run 16644 0.31% ~ 0.142187
(2.614278) 0.000009 (0.000157) 0.000007 (0.000022)
``

    ``          Matrices.py:103     <function __mul__ at

0x2ecf848> Matrix
Multiplication 16863 4.35% ~ 1.991480 (2.671900)
0.000118 (0.000158) 0.000080 (0.000107) ``

    ``          main.py:109         <function makeTorus at

0x2ed05f0> Make
Torus 1 0.00% ~ 0.000551
(0.000551) 0.000551 (0.000551) N/A
``

    ``                              <function plotPixel at

0x2cacd70>
plotPixel 730000 51.99% ~ 23.825558
(26.166163) 0.000033 (0.000036) 0.000009 (0.000009) `

I hope you can see that okay, basically plotPixel is the REALLY slow

one, which I cannot accelerate with C, the depth and colour buffers
are just massive lists of lists, I can speed them up later, matrix
multiplication was actually where I thought the bottle-neck would
be, but anyway, a lot of that can be improved with C.

I mentioned no dependencies earlier, the C acceleration will be

optional, basically, it’s more for my own interest, some Python
tuning should help a lot.

The plotPixel function is passed a co-ordinate and a colour (rgba (I

currently ignore alpha, it’s not really needed…) as values from
the closed interval [0, 1]), it then gets a brush, and a pen, and
draws a rectangle for the pixel (because of the performance, pixel
sizes are not one to one)

I could just... have an 8x8x8 (512 colours) colour-space, but I'd

like not to, I don’t expect 30fps, but I need a bit more, the map is
easy, it is just a memory buffer, is there a way to efficiently draw
that with wxPython?

Also, while I am here, I don't want to use a list of lists for my

buffer, the values are floats, any advice on speeding them up would
be awesome (going though c? Using doubles to save a conversion if I
did use C (Python uses doubles)? accessing with get and set
methods?) so yes please help!

I really don't expect this to be a "realistic" rendering thing, but

I cannot use C to speed up this part (that I know anyway) I have not
found away pens and brushes!

I look forward to your reply.

Alec

My first thought is -- is it THAT hard to install PyOpenGL?

But this is pretty cool none the less, so:

The point of writing it in Python was mainly "because I can", it means they
don't have to install anything other than Python and wxPython (A lot of them
use Windows), it's something I want to explore.

There are some things that Python itself isn't good for, and this on
eof them. If you could consider a dependency on numpy , that could
help a lot.

I think if hit like this:

Using python for number-crunching without numpy would be like using
python for text processing without the string object -- i.e. workign
with list of charactors...

Anyway, with or without numpy, my thoughts:

I've implemented Vertex and Fragment shaders, everything actually works,
which took all of 7 hours actually, with plenty of testing, because that
first hello-world Torus outline in the depth buffer was so cool to look at.

7 hours? pretty darn fast if you ask me!

I hope you can see that okay,

kind of hard to see in an email, when I don't know your code, but I'll
take your word for it:

basically plotPixel is the REALLY slow one,
which I cannot accelerate with C, the depth and colour buffers are just
massive lists of lists,

This is going to be pretty bad -- each pixel is then a python integer
object, I presume -- very heavy memory overhead. Rather, I'd use:

1) a numpy array -- perfect for this sort of thing!

2) an array.array -- not naturally 2-d but at least memory efficient

3) a memoryview -- this may be the "right" way to go...

I can speed them up later, matrix multiplication was
actually where I thought the bottle-neck would be,

numpy would help here, too

but anyway, a lot of that can be improved with C.

but much more so, and much easier if you use an object for your iiimge
buffers that stoes stuff more or less in a regular old C array:
memoryview, numpy array, etc.

Note that I'd recommend Cython -- it has built-in support for numpy
arrays and memory views -- perfect for this sort of thing.

I mentioned no dependencies earlier, the C acceleration will be optional,

well, if you provide binaries, then there would be no additional dependency.

The plotPixel function is passed a co-ordinate and a colour (rgba (I
currently ignore alpha, it's not really needed....) as values from the
closed interval [0, 1]), it then gets a brush, and a pen, and draws a
rectangle for the pixel (because of the performance, pixel sizes are not one
to one)

That's going to take a long time -- as you've seen:

1) A python function call, and transformation from python types to
wxPython types -- pretty slow! And seting teh Pen and brush for each
pixel is going to take some time too.

2) can you really not map pixel<->pixel? if you did you could use
DC.DrawPoint, or even better, DrawPointList -- that could really help.

I expect you could do the scaling in a smarter way than drawing rectangles....

I could just... have an 8x8x8 (512 colours) colour-space, but I'd like not
to, I don't expect 30fps, but I need a bit more, the map is easy, it is just
a memory buffer, is there a way to efficiently draw that with wxPython?

If you have a proper memory Buffer (something that has the buffer
interface), you should be able to turn it into a wx.Bitmap, and just
draw that -- MUCH faster. See:

http://wiki.wxpython.org/WorkingWithImages

Also, while I am here, I don't want to use a list of lists for my buffer,
the values are floats, any advice on speeding them up would be awesome
(going though c? Using doubles to save a conversion if I did use C (Python
uses doubles)? accessing with get and set methods?) so yes please help!

see above: numpy arrays, array.array or memoryview.

I'm not sure your imag buffer should be floats, but if so, then you
should still be able to do a fast conversion to a RGB or RGBA buffer
-- very easy in numpy, not so fast with memoryview or array.array, but
still probably a lot faster than this DrawRectangle stuff!

If you're willing to give Cython a shot -- maybe this example will
help give you an idea:

http://wiki.cython.org/examples/mandelbrot

It uses Cython to set values on a numpy array image buffer, and there
is wx code to display that....

HTH,
  -Chris

···

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov

This is a sort of bump, I've yet to find a way to efficiently copy a colour buffer (drawing rectangles and passing RGB values helped..... saved a lot of pens anyway), would still love an answer.

Alec

···

On 12/06/13 19:36, Alec Teal wrote:

Hi guys,

This is my first message here, but Robbin Dunn, you are a great man, and many a time searching for answers has lead me to your post, which has solved answers, searching has not saved me this time, hence this email.

I'm working on some software that'll replace some closed source bit of tat that students across the world are forced to buy, However they (users) actually liked a certain aspect a lot more than I thought, a 3d drawing of the "world", I can do this, I've basically created the OpenGL spec in Python, I know what you're thinking, what a crap idea, that'll be really slow, it's not slow for the reasons you'd think it would be.

The point of writing it in Python was mainly "because I can", it means they don't have to install anything other than Python and wxPython (A lot of them use Windows), it's something I want to explore.

I've implemented Vertex and Fragment shaders, everything actually works, which took all of 7 hours actually, with plenty of testing, because that first hello-world Torus outline in the depth buffer was so cool to look at.

So anyway, here's some profiling results:
Profile summary: Testing stuff
Location: Target: Name/Desc: Calls: Total (Abs) time: Avg time: Std dev:
                    <function draw at 0x2ed02a8> draw 73 16.18% ~ 7.415316 (44.429733) 0.101580 (0.608626) 0.017291 (0.021072)
                    <function set at 0x2ecf5f0> set 1065780 2.59% ~ 1.185920 (1.185920) 0.000001 (0.000001) 0.000001 (0.000001)
                    <function clearDepthBuffer at 0x2ecfcf8> clearDepthBuffer 73 1.18% ~ 0.541950 (0.541950) 0.007424 (0.007424) 0.002129 (0.002129)
                    <function __getitem__ at 0x2da0848> __getitem__ 2190000 5.11% ~ 2.340605 (2.340605) 0.000001 (0.000001) 0.000001 (0.000001)
                    <function get at 0x2ecf668> get 597432 1.40% ~ 0.642502 (0.642502) 0.000001 (0.000001) 0.000001 (0.000001)
                    <function clone at 0x2ecf758> clone 219 0.05% ~ 0.022992 (0.029213) 0.000105 (0.000133) 0.000040 (0.000051)
                    <function setVertexShaderAttribute at 0x2ecfed8> setVertexShaderAttribute 73 0.00% ~ 0.001987 (0.001987) 0.000027 (0.000027) 0.000007 (0.000007)
                    <function onDraw at 0x2ed0578> onDraw 73 0.04% ~ 0.017156 (45.102545) 0.000235 (0.617843) 0.000059 (0.020795)
                    <function Draw at 0x2cacc80> Draw 73 0.01% ~ 0.003555 (45.106995) 0.000049 (0.617904) 0.000015 (0.020795)
                    <function UpdateDrawing at 0x7f1f4940c500> UpdateDrawing 73 1.52% ~ 0.698443 (45.821557) 0.009568 (0.627693) 0.002209 (0.021146)
                    <function onPrepare at 0x2ed0de8> onPrepare 73 0.00% ~ 0.001627 (0.139565) 0.000022 (0.001912) 0.000008 (0.000728)
                    <function OnPaint at 0x7f1f4947be60> OnPaint 71 0.04% ~ 0.016845 (0.016845) 0.000237 (0.000237) 0.000072 (0.000072)
                    <function OnIdle at 0x7f1f4947b230> OnIdle 69 0.02% ~ 0.007404 (43.406363) 0.000107 (0.629078) 0.000068 (0.018467)
                    <function drawDepthBuffer at 0x2ed0320> drawDepthBuffer 73 15.21% ~ 6.970698 (33.136861) 0.095489 (0.453930) 0.000916 (0.004267)
                    <function requestDraw at 0x2ecfb90> requestDraw 73 0.00% ~ 0.000895 (45.103440) 0.000012 (0.617855) 0.000002 (0.020795)
                    <function setVertexShaderConstant at 0x2ecff50> setVertexShaderConstant 219 0.01% ~ 0.002566 (0.031779) 0.000012 (0.000145) 0.000005 (0.000055)
                    <function OnSize at 0x7f1f4947b6e0> OnSize 4 0.00% ~ 0.000660 (2.423258) 0.000165 (0.605815) 0.000030 (0.048200)
                    <function run at 0x2ed0e60> run 16644 0.31% ~ 0.142187 (2.614278) 0.000009 (0.000157) 0.000007 (0.000022)
Matrices.py:103 <function __mul__ at 0x2ecf848> Matrix Multiplication 16863 4.35% ~ 1.991480 (2.671900) 0.000118 (0.000158) 0.000080 (0.000107)
main.py:109 <function makeTorus at 0x2ed05f0> Make Torus 1 0.00% ~ 0.000551 (0.000551) 0.000551 (0.000551) N/A
                    <function plotPixel at 0x2cacd70> plotPixel 730000 51.99% ~ 23.825558 (26.166163) 0.000033 (0.000036) 0.000009 (0.000009)

I hope you can see that okay, basically plotPixel is the REALLY slow one, which I cannot accelerate with C, the depth and colour buffers are just massive lists of lists, I can speed them up later, matrix multiplication was actually where I thought the bottle-neck would be, but anyway, a lot of that can be improved with C.

I mentioned no dependencies earlier, the C acceleration will be optional, basically, it's more for my own interest, some Python tuning should help a lot.

The plotPixel function is passed a co-ordinate and a colour (rgba (I currently ignore alpha, it's not really needed....) as values from the closed interval [0, 1]), it then gets a brush, and a pen, and draws a rectangle for the pixel (because of the performance, pixel sizes are not one to one)

I could just... have an 8x8x8 (512 colours) colour-space, but I'd like not to, I don't expect 30fps, but I need a bit more, the map is easy, it is just a memory buffer, is there a way to efficiently draw that with wxPython?

Also, while I am here, I don't want to use a list of lists for my buffer, the values are floats, any advice on speeding them up would be awesome (going though c? Using doubles to save a conversion if I did use C (Python uses doubles)? accessing with get and set methods?) so yes please help!

I really don't expect this to be a "realistic" rendering thing, but I cannot use C to speed up this part (that I know anyway) I have not found away pens and brushes!

I look forward to your reply.

Alec
--
You received this message because you are subscribed to the Google Groups "wxPython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wxpython-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

This is a sort of bump, I've yet to find a way to efficiently copy a colour
buffer (drawing rectangles and passing RGB values helped..... saved a lot of
pens anyway), would still love an answer.

I remember that thread, but not what you ended up trying. But a few thoughts:

1) manipulating pixels one by one is going to be slow in python,
period. If you can handle a numpy dependency, I'd use a numpy array to
store your pixel array, and numpy "ufuncs" may help speed things up.
Even if they don't, a numpy array gives you a nice buffer you can use
with C or cython code, and/or pass to/from wxPython. I like to say
that working with a lo tof numbers in Python without numpy is ike
doing text processing without the string object -- you wouldn't try to
text process with lists of characters, would you ?

2) One of the issues is that a list of lists of python integers uses a
lot more memory than you require -- a python integer is 32 (or 64)
bits, and has a python object wrapper around it -- and list stores an
array of pointers to python objects, so a lo topf indirection, etc. --
so, first choice: numpy array, second choice: array.array -- 1-d but
efficient storage and supports the buffer interface.

Have you tried these? what have you got for results? where are the
bottlenecks now?

-Chris

···

On Fri, Jul 5, 2013 at 7:50 AM, Alec Teal <a.teal@warwick.ac.uk> wrote:

Alec

On 12/06/13 19:36, Alec Teal wrote:

Hi guys,

This is my first message here, but Robbin Dunn, you are a great man, and
many a time searching for answers has lead me to your post, which has solved
answers, searching has not saved me this time, hence this email.

I'm working on some software that'll replace some closed source bit of tat
that students across the world are forced to buy, However they (users)
actually liked a certain aspect a lot more than I thought, a 3d drawing of
the "world", I can do this, I've basically created the OpenGL spec in
Python, I know what you're thinking, what a crap idea, that'll be really
slow, it's not slow for the reasons you'd think it would be.

The point of writing it in Python was mainly "because I can", it means they
don't have to install anything other than Python and wxPython (A lot of them
use Windows), it's something I want to explore.

I've implemented Vertex and Fragment shaders, everything actually works,
which took all of 7 hours actually, with plenty of testing, because that
first hello-world Torus outline in the depth buffer was so cool to look at.

So anyway, here's some profiling results:
Profile summary: Testing stuff
Location: Target:
Name/Desc: Calls: Total (Abs) time: Avg
time: Std dev:
                    <function draw at 0x2ed02a8>
draw 73 16.18% ~ 7.415316 (44.429733)
0.101580 (0.608626) 0.017291 (0.021072)
                    <function set at 0x2ecf5f0>
set 1065780 2.59% ~ 1.185920 (1.185920)
0.000001 (0.000001) 0.000001 (0.000001)
                    <function clearDepthBuffer at 0x2ecfcf8>
clearDepthBuffer 73 1.18% ~ 0.541950 (0.541950)
0.007424 (0.007424) 0.002129 (0.002129)
                    <function __getitem__ at 0x2da0848>
__getitem__ 2190000 5.11% ~ 2.340605 (2.340605)
0.000001 (0.000001) 0.000001 (0.000001)
                    <function get at 0x2ecf668>
get 597432 1.40% ~ 0.642502 (0.642502)
0.000001 (0.000001) 0.000001 (0.000001)
                    <function clone at 0x2ecf758>
clone 219 0.05% ~ 0.022992 (0.029213)
0.000105 (0.000133) 0.000040 (0.000051)
                    <function setVertexShaderAttribute at 0x2ecfed8>
setVertexShaderAttribute 73 0.00% ~ 0.001987 (0.001987)
0.000027 (0.000027) 0.000007 (0.000007)
                    <function onDraw at 0x2ed0578>
onDraw 73 0.04% ~ 0.017156 (45.102545)
0.000235 (0.617843) 0.000059 (0.020795)
                    <function Draw at 0x2cacc80>
Draw 73 0.01% ~ 0.003555 (45.106995)
0.000049 (0.617904) 0.000015 (0.020795)
                    <function UpdateDrawing at 0x7f1f4940c500>
UpdateDrawing 73 1.52% ~ 0.698443 (45.821557)
0.009568 (0.627693) 0.002209 (0.021146)
                    <function onPrepare at 0x2ed0de8>
onPrepare 73 0.00% ~ 0.001627 (0.139565)
0.000022 (0.001912) 0.000008 (0.000728)
                    <function OnPaint at 0x7f1f4947be60>
OnPaint 71 0.04% ~ 0.016845 (0.016845)
0.000237 (0.000237) 0.000072 (0.000072)
                    <function OnIdle at 0x7f1f4947b230>
OnIdle 69 0.02% ~ 0.007404 (43.406363)
0.000107 (0.629078) 0.000068 (0.018467)
                    <function drawDepthBuffer at 0x2ed0320>
drawDepthBuffer 73 15.21% ~ 6.970698 (33.136861)
0.095489 (0.453930) 0.000916 (0.004267)
                    <function requestDraw at 0x2ecfb90>
requestDraw 73 0.00% ~ 0.000895 (45.103440)
0.000012 (0.617855) 0.000002 (0.020795)
                    <function setVertexShaderConstant at 0x2ecff50>
setVertexShaderConstant 219 0.01% ~ 0.002566 (0.031779)
0.000012 (0.000145) 0.000005 (0.000055)
                    <function OnSize at 0x7f1f4947b6e0>
OnSize 4 0.00% ~ 0.000660 (2.423258)
0.000165 (0.605815) 0.000030 (0.048200)
                    <function run at 0x2ed0e60>
run 16644 0.31% ~ 0.142187 (2.614278)
0.000009 (0.000157) 0.000007 (0.000022)
Matrices.py:103 <function __mul__ at 0x2ecf848>
Matrix Multiplication 16863 4.35% ~ 1.991480 (2.671900)
0.000118 (0.000158) 0.000080 (0.000107)
main.py:109 <function makeTorus at 0x2ed05f0>
Make Torus 1 0.00% ~ 0.000551 (0.000551)
0.000551 (0.000551) N/A
                    <function plotPixel at 0x2cacd70>
plotPixel 730000 51.99% ~ 23.825558 (26.166163)
0.000033 (0.000036) 0.000009 (0.000009)

I hope you can see that okay, basically plotPixel is the REALLY slow one,
which I cannot accelerate with C, the depth and colour buffers are just
massive lists of lists, I can speed them up later, matrix multiplication was
actually where I thought the bottle-neck would be, but anyway, a lot of that
can be improved with C.

I mentioned no dependencies earlier, the C acceleration will be optional,
basically, it's more for my own interest, some Python tuning should help a
lot.

The plotPixel function is passed a co-ordinate and a colour (rgba (I
currently ignore alpha, it's not really needed....) as values from the
closed interval [0, 1]), it then gets a brush, and a pen, and draws a
rectangle for the pixel (because of the performance, pixel sizes are not one
to one)

I could just... have an 8x8x8 (512 colours) colour-space, but I'd like not
to, I don't expect 30fps, but I need a bit more, the map is easy, it is just
a memory buffer, is there a way to efficiently draw that with wxPython?

Also, while I am here, I don't want to use a list of lists for my buffer,
the values are floats, any advice on speeding them up would be awesome
(going though c? Using doubles to save a conversion if I did use C (Python
uses doubles)? accessing with get and set methods?) so yes please help!

I really don't expect this to be a "realistic" rendering thing, but I cannot
use C to speed up this part (that I know anyway) I have not found away pens
and brushes!

I look forward to your reply.

Alec
--
You received this message because you are subscribed to the Google Groups
"wxPython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to wxpython-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"wxPython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to wxpython-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov