wxPython (and Matplotlib) Strangeness

wxPython Users
1

Hello NG,

    in my application (that is reaching 50,000 lines of code at the moment)
I use wxPython, matplotlib and other useful resources. I am having 2 strange
behaviors, one related to matplotlib and the other to wxPython.

1) wxPython Problem

When my app starts, it loads a bunch of modules and creates a main wx.Notebook,
with a lot of buttons, grid and textctrls and so on. When it starts, a wx.Gauge
+ a Splashscreen inform the user on the loading progress (usually it takes
5-6 seconds) and at the end the main frame is shown and maximized.
Well, I have launched the Windows Task Manager to monitor the RAM consuption.
This is what happens:

a) Loading and showing everything increase the RAM usage to 70 MB (!!!)
BUT;
b) If I minimize the main frame the RAM drops down to 4 MB AND;
c) By re-maximizing the main frame the RAM goes to 7 MB.

Now, if I do some interaction with the wxThings in all the tabs in the wx.Notebook,
the RAM keeps increasing. If at a certain point I minimize the main frame
the RAM goes back to 4 MB and re-maximizing it raises the RAM to 8 MB.

Question: what is going on? Is there something (probably easy) that I am
not aware of on Windows machines? Why the minimize/maximize actions have
this effect? Is there a way to reproduce the effect of minimize/maximize
with actually doing it? May it be a memory leaking problem somewhere?

2) Matplotlib problem

In the same app, I have a wx.Grid in which, by clicking with the right mouse
button, a row is selected and a popup menu appears. By choosing the right
item on this submenu, a frame with a Matplotlib figure (and other small
things) is created. Then I close the frame and I redo the same action. Every
time I do this action, it seems that Matplotlib is sucking 4 MB of RAM,
that does not decrease after I have closed the Matplotlib frame.
At a certain point, my app goes over 150 MB of RAM only by having opened/closed
some Matplotlib figures. The only solution I found is still to minimize/maximize
the main GUI frame (as in Problem 1).

Question: why the RAM consumption does not decrease when I close the Matplotlib
frame? Shouldn't be the gc that takes care of doing such action?

Thank you *a lot* for every suggestion, comment, opinion.
Sorry for the long post.

Andrea.

2

Hello NG,

    in my application (that is reaching 50,000 lines of code at the moment)
I use wxPython, matplotlib and other useful resources. I am having 2 strange
behaviors, one related to matplotlib and the other to wxPython.

1) wxPython Problem

When my app starts, it loads a bunch of modules and creates a main wx.Notebook,
with a lot of buttons, grid and textctrls and so on. When it starts, a wx.Gauge
+ a Splashscreen inform the user on the loading progress (usually it takes
5-6 seconds) and at the end the main frame is shown and maximized.
Well, I have launched the Windows Task Manager to monitor the RAM consuption.
This is what happens:

a) Loading and showing everything increase the RAM usage to 70 MB (!!!)
BUT;
b) If I minimize the main frame the RAM drops down to 4 MB AND;
c) By re-maximizing the main frame the RAM goes to 7 MB.

Now, if I do some interaction with the wxThings in all the tabs in the wx.Notebook,
the RAM keeps increasing. If at a certain point I minimize the main frame
the RAM goes back to 4 MB and re-maximizing it raises the RAM to 8 MB.

Question: what is going on? Is there something (probably easy) that I am
not aware of on Windows machines? Why the minimize/maximize actions have
this effect? Is there a way to reproduce the effect of minimize/maximize
with actually doing it? May it be a memory leaking problem somewhere?

2) Matplotlib problem

In the same app, I have a wx.Grid in which, by clicking with the right mouse
button, a row is selected and a popup menu appears. By choosing the right
item on this submenu, a frame with a Matplotlib figure (and other small
things) is created. Then I close the frame and I redo the same action. Every
time I do this action, it seems that Matplotlib is sucking 4 MB of RAM,
that does not decrease after I have closed the Matplotlib frame.
At a certain point, my app goes over 150 MB of RAM only by having opened/closed
some Matplotlib figures. The only solution I found is still to minimize/maximize
the main GUI frame (as in Problem 1).

Question: why the RAM consumption does not decrease when I close the Matplotlib
frame? Shouldn't be the gc that takes care of doing such action?

Thank you *a lot* for every suggestion, comment, opinion.
Sorry for the long post.

Andrea.

The memory usage reported by Windows is notoriously inaccurate and
unreliable. The minimize/restore issue is actually a common case, and
I believe the root cause has something to do with the Windows window
manager, and it's internal storage being charged to your application.
It happens with every windows application I've checked.

···

On 7/15/05, andrea_gavana@tin.it <andrea_gavana@tin.it> wrote:

---------------------------------------------------------------------
To unsubscribe, e-mail: wxPython-users-unsubscribe@lists.wxwidgets.org
For additional commands, e-mail: wxPython-users-help@lists.wxwidgets.org

3

The memory usage reported by Windows is notoriously inaccurate and
unreliable. The minimize/restore issue is actually a common case, and
I believe the root cause has something to do with the Windows window
manager, and it's internal storage being charged to your application.
It happens with every windows application I've checked.

Here you can find a little more information about the problem:

4

Hi Andrea,

andrea_gavana@tin.it wrote:

Hello NG,

   in my application (that is reaching 50,000 lines of code at the moment)
I use wxPython, matplotlib and other useful resources. I am having 2 strange
behaviors, one related to matplotlib and the other to wxPython.

1) wxPython Problem

When my app starts, it loads a bunch of modules and creates a main wx.Notebook,
with a lot of buttons, grid and textctrls and so on. When it starts, a wx.Gauge
+ a Splashscreen inform the user on the loading progress (usually it takes
5-6 seconds) and at the end the main frame is shown and maximized.
Well, I have launched the Windows Task Manager to monitor the RAM consuption.
This is what happens:

a) Loading and showing everything increase the RAM usage to 70 MB (!!!)
BUT;
b) If I minimize the main frame the RAM drops down to 4 MB AND;

Did you check the Virtual Memory Size column? I think/guess that the drop in the Mem Usage column has to do with Windows releasing the memory (RAM) of an application which is minimized but it is still in the Paging File.

Quote from the Task Manager help:
Memory Usage
In Task Manager, the current working set of a process, in kilobytes. The current working set is the number of pages currently resident in memory. On the *Task Manager Processes* tab, the column heading is *Mem Usage*.

c) By re-maximizing the main frame the RAM goes to 7 MB.

Now, if I do some interaction with the wxThings in all the tabs in the wx.Notebook,
the RAM keeps increasing. If at a certain point I minimize the main frame
the RAM goes back to 4 MB and re-maximizing it raises the RAM to 8 MB.

Question: what is going on? Is there something (probably easy) that I am
not aware of on Windows machines? Why the minimize/maximize actions have
this effect? Is there a way to reproduce the effect of minimize/maximize
with actually doing it? May it be a memory leaking problem somewhere?

For a similar problem I had on Win 9x I used gc.collect() and a progress dialog to allow Win 9x to recover resources during program start up. On Win XP or 2000 I don't see to need to progress dialog, just calling gc seems to be enough.

2) Matplotlib problem

In the same app, I have a wx.Grid in which, by clicking with the right mouse
button, a row is selected and a popup menu appears. By choosing the right
item on this submenu, a frame with a Matplotlib figure (and other small
things) is created. Then I close the frame and I redo the same action. Every
time I do this action, it seems that Matplotlib is sucking 4 MB of RAM,
that does not decrease after I have closed the Matplotlib frame.
At a certain point, my app goes over 150 MB of RAM only by having opened/closed
some Matplotlib figures. The only solution I found is still to minimize/maximize
the main GUI frame (as in Problem 1).

Question: why the RAM consumption does not decrease when I close the Matplotlib
frame? Shouldn't be the gc that takes care of doing such action?

Thank you *a lot* for every suggestion, comment, opinion.
Sorry for the long post.

Andrea.

See you
Werner

5

Hi list,

I seem to have a similar problem with a database application that is also
around 50,000 lines. the application is run under Linux, and seems to grow
during the day from 70 MB to more than 100 MB. We run the application on a
central server and distribute the screens to the user thru GoGlobal. For each
user there is 21 MB shared memory. So this means there is an average of about
65 MB per user. As we have many users the machine it is running on is a 4 GB
dual processor and it seems to run pretty good, but still, the growth of
memory use is bothering me. Is there anything I can do to minimize the memory
usage? I thought Python would take care of the memory management and garbage
collection, or should I take some special measures?

I read some articles about the garbage collector, and they stated that memory
once claimed by python is under certain circumstances never released. Are
there some guru's around that can comment on this?

Cheers,
Dick Kniep

···

Op zaterdag 16 juli 2005 12:57, schreef Werner F. Bruhin:

Hi Andrea,

andrea_gavana@tin.it wrote:
>Hello NG,
>
> in my application (that is reaching 50,000 lines of code at the
> moment) I use wxPython, matplotlib and other useful resources. I am
> having 2 strange behaviors, one related to matplotlib and the other to
> wxPython.
>
>1) wxPython Problem
>
>When my app starts, it loads a bunch of modules and creates a main
> wx.Notebook, with a lot of buttons, grid and textctrls and so on. When it
> starts, a wx.Gauge + a Splashscreen inform the user on the loading
> progress (usually it takes 5-6 seconds) and at the end the main frame is
> shown and maximized. Well, I have launched the Windows Task Manager to
> monitor the RAM consuption. This is what happens:
>
>a) Loading and showing everything increase the RAM usage to 70 MB (!!!)
>BUT;
>b) If I minimize the main frame the RAM drops down to 4 MB AND;

Did you check the Virtual Memory Size column? I think/guess that the
drop in the Mem Usage column has to do with Windows releasing the memory
(RAM) of an application which is minimized but it is still in the Paging
File.

Quote from the Task Manager help:
Memory Usage
In Task Manager, the current working set of a process, in kilobytes. The
current working set is the number of pages currently resident in memory.
On the *Task Manager Processes* tab, the column heading is *Mem Usage*.

>c) By re-maximizing the main frame the RAM goes to 7 MB.
>
>Now, if I do some interaction with the wxThings in all the tabs in the
> wx.Notebook, the RAM keeps increasing. If at a certain point I minimize
> the main frame the RAM goes back to 4 MB and re-maximizing it raises the
> RAM to 8 MB.
>
>Question: what is going on? Is there something (probably easy) that I am
>not aware of on Windows machines? Why the minimize/maximize actions have
>this effect? Is there a way to reproduce the effect of minimize/maximize
>with actually doing it? May it be a memory leaking problem somewhere?

For a similar problem I had on Win 9x I used gc.collect() and a progress
dialog to allow Win 9x to recover resources during program start up. On
Win XP or 2000 I don't see to need to progress dialog, just calling gc
seems to be enough.

>2) Matplotlib problem
>
>In the same app, I have a wx.Grid in which, by clicking with the right
> mouse button, a row is selected and a popup menu appears. By choosing the
> right item on this submenu, a frame with a Matplotlib figure (and other
> small things) is created. Then I close the frame and I redo the same
> action. Every time I do this action, it seems that Matplotlib is sucking
> 4 MB of RAM, that does not decrease after I have closed the Matplotlib
> frame.
>At a certain point, my app goes over 150 MB of RAM only by having
> opened/closed some Matplotlib figures. The only solution I found is still
> to minimize/maximize the main GUI frame (as in Problem 1).
>
>Question: why the RAM consumption does not decrease when I close the
> Matplotlib frame? Shouldn't be the gc that takes care of doing such
> action?

6

Hello NG,

thanks Chris, Werner & Nitro for your nice and useful responses.

The memory usage reported by Windows is notoriously inaccurate and
unreliable. The minimize/restore issue is actually a common case, and
I believe the root cause has something to do with the Windows window
manager, and it's internal storage being charged to your application.
It happens with every windows application I've checked.

Let's say I don't trust the memory usage reported by Windows. What does
it happen when this "false" memory usage reaches 512 MB (or 1GB or whatever
you want)? Suppose it reaches the RAM memory on my PC... will my PC begin
swap? Will it be slower? Will it eventually crash because of this RAM usage.
Because if this are the results, it doesn't matter if Windows reports correctly
the memory usage or not, because my application is *still* leaking memory
somewhere, with no reason.

Am I missing something else?

Andrea

7

Hello NG,

thanks Chris, Werner & Nitro for your nice and useful responses.

>The memory usage reported by Windows is notoriously inaccurate and
>unreliable. The minimize/restore issue is actually a common case, and
>I believe the root cause has something to do with the Windows window
>manager, and it's internal storage being charged to your application.
>It happens with every windows application I've checked.

Let's say I don't trust the memory usage reported by Windows. What does
it happen when this "false" memory usage reaches 512 MB (or 1GB or whatever
you want)? Suppose it reaches the RAM memory on my PC... will my PC begin
swap? Will it be slower? Will it eventually crash because of this RAM usage.
Because if this are the results, it doesn't matter if Windows reports correctly
the memory usage or not, because my application is *still* leaking memory
somewhere, with no reason.

Memory leaks and the like will be a problem, of course, the point is
that the task manager is not an accurate way of finding/identifying
these leaks. In C/C++, I use custom allocators to track the heap so I
can get very accurate measures of all the memory my code (as opposed
to Windows library code) uses. If you're up to it I believe that you
can compile Python with hooks to allow similiar functionality (but
note that tracking memory leaks in a garbage collected environment is
very difficult, especially with complex programs).

You can't do anything directly about any memory issues that may come
from leaks within Windows (especially from Python), and tracing enough
to track down exactly where they are requires some pretty low level
skills (and tools).

The specific case you're mentioning, with the memory usage fluctuating
as you minimize/restore, it's a specific case of the Task Manager not
accurately reporting memory usage, and even if it were, it's not
something you can affect - again, *all* windows applications will show
this behavior.

···

On 7/17/05, andrea_gavana@tin.it <andrea_gavana@tin.it> wrote:

Am I missing something else?

Andrea