Namespace Diff Tool import dichotomy (!)

Hi All,

this is not (yet) and announcement, just a request for suggestions as I am struggling a bit to do The Right Thing ™. I am still coding a little tool Robin has christened as “Namespace Diff Tool”:

http://svn.wxwidgets.org/viewvc/wx/wxPython/Phoenix/trunk/TODO.txt?view=markup

One of the (many) issues I am having is related to how the import mechanism works in Python and where the code for a class/method/attribute actually is. These are the given, for the moment:

  1. The tool is generic enough to be used on other libraries/packages. I have successfully used it to import and analyze more than 30 libraries[1];

  2. I use the Python import builtin function at runtime to find out the package/module/class hierarchies, code, methods signatures, attributes, docstrings and so on.

Now, if you look at the first screenshot I am attaching (NDT_Import_1.png), and specifically at the agw.aui tree, you will see that this tree contains all the classes, methods and attributes for all the widgets in agw.aui. We can focus, for example, on the AuiCenterDockingGuide class in the center of the screeenshot. However, the AuiCenterDockingGuide class is defined in the framemanager module and, when I import the framemanager module itself, another AuiCenterDockingGuide class appears as a child of the framemanager node, as you can see in the second screenshot (NDT_Import_2.png).

The dichotomy here is that, although the code for this class actually lives in agw.aui.framemanager, the objects themselves, once imported, live in the agw.aui namespace because the init.py file for agw.aui contains the following statements:

from aui_constants import *

from aui_utilities import *

from auibar import *

from auibook import *

from tabart import *

from dockart import *

from framemanager import *

from tabmdi import *

Now, I would like to preserve the hierarchy package => module => class inside the NDT trees, which means that the framemanager module should be there as a separate node (and as a child of the agw.aui package), containing all its children (classes, methods, attributes, etc…), but I have no idea how to avoid duplication of tree children because of the Python import mechanism. This effect is even worse on other libraries (like numpy, for example) which do some heavy voodoo importing stuff in the init.py files for their sub-packages. You can actually see what happens with numpy when I filter the package tree for the “complex128” class in the third screenshot (NDT_numpy.png). This “complex128” class lives in the numpy namespace but I believe its declaration is inside the numpy.core package or the numpy.core.numerictypes module.

On the same note, if we consider core classes like wx.TextCtrl, they do live in the wx namespace but their definition is inside the _controls.py Python file. At the moment I am filtering out the import of modules which name starts with an underscore, but I am not sure this is a comprehensive and fault-free approach.

Any suggestion on how to handle this issues is most welcome. I can live with the class/method/attribute duplication but it would be nice if it didn’t happen…

Thank you in advance.

[1] In my test database I currently have successfully imported the following libraries/packages:

apptools-4.0.0, Bio-1.57, cloud-2.2.4, coverage-3.5, Crypto-2.3, Cython-0.14.1, dateutil-1.5, docutils-0.7, fwrap-1.0, html5lib-0.90, IPython-0.11, jinja2-2.5.5, lxml-2.0, matplotlib-1.0.1, mayavi-4.0.0, networkx-1.5, numpy-1.6.1, OpenSSL-0.12, paramiko-1.7.7.1, PySide-1.0.5, pytz-2011g, reportlab-3788, scipy-0.9.0, setuptools-0.6, sphinx-1.0.7, sqlalchemy-0.7.1, sympy-0.7.1, traits-4.0.0, vtk-5.0, wx-2.9.2.4, xlrd-0.7.1, xlwt-0.7.1, zope-1.0

Andrea.

“Imagination Is The Only Weapon In The War Against Reality.”
http://xoomer.alice.it/infinity77/

import PyQt4.QtGui

Traceback (most recent call last):

File “”, line 1, in

ImportError: No module named PyQt4.QtGui

import pygtk

Traceback (most recent call last):

File “”, line 1, in

ImportError: No module named pygtk

···

import wx

Are you using the inspect module? The WIT uses inspect.getmodule(obj) to get the real module where the object's class is defined. I haven't used it much beyond that, but it looks like the inspect module is able to tell you just about anything you would ever want to know about python objects and code.

···

On 10/26/11 1:23 AM, Andrea Gavana wrote:

Now, I would like to preserve the hierarchy package => module => class
inside the NDT trees, which means that the framemanager module should be
there as a separate node (and as a child of the agw.aui package),
containing all its children (classes, methods, attributes, etc...), but
I have no idea how to avoid duplication of tree children because of the
Python import mechanism.

--
Robin Dunn
Software Craftsman

Oh, there is also the __module__ attribute in most (all?) objects that will give you the just the name of the module rather than the module itself. You could use to to see if the object is from the module you are currently looking at and only add it to the tree if it is.

···

On 10/26/11 10:11 AM, Robin Dunn wrote:

On 10/26/11 1:23 AM, Andrea Gavana wrote:

Now, I would like to preserve the hierarchy package => module => class
inside the NDT trees, which means that the framemanager module should be
there as a separate node (and as a child of the agw.aui package),
containing all its children (classes, methods, attributes, etc...), but
I have no idea how to avoid duplication of tree children because of the
Python import mechanism.

Are you using the inspect module? The WIT uses inspect.getmodule(obj) to
get the real module where the object's class is defined. I haven't used
it much beyond that, but it looks like the inspect module is able to
tell you just about anything you would ever want to know about python
objects and code.

--
Robin Dunn
Software Craftsman

Hi Robin,

Now, I would like to preserve the hierarchy package => module => class

inside the NDT trees, which means that the framemanager module should be

there as a separate node (and as a child of the agw.aui package),

containing all its children (classes, methods, attributes, etc…), but

I have no idea how to avoid duplication of tree children because of the

Python import mechanism.

Are you using the inspect module? The WIT uses inspect.getmodule(obj) to

get the real module where the object’s class is defined. I haven’t used

it much beyond that, but it looks like the inspect module is able to

tell you just about anything you would ever want to know about python

objects and code.

Oh, there is also the module attribute in most (all?) objects that will give you the just the name of the module rather than the module itself. You could use to to see if the object is from the module you are currently looking at and only add it to the tree if it is.

Thank you for the suggestions. I was somewhat aware of inspect.getmodule and the module attribute, although I hadn’t considered it in this way yet. However, my question was mostly related to how we would like to see these classes/methods/attributes displayed in the tree. For example, considering wx.TextCtrl, it lives in the wx namespace but:

import wx, numpy, inspect

inspect.getmodule(wx.TextCtrl)

<module ‘wx._controls’ from ‘C:\Python27\lib\site-packages\wx-2.9.2-msw\wx_controls.pyc’>

It will end up under the _controls sub-tree, so it might look counter-intuitive if someone was looking for wx.TextCtrl under the main wx tree. Moreover, inspect.getmodule can be unreliable on some (built-in) classes/methods/attributes:

inspect.getmodule(numpy.abs) # Nothing is printed

inspect.getmodule(numpy.zeros)

<module ‘numpy.core.multiarray’ from ‘C:\Python27\lib\site-packages\numpy\core\multiarray.pyd’>

inspect.getmodule(wx.BLACK)

<module ‘wx._gdi’ from ‘C:\Python27\lib\site-packages\wx-2.9.2-msw\wx_gdi.pyc’>

inspect.getmodule(wx.SYS_DEFAULT_GUI_FONT) # Nothing is printed

inspect.getmodule(wx.BORDER_NONE) # Nothing is printed

I am not clear if I am missing something here or not…

Andrea.

“Imagination Is The Only Weapon In The War Against Reality.”

http://xoomer.alice.it/infinity77/

import PyQt4.QtGui

Traceback (most recent call last):

File “”, line 1, in

ImportError: No module named PyQt4.QtGui

import pygtk

Traceback (most recent call last):

File “”, line 1, in

ImportError: No module named pygtk

···

On 26 October 2011 20:17, Robin Dunn wrote:

On 10/26/11 10:11 AM, Robin Dunn wrote:

On 10/26/11 1:23 AM, Andrea Gavana wrote:

import wx

Hi,

···

2011/10/26 Andrea Gavana andrea.gavana@gmail.com

this is not (yet) and announcement, just a request for suggestions as I am struggling a bit to do The Right Thing ™. I am still coding a little tool Robin has christened as “Namespace Diff Tool”:

http://svn.wxwidgets.org/viewvc/wx/wxPython/Phoenix/trunk/TODO.txt?view=markup

Sounds like a great tool!
Do you think it can be extended to the Python interpreter itself?

i.e. the library is the complete standard library, together with types and functions built into the interpreter?

It could be used to compare different versions of Python,

but also different implementations, like Pypy vs. CPython!


Amaury Forgeot d’Arc

Hi Amaury,

Hi,

this is not (yet) and announcement, just a request for suggestions as I am struggling a bit to do The Right Thing ™. I am still coding a little tool Robin has christened as “Namespace Diff Tool”:

http://svn.wxwidgets.org/viewvc/wx/wxPython/Phoenix/trunk/TODO.txt?view=markup

Sounds like a great tool!
Do you think it can be extended to the Python interpreter itself?

I think so, although at the moment it recursively import a library based on the init.py file of that library (the user interface asks you to choose the init.py file before starting the import stuff). I could simply provide a switch to the user, whereby he/she can choose the init.py file for a library or choose to import an entire directory tree.

Curiously enough, there is no init.py file in my C:\Python27\Lib folder (!).

i.e. the library is the complete standard library, together with types and functions built into the interpreter?

It could be used to compare different versions of Python,

but also different implementations, like Pypy vs. CPython!

As soon as I finish the current version I’ll start looking into this possibility. However, the user interface is based on wxPython, and in order to run the GUI and the import stuff at the same time you have to use the same interpreter, and I doubt PyPy supports wxPython.

NDT screenshots:

http://imageshack.us/f/811/ndt1.png

http://imageshack.us/f/706/ndt2.png

http://imageshack.us/f/560/ndtfilter.png

Andrea.

“Imagination Is The Only Weapon In The War Against Reality.”

http://xoomer.alice.it/infinity77/

import PyQt4.QtGui

Traceback (most recent call last):

File “”, line 1, in

ImportError: No module named PyQt4.QtGui

import pygtk

Traceback (most recent call last):

File “”, line 1, in

ImportError: No module named pygtk

···

On 26 October 2011 23:24, Amaury Forgeot d’Arc wrote:

2011/10/26 Andrea Gavana andrea.gavana@gmail.com

import wx

Since they are hashable you can detect duplicates by putting them in a
dict. So then the problem is to work out which is the documented one.
I would imagine that in the vast majority of packages the name
closest to the top of the tree is the documented one. A primitive way
of detecting that is to count the dots, and possibly skipping private
modules.

You could also provide an overrides file for those cases where this
heuristic isn't correct.

  namespace_overrides = {'wx.agw.extras.OtherCtrl', ...}
  classes = { TextCtrl : ['wx', 'wx._controls'],
              Button : ['wx', 'core'],
              OtherCtrl : ['wx._core', 'wx.agw.extras'}

  def find_override(class_, namespaces):
      for ns in namespaces:
          qualified = ns + '.' + class_.__name__
          if qualified in namespace_overrides:
              return qualified

  for class_, namespaces in classes.items():
      documented = find_override(class_, namespaces)
      if not documented:
          namespaces.sort(key=lambda x : sys.maxint if
x.startswith('_') else x.count('.'))
          documented = namespaces[0]

Sam

···

On 26 October 2011 20:55, Andrea Gavana <andrea.gavana@gmail.com> wrote:

Thank you for the suggestions. I was somewhat aware of inspect.getmodule and
the __module__ attribute, although I hadn't considered it in this way yet.
However, my question was mostly related to how we would like to see these
classes/methods/attributes displayed in the tree. For example, considering
wx.TextCtrl, it lives in the wx namespace but:

import wx, numpy, inspect
inspect.getmodule(wx.TextCtrl)

<module 'wx._controls' from
'C:\Python27\lib\site-packages\wx-2.9.2-msw\wx\_controls.pyc'>
It will end up under the _controls sub-tree, so it might look
counter-intuitive if someone was looking for wx.TextCtrl under the main wx
tree.

On 26 October 2011 21:46, I wrote badly:

     namespaces\.sort\(key=lambda x : sys\.maxint if

x.startswith('_') else x.count('.'))

Sorry, that test for private module is rubbish. "startswith('_) or
'._' in x" might be better

Anyway, you can get the idea I think.

Sam

Hi Sam,

Thank you for the suggestions. I was somewhat aware of inspect.getmodule and

the module attribute, although I hadn’t considered it in this way yet.

However, my question was mostly related to how we would like to see these

classes/methods/attributes displayed in the tree. For example, considering

wx.TextCtrl, it lives in the wx namespace but:

import wx, numpy, inspect

inspect.getmodule(wx.TextCtrl)

<module ‘wx._controls’ from

‘C:\Python27\lib\site-packages\wx-2.9.2-msw\wx_controls.pyc’>

It will end up under the _controls sub-tree, so it might look

counter-intuitive if someone was looking for wx.TextCtrl under the main wx

tree.

Since they are hashable you can detect duplicates by putting them in a

dict. So then the problem is to work out which is the documented one.

I would imagine that in the vast majority of packages the name

closest to the top of the tree is the documented one. A primitive way

of detecting that is to count the dots, and possibly skipping private

modules.

You could also provide an overrides file for those cases where this

heuristic isn’t correct.

namespace_overrides = {‘wx.agw.extras.OtherCtrl’, …}

classes = { TextCtrl : [‘wx’, ‘wx._controls’],

          Button : ['wx', 'core'],

          OtherCtrl : ['wx._core', 'wx.agw.extras'}

def find_override(class_, namespaces):

  for ns in namespaces:

      qualified = ns + '.' + class_.__name__

      if qualified in namespace_overrides:

          return qualified

for class_, namespaces in classes.items():

  documented = find_override(class_, namespaces)

  if not documented:

      namespaces.sort(key=lambda x : sys.maxint if

x.startswith(‘_’) else x.count(‘.’))

      documented = namespaces[0]

I don’t think that will work: wx.TextCtrl and wx._controls.TextCtrl are the same object, they have the same attributes, methods, docs and anything else you can think of:

import wx

wx.TextCtrl.doc

‘Proxy of C++ TextCtrl class’

wx._controls.TextCtrl.doc

‘Proxy of C++ TextCtrl class’

wx.TextCtrl == wx._controls.TextCtrl

True

wx.TextCtrl is wx._controls.TextCtrl

True

There is no way to differentiate between them. And even if I exclude modules starting with an underscore, other libraries (i.e., numpy) do not have any underscored module and they show the same issue (i.e., duplicated children in the library tree).

Andrea.

“Imagination Is The Only Weapon In The War Against Reality.”
http://xoomer.alice.it/infinity77/

import PyQt4.QtGui

Traceback (most recent call last):

File “”, line 1, in

ImportError: No module named PyQt4.QtGui

import pygtk

Traceback (most recent call last):

File “”, line 1, in

ImportError: No module named pygtk

···

On 26 October 2011 23:46, Sam Partington wrote:

On 26 October 2011 20:55, Andrea Gavana andrea.gavana@gmail.com wrote:

import wx

It does! http://morepypy.blogspot.com/2010/05/running-wxpython-on-top-of-pypy.html

···

2011/10/26 Andrea Gavana andrea.gavana@gmail.com

i.e. the library is the complete standard library, together with types and functions built into the interpreter?

It could be used to compare different versions of Python,

but also different implementations, like Pypy vs. CPython!

As soon as I finish the current version I’ll start looking into this possibility. However, the user interface is based on wxPython, and in order to run the GUI and the import stuff at the same time you have to use the same interpreter, and I doubt PyPy supports wxPython.


Amaury Forgeot d’Arc

i.e. the library is the complete standard library, together with types and functions built into the interpreter?

It could be used to compare different versions of Python,

but also different implementations, like Pypy vs. CPython!

As soon as I finish the current version I’ll start looking into this possibility. However, the user interface is based on wxPython, and in order to run the GUI and the import stuff at the same time you have to use the same interpreter, and I doubt PyPy supports wxPython.

It does! http://morepypy.blogspot.com/2010/05/running-wxpython-on-top-of-pypy.html

Jesus, you’ve got to be kidding… I wasn’t expecting that, at all, I thought PyPy was far from being able to run libraries I most often use (wxPython, numpy, matplotlib, etc…). My compliments to the PyPy guys (although I can not see the screenshots in the blog post for some reason…)

Andrea.

“Imagination Is The Only Weapon In The War Against Reality.”
http://xoomer.alice.it/infinity77/

import PyQt4.QtGui

Traceback (most recent call last):

File “”, line 1, in

ImportError: No module named PyQt4.QtGui

import pygtk

Traceback (most recent call last):

File “”, line 1, in

ImportError: No module named pygtk

···

On 26 October 2011 23:58, Amaury Forgeot d’Arc wrote:

2011/10/26 Andrea Gavana andrea.gavana@gmail.com

import wx

Ok, here's an idea. In the Dependency Walker tool they have a similar issue with DLLs where there are different types of DLL dependencies, such that the same DLL can show up in the tree in different places. They distinguish between the dependency types with different icons.

So in the NDT we'll have basically have three kinds of references, those that appear in their "real" module, those that are "imported" from another module, and those that you don't have enough information to be able to tell. Let's ignore the last type for a moment and just look at the first two... You could show both of them in the tree but make it clear which are the real items (those defined in that module) and those that are imported. I'm thinking that maybe the imported items could have an italics font or maybe use a mid-grey color. If the items in the module are sorted such that the real items are at the top and the imported items are after that then that would also help distinguish them. Finally, another idea that hits me is that perhaps selecting an imported item, or maybe double-clicking or using a context menu item, would jump the selection to the place in the tree where the corresponding real item is located.

For items that you can't get enough information to tell where they are actually located I'm not sure what to suggest. I guess you'll have to assume that they are in the module that you are scanning. It might make sense however to have options to be able to filter these items out of the list since they are likely to not be the kinds of items that the user wants to examine closely anyway. And when doing a diff between versions of the namespace just reporting that one has an item FOO and the other does not will probably be sufficient.

···

On 10/26/11 12:55 PM, Andrea Gavana wrote:

Hi Robin,

On 26 October 2011 20:17, Robin Dunn wrote:

    On 10/26/11 10:11 AM, Robin Dunn wrote:

        On 10/26/11 1:23 AM, Andrea Gavana wrote:

            Now, I would like to preserve the hierarchy package =>
            module => class
            inside the NDT trees, which means that the framemanager
            module should be
            there as a separate node (and as a child of the agw.aui
            package),
            containing all its children (classes, methods, attributes,
            etc...), but
            I have no idea how to avoid duplication of tree children
            because of the
            Python import mechanism.

        Are you using the inspect module? The WIT uses
        inspect.getmodule(obj) to
        get the real module where the object's class is defined. I
        haven't used
        it much beyond that, but it looks like the inspect module is able to
        tell you just about anything you would ever want to know about
        python
        objects and code.

    Oh, there is also the __module__ attribute in most (all?) objects
    that will give you the just the name of the module rather than the
    module itself. You could use to to see if the object is from the
    module you are currently looking at and only add it to the tree if
    it is.

Thank you for the suggestions. I was somewhat aware of inspect.getmodule
and the __module__ attribute, although I hadn't considered it in this
way yet. However, my question was mostly related to how we would like to
see these classes/methods/attributes displayed in the tree. For example,
considering wx.TextCtrl, it lives in the wx namespace but:

>>> import wx, numpy, inspect
>>> inspect.getmodule(wx.TextCtrl)
<module 'wx._controls' from
'C:\Python27\lib\site-packages\wx-2.9.2-msw\wx\_controls.pyc'>

It will end up under the _controls sub-tree, so it might look
counter-intuitive if someone was looking for wx.TextCtrl under the main
wx tree.

--
Robin Dunn
Software Craftsman

Argh! imagebin.ca does not work anymore. I fear the images are lost forever :frowning:
wxPython was not too hard to support, actually. But it was the SWIG version.

I’ll see how it behaves on Phonenix versions.

···

2011/10/26 Andrea Gavana andrea.gavana@gmail.com

As soon as I finish the current version I’ll start looking into this possibility. However, the user interface is based on wxPython, and in order to run the GUI and the import stuff at the same time you have to use the same interpreter, and I doubt PyPy supports wxPython.

It does! http://morepypy.blogspot.com/2010/05/running-wxpython-on-top-of-pypy.html

Jesus, you’ve got to be kidding… I wasn’t expecting that, at all, I thought PyPy was far from being able to run libraries I most often use (wxPython, numpy, matplotlib, etc…). My compliments to the PyPy guys (although I can not see the screenshots in the blog post for some reason…)


Amaury Forgeot d’Arc

Hi Andrea,

I don't think that will work: wx.TextCtrl and wx._controls.TextCtrl are the
*same* object, they have the same attributes, methods, docs and anything
else you can think of:

Yes exactly :- they are the same object so you can filter out the
duplicates. The only problem is to work out which module is the best
module to report that the object lives in.

Perhaps I don't understand the problem. As I understand it you walk
the package, building up a collection of things in each module.

So in module wx you have TextCtrl, Button whatever.
But as you also import wx._controls you get a TextCtrl and Button
there too and it appears in your tree twice.

modules = { 'wx' : [TextCtrl, Button, ...],
'wx._controls' : [TextCtrl, Button, ...] }

So turn that data structure around :

all_things = collections.defaultdict(list)
for module, things in modules:
    for thing in things:
        all_things[thing].append(module)

Then run the sort heuristic on all_things as in my previous email
(simplified for brevity).

thing_locations = {}
for thing, modules in all_things.items():
    modules.sort(key=lambda x.count('.'))
    thing_locations[thing] = modules[0]

Then as you walk the tree to display it you just need to see if the
parent module of the current node is equal to
class_locations[nodething], and if so show it. If not then don't. You
also probably want to mark each thing when you display it so you can
check that everything got displayed somewhere.

There is no way to differentiate between them. And even if I exclude modules
starting with an underscore, other libraries (i.e., numpy) do not have any
underscored module and they show the same issue (i.e., duplicated children
in the library tree).

I don't think the private module will be necessary normally, but it
may improve the guess in some packages.

Sorry if I've missed the point completely.

Sam

···

On 26 October 2011 21:55, Andrea Gavana <andrea.gavana@gmail.com> wrote:

Hi Andrea,

···

2011/10/26 Andrea Gavana <andrea.gavana@gmail.com>:

Now, I would like to preserve the hierarchy package => module => class
inside the NDT trees, which means that the framemanager module should be
there as a separate node (and as a child of the agw.aui package), containing
all its children (classes, methods, attributes, etc...), but I have no idea
how to avoid duplication of tree children because of the Python import
mechanism.

You're creating a Directed Acyclic Graph (DAG) and obviously the thing
you need to do is create wx.agw.DAGCtrl and then use that to display
the data :slight_smile:

Regards, Frank

Hi All,

Hi Robin,

        Now, I would like to preserve the hierarchy package =>

        module => class

        inside the NDT trees, which means that the framemanager

        module should be

        there as a separate node (and as a child of the agw.aui

        package),

        containing all its children (classes, methods, attributes,

        etc...), but

        I have no idea how to avoid duplication of tree children

        because of the

        Python import mechanism.





    Are you using the inspect module? The WIT uses

    inspect.getmodule(obj) to

    get the real module where the object's class is defined. I

    haven't used

    it much beyond that, but it looks like the inspect module is able to

    tell you just about anything you would ever want to know about

    python

    objects and code.





Oh, there is also the __module__ attribute in most (all?) objects

that will give you the just the name of the module rather than the

module itself.  You could use to to see if the object is from the

module you are currently looking at and only add it to the tree if

it is.

Thank you for the suggestions. I was somewhat aware of inspect.getmodule

and the module attribute, although I hadn’t considered it in this

way yet. However, my question was mostly related to how we would like to

see these classes/methods/attributes displayed in the tree. For example,

considering wx.TextCtrl, it lives in the wx namespace but:

import wx, numpy, inspect

inspect.getmodule(wx.TextCtrl)

<module ‘wx._controls’ from

‘C:\Python27\lib\site-packages\wx-2.9.2-msw\wx_controls.pyc’>

It will end up under the _controls sub-tree, so it might look

counter-intuitive if someone was looking for wx.TextCtrl under the main

wx tree.

Ok, here’s an idea. In the Dependency Walker tool they have a similar issue with DLLs where there are different types of DLL dependencies, such that the same DLL can show up in the tree in different places. They distinguish between the dependency types with different icons.

So in the NDT we’ll have basically have three kinds of references, those that appear in their “real” module, those that are “imported” from another module, and those that you don’t have enough information to be able to tell. Let’s ignore the last type for a moment and just look at the first two… You could show both of them in the tree but make it clear which are the real items (those defined in that module) and those that are imported. I’m thinking that maybe the imported items could have an italics font or maybe use a mid-grey color. If the items in the module are sorted such that the real items are at the top and the imported items are after that then that would also help distinguish them. Finally, another idea that hits me is that perhaps selecting an imported item, or maybe double-clicking or using a context menu item, would jump the selection to the place in the tree where the corresponding real item is located.

For items that you can’t get enough information to tell where they are actually located I’m not sure what to suggest. I guess you’ll have to assume that they are in the module that you are scanning. It might make sense however to have options to be able to filter these items out of the list since they are likely to not be the kinds of items that the user wants to examine closely anyway. And when doing a diff between versions of the namespace just reporting that one has an item FOO and the other does not will probably be sufficient.

Just an update on this, and possibly a request for suggestions/help as I am growing so weary of this Python importing mechanism stuff that I’ll soon give up.

The main issue goes back to how to import stuff and dynamically analyzing a package. Let’s suppose that I am using wxPython 2.9.2.4 to display the user interface, and I want to scan and analyze wxPython 2.8.11 as a package using NDT. No matter what the heck I do, Python keeps importing stuff from 2.9.2.4 because it is already in sys.modules (and playing with sys.modules doesn’t help either). This is what I have tried so far:

  1. Using the “imp” module without playing with sys.modules:

import_name = “wx”

directory = r"C:\Python27\Lib\site-packages\wx-2.8.10-msw\wx"

f, filename, description = imp.find_module(import_name, [os.path.dirname(directory)])

mainmod = imp.load_module(import_name, f, filename, description)

==> On subsequent import of sub-packages/sub-modules, 2.9.2.4 stuff gets imported.

  1. Using the “imp” module playing with sys.modules:

import_name = “wx”

directory = r"C:\Python27\Lib\site-packages\wx-2.8.10-msw\wx"

if import_name in sys.modules:

sys.modules.pop(import_name)

f, filename, description = imp.find_module(import_name, [os.path.dirname(directory)])

mainmod = imp.load_module(import_name, f, filename, description)

==> Traceback (most recent call last):

File “C:\MyProjects\Phoenix\NDT\inputpanel.py”, line 166, in OnHandlePackage

mainmod = Import(initName, importName)

File “C:\MyProjects\Phoenix\NDT\modulehunter.py”, line 978, in Import

mainmod = imp.load_module(import_name, f, filename, description)

File “C:\Python27\Lib\site-packages\wx-2.8.10-msw\wx_init_.py”, line 54, in

docfilter = wx._core.__DocFilter(globals())

AttributeError: ‘module’ object has no attribute ‘_core’

  1. Using “execfile” with the init.py file as an input:

mainmod = types.ModuleType(repr(import_name))

execfile(init_name, mainmod.dict)

mainmod.file = init_name + “c”

==> On subsequent import of sub-packages/sub-modules, 2.9.2.4 stuff gets imported (not to mention that other libraries, like numpy, scipy and so on, bomb with any kind of assertion errors and Python hard crashes with this approach).

Now, I believe the only possible remaining solution is to look at a somewhat "recursive approach with the “imp” module as described here:

http://docs.python.org/library/imp.html#imp.find_module

Although it is a crappy and tedious thing to implement. I am obviously open to all suggestions (I guess my Python knowledge is just not good enough), as I would be unhappy to abandon NDT at its almost finished state because of me growing sick of this import things.

Thank you in advance.

Andrea.

“Imagination Is The Only Weapon In The War Against Reality.”
http://xoomer.alice.it/infinity77/

···

On 26 October 2011 23:14, Robin Dunn wrote:

On 10/26/11 12:55 PM, Andrea Gavana wrote:

On 26 October 2011 20:17, Robin Dunn wrote:
On 10/26/11 10:11 AM, Robin Dunn wrote:
On 10/26/11 1:23 AM, Andrea Gavana wrote:

Hi,

Hi All,

Now, I believe the only possible remaining solution is to look at a somewhat
"recursive approach with the "imp" module as described here:
http://docs.python.org/library/imp.html#imp.find_module
Although it is a crappy and tedious thing to implement. I am obviously open
to all suggestions (I guess my Python knowledge is just not good enough), as
I would be unhappy to abandon NDT at its almost finished state because of me
growing sick of this __import__ things.
Thank you in advance.
Andrea.

IMHO, the best way to do it would be to do all the module analysis in
a separate process using the correct version of Python for the package
/ user choice.

For example a user may be running this tool using Python 2.7 but wants
to check against libraries in their 3.x install. The separate process
will also ensure the correct module is being imported each time since
the sys.modules will be clean for that process and using other
possibly dependent libs from the same version of Python.

Would try to design it to have a separate script that can be run to do
the analysis and either pipe the output back to the main program
through stdio or write to a shared database or something. This would
provide the most flexibility though while adding a slight amount of
complexity.

Regards,

Cody

···

On Wed, Nov 2, 2011 at 10:43 AM, Andrea Gavana <andrea.gavana@gmail.com> wrote:

Yep, that is basically what I was going to suggest as well. The analysis itself doesn't need a GUI, just the part that the user will use to look at the data collected by the analysis step.

···

On 11/2/11 8:55 AM, Cody wrote:

Hi,

On Wed, Nov 2, 2011 at 10:43 AM, Andrea Gavana<andrea.gavana@gmail.com> wrote:

Hi All,

Now, I believe the only possible remaining solution is to look at a somewhat
"recursive approach with the "imp" module as described here:
http://docs.python.org/library/imp.html#imp.find_module
Although it is a crappy and tedious thing to implement. I am obviously open
to all suggestions (I guess my Python knowledge is just not good enough), as
I would be unhappy to abandon NDT at its almost finished state because of me
growing sick of this __import__ things.
Thank you in advance.
Andrea.

IMHO, the best way to do it would be to do all the module analysis in
a separate process using the correct version of Python for the package
/ user choice.

For example a user may be running this tool using Python 2.7 but wants
to check against libraries in their 3.x install. The separate process
will also ensure the correct module is being imported each time since
the sys.modules will be clean for that process and using other
possibly dependent libs from the same version of Python.

Would try to design it to have a separate script that can be run to do
the analysis and either pipe the output back to the main program
through stdio or write to a shared database or something. This would
provide the most flexibility though while adding a slight amount of
complexity.

--
Robin Dunn
Software Craftsman

One thing:

···

-------------------
Didn't see it mentioned yet but don't forget testing with id() for
object sameness.
__import__( ...mymodule...)
all_objects =
for item in mymodule.items():
    all_objects.append( item )

Two thing:
-----------------

The main issue goes back to how to import stuff and dynamically analyzing a package.
... No matter what the heck I do, Python keeps importing stuff from 2.9.2.4 because it is already in sys.modules (and playing with sys.modules doesn't help either).

When you run the python program it "remembers" a lot of stuff.
    Environment variables
        PATH, PYTHONPATH, PYTHONHOME and more esoterically
IRONPYTHONPATH, LIB, INCLUDE

    and 3rd party packages in the path may also have versions of
python installed like:
        Panda3D, MinGW, Cygwin

    The envar PYTHONSTARTUP might effect the caching of import
        as it could possibly preempt/postprocess your modifing/setting up
        the "other-python-version" environment by using a fully qualified
        python module pathname.

    There there's:
        sys.flags.ignore_environment
        sys.path_importer_cache !! might be causing you headaches too
        sys.flags.no_site
        sys.flags.no_user_site

See also the python(.exe) command line options -s and -S

It could really get crazy with python VMs, customized __import__
functions as has some apps do. On and on.

I say just go simple for release NDT 1.0.0 and assume the most basic
stuff, like one version of python installed, just for starters. Let
individuals with "multi-Python-versions" setups work out the details,
for now. Once the core of the tool is mature and well tested, then go
multiversion.

Hi,

Hi,

Hi All,

Now, I believe the only possible remaining solution is to look at a somewhat

"recursive approach with the “imp” module as described here:

http://docs.python.org/library/imp.html#imp.find_module

Although it is a crappy and tedious thing to implement. I am obviously open

to all suggestions (I guess my Python knowledge is just not good enough), as

I would be unhappy to abandon NDT at its almost finished state because of me

growing sick of this import things.

Thank you in advance.

Andrea.

IMHO, the best way to do it would be to do all the module analysis in

a separate process using the correct version of Python for the package

/ user choice.

For example a user may be running this tool using Python 2.7 but wants

to check against libraries in their 3.x install. The separate process

will also ensure the correct module is being imported each time since

the sys.modules will be clean for that process and using other

possibly dependent libs from the same version of Python.

Would try to design it to have a separate script that can be run to do

the analysis and either pipe the output back to the main program

through stdio or write to a shared database or something. This would

provide the most flexibility though while adding a slight amount of

complexity.

Thanks for the suggestion, and I’ll definitely go this way at some point. However, it appears not to solve the underlying problem on how to import stuff correctly. Let’s assume for the moment the following:

  1. I run on Windows Vista and 7

  2. I’m using Python 2.7;

  3. I have wxPython 2.8.10 and 2.9.2.4 in site-packages. The default version (specified in wx.pth) is 2.9.2.4;

  4. I want to analyze wxPython 2.8.10 (not the default one).

These are the conclusions I could draw up to now, and they are true regardless if you run the importing/analyzing stuff in a separate thread or separate process or on the main thread or anywhere else:

  1. I can not use the builtin import function, as from Python versions >= 2.6 importing by filename raises an ImportError (“ImportError: Import by filename is not supported.”). If I use import the recommended way, i.e.:

module = import(“wx.html”, globals(), locals(), fromlist[“html”])

This will import stuff from 2.9.2.4 (because it is the default wxPython version) and not from 2.8.10, regardless of what I do to sys.path and/or sys.modules;

  1. Using the pkgutil and/or imp libraries yields any kind of error (you can see them running the attached Python script, just modify the DIRECTORY and IMPORT_NAME variables). Some examples of these errors:
  • Running the Import_1() function:

C:\MyProjects>imp_1.py

Module Name Module Path

wx.version C:\Python27\Lib\site-packages\wx-2.8.10-msw\wx_version_.pyc

wx._activex C:\Python27\Lib\site-packages\wx-2.8.10-msw\wx_activex.pyd

wx.lib.ClickableHtmlWindow

Traceback (most recent call last):

File “C:\MyProjects\imp_1.py”, line 48, in Import_1

submodule = loader.load_module(module_name)

File “C:\Python27\lib\pkgutil.py”, line 238, in load_module

mod = imp.load_module(fullname, self.file, self.filename, self.etc)

File “C:\Python27\Lib\site-packages\wx-2.9.2-msw\wx\lib\ClickableHtmlWindow.py”, line 18, in

import  wx.html as  html

AttributeError: ‘module’ object has no attribute ‘html’

Yeah, right, now I am really convinced that wx has no “html” module…

  • Running the Import_2() function with “add_path=False” (i.e., no modifications of sys.path):

Module Name Module Path

wx.version C:\Python27\Lib\site-packages\wx-2.8.10-msw\wx_version_.pyc

wx._activex C:\Python27\Lib\site-packages\wx-2.8.10-msw\wx_activex.pyd

wx.lib.agw.aquabutton C:\Python27\Lib\site-packages\wx-2.8.10-msw\wx\lib\agw\aquabutton.pyc

wx.lib.agw.artmanager

Traceback (most recent call last):

File “C:\MyProjects\imp_1.py”, line 89, in Import_2

submodule = imp.load_module(submodule_name, f, filename, description)

File “C:\Python27\Lib\site-packages\wx-2.8.10-msw\wx\lib\agw\artmanager.py”, line 6, in

from fmresources import *

ImportError: No module named fmresources

And if I hadn’t written the fmresources module myself (it’s in AGW), I would actually have believed this. No chance.

  • Running the Import_2() function with “add_path=True” (i.e., adding all the wxPython folders/sub-folders to sys.path):

Module Name Module Path

wx.version C:\Python27\Lib\site-packages\wx-2.8.10-msw\wx_version_.pyc

wx._activex C:\Python27\Lib\site-packages\wx-2.8.10-msw\wx_activex.pyd

wx.lib.agw.aui.aui_constants C:\Python27\Lib\site-packages\wx-2.8.10-msw\wx\lib\agw\aui\aui_constants.pyc

wx.lib.agw.aui.aui_switcherdialog

Traceback (most recent call last):

File “C:\MyProjects\imp_1.py”, line 89, in Import_2

submodule = imp.load_module(submodule_name, f, filename, description)

File “C:\Python27\Lib\site-packages\wx-2.8.10-msw\wx\lib\agw\aui\aui_switcherdialog.py”, line 151, in

import auibook

File “C:\Python27\Lib\site-packages\wx-2.8.10-msw\wx\lib\agw\aui\auibook.py”, line 21, in

import framemanager

File “C:\Python27\Lib\site-packages\wx-2.8.10-msw\wx\lib\agw\aui\framemanager.py”, line 103, in

import tabmdi

File “C:\Python27\Lib\site-packages\wx-2.8.10-msw\wx\lib\agw\aui\tabmdi.py”, line 586, in

class AuiMDIClientWindow(auibook.AuiNotebook):

AttributeError: ‘module’ object has no attribute ‘AuiNotebook’

Uhm, really? I thought the main purpose of the “auibook” module was to actually have an AuiNotebook class in it…

Now, I am not sure what I am missing here but I would extremely welcome any correction to my horribly weak understanding of the importing mechanism of Python, and any correction to my crappy code to make it work as it should.

Thank you in advance.

Andrea.

“Imagination Is The Only Weapon In The War Against Reality.”
http://xoomer.alice.it/infinity77/

import PyQt4.QtGui

Traceback (most recent call last):

File “”, line 1, in

ImportError: No module named PyQt4.QtGui

import pygtk

Traceback (most recent call last):

File “”, line 1, in

ImportError: No module named pygtk

imp_1.py (3.29 KB)

···

On 2 November 2011 16:55, Cody wrote:

On Wed, Nov 2, 2011 at 10:43 AM, Andrea Gavana andrea.gavana@gmail.com wrote:

import wx