EDIT:
OK, I managed to isolate the bug and the exact, complete code to to reproduce it. But it appears either something that's by design, or a bug in python.
Create two sibling packages: admin & General, each with it's own __init__.py, of course.
In the package admin put the file 'test.py' with the following code:
from General.test02 import run
import RunStoppedException
try:
run()
except RunStoppedException.RunStoppedException,e:
print 'right'
except Exception,e:
print 'this is what i got: %s'%type(e)
and also in admin put the file 'RunStoppedException.py' with the following code:
class RunStoppedException(Exception):
def __init__(self):
Exception.__init__(self)
In the package General put the file test02.py with the code:
import admin.RunStoppedException
def run():
raise admin.RunStoppedException.RunStoppedException()
the printout:
this is what i got: <class 'admin.RunStoppedException.RunStoppedException'>
When it should've been right. This only happens when one file sits in the same dir as the exception, so they import it differently.
Is this by design, or a bug of python?
I am using python2.6, running it under eclipse+pydev
import admin.RunStoppedException
This is an ambiguous relative import. Do you mean RunStoppedException from the admin top-level module? Or from mypackage.admin when you're in a package? If your current working directory (which is added to the module search path) happens to be inside the package, it could be either, depending on whether Python knows it's inside a package, which depends on how you're running the script.
If you've got both import admin.RunStoppedException and import RunStoppedException in different modules, that could very well import two copies of the same module: a top-level RunStoppedException and a submodule admin.RunStoppedException of the package admin, resulting in two instances of the exception, and the subsequent mismatch in except.
So don't use implicit relative imports. They are in any case going away (see PEP328). Always spell out the full module name, eg. import mypackage.admin.RunStoppedException. However avoid using the same identifier for your module name and your class name as this is terribly confusing. Note that Python will allow you to say:
except RunStoppedException:
where that identifier is referring to a module and not a subclass of Exception. This is for historical reasons and may also go away, but for the meantime it can hide bugs. A common pattern would be to use mypackage.exceptions to hold many exceptions. One-class-per-file is a Java habit that is frowned on in Python.
It's also a good idea generally try to keep the importing of module contents (like classes) down as much as possible. If something changes the copy of RunStoppedException inside the module, you'll now have different copies in different scripts. Though classes mostly don't change, module-level variables may, and monkey-patching and reloading become much harder when you're taking stuff outside of its owner module.
I can only see two reasons
You have two different Exception classes with same name
Edit: I think culprit is this part because you import Exception class two ways
from RunStoppedException import RunStoppedException
from admin.RunStoppedException import RunStoppedException
make them consistent and your problem will be gone.
You are using some IDE, which is interfering with your code, this sounds bizarre but try to run your code on command line if you aren't
Even 1 and 2 doesn't fix your problem, write a small piece of code demonstrating the problem, which we can run here, which we can fix, but I am sure we will not need to because once you have written such a small standalone script where you can replicate the problem you will find the solution too.
Works fine for me:
[/tmp] ls admin/
RunStoppedException.py __init__.py test.py
RunStoppedException.pyc __init__.pyc
[/tmp] ls General/
__init__.py __init__.pyc test02.py test02.pyc
[/tmp] python -m admin.test
right
[/tmp]
Running on:
Python 2.6.4 Stackless 3.1b3 060516 (release26-maint, Dec 14 2009, 23:28:06)
[GCC 4.2.1 (Apple Inc. build 5646) (dot 1)] on darwin
My guess is that you have another "General" on your path somewhere, perhaps from earlier tests, and thats why the exceptions don't match.
Did you try the id/inspect.getabsfile debugging? If so, what was the output?
Related
I have a project containing multiple packages for example:
production/
__init__.py
prod_module_1.py
prod_subpackage/
__init__.py
prod_submodule_1.py
...
computation/
__init__.py
computation_module_1.py
...
development/
...
Now I want to prevent that specific packages can be imported by others. To go with the above example I want development modules to be able to import all types of packages but I don't want any module in production being able to import code from development. And in the same gist I want computation to be "standalone" so it can only import modules from itself, but it can be imported by other modules.
In the Python documentation https://docs.python.org/3/reference/import.html I found the following excerpt:
To selectively prevent import of some modules from a hook early on the meta path (rather than disabling the standard import system entirely), it is sufficient to raise ModuleNotFoundError directly from find_spec() instead of returning None. The latter indicates that the meta path search should continue, while raising an exception terminates it immediately.
But I'm unsure how/where I need to overwrite this method. An insight on where in the code base these changes need to be made would be greatly appreciated.
I recently was asked to deliver a python project as part of an interview process.
I tested my project on Windows and MacOSX, with Pycharm, Spyder, jupyter notebook and command line and everything works fine.
However, the reviewer was unable to make the project work on his side, because of module import issues according to him.
My modules are organized like this:
my_project/
my_module.py
main_module.py
my_package/
__init__.py
my_submodule_1.py
my_submodule_2.py
my_submodule_1.py:
import my_module
import my_submodule_2
I haven't added any path related to this project in PYTHONPATH.
The project main function is located in main_module.py.
The reviewer seem to have problems with the modules imported in my_submodule_1.py.
Could anyone shed some light on possible mistakes here and why it would work on my side and not on his?
Your my_submodule_1 module is doing an implicit relative import when it imports my_submodule_2 directly.
That's not legal in Python 3. It is allowed in Python 2, though its usually a bad idea to use it. You can get the Python 3 semantics by putting from __future__ import absolute_import above the other import statements in your file. To fix the import, you'd want to change import my_submodule_2 to either import my_package.my_submodule_2 (an absolute import) or from . import my_submodule2 (an explicit relative import).
If your interviewer is using Python 3 and you're using Python 2, there are likely to be other issues with your code (especially if you're doing any sort of text processing), so I'd make sure you're testing your code in the version they expect!
I think since my_module.py is not in same directory as my_submodule1.py ,and on the reviewer pc the sys.path doesn't have that location of my_module.py, that's why it getting problem in importing the module from its parent directory.
if u give the details of error that the reviewer is getting it might help finding the right solution.
I am using the following command to run tests:
nosetests --with-coverage --cover-html --cover-package mypackage
I would like the coverage report to be updated, even if a developer adds new, untested, code to the package.
For example, imagine a developer adds a new module to the package but forgets to write tests for it. Since the tests may not import the new module, the code coverage may not reflect the uncovered code. Obviously this is something which could be prevented at the code review stage but it would be great to catch it even earlier.
My solution was to write a simple test which dynamically imports all modules under the top-level package. I used the following code snippet to do this:
import os
import pkgutil
for loader, name, is_pkg in pkgutil.walk_packages([pkg_dirname]):
mod = loader.find_module(name).load_module(name)
Dynamically importing sub-packages and sub-modules like this does not get picked up by the code coverage plugin in nose.
Can anyone suggest a better way to achieve this type of thing?
The problem seems to be the method for dynamically importing all packages/modules under the top-level package.
Using the method defined here seems to work. The key difference being the use of importlib instead of pkgutil. However, importlib was introduced in python 2.7 and 3.1 so this solution is not appropriate for older versions of python.
I have updated the original code snippet to use __import__ instead of the ImpLoader.load_module method. This also seems to do the trick.
import os
import pkgutil
for loader, name, is_pkg in pkgutil.walk_packages([pkg_dirname]):
mod = loader.find_module(name)
__import__(mod.fullname)
I'm self-taught in the Python world, so some of the structural conventions are still a little hazy to me. However, I've been getting very close to what I want to accomplish, but just ran into a larger problem.
Basically, I have a directory structure like this, which will sit outside of the normal python installation (this is to be distributed to people who should not have to know what a python installation is, but will have the one that comes standard with ArcGIS):
top_directory/
ArcToolbox.tbx
scripts/
ArcGIStool.py (script for the tool in the .tbx)
pythonmod/
__init__.py
general.py
xlrd/ (copied from my own python installation)
xlwt/ (copied from my own python installation)
xlutils/ (copied from my own python installation)
So, I like this directory structure, because all of the ArcGIStool.py scripts call functions within the pythonmod package (like those within general.py), and all of the general.py functions can call xlrd and xlwt functions with simple "import xlrd" statements. This means that if the user desired, he/she could just move the pythonmod folder to the python site-packages folder, and everything would run fine, even if xlrd/xlwt/xlutils are already installed.
THE PROBLEM:
Everything is great, until I try to use xlutils in general.py. Specifically, I need to "from xlutils.copy import copy". However, this sets off a cascade of import errors. One is that xlutils/copy.py uses "from xlutils.filter import process,XLRDReader,XLWTWriter". I solved this by modifying xlutils/copy.py like this:
try:
from xlutils.filter import process,XLRDReader,XLWTWriter
except ImportError:
from filter import process,XLRDReader,XLWTWriter
I thought this would work fine for other situations, but there are modules in the xlutils package that need to import xlrd. I tried following this advice, but when I use
try:
import xlrd
except ImportError:
import os, sys, imp
path = os.path.dirname(os.path.dirname(sys.argv[0]))
xlrd = imp.load_source("pythonmod.xlrd",os.path.join(path,"xlrd","__init__.py"))
I get a new import error: In xlrd/init.py, the info module is called (from xlrd/info.py), BUT when I use the above code, I get an error saying that the name "info" is not defined.
This leads me to believe that I don't really know what is going on, because I thought that when the init.py file was imported it would run just like normal and look within its containing folder for info.py. This does not seem to be the case, unfortunately.
Thanks for your interest, and any help would be greatly appreciated.
p.s. I don't want to have to modify the path variables, as I have no idea who will be using this toolset, and permissions are likely to be an issue, etc.
I realized I was using imp.load_source incorrectly. The correct syntax for what I wanted to do should have been:
imp.load_source("xlrd",os.path.join(path,"xlrd","__init__.py"))
In the end though, I ended up rewriting my code to not need xlutils at all, because I continued to have import errors that were causing many more problems than were worth dealing with.
Okay, so in the past, I've made my own Python packages with Python 2.x (most recently, 2.7.5). It has worked fine. Let me explain how I did that, for reference:
Make a directory within the working directory. We'll call it myPackage.
Make a file called __init__.py in the directory myPackage.
Make sure all the modules that you want to be part of the package are imported within __init__.py. These modules are typically in the myPackage folder.
From a Python program in the working directory, type import myPackage (and it imports fine, and is usable).
However, in Python 3, I get errors with that. (ImportError: No module named 'Whatever the first imported module is')
I researched the problem and found the following:
Starred imports don't work in Python 3.3.
The __init__.py file is not required in Python 3.3.
So, I removed the stars from my imports, and leaving the __init__.py file in, I still got errors (ImportError: No module named 'Whatever the first imported module is'). So, I removed the __init__.py file, and I don't get any errors, but my package doesn't include any of my modules.
Okay, so I discovered by doing a web search for python3 __init__.py or some such that I can do the following, although I don't have any clue if this is the standard way of doing things:
In the modules in the package, make sure there are no plain imports (not just no starred ones). Only do from myModule import stuff. However, you need to put a . in front of myModule: e.g. from .myModule import stuff. Then, I can import myPackage.oneOfMyModules
I found that by following this rule in the __init__.py file, it also works.
Once again, I don't know if this is how it's supposed to work, but it seems to work.
I found this page that is supposed to have something to do with the changes in packages in Python 3.something, but I'm not sure how it relates to what I'm doing:
http://legacy.python.org/dev/peps/pep-0420/
So, what is the standard way to do this? Where is it documented (actually saying the syntax)? Is the way I'm doing it right? Can I do regular imports instead of from package import module?
After analyzing some Python 3 packages installed on my system (I should have tried that to start with!) I discovered that they often seem to do things a little differently. Instead of just doing from .myModule import stuff they would do from myPackage.myModule import stuff (inside the modules in the package). So, that works, too, I suppose, and seems to be more frequently used.