I'm developing a package "jw.data" in Python 2.7.9, with a namespace of jw and a package data in in. I have put the canonical
try:
__import__('pkg_resources').declare_namespace(__name__)
except ImportError:
from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)
in jw/__init__.py. When I run ./setup.py develop (in setup.py I have put namespace_packages=['jw']), then I have "jw" in jw.data.egg-info/namespace_packages.txt. Doing a
import jw.data
import jw.data.model
just works fine. So I guess I have set up the namespace package correctly.
Now I have written a package "jw.util", also in namespace jw, with a package util in it. As soon as I install it, importing jw.data or anything below it fails:
>>> import jw.data
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named data
After uninstalling jw.util it works again.
I used jw.util elsewhere, but not yet in a package within the jw namespace. It looks like jw.util is reserving the jw namespace.
I had the same problem with another namespace. There I noticed the package name in setup.py is just the same as the namespace plus package. I renamed all the packages from "namespace.packagex" to "namespace-packagex", and astonishingly enough it worked. I tried the same with jw.util and jw.data, but here that trick doesn't work. And I don't really believe the package name in setup.py has anything to do with the package hierarchy it contains, or has it?
Anyway, anybody got an idea what's going on here?
Seems to be a long-known bug in Python.
But there's a solution in https://github.com/pypa/setuptools/issues/250:
Put simply, append
;import pkg_resources; pkg_resources.fixup_namespace_packages('')
to the single line in module-nspkg.pth of the competing package in site-packages. The semicolon is required.
Related
I'm trying to make a simple import and use the emailage third party library.
As per their documentation, the way to use their library is as follows:
pip install emailage-official
Then, simply import with:
from emailage.client import EmailageClient
The install works fine with pip - no errors. I double checked to see that the emailage package exists within the proper directory, and it does.
Package exists at:
C:\Users\aaron\AppData\Local\Programs\Python\Python37-32\Lib\site-packages\emailage
This folder has (seemingly) the correct files with an __init__.py and everything. However, both pylint and command line interpreter throw me a
'No module named 'emailage.client'; 'emailage' is not a package' error.
The output of my sys.path is:
[...
'C:\\Users\\aaron\\AppData\\Local\\Programs\\Python\\Python37-32\\lib\\site-packages'
...
]
So the directory where emailage is installed is a part of the path... and lastly I pip-installed numpy just to test if it worked properly. Numpy installed to the same site-packages folder as emailage, and it works fine when it is imported, so I'm stuck.
I don't typically use Python much, so any and all help would be appreciated.
The issue was in the naming of my file.
I hastily named my file emailage.py and then tried to import from emailage.client.
I'm assuming that Python looked in my current directory and matched the names of the file I was working on before checking the installed third party libraries.
After renaming my file everything seems ok.
For others who run into similar problems -- beware of conflicting naming. Sometimes the simplest things trip you up the longest.
I ran into something similar and the answer from OP about namespace collision is what finally clued me in.
I was using the same name for both a sub-package (directory) and a module (file) within it.
For example I had this:
/opt/mylib/myapi
/opt/mylib/myapi/__init__.py
/opt/mylib/myapi/myapi_creds.py # gitignored file for user/pass
/opt/mylib/myapi/myapi.py # base module, load creds and connect
/opt/mylib/myapi/myapi_dostuff.py # call myapi.py and do work
The script 'myapi.py' imports credentials from myapi_creds.py via this statement:
from myapi.myapi_creds import my_user, my_pass
Testing the module 'myapi.py' resulted in this error:
$ ./myapi.py
Traceback (most recent call last):
File "./myapi.py", line 12, in <module>
from myapi.myapi_creds import my_user, my_pass
File "/opt/mylib/myapi/myapi.py", line 12, in <module>
from myapi.myapi_creds import my_user, my_pass
ModuleNotFoundError: No module named 'myapi.myapi_creds'; 'myapi' is not a package
The solution was to rename myapi.py to myapi_base.py so it's name does not collide with the sub-package name.
I took a look at this problem, and even though it is not exactly the same error that I encountered, it helped me solve it. I'll explain the situation I had, since I think some users might find this handy.
So, I was getting the following error log:
Traceback (most recent call last):
File "/home/kemal/Programming/Python/Preference_Articulation/LocalSearch/LS_apriori.py", line 1, in <module>
from LocalSearch.LocalSearch import LocalSearch
ModuleNotFoundError: No module named 'LocalSearch.LocalSearch'; 'LocalSearch' is not a package
The structure of my project is the following (using PyCharm):
View of project structure
The important thing to notice is that I separated my code into several folders, since it makes it more readable. Now, in the folder named LocalSearch I have 4 files, LocalSearch, LS_apriori and some 2 tests files (not relevant). When trying to run the file LS_apriori (which uses methods and classes from file LocalSearch) I was getting the error provided above. The code specifically is not important, and the way I handled the imports was the following:
from LocalSearch.LocalSearch import LocalSearch
The fix was simple. I renamed the py-file LocalSearch to Local_Search (just added an underscore). Afterwards, the error was gone.
So my problem was possessing a folder(package) with the same name as a file(module) inside it, which has a class inside it with the same name. Python didn't like that.
Having modules with the same name as packages inside them is fine however, I guess the class just added extra confusion.
I've never seen an import issue like this before. I removed a directory from site-packages and the corresponding package is still importable.
python2
> import google
> print(google.__path__)
['/home/bamboo/.local/lib/python2.7/site-packages/google']
However this directory doesn't actually exist
ls: cannot access /home/bamboo/.local/lib/python2.7/site-packages/google: No such file or directory
I've removed everything that I'm aware of that is related to it, but there must still be something hanging around.
Digging another level deeper I tried to reload google.
python2
> import google;
> reload(google);
ImportError: No module named google
So apparently it recognizes it is gone on reload.
Checking out sys.modules you get
python2
> import sys
> print(sys.modules)
{'google': <module 'google' (built-in)>, 'copy_reg': <module 'copy_reg' from '/usr/lib/python2.7/copy_reg.pyc'> ...
which indicates that apparently google is a built-in.
Note on motivation: Usually this sort of issue would be weird, but not a show stopper. The problem for me is that the google package is masking a different package of the same name.
tl,dr: use pip to uninstall Google packages completely.
There are two issues here:
strange import/reload behaviour of the google package
removal of the google package
import/reload behavior
I can reproduce the import/reload behaviour by installing the (Google) protobuf package (many Google packages will behave in the same way).
$ mktmpenv -p $(which python2)
...
$ python --version
Python 2.7.13
$ pip install protobuf
...
Installing collected packages: six, protobuf
Successfully installed protobuf-3.5.1 six-1.11.0
>>> import google
>>> print google.__path__
['~/virtual-envs/tmp-66cd9b4d01a8dec6/lib/python2.7/site-packages/google']
>>> import sys
>>> print sys.modules['google']
<module 'google' (built-in)>
>>> reload(google)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named google
I suspect what's going on here is that Google prefer to have all Google packages installed under a single google package, but this package is not designed to be importable, hence the unexpected reload behaviour. However importing subpackages by name works as expected:
>>> import protobuf
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named protobuf
>>> from google import protobuf
>>> protobuf.__path__
['~/virtual-envs/tmp-66cd9b4d01a8dec6/lib/python2.7/site-packages/google/protobuf']
>>> reload(protobuf)
<module 'google.protobuf' from '~/virtual-envs/tmp-66cd9b4d01a8dec6/lib/python2.7/site-packages/google/protobuf/__init__.pyc'>
>>>
Removal of the google package
The question states:
I removed a directory from site-packages and the corresponding package is still importable.
This can also be reproduced:
($ rm -rf ~/virtual-envs/tmp-66cd9b4d01a8dec6/lib/python2.7/site-packages/google
$ python
>>> import google
>>> print google.__path__
['~/virtual-envs/tmp-66cd9b4d01a8dec6/lib/python2.7/site-packages/google']
>>>
The problem here is that simply removing the google directory and its contents is not enough to completely uninstall whatever Google packages are present.
The site-packages directory still contains the file protobuf-3.5.1-py2.7-nspkg.pth, which contains this code (split into separate lines for readability, the original is a single line of semi-colon separated statements):
import sys, types, os
has_mfs = sys.version_info > (3, 5)
p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('google',))
importlib = has_mfs and __import__('importlib.util')
has_mfs and __import__('importlib.machinery')
m = has_mfs and sys.modules.setdefault('google', importlib.util.module_from_spec(importlib.machinery.PathFinder.find_spec('google', [os.path.dirname(p)])))
m = m or sys.modules.setdefault('google', types.ModuleType('google'))
mp = (m or []) and m.__dict__.setdefault('__path__',[])
(p not in mp) and mp.append(p)
The line
m = m or sys.modules.setdefault('google', types.ModuleType('google'))
is creating the google module in sys.modules if it doesn't already exist - this is why the google module is importable even after the directory has been deleted.
The correct way to remove the google module is by uninstalling google packages using pip:
pip uninstall protobuf
If pip isn't available in the build environment the it's a case of identifying any related files and folders (*dist-info/, *.pth) in site-packages and removing them manually.
I'm not sure if the title of this question is correct, mods please feel free to change it.
I'm working through cs231n Convolutional Neural Networks for Visual Recognition course online and I've hit a weird error. This is my first attempt at any real Python programming so it could be a simple error, or something more complicated.
I'm using Python 2.7, Anaconda, and Windows 7.
When trying to run a script I get the following error;
NameError: global name 'im2col_cython' is not defined
My understanding so far is that this occurs because this fails (and calls the exception);
try:
from cs231n.im2col_cython import col2im_cython, im2col_cython
from cs231n.im2col_cython import col2im_6d_cython
except ImportError:
print 'run the following from the cs231n directory and try again:'
print 'python setup.py build_ext --inplace'
print 'You may also need to restart your iPython kernel'
I've tried to figure out why this may be the case. First off I have to run setup.py to turn im2col_cython.pyx into other files. This seems to run but does at one point have the warning;
warning: extension name 'im2col_cython' does not match fully qualified name 'cs231n.im2col_cython' of 'im2col_cython.pyx'
My figuring here is that it's an issue to do with the fact that the folder tree I have looks like this;
Assignment 2
->cs231n
Inside cs231n is where the setup.py and im2col_cython.pyx files are located. I've installed cython, and I have vc for python2.7 installed. When I run setup.py is creates 2 new files;
im2col_cython.c
im2col_cython.pyd
But I have no idea if that's what it's meant to do or anything. I'm in way over my head with this (0 Python knowledge), but I'm keen to learn!
EDIT 1
The script that imports im2col_cython is here
The im2col_cython file is here
EDIT 2
When including the following in the exception clause;
import traceback; traceback.print_exc()
I get this output;
Traceback (most recent call last):
File "fast_layers.py", line 3, in <module>
from cs231n.im2col_cython import col2im_cython, im2col_cython
ImportError: No module named cs231n.im2col_cython
Add this in the file fast_layer.py before import cs231n.im2col_cython:
import pyximport
pyximport.install()
The problem you mentioned should be fixed.
Ive been writing a little script to bootstrap an environment for me, but ran into some confusion when attempting to handle module import errors. My intention was to catch any import error for the yaml module, and then use apt to install the module, and re-import it...
def install_yaml():
print "Attempting to install python-yaml"
print "=============== Begining of Apt Output ==============="
if subprocess.call(["apt-get", "-y", "install", "python-yaml"]) != 0 :
print "Failure whilst installing python-yaml"
sys.exit(1)
print "================= End of Apt Output =================="
#if all has gone to plan attempt to import yaml
import yaml
reload(yaml)
try:
import yaml
except ImportError:
print "Failure whilst importing yaml"
install_yaml()
grains_config = {}
grains_config['bootstrap version'] = __version__
grains_config['bootstrap time'] = "{0}".format(datetime.datetime.now())
with open("/tmp/doc.yaml", 'w+') as grains_file:
yaml.dump(grains_config, grains_file, default_flow_style=False)
Unfortunately when run I get a NameError
Traceback (most recent call last):
File "importtest-fail.py", line 32, in <module>
yaml.dump(grains_config, grains_file, default_flow_style=False)
NameError: name 'yaml' is not defined
After some research I discovered the reload builtin (Reload a previously imported module), which sounded like what I wanted to do, but still results in a NameError on the yaml modules first use.
Does anyone have any suggestions that would allow me to handle the import exception, install the dependencies and "re-import" it?
I could obviously wrap the python script in some bash to do the initial dependency install, but its not a very clean solution.
Thanks
You imported yaml as a local in install_yaml(). You'd have to mark it as a global instead:
global yaml
inside the function, or better still, move the import out of the function and put it right after calling install_yaml().
Personally, I'd never auto-install a dependency this way. Just fail and leave it to the administrator to install the dependency properly. They could be using other means (such as a virtualenv) to manage packages, for example.
Using windows 7 and python 2.7. I have a package named Regetron in c:\Python27\Lib\site-packages\regetron which contains __init__.py and engine.py. When I try to run this library from the command prompt by typing regetron I get the following error:
Traceback (most recent call last):
File "C:\Python27\Scripts\regetron.py", line 6, in <module>
from regetron.engine import Regetron
File "C:\Python27\Scripts\regetron.py", line 6, in <module>
from regetron.engine import Regetron
ImportError: No module named engine
I added c:\Python27\Lib\site-packages\regetron to %PYTHONPATH% and can successfully import this module from other scripts located in other folders as well as the interactive prompt, but for some reason it refuses to run from the command prompt. What is going on?
You actually have two problems here. Fixing either one of them would actually eliminate your immediate error, but you need to fix both of them.
When I try to run this library from the command prompt by typing regetron
You shouldn't have a script named regetron and also have a module or package named regetron. Fix it by renaming your script. But if you want to understand why it's causing a problem:
The current working directory is always part of sys.path. So, you're in the directory with regetron.py in, and you run it with regetron. That means that regetron.py is on the path. So when you import regetron, it finds your script, not the package. Or, when you from regetron import engine, it finds your script, and tries to import a variable/function/class/whatever named engine from it, rather than finding the package and trying to import a module underneath it.
I added c:\Python27\Lib\site-packages\regetron to %PYTHONPATH%
Never add a package's directory to sys.path.
Since site-packages is already on your sys.path, the code in regetron/engine.py is already available as regetron.engine. You don't want it to also be available as engine. This will cause all kinds of problems.
So, rename you script to something else, remove regetron from %PYTHONPATH%, and everything will be fine.
But you may want to (re-)read the section on Packages in the tutorial.