I am having trouble creating an importable Python package/library/module or whatever the right nomenclature is. I am using Python 3.7
The file structure I am using is:
Python37//Lib//mypackage
mypackage
__init__.py
mypackage_.py
The code in __init__.py is:
from mypackage.mypackage_ import MyClass
The code in mypackage_.py is:
class MyClass:
def __init__(self, myarg = None):
self.myvar = myarg
And from my desktop I try running the following code:
import mypackage
x = MyClass(None)
But get the following error:
Traceback (most recent call last):
File "C:\Users\***\Desktop\importtest.py", line 3, in <module>
x = MyClass(None)
NameError: name 'MyClass' is not defined
You haven't imported the name MyClass into your current namespace. You've imported mypackage. To access anything within mypackage, you need to prefix the name with mypackage.<Name>
import mypackage
x = mypackage.MyClass(None)
As #rdas says, you need to prefix the name with mypackage.<Name>.
I don't recommend doing this, but you can wildcard import in order to make x = MyClass(None) work:
from mypackage import *
Now, everything from mypackage is imported and usable in the current namespace. However, you have to be careful with wildcard imports because they can create definition conflictions (if multiple modules have the same name for different things).
Related
I'm trying to make a library out of a Python project I don't own.
The project has the following directory layout:
.
├── MANIFEST.in
├── pyproject.toml
└── src
├── all.py
├── the.py
└── sources.py
In pyproject.toml I have:
[tool.setuptools]
packages = ["mypkg"]
[tool.setuptools.package-dir]
mypkg = "src"
The problem I'm facing is that when I build and install this package I can't use it because the author is importing stuff without mypkg prefix in the various source files.
F.ex. in all.py
from the import SomeThing
Since I don't own the package I can't go modify all the sources but I still want to be able to build a library from it by just adding MANIFEST.in and pyproject.toml.
Is it possible to somehow instruct setuptools to build a package that won't litter site-packages with all the sources while still allowing them to be imported without the mypkg prefix?
It isn't possible without adding a custom import hook with the package. The hook takes the form of a module that is shipped with the package, and it must be imported before usage from your module (e.g. in src/all.py)
src/mypkgimp.py
import sys
import importlib
class MyPkgLoader(importlib.abc.Loader):
def find_spec(self, name, path=None, target=None):
# update the list with modules that should be treated special
if name in ['sources', 'the']:
return importlib.util.spec_from_loader(name, self)
return None
def create_module(self, spec):
# Uncomment if "normal" imports should have precedence
# try:
# sys.meta_path = [x for x in sys.meta_path[:] if x is not self]
# return importlib.import_module(spec.name)
# except ImportError:
# pass
# finally:
# sys.meta_path = [self] + sys.meta_path
# Otherwise, this will unconditionally shadow normal imports
module = importlib.import_module('.' + spec.name, 'mypkg')
# Final step: inject the module to the "shortened" name
sys.modules[spec.name] = module
return module
def exec_module(self, module):
pass
if not hasattr(sys, 'frozen'):
sys.meta_path = [MyPkgLoader()] + sys.meta_path
Yes, the above uses different methods described by the thread I have linked previously, as importlib have deprecated those methods in Python 3.10, refer to documentation for details.
Anyway, for the demo, put some dummy classes in the modules:
src/the.py
class SomeThing: ...
src/sources.py
class Source: ...
Now, modify src/all.py to have the following:
import mypkg.mypkgimp
from the import SomeThing
Example usage:
>>> from sources import Source
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'sources'
>>> from mypkg import all
>>> all.SomeThing
<class 'mypkg.the.SomeThing'>
>>> from sources import Source
>>> Source
<class 'mypkg.sources.Source'>
>>> from sources import Error
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: cannot import name 'Error' from 'mypkg.sources' (/tmp/mypkg/src/sources.py)
Note how the import initially didn't work, but after mypkg.all got imported, the sources import now works globally. Hence care may be needed to not shadow "real" imports and I have provided the example to import using the "default"[*] import mechanism.
If you want the module names to look different (i.e. without the mypkg. prefix), that will be a separate question, as code typically don't check for their own module name for functionality (and never mind that this actually shows how the namespace is implicitly used - changing the actual name is more akin to a module relocation, yes this can be done, but a bit more complicated and this answer is long enough as it is).
[*] "default" as in not including behaviors introduced by this custom import hook - other import hooks may do their own other weird shenanigans.
I have a folder structure similar to this (my example has all the necessary bits):
web-scraper/
scraper.py
modules/
__init__.py
config.py
website_one_scraper.py
Where config.py just stores some global variables. It looks a bit like:
global var1
var1 = "This is a test!"
Within website_one_scraper.py it looks like this:
import config
def test_function():
# Do some web stuff...
return = len(config.var1)
if __name__ == "__main__":
print(test_function)
And scraper.py looks like this:
from module import website_one_scraper
print(website_one_scraper.test_function())
website_scraper_one.py works fine when run by itself, and thus the code under if __name__ == "__main__" is run. However, when I run scraper.py, I get the error:
ModuleNotFoundError: No module named 'config'
And this is the full error and traceback (albeit with different names, as I've changed some names for the example above):
Traceback (most recent call last):
File "c:\Users\User\Documents\Programming\Work\intrack-web-scraper\satellite_scraper.py", line 3, in
<module>
from modules import planet4589
File "c:\Users\User\Documents\Programming\Work\intrack-web-scraper\modules\planet4589.py", line 5, in
<module>
import config
ModuleNotFoundError: No module named 'config'
Also note that In scraper.py I've tried replacing from modules import website_one_scraper with import website_one_scraper, from .modules import website_one_scraper, and from . import website_one_scraper, but they all don't work.
What could the cause of my error be? Could it be something to do with how I'm importing everything?
(I'm using Python 3.9.1)
In your website_scraper_one.py, instead of import config.py try to use from . import config
Explanation:
. is the current package or the current folder
config is the module to import
I have a following directory structure:
source
source_1.py
__init__.py
source1.py has class Source defined
source1.py
class Source(object):
pass
I am able to import using this
>>> from source.source1 import Source
>>> Source
<class 'source.source1.Source'>
However when trying to import using the below method it fails.
>>> from source import *
>>> source1.Source
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'source1' is not defined
Please let me know how can we use the 2nd import ?
For importing from a package (unlike importing from a module) you need to specify what * means. To do that, in __init__.py add a line like this:
__all__ = ["source1"]
See the Python documentation for Importing * From a Package.
I have a module called spellnum. It can be used as a command-line utility (it has the if __name__ == '__main__': block) or it can be imported like a standard Python module.
The module defines a class named Speller which looks like this:
class Speller(object):
def __init__(self, lang="en"):
module = __import__("spelling_" + lang)
# use module's contents...
As you can see, the class constructor loads other modules at runtime. Those modules (spelling_en.py, spelling_es.py, etc.) are located in the same directory as the spellnum.py itself.
Besides spellnum.py, there are other files with utility functions and classes. I'd like to hide those files since I don't want to expose them to the user and since it's a bad idea to pollute the Python's lib directory with random files. The only way to achieve this that I know of is to create a package.
I've come up with this layout for the project (inspired by this great tutorial):
spellnum/ # project root
spellnum/ # package root
__init__.py
spellnum.py
spelling_en.py
spelling_es.py
squash.py
# ... some other private files
test/
test_spellnum.py
example.py
The file __init__.py contains a single line:
from spellnum import Speller
Given this new layout, the code for dynamic module loading had to be changed:
class Speller(object):
def __init__(self, lang="en"):
spelling_mod = "spelling_" + lang
package = __import__("spellnum", fromlist=[spelling_mod])
module = getattr(package, spelling_mod)
# use module as usual
So, with this project layout a can do the following:
Successfully import spellnum inside example.py and use it like a simple module:
# an excerpt from the example.py file
import spellnum
speller = spellnum.Speller(es)
# ...
import spellnum in the tests and run those tests from the project root like this:
$ PYTHONPATH="`pwd`:$PYTHONPATH" python test/test_spellnum.py
The problem
I cannot execute spellnum.py directly with the new layout. When I try to, it shows the following error:
Traceback (most recent call last):
...
File "spellnum/spellnum.py", line 23, in __init__
module = getattr(package, spelling_mod)
AttributeError: 'module' object has no attribute 'spelling_en'
The question
What's the best way to organize all of the files required by my module to work so that users are able to use the module both from command line and from their Python code?
Thanks!
How about keeping spellnum.py?
spellnum.py
spelling/
__init__.py
en.py
es.py
Your problem is, that the package is called the same as the python-file you want to execute, thus importing
from spellnum import spellnum_en
will try to import from the file instead of the package. You could fiddle around with relative imports, but I don't know how to make them work with __import__, so I'd suggest the following:
def __init__(self, lang="en"):
mod = "spellnum_" + lang
module = None
if __name__ == '__main__':
module = __import__(mod)
else:
package = getattr(__import__("spellnum", fromlist=[mod]), mod)
Here's the structure I'm working with:
directory/
script.py
subdir/
__init__.py
myclass01.py
myclass02.py
What I want to do is import in script.py the classes defined in myclass01.py and myclass02.py. If I do:
from subdir.myclass01 import *
It works fine for the class defined in myclass01.py. But with this solution if there are many classes defined in different files in subdir and I want to import all of them, I'd have to type one line for each file. There must be a shortcut for this. I tried:
from subdir.* import *
But it didn't work out.
EDIT: here are the contents of the files:
This is __init__.py (using __all__ as Apalala suggested):
__all__ = ['MyClass01','MyClass02']
This is myclass01.py:
class MyClass01:
def printsomething():
print 'hey'
This is myclass02.py:
class MyClass02:
def printsomething():
print 'sup'
This is script.py:
from subdir import *
MyClass01().printsomething()
MyClass02().printsomething()
This is the traceback that I get when I try to run script.py:
File "script.py", line 1, in <module>
from subdir import *
AttributeError: 'module' object has no attribute 'MyClass01'
Although the names used there are different from what's shown in your question's directory structure, you could use my answer to the question titled Namespacing and classes. The __init__.py shown there would have also allowed the usepackage.py script to have been written this way (package maps to subdir in your question, and Class1 to myclass01, etc):
from package import *
print Class1
print Class2
print Class3
Revision (updated):
Oops, sorry, the code in my other answer doesn't quite do what you want — it only automatically imports the names of any package submodules. To make it also import the named attributes from each submodule requires a few more lines of code. Here's a modified version of the package's __init__.py file (which also works in Python 3.4.1):
def _import_package_files():
""" Dynamically import all the public attributes of the python modules in this
file's directory (the package directory) and return a list of their names.
"""
import os
exports = []
globals_, locals_ = globals(), locals()
package_path = os.path.dirname(__file__)
package_name = os.path.basename(package_path)
for filename in os.listdir(package_path):
modulename, ext = os.path.splitext(filename)
if modulename[0] != '_' and ext in ('.py', '.pyw'):
subpackage = '{}.{}'.format(package_name, modulename) # pkg relative
module = __import__(subpackage, globals_, locals_, [modulename])
modict = module.__dict__
names = (modict['__all__'] if '__all__' in modict else
[name for name in modict if name[0] != '_']) # all public
exports.extend(names)
globals_.update((name, modict[name]) for name in names)
return exports
if __name__ != '__main__':
__all__ = ['__all__'] + _import_package_files() # '__all__' in __all__
Alternatively you can put the above into a separate .py module file of its own in the package directory—such as _import_package_files.py—and use it from the package's __init__.py like this:
if __name__ != '__main__':
from ._import_package_files import * # defines __all__
__all__.remove('__all__') # prevent export (optional)
Whatever you name the file, it should be something that starts with an _ underscore character so it doesn't try to import itself recursively.
Your best option, though probably not the best style, is to import everything into the package's namespace:
# this is subdir/__init__.py
from myclass01 import *
from myclass02 import *
from myclass03 import *
Then, in other modules, you can import what you want directly from the package:
from subdir import Class1
I know it's been a couple months since this question was answered, but I was looking for the same thing and ran across this page. I wasn't very satisfied with the chosen answer, so I ended up writing my own solution and thought I'd share it. Here's what I came up with:
# NOTE: The function name starts with an underscore so it doesn't get deleted by iself
def _load_modules(attr_filter=None):
import os
curdir = os.path.dirname(__file__)
imports = [os.path.splitext(fname)[0] for fname in os.listdir(curdir) if fname.endswith(".py")]
pubattrs = {}
for mod_name in imports:
mod = __import__(mod_name, globals(), locals(), ['*'], -1)
for attr in mod.__dict__:
if not attr.startswith('_') and (not attr_filter or attr_filter(mod_name, attr)):
pubattrs[attr] = getattr(mod, attr)
# Restore the global namespace to it's initial state
for var in globals().copy():
if not var.startswith('_'):
del globals()[var]
# Update the global namespace with the specific items we want
globals().update(pubattrs)
# EXAMPLE: Only load classes that end with "Resource"
_load_modules(attr_filter=lambda mod, attr: True if attr.endswith("Resource") else False)
del _load_modules # Keep the namespace clean
This simply imports * from all .py files in the package directory and then only pulls the public ones into the global namespace. Additionally, it allows a filter if only certain public attributes are desired.
I use this simple way:
add the directory to the system path, then
import module or from module import function1, class1 in that directory.
notice that the module is nothing but the name of your *.py file, without the extension part.
Here is a general example:
import sys
sys.path.append("/path/to/folder/")
import module # in that folder
In your case it could be something like this:
import sys
sys.path.append("subdir/")
import myclass01
# or
from myclass01 import func1, class1, class2 # .. etc
from subdir.* import *
You can not use '*' this way directly after the 'from' statement.
You need explict imports. Please check with the Python documentation on imports and packages.