I have a module called spellnum. It can be used as a command-line utility (it has the if __name__ == '__main__': block) or it can be imported like a standard Python module.
The module defines a class named Speller which looks like this:
class Speller(object):
def __init__(self, lang="en"):
module = __import__("spelling_" + lang)
# use module's contents...
As you can see, the class constructor loads other modules at runtime. Those modules (spelling_en.py, spelling_es.py, etc.) are located in the same directory as the spellnum.py itself.
Besides spellnum.py, there are other files with utility functions and classes. I'd like to hide those files since I don't want to expose them to the user and since it's a bad idea to pollute the Python's lib directory with random files. The only way to achieve this that I know of is to create a package.
I've come up with this layout for the project (inspired by this great tutorial):
spellnum/ # project root
spellnum/ # package root
__init__.py
spellnum.py
spelling_en.py
spelling_es.py
squash.py
# ... some other private files
test/
test_spellnum.py
example.py
The file __init__.py contains a single line:
from spellnum import Speller
Given this new layout, the code for dynamic module loading had to be changed:
class Speller(object):
def __init__(self, lang="en"):
spelling_mod = "spelling_" + lang
package = __import__("spellnum", fromlist=[spelling_mod])
module = getattr(package, spelling_mod)
# use module as usual
So, with this project layout a can do the following:
Successfully import spellnum inside example.py and use it like a simple module:
# an excerpt from the example.py file
import spellnum
speller = spellnum.Speller(es)
# ...
import spellnum in the tests and run those tests from the project root like this:
$ PYTHONPATH="`pwd`:$PYTHONPATH" python test/test_spellnum.py
The problem
I cannot execute spellnum.py directly with the new layout. When I try to, it shows the following error:
Traceback (most recent call last):
...
File "spellnum/spellnum.py", line 23, in __init__
module = getattr(package, spelling_mod)
AttributeError: 'module' object has no attribute 'spelling_en'
The question
What's the best way to organize all of the files required by my module to work so that users are able to use the module both from command line and from their Python code?
Thanks!
How about keeping spellnum.py?
spellnum.py
spelling/
__init__.py
en.py
es.py
Your problem is, that the package is called the same as the python-file you want to execute, thus importing
from spellnum import spellnum_en
will try to import from the file instead of the package. You could fiddle around with relative imports, but I don't know how to make them work with __import__, so I'd suggest the following:
def __init__(self, lang="en"):
mod = "spellnum_" + lang
module = None
if __name__ == '__main__':
module = __import__(mod)
else:
package = getattr(__import__("spellnum", fromlist=[mod]), mod)
Related
I'm trying to make a library out of a Python project I don't own.
The project has the following directory layout:
.
├── MANIFEST.in
├── pyproject.toml
└── src
├── all.py
├── the.py
└── sources.py
In pyproject.toml I have:
[tool.setuptools]
packages = ["mypkg"]
[tool.setuptools.package-dir]
mypkg = "src"
The problem I'm facing is that when I build and install this package I can't use it because the author is importing stuff without mypkg prefix in the various source files.
F.ex. in all.py
from the import SomeThing
Since I don't own the package I can't go modify all the sources but I still want to be able to build a library from it by just adding MANIFEST.in and pyproject.toml.
Is it possible to somehow instruct setuptools to build a package that won't litter site-packages with all the sources while still allowing them to be imported without the mypkg prefix?
It isn't possible without adding a custom import hook with the package. The hook takes the form of a module that is shipped with the package, and it must be imported before usage from your module (e.g. in src/all.py)
src/mypkgimp.py
import sys
import importlib
class MyPkgLoader(importlib.abc.Loader):
def find_spec(self, name, path=None, target=None):
# update the list with modules that should be treated special
if name in ['sources', 'the']:
return importlib.util.spec_from_loader(name, self)
return None
def create_module(self, spec):
# Uncomment if "normal" imports should have precedence
# try:
# sys.meta_path = [x for x in sys.meta_path[:] if x is not self]
# return importlib.import_module(spec.name)
# except ImportError:
# pass
# finally:
# sys.meta_path = [self] + sys.meta_path
# Otherwise, this will unconditionally shadow normal imports
module = importlib.import_module('.' + spec.name, 'mypkg')
# Final step: inject the module to the "shortened" name
sys.modules[spec.name] = module
return module
def exec_module(self, module):
pass
if not hasattr(sys, 'frozen'):
sys.meta_path = [MyPkgLoader()] + sys.meta_path
Yes, the above uses different methods described by the thread I have linked previously, as importlib have deprecated those methods in Python 3.10, refer to documentation for details.
Anyway, for the demo, put some dummy classes in the modules:
src/the.py
class SomeThing: ...
src/sources.py
class Source: ...
Now, modify src/all.py to have the following:
import mypkg.mypkgimp
from the import SomeThing
Example usage:
>>> from sources import Source
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'sources'
>>> from mypkg import all
>>> all.SomeThing
<class 'mypkg.the.SomeThing'>
>>> from sources import Source
>>> Source
<class 'mypkg.sources.Source'>
>>> from sources import Error
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: cannot import name 'Error' from 'mypkg.sources' (/tmp/mypkg/src/sources.py)
Note how the import initially didn't work, but after mypkg.all got imported, the sources import now works globally. Hence care may be needed to not shadow "real" imports and I have provided the example to import using the "default"[*] import mechanism.
If you want the module names to look different (i.e. without the mypkg. prefix), that will be a separate question, as code typically don't check for their own module name for functionality (and never mind that this actually shows how the namespace is implicitly used - changing the actual name is more akin to a module relocation, yes this can be done, but a bit more complicated and this answer is long enough as it is).
[*] "default" as in not including behaviors introduced by this custom import hook - other import hooks may do their own other weird shenanigans.
I am currently having difficulties import some functions which are located in a python file which is in the parent directory of the working directory of the main flask application script. Here's how the structure looks like
project_folder
- public
--app.py
-scripts.py
here's a replica code for app.py:
def some_function():
from scripts import func_one, func_two
func_one()
func_two()
print('done')
if __name__ == "__main__":
some_function()
scripts.py contain the function as such:
def func_one():
print('function one successfully imported')
def func_two():
print('function two successfully imported')
What is the pythonic way of importing those functions in my app.py?
Precede it with a dot so that it searches the current directory (project_folder) instead of your python path:
from .scripts import func_one, func_two
The details of relative imports are described in PEP 328
Edit: I assumed you were working with a package. Consider adding an __init__.py file.
Anyways, you can import anything in python by altering the system path:
import sys
sys.path.append("/path/to/directory")
from x import y
1.
import importlib.util
def loadbasic():
spec = importlib.util.spec_from_file_location("basic", os.path.join(os.path.split(__file__)[0], 'basic.py'))
basic = importlib.util.module_from_spec(spec)
spec.loader.exec_module(basic)
return basic #returns a module
Or create an empty file __init__.py in the directory.
And
do not pollute your path with appends.
I am having trouble creating an importable Python package/library/module or whatever the right nomenclature is. I am using Python 3.7
The file structure I am using is:
Python37//Lib//mypackage
mypackage
__init__.py
mypackage_.py
The code in __init__.py is:
from mypackage.mypackage_ import MyClass
The code in mypackage_.py is:
class MyClass:
def __init__(self, myarg = None):
self.myvar = myarg
And from my desktop I try running the following code:
import mypackage
x = MyClass(None)
But get the following error:
Traceback (most recent call last):
File "C:\Users\***\Desktop\importtest.py", line 3, in <module>
x = MyClass(None)
NameError: name 'MyClass' is not defined
You haven't imported the name MyClass into your current namespace. You've imported mypackage. To access anything within mypackage, you need to prefix the name with mypackage.<Name>
import mypackage
x = mypackage.MyClass(None)
As #rdas says, you need to prefix the name with mypackage.<Name>.
I don't recommend doing this, but you can wildcard import in order to make x = MyClass(None) work:
from mypackage import *
Now, everything from mypackage is imported and usable in the current namespace. However, you have to be careful with wildcard imports because they can create definition conflictions (if multiple modules have the same name for different things).
I am trying to calling a function from a module which is located in different directory than the current.
This is code I am using
c:\commands contains a file nlog.py which contains function parselog . i would like to importing this function
into a python file in other directory c:\applications
def parselog(inFileDir = )
### This is in directory c:\commands which need to imported ####
c:\applications\pscan.py is the script which is calling / importing from the above file / directory
if __name__ == '__main__':
#sys.path.append("C:/commands")
sys.path.insert(0,'C:/commands')
from commands import nlog
def pscan (infileDir = )
parselog(inFileDir=r'os.getcwd()') # I am calling the function here
We are getting error
NameError: global name 'parselog' is not defined
I am not sure if I am importing in a wrong way. The C:\commands folder contains _init_.py file. Please let me know what I am missing.
Make and empty __init__.py file in the folder of the module you want to import.
Below code should work.
import sys
sys.path.append(r'c:\commands')
import nlog
parselog(inFileDir=r'os.getcwd()')
I started using Python few days back and I think I have a very basic question where I am stuck. Maybe I am not doing it correctly in Python so wanted some advice from the experts:
I have a config.cfg & a class test in one package lib as follows:
myProj/lib/pkg1/config.cfg
[api_config]
url = https://someapi.com/v1/
username=sumitk
myProj/lib/pkg1/test.py
class test(object):
def __init__(self, **kwargs):
config = ConfigParser.ConfigParser()
config.read('config.cfg')
print config.get('api_config', 'username')
#just printing here but will be using this as a class variable
def some other foos()..
Now I want to create an object of test in some other module in a different package
myProj/example/useTest.py
from lib.pkg1.test import test
def temp(a, b, c):
var = test()
def main():
temp("","","")
if __name__ == '__main__':
main()
Running useTest.py is giving me error:
...
print config.get('api_config', 'username')
File "C:\Python27\lib\ConfigParser.py", line 607, in get
raise NoSectionError(section)
ConfigParser.NoSectionError: No section: 'api_config'
Now if I place thie useTest.py in the same package it runs perfectly fine:
myProj/lib/pkg1/useTest.py
myProj/lib/pkg1/test.py
myProj/lib/pkg1/config.cfg
I guess there is some very basic package access concept in Python that I am not aware of or is there something I am doing wrong here?
The issue here is that you have a different working directory depending on which module is your main script. You can check the working directory by adding the following lines to the top of each script:
import os
print os.getcwd()
Because you just provide 'config.cfg' as your file name, it will attempt to find that file inside of the working directory.
To fix this, give an absolute path to your config file.
You should be able to figure out the absolute path with the following method since you know that config.cfg and test.py are in the same directory:
# inside of test.py
import os
config_path = os.path.join(os.path.dirname(os.path.abspath(__file__)),
'config.cfg')