We Keep Coding

Can not import pyi file [duplicate]

I am using django and I have a file named models.admin.py and I want to do the following idea in models.py:
from "models.admin" import *
however, I get a syntax error for having double quotes. But if I just do
from models.admin import *
then I get "ImportError: No module named admin"
Is there any way to import from a python file that has a period in its name?

Actually, you can import a module with an invalid name. But you'll need to use imp for that, e.g. assuming file is named models.admin.py, you could do
import imp
with open('models.admin.py', 'rb') as fp:
models_admin = imp.load_module(
'models_admin', fp, 'models.admin.py',
('.py', 'rb', imp.PY_SOURCE)
)
But read the docs on imp.find_module and imp.load_module before you start using it.

If you really want to, you can import a module with an unusual filename (e.g., a filename containing a '.' before the '.py') using the imp module:
>>> import imp
>>> a_b = imp.load_source('a.b', 'a.b.py')
>>> a_b.x
"I was defined in a.b.py!"
However, that's generally a bad idea. It's more likely that you're trying to use packages, in which case you should create a directory named "a", containing a file named "b.py"; and then "import a.b" will load a/b.py.

The file is called models/admin.py. (Source)
That is, it should be called admin.py in a directory called models.
Then you can import using from models.admin import *, assuming that it is in your Python path.

Like below
Assume dir structure is like this:
C:.
│ script.py
│
└───Parent
└───Child
├───1.1
│ main.py
│
└───1.2
**assume you want to import main.py in script.py **
your main.py looks like below
def my_function():
print("Hello from a function")
your script.py looks like below
from os import path
import importlib
from os.path import dirname
import sys
import importlib.util
def getPath():
# your logic to get to the path
return path.join(dirname(__file__),'Parent','Child','1.1','main.py')
file_path = getPath()
module_name = 'main'
spec = importlib.util.spec_from_file_location(module_name, file_path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
#call functions like this
module.my_function()
Check out this gist

No, you can't import a python file as a module if its name contains a period (or a question mark, or exclamation mark, etc). A python module's name (not including the .py) must be a valid python name (ie can be used as a variable name).

In my case, I am using grafanalib, and the filename has to be xx.dashboard.py based on the doc. However, I do want to import this file to simplify the uploading step.
I got warning when I use import imp:
the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses
Here is the simple demo using importlib and pathlib:
foo.bar.py and main.py are in the same foler.
# foo.bar.py
num = 42
# main.py
import importlib.machinery
import pathlib
module = importlib.machinery.SourceFileLoader(
"foo_bar",
pathlib.Path(__file__).parent.joinpath("foo.bar.py").resolve().as_posix(),
).load_module()
print(module.num) # 42

You are not referencing files in the import statement, you are referencing modules and packages.
Please read the docs, they are very clear on that matter.
Anyway, since you are using django, the usual approach won't work. If you want to keep models in separate files, rather than in models.py, you have to take extra steps, outlined, for example, here.
Edit:
Well, I don't really know what the questioneer means when he mentions admin and whether or not it is related to the admin interface of django. My points still stand.

How to replace usages of deprecated imp.load_dynamic?

The imp module is deprecated (since version 3.4), however some parts of the infrastructure (e.g. pyximport) still use imp.load_dynamic, which leads to deprecation-warnings with newer Python versions.
Internally, imp.load_dynamic uses importlib-machinery:
from importlib._bootstrap import _load
def load_dynamic(name, path, file=None):
"""**DEPRECATED**
Load an extension module.
"""
import importlib.machinery
loader = importlib.machinery.ExtensionFileLoader(name, path)
# Issue #24748: Skip the sys.modules check in _load_module_shim;
# always load new extension
spec = importlib.machinery.ModuleSpec(
name=name, loader=loader, origin=path)
return _load(spec)
But it feels silly to duplicate this code (which at least once had to be improved) for all kind of projects.
importlib's documentation proposes the following:
import importlib.util
import sys
def alternative_load_dynamic(name, path, file=None):
spec = importlib.util.spec_from_file_location(name, path)
module = importlib.util.module_from_spec(spec)
sys.modules[name] = module
spec.loader.exec_module(module)
return sys.modules[name]
While it has the advantage of not using the private API compared to the first implementation (i.e. _load) and gets the behavior at the core of issue #24748 right, it is not a drop-in replacement for imp.load_dynamic:
this alternative can more: it can also load pure python modules (*.py or *.pyc).
this alternative can less: it can only load dynamic modules with ending *.so (or *.pyd on Windows), but not for example foo.my_ending.
What are other options to replace imp.load_dynamic?

How about importlib.import_module ?
It also notes:
If you are dynamically importing a module that was created since the interpreter began execution (e.g., created a Python source file), you may need to call invalidate_caches() in order for the new module to be noticed by the import system.
https://docs.python.org/3/library/importlib.html#importlib.import_module

Python 3.5+: How to dynamically import a module given the full file path (in the presence of implicit sibling imports)?

Question
The standard library clearly documents how to import source files directly (given the absolute file path to the source file), but this approach does not work if that source file uses implicit sibling imports as described in the example below.
How could that example be adapted to work in the presence of implicit sibling imports?
I already checked out this and this other Stackoverflow questions on the topic, but they do not address implicit sibling imports within the file being imported by hand.
Setup/Example
Here's an illustrative example
Directory structure:
root/
- directory/
- app.py
- folder/
- implicit_sibling_import.py
- lib.py
app.py:
import os
import importlib.util
# construct absolute paths
root = os.path.abspath(os.path.dirname(os.path.dirname(os.path.realpath(__file__))))
isi_path = os.path.join(root, 'folder', 'implicit_sibling_import.py')
def path_import(absolute_path):
'''implementation taken from https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly'''
spec = importlib.util.spec_from_file_location(absolute_path, absolute_path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
isi = path_import(isi_path)
print(isi.hello_wrapper())
lib.py:
def hello():
return 'world'
implicit_sibling_import.py:
import lib # this is the implicit sibling import. grabs root/folder/lib.py
def hello_wrapper():
return "ISI says: " + lib.hello()
#if __name__ == '__main__':
# print(hello_wrapper())
Running python folder/implicit_sibling_import.py with the if __name__ == '__main__': block commented out yields ISI says: world in Python 3.6.
But running python directory/app.py yields:
Traceback (most recent call last):
File "directory/app.py", line 10, in <module>
spec.loader.exec_module(module)
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed
File "/Users/pedro/test/folder/implicit_sibling_import.py", line 1, in <module>
import lib
ModuleNotFoundError: No module named 'lib'
Workaround
If I add import sys; sys.path.insert(0, os.path.dirname(isi_path)) to app.py, python app.py yields world as intended, but I would like to avoid munging the sys.path if possible.
Answer requirements
I'd like python app.py to print ISI says: world and I'd like to accomplish this by modifying the path_import function.
I'm not sure of the implications of mangling sys.path. Eg. if there was directory/requests.py and I added the path to directory to the sys.path, I wouldn't want import requests to start importing directory/requests.py instead of importing the requests library that I installed with pip install requests.
The solution MUST be implemented as a python function that accepts the absolute file path to the desired module and returns the module object.
Ideally, the solution should not introduce side-effects (eg. if it does modify sys.path, it should return sys.path to its original state). If the solution does introduce side-effects, it should explain why a solution cannot be achieved without introducing side-effects.
PYTHONPATH
If I have multiple projects doing this, I don't want to have to remember to set PYTHONPATH every time I switch between them. The user should just be able to pip install my project and run it without any additional setup.
-m
The -m flag is the recommended/pythonic approach, but the standard library also clearly documents How to import source files directly. I'd like to know how I can adapt that approach to cope with implicit relative imports. Clearly, Python's internals must do this, so how do the internals differ from the "import source files directly" documentation?

The easiest solution I could come up with is to temporarily modify sys.path in the function doing the import:
from contextlib import contextmanager
#contextmanager
def add_to_path(p):
import sys
old_path = sys.path
sys.path = sys.path[:]
sys.path.insert(0, p)
try:
yield
finally:
sys.path = old_path
def path_import(absolute_path):
'''implementation taken from https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly'''
with add_to_path(os.path.dirname(absolute_path)):
spec = importlib.util.spec_from_file_location(absolute_path, absolute_path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
This should not cause any problems unless you do imports in another thread concurrently. Otherwise, since sys.path is restored to its previous state, there should be no unwanted side effects.
Edit:
I realize that my answer is somewhat unsatisfactory but, digging into the code reveals that, the line spec.loader.exec_module(module) basically results in exec(spec.loader.get_code(module.__name__),module.__dict__) getting called. Here spec.loader.get_code(module.__name__) is simply the code contained in lib.py.
Thus a better answer to the question would have to find a way to make the import statement behave differently by simply injecting one or more global variables through the second argument of the exec-statement. However, "whatever you do to make the import machinery look in that file's folder, it'll have to linger beyond the duration of the initial import, since functions from that file might perform further imports when you call them", as stated by #user2357112 in the question comments.
Unfortunately the only way to change the behavior of the import statement seems to be to change sys.path or in a package __path__. module.__dict__ already contains __path__ so that doesn't seem to work which leaves sys.path (Or trying to figure out why exec does not treat the code as a package even though it has __path__ and __package__ ... - But I don't know where to start - Maybe it has something to do with having no __init__.py file).
Furthermore this issue does not seem to be specific to importlib but rather a general problem with sibling imports.
Edit2: If you don't want the module to end up in sys.modules the following should work (Note that any modules added to sys.modules during the import are removed):
from contextlib import contextmanager
#contextmanager
def add_to_path(p):
import sys
old_path = sys.path
old_modules = sys.modules
sys.modules = old_modules.copy()
sys.path = sys.path[:]
sys.path.insert(0, p)
try:
yield
finally:
sys.path = old_path
sys.modules = old_modules

add to the PYTHONPATH environment variable the path your application is on
Augment the default search path for module files. The format is the same as the shell’s PATH: one or more directory pathnames
separated by os.pathsep (e.g. colons on Unix or semicolons on
Windows). Non-existent directories are silently ignored.
on bash its like this:
export PYTHONPATH="./folder/:${PYTHONPATH}"
or run directly:
PYTHONPATH="./folder/:${PYTHONPATH}" python directory/app.py

The OP's idea is great, this work only for this example by adding sibling modules with proper name to the sys.modules, I would say it is the SAME as adding PYTHONPATH. tested and working with version 3.5.1.
import os
import sys
import importlib.util
class PathImport(object):
def get_module_name(self, absolute_path):
module_name = os.path.basename(absolute_path)
module_name = module_name.replace('.py', '')
return module_name
def add_sibling_modules(self, sibling_dirname):
for current, subdir, files in os.walk(sibling_dirname):
for file_py in files:
if not file_py.endswith('.py'):
continue
if file_py == '__init__.py':
continue
python_file = os.path.join(current, file_py)
(module, spec) = self.path_import(python_file)
sys.modules[spec.name] = module
def path_import(self, absolute_path):
module_name = self.get_module_name(absolute_path)
spec = importlib.util.spec_from_file_location(module_name, absolute_path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return (module, spec)
def main():
pathImport = PathImport()
root = os.path.abspath(os.path.dirname(os.path.dirname(os.path.realpath(__file__))))
isi_path = os.path.join(root, 'folder', 'implicit_sibling_import.py')
sibling_dirname = os.path.dirname(isi_path)
pathImport.add_sibling_modules(sibling_dirname)
(lib, spec) = pathImport.path_import(isi_path)
print (lib.hello())
if __name__ == '__main__':
main()

Try:
export PYTHONPATH="./folder/:${PYTHONPATH}"
or run directly:
PYTHONPATH="./folder/:${PYTHONPATH}" python directory/app.py
Make sure your root is in a folder that is explicitly searched in the PYTHONPATH. Use an absolute import:
from root.folder import implicit_sibling_import #called from app.py

Make sure your root is in a folder that is explicitly searched in the PYTHONPATH
Use an absolute import:
from root.folder import implicit_sibling_import # called from app.py

How to import members of all modules within a package?

I am developing a package that has a file structure similar to the following:
test.py
package/
__init__.py
foo_module.py
example_module.py
If I call import package in test.py, I want the package module to appear similar to this:
>>> vars(package)
mapping_proxy({foo: <function foo at 0x…}, {example: <function example at 0x…})
In other words, I want the members of all modules in package to be in package's namespace, and I do not want the modules themselves to be in the namespace. package is not a sub-package.
Let's say my files look like this:
foo_module.py:
def foo(bar):
return bar
example_module.py:
def example(arg):
return foo(arg)
test.py:
print(example('derp'))
How do I structure the import statements in test.py, example_module.py, and __init__.py to work from outside the package directory (i.e. test.py) and within the package itself (i.e. foo_module.py and example_module.py)? Everything I try gives Parent module '' not loaded, cannot perform relative import or ImportError: No module named 'module_name'.
Also, as a side-note (as per PEP 8): "Relative imports for intra-package imports are highly discouraged. Always use the absolute package path for all imports. Even now that PEP 328 is fully implemented in Python 2.5, its style of explicit relative imports is actively discouraged; absolute imports are more portable and usually more readable."
I am using Python 3.3.

I want the members of all modules in package to be in package's
namespace, and I do not want the modules themselves to be in the
namespace.
I was able to do that by adapting something I've used in Python 2 to automatically import plug-ins to also work in Python 3.
In a nutshell, here's how it works:
The package's __init__.py file imports all the other Python files in the same package directory except for those whose names start with an '_' (underscore) character.
It then adds any names in the imported module's namespace to that of __init__ module's (which is also the package's namespace). Note I had to make the example_module module explicitly import foo from the .foo_module.
One important aspect of doing things this way is realizing that it's dynamic and doesn't require the package module names to be hardcoded into the __init__.py file. Of course this requires more code to accomplish, but also makes it very generic and able to work with just about any (single-level) package — since it will automatically import new modules when they're added and no longer attempt to import any removed from the directory.
test.py:
from package import *
print(example('derp'))
__init__.py:
def _import_all_modules():
""" Dynamically imports all modules in this package. """
import traceback
import os
global __all__
__all__ = []
globals_, locals_ = globals(), locals()
# Dynamically import all the package modules in this file's directory.
for filename in os.listdir(__name__):
# Process all python files in directory that don't start
# with underscore (which also prevents this module from
# importing itself).
if filename[0] != '_' and filename.split('.')[-1] in ('py', 'pyw'):
modulename = filename.split('.')[0] # Filename sans extension.
package_module = '.'.join([__name__, modulename])
try:
module = __import__(package_module, globals_, locals_, [modulename])
except:
traceback.print_exc()
raise
for name in module.__dict__:
if not name.startswith('_'):
globals_[name] = module.__dict__[name]
__all__.append(name)
_import_all_modules()
foo_module.py:
def foo(bar):
return bar
example_module.py:
from .foo_module import foo # added
def example(arg):
return foo(arg)

I think you can get the values you need without cluttering up your namespace, by using from module import name style imports. I think these imports will work for what you are asking for:
Imports for example_module.py:
from package.foo_module import foo
Imports for __init__.py:
from package.foo_module import foo
from package.example_module import example
__all__ = [foo, example] # not strictly necessary, but makes clear what is public
Imports for test.py:
from package import example
Note that this only works if you're running test.py (or something else at the same level of the package hierarchy). Otherwise you'd need to make sure the folder containing package is in the python module search path (either by installing the package somewhere Python will look for it, or by adding the appropriate folder to sys.path).

Creating aliases for Python packages?

I have a directory, let's call it Storage full of packages with unwieldy names like mypackage-xxyyzzww, and of course Storage is on my PYTHONPATH. Since packages have long unmemorable names, all of the packages are symlinked to friendlier names, such as mypackage.
Now, I don't want to rely on file system symbolic links to do this, instead I tried mucking around with sys.path and sys.modules. Currently I'm doing something like this:
import imp
imp.load_package('mypackage', 'Storage/mypackage-xxyyzzww')
How bad is it to do things this way, and is there a chance this will break in the future? One funny thing is that there's even no mention of imp.load_package function in the docs.
EDIT: besides not relying on symbolic links, I can't use PYTHONPATH variable anymore.

Instead of using imp, you can assign different names to imported modules.
import mypackage_xxyyzzww as mypackage
If you then create a __init__.py file inside of Storage, you can add several of the above lines to make importing easier.
Storage/__init__.py:
import mypackage_xxyyzzww as mypackage
import otherpackage_xxyyzzww as otherpackage
Interpreter:
>>> from Storage import mypackage, otherpackage

importlib may be more appropriate, as it uses/implements the PEP302 mechanism.
Follow the DictImporter example, but override find_module to find the real filename and store it in the dict, then override load_module to get the code from the found file.
You shouldn't need to use sys.path once you've created your Storage module
#from importlib import abc
import imp
import os
import sys
import logging
logging.basicConfig(level=logging.DEBUG)
dprint = logging.debug
class MyImporter(object):
def __init__(self,path):
self.path=path
self.names = {}
def find_module(self,fullname,path=None):
dprint("find_module({fullname},{path})".format(**locals()))
ml = imp.find_module(fullname,path)
dprint(repr(ml))
raise ImportError
def load_module(self,fullname):
dprint("load_module({fullname})".format(**locals()))
return imp.load_module(fullname)
raise ImportError
def load_storage( path, modname=None ):
if modname is None:
modname = os.path.basename(path)
mod = imp.new_module(modname)
sys.modules[modname] = mod
assert mod.__name__== modname
mod.__path__=[path]
#sys.meta_path.append(MyImporter(path))
mod.__loader__= MyImporter(path)
return mod
if __name__=="__main__":
load_storage("arbitrary-path-to-code/Storage")
from Storage import plain
from Storage import mypkg
Then when you import Storage.mypackage, python will immediately use your importer without bothering to look on sys.path
That doesn't work. The code above does work to import ordinary modules under Storage without requiring Storage to be on sys.path, but both 3.1 and 2.6 seem to ignore the loader attribute mentioned in PEP302.
If I uncomment the sys.meta_path line, 3.1 dies with StackOverflow, and 2.6 dies with ImportError. hmmm... I'm out of time now, but may look at it later.

Packages are just entries in the namespace. You should not name your path components with anything that is not a legal python variable name.