I have a project containing multiple packages for example:
production/
__init__.py
prod_module_1.py
prod_subpackage/
__init__.py
prod_submodule_1.py
...
computation/
__init__.py
computation_module_1.py
...
development/
...
Now I want to prevent that specific packages can be imported by others. To go with the above example I want development modules to be able to import all types of packages but I don't want any module in production being able to import code from development. And in the same gist I want computation to be "standalone" so it can only import modules from itself, but it can be imported by other modules.
In the Python documentation https://docs.python.org/3/reference/import.html I found the following excerpt:
To selectively prevent import of some modules from a hook early on the meta path (rather than disabling the standard import system entirely), it is sufficient to raise ModuleNotFoundError directly from find_spec() instead of returning None. The latter indicates that the meta path search should continue, while raising an exception terminates it immediately.
But I'm unsure how/where I need to overwrite this method. An insight on where in the code base these changes need to be made would be greatly appreciated.
Related
I wrote a custom python package for Ansible to handle business logic for some servers I manage. I have multiple files and they reference each other by re-importing the package.
So my package named <MyCustomPackage> has functions <Function1> <Function2> <Function3>, etc all in their own files... Some of these functions reference functions in the same package, so to do that the file has:
import MyCustomPackage
at the top. I did it this way instead of a relative import because I'm also unit testing these and mocking would not work with relative paths because of a __init__ file in the test directory which was needed for test discovery. The only way I could mock was through importing the package itself. Seemed simple enough.
The problem is with Ansible. These packages are in module_utils. I import them with:
from ansible.module_utils.MyCustomPackage import MyCustomPackage
but when I use the commands I get module not found errors - and traced it back to the import MyCustomPackage statement in the package itself.
So - how should I be structuring my package? Should I try again with relative file imports, or have the package modify the path so it's found with the friendly name?
Any tips would be helpful! Or if someone has a module they've written with Python modules in module_utils and unit tests that they'd be willing to share, that'd be great also!
Many people have problems with relative imports and imports in general in Python because they are ambiguous and surprisingly depend on your current working directory (and other things).
Thus I've created an experimental, new import library: ultraimport
It gives you more control over your imports and lets you do file system based, relative imports.
Given that you have a file function1.py, to import a function from function2.py, you would then write:
import ultraimport
Function2 = ultraimport('__dir__/function2.py', 'Function2')
This will always work, no matter how you run your code. It also does not force you to a specific package structure. You can just have any files you like.
I have a solution with different folders and I wanted to create a common package for them to share reusable code. There are lot of examples for python packages are there. None of them addresses separate folder scenario in a simple way.
My folder structure is :
Client/
__init__.py
--logic.py
Common/
__init__.py
-- Constants.py
Server/
__init__.py
-- Test.py
My Test.py looks like this :
from Common import Constant
print(Constant.TEST_VALUE) #-- From common package
But the code not even passes the import statements.
Exception has occurred: ModuleNotFoundError No module named 'Common'
If I try with relative path : from ..Common import Constant
Error:
Exception has occurred: ImportError attempted relative import with no known parent package
My purpose is to reuse the constant.py file for both Cars and Server solution. There are examples for sys.path which I am not sure if its a good practice. Also the code in Common will not be having complex codes to proceed with installing them.
If these packages are meant to be part of the same conceptual project, then simply make a containing package. Now the relative import will work as long as you start from outside the package hierarchy, and ensure the containing folder is on sys.path. The easy ways to do that are to install the package (preferably to a virtual environment), or start python from that folder (in which case Cars, Common and Server appear as separate packages) or just outside (in which case the containing folder is a package and Cars, Common and Server are sub-packages - which is probably what you want; it's more consistent with how you will most likely install the code later).
If these packages are separate projects that happen to depend upon one another, then Python has no reason by default to look in any particular place for the dependency packages - aside from the places it ordinarily looks for anything. So, again, you can start python from the containing folder if there actually is such a containing folder; but there isn't a good project-organization reason for there to be a containing folder - so set up an actual project for each package, and install them all to the same virtual environment. Now you can do cross-package imports with absolute syntax (but not relative syntax, because that only works within a package hierarchy).
You can use this code found here:
# Test.py
import sys
# insert at 1, 0 is the script path (or '' in REPL)
sys.path.insert(1, '/path/to/Cars/')
sys.path.insert(1, '/path/to/Common/')
import Audi
import Constants
I've got an application that is designed as a sort of wrapper for a number of open source repositories to run and then visualise the results from them.
The relevant parts of the project layout is as below:
project/
|--repos/
|--repo1/
|--utils/
|--somecode.py
|--repo2/
|--utils/
|--somemorecode.py
|--myapp/
|--main.py
The file that is the entry point into my application is main.py within myapp. This calls into methods contained within somecode.py and somemorecode.py which return results.
The issue that I am coming across is that Python is looking at the wrong utils folder and therefore the imports are failing. For example, when I import a function from somemorecode.py into main.py, the file I am importing tries to access the utils folder located in repo1.
So far I have attempted modifications to PYTHONPATH, editing sys.path at runtime (obviously not recommended), but the former does not appear to solve the issue, and the latter threw up presumably caching issues due to having to flick between code from the two repos in fairly quick succession.
The repositories are open source GitHub repos which need to remain unmodified. These are not set up as packages as they are meant to be used as self contained applications, but I running them together for comparison purposes. Both of these repos contain utils packages with init.py files.
somecode.py uses imports such as from utils.file import function, and somemorecode.py uses the exact same sort of import but should be accessing the utils folder of repo2.
You can use as keyword. It allows you to give custom name to imported module/object.
from repo1 import utils as repo1_utils
from repo2 import utils as repo2_utils
Then just call whatewer you need by accessing this new name that you've specified:
repo1_utils.some_function()
First, I need to describe the environment I'm writing for. I'm writing Python code that will be loaded and executed by a Python runtime running within a CAD application. The CAD application uses Python as its scripting engine. As a result, I don't have access to the Python runtime and as a good citizen to all other scripts shouldn't modify any system settings. My script is just one of many that are loaded and running.
This all works fine except when I want to use non-standard libraries. In that case, I need to install a local copy of the library for my script to access. The problem I'm having is that most libraries expect to be installed and added to the sys path which is something I shouldn't do because it could create conflicts with what other scripts are doing. What I'm attempting to do instead is to set up a local copy of the library(s) and then edit their source so their imports are relative and they don't depend on the sys path. That way my program will have its own local copy of the libraries and not depend on anything else and won't disturb any other scripts.
I'm using the -t option of PIP to install Requests and PyOpenSSL into a "Packages" subfolder in my script folder. Here's an abbreviated list of what I have.
RequestsTest/
RequestsTest.py
Packages/
OpenSSL/
cryptography/
x509/
__init__.py
base.py
hazmat/
__init__.py
backends/
__init__.py
interfaces.py
openssl/
__init__.py
backend.py
x509.py
OpenSSL/
__init__.py
SSL.py
Requests/
chardet/
__init__.py
requests/
__init__.py
urllib3/
__init__.py
request.py
contrib/
__init__.py
pyopenssl.py
util/
__init__.py
request.py
ssl_.py
Although it's tedious to track down the various import statements and make then relative, it does seem to work. However, I'm having problems with one particular set of imports.
In Packages/Requests/urllib3/contrib/pyopenssl.py it contains the following imports, which I've modified:
from ....OpenSSL.OpenSSL import SSL
from ....OpenSSL.cryptography import x509
They were originally:
from OpenSSL import OpenSSL.SSL
from cryptography import x509
I get the error "ImportError: No module named 'OpenSSL'" for the first line and "ImportError: No module named 'cryptography'" for the second line. I'm fairly certain the path is correct because if I change the number of dots I get the no module named error but it lists the full path of the what it's trying to load and not just the name of the module.
I would appreciate some help with this specific issue but can also use some overall advice of how to set up and use private copies of libraries. Remember that my program is just one of many that the system is loading changing the system or setting up a virtual environment is not an option.
Check out the localimport module, which seems to be a solution for your particular use case. From the README:
Given your Python script, application or plugin comes with a directory that contains modules for import, you can use localimport to keep the global importer state clean.
app.py
res/modules/
some_package/
__init__.py
# app.py
with localimport('res/modules') as _importer:
import some_package
assert 'some_package' not in sys.modules
The tagline is "Isolated import of Python Modules for embedded applications." so it seems pretty relevant.
When using that module, the following may help keep things neat:
Put your actual script logic into its own file.
Have a wrapper script (which will be the one loaded by the CAD software) which does localimport as mentioned in the README then does a relative import of your module. If your module gets big enough maybe put it into its own package and consume it in the same way as everything else (just do from RequestsTest import * in the body of the with localimport(): ....
Try to have a clear boundary between the source code you write and the final organized set of files required to use that source code in the context of the CAD Python runtime. It is OK to have a build/packaging step that creates the localimport script, downloads the required packages, etc. It's better even because then it is automated and not something that was done manually that someone in the future may have to recreate.
I developed a python3 package in Pycharm but am running into some confusing behavior when I try to import modules in my test cases. The problem seems to be with the directory path to the internal package modules. It is a bit difficult to explain the issue, but here is the gist.
So the python package name is pyugend. Now when I try and import a module--inside the package--named Models into a test case, pycharm forces me to reference the path as pyugend.pyugend.Models. So I need to reference pyugend twice.
However, when I build, install, and import this package into a jupyter notebook or some script, then I run into errors about the pyugend package not finding the internal modules. The only way to fix these errors is to change the paths inside of the module to references like pyugend.Models.
So basically, to run tests inside of pycharm I have make sure all of the internal package imports use a directory path like from pyugend.pyugend.Models import ... But when I want to use the package outside of pycharm then I actually have to go into the package, convert all of the import pyugend.pyugend... references to just single import pyugend.Models ... references.
I have included a picture of the directory structure as well as a picture of the __init__.py.
You can add the linesys.path.append(os.path.dirname(os.path.abspath(__file__))) before the imports in __init.py__ and then change the imports to from pyugend.Models import Base_model and so on, to enable consistent behavior wherever you use the package.