First, I need to describe the environment I'm writing for. I'm writing Python code that will be loaded and executed by a Python runtime running within a CAD application. The CAD application uses Python as its scripting engine. As a result, I don't have access to the Python runtime and as a good citizen to all other scripts shouldn't modify any system settings. My script is just one of many that are loaded and running.
This all works fine except when I want to use non-standard libraries. In that case, I need to install a local copy of the library for my script to access. The problem I'm having is that most libraries expect to be installed and added to the sys path which is something I shouldn't do because it could create conflicts with what other scripts are doing. What I'm attempting to do instead is to set up a local copy of the library(s) and then edit their source so their imports are relative and they don't depend on the sys path. That way my program will have its own local copy of the libraries and not depend on anything else and won't disturb any other scripts.
I'm using the -t option of PIP to install Requests and PyOpenSSL into a "Packages" subfolder in my script folder. Here's an abbreviated list of what I have.
RequestsTest/
RequestsTest.py
Packages/
OpenSSL/
cryptography/
x509/
__init__.py
base.py
hazmat/
__init__.py
backends/
__init__.py
interfaces.py
openssl/
__init__.py
backend.py
x509.py
OpenSSL/
__init__.py
SSL.py
Requests/
chardet/
__init__.py
requests/
__init__.py
urllib3/
__init__.py
request.py
contrib/
__init__.py
pyopenssl.py
util/
__init__.py
request.py
ssl_.py
Although it's tedious to track down the various import statements and make then relative, it does seem to work. However, I'm having problems with one particular set of imports.
In Packages/Requests/urllib3/contrib/pyopenssl.py it contains the following imports, which I've modified:
from ....OpenSSL.OpenSSL import SSL
from ....OpenSSL.cryptography import x509
They were originally:
from OpenSSL import OpenSSL.SSL
from cryptography import x509
I get the error "ImportError: No module named 'OpenSSL'" for the first line and "ImportError: No module named 'cryptography'" for the second line. I'm fairly certain the path is correct because if I change the number of dots I get the no module named error but it lists the full path of the what it's trying to load and not just the name of the module.
I would appreciate some help with this specific issue but can also use some overall advice of how to set up and use private copies of libraries. Remember that my program is just one of many that the system is loading changing the system or setting up a virtual environment is not an option.
Check out the localimport module, which seems to be a solution for your particular use case. From the README:
Given your Python script, application or plugin comes with a directory that contains modules for import, you can use localimport to keep the global importer state clean.
app.py
res/modules/
some_package/
__init__.py
# app.py
with localimport('res/modules') as _importer:
import some_package
assert 'some_package' not in sys.modules
The tagline is "Isolated import of Python Modules for embedded applications." so it seems pretty relevant.
When using that module, the following may help keep things neat:
Put your actual script logic into its own file.
Have a wrapper script (which will be the one loaded by the CAD software) which does localimport as mentioned in the README then does a relative import of your module. If your module gets big enough maybe put it into its own package and consume it in the same way as everything else (just do from RequestsTest import * in the body of the with localimport(): ....
Try to have a clear boundary between the source code you write and the final organized set of files required to use that source code in the context of the CAD Python runtime. It is OK to have a build/packaging step that creates the localimport script, downloads the required packages, etc. It's better even because then it is automated and not something that was done manually that someone in the future may have to recreate.
Related
I have a solution with different folders and I wanted to create a common package for them to share reusable code. There are lot of examples for python packages are there. None of them addresses separate folder scenario in a simple way.
My folder structure is :
Client/
__init__.py
--logic.py
Common/
__init__.py
-- Constants.py
Server/
__init__.py
-- Test.py
My Test.py looks like this :
from Common import Constant
print(Constant.TEST_VALUE) #-- From common package
But the code not even passes the import statements.
Exception has occurred: ModuleNotFoundError No module named 'Common'
If I try with relative path : from ..Common import Constant
Error:
Exception has occurred: ImportError attempted relative import with no known parent package
My purpose is to reuse the constant.py file for both Cars and Server solution. There are examples for sys.path which I am not sure if its a good practice. Also the code in Common will not be having complex codes to proceed with installing them.
If these packages are meant to be part of the same conceptual project, then simply make a containing package. Now the relative import will work as long as you start from outside the package hierarchy, and ensure the containing folder is on sys.path. The easy ways to do that are to install the package (preferably to a virtual environment), or start python from that folder (in which case Cars, Common and Server appear as separate packages) or just outside (in which case the containing folder is a package and Cars, Common and Server are sub-packages - which is probably what you want; it's more consistent with how you will most likely install the code later).
If these packages are separate projects that happen to depend upon one another, then Python has no reason by default to look in any particular place for the dependency packages - aside from the places it ordinarily looks for anything. So, again, you can start python from the containing folder if there actually is such a containing folder; but there isn't a good project-organization reason for there to be a containing folder - so set up an actual project for each package, and install them all to the same virtual environment. Now you can do cross-package imports with absolute syntax (but not relative syntax, because that only works within a package hierarchy).
You can use this code found here:
# Test.py
import sys
# insert at 1, 0 is the script path (or '' in REPL)
sys.path.insert(1, '/path/to/Cars/')
sys.path.insert(1, '/path/to/Common/')
import Audi
import Constants
I used below solution for importing dependencies.
I found this solution works if I run the code in Pycharm but not in Terminal.
The error message in Terminal is "cannot find graphics.primitive".
I'm using Mac and Python 3.5.
Why I see different behaviors from the Terminal and Pycharm?
How may I make the solution work for both?
http://chimera.labs.oreilly.com/books/1230000000393/ch10.html#_solution_169
Making a Hierarchical Package of Modules
Problem
You want to organize your code into a package consisting of a hierarchical collection of modules.
Solution
Making a package structure is simple. Just organize your code as you wish on the file-system and make sure that every directory defines an init.py file. For example:
graphics/
__init__.py
primitive/
__init__.py
line.py
fill.py
text.py
formats/
__init__.py
png.py
jpg.py
Once you have done this, you should be able to perform various import statements, such as the following:
import graphics.primitive.line
from graphics.primitive import line
import graphics.formats.jpg as jpg
You need to make sure that the graphics package is in the Python search path. PyCharm does this by extending sys.path as follows:
import sys
sys.path.extend(['/Users/hackworth/Development/graphics_parent_dir', '/Applications/PyCharm.app/Contents/helpers/pycharm', '/Applications/PyCharm.app/Contents/helpers/pydev'])
You can do the same in your code replacing /Users/hackworth/graphics_parent_dir with the appropriate path, or you can include the full path to graphics_parent_dir in the PYTHONPATH environment variable. See the Python documentation for details.
Another option would be to place the graphics package into a location the is searched by default on your system.
I have the following structure on a Python program:
my_program/
main.py
packages/
__init.py__
package_to_share/
__init__.py
main_to_share.py
module_to_share.py
package_A/
__init__.py
main_A.py
some_module_A.py
package_B/
__init__.py
main_B.py
some_module_B.py
The package package_to_share provides functionality that every package in the packages folder uses and that main.py at the root folder uses.
I also want to be able to cd into each package and be able to run main_X.py.
So far I figured out how to access functionality from main.py:
import packages.package_A.some_module_A
import packages.package_to_share.module_to_share
but I am having problems accessing the functionality in package_to_share from regular packages (e.g. package_A)
For example, when in main_A.py or some_module_A.py, typing import packages.package_to_share.module_to_share fails.
This leads me to following questions questions:
Given the specifics of my problem, with packages to be shared (accessed) by files at the root folder and by other packages, is there a better way to organize my folders ? Does this organization of modules and files in general conform to good standards in Python?
The following looks incredibly hacky to me, but it's the best thing I came up with to make sure my regular modules see the "shared" modules:
p_this_file = os.path.dirname(os.path.realpath(__file__))
new_sys_path.append(os.path.join(p_cwd, '..')
new_sys_path.extend(sys.path)
sys.path = new_sys_path
It also does not prevent my regular packages from importing each other.
Rather than manipulating the path for imports (which I don't recommend), you could use relative imports within module_A:
from .. import shared
That is what I sometimes do, though I generally have my packages installed so I can reference them fully where I need to use them:
from my_module import shared
I am experimenting with python, mostly troubleshooting other people's code. I am trying to get a program to run, "path\folderA\program.py".
I am running the program from path\folderA
I am getting an error:
ImportError: No module named fff.ggg.ppp
program.py contains an import:
from fff.ggg.ppp import mmm
In the folder "path\folderB" there are:
"path\folderB\fff\__init__.py"
"path\folderB\fff\ggg"
folder ggg also contains __init__.py, as well as program ppp.py
From reading other posts, like Python error "ImportError: No module named" I understand that having the __init__.py makes a folder a "package" which makes imports from it possible - but it doesn't work, since I am getting an error.
This has been working for other people that worked with these projects, so there is something wrong with my setup.
I read something about the directories having to be in the sys.path. Does that mean I have to add them to the environment variable path ? That would mean adding a lot of directories to the PATH though, so it can't be.
So I also found the following:
import sys
sys.path.append( <path to FolderB> )
But that means changing the code (which has not been necessary for other people) and hard-coding a path to what it is on my local machine - which I shouldn't have to, right ?
I can't visualize it - apparently I am not supposed to change the code and hard-code the physical path to the import module - so how can a program from folderA even know to look in folderB for an import ?
How does the magic of __init__.py work ?
I can't visualize it - apparently I am not supposed to change the code
and hard-code the physical path to the import module - so how can a
program from folderA even know to look in folderB for an import ?
You are correct. Somehow you have to tell python to look for imported modules in folderB. There is no __init__.py magic that lets you import from other folders on your hard drive.
Usually, if you've got various different python packages like that, they work by being installed into python's library. That way they can imported from anywhere. This is usually accomplished by a setup.py script. Check if folderB has one. Run it with python setup.py install.
If that doesn't work, we'll need more information about how this code is structured.
Folder B must be on the sys.path, so you would either need to move mmm to A, or modify sys.path from within A (not sure if that works). __init__.py tells python that the folder is a package, so you could have folders with __init__.py within folders with __init__.py and python treats the folders inside as parts of the parent folder. Check out sympy or almost any large python library and you will find such a structure. It can also contain code to be run on import, but can also be empty.
My situation is similar to one in this question... The difference is,
In our python/django project, we have a directory called utils, which keeps basic functions...
Sometimes, we need to test some modules by running thm from console like
python myproject/some_module.py
It is all good, until python tries to import something from our utils directory...
from utils.custom_modules import some_function
ImportError: No module named custom_modules
I check my python path, and our project is on the list, each folder under the project file has __init__.py files, and when i run ipython within project directory... Everything is ok, otherwise, python imports from its own utils directory...
My collegues use the sama method without any problem, but it throws ImportError on my environment... What could be the problem that all of us was missing?
UPDATE: My project directory, and each sub-drectory have __init__.py file, and i can import other modules from my project without any problem... When i am in a diffrent folder than my procekt and i run ipython, a such import have no problem...
from someothermodule.submodule imprort blahblahblah
But, when it comes to importing utils, it fails...
UPATE 2: What caused the problem was the utils directory under django folder, which is also in the python path...
See the PEP on absolute and relative imports for the semantics. You probably want
from .utils.custom_modules import some_function
if you're in a file at the top level of your package.
Edit: This can only be done from inside a package. This is for a good reason -- if you're importing something that is part of your project, then you're already treating it like a Python package, and you should actually make it one. You do this by adding an __init__.py file to project directory.
Edit 2: You've completely changed the question. It may be possible to work around the problem, but the correct thing to do is not refer to your packed the same way as a builtin package. You either need to rename utils, or make it a subpackage of another package, so you can refer to it by a non-conflicting name (like from mydjangoapp.utils.custom_modules import some_function).
I'm not going to bother telling you to try and make sure you don't name your own modules after the stdlib modules;
If you want to leave the name like this, you'll have to use something like this in everything that imports your own utils module:
import sys, imp
utils = sys.modules.get('utils')
if not utils: utils = imp.load_module('utils',*imp.find_module('utils/utils'))
However, unless there's a lot of stuff that you'd have to change after renaming it, I'd suggest you rename it.