In some Python scripts I see the following imports:
import fileA
import someDir.fileB
from fileC import functionA
There exist corresponding files fileA.py, someDir/fileB.py and fileC.py. However, while looking in the Requests source code, I found this in the __init__.py file:
from requests.packages.urllib3.contrib import pyopenssl
In this case, requests is the CWD and packages.urllib3.contrib.pyopenssl.py is the file. Why does this defy convention? I do see that the packages.urllib3.contrib directory does also have a __init__.py file, which seems to be related.
Furthermore, I'm not sure if it is related but I think it is so I post it here. In my script I have the folder kennethreitz/requests, since the application depends on the Requests module but I'm deploying it to environments which might not have Requests installed. However, simply adding to the file import kennethreitz.requests is not including the Requests module. I import kennethreitz.requests.__init__ and a few other obvious permutations but I cannot get the module to import. How can I package Requests with my code? The obvious Google searches are not helping.
requests is using an absolute import. You cannot arbitrarily nest packages into other directories and still expect things to work.
Instead, add the kennethreitz directory (which should not have a __init__.py file) to your sys.path module search path. That way the requests module will still be importable as a top-level package.
Next, you may want to look into Python packaging, dependencies and using a tool like pip or zc.buildout to deploy your code for you. Those tools handle dependencies for you and will install requests as required. See the Python Packaging User Guide for an introduction.
Related
I have searched through the existing questions on stackoverflow and wasn't able to find what I am looking for. Firstly, I am new to Python, I come from Ruby so some things seem unclear to me in Python. I learn by doing so I am writing my own python REST API client for a payment gateway, which I plan on releasing to PyPi. The problem I have is importing modules from different folders.
Let's say I have the folder structure like this:
my_project/src/lib/directory1/module1.py
my_project/src/lib/directory2/module2.py
In my_project/src/lib/directory1/module1.py I want to import a function defined in my_project/src/lib/directory2/module2.py so
# my_project/src/lib/directory2/module2.py
from lib.directory2 import module1
This doesn't work, it says ImportError: No module named directory2. I read that in Python you need to add the module to the PATH but I have gone to PyPi and took the first library from the top (SeleniumBase) to look at how the code is organised in the GitHub project and I don't see they do any stuff like that. Could you please explain to me how this works and how I can organise my codebase to be able to import modules in one module from different folders to build my own library?
I read this article https://docs.python.org/3/reference/import.html and tried what they say in section 5.7. Package Relative Imports but it doesn't work
In theory this should work
from ..subpackage2.moduleZ import eggs
But it doesn't. I get SystemError: Parent module '' not loaded, cannot perform relative import.
Import will only work if the module you are trying to import is in the same directory as the script that is doing the import.
Otherwise you have to specify the path.
Like this :
import my_project/src/lib/directory1/module1.py
First, I need to describe the environment I'm writing for. I'm writing Python code that will be loaded and executed by a Python runtime running within a CAD application. The CAD application uses Python as its scripting engine. As a result, I don't have access to the Python runtime and as a good citizen to all other scripts shouldn't modify any system settings. My script is just one of many that are loaded and running.
This all works fine except when I want to use non-standard libraries. In that case, I need to install a local copy of the library for my script to access. The problem I'm having is that most libraries expect to be installed and added to the sys path which is something I shouldn't do because it could create conflicts with what other scripts are doing. What I'm attempting to do instead is to set up a local copy of the library(s) and then edit their source so their imports are relative and they don't depend on the sys path. That way my program will have its own local copy of the libraries and not depend on anything else and won't disturb any other scripts.
I'm using the -t option of PIP to install Requests and PyOpenSSL into a "Packages" subfolder in my script folder. Here's an abbreviated list of what I have.
RequestsTest/
RequestsTest.py
Packages/
OpenSSL/
cryptography/
x509/
__init__.py
base.py
hazmat/
__init__.py
backends/
__init__.py
interfaces.py
openssl/
__init__.py
backend.py
x509.py
OpenSSL/
__init__.py
SSL.py
Requests/
chardet/
__init__.py
requests/
__init__.py
urllib3/
__init__.py
request.py
contrib/
__init__.py
pyopenssl.py
util/
__init__.py
request.py
ssl_.py
Although it's tedious to track down the various import statements and make then relative, it does seem to work. However, I'm having problems with one particular set of imports.
In Packages/Requests/urllib3/contrib/pyopenssl.py it contains the following imports, which I've modified:
from ....OpenSSL.OpenSSL import SSL
from ....OpenSSL.cryptography import x509
They were originally:
from OpenSSL import OpenSSL.SSL
from cryptography import x509
I get the error "ImportError: No module named 'OpenSSL'" for the first line and "ImportError: No module named 'cryptography'" for the second line. I'm fairly certain the path is correct because if I change the number of dots I get the no module named error but it lists the full path of the what it's trying to load and not just the name of the module.
I would appreciate some help with this specific issue but can also use some overall advice of how to set up and use private copies of libraries. Remember that my program is just one of many that the system is loading changing the system or setting up a virtual environment is not an option.
Check out the localimport module, which seems to be a solution for your particular use case. From the README:
Given your Python script, application or plugin comes with a directory that contains modules for import, you can use localimport to keep the global importer state clean.
app.py
res/modules/
some_package/
__init__.py
# app.py
with localimport('res/modules') as _importer:
import some_package
assert 'some_package' not in sys.modules
The tagline is "Isolated import of Python Modules for embedded applications." so it seems pretty relevant.
When using that module, the following may help keep things neat:
Put your actual script logic into its own file.
Have a wrapper script (which will be the one loaded by the CAD software) which does localimport as mentioned in the README then does a relative import of your module. If your module gets big enough maybe put it into its own package and consume it in the same way as everything else (just do from RequestsTest import * in the body of the with localimport(): ....
Try to have a clear boundary between the source code you write and the final organized set of files required to use that source code in the context of the CAD Python runtime. It is OK to have a build/packaging step that creates the localimport script, downloads the required packages, etc. It's better even because then it is automated and not something that was done manually that someone in the future may have to recreate.
Is it possible to use modules installed via python pip in gcloud deployment manager templates (python templates, not jinja)?
I have only being able to find reference of how to import .py files through a deployment manager schema file. e.g.
app.py.schema
info:
title: app
author: me
description: this is a description
imports:
- path: helper.py
i.e. i can only import a single .py at a time, so not useful for importing pip modules.
this link explains that to use libraries that is not explicitly supported we need to import the full library source. Although it does not mention if this full library source can actually be a pip module, or is it only referring to single .py files.
The module i'm trying to use inside my python templates is netaddr for manipulating ip address and subnets.
Any help is appreciated.
what you are looking for it not possible, you cannot install module using pip with interacting the the API, unless if you want to import the whole netaddr module as source code in your *.yaml config file (by adding the path for all the files related to the module) then importing which function your *.py file as Google mention in the documentation some library are supported, even with that some sys and network call will be rejected, you may think about using template_module
Original Answer:
Yes, you can check the link Here for importing multiple python files and using multiple templates.
I used below solution for importing dependencies.
I found this solution works if I run the code in Pycharm but not in Terminal.
The error message in Terminal is "cannot find graphics.primitive".
I'm using Mac and Python 3.5.
Why I see different behaviors from the Terminal and Pycharm?
How may I make the solution work for both?
http://chimera.labs.oreilly.com/books/1230000000393/ch10.html#_solution_169
Making a Hierarchical Package of Modules
Problem
You want to organize your code into a package consisting of a hierarchical collection of modules.
Solution
Making a package structure is simple. Just organize your code as you wish on the file-system and make sure that every directory defines an init.py file. For example:
graphics/
__init__.py
primitive/
__init__.py
line.py
fill.py
text.py
formats/
__init__.py
png.py
jpg.py
Once you have done this, you should be able to perform various import statements, such as the following:
import graphics.primitive.line
from graphics.primitive import line
import graphics.formats.jpg as jpg
You need to make sure that the graphics package is in the Python search path. PyCharm does this by extending sys.path as follows:
import sys
sys.path.extend(['/Users/hackworth/Development/graphics_parent_dir', '/Applications/PyCharm.app/Contents/helpers/pycharm', '/Applications/PyCharm.app/Contents/helpers/pydev'])
You can do the same in your code replacing /Users/hackworth/graphics_parent_dir with the appropriate path, or you can include the full path to graphics_parent_dir in the PYTHONPATH environment variable. See the Python documentation for details.
Another option would be to place the graphics package into a location the is searched by default on your system.
I have the following structure on a Python program:
my_program/
main.py
packages/
__init.py__
package_to_share/
__init__.py
main_to_share.py
module_to_share.py
package_A/
__init__.py
main_A.py
some_module_A.py
package_B/
__init__.py
main_B.py
some_module_B.py
The package package_to_share provides functionality that every package in the packages folder uses and that main.py at the root folder uses.
I also want to be able to cd into each package and be able to run main_X.py.
So far I figured out how to access functionality from main.py:
import packages.package_A.some_module_A
import packages.package_to_share.module_to_share
but I am having problems accessing the functionality in package_to_share from regular packages (e.g. package_A)
For example, when in main_A.py or some_module_A.py, typing import packages.package_to_share.module_to_share fails.
This leads me to following questions questions:
Given the specifics of my problem, with packages to be shared (accessed) by files at the root folder and by other packages, is there a better way to organize my folders ? Does this organization of modules and files in general conform to good standards in Python?
The following looks incredibly hacky to me, but it's the best thing I came up with to make sure my regular modules see the "shared" modules:
p_this_file = os.path.dirname(os.path.realpath(__file__))
new_sys_path.append(os.path.join(p_cwd, '..')
new_sys_path.extend(sys.path)
sys.path = new_sys_path
It also does not prevent my regular packages from importing each other.
Rather than manipulating the path for imports (which I don't recommend), you could use relative imports within module_A:
from .. import shared
That is what I sometimes do, though I generally have my packages installed so I can reference them fully where I need to use them:
from my_module import shared