I am new to Python. I am at a company where they built a large system in Python. They use a proprietary system to manage the paths when the system is running, but now I have been asked to build a standalone script that interacts with some of the code in their system. Sadly, my standalone script won't be running under the path-manager they use, so I need to figure out the paths on my own.
So, for instance, I have this line:
from hark.tasks import REPLY_LINE
This is actually copied from some of their older code. In this case, the script can find hark, but hark has an __init__.py file, and that is where the problems start. So I get this:
meg/src/python2/hark/hark/__init__.py in <module>()
5 from flask import jsonify, render_template, request
6 import jinja2
----> 7 import logbook, logbook.compat
8
9 from healthhark.context import Ghost, g
The project that they built actually includes logbook 3 times. If I do:
find . -name "*logbook*"
I see:
meg/zurge/opt/python2.7/lib/python2.7/site-packages/logbook
meg/zurge/opt/python2.7-hark/lib/python2.7/site-packages/logbook
meg/zurge/opt/python3.4/lib/python3.4/site-packages/logbook
Like I said, they have a proprietary path manager that usually tells each piece of code where it can find the packages that it should include, but I am building a standalone app.
I don't know much about Python, but I am wondering if their is an idiomatic and Pythonic way of including packages that are in such distant directories?
And, before anyone suggests pip install, we don't rely on global installs at all.
The best solution would probably be virtualenv or virtualenvwrapper. This would allow you to define an environment which contains all of the libraries that your script requires. This would not be global.
This can be done as follows:
Create a requirements.txt file defining the required libraries
Install pip and virtualenvwrapper
source /usr/local/bin/virtualenvwrapper.sh
workon hark_task_script_env || ( mkvirtualenv hark_task_script_env && pip install -r requirements.txt )
python your-script.py
The virtualenvwrapper installs the environment into a folder in your home directory. The plain virtualenv library installs the environment into a folder in the project. Other than that they are equivalent.
I would really recommend using them over a proprietry package manager. If you must use the proprietry package manager then it is reasonable to have access to the package loader for it!
If this really isn't satisfactory then you can hack a package loader as follows (assuming you use a *nix system). This is a shell script written in zsh. Hopefully it is clear enough for you to rewrite it in a supported shell if your system does not have that available:
#!/bin/zsh
setopt extended_glob
function find_package_init_files () {
locate __init__.py
}
# Get the containing folder of a file or a folder
function file_or_folder_to_parent_folder () {
while read file_or_folder
do
echo ${file_or_folder:h}
done
}
# Exclude folders where the parent folder also has an __init__.py file in it
function exclude_inner_packages () {
while read init_file_folder
do
init_file_parent_folder=${init_file_folder:h}
if [ ! -e ${init_file_parent_folder}/__init__.py ]
then
echo ${init_file_folder}
fi
done
}
# This produces an array of all folders
# that contain at least one python package
function get_distinct_python_package_folders () {
find_package_init_files |
file_or_folder_to_parent_folder |
exclude_inner_packages |
file_or_folder_to_parent_folder |
sort |
uniq
}
PYTHONPATH=${(j/:/)$(get_distinct_python_package_folders)} YOUR_SCRIPT_HERE
You may well need to update this script to put the default python path first, and remember that this is an incredibly clumsy approach. If there are multiple versions of libraries installed on a system then the one that you may end up using will be ill defined.
Using a proper package manager, even if that is the proprietry one, is the best way.
Related
Just working my way through a (very good) book call Test Driven Development using Python.
This makes use of Python3.4 by the way. By the way, I am running in a Windows 7 OS.
I've got all the stuff working using a simple text editor and running from the command line... in the course of which in particular I used "pip install" to install Django and Selenium, as per book's instructions.
This created folders "selenium" and "django" under ...\Python34\Lib\site-packages\ ... so I added these to the PythonPath for my Eclipse/PyDev project.
With the correct interpreter selected I then tried to run a file which runs fine on the command line: "> python3 functional_tests.py"... but I get
File "D:\apps\Python34\lib\site-packages\django\http\__init__.py", line 1, in <module>
from django.http.cookie import SimpleCookie, parse_cookie
File "D:\apps\Python34\lib\site-packages\django\http\cookie.py", line 5, in <module>
from django.utils.six.moves import http_cookies
ImportError: cannot import name 'http_cookies'
... to me this looks like a dependency thing... as though "pip install" handles dependency matters in a way just including a single folder doesn't.
Question boils down to this: what's the "proper" way to install a python module using PyDev?
several days later
wow... nothing? Nothing! I suppose this must mean that you either have to add dependencies manually or use something like Ant, Maven or Gradle within Eclipse itself. These latter are not my strong areas, even outside an IDE. Would still be nice to have an answer from a PyDev expert!
Well, pip install should work for PyDev (it should automatically recognize the dependency)...
I.e.: in your use case, the only folder that should be in the PYTHONPATH is D:\apps\Python34\lib\site-packages (and pip should install packages to that folder -- make sure you don't add extra folders for "D:\apps\Python34\lib\site-packages\django" nor anything else inside the site-packages to the PYTHONPATH).
If it's still not working, please check if the module django.utils.six.moves.http_cookies is indeed where you expect it to be. Also, you can print the PYTHONPATH being used in runtime with:
import sys
print('\n'.join(sorted(sys.path)))
To check if that's really what you expect.
I'm trying to use python in a new OS X application for plugin scripting. I'm looking to offload some of the program logic to python for easy, on-the-fly modification. I had started by looking at the Big Nerd Ranch tutorial. That seemed to work, but it suggested an older method of linking Python. It appears that since Xcode 5, we are supposed to install python using this Mac Developer Library Tech Note. This process, however, seems to create a linked python instance, not an embedded one. I've tried following the guidelines in the answer to this question but it seems to break down.
So my question is this: what are the current best practices for building python for use as a plugin runtime in an Objective C Mac OS X app? How does one go about making sure that it is bundled with the application, and that it is possible to install any additional libraries that one might want before the final Objective C app is built?
I've come up with a method that seems to work fairly well.
First, download source of the python version you want to use from the official website. Extract the archive somewhere. I'm using Python 3.4.2. Adjust the commands on your system for the specific version you're using.
Create a build directory that you will use for this development python version. The entire directory should have no spaces in it to make sure that bash interprets the she-bang (#!) lines correctly. I used /Users/myaccount/development/python/devbuild/python3.4.2.
Go into the extracted Python directory and run the following commands:
./configure --prefix="/Users/myaccount/development/python/devbuild/python3.4.2"
make
make install
This will install python in that development build directory. Setup the Python path to use the correct directory:
export PYTHONPATH="/Users/myaccount/development/python/devbuild/python3.4.2/lib/python3.4/site-packages/"
Go into the python bin directory (/Users/myaccount/development/python/devbuild/python3.4.2/bin) and use the pip3 there to install any modules that you need. The $PYTHONPATH setting will ensure that the modules get installed into the correct site-packages directory.
Find a handy home for the PyObjC repository and clone it there. Then checkout the latest version tag and install it, making sure that your $PYTHONPATH is still correct:
hg clone https://bitbucket.org/ronaldoussoren/pyobjc
cd pyobjc
hg tags
hg checkout [the id of the latest version from the tag list]
/Users/myaccount/development/python/devbuild/python3.4.2/bin/python3 ./install.py
Whenever you need to update the python modules, just make sure to use the correct python bin and $PYTHONPATH.
Now to add python to an Xcode project.
Drag the /Users/myaccount/development/python/devbuild/python3.4.2 directory into the Xcode project, setting it to not copy items as necessary, and to create a folder reference.
Add /Users/myaccount/development/python/devbuild/python3.4.2/include/python3.4m to the Header Search Paths setting in the Xcode project's Build Settings. Not sure if there's a way to do this as a generalized step to just search the folder referenced directory we had just added.
Drag the `/Users/myaccount/development/python/devbuild/python3.4.2/lib/libpython3.4m.a library into the Xcode project, setting it to be added as a reference without copying as well.
The code from the Big Nerd Ranch scripting tutorial repository can now be used with a few modifications.
The Plugin Manager code will need an NSString extension to work with the wchar_t strings that the Python API seems to like so much:
#interface NSString (WcharEncodedString)
- (wchar_t*) getWideString;
#end
#implementation NSString (WcharEncodedString)
- (wchar_t*) getWideString {
const char* tmp = [self cStringUsingEncoding:NSUTF8StringEncoding];
unsigned long buflen = strlen(tmp) + 1;
wchar_t* buffer = malloc(buflen * sizeof(wchar_t));
mbstowcs(buffer, tmp, buflen);
return buffer;
}
#end
The Python header should be included as follows:
#include "Python.h"
The following code needs to be run before Py_Initialize() is called in order to set up the correct python executable, PYTHONPATH, and PYTHONHOME as suggested by Zorg on that other question.
NSString* executablePath = [[NSBundle mainBundle] pathForResource:#"python3.4" ofType:nil inDirectory:#"python3.4.2/bin"];
Py_SetProgramName([executablePath getWideString]);
NSString* pythonDirectory = [[NSBundle mainBundle] pathForResource:#"python3.4" ofType:nil inDirectory:#"python3.4.2/lib"];
Py_SetPath([pythonDirectory getWideString]);
Py_SetPythonHome([pythonDirectory getWideString]);
Finally, the python path needs to be expanded in the PluginExecutor.py file to include the various subdirectories of the high level lib path. Add the following code to the top of the Plugin Executor file:
import sys
from os import walk
path = sys.path.copy()
for p in path:
for root,dirs,files in walk(p):
if p is not root:
sys.path.append(root)
I'll post updates if things start to break down, but this seems a working solution for now.
My context is appengine_config.py, but this is really a general Python question.
Given that we've cloned a repo of an app that has an empty directory lib in it, and that we populate lib with packages by using the command pip install -r requirements.txt --target lib, then:
dirname ='lib'
dirpath = os.path.join(os.path.dirname(__file__), dirname)
For importing purposes, we can add such a filesystem path to the beginning of the Python path in the following way (we use index 1 because the first position should remain '.', the current directory):
sys.path.insert(1, dirpath)
However, that won't work if any of the packages in that directory are namespace packages.
To support namespace packages we can instead use:
site.addsitedir(dirpath)
But that appends the new directory to the end of the path, which we don't want in case we need to override a platform-supplied package (such as WebOb) with a newer version.
The solution I have so far is this bit of code which I'd really like to simplify:
sys.path, remainder = sys.path[:1], sys.path[1:]
site.addsitedir(dirpath)
sys.path.extend(remainder)
Is there a cleaner or more Pythonic way of accomplishing this?
For this answer I assume you know how to use setuptools and setup.py.
Assuming you would like to use the standard setuptools workflow for development, I recommend using this code snipped in your appengine_config.py:
import os
import sys
if os.environ.get('CURRENT_VERSION_ID') == 'testbed-version':
# If we are unittesting, fake the non-existence of appengine_config.
# The error message of the import error is handled by gae and must
# exactly match the proper string.
raise ImportError('No module named appengine_config')
# Imports are done relative because Google app engine prohibits
# absolute imports.
lib_dir = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'libs')
# Add every library to sys.path.
if os.path.isdir(lib_dir):
for lib in os.listdir(lib_dir):
if lib.endswith('.egg'):
lib = os.path.join(lib_dir, lib)
# Insert to override default libraries such as webob 1.1.1.
sys.path.insert(0, lib)
And this piece of code in setup.cfg:
[develop]
install-dir = libs
always-copy = true
If you type python setup.py develop, the libraries are downloaded as eggs in the libs directory. appengine_config inserts them to your path.
We use this at work to include webob==1.3.1 and internal packages which are all namespaced using our company namespace.
You may want to have a look at the answers in the Stack Overflow thread, "How do I manage third-party Python libraries with Google App Engine? (virtualenv? pip?)," but for your particular predicament with namespace packages, you're running up against a long-standing issue I filed against site.addsitedir's behavior of appending to sys.path instead of inserting after the first element. Please feel free to add to that discussion with a link to this use case.
I do want to address something else that you said that I think is misleading:
My context is appengine_config.py, but this is really a general Python
question.
The question actually arises from the limitations of Google App Engine and the inability to install third-party packages, and hence, seeking a workaround. Rather than manually adjusting sys.path and using site.addsitedir. In general Python development, if your code uses these, you're Doing It Wrong.
The Python Packaging Authority (PyPA) describes the best practices to put third party libraries on your path, which I outline below:
Create a virtualenv
Mark out your dependencies in your setup.py and/or requirements files (see PyPA's "Concepts and Analyses")
Install your dependencies into the virtualenv with pip
Install your project, itself, into the virtualenv with pip and the -e/--editable flag.
Unfortunately, Google App Engine is incompatible with virtualenv and with pip. GAE chose to block this toolset in an attempt sandbox the environment. Hence, one must use hacks to work around the limitations of GAE to use additional or newer third party libraries.
If you dislike this limitation and want to use standard Python tooling for managing third-party package dependencies, other Platform as a Service providers out there eagerly await your business.
I just installed ycm, everything looks good, but I found small problem. The problem is as following:
import os # os is built-in library
os. # ycm helps to complete members of the class.
import numpy # numpy is not built-in library, where its location is site-packages.
numpy. # nothing happened. ycm shows 'pattern not found' message.
I think, this would be a simple problem. But I could not find the solution yet. I think, there is some configuration file in which I can define 'search path' for my project.
It would be grateful if I can find a way to solve it.
Best,
Je-Hoon Song
I had the same issue with module 'mpmath' and fixed it in the following manner:
First I retrieved the path where the module was located:
%python3
>>>import mpmath
>>>print(mpmath.__file__)
/usr/lib/python3.4/site-packages/mpmath/__init__.py
Here I found the path of all my "installed" python3 packages to be:
/usr/lib/python3.4/site-packages/
I then simply added to my PYTHONPATH environment variable this path:
%export PYTHONPATH=/usr/lib/python3.4/site-packages/
Then when I used vim sample.py typing import mpmath and following it up with mpmath. YCM showed me all the autocompletions for the mpmath module.
Hope this helps.
I use anaconda python to be my python interpreter in ycm to solve this.
First I modified my vimrc according to full pythong setting in vim.
Then I change g:ycm_python_interpreter_path by
let g:ycm_python_interpreter_path = '/usr/local/anaconda3/bin/python3.8'
In this way I didn't change the system environment variables.
Addtional Info 1:
I think the main problem is that,
my Python interpreter for YCM is my system python (/usr/local/opt/python#3.9/bin/python3.9 ),
which only has limited locally built libraries.
So using the libraries comes with anaconda (/usr/local/anaconda3/bin/python3.8 ) can solve.
Additional Info 2:
By reading :YcmDebugInfo, the main different after edited g:ycm_python_interpreter_path is that:
-- Python completer debug information:
-- Python interpreter: /usr/local/opt/python#3.9/bin/python3.9
-- Python path: ['/usr/local/Cellar/python#3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python39.zip', '/usr/local/Cellar/python#3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9', '/usr/local/Cellar/python#3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload', '/usr/local/lib/python3.9/site-packages']
-- Python version: 3.9.6
change to
-- Python completer debug information:
-- Python interpreter: /usr/local/anaconda3/bin/python3.8
-- Python path: ['/usr/local/anaconda3/lib/python38.zip', '/usr/local/anaconda3/lib/python3.8', '/usr/local/anaconda3/lib/python3.8/lib-dynload', '/usr/local/anaconda3/lib/python3.8/site-packages', '/usr/local/anaconda3/lib/python3.8/site-packages/aeosa', '/usr/local/anaconda3/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg']
-- Python version: 3.8.8
Additional Info 3: about how to read list of locally installed pyton modules
use https://stackoverflow.com/a/740018/11226687
e.g. in my case
$ /usr/local/opt/python#3.9/bin/python3.9
>>> help('modules')
# only return limitted modules
$ /usr/local/anaconda3/bin/python3
>>> help('modules')
# list out all the modules included in Anaconda, including numpy/matplotlib/scipy ect
numpy is kind of a difficult library because it dynamically builds its namespace on import, making it hard for static code analysis tools to know when you're write the code what names should be available. Because the names available in the namespace numpy are only really known at runtime, YCM probably doesn't have any useful suggestions for you.
One simple way to fix is activate your python environment, then open vim. For example
(django_mdn) ➜ locallibrary git:(master) ✗ vim
and in the vim run :echo $PATH.
Then you should be able to see that your venv path is at the first like this:
/Users/gwanghyeongim/.virtualenvs/django_mdn/bin:/usr/local/opt/tcl-tk/bin:...
Then see if your python packages are auto-complete.
It worked.
If you want to set a certain site-packages to be auto complete permanently, you need to make a file called .ycm_ extra_conf.py in your project root directory or global_extra_conf.py and set vim configuration if you want to set it globally.
P.S.
By running export PYTHONPATH=/usr/lib/python3.4/site-packages/ in the shell before opening vim didn't work for me. Besides, unless setting PYTHONPATH permanently, which will cause issue, you will have to set export PYTHONPATH everytime you want dependencies to be auto complete.
I'm using suds (brilliant library, btw), and I'd like to make it portable (so that everyone who uses the code that relies on it, can just checkout the files and run it).
I have tracked down 'suds-0.4-py2.6.egg' (in python/lib/site-packages), and put it in with my files, and I've tried:
import path.to.egg.file.suds
from path.to.egg.file.suds import *
import path.to.egg.file.suds-0.4-py2.6
The first two complain that suds doesn't exist, and the last one has invalid syntax.
In the __init__.py file, I have:
__all__ = [ "FileOne" ,
"FileTwo",
"suds-0.4-py2.6"]
and have previously tried
__all__ = [ "FileOne" ,
"FileTwo",
"suds"]
but neither work.
Is this the right way of going about it? If so, how can I get my imports to work. If not, how else can I achieve the same result?
Thanks
You must add your egg file to sys.path, like this:
import sys
# insert at 0 instead of appending to end to take precedence
# over system-installed suds (if there is one).
sys.path.insert(0, "suds-0.4-py2.6.egg")
import suds
.egg files are zipped archives; hence you cannot directly import them as you have discovered.
The easy way is to simply unzip the archive, and then copy the suds directory to your application's source code directory. Since Python will stop at the first module it discovers; your local copy of suds will be used even if it is not installed globally for Python.
One step up from that, is to add the egg to your path by appending it to sys.path.
However, the proper way would be to package your application for distribution; or provide a requirements file that lets other people know what external packages your program depends on.
Usually I distribute my program with a requirements.txt file that contain all dependencies and their version.
The users can then install these libraries with:
pip install -r requirements.txt
I don't think including eggs with your code is a good idea, what if the user use python2.7 instead of python2.6
More info about requirement file: http://www.pip-installer.org/en/latest/requirements.html