I would like to include a third party library into my Python script folder to distribute it all togehter (I am awary of the distribution license, and this library is fine to distribute). This is in order to avoid installing the library on another machine.
Saying I have a script (my_script.py), which calls this external library. I tried to copy this library from the site-packages subdirectory of Python directory into the directory where I have my files, but it seems not to be enough (I think th reason is in the __init__.py of this library which probably needs the folder to be in the PYTHONPATH).
Would it be reasonable to insert some lines of code in my_script.py to temporary append its folder to sys.path in order to make the all things working?
For instance, if I have a structure similar to this:
Main_folder
my_script.py
/external_lib_folder
__init__.py
external_lib.py
and external_lib_folder is the external library I copied from site-packages and inserted in my Main_folder, would it be fine if I write these lines (e.g.) in my_script.py?
import os,sys
main_dir = os.path.dirname(os.path.abspath(__file__))
sys.path.append(main_dir)
EDIT
I ended up choosing the sys.path.append solution. I added these lines to my my_script.py:
import os, sys
# temporarily appends the folder containing this file into sys.path
main_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)),'functions')
sys.path.append(main_dir)
Anyway, I chose to insert this as an edit in my question and accept the answer of Torxed because of the time he spent in helping me (and of course because his solution works as well).
Python3
import importlib.machinery, imp
namespace = 'external_lib'
loader = importlib.machinery.SourceFileLoader(namespace, '/home/user/external_lib_folder/external_lib.py')
external_lib = loader.load_module(namespace)
# How to use it:
external_lib.function(data_or_something)
This would be an ideal way to load custom paths in Python 3.
Not entirely sure this is what you wanted but It's relevant enough to post an alternative to adding to sys.path.
Python2
In python 2 you could just do (if i'm not mistaken, been a while since i used an older version of Python):
external_lib = __import__('external_lib_folder')
This does however require you to keep the __init__.py and a proper declaration of functions in sad script, otherwise it will fail.
**It's also important that the folder you're trying to import from is of the same name that the __init__.py script in sad folder is trying to import it's sub-libraries from, for instance geopy would be:
./myscript.py
./geopy/
./geopy/__init__.py
./geopy/compat.py
...
And the code of myscript.py would look like this:
handle = __import__('geopy')
print(handle)
Which would produce the following output:
[user#machine project]$ python2 myscript.py
<module 'geopy' from '/home/user/project/geopy/__init__.pyc'>
[user#machine project]$ tree -L 2
.
├── geopy
│ ├── compat.py
│ ├── compat.pyc
│ ├── distance.py
│ ├── distance.pyc
│ ├── exc.py
│ ├── exc.pyc
│ ├── format.py
│ ├── format.pyc
│ ├── geocoders
│ ├── __init__.py
│ ├── __init__.pyc
│ ├── location.py
│ ├── location.pyc
│ ├── point.py
│ ├── point.pyc
│ ├── units.py
│ ├── units.pyc
│ ├── util.py
│ ├── util.pyc
│ └── version.pyc
└── myscript.py
2 directories, 20 files
Because in __init__.py of geopy, it's defined imports such as from geopy.point import Point which requires a namespace or a folder of geopy to be present.
There for you can't rename the folder to functions and place a folder called geopy in there because that won't work, nor will placing the contents of geopy in a folder called functions because that's not what geopy will look for.
Adding the path to sys.path (Py2 + 3)
As discussed in the comments, you can also add the folder to your sys.path variable prior to imports.
import sys
sys.path.insert(0, './functions')
import geopy
print(geopy)
>>> <module 'geopy' from './functions/geopy/__init__.pyc'>
Why this is a bad idea: It will work, and is used by many. The problems that can occur is that you might replace system functions or other modules might get loaded from other folders if you're not careful where you import stuff from. There for use .insert(0, ...) for most and be sure you actually want to risk replacing system built-ins with "shady" path names.
What you suggest is bad practice, it is a weak arrangement. The best solution (which is also easy to do) is to package it properly and add an explicit dependency, like this:
from setuptools import setup
setup(name='funniest',
version='0.1',
description='The funniest joke in the world',
url='http://github.com/storborg/funniest',
author='Flying Circus',
author_email='flyingcircus#example.com',
license='MIT',
packages=['funniest'],
install_requires=[
'markdown',
],
zip_safe=False)
This will work if the third party library is on pipy. If it's not, use this:
setup(
...
dependency_links=['http://github.com/user/repo/tarball/master#egg=package-1.0']
...
)
(See this explanation for packaging).
Related
My project structure is the following. Inside api.py i need some functions written in the upper level.
Project1
├── model.py
├── audio_utils.py
├── audio.py
└── backend
├── static
│ ├──js
│ ├──img
└── api.py
Why am I unable to import inside api.py the functions in the upper level?
When i try to do:
from audio_utils import *
I got the following:
No module named 'audio_utils'
Modules are imported from paths prefixes specified in sys.path. It usually contains '' that means that modules from current working directory are gonna be loaded.
(https://docs.python.org/3/tutorial/modules.html#packages)
I think you are starting your Python interpret while being in the backend directory. Then I think there is no way to access the modules in the upper directory -- not even with the .. (https://realpython.com/absolute-vs-relative-python-imports/#syntax-and-practical-examples_1) unless you change the sys.path which would be a really messy solution.
I suggest you create __init__.py files to indicate that the directories containing them are Python packages:
Project1
├── model.py
├── audio_utils.py
├── audio.py
└── backend
|-- __init__.py
├── static
│ ├──js
│ ├──img
└── api.py
And always start the interpret from the Project1 dir. Doing so, you should be able to import any module like this:
import model
from backed import api
import audio_utils
no matter in which module in the Project1 you are writing this in. The current directory of the interpret will be tried.
Note there is also the PYTHONPATH env variable and that you can use to your advantage.
Note that for publishing your project it is encouraged to put all the modules in a package (in other words: don't put the modues to the top level). This is to help prevent name collisions. I think this may help you to understand: https://realpython.com/pypi-publish-python-package/
You have __init__.py files in both directories right?
Try from ..audio_utils import *
If you create the dir structure this way:
$ tree
.
├── bar
│ ├── den.py
│ └── __init__.py # This indicates the bar is python package.
└── baz.py
1 directory, 3 files
$ cat bar/den.py
import baz
Then in the dir containing the bar/ and baz.py (the top level) you can start the Python interpret and use the absolute imports:
In [1]: import bar.den
In [2]: import baz
In [3]: bar.den.baz
Out[3]: <module 'baz' from '/tmp/Project1/baz.py'>
As you can see, we were able to import bar.den which also could import the baz from the top-level.
I'm using pyright for type checking and I'm also using pytest for testing inside Visual Studio Code. The folder structure for my tests is to have a 'test' subfolder in the package root . For example
|
MyPackage
|-- __init__.py
|-- MyModule.py
|--test
|-- __init__.py
|--MyModule_test.py
I'm organizing things like this as there will be many packages and I want to keep things organized.
Inside pytest I have
import pytest
import MyPackage.MyModule
...
Pytest is able to discover the tests and run them OK because it has some special ability to adjust its sys.path (or something).
However, pyright will just complain that it cannot import the module,
Import 'MyPackage.MyModule' could not be resolvedpyright (reportMissingImports). This makes sense, but is there some way to deal with this, either in pyright or in the Visual Studio Code settings to stop this from complaining?
You can add the library path to the path variable.
import sys
sys.path.insert(1, str('..'))
import MyModule
To enable Pylance to use your library properly (for auto-complete ...), use the following steps:
Pylance, by default, includes the root path of your workspace. If you want to include other subdirectories as import resolution paths, you can add them using the python.analysis.extraPaths setting for the workspace.
In VS Code press +<,> to open Settings.
Type in python.analysis.extraPaths
Select "Add Item"
Type in the path to your library `..'
Ok, a relative import as illustrated here was able to solve this. So in my case I should have
# MyModule_test.py
import pytest
from .. import MyModule
You should create a pyrightconfig.json file or pyproject.toml file at the root of your project. For example, if it's a Django project, you should have one of those files where manage.py is placed. Then, set include parameter and add the subdirectories (or app folders in Django terms).
You can consult this sample config file. See this issue ticket.
For example, if this were my project structure:
├── manage.py
├── movie
│ ├── admin.py
│ ├── apps.py
│ ├── __init__.py
│ ├── models.py
│ ├── tests.py
│ └── views.py
├── moviereviews
│ ├── asgi.py
│ ├── __init__.py
│ ├── settings.py
│ ├── urls.py
│ └── wsgi.py
└── pyproject.toml
my pyproject.toml would be:
[tool.pyright]
include = ["movie", "moviereviews"]
If you are working within a Python virtual environment, set venvPath and venv. Consult the documentation for an exhaustive list of options.
So, here's my (okay.. messy) dir:
.
├── app
│ ├── __init__.py
│ ├── analyze_text.py
│ ├── images.py
│ ├── main.py
│ ├── messages.py
│ ├── process_text.py
│ ├── requirements.txt
│ ├── response.py
│ └── tests
│ ├── __init__.py
│ ├── analyze_text_test.py
│ ├── test_process_text.py
│ └── unit_tests.py
└── setup.py # no idea what's going on with this
All I want to do is, simply use
from analyze_text import AnalyzeText
in the analyze_text_test.py file without seeing
"You're an idiot and you don't know what you're doing" in the terminal.. a.k.a:
ImportError: No module named (whatever)
I found this solution:
https://stackoverflow.com/a/11158224/2738183
import os,sys,inspect
currentdir = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
parentdir = os.path.dirname(currentdir)
sys.path.insert(0,parentdir)
import mymodule
Which works, but it's Janky.
Why?
Because..
I'm using it in a Janky way. I don't wanna have to re-paste this code to every single one of my unittests in the tests folder (which are in different files.)
(So what if i just paste it once in init.py? You get an error that's what. But I did randomly try that just to see what happened)
So what is the most elegant way to approach this problem? ( without repasting code (or just sticking it in a function and calling it multiple times) )
Edit:
The comments so far haven't solved anything, so I'll try to make this a bit more clear. I found a solution that works. So in each file in the tests directory I have to re-paste that solution (or call the same function as many times as there are files.) That's exactly what I'm trying to avoid. I'd like a solution that can apply to every file in the test directory so that I can use imports from the parent directory like normal, instead of appending the parent path inside every single one of those files.
Never mind. I found an elegant solution on stack overflow that I like.
Which amounted to placing
import os,sys,inspect
currentdir = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
parentdir = os.path.dirname(currentdir)
sys.path.insert(0,parentdir)
inside a file within the tests directory named env.py
and simply adding
import env
before importing as usual
see: https://stackoverflow.com/a/23386287/2738183
I'm making a simple python app. I don't know if I am doing it correctly or not so please correct me in the comments or if you have an answer for this
Error:
ImportError: No module named 'taskhandler'
and:
ImportError: No module named 'styles' while doing `python3 setup.py test
File Structure:
.
├── MANIFEST.in
├── pydotask.egg-info
│ ├── dependency_links.txt
│ ├── not-zip-safe
│ ├── PKG-INFO
│ ├── SOURCES.txt
│ └── top_level.txt
├── README.md
├── setup.py
├── task_mod
│ ├── __init__.py
│ ├── __pycache__
│ │ ├── __init__.cpython-35.pyc
│ │ ├── pydo.cpython-35.pyc
│ │ └── taskhandler.cpython-35.pyc
│ ├── pydo.py
│ ├── styles
│ │ ├── __init__.py
│ │ ├── __pycache__
│ │ │ ├── __init__.cpython-35.pyc
│ │ │ ├── termcolor.cpython-35.pyc
│ │ │ └── text_style.cpython-35.pyc
│ │ ├── termcolor.py
│ │ └── text_style.py
│ ├── taskhandler.py
│ └── tasks.csv
└── update.txt
5 directories, 22 files
'task_mod/pydo.py':
#!/usr/bin/env python3
''' To Do App in Python '''
import sys, os
import taskhandler as task
from styles import text_style as text
from styles import termcolor
task_mod/taskhandler.py:
#!/usr/bin/env python3
import sys, os
import csv
from styles import termcolor
from styles import text_style as text
setup.py
from setuptools import setup
def readme():
with open('README.md') as readme:
return readme.read()
setup(
name = 'pydotask',
version = '0.2',
description = 'PyDo is a CLI Application to keep you on track with your tasks and projects',
long_description = readme(),
classifiers = [
'Development Status :: 3 - Alpha',
'Programming Language :: Python :: 3.5',
'Topic :: Office/Business :: Scheduling'
],
keywords = 'utilities office schedule task reminder',
url = '',
author = 'Abhishta Gatya',
author_email = 'abhishtagatya#yahoo.com',
packages = ['task_mod'],
scripts = ['task_mod/pydo'],
python_requires = '>=3',
include_package_data = True,
zip_safe = False
)
How do I get around this problem?
Note: If I run python3 task_mod/pydo.py it works fine! But when I try testing it it, it gives 2 ImportErrors.
You have to specify all modules used in setup.py, not just the top-level folder. So in your setup.py file, replace the line packages = ['task_mod'], with packages = ['task_mod', 'task_mod.styles', 'task_mod.taskhandler'],.
Alternatively, without changing your setup.py, you can import with import task_mod.styles or with from task_mod import styles. Then you can use files in styles like styles.termcolor.
Or, you can use setuptool's black magic function find_packages like so: packages = find_packages(),
Related SO post
First, note that you provide a package named task_mod in setup.py. It means, you should only import task_mod or import task_mod.blah, never import blah. Because you do not provide blah in your library. Trying changing the imports to the absolute ones.
Second, if you still want a relative import (this is a common practice to do withing a single library — it is easier to maintain the code then), you should import relatively: from .styles import termcolor (note the dot).
Third, the relative imports only affect the modules & packages, not the scripts that you execute direcly (because pydo.py is the package __main__, not task_mod.pydo, and it changes everything). For the scripts, you have two choices to do it properly:
Choice A (so-so): Always import absolute package/module name (import task_mod.taskhandler as tashhandler in your pydo.py; also from task_mod.styles import termcolor, etc).
Choice B (best practice): Never export any scripts as part of the library (only for building/testing/CI/CD purposes). Instead, export the entry points for the console scripts (google: setuptools entry points).
setup(
....
entry_points={
'console_scripts': [
'pydo = task_mod.pydo:main',
],
},
)
And, of course, define the main() function in that module.
The shebangs (#!...python3) are completely irrelevant here.
UPD:
If you have problems with importing, please keep this in mind:
You should make your package to be on the PYTHONPATH env var (or sys.path internal variable), one way or another. This is exactly where Python looks for things when you do import things.
You can print(sys.path) before any imports to see why it happens this way — the first element will be your script's dir, and it will be varying between task_mod/pydo.py & setup.py calls.
When you run a script in the main project dir (python3 setup.py), sys.path starts with the project's dir. task_mod is there, you can import it. Also, when you do something like pip install -e ., the packages will also be "installed" into the Python/virtualenv list of libraries, but in a different way.
However, when you run python3 task_mod/pydo.py, you current dir is .../task_mod/. There, you cannot find task_mod package (because it is one level upper).
On the opposite side, when your run python3 setup.py, your current dir is the project's dir, and you cannot import styles & taskhandler directly as you did innitially (without prefixing them by task_mod.).
This is exactly the reason why you should never rely on the sys.path, and never use the scripts directly, and assume its location. Because it varies.
You can try running PYTHONPATH=. python3 task_mod/pydo.py to make it work the same as for setup.py (or PYTHONPATH=./task_mod/ python3 setup.py to make it the same as for task_mod/pydo.py). But this is a dirty hack to make it work. Instead, you should layout your library properly, according to Python's conventions.
I have the following directory structure:
├── DynamicProgramming
│ ├── 0-1_kp_problem.py
│ ├── b.py
│ ├── largest_contigous_subarray.py
│ ├── longest_common_substring.py
│ ├── min_change_for_given_money.py
│ ├── optimal_matrix_chain.py
│ ├── Readme.md
│ └── wis.py
├── helper
│ ├── a.py
│ └── __init__.py
└── Readme.md
The helper directory contains the library functions which will be used all over the code. How can I import the helper package from the scripts inside DynamicProgramming without adding it to the path?
Edit=>
I cannot move helper directory inside dynamicProgramming because there can be more than one directories using it.
You could use something like:
from ..helper import a
See python docs on packages.
If you run your code from project root folder, you are likely to succeed with import helper or import helper.a. If not, you would have to add current directory to PYTHONPATH:
$ export PYTHONPATH="."
better use project setup.py
Instead of playing with PYTHONPATH (what can be tricky business sometime), you shall create your project as python package.
You add setup.py into your project root, specify attributes of that package and build it from it.
setup.py can define multiple packages at once, but generally it is more often
using only one. For this purpose it would be better moving the helper package
into DynamicProgramming structure and import it from there.
Search for setup.py python packaging tutorials, it requires some study, but it will pay back.