This is my first time trying to set up a vagrant environment or a python virtuelenv, so forgive me if I am missing something basic.
Right now, I ssh into my vagrant box and in the home directory I have placed my venv folder. I have run
source venv/bin/activate
From my home directory I move to /vagrant, and within here I have my project files laid out something like this:
├──project
├── LICENSE
│
├── project
│ │ ├── exceptions.py
│ │ ├── __init__.py
│ │ ├── resources
│ │ │ ├── base.py
│ │ │ ├── __init__.py
│ │ └── target
│ │ └── __init__.py
│ │ └── test.py
│ ├── README.md
My problem is I am unable to import my modules in different directories. For example, if I am in /vagrant/project/project/target/test.py and I attempt:
import project.exceptions
I will get the error
ImportError: No module named project.exceptions
If I am in the /vagrant/project/project directory and I run
import exceptions
that works fine.
I have read up on similar problems people have experienced on StackOverflow.
Based on this question: Can't import package from virtualenv I have checked that my sys.executable path is the same in both my python interpreter as well as when I run a script (home/vagrant/venv/bin/python)
Based on this question: Import error with virtualenv. I have run ~/venv/bin/python directly and attempted to import, but the import still fails.
Let me know if there is more information I can provide. Thank you.
You have two options:
You can install your project into the virtual environment, by writing a setup.py file and by calling python setup.py install. See the Python Packaging User Guide.
You can set the PYTHONPATH environment variable to point to your project, like this:
$ export PYTHONPATH=$PYTHONPATH:/vagrant/project
Related
I have a project
testci/
├── __init__.py
├── README.md
├── requirements.txt
├── src
│ ├── __init__.py
│ └── mylib.py
└── test
├── __init__.py
└── pow_test.py
When I run python3.6 test/pow_test.py I see an error:
File "test/pow_test.py", line 3, in
import testci.src.mylib as mylib
ModuleNotFoundError: No module named 'testci'
pow_test.py
from testci.src.mylib import get_abs
def test_abs():
assert get_abs(-10) == 10
How can I fix this error?
System details: Ububntu 16.04 LTS, Python 3.6.10
try this
from .src import mylib
from mylib import get_abs
if it won't work then import one by one. But don't import the root folder since the file you are importing to is on the same folder you are trying to import then it will always raise an error
Run Python with the -m argument within the base testsci package to execute as a submodule.
I made a similar mock folder structure:
├───abc_blah
│ │ abc_blah.py
│ │ __init__.py
│
└───def
│ def.py
│ __init__.py
abc_blah.py
print('abc')
def.py
import abc_blah.abc_blah
Execute like such:
python -m def.def
Correctly prints out 'abc' as expected here.
simply add __package__ = "testci" and also it is a good practice to add a try and except block
Your final code should look something like
try:
from testci.src.mylib import get_abs
except ModuleNotFoundError:
from ..testci.src.mylib import get_abs
for running it, type python -m test.pow_test
I think your issue is how the package is installed. The import looks fine to me. As it says CI I'm guessing you're having the package installed remotely with only the test folder somehow.
Try adding a setup.py file where you define that both the test as well as the src packages are part of your testci package.
there are many ways to organize a project, keep things consider in mind, structure should be simple and more scaleable, can differentiate the things in codebase easily.
one of the few good possible ways to structure a project is below
project/
├── app.py
├── dockerfile
├── pipfile
├── Readme.md
├── requiements.txt
├── src_code
│ ├── code
│ │ ├── __init__.py
│ │ └── mylib.py
│ └── test
│ ├── __init__.py
│ └── test_func.py
└── travisfile
here app.py is main file which is responsible to run your entire project
I have a below structure
in migrations/env.py file I am trying to import from database import *
but it shows no module name database
I tried from ..database imprt * and adding file in pythonpath also but no luck :(
Your directory structure looks a bit suspicious to me. The alembic.ini shouldn't normally be part of the package (and setuptools won't by default pick it up when packaging). I think this would better be placed into the project-root.
Something like this would be more standard:
├── alembic.ini
├── migrations
│ ├── env.py
│ ├── script.py.mako
│ └── versions
│ └── ...
├── package_name
│ └── database
│ ├── __init__.py
│ └── ...
│ └── models
│ └── __init__.py
│ └── ...
├── README.md
└── setup.py
└── ...
Now, this alone would not make database available from env.py. For this to work you have to somehow make your package discoverable. Usually this would be done by installing package_name into some virtualenv. In that environment you could then use from package_name.database import * in your env.py.
Migrations needs to know where to import from, they either belong to the same package:
A:
migrations
database
init.py
And then in migrations:
from A.database.whatever import whatever else
Or you install them as packages separatedly inside your virtualenv:
And then each of them is dependent on the other, but because they are installed they can be invoked:
database/setup.py
migrations/setup.py
Then both are installed and migrations/env.py can call the installed package database
In order to manage all my python paths for my project and have them available as soon as I start python interpreter, I created a project.pth in the project home directory having relative paths in it,
Actually, to be read, I need to do a site.addsitedir(my_project_home_dir) each time I start the interpreter.
I tried setting PYTHONPATH or create a .pth in site-packages pointing to my project home directory, but project.pth is still not read automatically when I start the interpreter.
The only thing that works is to put my project.pth in site-packages, but by doing that, I have to transform my project relative paths to absolute paths.
So it there a master .pth file where I can specify my project home directory so I can have my project.pth located in that directory to be read automatically ?
While I know this doesn't specifically answer your question, might I suggest a different approach? The use of virtual environments (venv).
Given a project (I used pyscaffold) to create.
- Create a virtual environment at the highest level (python -m venv venv).
- Add PYTHONPATH to your venv\Scripts\activate.bat file.
- PYTHONPATH should contain at least three entries (path\to\MyTestProject; pat\to\MyTestProject\src; path\to\MyTestProject\tests). Activate your virtual environment from cmd by "venv\Scripts\activate.bat". - From within your modules, import from src (import src.mytestproject.package1.module1)
You will have your paths available to you each time you activate your virtual environment - and, you will have all the goodness associated with a virtual environment as a bonus.
MyTestProject
├── AUTHORS.rst
├── CHANGELOG.rst
├── docs
│ ├── authors.rst
│ ├── changelog.rst
│ ├── conf.py
│ ├── index.rst
│ ├── license.rst
│ ├── Makefile
│ └── _static
├── LICENSE.txt
├── README.rst
├── requirements.txt
├── setup.cfg
├── setup.py
├── src
│ └── mytestproject
│ ├── __init__.py
│ ├── package1
│ │ ├── __init__.py
│ │ └── module1.py
│ ├── package2
│ │ ├── __init__.py
│ │ └── module2.py
│ └── skeleton.py
└── tests
├── conftest.py
└── test_skeleton.py
Finally I found 2 ways :
1) First, note that .pth files also accept lines beginning with 'import' and execute them. So the solution was to create project.pth inside site-packages importing a module in my project home dir that do actually the site.addsitedir()
2) Other possibility : use the environment variable PYTHONSTARTUP : it will execute any module you want at python startup. You can specify a module that do the site.addsitedir() or even directly add paths into sys.path
I'm using pyinstaller to pack a splash screen, these are the import of the python script:
import subprocess
import time
import sys
import os
import signal
from multiprocessing import Process, Queue
import gi
gi.require_version('Gtk', '3.0')
from gi.repository import Gtk, Gdk, Pango
I use the following command to create the dist folder:
pyinstaller splash_gui.py
The problem is that the produced folder is 630Mb which is an overkill for just a splash screen program, so I investigated further and i find out that i could remove a LOT of files (the most heavy were the one in the share folder containing my themes, all of them) but more importantly i found out that the vast majority of dynamic libraries were useless.
With trial and error I managed to remove all the things that were not necessary (often generating a warning which i don't care because it is just a splash screen). Is there a direct way to avoid this mess? Briefly i want to keep just the file containing actual called functions.
This is the before and after clean situation:
630Mb 8Mb
. .
└── splash_gui └── splash_gui
├── array.so ├── binascii.so
... ├── _collections.so
├── share ├── cPickle.so
│ ├── fontconfig ├── cStringIO.so
│ ├── glib-2.0 ├── fcntl.so
│ ├── icons ├── _functools.so
│ ├── locale ├── gi._gi.so
│ ├── mime ├── _io.so
│ └── themes ├── itertools.so
... ├── libpython2.7.so.1.0
├── _sha.so ├── math.so
├── _socket.so ├── _multiprocessing.so
├── splash_gui ├── operator.so
├── _ssl.so ├── _random.so
├── strop.so ├── select.so
├── _struct.so ├── _socket.so
├── termios.so ├── splash_gui
├── time.so ├── _struct.so
├── unicodedata.so ├── time.so
└── zlib.so └── zlib.so
Except for the warnings the splash screen works normal
Pyinstaller packs all your installed librarieswith pip ,so you need to create a clean new virtual environment and only install packages that you need in that environment, when you do that install pyinstaller inside and run it
If a .exe file (and no other folders/files) is acceptable (really just try it out if you need to chack the file size), Here is my solution, running the command:
pyinstaller --onefile "directory\script_name.py"
In the current directory, a bunch of new folders will be made. Here's what to do with them:
Your file will be in "dist", and will be called script_name (without ".py", of course
The folders "dist" and "build" are unnecessary, as well as __pycache__ if it comes up
Also, the file "script_name".spec will be created.
I have the following directory structure:
├── DynamicProgramming
│ ├── 0-1_kp_problem.py
│ ├── b.py
│ ├── largest_contigous_subarray.py
│ ├── longest_common_substring.py
│ ├── min_change_for_given_money.py
│ ├── optimal_matrix_chain.py
│ ├── Readme.md
│ └── wis.py
├── helper
│ ├── a.py
│ └── __init__.py
└── Readme.md
The helper directory contains the library functions which will be used all over the code. How can I import the helper package from the scripts inside DynamicProgramming without adding it to the path?
Edit=>
I cannot move helper directory inside dynamicProgramming because there can be more than one directories using it.
You could use something like:
from ..helper import a
See python docs on packages.
If you run your code from project root folder, you are likely to succeed with import helper or import helper.a. If not, you would have to add current directory to PYTHONPATH:
$ export PYTHONPATH="."
better use project setup.py
Instead of playing with PYTHONPATH (what can be tricky business sometime), you shall create your project as python package.
You add setup.py into your project root, specify attributes of that package and build it from it.
setup.py can define multiple packages at once, but generally it is more often
using only one. For this purpose it would be better moving the helper package
into DynamicProgramming structure and import it from there.
Search for setup.py python packaging tutorials, it requires some study, but it will pay back.