I am working on a program and access a set of custom modules. I organized the modules in a subfolder as own package with an __init__.py. Moreover, on the main level of the directory I have created a virtual environment that holds my dependencies. The folder structure is as follows:
project
+-- main_program.py
+-- venv
| +-- cool_package.py
+---mypackage
| +-- module1.py
| +-- module2.py
| +-- __init__.py
The issue is, that module2.py depends on a package I installed in venv. Running module2.py from main_program.py gives an error "cool_package.py" not found.
How do I organize stuff like that so that I can accesses cool_package.py from main_program.py with all the other needed packaged. And make cool_package.py accessible for the custom package with module2.py as well?
I may have misunderstood what you mean by your virtual env but based on your folder and file layout I think you need to add an __init.py__ file to your venv folder to make it a package and then you should be able to import venv.cool_package.
thanks for all answers - it eventually worked by properly activating the environment before running the script. Something must have gone wrong the first time - all works now and the folder structure is correct.
Best
Moritz
Related
Project structure
I have the following folder structure
|
|- src
| |- mypackage
| | |- __init__.py
| | |- mymodule.py
| |- utils.egg
|- main.py
in mymodule.py file I can import the egg adding it to the sys.path as
import sys
sys.path.append('src/utils.egg')
import utils
When calling main.py everything works fine (python -m main).
Problem
The problem comes from pylint. First, it shows the following message in mymodule.py file
Unable to import 'utils' pylint(import-error)
if I ask for suggestions (CRTL + Space) when importing I got
utils.build
.dist
.utils
.setup
# |- suggestions
And from utils.utils I can acces the actual classes / functions in utils module. Of course if I import utils.utils, when executing the main script, an importing error pops up.
How can I configure my vscode setting in order fix pylint?
should I install the egg instead of copy it to the working folder?
Is my project's folder-structure ok, or it goes against recommended practices?
Extra info
In case you wonder the EGG-INFO/SOURCE.txt file looks like
setup.py
utils/__init__.py
utils/functions.py
utils.egg-info/PKG-INFO
utils.egg-info/SOURCES.txt
utils.egg-info/dependency_links.txt
utils.egg-info/top_level.txt
utils/internals/__init__.py
utils/internals/somemodule.py
utils/internals/someothermodule.py
Also, there aren't build nor dist folder in the egg.
This is an issue with Pylint itself and not the Python extension, so it will come down to however you need to configure Pylint.
As for whether you should copy an egg around or install it, you should be installing it into your virtual environment, or at least copying over the appropriate .pth file to make the egg directory work appropriately.
I have been working on a python project and I am new to it.
I have made a small library for my project in which I have several different modules doing different tasks.
For example: I have 5 modules namely add, subtract, multiply, divide and root.
I call all these .pyc files into my main.py file and my code runs properly if all of them are in the same folder.
Now, I want to store my main.py at: D:\project\main.py
and these 5 .pyc files at : D:\project\Lib\ (In the Lib folder)
I found a solution as to mention the path of the folder Lib into the code but I can not do so as I need to submit the code somewhere and if they try to run this on their PC, it might not import these files.
What would be the possible solution to this?
Try creating a package.
Use a directory structure like this:
.
+-- main.py
+-- lib
+-- __init__.py
+-- add.pyc
+-- substract.pyc
+-- ...
Then, in your main.py file, you can import them like this:
from lib import add
More on packages on Python docs
Inside D:\project\Lib create an __init__.py file. and put all your modules in D:\project\Lib now lib works as a python package.you dir structure should now look like this:
D:\project\Lib
|
+--- __init__.py
+--- add.py
+--- sub.py
+--- multiply.py
Now from any file inside (say for ex main.py) D:\project call any module you want like this.
from Lib.add import something.
final dir structure will roughly look like this.
D:\project
|
+-- main.py
+-- Lib
|
+--- __init__.py
+--- add.py
+--- sub.py
+--- multiply.py
I’m looking for solution for designing my program.
My program consists of 3 blocks:
Classes
Functions
Other utilities
I want to structure my program this way:
program_folder/
main.py
classes_folder/
class_1.py
class_2.py
functions_folder/
set_of_func_1.py
set_of_func_1.py
utilities_folder/
set_of_utilities_1.py
set_of_utilities_1.py
I want to:
any scripts in «classes_folder» were able to import any of scripts in
«functions_folder».
any scripts in «functions_folder» were able
to import any of scripts in «utilities_folder».
all scripts were
normally used by main.py.
all scripts in «classes_folder»,
«functions_folder» and «utilities_folder» could be tested when worked
as «main» (if __name__ == “__main__”: some tests)
«program_folder»
could be in any place in my computer (there shouldn’t be dependency
on exact path to «program_folder»).
From all the above I thought I have to:
Change import search path for all scripts in «classes_folder»,
«functions_folder» and «utilities_folder».
Set current working
directory to «program_folder» for all scripts?
Is there a way I can do it?
Does my idea look good or have I put there some unexpected problems?
You can create a skeleton project like the following:
/path/to/project/
setup.py
my_project/
__init__.py
a/
__init__.py
b/
__init__.py
==> ./my_project/__init__.py <==
print('my_project/__init__.py')
==> ./my_project/a/__init__.py <==
import my_project
print('my_project/a/__init__.py')
==> ./my_project/b/__init__.py <==
import my_project.a
print('my_project/b/__init__.py')
==> ./setup.py <==
from distutils.core import setup
setup(name='my_project',
version='1.0',
description='my_project',
author='author',
packages=['my_project'])
Then you can install the project locally using pip install -e /path/to/project/ (the project folder is not copied, just gets registered; there's a dependency on the exact path, but this dependency is not hard-coded in project files themselves).
As the result, import my_project, import my_project.a etc. do that they mean:
$ python my_project/b/__init__.py
my_project/__init__.py
my_project/a/__init__.py
my_project/b/__init__.py
A common Python project structure could look like this:
project_name/
setup.py
requirements.txt
project_name/
__main__.py
classes/
__init__.py
class1.py
class2.py
functions/
__init__.py
functions.py
utils/
__init__.py
utils.py
Then, you could modify your imports from absolute to relative and run your package using something like:
$ /path/to/project_name> python -m project_name
Note that setup.py is only required if you want to install your package under some of your interpreters.
Note: see comments below also
My project has the following structure:
server/ (root of project)
|
|--- __init__.py
|--- requirements.txt
|--- env/ (virtual environment)
|--- app/ (main app folder)
|--- __init__.py (defines a 'app = Flask(__name__)' object)
|--- app.py (runs app on local server)
|--- models.py
|--- views.py
The way I import different modules in app.py on my local machine is do:
# /server/app/app.py
from server.app import app
from server.app.models import *
from server.app.views import *
It works fine on my local machine (using PyCharm IDE, and the Python binary inside virtual environment folder /server/env/bin/.
However, when I push this to the production server running Ubuntu, where I install all dependencies globally, it keeps throwing the error no module named server.app when I run:
python server/app/app.py
Does anyone know why?
Any IDE environment usually sets the pythonpath. E.g. in eclipse right click your project and see the properties. You will see that your main project is listed in the pythonpath. This path is used to locate modules.
Now in your production code you are not in your IDE. So normal python interpreter cannot find your path. Hence you need to specify this.
One way is to add sys.path.append('/path/to/the/project') before you do your import (This should be done for the first script that got executed, in this case app.py, that way you only need to do this once).
You can also add your path permanently to your production environment. See this post.
As #Rash mentionned, your IDE very probably add the directory containing /server to your python path. You can check this in your app.py by adding
import sys
print "\n".join(sys.path)
before your imports.
When you're manually running your app, ie python server/app/app.py, the parent of your server directory is obviously not in your python path, so you have to add yourself.
I've seen all sorts of examples and other similar questions, but I can't seem to find an example that exactly matches my scenario. I feel like a total goon asking this because there are so many similar questions, but I just can't seem to get this working "correctly." Here is my project:
user_management (package)
|
|------- __init__.py
|
|------- Modules/
| |
| |----- __init__.py
| |----- LDAPManager.py
| |----- PasswordManager.py
|
|------- Scripts/
| |
| |----- __init__.py
| |----- CreateUser.py
| |----- FindUser.py
If I move "CreateUser.py" to the main user_management directory, I can easily use: "import Modules.LDAPManager" to import LDAPManager.py --- this works. What I can't do (which I want to do), is keep CreateUser.py in the Scripts subfolder, and import LDAPManager.py. I was hoping to accomplish this by using "import user_management.Modules.LDAPManager.py". This doesn't work. In short, I can get Python files to easily look deeper in the hierarchy, but I can't get a Python script to reference up one directory and down into another.
Note that I am able to solve my problem using:
sys.path.append(os.path.join(os.path.dirname(__file__), '..'))
import Modules.LDAPManager as LDAPManager
I've heard that this is bad practice and discouraged.
The files in Scripts are intended to be executed directly (is the init.py in Scripts even necessary?). I've read that in this case, I should be executing CreateUser.py with the -m flag. I've tried some variations on this and just can't seem to get CreateUser.py to recognize LDAPManager.py.
If I move CreateUser.py to the main user_management directory, I can
easily use: import Modules.LDAPManager to import LDAPManager.py
--- this works.
Please, don't. In this way the LDAPManager module used by CreateUser will not be the same as the one imported via other imports. This can create problems when you have some global state in the module or during pickling/unpickling. Avoid imports that work only because the module happens to be in the same directory.
When you have a package structure you should either:
Use relative imports, i.e if the CreateUser.py is in Scripts/:
from ..Modules import LDAPManager
Note that this was (note the past tense) discouraged by PEP 8 only because old versions of python didn't support them very well, but this problem was solved years ago. The current version of PEP 8 does suggest them as an acceptable alternative to absolute imports. I actually like them inside packages.
Use absolute imports using the whole package name(CreateUser.py in Scripts/):
from user_management.Modules import LDAPManager
In order for the second one to work the package user_management should be installed inside the PYTHONPATH. During development you can configure the IDE so that this happens, without having to manually add calls to sys.path.append anywhere.
Also I find it odd that Scripts/ is a subpackage. Because in a real installation the user_management module would be installed under the site-packages found in the lib/ directory (whichever directory is used to install libraries in your OS), while the scripts should be installed under a bin/ directory (whichever contains executables for your OS).
In fact I believe Script/ shouldn't even be under user_management. It should be at the same level of user_management.
In this way you do not have to use -m, but you simply have to make sure the package can be found (this again is a matter of configuring the IDE, installing the package correctly or using PYTHONPATH=. python Scripts/CreateUser.py to launch the scripts with the correct path).
In summary, the hierarchy I would use is:
user_management (package)
|
|------- __init__.py
|
|------- Modules/
| |
| |----- __init__.py
| |----- LDAPManager.py
| |----- PasswordManager.py
|
Scripts/ (*not* a package)
|
|----- CreateUser.py
|----- FindUser.py
Then the code of CreateUser.py and FindUser.py should use absolute imports to import the modules:
from user_management.Modules import LDAPManager
During installation you make sure that user_management ends up somewhere in the PYTHONPATH, and the scripts inside the directory for executables so that they are able to find the modules. During development you either rely on IDE configuration, or you launch CreateUser.py adding the Scripts/ parent directory to the PYTHONPATH (I mean the directory that contains both user_management and Scripts):
PYTHONPATH=/the/parent/directory python Scripts/CreateUser.py
Or you can modify the PYTHONPATH globally so that you don't have to specify this each time. On unix OSes (linux, Mac OS X etc.) you can modify one of the shell scripts to define the PYTHONPATH external variable, on Windows you have to change the environmental variables settings.
Addendum I believe, if you are using python2, it's better to make sure to avoid implicit relative imports by putting:
from __future__ import absolute_import
at the top of your modules. In this way import X always means to import the toplevel module X and will never try to import the X.py file that's in the same directory (if that directory isn't in the PYTHONPATH). In this way the only way to do a relative import is to use the explicit syntax (the from . import X), which is better (explicit is better than implicit).
This will make sure you never happen to use the "bogus" implicit relative imports, since these would raise an ImportError clearly signalling that something is wrong. Otherwise you could use a module that's not what you think it is.
From Python 2.5 onwards, you can use
from ..Modules import LDAPManager
The leading period takes you "up" a level in your heirarchy.
See the Python docs on intra-package references for imports.
In the "root" __init__.py you can also do a
import sys
sys.path.insert(1, '.')
which should make both modules importable.
I faced the same issues. To solve this, I used export PYTHONPATH="$PWD". However, in this case, you will need to modify imports in your Scripts dir depending on the below:
Case 1: If you are in the user_management dir, your scripts should use this style from Modules import LDAPManager to import module.
Case 2: If you are out of the user_management 1 level like main, your scripts should use this style from user_management.Modules import LDAPManager to import modules.