I have the following folder structure:
data_plugin_main/
/common
__init__.py
utils.py
/data
__init__.py
commmon_dal.py
plugin1
/data
__init__.py
data_plugin_dal.py
plugin2
/data
__init__.py
another_plugin_dal.py
I need to maintain this structure, but be able to write code that either imports or can reference:
data.common_dal
data.data_plugin_dal
data.another_plugin_dal
It seems like python will only import/load modules in the first data folder it encounters. so in above case, since there is in __init__.py under data_plugin_main/data/, common_dal.py is loaded. however, even if I add the other paths to the PYTHONPATH, data_plugin_dal and another_plugin_dal are not able to be imported or referenced.
Does anyone know how to solve this problem?
Thanks
If I’m understanding correctly, you may want to try Pydev environment installed freely in Eclipse.
Each module is also the directory name with a parent module that fathers them all, so in pydev if plugin_2 source needs to reference common_dal.py the import looks like from data_plugin_main import data.common_dal
So in the project explorer you would create a parent module That acts as a super __init__ and then you can create child modules as you wish. It’s kinda Javaishly inspired.
I was able to solve this using namespace-packaging: https://packaging.python.org/guides/packaging-namespace-packages/
Related
I defined a directory named common in the root directory of my project. Basically, there isn't a main file, but rather lots of scripts which I want to directly run.
What I want to do is something like that:
from ../../common/ import <python_file>
Is it possible without too much of a hustle?
Would I need to put __init__.py to make it work?
The use of .. for imports is really a bad practice. What you should do is that. Create a virtualenv using virtualenvwrapper and then use add2virtualenv common
OR
Add the path of the root directory of your project into your PYTHONPATH env variable
export PYTHONPATH=$PYTHONPATH:/path/of/my/project
An __init__.py file must be placed in every folder containing python code
I've been developing a project in python for some time using pycharm. I have no problem running it in pycharm, but I want to try and running it from the command line (I'm using windows).
When I try to run my main file using python <filename> from within the root directory, I get a module not found error. What is pycharm doing/how can i replicate it?
Also, pycharm creates pycache folders. If my understanding is correct its compiling the files, and that makes the runtime faster. Since my runtime is already long i'd like to do the same.
I'm using python 3.6
edit
File Structure
-Root\
----scheduler\
------main.py
------matrices
------models
------rhc
------Singleton.py
------utils.py
------__init__.py
------apis\
acedb.py
metrics.py
__init__.py
------matrices\
distancematrix.py
__init__.py
------models\
branch.py
constants.py
customer.py
dispatch.py
technician.py
workorder.py
__init__.py
------rhc\
pathfinder.py
rhc.py
schedule.py
sched_time.py
tech_schedule.py
__init__.py
Edit 2
Found the solution, i had to move my main files outside of the modules
Thanks!
If you have the below folder structure for instance. add an empty file named init.py and import from your app.py, in case you have a Module1Class , you can always import it like this
from modules.module1 import Module1Class
from modules.module2 import Module2Class
Folder structure
/app.py
/modules
/module1.py
/module2.py
/__init__.py
The first time you rum app.py from terminal like python app.py, the .pyc files will be created, leaving you with the following folder structure
/app.py
/modules
/module1.py
/module1.pyc
/module2.py
/module2.pyc
/__init__.py
Please refer to the python documentation as it's very well documented on how to create modules and importing them. I'm more used to python2.7 , so my answer might not be an exact fit to newer python versions.
https://docs.python.org/3/tutorial/modules.html
From there you can learn more about __ init __.py and module creation , exporting and importing
PS: I use only text editors to develop in python, as I find pycharm a bit on the heavy side, so I cannot explain how exactly pycharm works behind the curtains.
see ModuleNotFoundError: What does it mean __main__ is not a package? for a good example of ModuleNotFoundError description (was answered fast)
I bet that pycharm is configured to use a different python interpreter of virtual environment.
There're a lot of threads on importing modules from sibling directories, and majority recommends to either simply add init.py to source tree, or modify sys.path from inside those init files.
Suppose I have following project structure:
project_root/
__init__.py
wrappers/
__init__.py
wrapper1.py
wrapper2.py
samples/
__init__.py
sample1.py
sample2.py
All init.py files contain code which inserts absolute path to project_root/ directory into the sys.path. I get "No module names x", no matter how I'm trying to import wrapperX modules into sampleX. And when I try to print sys.path from sampleX, it appears that it does not contain path to project_root.
So how do I use init.py correctly to set up project environment variables?
Do not run sampleX.py directly, execute as module instead:
# (in project root directory)
python -m samples.sample1
This way you do not need to fiddle with sys.path at all (which is generally discouraged). It also makes it much easier to use the samples/ package as a library later on.
Oh, and init.py is not run because it only gets run/imported (which is more or less the same thing) if you import the samples package, not if you run an individual file as script.
I am developing several Python projects for several customers at the same time. A simplified version of my project folder structure looks something like this:
/path/
to/
projects/
cust1/
proj1/
pack1/
__init__.py
mod1.py
proj2/
pack2/
__init__.py
mod2.py
cust2/
proj3/
pack3/
__init__.py
mod3.py
When I for example want to use functionality from proj1, I extend sys.path by /path/to/projects/cust1/proj1 (e.g. by setting PYTHONPATH or adding a .pth file to the site_packages folder or even modifying sys.path directly) and then import the module like this:
>>> from pack1.mod1 import something
As I work on more projects, it happens that different projects have identical package names:
/path/
to/
projects/
cust3/
proj4/
pack1/ <-- same package name as in cust1/proj1 above
__init__.py
mod4.py
If I now simply extend sys.path by /path/to/projects/cust3/proj4, I still can import from proj1, but not from proj4:
>>> from pack1.mod1 import something
>>> from pack1.mod4 import something_else
ImportError: No module named mod4
I think the reason why the second import fails is that Python only searches the first folder in sys.path where it finds a pack1 package and gives up if it does not find the mod4 module in there. I've asked about this in an earlier question, see import python modules with the same name, but the internal details are still unclear to me.
Anyway, the obvious solution is to add another layer of namespace qualification by turning project directories into super packages: Add __init__.py files to each proj* folder and remove these folders from the lines by which sys.path is extended, e.g.
$ export PYTHONPATH=/path/to/projects/cust1:/path/to/projects/cust3
$ touch /path/to/projects/cust1/proj1/__init__.py
$ touch /path/to/projects/cust3/proj4/__init__.py
$ python
>>> from proj1.pack1.mod1 import something
>>> from proj4.pack1.mod4 import something_else
Now I am running into a situation where different projects for different customers have the same name, e.g.
/path/
to/
projects/
cust3/
proj1/ <-- same project name as for cust1 above
__init__.py
pack4/
__init__.py
mod4.py
Trying to import from mod4 does not work anymore for the same reason as before:
>>> from proj1.pack4.mod4 import yet_something_else
ImportError: No module named pack4.mod4
Following the same approach that solved this problem before, I would add yet another package / namespace layer and turn customer folders into super super packages.
However, this clashes with other requirements I have to my project folder structure, e.g.
Development / Release structure to maintain several code lines
other kinds of source code like e.g. JavaScript, SQL, etc.
other files than source files like e.g. documents or data.
A less simplified, more real-world depiction of some project folders looks like this:
/path/
to/
projects/
cust1/
proj1/
Development/
code/
javascript/
...
python/
pack1/
__init__.py
mod1.py
doc/
...
Release/
...
proj2/
Development/
code/
python/
pack2/
__init__.py
mod2.py
I don't see how I can satisfy the requirements the python interpreter has to a folder structure and the ones that I have at the same time. Maybe I could create an extra folder structure with some symbolic links and use that in sys.path, but looking at the effort I'm already making, I have a feeling that there is something fundamentally wrong with my entire approach. On a sidenote, I also have a hard time believing that python really restricts me in my choice of source code folder names as it seems to do in the case depicted.
How can I set up my project folders and sys.path so I can import from all projects in a consistent manner if there are project and packages with identical names ?
This is the solution to my problem, albeit it might not be obvious at first.
In my projects, I have now introduced a convention of one namespace per customer. In every customer folder (cust1, cust2, etc.), there is an __init__.py file with this code:
import pkgutil
__path__ = pkgutil.extend_path(__path__, __name__)
All the other __init__.py files in my packages are empty (mostly because I haven't had the time yet to find out what else to do with them).
As explained here, extend_path makes sure Python is aware there is more than one sub-package within a package, physically located elsewhere and - from what I understand - the interpreter then does not stop searching after it fails to find a module under the first package path it encounters in sys.path, but searches all paths in __path__.
I can now access all code in a consistent manner criss-cross between all projects, e.g.
from cust1.proj1.pack1.mod1 import something
from cust3.proj4.pack1.mod4 import something_else
from cust3.proj1.pack4.mod4 import yet_something_else
On a downside, I had to create an even deeper project folder structure:
/path/
to/
projects/
cust1/
proj1/
Development/
code/
python/
cust1/
__init__.py <--- contains code as described above
proj1/
__init__.py <--- empty
pack1/
__init__.py <--- empty
mod1.py
but that seems very acceptable to me, especially considering how little effort I need to make to maintain this convention. sys.path is extended by /path/to/projects/cust1/proj1/Development/code/python for this project.
On a sidenote, I noticed that of all the __init__.py files for the same customer, the one in the path that appears first in sys.path is executed, no matter from which project I import something.
You should be using the excellent virtualenv and virtualenvwrapper tools.
What happens if you accidentally import code from one customer/project in another and don't notice? When you deliver it will almost certainly fail. I would adopt a convention of having PYTHONPATH set up for one project at a time, and not try to have everything you've ever written be importable at once.
You can use a wrapper script per-project to set PYTHONPATH and start python, or use scripts to switch environments when you switch projects.
Of course some projects well have dependencies on other projects (those libraries you mentioned), but if you intend for the customer to be able to import several projects at once then you have to arrange for the names to not clash. You can only have this problem when you have multiple projects on the PYTHONPATH that aren't supposed to be used together.
I have a python file in the location 'lib/lib_add_participant.py'.And I declared a class in the file.
Now I want to call the class functions from the file call_participant.py.
I tried this code from lib/lib_add_participant.py import LibAddParticipant
Its not worked.Please correct me(I am coming from Ruby).
If your file structure is as follows:
myproject/
__init__.py
call_participant.py
lib/
__init__.py
lib_add_participant.py
In this case, your __init__.py files can be empty, if you like (or they can define any number of things - see this fine answer from Alex Martelli for more details.
then you can add it by using
from .lib.lib_add_participant import LibAddParticipant
However, if call_participant.py is your end script, then this tactic will not work, because you "can't do a relative import in a non-package", as the interpreter will tell you. In that case, your best bet (in my opinion, at least) is to make lib into a package in your python path (either in your site-packages directory, or in a path referred to by your pythonpath environment variable).
Assuming lib is in a directory listed in your PYTHONPATH, try
from lib.lib_add_participant import LibAddParticipant