I am experimenting with python, mostly troubleshooting other people's code. I am trying to get a program to run, "path\folderA\program.py".
I am running the program from path\folderA
I am getting an error:
ImportError: No module named fff.ggg.ppp
program.py contains an import:
from fff.ggg.ppp import mmm
In the folder "path\folderB" there are:
"path\folderB\fff\__init__.py"
"path\folderB\fff\ggg"
folder ggg also contains __init__.py, as well as program ppp.py
From reading other posts, like Python error "ImportError: No module named" I understand that having the __init__.py makes a folder a "package" which makes imports from it possible - but it doesn't work, since I am getting an error.
This has been working for other people that worked with these projects, so there is something wrong with my setup.
I read something about the directories having to be in the sys.path. Does that mean I have to add them to the environment variable path ? That would mean adding a lot of directories to the PATH though, so it can't be.
So I also found the following:
import sys
sys.path.append( <path to FolderB> )
But that means changing the code (which has not been necessary for other people) and hard-coding a path to what it is on my local machine - which I shouldn't have to, right ?
I can't visualize it - apparently I am not supposed to change the code and hard-code the physical path to the import module - so how can a program from folderA even know to look in folderB for an import ?
How does the magic of __init__.py work ?
I can't visualize it - apparently I am not supposed to change the code
and hard-code the physical path to the import module - so how can a
program from folderA even know to look in folderB for an import ?
You are correct. Somehow you have to tell python to look for imported modules in folderB. There is no __init__.py magic that lets you import from other folders on your hard drive.
Usually, if you've got various different python packages like that, they work by being installed into python's library. That way they can imported from anywhere. This is usually accomplished by a setup.py script. Check if folderB has one. Run it with python setup.py install.
If that doesn't work, we'll need more information about how this code is structured.
Folder B must be on the sys.path, so you would either need to move mmm to A, or modify sys.path from within A (not sure if that works). __init__.py tells python that the folder is a package, so you could have folders with __init__.py within folders with __init__.py and python treats the folders inside as parts of the parent folder. Check out sympy or almost any large python library and you will find such a structure. It can also contain code to be run on import, but can also be empty.
Related
I know that python mechanism doesn't allow doing relative imports without a known parent package, but I want to know which the reason is.
If relative imports worked without known parent package it would make developers life much easier (I think, correct me if I am wrong)
Example:
From a python script a have to import a global definitions file. Right now I have to do it like this:
DEFS_PATH = '../../../'
sys.path.append(DEFS_PATH)
import DEFINITIONS as defs
If I could import this file just like this without having to specify the -m flag when executing the script or creating a __init__.py file that collects all packages. It would make everything much more easier.
from .... import DEFINITIONS as defs
Of course doing this raises the famous import error:
Obviously this is a toy example, but imagine having to repeat this in hundreds of python scripts...
Is there any workaround for importing relative packages without a known parent package that doesn't involve tha hacky ugly way (sys.path.append(...) or python -m myscript )?
I solved this problem but in a different way. I have a folder where I have lots of global functions that I used in different packages, I could create a Python Package of this folder, however I would have to rebuild it each time I changed something.
The solution that fitted me was to add user-packages.pth file in site-packages directory of my current environment, but it could also be added to global site-packages folder. Inside this user-packages.pth I added the absolute path to my directory where all the global utils are. And now I just have to do from any python script
from utils import data_processing as dp
from utils.database import database_connection as dc
Now I don't need to add in each file sys.path.append("path/to/myutils/")
Note:
The .pth file could have any file name (customName.pth) and paths inside the file should be separated by carriage return ("\n"). Also, paths should be absoulte.
For example:
C:\path\to\utils1
C:\path\to\other\utils2
I am trying to understand how import works in jupyter notebook.
My present working directory is "home/username". I have three python modules.
The path names of these modules are as given below.
"/home/username/module1.py"
"/home/username/folder/module2.py"
"/home/username/anaconda3/lib/python3.7/os.py" (which is an inbuilt python module)
Jupyter Notebook:
cell 1:
import module1
Works just fine
cell 2:
import module2 gives
ModuleNotFoundError: No module named 'module2'
cell 3:
import os
Works just fine
It seems like modules in the working directory can be imported without any problem. So, module1.py can be imported. Modules in other directories that are not packages cannot be imported directly. So, module2.py throws an error. But if this is the case how can os.py, which is not the working directory or in another package in the same directory, be imported directly?
This is really more about how python itself works.
You should be able to import module2 with from folder import module2. You should declare /home/username/folder as a package by create a blank init file /home/username/folder/__init__py. I recommend naming the package something more unique, like potrus_folder, that way you don't get naming conflicts down the line.
To explain: Python keeps track of what modules it has available through it's path, it is usually set in your environment variables. To see what folders it looks in for modules you can do import sys then print(sys.path). By default your working directory (/home/username/) will be included, with highest priority (it should thus be either first or last in sys.path, I don't remember). You can add your own folder with sys.path.append('/some/folder'), although it is frowned upon, and you should really add it to your system path, or just keep it as a package in your working directory.
Packages are really just subfolders of paths which have already been added. You access them, as I explained earlier, by using the from X import Y syntax, or if you want to go deeper from X.Z import Y. Remember the __init__.py file.
The path of os library is set in environment*
Whenever you give import it would search all the directories which are added in your environment + the pwd , so you could just add the directory in environment and that would work
By default /home/username/anaconda3/lib/python3.7/ is added by default at the time of installation since there is where most of the module lies, but you can add urs too
First, I need to describe the environment I'm writing for. I'm writing Python code that will be loaded and executed by a Python runtime running within a CAD application. The CAD application uses Python as its scripting engine. As a result, I don't have access to the Python runtime and as a good citizen to all other scripts shouldn't modify any system settings. My script is just one of many that are loaded and running.
This all works fine except when I want to use non-standard libraries. In that case, I need to install a local copy of the library for my script to access. The problem I'm having is that most libraries expect to be installed and added to the sys path which is something I shouldn't do because it could create conflicts with what other scripts are doing. What I'm attempting to do instead is to set up a local copy of the library(s) and then edit their source so their imports are relative and they don't depend on the sys path. That way my program will have its own local copy of the libraries and not depend on anything else and won't disturb any other scripts.
I'm using the -t option of PIP to install Requests and PyOpenSSL into a "Packages" subfolder in my script folder. Here's an abbreviated list of what I have.
RequestsTest/
RequestsTest.py
Packages/
OpenSSL/
cryptography/
x509/
__init__.py
base.py
hazmat/
__init__.py
backends/
__init__.py
interfaces.py
openssl/
__init__.py
backend.py
x509.py
OpenSSL/
__init__.py
SSL.py
Requests/
chardet/
__init__.py
requests/
__init__.py
urllib3/
__init__.py
request.py
contrib/
__init__.py
pyopenssl.py
util/
__init__.py
request.py
ssl_.py
Although it's tedious to track down the various import statements and make then relative, it does seem to work. However, I'm having problems with one particular set of imports.
In Packages/Requests/urllib3/contrib/pyopenssl.py it contains the following imports, which I've modified:
from ....OpenSSL.OpenSSL import SSL
from ....OpenSSL.cryptography import x509
They were originally:
from OpenSSL import OpenSSL.SSL
from cryptography import x509
I get the error "ImportError: No module named 'OpenSSL'" for the first line and "ImportError: No module named 'cryptography'" for the second line. I'm fairly certain the path is correct because if I change the number of dots I get the no module named error but it lists the full path of the what it's trying to load and not just the name of the module.
I would appreciate some help with this specific issue but can also use some overall advice of how to set up and use private copies of libraries. Remember that my program is just one of many that the system is loading changing the system or setting up a virtual environment is not an option.
Check out the localimport module, which seems to be a solution for your particular use case. From the README:
Given your Python script, application or plugin comes with a directory that contains modules for import, you can use localimport to keep the global importer state clean.
app.py
res/modules/
some_package/
__init__.py
# app.py
with localimport('res/modules') as _importer:
import some_package
assert 'some_package' not in sys.modules
The tagline is "Isolated import of Python Modules for embedded applications." so it seems pretty relevant.
When using that module, the following may help keep things neat:
Put your actual script logic into its own file.
Have a wrapper script (which will be the one loaded by the CAD software) which does localimport as mentioned in the README then does a relative import of your module. If your module gets big enough maybe put it into its own package and consume it in the same way as everything else (just do from RequestsTest import * in the body of the with localimport(): ....
Try to have a clear boundary between the source code you write and the final organized set of files required to use that source code in the context of the CAD Python runtime. It is OK to have a build/packaging step that creates the localimport script, downloads the required packages, etc. It's better even because then it is automated and not something that was done manually that someone in the future may have to recreate.
So Im a beginner to python/programming and came upon this code in a tutorial, which Im having trouble understanding.
from pythonds.basic.stack import Stack
What I did was , I went to the site-packages folder in my python directory (which holds all modules). There I could find the directory structure to be : -
pythonds/basic/stack.py
The file stack.py has a "class Stack" inside it.
So am I correct in interpreting/relating the import command to this directory structure ?
Also , whenever such a long chaining of modules happen in python, can it always be understood in such a manner.
In command line, you can do like this:
C:\Python27\Lib>pip intall pythonds
Then this module can work.
Not all the time.
It's probably better to not try and compare the directory structure with the module path, unless you have to debug modules or install them manually.
Sometimes, your PYTHONPATH will be extended to include subdirectories in site-packages, and then there'll be an extra subdirectory.
Other times, there can be an __init__.py file in the pythonds/basic/ directory (there likely is), that can contain
from .stack import Stack
in which case the import path could be
from pythonds.basic import Stack
Your understanding is right.
import pythonds.basic.stack
This will make all the classes in the module accessible by your script. Whereas,
from pythonds.basic.stack import Stack
will make only the Stack class accessible by your script.
My situation is similar to one in this question... The difference is,
In our python/django project, we have a directory called utils, which keeps basic functions...
Sometimes, we need to test some modules by running thm from console like
python myproject/some_module.py
It is all good, until python tries to import something from our utils directory...
from utils.custom_modules import some_function
ImportError: No module named custom_modules
I check my python path, and our project is on the list, each folder under the project file has __init__.py files, and when i run ipython within project directory... Everything is ok, otherwise, python imports from its own utils directory...
My collegues use the sama method without any problem, but it throws ImportError on my environment... What could be the problem that all of us was missing?
UPDATE: My project directory, and each sub-drectory have __init__.py file, and i can import other modules from my project without any problem... When i am in a diffrent folder than my procekt and i run ipython, a such import have no problem...
from someothermodule.submodule imprort blahblahblah
But, when it comes to importing utils, it fails...
UPATE 2: What caused the problem was the utils directory under django folder, which is also in the python path...
See the PEP on absolute and relative imports for the semantics. You probably want
from .utils.custom_modules import some_function
if you're in a file at the top level of your package.
Edit: This can only be done from inside a package. This is for a good reason -- if you're importing something that is part of your project, then you're already treating it like a Python package, and you should actually make it one. You do this by adding an __init__.py file to project directory.
Edit 2: You've completely changed the question. It may be possible to work around the problem, but the correct thing to do is not refer to your packed the same way as a builtin package. You either need to rename utils, or make it a subpackage of another package, so you can refer to it by a non-conflicting name (like from mydjangoapp.utils.custom_modules import some_function).
I'm not going to bother telling you to try and make sure you don't name your own modules after the stdlib modules;
If you want to leave the name like this, you'll have to use something like this in everything that imports your own utils module:
import sys, imp
utils = sys.modules.get('utils')
if not utils: utils = imp.load_module('utils',*imp.find_module('utils/utils'))
However, unless there's a lot of stuff that you'd have to change after renaming it, I'd suggest you rename it.