Related
I am an experienced java enterprise developer but very new to python enterprise development shop. I am currently, struggling to understand why some imports work while others don't.
Some background: Our dev team recently upgraded python from 3.6 to 3.10.5 and following is our package structure
src/
bunch of files (dockerfile, Pipfile, requrirements.txt, shell scripts, etc)
package/
__init__.py
moduleA.py
subpackage1/
__init__.py
moduleX.py
moduleY.py
subpackage2/
__init__.py
moduleZ.py
tests/
__init__.py
test1.py
Now, inside the moduleA.py, I am trying to import subpackage2/moduleZ.py like so
from .subpackage2 import moduleZ
But, I get the error saying
ImportError: attempted relative import with no known parent package
The funny thing is that if I move moduleA.py out of package/ and into src/ then it is able to find everything. I am not sure why is this the case.
I run the moduleA.py by executiong python package/moduleA.py.
Now, I read that maybe there is a problem becasue you have you give a -m parameter if running a module as a script (or something on those lines). But, if I do that, I get the following error:
ModuleNotFoundError: No module names 'package/moduleA.py'
I even try running package1/moduleA and remove the .py, but that does not work either. I can understand why as I technically never installed it ?
All of this happened because apparently, the tests broke and to make it work they added relative imports. They changed the import from "from subpackage2 import moduleZ" to "from .subpackage2 import moduleZ" and the tests started working, but the app started failing.
Any understanding I can get would be much appreciated.
The -m parameter is used with the import name, not the path. So you'd use python3 -m package.moduleA (with . instead of /, and no .py), not python3 -m package/moduleA.py.
That said, it only works if package.moduleA is locatable from one of the roots in sys.path. Shy of installing the package, the simplest way to make it work is to ensure your working directory is src (so package exists in the working directory):
$ cd path/to/src
$ python3 -m package.moduleA
and, with your existing setup, if moduleA.py includes a from .subpackage2 import moduleZ, the import should work; Python knows package.moduleA is a module within package, so it can use a relative import to look for a sibling package to moduleA named subpackage2, and then inside it it can find moduleZ.
Obviously, this is brittle (it only works if you cd to the src root directory before running Python, or hack the path to src in PYTHONPATH, which is terrible hack if the code ever has to be run by anyone else); ideally you make this an installable package, install it (in global site-packages, user site-packages, or within a virtual environment created with the built-in venv module or the third-party virtualenv module), and then your working directory no longer matters (since the site-packages will be part of your sys.path automatically). For simple testing, as long as the working directory is correct (not sure what it was for you), and you use -m correctly (you were using it incorrectly), relative imports will work, but it's not the long term solution.
So first of all - the root importing directory is the directory from which you're running the main script.
This directory by default is the root for all imports from all scripts.
So if you're executing script from directory src you can do such imports:
from package.moduleA import *
from package.subpackage1.moduleX import *
But now in files moduleA and moduleX you need to make imports based on root folder. If you want to import something from module moduleY inside moduleX you need to do:
# this is inside moduleX
from package.subpackage1.moduleY import *
This is because python is looking for modules in specific locations.
First location is your root directory - directory from which you execute your main script.
Second location is directory with modules installed by PIP.
You can check all directories using following:
import sys
for p in sys.path:
print(p)
Now to solve your problem there are couple solutions.
The fast one but IMHO not the best one is to add all paths with submodules to sys.path - list variable with all directories where python is looking for modules.
new_path = "/path/to/application/app/folder/src/package/subpackage1"
if new_path not in sys.path:
sys.path.append(new_path)
Another solution is to use full path for imports in all package modules:
from package.subpackage1.moduleX import *
I think in your case it will be the correct solution.
You can also combine 2 solutions.
First add folders with subpackages to sys.path and use subpackage folders as a root folders for imports. But it's good solution only if you have complex submodule structure. And it's not the best solution if in future you will need to deploy your package as a wheel or share between multiple projects.
A newb where pytest is concerned.
I set up my structure: my_proj\src with the app files under src, and my_proj\tests with the test files under there. Following this book, I then added files like my_proj\setup.py, which looks like this currently:
from setuptools import setup, find_packages
setup(
name='my_proj',
version='0.1.0',
license='proprietary',
...
packages=find_packages(where='src'),
package_dir={'': 'src'},
install_requires=[],
extras_require={},
entry_points={
'console_scripts': [
'py_lib = py_lib.cli:py_lib_cli',
]
},
)
Then, as per the book, I ran
> pip install -e .
... and by some magic beyond my understanding this then set up things so that under the my_proj\tests side of things, the test files were able to import .py files from under my_proj\src.
Then I wanted to add a new .py file to the src files, i.e. the application files, the files under my_proj\src. But the tests are completely unable to see it: "No module named 'xxx'".
I tried pip install -e . again. I also tried pip develop ., and I tried the --no-cache-dir flag. I even tried running setup.py directly, which apparently is not recommended.
Then I found that under my_proj\src there is a directory which has been generated, my_proj.egg-info. One of the files under there is SOURCES.txt, which appears to list all the files in the project. My new .py file under \src is indeed listed there. And yet the test framework apparently can't see it.
Finally I tried uninstalling the thing:
pip uninstall my_proj
...
Proceed (y/n)? y
...
and reinstalling
pip install -e .
Tests still can't see (i.e. import) the new file. Grrr.
Can anyone explain this and tell me how to resolve the problem?
Edit
In answer to the comment: I added a new file, under src\utils: new_file.py, like this:
# -*- coding: utf-8 -*-
conc_fh = False
line_of_stars=50*'*' + '\n'
The test file is my_proj\tests\unit\test_misc_utils.py. I am usually running pytest from my_proj\tests\unit, although I have also tried the parent and grandparent directories. In that file this is the import line:
import utils.new_file
Same fail (for the new file). Then I tried all those possible remedies again.
I also tried deleting things like the my_proj.egg-info directory, and the my_proj\tests.pytest_cache and my_proj\tests\__pycache__ directories.
I can only conclude (provisionally) that something under site-packages in the virtual environment is incapable of updating correctly.
I have even tried pip uninstall pytest and reinstall, followed by pip install -e . again. Even then: NO!
Can someone confirm that adding a file works without problems for them (ideally on a W10 OS)?
NB this is the fail output:
______________________________________ ERROR collecting unit/test_misc_utils.py _______________________________________
ImportError while importing test module 'D:\My documents\software projects\EclipseWorkspace\my_proj\tests\unit\test_misc_utils.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
c:\users\mike\appdata\local\programs\python\python39\lib\importlib\__init__.py:127: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests\unit\test_misc_utils.py:3: in <module>
import utils.new_file
E ModuleNotFoundError: No module named 'utils.new_file'
Later
I completely created a new project, by copying the old one, but setting up an entirely new virtual environment.
I then installed pytest again, and again ran pip install -e ., and then pytest tests/unit from the project directory.
To my utter amazement, once again, these two new files, the ones recently created (and copied in the process of making a new project), and only them, failed to be recognised and could not be imported.
Now I am totally baffled. It would appear that there is something about the nature of these files which is "defective". I've tried looking at the lines in them, for things like spurious Unicode characters. Nothing so far...
Think I finally worked it out.
The question is: what is available (for import) to the modules and files under "my_proj/tests". I believe this is determined by setup.py (as outlined in the referenced book). The most bare-boned version of this is:
from setuptools import setup
setup(
name='my_proj2',
py_modules=['my_proj2']
)
When you then run pip install -e . (note dot), this means that modules under my_proj2/src become available for import. But perhaps not everything. The downloadable files (again for the book) include a module [root]/src/tasks ... and under that is an __init__.py, which explicitly imports various variables and classes from that module, for re-export to the test classes.
This approach (including anything you want to import into tests in __init__.py of an application module) is not the only way of accessing stuff in modules under [root]/src when running tests. Any valid path under [root]/src appears to work. I have no idea why it failed on my previous attempts. Maybe the previous setup.py file (tweaked from the downloaded version) contained something complicated which interfered with things.
I created project and helper modules for it. But some of modules are using each other like worker 1 uses helper1 and helper2 also uses helper1. So I completle confused how I need to import all those modules so so can work standalone (for example I want to debug helper2 out of main script) and they still will be functional. Summarizing - how to correctly import modules so maint_script will work and other modules when using out of main_script. Sorry for my English.
main program dir/
main_script.py
-classes/
|
|--helper1.py
|--helper2.py
-worker_classes/
|
|--worker1.py
At the moment I'am using this constructions in the begging of each script, but I feel that this approach isn't appropriate for python
import os
import sys
sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), 'shell_modules')))
The way I deal with imports inside a project is to install the project in editable mode. This way, all files will be able to locate each other, always starting from your project root directory.
In order to do this, follow these steps:
1) write a setup.py file and add it to your project root folder - it doesn't need much info at all:
# setup.py
from setuptools import setup, find_packages
setup(name='MyPackageName', version='1.0.0', packages=find_packages())
2) install your package in editable mode (ideally from a virtual environment). From a terminal in your project folder, write
$ pip install -e .
Note the dot - this means "install the package from the current directory in editable mode".
3) your files are now able to locate each other, always starting from the project root. To import helper1.py, for example, you write:
from classes import helper1
or alternatively:
from classes.helper1 import foo, bar
This will be true to import helper1.py for any file, no matter where it is located in the project structure.
Like I said, you should use a virtual environment for this, so that pip does not install your package to your main Python installation (which could be messy if your project has many dependencies).
Currently my favorite tool for this is pipenv. When using it, replace the terminal command with
$ pipenv install -e .
So that your project gets added to the Pipfile.
I am a beginner with Python. Before I start, here's my Python folder structure
-project
----src
------model
--------order.py
------hello-world.py
Under src I have a folder named model which has a Python file called order.py which contents follow:
class SellOrder(object):
def __init__(self,genericName,brandName):
self.genericName = genericName
self.brandName = brandName
Next my hello-world.py is inside the src folder, one level above order.py:
import model.order.SellOrder
order = SellOrder("Test","Test")
print order.brandName
Whenever I run python hello-world.py it results in the error
Traceback (most recent call last):
File "hello-world.py", line 1, in <module>
import model.order.SellOrder
ImportError: No module named model.order.SellOrder
Is there anything I missed?
All modules in Python have to have a certain directory structure. You can find details here.
Create an empty file called __init__.py under the model directory, such that your directory structure would look something like that:
.
└── project
└── src
├── hello-world.py
└── model
├── __init__.py
└── order.py
Also in your hello-world.py file change the import statement to the following:
from model.order import SellOrder
That should fix it
P.S.: If you are placing your model directory in some other location (not in the same directory branch), you will have to modify the python path using sys.path.
If it's your root module just add it to PYTHONPATH (PyCharm usually does that)
export PYTHONPATH=$PYTHONPATH:<root module path>
for Docker:
ENV PYTHONPATH="${PYTHONPATH}:<root module path in container>"
you need a file named __init__.py (two underscores on each side) in every folder in the hierarchy, so one in src/ and one in model/. This is what python looks for to know that it should access a particular folder. The files are meant to contain initialization instructions but even if you create them empty this will solve it.
It's easier if you use this code
python3 -m module.sub_module
For example:
python3 -m entrypoint.settings
Just add your project root directory to environment variable: PYTHONPATH.
so for the below project structure, just add Rootdir path(For e.g: add E:\Projects\Rootdir) in PYTHONPATH.
Rootdir
└── pkg2
├── b.py
├── c.py
└── pkg2
├── b.py
├── c.py
You need to make sure the module is installed for all versions of python
You can check to see if a module is installed for python by running:
pip uninstall moduleName
If it is installed, it will ask you if you want to delete it or not. My issue was that it was installed for python, but not for python3. To check to see if a module is installed for python3, run:
python3 -m pip uninstall moduleName
After doing this, if you find that a module is not installed for one or both versions, use these two commands to install the module.
pip install moduleName
python3 -m pip install moduleName
After trying to add the path using:
pip show
on command prompt and using
sys.path.insert(0, "/home/myname/pythonfiles")
and didn't work. Also got SSL error when trying to install the module again using conda this time instead of pip.
I simply copied the module that wasn't found from the path "Mine was in
C:\Users\user\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages
so I copied it to 'C:\Users\user\Anaconda3\Lib\site-packages'
I only use Python as a secondary language and probably made a newbie-error. I had similar problem and my error was calling:
import requests
I got the error
ModuleNotFoundError: No module named 'requests.adapters'; 'requests' is not a package
Turns out the file I created in the same folder named "requests.py" made a conflict. Renaming the file made it work again.
I had same error. For those who run python scripts on different servers, please check if the python path is correctly specified in shebang. For me on each server it was located in different dirs.
If you are using VSCode, what worked for me was I changed the interpreter of my IDE, here is a quick snapshot:
I install my packages through pip3, it appears to be like my Homebrew handles all of the packages I installed previously, so that's the tweak I had to make!!
Another solution depends on where you are running this code from.
If you try running python hello-world.py (from the src directory), you would have to do the following two things for this to work:
Change the import line in hello-world.py to from model.order import SellOrder
If not already there, add to your system environment variable. Key PYTHONPATH, value .
if you are using python 3 then try the below command. I was facing similar issue , this fixed my problem
pip3 install
I solved it by deleting previous python2 and only using python3 which is working fine on windows 10
you need to import the function so the program know what that is here is example:
import os
import pyttsx3
i had the same problem first then i import the function and it work so i would really recommend to try it
I'm having a hard time understanding how module importing works in Python (I've never done it in any other language before either).
Let's say I have:
myapp/__init__.py
myapp/myapp/myapp.py
myapp/myapp/SomeObject.py
myapp/tests/TestCase.py
Now I'm trying to get something like this:
myapp.py
===================
from myapp import SomeObject
# stuff ...
TestCase.py
===================
from myapp import SomeObject
# some tests on SomeObject
However, I'm definitely doing something wrong as Python can't see that myapp is a module:
ImportError: No module named myapp
In your particular case it looks like you're trying to import SomeObject from the myapp.py and TestCase.py scripts. From myapp.py, do
import SomeObject
since it is in the same folder. For TestCase.py, do
from ..myapp import SomeObject
However, this will work only if you are importing TestCase from the package. If you want to directly run python TestCase.py, you would have to mess with your path. This can be done within Python:
import sys
sys.path.append("..")
from myapp import SomeObject
though that is generally not recommended.
In general, if you want other people to use your Python package, you should use distutils to create a setup script. That way, anyone can install your package easily using a command like python setup.py install and it will be available everywhere on their machine. If you're serious about the package, you could even add it to the Python Package Index, PyPI.
The function import looks for files into your PYTHONPATH env. variable and your local directory. So you can either put all your files in the same directory, or export the path typing into a terminal::
export PYTHONPATH="$PYTHONPATH:/path_to_myapp/myapp/myapp/"
exporting path is a good way. Another way is to add a .pth to your site-packages location.
On my mac my python keeps site-packages in /Library/Python shown below
/Library/Python/2.7/site-packages
I created a file called awesome.pth at /Library/Python/2.7/site-packages/awesome.pth and in the file put the following path that references my awesome modules
/opt/awesome/custom_python_modules
You can try
from myapp.myapp import SomeObject
because your project name is the same as the myapp.py which makes it search the project document first
You need to have
__init__.py
in all the folders that have code you need to interact with.
You also need to specify the top folder name of your project in every import even if the file you tried to import is at the same level.
In your first myapp directory ,u can add a setup.py file and add two python code in setup.py
from setuptools import setup
setup(name='myapp')
in your first myapp directory in commandline , use pip install -e . to install the package
pip install on Windows 10 defaults to installing in 'Program Files/PythonXX/Lib/site-packages' which is a directory that requires administrative privileges. So I fixed my issue by running pip install as Administrator (you have to open command prompt as administrator even if you are logged in with an admin account). Also, it is safer to call pip from python.
e.g.
python -m pip install <package-name>
instead of
pip install <package-name>
In my case it was Windows vs Python surprise, despite Windows filenames are not case sensitive, Python import is. So if you have Stuff.py file you need to import this name as-is.
let's say i write a module
import os
my_home_dir=os.environ['HOME'] // in windows 'HOMEPATH'
file_abs_path=os.path.join(my_home_dir,"my_module.py")
with open(file_abs_path,"w") as f:
f.write("print('I am loaded successfully')")
import importlib
importlib.util.find_spec('my_module') ==> cannot find
we have to tell python where to look for the module. we have to add our path to the sys.path
import sys
sys.path.append(file_abs_path)
now importlib.util.find_spec('my_module') returns:
ModuleSpec(name='my_module', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7fa40143e8e0>, origin='/Users/name/my_module.py')
we created our module, we informed python its path, now we should be able to import it
import my_module
//I am loaded successfully
This worked for me:
from .myapp import SomeObject
The . signifies that it will search any local modules from the parent module.
Short Answer:
python -m ParentPackage.Submodule
Executing the required file via module flag worked for me. Lets say we got a typical directory structure as below:
my_project:
| Core
->myScript.py
| Utils
->helpers.py
configs.py
Now if you want to run a file inside a directory, that has imports from other modules, all you need to do is like below:
python -m Core.myscript
PS: You gotta use dot notation to refer the submodules(Files/scripts you want to execute). Also I used python3.9+. So I didnt require neither any init.py nor any sys path append statements.
Hope that helps! Happy Coding!
If you use Anaconda you can do:
conda develop /Path/To/Your/Modules
from the Shell and it will write your path into a conda.pth file into the standard directory for 3rd party modules (site-packages in my case).
If you are using the IPython Console, make sure your IDE (e.g., spyder) is pointing to the right working directory (i.e., your project folder)
Besides the suggested solutions like the accepted answer, I had the same problem in Pycharm, and I didn't want to modify imports like the relative addressing suggested above.
I finally found out that if I mark my src/ (root directory of my python codes) as the source in Interpreter settings, the issue will be resolved.