I create two python files, and the directory/file relations is as follows:
mytest---
|---mycommon.py
|---myMainDir---
|----myMain.py
In mycommon.py:
def myFunc(a):
...
And in myMain.py:
import sys
sys.path.append(os.path.join(os.path.dirname(os.path.abspath('__file__')), '..'))
import mycommon.py
mycommon.myFunc("abc")
Then I created exe using pyinstaller:
pyinstall.py -F mytest\myMainDir\myMain.py
MyMain.exe is created, but when run, is tells that can not find mycommon module.
PyInstaller's official manual describes this issue:
Some Python scripts import modules in ways that PyInstaller cannot detect: for example, by using the __import__() function with variable data, or manipulating the sys.path value at run time. If your script requires files that PyInstaller does not know about, you must help it.
It also suggests what should be done in such a case:
If Analysis recognizes that a module is needed, but cannot find that module, it is often because the script is manipulating sys.path. The easiest thing to do in this case is to use the --paths= option to list all the other places that the script might be searching for imports:
pyi-makespec --paths=/path/to/thisdir --paths=/path/to/otherdir myscript.py
These paths will be added to the current sys.path during analysis.
Therefore, please specify the --paths argument while building the application. The manual states that specifying the -p argument is equivalent:
-p dir_list, --paths=dir_list
Set the search path(s) for imported modules (like using PYTHONPATH). Use this option to help PyInstaller to search in the right places when your code modifies sys.path for imports. Give one or more paths separated by ; (under Windows) or : (all other platforms), or give the option more than once to give multiple paths to search.
Also I had to fight a bit to get pyinstaller correctly import python scripts in a subfolder where the path to the subfolder was set relatively via sys.path.insert.
The answer by Yoel was correct for me but I needed careful setting of paths in Windows. Here is what I did:
My main py is:
D:\_Development\pCompareDBSync\pCompareDBSync\pCompareDBSync.py
My imported py is:
D:\_Development\pCompareDBSync\pCompareDBSync\py\pCompareNVR.py
(I have many of such imported py's in folder .\py\ but here i just use a single one as example)
So my main PY, I have the following include:
sys.path.insert(0, 'py')
try:
from pCompareNVR import fgetNV_sN_dict
from pCompareNVR import findNVRJobInDBSync
from pCompareNVR import getNVRRecords
from pCompareNVR import saveNVRRecords
from pCompareNVR import compareNVRs
except Exception as e:
print('Can not import files:' + str(e))
input("Press Enter to exit!")
sys.exit(0)
pyinstaller --onefile pCompareDBSync.py
-> pCompareDBSync.exe that did NOT include py/pCompareNVR.py
I had to include the absolute pad to the main PY and the imported PY's:
pyinstaller --onefile --paths=D:\_Development\pCompareDBSync\pCompareDBSync\ --paths=D:\_Development\pCompareDBSync\pCompareDBSync\py pCompareDBSync.py
-> pCompareDBSync.exe that did now include py/pCompareNVR.py -> OK
And that solved this issue for me!
I am having the same issue as OP (and as this comes up a lot in google searches I though I would add my experience).
Similar folder layout, save for a common folder containing mycommon.py in the same location. I am running PyInstaller from myMainDir as part of a CI build step.
I had tried the suggested solutions: setting --paths, declaring the hidden imports in the spec file etc. I still could not get it working.
I ended up 'solving' (read hacking) the problem by adding a step in the build script to copy the common folder into myMainDir before running PyInstaller.
Related
I'm trying to import python modules within the same directory using PyCharm which should be a very easy task. I've been struggling with this bug for a while and since I had to finish projects I regularly used the 'SourceFileLoader' from 'importlib.machinery' given the related base directory.
Since this method is quite inefficient for sharing my codes I'd like to finally deal with the problem for good.
Here one example:
Directory
C:\Users\Hallo.DESKTOP-4OVSO14\PycharmProjects\Uni\Tello_Video
contains files:
__init__.py
main.py
tello.py
tello_control_ui.py
main.py contains following code:
import tello from tello_control_ui
import TelloUI
def main():
drone = tello.Tello('', 8889)
vplayer = TelloUI(drone,"./img/")
# start the Tkinter mainloop
vplayer.root.mainloop()
if __name__ == "__main__":
main()
which produces error message:
No module named 'tello'
If i change the import lines of this script to find the overall directory 'Tello_Video' it does the trick.
from Tello_Video import tello, tello_control_ui
Still this solution is very unconvenient since a lot of open source scripts are using the direct way for scripts in the same directory.
I'm running Python 3.7 on Windows 10 using Pycharm. I've read other question concerning this matter and already tried following methods:
1.) update package setuptools
2.) creating overall directory (in this example /Tello_Video) as 'Python Package'
3.) execute Pycharm in the same directory (/Tello_Video) where python files are stored
(4.) __init__.py is empty)
Thank you for any advice!
I am trying to have an easy way to update a program I am working on, by being able to just place replacement .py files in a folder which I then put at the top of the sys.path list so all import statements will first check that folder and not my top level directory, importing the newer .py files and not the ones used to build the program originally with PyInstaller. Right now, if I use this code:
import sys,os
bundlePath = 'c:/tests/importTest'
print ("Bundle path: {}".format(bundlePath))
importPath = bundlePath + '/uploads'
sys.path.insert(0,importPath)
print ("Import path: {}".format(sys.path[0]))
os.chdir(sys.path[0])
import testyTastery
print(testyTastery.message)
My Python IDE or running this script through the command line will print the message from uploads/testyTastery.py (there is also a testyTastery.py in the top level directory). However, when I change line 2 to bundlePath = sys.path[1] (because builds in PyInstaller put the top level directory in sys.path[1], at least in Windows, sys.path[0] points to a base_library.zip that PyInstaller creates) and build this into a folder bundle for PyInstaller (making sure to either copy uploads/testyTastery.py or include it in the .spec file), I get the message from the testyTastery.py script, not the uploads/testyTastery.py script.
Any idea on how to easily change this? To be clear, the first two print statements result in the same output for both running it before and after compiling with PyInstaller, after changing the second line as mentioned.
I don't know, but maybe it has something to do with PyInstaller always looking at files it packages before looking in sys.path for imports?
Edit: To be clear, if I do a build with none of the files available, then put the files in the appropriate folders afterward, the import precedence is just like in the command line / IDE. This suggests to me that files packaged by PyInstaller are higher up in the import hierarchy.
I finally found a solution to this problem. To be clear, the original problem was to package up files (in folder option) with PyInstaller, and later just drop in files to replace the ones that were originally packed up. Because PyInstaller inserts its method of loading files into sys.meta_path, you need to install your own custom import method before PyInstaller's. Here is what I've got (changing the order of system path is because PyInstaller also changes sys.path[0] to a zip file, sys.path[1] is what you really want to be at the top of the sys.path chain if you are doing this or using sys.path[0] in your program:
import os
import sys
sys.path = [sys.path[1]] + [sys.path[0]] + sys.path[2:]
from importlib.machinery import SourceFileLoader
class MyImporter(object):
def find_module(self, module_name, package_path):
print ('entered again')
if os.path.exists("{}/{}.py".format(sys.path[0],module_name)):
return SourceFileLoader(module_name,"{}/{}.py".format(sys.path[0],module_name))
else:
return None
def load_module(self, module_name):
if os.path.exists("{}/{}.py".format(sys.path[0],module_name)):
return SourceFileLoader(module_name,"{}/{}.py".format(sys.path[0],module_name)).load_module()
else:
return None
sys.meta_path.insert(1,MyImporter())
If you do this in your main file that PyInstaller imports then your entire program will search sys.path[0] (It would be trivial to add more folders to your system path and search them as well, of course) before searching for PyInstaller packaged modules.
I created a test file called test.py (contained a class named test) and saved it into documents(Mac OS X 10.9.3)
I then attempted to use this file by writing from test import test. However, I got an error telling me that there was no module named test. Please can you shed some light into this issue.
The problem is that the Documents folder isn't usually in the PATH or PYTHONPATH, so the interpreter can't see scripts that are stored there (unless Documents happens to be the current working directory).
I can think of two solutions:
Move the file test.py to somewhere within Python's path.
You can find your PYTHONPATH with this script:
import os
try:
user_paths = os.environ['PYTHONPATH'].split(os.pathsep)
except KeyError:
user_paths = []
(From Mark Ransom’s answer to How do I find out my python path using python?)
You can find the system PATH with
import sys
print sys.path
Both of these return a list of folders. If you put test.py in one of those folders, then it will be seen by the interpreter.
For example, I usually install custom modules at
/Library/Python/2.7/site-packages
and then I can import them as normal.
Manually append Documents to your path.
You can use sys.path.append to add a directory to the path.
If you only need to use test.py in a handful of scripts, then you can add this code to each script you want to use test.py in:
import os
import sys
sys.path.append(os.path.join(os.environ['HOME'], 'Documents'))
from test import test
I'm having a hard time understanding how module importing works in Python (I've never done it in any other language before either).
Let's say I have:
myapp/__init__.py
myapp/myapp/myapp.py
myapp/myapp/SomeObject.py
myapp/tests/TestCase.py
Now I'm trying to get something like this:
myapp.py
===================
from myapp import SomeObject
# stuff ...
TestCase.py
===================
from myapp import SomeObject
# some tests on SomeObject
However, I'm definitely doing something wrong as Python can't see that myapp is a module:
ImportError: No module named myapp
In your particular case it looks like you're trying to import SomeObject from the myapp.py and TestCase.py scripts. From myapp.py, do
import SomeObject
since it is in the same folder. For TestCase.py, do
from ..myapp import SomeObject
However, this will work only if you are importing TestCase from the package. If you want to directly run python TestCase.py, you would have to mess with your path. This can be done within Python:
import sys
sys.path.append("..")
from myapp import SomeObject
though that is generally not recommended.
In general, if you want other people to use your Python package, you should use distutils to create a setup script. That way, anyone can install your package easily using a command like python setup.py install and it will be available everywhere on their machine. If you're serious about the package, you could even add it to the Python Package Index, PyPI.
The function import looks for files into your PYTHONPATH env. variable and your local directory. So you can either put all your files in the same directory, or export the path typing into a terminal::
export PYTHONPATH="$PYTHONPATH:/path_to_myapp/myapp/myapp/"
exporting path is a good way. Another way is to add a .pth to your site-packages location.
On my mac my python keeps site-packages in /Library/Python shown below
/Library/Python/2.7/site-packages
I created a file called awesome.pth at /Library/Python/2.7/site-packages/awesome.pth and in the file put the following path that references my awesome modules
/opt/awesome/custom_python_modules
You can try
from myapp.myapp import SomeObject
because your project name is the same as the myapp.py which makes it search the project document first
You need to have
__init__.py
in all the folders that have code you need to interact with.
You also need to specify the top folder name of your project in every import even if the file you tried to import is at the same level.
In your first myapp directory ,u can add a setup.py file and add two python code in setup.py
from setuptools import setup
setup(name='myapp')
in your first myapp directory in commandline , use pip install -e . to install the package
pip install on Windows 10 defaults to installing in 'Program Files/PythonXX/Lib/site-packages' which is a directory that requires administrative privileges. So I fixed my issue by running pip install as Administrator (you have to open command prompt as administrator even if you are logged in with an admin account). Also, it is safer to call pip from python.
e.g.
python -m pip install <package-name>
instead of
pip install <package-name>
In my case it was Windows vs Python surprise, despite Windows filenames are not case sensitive, Python import is. So if you have Stuff.py file you need to import this name as-is.
let's say i write a module
import os
my_home_dir=os.environ['HOME'] // in windows 'HOMEPATH'
file_abs_path=os.path.join(my_home_dir,"my_module.py")
with open(file_abs_path,"w") as f:
f.write("print('I am loaded successfully')")
import importlib
importlib.util.find_spec('my_module') ==> cannot find
we have to tell python where to look for the module. we have to add our path to the sys.path
import sys
sys.path.append(file_abs_path)
now importlib.util.find_spec('my_module') returns:
ModuleSpec(name='my_module', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7fa40143e8e0>, origin='/Users/name/my_module.py')
we created our module, we informed python its path, now we should be able to import it
import my_module
//I am loaded successfully
This worked for me:
from .myapp import SomeObject
The . signifies that it will search any local modules from the parent module.
Short Answer:
python -m ParentPackage.Submodule
Executing the required file via module flag worked for me. Lets say we got a typical directory structure as below:
my_project:
| Core
->myScript.py
| Utils
->helpers.py
configs.py
Now if you want to run a file inside a directory, that has imports from other modules, all you need to do is like below:
python -m Core.myscript
PS: You gotta use dot notation to refer the submodules(Files/scripts you want to execute). Also I used python3.9+. So I didnt require neither any init.py nor any sys path append statements.
Hope that helps! Happy Coding!
If you use Anaconda you can do:
conda develop /Path/To/Your/Modules
from the Shell and it will write your path into a conda.pth file into the standard directory for 3rd party modules (site-packages in my case).
If you are using the IPython Console, make sure your IDE (e.g., spyder) is pointing to the right working directory (i.e., your project folder)
Besides the suggested solutions like the accepted answer, I had the same problem in Pycharm, and I didn't want to modify imports like the relative addressing suggested above.
I finally found out that if I mark my src/ (root directory of my python codes) as the source in Interpreter settings, the issue will be resolved.
I have two directories in my project:
project/
src/
scripts/
"src" contains my polished code, and "scripts" contains one-off Python scripts.
I would like all the scripts to have "../src" added to their sys.path, so that they can access the modules under the "src" tree. One way to do this is to write a scripts/__init__.py file, with the contents:
scripts/__init__.py:
import sys
sys.path.append("../src")
This works, but has the unwanted side-effect of putting all of my scripts in a package called "scripts". Is there some other way to get all my scripts to automatically call the above initialization code?
I could just edit the PYTHONPATH environment variable in my .bashrc, but I want my scripts to work out-of-the-box, without requiring the user to fiddle with PYTHONPATH. Also, I don't like having to make account-wide changes just to accommodate this one project.
Even if you have other plans for distribution, it might be worth putting together a basic setup.py in your src folder. That way, you can run setup.py develop to have distutils put a link to your code onto your default path (meaning any changes you make will be reflected in-place without having to "reinstall", and all modules will "just work," no matter where your scripts are). It'd be a one-time step, but that's still one more step than zero, so it depends on whether that's more trouble than updating .bashrc. If you use pip, the equivalent would be pip install -e /path/to/src.
The more-robust solution--especially if you're going to be mirroring/versioning these scripts on several developers' machines--is to do your development work inside a controlled virtual environment. It turns out virtualenv even has built-in support for making your own bootstrap customizations. It seems like you'd just need an after_install() hook to either tweak sitecustomize, run pip install -e, or add a plain .pth file to site-packages. The custom bootstrap could live in your source control along with the other scripts, and would need to be run once for each developer's setup. You'd also have the normal benefits of using virtualenv (explicit dependency versioning, isolation from system-wide configuration, and standardization between disparate machines, to name a few).
If you really don't want to have any setup steps whatsoever and are willing to only run these scripts from inside the 'project' directory, then you could plop in an __init__.py as such:
project/
src/
some_module.py
scripts/
__init__.py # special "magic"
some_script.py
And these are what your files could look like:
# file: project/src/some_module.py
print("importing %r" % __name__)
def some_function():
print("called some_function() inside %s" % __name__)
--------------------------------------------------------
# file: project/scripts/some_script.py
import some_module
if __name__ == '__main__':
some_module.some_function()
--------------------------------------------------------
# file: project/scripts/__init__.py
import sys
from os.path import dirname, abspath, join
print("doing magic!")
sys.path.insert(0, join(dirname(dirname(abspath(__file__))), 'src'))
Then you'd have to run your scripts like so:
[~/project] $ python -m scripts.some_script
doing magic!
importing 'some_module'
called some_function() inside some_module
Beware! The scripts can only be called like this from inside project/:
[~/otherdir] $ python -m scripts.some_script
ImportError: no module named scripts
To enable that, you're back to editing .bashrc, or using one of the options above. The last option should really be a last resort; as #Simon said, you're really fighting the language at that point.
If you want your scripts to be runnable (I assume from the command line), they have to be on the path somewhere.
Something sounds odd about what you're trying to do though. Can you show us an example of exactly what you're trying to accomplish?
You can add a file called 'pathHack.py' in the project dir and put something like this into it:
import os
import sys
pkgDir = os.path.dirname(__file__)
sys.path.insert(os.path.join(pkgDir, 'scripts')
Then, in a python file in your project dir, start by:
import pathHack
And now you can import stuff from the scripts dir without the 'scripts.' prefix. If you have only one file in this directory, and you don't care about hiding this kind of thing, you may inline this snippet.