I am trying to have an easy way to update a program I am working on, by being able to just place replacement .py files in a folder which I then put at the top of the sys.path list so all import statements will first check that folder and not my top level directory, importing the newer .py files and not the ones used to build the program originally with PyInstaller. Right now, if I use this code:
import sys,os
bundlePath = 'c:/tests/importTest'
print ("Bundle path: {}".format(bundlePath))
importPath = bundlePath + '/uploads'
sys.path.insert(0,importPath)
print ("Import path: {}".format(sys.path[0]))
os.chdir(sys.path[0])
import testyTastery
print(testyTastery.message)
My Python IDE or running this script through the command line will print the message from uploads/testyTastery.py (there is also a testyTastery.py in the top level directory). However, when I change line 2 to bundlePath = sys.path[1] (because builds in PyInstaller put the top level directory in sys.path[1], at least in Windows, sys.path[0] points to a base_library.zip that PyInstaller creates) and build this into a folder bundle for PyInstaller (making sure to either copy uploads/testyTastery.py or include it in the .spec file), I get the message from the testyTastery.py script, not the uploads/testyTastery.py script.
Any idea on how to easily change this? To be clear, the first two print statements result in the same output for both running it before and after compiling with PyInstaller, after changing the second line as mentioned.
I don't know, but maybe it has something to do with PyInstaller always looking at files it packages before looking in sys.path for imports?
Edit: To be clear, if I do a build with none of the files available, then put the files in the appropriate folders afterward, the import precedence is just like in the command line / IDE. This suggests to me that files packaged by PyInstaller are higher up in the import hierarchy.
I finally found a solution to this problem. To be clear, the original problem was to package up files (in folder option) with PyInstaller, and later just drop in files to replace the ones that were originally packed up. Because PyInstaller inserts its method of loading files into sys.meta_path, you need to install your own custom import method before PyInstaller's. Here is what I've got (changing the order of system path is because PyInstaller also changes sys.path[0] to a zip file, sys.path[1] is what you really want to be at the top of the sys.path chain if you are doing this or using sys.path[0] in your program:
import os
import sys
sys.path = [sys.path[1]] + [sys.path[0]] + sys.path[2:]
from importlib.machinery import SourceFileLoader
class MyImporter(object):
def find_module(self, module_name, package_path):
print ('entered again')
if os.path.exists("{}/{}.py".format(sys.path[0],module_name)):
return SourceFileLoader(module_name,"{}/{}.py".format(sys.path[0],module_name))
else:
return None
def load_module(self, module_name):
if os.path.exists("{}/{}.py".format(sys.path[0],module_name)):
return SourceFileLoader(module_name,"{}/{}.py".format(sys.path[0],module_name)).load_module()
else:
return None
sys.meta_path.insert(1,MyImporter())
If you do this in your main file that PyInstaller imports then your entire program will search sys.path[0] (It would be trivial to add more folders to your system path and search them as well, of course) before searching for PyInstaller packaged modules.
Related
here is my Maya.env
λ cat C:\Users\roroco\Documents\maya\2018\Maya.env
MAYA_SCRIPT_PATH=C:/Users/roroco/OneDrive/maya/script
MAYA_PLUGIN_IN_PATH=C:/Users/roroco/OneDrive/maya/plugin
I'm sure my script exist:
λ cat C:\Users\roroco\OneDrive\maya\script\ro.py
def init():
print("prpr")
and in maya script editor, my script dir exist in MAYA_PATH_DIR
getenv MAYA_SCRIPT_PATH;
// Result: C:/Users/roroco/Documents/maya/projects/default/scripts;C:/Users/roroco/OneDrive/maya/script;C:/Users/roroco/Documents/maya/2018/scripts;C:/Users/roroco/Documents/maya/scripts;C:/Users/roroco/Documents/maya/2018/presets;C:/Users/roroco/Documents/maya/2018/prefs/shelves;C:/Users/roroco/Documents/maya/2018/prefs/markingMenus...
but when I import ro in script editor, it raise:
# Error: ImportError: file <maya console> line 1: No module named ro #
I hope I can write my maya script and auto sync to onedrive, how should i do
I don't understand maya doc say i can set MAYA_SCRIPT_PATH why it doesn't work
My tmp solution is add my script_dir in maya startup script
in C:\Users\roroco\Documents\maya\scripts\userSetup.py, add following code:
import imp
import os
imp.load_source("", "c:/users/" + os.environ["USERNAME"] + "/OneDrive/maya/script/userSetup.py")
and in my_script_dir/userSetup.py add my_script_dir to sys.path
import sys
import os
ro_script = os.path.dirname(__file__)
if ro_script not in sys.path:
sys.path.insert(0, ro_script)
you should be able to add python paths to your running maya dynamically with sys.path.insert or sys.path.append. You should not need to do anything more elaborate than that; you script should be importable if it's on the sys path.
For what you're trying to do a very common and clean solution is to use maya modules. Modules will allow you to append to the usual search directories, and they support network shares -- so you would add a module (in your maya modules path) and point its scripts directory to your onedrive. Modules can have their own userSetup.pys as well, which is very nice for keeping a clean separation between your code and anything else running on a particular machine.
More on modules here and here. The docs are here
You shouldn't have to add your path dynamically through userSetup.py.
Make sure you include PYTHONPATH in your Maya.env:
PYTHONPATH=C:\Users\roroco\OneDrive\maya\script
Checking my Windows home setup, I'm using back slashes in Maya.env, so you can see if switching helps. Instead of checking MAYA_PATH_DIR, check your sys.path to see if your directory is there. All Python modules need to be in one of these directories to import it:
import sys
for p in sys.path:
print p
You can also copy and paste your path to os.path.exists to confirm it really does exist. Sometimes it's the right path but the slashes make it so it doesn't resolve properly.
I create two python files, and the directory/file relations is as follows:
mytest---
|---mycommon.py
|---myMainDir---
|----myMain.py
In mycommon.py:
def myFunc(a):
...
And in myMain.py:
import sys
sys.path.append(os.path.join(os.path.dirname(os.path.abspath('__file__')), '..'))
import mycommon.py
mycommon.myFunc("abc")
Then I created exe using pyinstaller:
pyinstall.py -F mytest\myMainDir\myMain.py
MyMain.exe is created, but when run, is tells that can not find mycommon module.
PyInstaller's official manual describes this issue:
Some Python scripts import modules in ways that PyInstaller cannot detect: for example, by using the __import__() function with variable data, or manipulating the sys.path value at run time. If your script requires files that PyInstaller does not know about, you must help it.
It also suggests what should be done in such a case:
If Analysis recognizes that a module is needed, but cannot find that module, it is often because the script is manipulating sys.path. The easiest thing to do in this case is to use the --paths= option to list all the other places that the script might be searching for imports:
pyi-makespec --paths=/path/to/thisdir --paths=/path/to/otherdir myscript.py
These paths will be added to the current sys.path during analysis.
Therefore, please specify the --paths argument while building the application. The manual states that specifying the -p argument is equivalent:
-p dir_list, --paths=dir_list
Set the search path(s) for imported modules (like using PYTHONPATH). Use this option to help PyInstaller to search in the right places when your code modifies sys.path for imports. Give one or more paths separated by ; (under Windows) or : (all other platforms), or give the option more than once to give multiple paths to search.
Also I had to fight a bit to get pyinstaller correctly import python scripts in a subfolder where the path to the subfolder was set relatively via sys.path.insert.
The answer by Yoel was correct for me but I needed careful setting of paths in Windows. Here is what I did:
My main py is:
D:\_Development\pCompareDBSync\pCompareDBSync\pCompareDBSync.py
My imported py is:
D:\_Development\pCompareDBSync\pCompareDBSync\py\pCompareNVR.py
(I have many of such imported py's in folder .\py\ but here i just use a single one as example)
So my main PY, I have the following include:
sys.path.insert(0, 'py')
try:
from pCompareNVR import fgetNV_sN_dict
from pCompareNVR import findNVRJobInDBSync
from pCompareNVR import getNVRRecords
from pCompareNVR import saveNVRRecords
from pCompareNVR import compareNVRs
except Exception as e:
print('Can not import files:' + str(e))
input("Press Enter to exit!")
sys.exit(0)
pyinstaller --onefile pCompareDBSync.py
-> pCompareDBSync.exe that did NOT include py/pCompareNVR.py
I had to include the absolute pad to the main PY and the imported PY's:
pyinstaller --onefile --paths=D:\_Development\pCompareDBSync\pCompareDBSync\ --paths=D:\_Development\pCompareDBSync\pCompareDBSync\py pCompareDBSync.py
-> pCompareDBSync.exe that did now include py/pCompareNVR.py -> OK
And that solved this issue for me!
I am having the same issue as OP (and as this comes up a lot in google searches I though I would add my experience).
Similar folder layout, save for a common folder containing mycommon.py in the same location. I am running PyInstaller from myMainDir as part of a CI build step.
I had tried the suggested solutions: setting --paths, declaring the hidden imports in the spec file etc. I still could not get it working.
I ended up 'solving' (read hacking) the problem by adding a step in the build script to copy the common folder into myMainDir before running PyInstaller.
I created a test file called test.py (contained a class named test) and saved it into documents(Mac OS X 10.9.3)
I then attempted to use this file by writing from test import test. However, I got an error telling me that there was no module named test. Please can you shed some light into this issue.
The problem is that the Documents folder isn't usually in the PATH or PYTHONPATH, so the interpreter can't see scripts that are stored there (unless Documents happens to be the current working directory).
I can think of two solutions:
Move the file test.py to somewhere within Python's path.
You can find your PYTHONPATH with this script:
import os
try:
user_paths = os.environ['PYTHONPATH'].split(os.pathsep)
except KeyError:
user_paths = []
(From Mark Ransom’s answer to How do I find out my python path using python?)
You can find the system PATH with
import sys
print sys.path
Both of these return a list of folders. If you put test.py in one of those folders, then it will be seen by the interpreter.
For example, I usually install custom modules at
/Library/Python/2.7/site-packages
and then I can import them as normal.
Manually append Documents to your path.
You can use sys.path.append to add a directory to the path.
If you only need to use test.py in a handful of scripts, then you can add this code to each script you want to use test.py in:
import os
import sys
sys.path.append(os.path.join(os.environ['HOME'], 'Documents'))
from test import test
I have in my python workspace two Modules which need sip.pyd
Module1.pyd needs sip.pyd (which implements v 8.0-8.1)
Module2.pyd needs sip.pyd (another file, that implements v6.0)
So I can't just choose the newer one, it doesn't work: I have to keep them both!
(RuntimeError: the sip module implements API v6.0 but the fbx module requires API v8.1)
How can I import a module in .pyd extension (a python dll, not editable), and specify which sip.pyd to source?
As for a workaround, I manage to do that:
One sip.pyd is in my root site-packages location.
If I have to import the module that need the other sip.pyd, I remove root path form sys.path, and I append the precise folder path where the other sip.pyd are.
I can import my Module and restore previous sys.path.
Assuming you don't have a piece of code needing both files at once. I'd recommend the following:
install both files in 2 separate directories (call them e.g. sip-6.0 and sip-8.0), that you'll place in site-packages/
write a sip_helper.py file with code looking like
sip_helper.py contents:
import sys
import re
from os.path import join, dirname
def install_sip(version='6.0'):
assert version in ('6.0', '8.0'), "unsupported version"
keep = []
if 'sip' in sys.modules:
del sys.modules['sip']
for path in sys.path:
if not re.match('.*sip\d\.\d', path):
keep.append(path)
sys.path[:] = keep # remove other paths
sys.path.append(join(dirname(__file__), 'sip-%s' % version))
put sip_helper.py in site_packages (the parent directory of the sip-6.0 and sip-8.0 directories)
call sip_helper.install_sip at the startup of your programs
VirtualEnv is done to handle those case.
virtualenv is a tool to create isolated Python environments.
Using virtualenv, you will be able to create 2 environements, one with the sip.pyd in version 8.x another in version 6.0
I don't know if that works (if a module's name has to match its contents), but can't you just rename them to sip6.pyd resp. sip8.pyd and then do
if need6:
import sip6 as sip
else:
import sip8 as sip
?
When I use a fixture using a file conftest.py the directory I get through the os.getcwd() is not correct! I get C: \ Program Files \ Java \ jdk1.8.0_60 \ jre \ bin
If I run the test from the command line, everything works correctly
Example:
conftest.py
import os
import pytest
#pytest.fixture()
def curdir():
directory = os.getcwd()
print directory
return directory
PyCharm
I get C:\Program Files\Java\jdk1.8.0_60\jre\bin
Incorrect directory
CMD
I get C:\python\testing\scripts\example
Correct directory
test_simple.py
# coding=utf-8
import json
import os
def test_simple(curdir):
print curdir
Why is this happening?
I’ve never run PyCharm on Windows and I don’t know Windows very well at all, but here’s my guess: PyCharm (a Java application) sets the JRE’s binary directory as the current directory when it is run, and any process spawned by PyCharm (such as the Python interpreter) simply inherits this.
So I wouldn’t necessarily think that what you’re seeing is weird at all.
If you need your current directory to be something specific, you’d probably best force it by using os.chdir().
I found a solution. Instead
os.getcwd()
use
os.path.dirname(os.path.abspath(__file__))