In my IronPython script, I'm using standard libary modules like ConfigParser, logging and JSON.
Then I use pyc.py to create an executable. At first I ran into problems, namely '...ImportException: no module named ...'
since they weren't being included in the exe and accompanying dlls.
So I ran a solution from here: IronPython: EXE compiled using pyc.py cannot import module "os" and it mostly worked.
For example, importing 'ConfigParser' would work since in the IronPython 'Lib' folder as a module, it's there as 'ConfigParser.py'. However I'm still having trouble using JSON and logging since they're inside of folders with their name (packages?).
I'm feeling that I'm just missing something simple, and probably need to read up more on python modules and how they really work, but I'm not sure what I should be looking for.
Any help would be greatly appreciated.
Thanks!
EDIT:
I can't answer my own question yet, so I'll leave this here.
Somehow got it to work in a really 'hacky' way. There must be another much cleaner solution to this that I'm missing (some option in pyc.py?)
Here's what I did:
1) Made the StdLib.dll file generated from the link above (IronPython: EXE compiled using pyc.py cannot import module "os"). This would be missing the std lib packages.
2) Used SharpDevelop to compile the standard lib packages that weren't included in the above dll following the method here: http://community.sharpdevelop.net/blogs/mattward/archive/2010/03/16/CompilingPythonPackagesWithIronPython.aspx
3) Used SharpDevelop to build my program and tie together all the references.
- Reference to the dlls made in step 2
- Reference to the StdLib.dll made in step 1
Again, there must be a better solution to this.
I've found two ways to compile standard library python packages:
1st way: Individually compile each package into a dll
Using pyc.py, run something like (this example compiles logging package):
ipy pyc.py ".\Lib\logging\__init__.py" ".\Lib\logging\config.py" ".\Lib\logging\handlers.py" /target:dll /out:logging
This creates a logging.dll file, which you can then use like this:
import clr
clr.AddReference('StdLib') #from the compilation of non-package std libraries
clr.AddReference('logging')
import logging
**Note: This is assuming you've run the solution from IronPython: EXE compiled using pyc.py cannot import module "os" to create StdLib.dll
2nd way: Modify the compilation script that generated StdLib.dll
I changed this line:
#Build StdLib.DLL
gb = glob.glob(r".\Lib\*.py")
gb.append("/out:StdLib")
To this:
#Build StdLib.DLL
gb1 = glob.glob(r".\Lib\*.py")
gb2 = glob.glob(r".\Lib\*\*.py")
gb3 = glob.glob(r".\Lib\*\*\*.py")
gb = list(set(gb1 + gb2 + gb3))
gb.append("/out:StdLib")
This includes the subfolders in the Lib directory which get missed in the original regex (only modules get included). Now, packages like xml, json, logging, etc. get included into StdLib.dll
Related
I am trying to follow the tutorial for creating python packages from shared objects compiled from C++ via boost::python, but I am running into some problems I need clarification about.
Assume I have a local $install_dir into which I install the compiled shared objects in the form of a python package via CMake. Parallel to the tutorial liked above, my structure is:
$installdir/
my_package/
__init__.py
module/
__init__.py
_module.so
I have added $installdir to my $PYTHONPATH.
$installdir/my_package/__init__.py is empty.
$installdir/my_package/module/__init__.py contains:
from _module import *
When I then import my_package.module I get ModuleNotFoundError: No module named '_module' raised from $installdir/my_package/module/__init__.py.
The issue seems to be that _module.so is not found from $installdir/my_package/module/__init__.py.
Why is the approach from the tutorial not working?
If I add $installdir/my_package/module to $PYTHONPATH directly everything works fine, but it feels like that should not be neccessary, as $installdir/my_package/module/__init__.py should find _module.so locally.
I implemented the following portable workaround for now within $installdir/my_package/module/__init__.py:
import sys, pathlib
sys.path.insert(0,str(pathlib.Path(__file__).parent.absolute()))
from _module import *
Bonus Question:
Changing the file name extension from .so to .pyd breaks the import (ModuleNotFoundError) even without any packaging and .pyd being accessible directly via $PYTHONPATH. I define the extension via CMake's SUFFIX target property. This is obviously mostly cosmetic, but I would still like to understand the reason and how to fix it.
Edit:
This is Ubuntu 20.04 with python 3.8 and boost 1.71
I have been coding in python for about 2 months, but I'm only familiar with basic object-oriented programming, so I do not really understand things like how searching for modules is implemented. (Basically I'm a noob.)
I pip installed a package called Opentrons Opentrons 2.5.2 and all its dependencies into the samefolder as a python script I'm currently writing. However when I tried to import the module below[1], I get an error saying that "Opentrons is not a module". Then, I tried shifting it into the python library because I found out the search path using the pprint module and it seems to work. I was wondering if I can specify the search path from the .py file itself instead of manually printing the search path and putting the file into the library that the script searches for. (Willing to put in images of the directories I put the opentrons package in if it helps.)
[1]
import sys
import pprint
pprint.pprint(search.path)
from opentrons import robot, containers, instruments
Edit: I realise that the fact that I am running all my scripts in a Spyder console located in a python 3.6 environment might be important.
You can try using the __import__ function, or importlib. This should allow you to specify the path.
So I'm trying to compile a python script named "File.py" using this code:
from cx_Freeze import setup, Executable
# Define packages needed(not builtin)
packages = ['scipy' , 'numpy']
options = {
'build_exe': {
'packages': packages,
},
}
setup(name='Monitor',
version = '0.8',
description = 'Monitors stuff.',
options = options,
executables = [Executable("File.py")])
"File.py" only uses scipy and numpy, which I have included in the packages variable. However, after compiling and running File.exe, I get the error: "ModuleNotFoundError: No module named 'scipy.spatial.ckdtree'". I am using python 3.6 and cx_Freeze 6 (which supports python 3.6). How can I get cx_Freeze to include all of the necessary modules?
Try giving the actual address of the library. So find where the scipy and numpy files are and in the packages, include something like this: r"C:\Users\yourname\AppData\Local\Programs\Python\Python36\DLLs\tcl86t.dll",
r"C:\Users\yourname\AppData\Local\Programs\Python\Python36\DLLs\tk86t.dll",
Probably dead, but I'll put in my two cents having encountered a couple issues like this with scipy over the years in combination with cx_Freeze. I had trouble finding documentation of this issue on SO.
First, when you run File.exe, check the full stack trace in the dialog box when you get the error. ModuleNotFoundError is your symptom, but root cause is in which file is attempting to load the module, usually an __init__.py file somewhere in scipy.
Second, go to that file and compare the module name being loaded against the package name and aliases. You are looking for discrepancies in the case of the module name or path to the module. For example, one ckdtree load issue I have come across was __init__.py was attempting to load 'ckdtree', but the module in the distribution was 'cKDTree' or had a capital letter where there wasn't supposed to be one. Changing the case in the file attempting to load the module fixes the issue.
I am trying to package my program using pyinstaller. The code runs fine on windows, and uses SqlAlchemy, OpenCV and pyodbc packages.
I ran pyinstaller to create the executable and tried to run it. I'm getting an error:
ImportError: No module named ConfigParser
now, I reran the same thing and looked at logs from pyinstaller and got a warning:
WARNING: Hidden import "sqlalchemy.sql.functions.func" not found!
along with a few others.
then there was a warning about trying to import ConfigParser in lower and uppercase.
Attempted to add Python module twice with different upper/lowercases: ConfigParser
What might be the issue here?
So, I figured it out. To an extent.
Seems like pyInstaller doesn't deal with SWIG files that well.
In sqlalchemy.utils there's a file called compat.py. It is there to make the module compatible with all versions of python.
for example, in python2.x, there's ConfigParser whereas in py3, it is named configparser
So there is a part in compat.py to deal with it:
if py3:
import configparser
# Some other such import statements
elif py2:
import ConfigParser as configparser
Now, pyinstaller gets stumped here as it just focuses on the import, and hence it tries to import both and fails miserably.
My crude workaround to this involved modifying the compat.py file and retaining only the parts relevant to the python version I have (2.x).
Running pyinstaller again proved to be a success! :)
Although this is all very crude and there's probably something better out there, but I couldn't find anything, so I'm sharing what worked for me.
Let's say I have vtk module in my Python site packages, and from application with own Python distribution I want to access this module.
I tried couple of things like:
import sys
sys.path.append("C:\Python27\Lib\site-packages")
sys.path.append("C:\Python27\Lib\site-packages\vtk")
import vtk
lut = vtk.vtkLookupTable()
but it fails to load module properly:
AttributeError: 'module' object has no attribute 'vtkLookupTable'
If I do same from default Python interpreter all is fine.
Now I thought to make a wrapper of vtk in this application site packages, with simple __init__.py resolving paths, so that when I do import vtk it will hopefully load right thing, but I have no experience with Python packages to try to make this work
To put it simple, how can I wrap module from arbitrary folder, in Python site packages by making folder with same name as referenced package and simple __init__.py file?
Remove these lines:
sys.path.append("C:\Python27\Lib\site-packages")
sys.path.append("C:\Python27\Lib\site-packages\vtk")
The site-packages will already be on your python path. Adding a package/folder within that python path (especially at the first level), will just mess with your imports. How is this vtk package structured?
/path/to/site-packages/
vtk/
__init__.py
vtk.py
In this case, to access a function within vtk:
from vtk import vtk
lut = vtk.vtkLookupTable()
It all comes down to how the folder is arranged. You could also do this:
import vtk
lut = vtk.vtk.vtkLookupTable()
Do not try to hack python importing by creating proxy modules simply because you're not understanding how python importing is working. The error was quite clear. The attribute vtkLookupTable did not exist on whatever it was you imported. You imported the wrong thing. Fix it.
You should very very very very rarely have to manipulate the sys.path manually. When you do have to, you should know that it's the right reason - not to work around something you're not fully understanding.
I had trouble with python paths when I first started with python. It can be frustrating, but coming to understand how it works is necessary. What can help you is something like the following:
import vtk
print dir(vtk)
That will print the attributes of vtk, so you can explore exactly what is in the package or module in cases like this where you think you're importing the right thing.
After re-reading your question, it seems like this is a different python install you're talking about. The answer is to install this package into the other python install, or include this package as a top level import by copying the folder into the root level of your application.
"C:\Python27\Lib\site-packages" is already on your python path. So appending path is unnecessary. Remove:
import sys
sys.path.append("C:\Python27\Lib\site-packages")
sys.path.append("C:\Python27\Lib\site-packages\vtk")
Create a new folder called 'vtk\' in "C:\Python27\Lib\site-packages", then create a new python file named __init__.py in "C:\Python27\Lib\site-packages\vtk" and put your own module vtk.py in this directory.
Using:
import vtk
or
from vtk import vtk
to use your own module.