Created a egg and whl file of pyarrow and put this on s3, for call this in pythonshell job. Received this message:
Job code:
import pyarrow
raise
Error, same structure for whl:
Traceback (most recent call last):
File "/tmp/runscript.py", line 118, in <module>
runpy.run_path(temp_file_path, run_name='__main__')
File "/usr/local/lib/python3.6/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/usr/local/lib/python3.6/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/tmp/glue-python-scripts-e67xuz2j/genos.py", line 1, in <module>
File "/glue/lib/installation/kanna-0.1-py3.6.egg/pyarrow/__init__.py", line 49, in <module>
from pyarrow.lib import cpu_count, set_cpu_count
ModuleNotFoundError: No module named 'pyarrow.lib'
PD: Cannot found the lib.py or lib folder in local files.
I was having the same problem with AWS Lambda and came across this question.
For Glue, AWS docs state only pure python libraries can be used.
For Lambda:
The underlying problem is that modules like pyarrow port their code from C/ C++. When you check pyarrow codebase, you will find in fact two pyarrow.lib files exist, but they have .pyx and .pxd file extensions. This is not pure Python code and therefore depends on underlying CPU architecture.
I had to manually download .whl files for my required version for pyarrow and its dependency numpy. From http://pypi.org/project/pyarrow/, click on Download files and search for your matching version. cp39 means cpython 3.9. and x86 represents the CPU architecture. Follow the same steps for Numpy. I ended up downloading these files: pyarrow-8.0.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl and numpy-1.22.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.
You then have to unzip them and create an archive where both sit together in a folder named Python. This folder can be used to create a layer in Lambda. Attach this layer to your project and import pyarrow should work.
The other solution is to use custom Docker images. This worked for me as well. I believe the AWS docs are exhaustive on that topic. I have written a PoC and all the steps that I followed here.
I followed this guide for creating a pyarrow layer.
pyarrow won't work as it is with glue as it needs support for C and glue doesn't support it.
What you could do is try installing the library on local machine and creating a package manually, then using that egg file.
That worked for my colleage, haven't tested personally.
Related
my issue is that i've been trying to make an exe file with a .py that I created, but it doesn't work. I executed the converted .exe file from the console to see the log and I got this:
Traceback (most recent call last):
File "professors.py", line 10, in <module>
File "PyInstaller\loader\pyimod03_importers.py", line 531, in exec_module
File "docx2pdf\__init__.py", line 13, in <module>
File "importlib\metadata.py", line 551, in version
File "importlib\metadata.py", line 524, in distribution
File "importlib\metadata.py", line 187, in from_name
importlib.metadata.PackageNotFoundError: docx2pdf
[6940] Failed to execute script professors
I've been reading about the issue and it seems that pyinstaller doesn't recognize the docx2pdf import. I read in a website that I have to add it in a folder called site-packages, but it is supposed to have another folder inside called pyInstaller, which is not there. I also tried adding the folder manually but it still didn't work. I also tried with the --hidden-imports method but im not sure how to exactly use it.
I will have to apply this to other imports that I have also.
Any ideas? Thanks in advance!
I had a identical issue. I removed lines 7-13 from the docx2pdf module:
try:
# 3.8+
from importlib.metadata import version
except ImportError:
from importlib_metadata import version
__version__ = version(__package__)
and line 119...
print(__version__)
After that, it worked. Modifying the module probably isn't the best solution, and of course I'd recommend backing up the module before modifying it, but I hope it works for you too.
I have been trying to deploy a Cloud Function with some private dependencies (pyodbc) as I couldn't get it working thru requirements.txt. Please note, I don't want to use Docker here. So all I have built here is below files,
1. main.py
2. process.py ( this one use pyodbc to connect to teradata)
3. libs (folder)
3.1 pyodbc-4.0.30.dist-info (package)
3.2 pyodbc (python extension module)
3.3 __init.py__ ( this is to make this folder as module)
4.requirements.txt
I also updated process.py file to import pyodbc module as below,
import libs.pyodbc
Please note: I used GCP docs to install the pyodbc package and put in libs using https://cloud.google.com/functions/docs/writing/specifying-dependencies-python
On top this, I am also requirements.txt to import as default.
But I am still getting module error as below.
Error message: Code in file main.py can't be loaded.
Did you list all required modules in requirements.txt?
Detailed stack trace: Traceback (most recent call last):
File "/env/local/lib/python3.7/site-packages/google/cloud/functions/worker.py", line 305, in check_or_load_user_function
_function_handler.load_user_function()
File "/env/local/lib/python3.7/site-packages/google/cloud/functions/worker.py", line 184, in load_user_function
spec.loader.exec_module(main)
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/user_code/main.py", line 9, in <module>
from process import process
File "/user_code/process.py", line 6, in <module>
import libs.pyodbc
ModuleNotFoundError: No module named 'libs.pyodbc'
Any leads or help from here is really appreciated. All I am trying to achieve here is, Read CSV files from GCP bucket and process it thru dataframe which loads into teradata and generate output file back into another GCP bucket. I am trying to achieve all with Cloud Functions only. Thank you
The pyodbc project might be a bit of a special case here, because:
The project requires some platform-specific code;
They haven't published source distributions for their latest release (only built distributions).
Here's what I did to get this to work. Starting with an empty libs directory, first download the latest available source distribution:
$ pip download pyodbc --no-binary :all:
Make a directory for the module:
$ mkdir libs/pyodbc
Untar the source distribution into the module:
$ tar xf pyodbc-4.0.28.tar.gz -C libs/pyodbc
Then, in the function you can do:
import libs.pyodbc
My helloworld code doesnt run on google app engine...
I use python 2.7.6 and google app engine version 1.8.8.
Here is what I get:
Traceback (most recent call last):
File "/home/yash/Google/google_appengine/old_dev_appserver.py", line 196, in <module>
run_file(__file__, globals())
File "/home/yash/Google/google_appengine/old_dev_appserver.py", line 192, in run_file
execfile(script_path, globals_)
File "/home/yash/Google/google_appengine/google/appengine/tools/dev_appserver_main.py", line 173, in <module>
from google.appengine.tools import appcfg
File "/home/yash/Google/google_appengine/google/appengine/tools/appcfg.py", line 74, in <module>
from google.appengine.tools import appengine_rpc
File "/home/yash/Google/google_appengine/google/appengine/tools/appengine_rpc.py", line 28, in <module>
import fancy_urllib
File "/home/yash/Google/google_appengine/lib/fancy_urllib/fancy_urllib/__init__.py", line 354, in <module>
class FancyHTTPSHandler(urllib2.HTTPSHandler):
AttributeError: 'module' object has no attribute 'HTTPSHandler'</i>
well...problem solved...i guess few libraries werent installed...compiled python 5-6 times after each library installation...
the problem wasnt in the GAE...it was the lack of required python modules
m still a noob...
Thanks for your time and help :)
Looks like you don't have python support for SSL.
Try this
pip install ssl
Read python-ssl docs for details.
From the docs
If you want to use native python ssl, you must enable it using the
libraries configuration in your application's app.yaml file where you
specify the library name "ssl", as shown here:
libraries:
- name: ssl
version: latest
Does your app.yaml contain this?
Your google appengine environment probably isn't installed properly. I'd recommend reinstalling the SDK and checking for errors in the installation.
If you're using something like virtualenv, it could be affecting the way your install works.
I have a python application primarily dependent on matplotlib, tkinter, numpy and scipy I have been distributing for multiple operating systems. I build these on top of the python.org version of python 2.6. For mac, I have had good success in the past with py2app and pyinstaller to build standalone versions of my application. Recently, however, I have included functions that require patsy, pandas and igraph and have had to move to the mac distributed version of python 2.6 in order to get igraph to work. All works fine in building a py2app version of my software on the Mac for my OS (10.6) but when I run on 10.8 I encountered the following error:
Traceback (most recent call last):
File "UI.pyc", line 603 createHeatMap
File "clustering.pyc", line 31, in <module>
File "mpl_toolkits/__init__.pyc", line 2, in <module>
File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/pkg_resources.py", line 698, in <module>
File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/pkg_resources.py", line 701, in Environment
File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/pkg_resources.py", line 96, in get_supported_platform
File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/pkg_resources.py", line 222, in get_build_platform
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/distutils/util.py", line 98, in get_platform
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/distutils/sysconfig.py", line 537, in get_config_vars
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/distutils/sysconfig.py", line 405, in _init_posix
DistutilsPlatformError: invalid Python installation: unable to open /user/include/python2.6/pyconfig.h (No such file or directory)
This is fixed by saving the file pyconfig.h to the designated location on the destination machine (not present in 10.8 but present in 10.6), which allows my application to work fine. In my py2app pacakge directory, under "Contents/Resources" there is a sym-link to the directory "include" which points to: /System/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6/pyconfig.h. If I include the pyconfig.h file as an addition resource in my py2app setup file, it also includes it in the package, but it still is not recognized.
Can anyone suggest a good work around for this that doesn't require me to use the python.org version of python (I've had horrible luck getting igraph to work with this).
Thanks...
This is a known issue in py2app, see https://bitbucket.org/ronaldoussoren/py2app/issue/36/py2app-fails-to-build-a-working-executable. The pyconfig.h file is only present on machines where the developer tools are installed, which is why I haven't noticed the problem myself.
The easiest workaround is probably to replace the pyconfig.h symlink by a copy of the file after creating the app bundle.
I have created a simple program which uses pywin32. I want to deploy it as an executable, so I py2exe'd it. I also didn't want a huge amount of files, so I set bundle_files to 1 (meaning bundle everything together). However, when I attempt running it, I get:
Traceback (most recent call last):
File "pshelper.py", line 4, in <module>
File "zipextimporter.pyc", line 82, in load_module
File "win32.pyc", line 8, in <module>
File "zipextimporter.pyc", line 98, in load_module
ImportError: MemoryLoadLibrary failed loading win32ui.pyd
In my setup script, I tried doing packages=["win32ui"] and includes=["win32ui"] as options, but that didn't help. How can I get py2exe to include win32ui.pyd?
I don't have this problem if I don't ask it to bundle the files, so I can do that, for now, but I'd like to know how to get it to work properly.
Are you sure that the problem is that win32ui.pyd is not included? The stack trace isn't exactly the same as noted in the wiki, but check this out: http://www.py2exe.org/index.cgi/Py2exeAndWin32ui.
The work-around that has worked best so far is to simply re-implement the pywin32 functions using ctypes. That doesn't require another .pyd or .dll file so the issue is obviated.
Do care to try out PyInstaller? I've used it both on Windows 7 and Ubuntu 10.04 and it worked like magic, even when I compiled to .pyd on Windows.
I've been able to bundle all sorts of applications that I've developed with it.
I have the same problem trying to bundle psutil with py2exe. Here is what I found so far.
Traceback (most recent call last):
File "wx_gui.py", line 43, in <module>
File "zipextimporter.pyc", line 82, in load_module
File "psutil\__init__.pyc", line 85, in <module>
File "zipextimporter.pyc", line 82, in load_module
File "psutil\_psmswindows.pyc", line 15, in <module>
File "zipextimporter.pyc", line 98, in load_module
ImportError: MemoryLoadLibrary failed loading _psutil_mswindows.pyd
I get this traceback when bundle_files = 3. zipfile may be specified or may be None, I still get the problem.
First, I thought this was a missing dll because of this page:
http://www.py2exe.org/index.cgi/ProblemsToBeFixed
I've copied all the dlls I found in Python27 into the same directory as the executable and added that directory path to os.environ['path']. That didn't worked.
Then I tried to import my package directly from site-packages.
I've replaced the whole sys.path of my compiled executable with my normal sys.path
sys.path = [r'C:\Python27\Lib\idlelib', ...]
I think the .pyd module got imported but Visual c++ threw me this really ugly error message:
Runtime Error!
Program: {path}.exe
This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.
I suggest you try to replace the whole sys.path to see if it is working. If it is, you could always make your single executable write the necessary module into a temp directory and add it to your path. If not, I feel like this problem is going to be hard to solve.