"Can't extract file(s) to egg cache" error in GAE - python

I have Windows 7 and Python 2.7 with setuptools installed.
After i download pitz module (easy_install pitz), Google App Engine stop working:
bad runtime process port ['']
Traceback (most recent call last):
File "G:\Program Files (x86)\Google\google_appengine\_python_runtime.py", line 184, in <module>
_run_file(__file__, globals())
File "G:\Program Files (x86)\Google\google_appengine\_python_runtime.py", line 180, in _run_file
execfile(script_path, globals_)
File "G:\Program Files (x86)\Google\google_appengine\google\appengine\tools\devappserver2\python\runtime.py", line 28, in <module>
from google.appengine.ext.remote_api import remote_api_stub
File "G:\Program Files (x86)\Google\google_appengine\google\appengine\ext\remote_api\remote_api_stub.py", line 75, in <module>
import yaml
File "G:\Program Files (x86)\Google\google_appengine\lib\yaml-3.10\yaml\__init__.py", line 14, in <module>
from cyaml import *
File "G:\Program Files (x86)\Google\google_appengine\lib\yaml-3.10\yaml\cyaml.py", line 5, in <module>
from _yaml import CParser, CEmitter
File "C:\Python27\lib\site-packages\pyyaml-3.11-py2.7-win-amd64.egg\_yaml.py", line 7, in <module>
File "C:\Python27\lib\site-packages\pyyaml-3.11-py2.7-win-amd64.egg\_yaml.py", line 4, in __bootstrap__
File "C:\Python27\lib\site-packages\pkg_resources.py", line 950, in resource_filename
self, resource_name
File "C:\Python27\lib\site-packages\pkg_resources.py", line 1607, in get_resource_filename
self._extract_resource(manager, self._eager_to_zip(name))
File "C:\Python27\lib\site-packages\pkg_resources.py", line 1667, in _extract_resource
manager.extraction_error()
File "C:\Python27\lib\site-packages\pkg_resources.py", line 996, in extraction_error
raise err
pkg_resources.ExtractionError: Can't extract file(s) to egg cache
The following error occurred while trying to extract file(s) to the Python egg
cache:
[Error 5] : 'C:\\Users\\Kostr\\AppData\\Roaming\\Python-Eggs\\pyyaml-3.11-py2.7-win-amd64.egg-tmp\\_yaml.pyd'
The Python egg cache directory is currently set to:
C:\Users\Kostr\AppData\Roaming\Python-Eggs
Perhaps your account does not have write access to this directory? You can
change the cache directory by setting the PYTHON_EGG_CACHE environment
variable to point to an accessible directory.
How to solve this issue?

from: https://code.google.com/p/modwsgi/wiki/ApplicationIssues
To avoid this particular problem you can set the PYTHON_EGG_CACHE cache environment variable at the start of the WSGI application script file. The environment variable should be set to a directory which is owned and/or writable by the user that Apache runs as.
import os
os.environ['PYTHON_EGG_CACHE'] = '/usr/local/pylons/python-eggs'
Again, make sure this exists. For Windows users, maybe something like:
os.environ['PYTHON_EGG_CACHE'] = '/tmp'
Alternatively, if using mod_wsgi 2.0, one could also use the WSGIPythonEggs directive for applications running in embedded mode, or the python-eggs option to the WSGIDaemonProcess directive when using daemon mode.
Note that you should refrain from ever using directories or files which have been made writable to anyone as this could compromise security. Also be aware that if hosting multiple applications under the same web server, they will all run as the same user and so it will be possible for each to both see and modify each others files. If this is an issue, you should host the applications on different web servers running as different users or on different systems. Alternatively, any data required or updated by the application should be hosted in a database with separate accounts for each application.

Sounds like you don't have permissions to write to the directories. Uninstall pitz, and reinstall using the -Z tag to install it as the unzipped version. Then, you don't need the cache directory to extract each time. That should solve the directories / permissions issue.

Related

can't get appengine python sample for flask or django to work

I have downloaded the appengine samples.
When I enter directory python-docs-samples/appengine/standard/django and install it on my appengine project, and then look at the project web page, I get this error:
Traceback (most recent call last):
File "/base/alloc/tmpfs/dynamic_runtimes/python27/b93f87cfaa0115e5_unzipped/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 240, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "/base/alloc/tmpfs/dynamic_runtimes/python27/b93f87cfaa0115e5_unzipped/python27_lib/versions/1/google/appengine/api/lib_config.py", line 358, in __getattr__
self._update_configs()
File "/base/alloc/tmpfs/dynamic_runtimes/python27/b93f87cfaa0115e5_unzipped/python27_lib/versions/1/google/appengine/api/lib_config.py", line 294, in _update_configs
self._registry.initialize()
File "/base/alloc/tmpfs/dynamic_runtimes/python27/b93f87cfaa0115e5_unzipped/python27_lib/versions/1/google/appengine/api/lib_config.py", line 165, in initialize
import_func(self._modname)
File "/base/alloc/tmpfs/dynamic_runtimes/python27/b93f87cfaa0115e5_unzipped/python27_dist/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/base/data/home/apps/s~forcediagnosticsclinic/20180707t052537.410964682053028174/appengine_config.py", line 18, in <module>
vendor.add('lib')
File "/base/alloc/tmpfs/dynamic_runtimes/python27/b93f87cfaa0115e5_unzipped/python27_lib/versions/1/google/appengine/ext/vendor/__init__.py", line 44, in add
'No such virtualenv or site directory' % path)
ValueError: virtualenv: cannot access lib: No such virtualenv or site directory
What is going on?
You app config refers to folder called lib and this folder is missing or you forgot to copy it properly. Try to get the same project working on your local computer.
Lenik is right. Specifically, it looks like you didn't do this step in the tutorial:
pip install -r requirements-vendor.txt -t lib/
https://cloud.google.com/python/django/appengine#run_the_app_on_your_local_computer

Google App Engine dev_appserver.py IOError when importing apache_beam

I have an App Engine application which breaks the moment I do
import apache_beam
I can confirm the dependencies are installed within the virtual environment the app runs in. Judging by the stack trace below it looks like it's trying to access /dev/null for some reason and it fails to.
INFO 2017-08-13 13:03:37,980 stubs.py:50] Sandbox prevented access to file "/home/raul/Downloads/google-cloud-sdk"
INFO 2017-08-13 13:03:37,980 stubs.py:51] If it is a static file, check that `application_readable: true` is set in your app.yaml
INFO 2017-08-13 13:03:37,980 stubs.py:50] Sandbox prevented access to file "/home/raul/Downloads/google-cloud-sdk/platform"
INFO 2017-08-13 13:03:37,980 stubs.py:51] If it is a static file, check that `application_readable: true` is set in your app.yaml
INFO 2017-08-13 13:03:38,260 stubs.py:50] Sandbox prevented access to file "/dev/null"
INFO 2017-08-13 13:03:38,260 stubs.py:51] If it is a static file, check that `application_readable: true` is set in your app.yaml
ERROR 2017-08-13 13:03:38,261 wsgi.py:263]
Traceback (most recent call last):
File "/home/raul/Downloads/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 240, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "/home/raul/Downloads/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
handler, path, err = LoadObject(self._handler)
File "/home/raul/Downloads/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 85, in LoadObject
obj = __import__(path[0])
File "/home/raul/Documents/football/main.py", line 7, in <module>
import apache_beam
File "/home/raul/Documents/football/env/lib/python2.7/site-packages/apache_beam/__init__.py", line 75, in <module>
import apache_beam.internal.pickler
File "/home/raul/Documents/football/env/lib/python2.7/site-packages/apache_beam/internal/pickler.py", line 38, in <module>
import dill
File "/home/raul/Documents/football/env/lib/python2.7/site-packages/dill/__init__.py", line 27, in <module>
from .dill import dump, dumps, load, loads, dump_session, load_session, \
File "/home/raul/Documents/football/env/lib/python2.7/site-packages/dill/dill.py", line 165, in <module>
FileType = type(open(os.devnull, 'rb', buffering=0))
File "/home/raul/Downloads/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/stubs.py", line 260, in __init__
raise IOError(errno.EACCES, 'file not accessible', filename)
IOError: [Errno 13] file not accessible: '/dev/null'
Any ideas what might be causing this and how to fix it?
The GAE Python sandbox is pretty restrictive in terms of what your app code is allowed to do. It looks like the dill package used by apache-beam needs to access special files (the /dev/null device file in this case) which is not allowed on the sandbox.
You either change your code to meet all sandbox requirements or try to use the flexible environment, which is a lot more tolerant (but it's a different beast overall). See Choosing an App Engine Environment and Migrating Services from the Standard Environment to the Flexible Environment.
Side note: the traceback indicates your app tried to execute apache-beam from the local system installation, which also won't work in the standard environment - it needs to be vendored into your application, see Installing a third-party library.

Python - How to import module from .so file from egg file without using an absolute file path

I built the pyahocorasick library with python setup.py bdist_egg command and uploaded it onto Spark for my PySpark job.
However, the .so file inside pyahocorasick can't be imported through pkg_resources.resource_filename() method on Spark cluster for security reasons.
Traceback (most recent call last):
File "spark_datawash.py", line 251, in <module>
import ahocorasick
File "build/bdist.linux-x86_64/egg/ahocorasick.py", line 7, in <module>
File "build/bdist.linux-x86_64/egg/ahocorasick.py", line 4, in __bootstrap__
File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 1152, in resource_filename
File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 1696, in get_resource_filename
File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 1726, in _extract_resource
File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 1219, in get_cache_path
File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 1199, in extraction_error
pkg_resources.ExtractionError: Can't extract file(s) to egg cache
The following error occurred while trying to extract file(s) to the Python egg
cache:
[Errno 13] Permission denied: '/home/.python-eggs'
The Python egg cache directory is currently set to:
/home/.python-eggs
Perhaps your account does not have write access to this directory? You can
change the cache directory by setting the PYTHON_EGG_CACHE environment
variable to point to an accessible directory.
This is how pyahocorasick import the .so:
def __bootstrap__():
global __bootstrap__, __loader__, __file__
import sys, pkg_resources, imp
__file__ = pkg_resources.resource_filename(__name__, 'ahocorasick.so')
__loader__ = None; del __bootstrap__, __loader__
imp.load_dynamic(__name__,__file__)
__bootstrap__()
Can I import the .so by resource_stream() instead of resource_filename() or some other way without the need to read from an absolute file path?
Thanks all.
BTW, I can't install pyahocorasick on every node on Spark cluster for other reasons. So I have to upload an egg-zipped distribution for later use.

Using virtualenv in VMWare with shared Windows folder

I currently use a linux VM in VMWare running on a Windows machine. I have a shared Windows folder for my projects that I access in linux via /mnt/hgfs/
When I try to use virtualenv for my python projects I get an 'Operation not supported' error:
joe#myserver:/mnt/hgfs/winwww/envtest# virtualenv env
Traceback (most recent call last):
File "/usr/local/bin/virtualenv", line 9, in <module>
load_entry_point('virtualenv==1.7', 'console_scripts', 'virtualenv')()
File "/usr/local/lib/python2.6/dist-packages/virtualenv.py", line 928, in main
never_download=options.never_download)
File "/usr/local/lib/python2.6/dist-packages/virtualenv.py", line 1029, in create_environment
site_packages=site_packages, clear=clear))
File "/usr/local/lib/python2.6/dist-packages/virtualenv.py", line 1164, in install_python
copyfile(join(stdlib_dir, fn), join(lib_dir, fn))
File "/usr/local/lib/python2.6/dist-packages/virtualenv.py", line 430, in copyfile
copyfileordir(src, dest)
File "/usr/local/lib/python2.6/dist-packages/virtualenv.py", line 405, in copyfileordir
shutil.copytree(src, dest, True)
File "/usr/lib/python2.6/shutil.py", line 173, in copytree
raise Error, errors
shutil.Error: [('/usr/lib/python2.6/config/libpython2.6.so', 'env/lib/python2.6/config/libpython2.6.so', '[Errno 95] Operation not supported')]
I'm assuming this is due to Windows not being able to handle symbolic links because I can create my virtualenvs in other directories.
Has anyone worked out a way to make virtualenv work in shared Windows folders? I edit my code on my Windows machine which is why I use shared folders.
Thanks.
The virtualenv does not need to be located next to your code, so even if you can't create it within your shared folder this shouldn't affect your ability to edit the code on windows and run it on your host platform. It sounds to me like a bad idea anyway to attempt to share platform-specific files with multiple platforms.
This is the usefulness of the .pth files and if you're using setuptools with a setup.py within your project you can just $VENV/python setup.py develop which will link your project into the virtualenv, wherever it may be located. You can then run $VENV/python and your package will be in the sys.path and available for import.

py2exe and sqlalchemy-migrate

I'm using python 2.6, sqlalchemy 0.6 for a tiny desktop application. Also i'm using py2exe and sqlalchemy-migrate and i'm trying to integrate migrate inside the exe. In this way when the user starts the application the database gets automatically upgraded.
If i try to use migrate in my eclipse project it works well but i want to release the project itself in an exe using py2exe. Unfortunately when i start the exe i got the error:
Traceback (most recent call last):
File "sagra.py", line 7, in <module>
File "guiutil.pyc", line 3, in <module>
File "bo.pyc", line 4, in <module>
File "database.pyc", line 26, in <module>
File "<string>", line 2, in version_control
File "migrate\versioning\util\__init__.pyc", line 160, in with_engine
File "migrate\versioning\api.pyc", line 248, in version_control
File "migrate\versioning\schema.pyc", line 128, in create
File "migrate\versioning\repository.pyc", line 76, in __init__
File "migrate\versioning\repository.pyc", line 97, in verify
migrate.versioning.exceptions.InvalidRepositoryError: db_repository
but the directory db_repository and the migration scripts are correctly inside the "library.zip" used by py2exe to incorporate all the python resources used by my software.
It seems sqlalchemy-migrate doesn't search inside the zip for the directory of the repository.
Here the problem (repository.py):
def verify(cls, path):
"""
Ensure the target path is a valid repository.
:raises: :exc:`InvalidRepositoryError <migrate.versioning.exceptions.InvalidRepositoryError>`
"""
# Ensure the existence of required files
try:
cls.require_found(path)
cls.require_found(os.path.join(path, cls._config))
cls.require_found(os.path.join(path, cls._versions))
except exceptions.PathNotFoundError, e:
raise exceptions.InvalidRepositoryError(path)
How to read the repository from a zip ?
Is there anyone who have tried (and succeded) to use sqlalchemy-migrate inside the exe generated by py2exe ?
Thanks for any help

Categories