Matplotlib with Google App Engine local development server - python

I want to use matplotlib in my Google App Engine project. I followed the steps, described here in the official docs. What I did:
1) Created a directory named lib in my application root directory.
2) Created a file appengine_config.py in my application root directory and added there these lines:
from google.appengine.ext import vendor
vendor.add('lib')
3) Since the docs say, that the only version of matplotlib working is 1.2.0, I executed the following command in the Terminal:
pip install -t lib matplotlib==1.2.0
There is also step 0 in the docs, which says
Use pip to install the library and the vendor module to enable importing packages from the third-party library directory.
But I don't understand what it actually means. If this is something essential, please, explain to me what it's meant here. I found this answer here on stackoverflow, and It seems there is nothing different from what I have done.
Also, I added
libraries:
- name: matplotlib
version: "1.2.0"
to app.yaml.
So, after all these steps I add the line
import matplotlib
to main.py and start a local server with
python ~/path/google_appengine/dev_appserver.py app.yaml
But when I try to access http://localhost:8080/, the error is raised:
raise ImportError('No module named %s' % fullname)
ImportError: No module named _ctypes
The whole output, if needed, looks like this:
ERROR 2016-08-11 16:26:51,621 wsgi.py:263]
Traceback (most recent call last):
File "/home/magnitofon/Загрузки/google_appengine/google/appengine/runtime/wsgi.py", line 240, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "/home/magnitofon/Загрузки/google_appengine/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
handler, path, err = LoadObject(self._handler)
File "/home/magnitofon/Загрузки/google_appengine/google/appengine/runtime/wsgi.py", line 85, in LoadObject
obj = __import__(path[0])
File "/home/magnitofon/realec-inspector/main.py", line 20, in <module>
import matplotlib
File "/home/magnitofon/realec-inspector/lib/matplotlib/__init__.py", line 151, in <module>
from matplotlib.rcsetup import (defaultParams,
File "/home/magnitofon/realec-inspector/lib/matplotlib/rcsetup.py", line 20, in <module>
from matplotlib.colors import is_color_like
File "/home/magnitofon/realec-inspector/lib/matplotlib/colors.py", line 52, in <module>
import numpy as np
File "/home/magnitofon/Загрузки/google_appengine/google/appengine/tools/devappserver2/python/sandbox.py", line 705, in load_module
module = self._find_and_load_module(fullname, fullname, [module_path])
File "/home/magnitofon/Загрузки/google_appengine/google/appengine/tools/devappserver2/python/sandbox.py", line 446, in _find_and_load_module
return imp.load_module(fullname, source_file, path_name, description)
File "/usr/local/lib/python2.7/dist-packages/numpy/__init__.py", line 180, in <module>
from . import add_newdocs
File "/usr/local/lib/python2.7/dist-packages/numpy/add_newdocs.py", line 13, in <module>
from numpy.lib import add_newdoc
File "/usr/local/lib/python2.7/dist-packages/numpy/lib/__init__.py", line 8, in <module>
from .type_check import *
File "/usr/local/lib/python2.7/dist-packages/numpy/lib/type_check.py", line 11, in <module>
import numpy.core.numeric as _nx
File "/usr/local/lib/python2.7/dist-packages/numpy/core/__init__.py", line 22, in <module>
from . import _internal # for freeze programs
File "/usr/local/lib/python2.7/dist-packages/numpy/core/_internal.py", line 14, in <module>
import ctypes
File "/usr/lib/python2.7/ctypes/__init__.py", line 10, in <module>
from _ctypes import Union, Structure, Array
File "/home/magnitofon/Загрузки/google_appengine/google/appengine/tools/devappserver2/python/sandbox.py", line 963, in load_module
raise ImportError('No module named %s' % fullname)
ImportError: No module named _ctypes
What am I doing wrong?

matplotlib is one of the Google-provided 3rd party libs, so you should be following just the Requesting a library instructions and
not the Installing a library ones.
Sadly they're now both on the same documentation page, called Using Built-in Libraries in Python 2.7 - very confusing for the unaware as the vendoring technique should be used for libraries which are not GAE built-in/provided. Filed Issue 13202.
Note: pay attention to the Using libraries with the local development server section, it applies to matplotlib. You may need to install some packages on your system, but not in the application itself (which could negatively affect your deployment on GAE) - they need to be accessible by the development server, not directly by your application.
Duh, I just noticed the Using matplotlib section, on the same page :)
It mentions:
Note: The experimental release of matplotlib is not supported on the development server. You can still add matplotlib to the
libraries list, but it will raise an ImportError exception when
imported.

Some searching into this issue surfaced this (old) recipe:
http://code.activestate.com/recipes/578393-gae-matplotlib-demo/
be sure to read the comments.
However I tried, I was not able to make it work. It be this approach just does not work with the current version of the app engine sandbox anymore, or I just was not able to follow all the steps in the recipe correctly.
Maybe an even better read are the comments of Matt Giuca (a google engineer who ported matplotlib to work in GAE production env.) and his pull request in github.
https://github.com/matplotlib/matplotlib/issues/1823/
His comment of April 17, 2013 gives a recipe for matching the dev_appserver (version 1.77) to work with the matplotlib locally.
Maybe this helps someone dealing with this issue.

Related

Python version error when running the Splunk TA-geoip2 app

I am using Splunk 8+ and Python 3.6. I am trying to run this https://github.com/0x616c6578/TA-geoip2 plugin.
I get this error message when I look into search.log and try to run the Python command in the log by itself:
C:\Users\strozllc>"C:\Program Files\Splunk\bin\Python3.exe" "C:\Program Files\Splunk\etc\apps\TA-geoip2-main\bin\geoip-command.py
Traceback (most recent call last):
File "C:\Program Files\Splunk\etc\apps\TA-geoip2-main\bin\geoip-command.py", line 8, in <module>
from splunklib.searchcommands import \
File "C:\Program Files\Splunk\etc\apps\TA-geoip2-main\bin\..\lib\splunklib\searchcommands\__init__.py", line 145, in <module>
from .environment import *
File "C:\Program Files\Splunk\etc\apps\TA-geoip2-main\bin\..\lib\splunklib\searchcommands\environment.py", line 20, in <module>
from logging.config import fileConfig
File "C:\Program Files\Splunk\Python-3.7\lib\logging\config.py", line 30, in <module>
import logging.handlers
File "C:\Program Files\Splunk\Python-3.7\lib\logging\handlers.py", line 26, in <module>
import logging, socket, os, pickle, struct, time, re
File "C:\Program Files\Splunk\Python-3.7\lib\socket.py", line 49, in <module>
import _socket
ImportError: Module use of python27.dll conflicts with this version of Python
n.
Huh? I looked at line 8 in geocommand.py and it is from
splunklib.searchcommands import \
dispatch, StreamingCommand, Configuration, Option, validators
I look it C:\Program Files\Splunk\bin, and there is indeed a python27.dll. But how is it getting called?
This is a case of Python 3 trying to run Python 2 code. The latest versions of Splunk (you didn't say which one you're using) only support Python 3 so an error will be thrown when an outdated library file is encountered.
I'd suggest filing an issue on GitHub, but the app appears to be abandonware since it hasn't been touched in a year despite having 4 issues.
Consider forking the code and updating it yourself.

Exception during calling gensim?

I tried to load gensim in my code. Often it works fine. Today, I get the following exception:
Traceback (most recent call last):
File "/project/6008168/tamouze/just.py", line 2, in <module>
import gensim
File "/project/6008168/tamouze/Python_directory/ENV2.7_new/lib/python2.7/site-packages/gensim/__init__.py", line 5, in <module>
from gensim import parsing, corpora, matutils, interfaces, models, similarities, summarization, utils # noqa:F401
File "/project/6008168/tamouze/Python_directory/ENV2.7_new/lib/python2.7/site-packages/gensim/parsing/__init__.py", line 4, in <module>
from .preprocessing import (remove_stopwords, strip_punctuation, strip_punctuation2, # noqa:F401
File "/project/6008168/tamouze/Python_directory/ENV2.7_new/lib/python2.7/site-packages/gensim/parsing/preprocessing.py", line 40, in <module>
from gensim import utils
File "/project/6008168/tamouze/Python_directory/ENV2.7_new/lib/python2.7/site-packages/gensim/utils.py", line 44, in <module>
from smart_open import smart_open
File "/project/6008168/tamouze/Python_directory/ENV2.7_new/lib/python2.7/site-packages/smart_open/__init__.py", line 1, in <module>
from .smart_open_lib import *
File "/project/6008168/tamouze/Python_directory/ENV2.7_new/lib/python2.7/site-packages/smart_open/smart_open_lib.py", line 29, in <module>
import requests
File "/project/6008168/tamouze/Python_directory/ENV2.7_new/lib/python2.7/site-packages/requests/__init__.py", line 97, in <module>
from . import utils
File "/project/6008168/tamouze/Python_directory/ENV2.7_new/lib/python2.7/site-packages/requests/utils.py", line 26, in <module>
from ._internal_utils import to_native_string
ImportError: cannot import name to_native_string
Im using python 2.7.14 and gensim 3.4.0.
How can I solve this problem?
The error isn't really occurring in gensim, even though that's how you found it. If you look at the stack, it's only triggered because gensim uses smart_open which in turn uses requests. It is in requests that the error happens.
If this was working, but now stopped, something likely changed in your environment, or how you're launching this code, related to the relationship between Python and the requests package.
For such errors, you should try searching Google for the final-two lines of your error stack – those most connected to the problem. Those are:
from ._internal_utils import to_native_string
ImportError: cannot import name to_native_string
(These leave out the file path that's specific to you, but have a number of unique tokens likely to have also been reported by any others.)
A number of people have hit this, from a variety of other projects, but always triggered through requests. Some have reported re-installing requests (perhaps to ensure it's version 2.0.0 or later) helps.
If a simple re-install doesn't help, you could also try one or all of:
uninstall, verify it's not present at all (that requests itself isn't found), then install – this could make sure you don't have overlapping redundant installations in different places that are both confusing the issue
start from a fresh Python environment, reinstalling all packages
double-check that all packages share the same Python2/Python3 compatibility

Python shell working differently on different location in the same machine inside same virtual environment

I have a cloud instance of a Linux machine (openSuSE) with multiple users.
I have created a virtual environment and installed all my required libraries (including Klein).
I have two users "a" and "b".
While logged in as "a" and inside virtualenv, when I open python shell at home directory and type
import klein
it imports normally.
Now when I change directory to
/home/b/
and run the same (open python shell, import klein) while being in the same virtualenv, it gives me an error.
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/a/.local/lib/python3.6/site-packages/klein/__init__.py", line 3, in <module>
from klein._plating import Plating
File "/home/a/.local/lib/python3.6/site-packages/klein/_plating.py", line 16, in <module>
from .app import _call
File "/home/a/.local/lib/python3.6/site-packages/klein/app.py", line 19, in <module>
from twisted.internet import endpoints, reactor
File "/home/a/.local/lib/python3.6/site-packages/twisted/internet/endpoints.py", line 58, in <module>
from twisted.protocols.tls import TLSMemoryBIOFactory
File "/home/a/.local/lib/python3.6/site-packages/twisted/protocols/tls.py", line 63, in <module>
from twisted.internet._sslverify import _setAcceptableProtocols
File "/home/a/.local/lib/python3.6/site-packages/twisted/internet/_sslverify.py", line 158, in <module>
verifyHostname, VerificationError = _selectVerifyImplementation()
File "/home/a/.local/lib/python3.6/site-packages/twisted/internet/_sslverify.py", line 141, in _selectVerifyImplementation
from service_identity import VerificationError
File "/home/a/.conda/envs/mm/lib/python3.6/site-packages/service_identity/__init__.py", line 7, in <module>
from . import cryptography, pyopenssl
File "/home/a/.conda/envs/mm/lib/python3.6/site-packages/service_identity/cryptography.py", line 16, in <module>
from .exceptions import SubjectAltNameWarning
File "/home/a/.conda/envs/mm/lib/python3.6/site-packages/service_identity/exceptions.py", line 21, in <module>
#attr.s
AttributeError: module 'attr' has no attribute 's'
Command "which python" gives same address at both location which is my virtualenv python address and that should be expected.
But what causes this weird python shell behavior.
Thank you
I solved it and a very shameful reason caused the error.
One of the modules Twisted uses is "attr" module. I had named one of my files attr.py and that is what was causing all the error.
I myself am not deleting this question if moderation has no problem, maybe somebody like me might be stuck at the same situation. It may help them.
Never name your python files same as that of any standard module unless overriding.
Also if your issue persists, then Jean's answer will definitely resolve it.
There can be multiple different Python packages that provide the same Python module. For example, there are at least two packages that provide the attr module:
https://pypi.org/project/attr/
https://pypi.org/project/attrs/
It's possible you've installed the wrong package based on the requirements. You can check what you have installed with pip freeze.

"ImportError: No module named pkg_resources" when importing google.cloud.datastore

I am attempting to use the Google Cloud Datastore, but importing google.cloud.datastore gives an ImportError:
ERROR 2018-03-13 19:28:29,013 wsgi.py:263]
Traceback (most recent call last):
File "/home/<user>/Software/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 240, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "/home/<user>/Software/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
handler, path, err = LoadObject(self._handler)
File "/home/<user>/Software/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 85, in LoadObject
obj = __import__(path[0])
File "/home/<user>/Projects/<my_project>/main.py", line 1, in <module>
from my_project import app
File "/home/<user>/Projects/<my_project>/<my_project>/__init__.py", line 2, in <module>
from my_project.submit.controllers import submit
File "/home/<user>/Projects/<my_project>/<my_project>/submit/controllers.py", line 6, in <module>
from . import model_datastore
File "/home/<user>/Projects/<my_project>/<my_project>/submit/model_datastore.py", line 2, in <module>
from google.cloud import datastore
File "/home/<user>/Projects/<my_project>/env/local/lib/python2.7/site-packages/google/cloud/datastore/__init__.py", line 57, in <module>
from pkg_resources import get_distribution
ImportError: No module named pkg_resources
I am on Linux Mint attempting to run a Google App Engine local dev server.
I am using a virtual environment. Both setuptools and pkg_resources are installed and updated in the virtual environment. When I enter the python cmd line interpreter from the virtual env and import pkg_resources, it works fine. When I run the google app engine dev server by doing dev_appserver app.yaml, everything works fine until I access the page that activates the handler that imports datastore, then I get this error.
None of the other similar posts about 'pkg_resources` import error were helpful.
If there is any other info I can provide that would help, please let me know. Thanks!
In the standard environment you need to install all your dependencies inside your application. See Using third-party libraries.
Whatever you have installed in the local environment (virtual or not) doesn't matter, GAE doesn't know how to use those and your app might not work properly locally and definitely won't work when deployed on GAE.
You traceback indicates that you're loading the datastore library from your virtual env, not from your app, which is most likely why it doesn't work:
.../env/local/lib/python2.7/site-packages/google/cloud/datastore/__init__.py
You need to fix your app dependency installation.
This discussion might be of interest as well: No module named warnings when starting GAE inside virtualenv locally

Google app engine does not run local server

My helloworld code doesnt run on google app engine...
I use python 2.7.6 and google app engine version 1.8.8.
Here is what I get:
Traceback (most recent call last):
File "/home/yash/Google/google_appengine/old_dev_appserver.py", line 196, in <module>
run_file(__file__, globals())
File "/home/yash/Google/google_appengine/old_dev_appserver.py", line 192, in run_file
execfile(script_path, globals_)
File "/home/yash/Google/google_appengine/google/appengine/tools/dev_appserver_main.py", line 173, in <module>
from google.appengine.tools import appcfg
File "/home/yash/Google/google_appengine/google/appengine/tools/appcfg.py", line 74, in <module>
from google.appengine.tools import appengine_rpc
File "/home/yash/Google/google_appengine/google/appengine/tools/appengine_rpc.py", line 28, in <module>
import fancy_urllib
File "/home/yash/Google/google_appengine/lib/fancy_urllib/fancy_urllib/__init__.py", line 354, in <module>
class FancyHTTPSHandler(urllib2.HTTPSHandler):
AttributeError: 'module' object has no attribute 'HTTPSHandler'</i>
well...problem solved...i guess few libraries werent installed...compiled python 5-6 times after each library installation...
the problem wasnt in the GAE...it was the lack of required python modules
m still a noob...
Thanks for your time and help :)
Looks like you don't have python support for SSL.
Try this
pip install ssl
Read python-ssl docs for details.
From the docs
If you want to use native python ssl, you must enable it using the
libraries configuration in your application's app.yaml file where you
specify the library name "ssl", as shown here:
libraries:
- name: ssl
version: latest
Does your app.yaml contain this?
Your google appengine environment probably isn't installed properly. I'd recommend reinstalling the SDK and checking for errors in the installation.
If you're using something like virtualenv, it could be affecting the way your install works.

Categories