We have a Python Application running on Google App Engine. For local development, we have the Google Appengine SDK installed. The setup is a little messed up as the application itself adds things to the sys.path. PyLint of course does not pick up those changes happening somewhere in some function.
I tried cleaning up the PyLint log by adding the Google Appengine SDK to the PYTHONPATH, which worked pretty good. Locally, I can now run PyLint for a specific set of folders and it reports a 10/10 with no errors (but a lot of locally-disabled statements). The same code is reporting exactly 1 issue on our CI system (We use Codeship). The message is Module 'google.appengine.ext.ndb' has no 'JsonProperty' member (no-member).
The Googe Appengine SDK is installed inside the Codeship container and is added to the PYTHONPATH. Other functions/classes from the SDK get picked up correctly. The google.appengine.ext.ndb.init.py looks like this:
"""NDB -- A new datastore API for the Google App Engine Python runtime."""
__version__ = '1.0.10'
__all__ = []
from tasklets import *
__all__ += tasklets.__all__
from model import * # This implies key.*
__all__ += model.__all__
from query import *
__all__ += query.__all__
from context import *
__all__ += context.__all__
The google.appengine.ext.ndb.model.py contains the JsonProperty class and has some magic as well:
# Update __all__ to contain all Property and Exception subclasses.
for _name, _object in globals().items():
if ((_name.endswith('Property') and issubclass(_object, Property)) or
(_name.endswith('Error') and issubclass(_object, Exception))):
__all__.append(_name)
As I said, it works locally but does not on Codeship. Locally, `pylint --version prints
pylint 1.5.0,
astroid 1.4.5
Python 2.7.9 (default, Apr 2 2015, 15:33:21)
[GCC 4.9.2]
while on Codeship, it says
pylint 1.5.0,
astroid 1.4.4
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
[GCC 4.8.2]
I am not sure what is going on here. There should be no major differences between Python 2.7.6 and 2.7.9, right? I have no idea how to further debug this either. Any hints are appreciated.
Related
I'm working on a small app running on GAE. When I deploy my application or try and use virtualenv (per the docs), I see the following in the logs in Cloud Platform (they're the same locally, too) when I try and access the app:
File "/base/data/home/apps/d~my-project/gen-debug:413039252702584887/main.py", line 10, in <module>
from google.cloud import logging_v2
File "/base/data/home/apps/d~my-project/gen-debug:413039252702584887/lib/google/cloud/logging_v2/__init__.py", line 17, in <module>
from google.cloud.logging_v2 import types
ImportError: cannot import name types
My code that's throwing this error looks like this:
from google.cloud import logging_v2
client = logging_v2.LoggingServiceV2Client()
c = client.list_log_entries(["my-project"], filter_='resource.type="global" AND jsonPayload.hello="there"')
l = list(c)
l1 = l[0]
desc, value = l1.ListFields()[1]
print desc.camelcase_name, dict(value.items())
The thing that is so freaking weird is that in Google, I can go into a Cloud Shell session, get into ipython, and this code totally works.
me#cloudshell:~ (my-project)$ ipython2
Python 2.7.13 (default, Sep 26 2018, 18:42:22)
Type "copyright", "credits" or "license" for more information.
IPython 5.8.0 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
In [1]: from google.cloud import logging_v2
...: client = logging_v2.LoggingServiceV2Client()
...: c = client.list_log_entries(["my-project"], filter_='resource.type="global" AND jsonPayload.hello="there"')
...: l = list(c)
...: l1 = l[0]
...: desc, value = l1.ListFields()[1]
...: print desc.camelcase_name, dict(value.items())
...:
jsonPayload {u'hello': u'there'}
What's going on that makes running this on GAE different than running it in the Google Cloud Shell? This import error seems like a Google issue, but I'm not sure.
That looks like the generic python client for Stackdriver logging. Which isn't designed for/supported on GAE first-generation (i.e., Python 2.7) apps as this runtime already has it's own (customized) stackdriver logging integrated.
From Supported environments:
The following VM instances support Stackdriver Logging using their own
software, possibly including custom versions or configurations of the
Stackdriver Logging agent. Manually installing the Stackdriver Logging
agent on them is not supported:
App Engine standard environment VM instances. App Engine includes built-in support for Stackdriver Logging. For more
information, see Stackdriver Logging in App Engine Apps.
App Engine flexible environment VM instances. Apps running in the App Engine flexible environment can write logs that are in
addition to what is included in the App Engine standard environment.
For more information, see Stackdriver Logging and the App Engine
flexible environment.
I am trying to deploy a Flask web app via mod_wsgi on Apache. I cannot use the default Python environment because it was compiled with UCS-2 Unicode instead of UCS-4, and I cannot recompile it for this one case. Thus, a virtual environment. Virtual environments would have been used anyway, but that error means that I can't get away with using the default Python install and just adding the virtual environment's modules to the PYTHONPATH, which otherwise would have let me avoid the current problem entirely by accident.
I found the documentation for mod_wsgi to change which Python executable to use. However, when attempting to do so, the server fails to work properly. /var/log/httpd/error_log rapidly floods with the line ImportError: No module named site.
I have checked every similar question I can find here and elsewhere, and not yet had success. Experimentation has shown that as far as I can tell, the problem occurs when changing PYTHONHOME without activating a virtual environment - and the way the automated deployment works (via Fabric), as far as I can tell I can't activate a virtual environment.
Apache config
My current httpd.conf for the app:
WSGIPythonPath /path/to/dir/containing/wsgi/file/and/app:/path/to/virtualenv/lib:/path/to/virtualenv/lib/site-packages
WSGIPythonHome /path/to/virtualenv
WSGISocketPrefix /var/run/wsgi
User user
Group group
<VirtualHost *>
ServerName servername.generic.com
DocumentRoot /path/to/dir/containing/wsgi/file/and/app/static_dev/
WSGIDaemonProcess appname user=user group=group threads=2
WSGIScriptAlias / /path/to/dir/containing/wsgi/file/and/app/app.wsgi
<Directory /path/to/dir/containing/wsgi/file/and/app>
WSGIProcessGroup appname
WSGIApplicationGroup %{GLOBAL}
Require all granted
</Directory>
</VirtualHost>
Data I've found from failed attempts
I know that the error is not in my app.wsgi, because when I added the line raise Exception('tried to open the file') at the very top to check that, the existing ImportError kept happening instead of that new Exception.
I have confirmed via ldd that my version of mod_wsgi is for Python 2.7.
I have tried setting WSGIPythonHome /path/to/virtualenv/bin/ and WSGIPythonHome /path/to/virtualenv/bin/python, with the same result as the current state.
I have tried omitting the WSGIPythonHome directive, in which case it loads the app.wsgi as it should, but breaks on a later import as described at the top (the reason I can't just do that).
I have tried omitting the WSGIPythonPath directive and leaving it up to app.wsgi to add things to the PYTHONPATH, with the same result as the current state.
I have tried putting the path-setting as an argument to WSGIDaemonProcess instead of as the WSGIPythonPath directive, with the same result as the current state.
I have confirmed that there is a site.py in /path/to/virtualenv/lib.
I have confirmed that no other non-app-specific Apache .conf files being used (default settings, automatic module loads, etc) contain the string "WSGI", so I don't think there's any conflicts here.
If I activate the virtual environment from the command line I can import site without an error, just for the sake of testing that it does in fact exist in the environment. However, this is insufficient because it needs to start smoothly from a single call to sudo systemctl start httpd.service due to the deployment tools in use, and that seems to not care about the venv of the current shell session.
If, from a default state, I export PYTHONHOME=/path/to/virtualenv, attempting to open the Python REPL exits immediately with ImportError: No module named site.
If I activate the virtual environment and then set PYTHONHOME, I get the same import error.
If I activate the virtual environment and don't touch PYTHONHOME, echo $PYTHONHOME outputs a blank line, and the Python REPL works fine. In the Python REPL while in the virtualenv:
(virtualenv)-bash-4.2$ python
Python 2.7.8 (default, Aug 14 2014, 13:26:38)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.prefix
'/path/to/virtualenv'
>>> sys.exec_prefix
'/path/to/virtualenv'
Even though setting PYTHONHOME to the same value didn't work.
If I try export PYTHONHOME=:/path/to/virtualenv or export PYTHONHOME=/path/to/virtualenv:, explicitly setting only one of prefix and exec_prefix, it fails with the same import error in either case.
If I activate the virtual environment and set PYTHONHOME in one of those latter two ways, the unset one appears to default to / rather than to the usual default value, but the Python REPL runs fine:
# Setting only exec_prefix
(virtualenv)-bash-4.2$ export PYTHONHOME=:/path/to/virtualenv
(virtualenv)-bash-4.2$ python
Python 2.7.8 (default, Aug 14 2014, 13:26:38)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.prefix
'/'
>>> sys.exec_prefix
'/path/to/virtualenv'
>>> quit()
# Setting only prefix
(.virtualenv)-bash-4.2$ export PYTHONHOME=/path/to/virtualenv:
(.virtualenv)-bash-4.2$ python
Python 2.7.8 (default, Aug 14 2014, 13:26:38)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.prefix
'/path/to/virtualenv'
>>> sys.exec_prefix
'/'
Unfortunately, since the deployment script doesn't care what environment is activated, that doesn't solve it. Trying to set WSGIPythonHome in such a fashion makes no difference whatsoever.
I have noticed one further thing: The Python in the virtualenv is 2.7.8. The Python run outside the virtualenv (usr/bin/python) is 2.7.5. I do not know - would this affect the setting of PYTHONHOME somehow? I would hope not - since that seems to defeat the entire purpose of using WSGIPythonHome to run a virtualenv as compared to just setting sys.path inside the app.wsgi file, the ability to start from a different executable - but I cannot rule it out, clueless as I am.
The 2.7.8 Python in /path/to/virtualenv/bin/python has a sys.real_prefix of /network-mounted-drive/sw/python/python-2.7.8.
I changed the deployment to build from /network-mounted-drive/sw/python/python-2.7.5, then did more tests. Results as follows:
Attempting to start httpd gives the same import error as before.
Setting PYTHONHOME to the location of the virtual environment, then running python:
-bash-4.2$ echo $PYTHONHOME
/path/to/virtualenv
-bash-4.2$ python
ImportError: No module named site
Setting PYTHONHOME to the location of the virtual environment, then explicitly running the virtual environment's python binary (activating the virtual environment and then running python gives the same result):
# In the directory just above the virtualenv
-bash-4.2$ ./virtualenv/bin/python
Python 2.7.5 (default, Mar 14 2016, 14:13:09)
[GCC 4.8.3 20140911 (Red Hat 4.8.3-9)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.prefix
'/path/to/virtualenv'
>>> sys.exec_prefix
'/path/to/virtualenv'
>>> sys.real_prefix
'/network-mounted-drive/sw/python/python-2.7.5
Does anyone have any idea?
Resolution found: The issue seems to have been in trying to use a virtual environment built from something other than the local python install on the system.
Solved by pushing the problem of "local python install on the deployment VM doesn't have pip installed" up the chain to someone with the permissions required to install pip, since no attempted workarounds via networked python installs worked.
The issue of actually using a virtual environment chained from a Python install on a network drive for mod_wsgi may be insoluble, or at least I couldn't figure it out in a reasonable amount of time relative to the bureaucratic solution.
We are using the ElementTree library in our project. This comes from the standard library, but for some reason pylint chokes on it. It always reports an E1101 on any reference on an ElementTree object. Here is a sample bit of code to demonstrate this:
import xml.etree.ElementTree as ET
test = ET.parse('test.xml')
config = test.find('.//configuration')
print(config)
for role in config.findall('.//role'):
print(role)
The output from pylint always throws an E1101:
$ pylint --rcfile=pylint.cfg -E test.py
************* Module orion-scripts.test
E: 7,12: Instance of 'int' has no 'findall' member (no-member)
But the program runs just fine:
$ python test.py
<Element 'role' at 0x1052dfe50>
<Element 'configuration' at 0x1052dfe10>
I found a number of solutions to similar problems, but so far none have worked for this one. I've tried:
Setting an ignored class with ignored-classes=xml.etree.ElementTree
Setting it as a generated member with generated-members=ET
Allowing any extension to run with unsafe-load-any-extension=yes
Nothing has worked and I'm currently just ignoring E1101 messages. However, this is a really bad option and I would much prefer to just ignore this particular class. If it helps, this is the version info for pylint:
$ pylint --version
No config file found, using default configuration
pylint 1.6.3,
astroid 1.4.7
Python 2.7.11 (default, Jan 22 2016, 08:28:37)
[GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)]
While trying to figure out how to properly import the Google Identity Toolkit into my GAE project I observed that when running the project in PyCharm (v. 4.5.2 pro) the PYTHONPATH is NOT the same as in PyCharm's Python Console.
For example in the Console I properly find the oauth2client library from the GAE SDK (v. 1.9.23):
/usr/bin/python2.7 -u /home/usr_local/pycharm-4.5.2/helpers/pydev/pydevconsole.py 43085 43754
Python 2.7.8 (default, Sep 30 2014, 15:34:38) [GCC]
Type "copyright", "credits" or "license" for more information.
IPython 2.2.0 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
PyDev console: using IPython 2.2.0
import sys; print('Python %s on %s' % (sys.version, sys.platform))
sys.path.extend(['/home/user/src/myapp', '/usr/local/google_appengine', '/usr/local/google_appengine/lib/antlr3', '/usr/local/google_appengine/lib/apiclient', '/usr/local/google_appengine/lib/endpoints-1.0', '/usr/local/google_appengine/lib/yaml-3.10', '/usr/local/google_appengine/lib/markupsafe-0.15', '/usr/local/google_appengine/lib/django-1.3', '/usr/local/google_appengine/lib/requests', '/usr/local/google_appengine/lib/fancy_urllib', '/usr/local/google_appengine/lib/webob_0_9', '/usr/local/google_appengine/lib/graphy', '/usr/local/google_appengine/lib/distutils', '/usr/local/google_appengine/lib/concurrent', '/usr/local/google_appengine/lib/cacerts', '/usr/local/google_appengine/lib/six', '/usr/local/google_appengine/lib/pyasn1', '/usr/local/google_appengine/lib/setuptools-0.6c11', '/usr/local/google_appengine/lib/jinja2-2.6', '/usr/local/google_appengine/lib/portpicker', '/usr/local/google_appengine/lib/django-0.96', '/usr/local/google_appengine/lib/django-1.2', '/usr/local/google_appengine/lib/PyAMF-0.6.1', '/usr/local/google_appengine/lib/sqlcmd', '/usr/local/google_appengine/lib/oauth2client', '/usr/local/google_appengine/lib/uritemplate', '/usr/local/google_appengine/lib/httplib2', '/usr/local/google_appengine/lib/protorpc-1.0', '/usr/local/google_appengine/lib/python-gflags', '/usr/local/google_appengine/lib/rsa', '/usr/local/google_appengine/lib/grizzled', '/usr/local/google_appengine/lib/django-1.5', '/usr/local/google_appengine/lib/ipaddr', '/usr/local/google_appengine/lib/django-1.4', '/usr/local/google_appengine/lib/argparse', '/usr/local/google_appengine/lib/google-api-python-client', '/usr/local/google_appengine/lib/mox', '/usr/local/google_appengine/lib/cherrypy', '/usr/local/google_appengine/lib/prettytable', '/usr/local/google_appengine/lib/webapp2-2.3', '/usr/local/google_appengine/lib/oauth2', '/usr/local/google_appengine/lib/pyasn1_modules', '/usr/local/google_appengine/lib/webapp2-2.5.2', '/usr/local/google_appengine/lib/websocket', '/usr/local/google_appengine/lib/yaml', '/usr/local/google_appengine/lib/docker', '/usr/local/google_appengine/lib/simplejson', '/usr/local/google_appengine/lib/webapp2-2.5.1', '/usr/local/google_appengine/lib/deprecated_enum'])
Python 2.7.8 (default, Sep 30 2014, 15:34:38) [GCC] on linux2
In[2]: from oauth2client import crypt
In[3]: crypt
Out[3]: <module 'oauth2client.crypt' from '/usr/local/google_appengine/lib/oauth2client/oauth2client/crypt.py'>
In[4]:
But when running the project I get:
File "/home/user/src/myapp/main/identitytoolkit/gitkitclient.py", line 47, in <module>
from oauth2client import crypt
ImportStringError: import_string() failed for 'gitkit.GitKitHandler'. Possible reasons are:
- missing __init__.py in a package;
- package or module path not included in sys.path;
- duplicated package or module name taking precedence in sys.path;
- missing module, class, function or variable;
Original exception:
ImportError: No module named oauth2client
I'm also dumping the sys.path before the exception (real paths are displayed, /usr/local/google_appengine is a symlink to /home/usr_local/google_appengine_1.9.23):
ERROR 2015-06-30 18:59:46,199 gitkit.py:13] ['/home/user/src/myapp/main', '/home/usr_local/google_appengine_1.9.23', '/home/usr_local/google_appengine_1.9.23', '/usr/lib64/python2.7', '/usr/lib64/python2.7/lib-dynload', '/home/usr_local/google_appengine_1.9.23/lib/webapp2-2.5.2', '/home/usr_local/google_appengine_1.9.23/lib/jinja2-2.6', '/home/usr_local/google_appengine_1.9.23/lib/pycrypto-2.6', '/home/usr_local/google_appengine_1.9.23/lib/markupsafe-0.15', '/home/usr_local/google_appengine_1.9.23/lib/setuptools-0.6c11', '/home/usr_local/google_appengine_1.9.23/lib/protorpc-1.0', '/home/usr_local/google_appengine_1.9.23/lib/webob-1.1.1', '/home/usr_local/google_appengine_1.9.23/lib/yaml-3.10']
Is there a way to configure PyCharm to apply the same PYTHONPATH from the Console (which is IMHO correct) to the development server execution?
Alternatively - how does PyCharm determine/configure the development server execution path? (I don't mind even configuring the correct path manually if needed).
Donno if relevant, but I've upgraded the PyCharm and GAE SDK versions several times while working on this project.
It turned out that the GAE development server adjusts its own execution path depending on the SDK installation and the executed app code itself.
The failures in my case were caused by incorrect 3rd party lib vendoring in a multi-modules app. I documented my fix here: Is Google Identity Toolkit (v3) compatible with GAE/python sandbox?
With the following code:
import pytest
def test_a():
with pytest.raises(Exception):
1/0
If I run pylint on it, it will make a complain that "raises" is not a member of module pytest:
E: 3,9:test_a: Module 'pytest' has no 'raises' member
Which is obviously not true. Any idea why pylint is making such a mistake? Is this a known bug?
py.test version:
> py.test --version
This is py.test version 2.2.3, imported from C:\Python27\lib\site-packages\pytest.pyc
PyLint version:
> pylint --version
No config file found, using default configuration
pylint 0.25.1,
astng 0.23.1, common 0.57.1
Python 2.7.2 (default, Jun 24 2011, 12:22:14) [MSC v.1500 64 bit (AMD64)]
You can silence this in a pylintrc file with:
ignored-classes=pytest
Last time I looked pylib does some heavy dynamic in low level python stuff, such as completely redefining the import code. It is very likely that this completely baffles pylint/astng, and prevents it from getting what is inside the pytest module: pylint/astng does not import the code it analyzes, it parses it, meaning that stuff which is dynamically initialized at import time will usually go unnoticed, which in turn generates false positives such as the one you report.
From there, you face the following choices:
use another unittest framework, less dynamic than py.test
silence the warnings / errors on your test code manually
use another linter which is happier than pylint on py.test (I'm interested to know how pychecker / pyflakes fare on that code)
write the astng plugin which will help astng grok the pylib tricks and submit it as a patch to the astng maintainers (and get extra credit from that)