We Keep Coding

Python urllib3 module could it be monkeypatched by eventlet?

I use Python eventlet pool and requests module to speed up my HTTPS request.
But it makes the requests process slower, I try to find the answer to the question these days.
I find some use cases for Python eventlet. such as Speed of fetching web pages with Eventlet and Python? and https://eventlet.net/doc/ these documents mentioned.
urls = ["http://www.google.com/intl/en_ALL/images/logo.gif",
"https://wiki.secondlife.com/w/images/secondlife.jpg",
"http://us.i1.yimg.com/us.yimg.com/i/ww/beta/y3.gif"]
import eventlet
from eventlet.green import urllib2
def fetch(url):
return urllib2.urlopen(url).read()
pool = eventlet.GreenPool()
for body in pool.imap(fetch, urls):
print "got body", len(body)
urls = [
"http://www.google.com/intl/en_ALL/images/logo.gif",
"http://python.org/images/python-logo.gif",
"http://us.i1.yimg.com/us.yimg.com/i/ww/beta/y3.gif",
]
import eventlet
from eventlet.green.urllib.request import urlopen
def fetch(url):
return urlopen(url).read()
pool = eventlet.GreenPool()
for body in pool.imap(fetch, urls):
print("got body", len(body))
I notice only Python module urllib and urllib2 are used in these examples, and I can not import urllib3 from event.green module.
Python 2.7.5 (default, Nov 16 2020, 22:23:17)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from eventlet.green import urllib3
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: cannot import name urllib3
>>>
When I check the Python requests module, I find it uses urllib3 module.
[root#ci-4183549-pzhang requests]# ls
adapters.py api.py auth.py certs.py compat.py cookies.py exceptions.py hooks.py __init__.py models.py packages sessions.pyo status_codes.pyo structures.pyo utils.pyo
adapters.pyc api.pyc auth.pyc certs.pyc compat.pyc cookies.pyc exceptions.pyc hooks.pyc __init__.pyc models.pyc sessions.py status_codes.py structures.py utils.py
adapters.pyo api.pyo auth.pyo certs.pyo compat.pyo cookies.pyo exceptions.pyo hooks.pyo __init__.pyo models.pyo sessions.pyc status_codes.pyc structures.pyc utils.pyc
[root#ci-4183549-pzhang requests]# ls packages/
chardet __init__.py __init__.pyc __init__.pyo urllib3
The question is:
Does it mean the requests module can not be eventlet monkeypatched, even if I do eventlet.monkey_patch() at the beginning of my script? it will not work correctly, will it?
Any help or hint is appreciated.

cPanel App runs on Python2 instead of Python3

I have a VPS Hosting in cPanel, and a Flask-App.
I followed the instructions of How to install WSGI Application in cPanel.
Everything is working fine, when I run the application using the terminal, but when I open the App URL, it shows the following error:
Traceback (most recent call last):
File "/opt/cpanel/ea-ruby24/root/usr/share/passenger/helper-scripts/wsgi-loader.py", line 369, in <module>
app_module = load_app()
File "/opt/cpanel/ea-ruby24/root/usr/share/passenger/helper-scripts/wsgi-loader.py", line 76, in load_app
return imp.load_source('passenger_wsgi', startup_file)
File "/home/qsemh/Maidan/passenger_wsgi.py", line 8, in <module>
wsgi = imp.load_source('wsgi', 'wsgi.py')
File "wsgi.py", line 1, in <module>
from flaskr import create_app
File "/home/qsemh/Maidan/flaskr/__init__.py", line 133
print(f"Validate {len(users)} Users At {current_time}")
^
SyntaxError: invalid syntax
So I've decided to create a simpler app in order to detect the issue, the app is as following :
passenger_wsgi.py
#!/usr/bin/env python3
import sys
from flask import Flask
application = Flask(__name__)
application.route("/")
def index():
return sys.version
When I run this simple Application Using the URL it shows the following as a respond :
2.7.5 (default, Apr 2 2020, 13:16:51) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]
even though I've used the shebang #!/usr/bin/env python3 at the start of the file, and when I run it using the terminal, it works as if it using python3.
I've tried changing the shebang to the following formats :
#!/usr/bin/python3
#!/usr/bin/env 'python3'
but they gave the same result.
What is the problem here, and How can I solve it?

I've found the Correct approach to solve this issue;
Basically, I only needed to add the following lines at the beginning of my passenger_wsgi.py file :
import os
INTERP = "/usr/bin/python3"
if sys.executable != INTERP: os.execl(INTERP, INTERP, *sys.argv)
so the final result would be :
passenger_wsgi.py
#!/usr/bin/env python3
import sys
import os
# Solution
INTERP = "/usr/bin/python3"
if sys.executable != INTERP: os.execl(INTERP, INTERP, *sys.argv)
from flask import Flask
application = Flask(__name__)
application.route("/")
def index():
return sys.version
and the respond is correctly as I intended in the first place:
3.6.8 (default, Apr 2 2020, 13:34:55) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]

This answer might be for some cPanel users. My cPanel is using passenger and wsgi, and even though my cPanel Python application is set to use python 3.9. it was running 2.7.5. Worse, it set all the 2.7 folders in the path.
So I ended up with this helper script called from
passenger_wsgi.py. I'm moved all the code out of this file into an import because this file may get blown away and re-created in some cases. It's not safe to rely on code in this file surviving. It's an auto generated file.
Put set_python_environment.py file in the same directory as passenger_wsgi.py
set_python_environment.py
# -*- coding: UTF-8 -*-
# insert these lines into passenger_wsgi.py if that file is rebuilt:
# import set_python_environment
# set_python_environment.upgrade_python()
import sys
import os
def upgrade_python():
try:
python_interpreter = "/home/username/virtualenv/YourApplicationPath/3.9/bin/python3.9_bin"
if sys.executable != python_interpreter:
print('switching from ' + sys.version + ' to ' + python_interpreter + '...')
print('directory: ' + os.path.dirname(__file__))
print('file: ' + __file__)
print('arg 0: ' + sys.argv[0])
# rebuild the env path variable
print('old system path:\n' + "\n".join(str(x) for x in sys.path))
print('\n')
for x in sys.path:
sys.path.remove(x)
sys.path.append('/opt/cpanel/ea-ruby24/root/usr/share/passenger/helper-scripts')
# App 3956 output: /usr/lib/python2.7/site-packages/pyzor-1.0.0-py2.7.egg
# App 3956 output: /usr/lib64/python27.zip
# App 3956 output: /usr/lib64/python2.7
sys.path.append('/home/username/virtualenv/YourApplicationPath/3.9/lib64/python3.9')
# App 3956 output: /usr/lib64/python2.7/plat-linux2
# App 3956 output: /usr/lib64/python2.7/lib-tk
# App 3956 output: /usr/lib64/python2.7/lib-old
# App 3956 output: /usr/lib64/python2.7/lib-dynload
# App 3956 output: /home/username/.local/lib/python2.7/site-packages
# App 3956 output: /usr/lib64/python2.7/site-packages
sys.path.append('/home/username/virtualenv/YourApplicationPath/3.9/lib64/python3.9/site-packages')
# App 3956 output: /usr/lib64/python2.7/site-packages/gtk-2.0
# App 3956 output: /usr/lib/python2.7/site-packages
sys.path.append('/home/username/virtualenv/YourApplicationPath/3.9/lib/python3.9/site-packages')
sys.path.append('/home/username/virtualenv/YourApplicationPath/3.9/bin')
print('new system path:\n' + "\n".join(str(x) for x in sys.path))
print('\n')
os.execl(python_interpreter, python_interpreter, *sys.argv)
else:
print('...continuing with ' + sys.executable)
except Exception as e:
print(str(e))
Modified passenger_wsgi.py
# -*- coding: UTF-8 -*-
import sys
import os
import imp
import set_python_environment
set_python_environment.upgrade_python()
sys.path.insert(0, os.path.dirname(__file__))
wsgi = imp.load_source('wsgi', 'app.py')
application = wsgi.app
In order to get the right path, I typed whereis python in the ssh environment after running the source command given on the python application page, to find the location of all the python versions, and the one ending in bin turned out to be the one which actually works.

How do I fix this Python ModuleNotFoundError

I am trying to figure out what is causing this file called builder.py to not run on mac even though it runs on windows. Code:
import cffi
import glob
import platform
# relative to build dir
LIB_BASE = '../libs/'
# compiling libraries statically to get a single binary
EXTRA_SRC = [LIB_BASE + 'subhook/subhook.c']
pltsysname = {'Windows': 'win32', 'Darwin': 'osx', 'Linux': 'elf'}
pltsrc = pltsysname[platform.system()]
pltsrc = LIB_BASE + 'plthook/plthook_{}.c'.format(pltsrc)
# EXTRA_SRC.append(pltsrc) # disabled until it is actually useful
LIBDIRS = []
if platform.system() == 'Windows':
LIBDIRS.append('../libs/SDL/lib/x86/')
CDEFS = 'generated internals SDL XDL subhook xternPython'.split()
def readfile(name):
with open(name, 'r') as f:
content = f.read()
return content
def build():
ffibuilder = cffi.FFI()
for fname in CDEFS:
ffibuilder.cdef(readfile('cdefs/{}.h'.format(fname)))
ffibuilder.embedding_api('uint32_t kickstart();')
ffibuilder.embedding_init_code(readfile('remote.py'))
ffibuilder.set_source(
'_remote', readfile('cdefs/remote.c'), sources=EXTRA_SRC,
libraries=['SDL2'], library_dirs=LIBDIRS,
define_macros=[('SUBHOOK_STATIC', None)])
ffibuilder.compile(tmpdir='build', target='remote.bin')
if __name__ == '__main__':
build()
When ever I run it I expect it to run but instead it comes up with the following error:
Traceback (most recent call last):
File "/Users/alexanderlee/Desktop/sbpe-1.6.1/builder.py", line 1, in <module>
import cffi
ModuleNotFoundError: No module named 'cffi'
>>>
How do I fix it?

It's probably because you only installed cffi on your Windows, so you probably need to install it on your Mac also.
You can follow rules on the docs:
pip install cffi

cffi is a third-party module. It's installed on your Windows computer but not on your Mac.

Python os.environ throws key error?

I'm accessing an environment variable in a script with os.environ.get and it's throwing a KeyError. It doesn't throw the error from the Python prompt. This is running on OS X 10.11.6, and is Python 2.7.10.
What is going on?
$ python score.py
Traceback (most recent call last):
File "score.py", line 4, in <module>
setup_logging()
File "/score/log.py", line 29, in setup_logging
config = get_config()
File "/score/log.py", line 11, in get_config
environment = os.environ.get('NODE_ENV')
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/UserDict.py", line 23, in __getitem__
raise KeyError(key)
KeyError: 'NODE_ENV'
$ python -c "import os; os.environ.get('NODE_ENV')"
$
As requested, here's the source code for score.py
from __future__ import print_function
from log import get_logger, setup_logging
setup_logging()
log = get_logger('score')
And here's log.py
import json
import os
import sys
from iron_worker import IronWorker
from logbook import Logger, Processor, NestedSetup, StderrHandler, SyslogHandler
IRON_IO_TASK_ID = IronWorker.task_id()
def get_config():
environment = os.environ.get('NODE_ENV')
if environment == 'production':
filename = '../config/config-production.json'
elif environment == 'integration':
filename = '../config/config-integration.json'
else:
filename = '../config/config-dev.json'
with open(filename) as f:
return json.load(f)
def setup_logging():
# This defines a remote Syslog handler
# This will include the TASK ID, if defined
app_name = 'scoreworker'
if IRON_IO_TASK_ID:
app_name += '-' + IRON_IO_TASK_ID
config = get_config()
default_log_handler = NestedSetup([
StderrHandler(),
SyslogHandler(
app_name,
address = (config['host'], config['port']),
level = 'ERROR',
bubble = True
)
])
default_log_handler.push_application()
def get_logger(name):
return Logger(name)

Try running:
find . -name \*.pyc -delete
To delete your .pyc files.
Researching your problem I came across this question, where a user was experiencing the same thing: .get() seemingly raising a KeyError. In that case, it was caused, according to this accepted answer, by a .pyc file which contained code where a dict value was being accessed by key (i.e., mydict['potentially_nonexistent_key']), while the traceback was showing the code from the updated .py file where .get() was used. I have never heard of this happening, where the traceback references current code from a .py file, but shows an error raised by an outdated .pyc file, but it seems to have happened at least once in the history of Python...
It is a long shot, but worth a try I thought.

I encountered a similar error when I set the environment variable without exporting it. So if you do this:
me#host:/# NODE_ENV=foo
You will get this:
me#host:/# python3
Python 3.8.2 (default, Apr 27 2020, 15:53:34)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> node_env = os.environ['NODE_ENV']
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.8/os.py", line 675, in __getitem__
raise KeyError(key) from None
KeyError: 'NODE_ENV'
>>>
But if you do this:
me#host:/# NODE_ENV=foo
me#host:/# export NODE_ENV
It works:
me#host:/# python3
Python 3.8.2 (default, Apr 27 2020, 15:53:34)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> node_env = os.environ['NODE_ENV']
>>> print(node_env)
foo
>>>

Command for windows to delete the .pyc files:
del /S *.pyc

I had the same problem. I solved that by making some corrections on the .env file:
Before:
Key = Value
After my correction:
Key=Value
without blank spaces and worked!

I was getting this error while trying to source from a .env file.
I didn't explicitly export the env vars so I had to change this.
ENVIRONMENT=DEV
to this
export ENVIRONMENT=DEV

Use export a=10 instead of a=10 while setting env variable. Add the same in ~./bashrc to reload the env var wherever you login.
Doing this resolved the issue

I'd recommend you start debugging os.py, for instance, on windows it's being used this implementation:
def get(self, key, failobj=None):
print self.data.__class__
print key
return self.data.get(key.upper(), failobj)
And if I test it with this:
import os
try:
os.environ.get('NODE_ENV')
except Exception as e:
print("-->{0}".format(e.__class__))
os.environ['NODE_ENV'] = "foobar"
try:
os.environ.get('NODE_ENV')
except Exception as e:
print("{0}".format(e.__class__))
The output will be:
<type 'dict'>
PYTHONUSERBASE
<type 'dict'>
APPDATA
<type 'dict'>
NODE_ENV
<type 'dict'>
NODE_ENV
So it makes sense the exception is not spawned reading dict.get docs.
In any case, if you don't want to mess up or debugging the python modules, try cleaning up the *.pyc files, try to set up properly NODE_ENV. And if all that don't work, restart your terminal to clear up.

keyring module is not included while packaging with py2exe

I am making an app using python 2.7 on windows and keyring-3.2.1 . In my python code on eclipse, I used
import keyring
keyring.set_password("service","jsonkey",json_res)
json_res= keyring.get_password("service","jsonkey")
is working fine as I am storing json response in keyring. But, when I converted python code into exe by using py2exe, it shows import error keyring while making dist. Please suggest how to include keyring in py2exe.
Traceback (most recent call last):
File "APP.py", line 8, in <module>
File "keyring\__init__.pyc", line 12, in <module>
File "keyring\core.pyc", line 15, in <module>
File "keyring\util\platform_.pyc", line 4, in <module>
File "keyring\util\platform.pyc", line 29, in <module>
AttributeError: 'module' object has no attribute 'system'
platform_.py code is :
from __future__ import absolute_import
import os
import platform
def _data_root_Windows():
try:
root = os.environ['LOCALAPPDATA']
except KeyError:
# Windows XP
root = os.path.join(os.environ['USERPROFILE'], 'Local Settings')
return os.path.join(root, 'Python Keyring')
def _data_root_Linux():
"""
Use freedesktop.org Base Dir Specfication to determine storage
location.
"""
fallback = os.path.expanduser('~/.local/share')
root = os.environ.get('XDG_DATA_HOME', None) or fallback
return os.path.join(root, 'python_keyring')
# by default, use Unix convention
data_root = globals().get('_data_root_' + platform.system(), _data_root_Linux)
platform.py code is:
import os
import sys
# While we support Python 2.4, use a convoluted technique to import
# platform from the stdlib.
# With Python 2.5 or later, just do "from __future__ import absolute_import"
# and "import platform"
exec('__import__("platform", globals=dict())')
platform = sys.modules['platform']
def _data_root_Windows():
try:
root = os.environ['LOCALAPPDATA']
except KeyError:
# Windows XP
root = os.path.join(os.environ['USERPROFILE'], 'Local Settings')
return os.path.join(root, 'Python Keyring')
def _data_root_Linux():
"""
Use freedesktop.org Base Dir Specfication to determine storage
location.
"""
fallback = os.path.expanduser('~/.local/share')
root = os.environ.get('XDG_DATA_HOME', None) or fallback
return os.path.join(root, 'python_keyring')
# by default, use Unix convention
data_root = globals().get('_data_root_' + platform.system(), _data_root_Linux)

The issue you're reporting is due to an environment that contains invalid modules, perhaps from an improper installation of one version of keyring over another. You will want to ensure that you've removed remnants of the older version of keyring. In particular, make sure there's no file called keyring\util\platform.* in your site-packages.
After doing that, however, you'll encounter another problem. Keyring loads its backend modules programmatically, so py2exe won't detect them.
To work around that, you'll want to add a 'packages' declaration to your py2exe options to specifically include the keyring.backends package. I invoked the following setup.py script with Python 2.7 to convert 'app.py' (which imports keyring) to an exe:
from distutils.core import setup
import py2exe
setup(
console=['app.py'],
options=dict(py2exe=dict(
packages='keyring.backends',
)),
)
The resulting app.exe will import and invoke keyring.