I use Python eventlet pool and requests module to speed up my HTTPS request.
But it makes the requests process slower, I try to find the answer to the question these days.
I find some use cases for Python eventlet. such as Speed of fetching web pages with Eventlet and Python? and https://eventlet.net/doc/ these documents mentioned.
urls = ["http://www.google.com/intl/en_ALL/images/logo.gif",
"https://wiki.secondlife.com/w/images/secondlife.jpg",
"http://us.i1.yimg.com/us.yimg.com/i/ww/beta/y3.gif"]
import eventlet
from eventlet.green import urllib2
def fetch(url):
return urllib2.urlopen(url).read()
pool = eventlet.GreenPool()
for body in pool.imap(fetch, urls):
print "got body", len(body)
urls = [
"http://www.google.com/intl/en_ALL/images/logo.gif",
"http://python.org/images/python-logo.gif",
"http://us.i1.yimg.com/us.yimg.com/i/ww/beta/y3.gif",
]
import eventlet
from eventlet.green.urllib.request import urlopen
def fetch(url):
return urlopen(url).read()
pool = eventlet.GreenPool()
for body in pool.imap(fetch, urls):
print("got body", len(body))
I notice only Python module urllib and urllib2 are used in these examples, and I can not import urllib3 from event.green module.
Python 2.7.5 (default, Nov 16 2020, 22:23:17)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from eventlet.green import urllib3
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: cannot import name urllib3
>>>
When I check the Python requests module, I find it uses urllib3 module.
[root#ci-4183549-pzhang requests]# ls
adapters.py api.py auth.py certs.py compat.py cookies.py exceptions.py hooks.py __init__.py models.py packages sessions.pyo status_codes.pyo structures.pyo utils.pyo
adapters.pyc api.pyc auth.pyc certs.pyc compat.pyc cookies.pyc exceptions.pyc hooks.pyc __init__.pyc models.pyc sessions.py status_codes.py structures.py utils.py
adapters.pyo api.pyo auth.pyo certs.pyo compat.pyo cookies.pyo exceptions.pyo hooks.pyo __init__.pyo models.pyo sessions.pyc status_codes.pyc structures.pyc utils.pyc
[root#ci-4183549-pzhang requests]# ls packages/
chardet __init__.py __init__.pyc __init__.pyo urllib3
The question is:
Does it mean the requests module can not be eventlet monkeypatched, even if I do eventlet.monkey_patch() at the beginning of my script? it will not work correctly, will it?
Any help or hint is appreciated.
Related
Im trying to run my Python app but I am getting the following error message:
File "services/user.py", line 1, in <module>
2020-12-01 15:28:34 -0800 [web][cinema-app-web-3-6774f77795-k5mqz]: from services import root_dir, nice_json
2020-12-01 15:28:34 -0800 [web][cinema-app-web-3-6774f77795-k5mqz]: ImportError: cannot import name root_dir
This is the folder structure I have today:
cinema-app
setup.py
requirements.txt
database
bookings.json
movies.json
showtimes.json
users.json
services
__init__.py
bookings.py
movies.py
showtimes.py
user.py
I then run the app by using:
python services/user.py
but then I get the error message above.
Here is the content of my __init__.py:
import os
import json
from flask import make_response
def root_dir():
""" Returns root director for this project """
return os.path.dirname(os.path.realpath(__file__ + '/..'))
def nice_json(arg):
response = make_response(json.dumps(arg, sort_keys = True, indent=4))
response.headers['Content-type'] = "application/json"
return response
Here is the user.py file content:
from services import root_dir, nice_json
from flask import Flask
from werkzeug.exceptions import NotFound, ServiceUnavailable
import json
import requests
app = Flask(__name__)
with open("{}/database/users.json".format(root_dir()), "r") as f:
users = json.load(f)
#app.route("/", methods=['GET'])
def hello():
return nice_json({
"uri": "/",
"subresource_uris": {
"users": "/users",
"user": "/users/<username>",
"bookings": "/users/<username>/bookings",
"suggested": "/users/<username>/suggested"
}
})
#app.route("/users/<username>/suggested", methods=['GET'])
def user_suggested(username):
"""
Returns movie suggestions. The algorithm returns a list of 3 top ranked
movies that the user has not yet booked.
:param username:
:return: Suggested movies
"""
raise NotImplementedError()
if __name__ == "__main__":
app.run(port=5000, debug=True)
Any help would be much appreciated
If you want to make a package, your code is right. To simplify, I rewrite the __init__.py and user.py:
user.py:
from services import root_dir, nice_json
def main():
print(root_dir)
print(nice_json)
if __name__ == '__main__':
main()
__init__.py:
def root_dir():
pass
def nice_json():
pass
And you can run in terminal this way:
user#computer$ python -m services/user
<function root_dir at 0x7fd1d0083670>
<function nice_json at 0x7fd1d0083700>
Here dont use the 'py' extension, the '-m' options tells python this is a modular import. You can still import your package from python:
user#computer$ python
Python 2.7.18 (default, Aug 4 2020, 11:16:42)
[GCC 9.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from services import user
>>> user.main()
<function root_dir at 0x7fa2e21a2dd0>
<function nice_json at 0x7fa2e21b01d0>
Code this way you can use your package when it is in PATH or in site-packages python dir.
I am facing a problem in python. Tho the error is quite common, but since i am bit new to python, unable to comprehend the source hence asking you all. There are 2 modules: session.py and objects.py.
session.py
import copy
import pymongo
import spacy
import tweepy
import objects
objects.py:
import re
def refresh (sv = sv, obj = ''):
return 0
now, in python shell, i am getting the error before even executing objects.py:
$ python
Python 2.7.13 (default, Sep 26 2018, 18:42:22)
[GCC 6.3.0 20170516] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import session
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "session.py", line 6, in <module>
import objects
File "objects.py", line 3, in <module>
def refresh (sv = sv, obj = ''):
NameError: name 'sv' is not defined
>>>
I came from perl background to maybe missing some very common thing, but still i am able to do this:
>>> def ff(t): print t
...
In above, whitout defining t, it is working while in objects.py, how can i define sv without starting execution?
To do
import itertools
dynamically, I can do the following
import importlib
importlib.import_module('itertools')
But what should I do to do the following dynamically
import itertools as iters
Problem Context:
I need to import a different version (0.10) of a module('pika' in my case), installed in a separate directory instead of default version(0.9).
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Welcome to Python !!
>>> import importlib
>>> import pika
>>> pika.__version__
'0.9.14'
>>> import scale.lib.hypervisor.esx65.pika_3_5 as pika35
>>> pika35.__version__
'0.10.0'
>>> importlib.import_module('scale.lib.hypervisor.esx65.pika_3_5')
<module 'scale.lib.hypervisor.esx65.pika_3_5' from 'scale/lib/hypervisor/esx65/pika_3_5/__init__.pyc'>
As we can see regular imports are working fine. However when importing dynamically, importing relative to the location is causing issues. As per importlib.import_module documentaion , following should work but it doesn't.
>>> importlib.import_module('pika_3_5', 'scale.lib.hypervisor.esx65')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
ImportError: No module named pika_3_5
And when trying to import 'pika' from a relative path, which should fail as there is no module pika under the relative path, it is still importing it from default module.
>>> importlib.import_module('pika', 'scale.lib.hypervisor.esx65.pika_3_5')
<module 'pika' from '/usr/local/lib/python2.7/dist-packages/pika/__init__.pyc'>
>>>
What is it that I am missing ? I mainly want to do the following dynamically.
import scale.lib.hypervisor.esx65.pika_3_5 as pika
To do
import itertools
dynamically, I can do the following
import importlib
importlib.import_module('itertools')
No, you do the following:
import importlib
itertools = importlib.import_module('itertools')
Similarly, to replicate import itertools as iters, you do
import importlib
iters = importlib.import_module('itertools')
importlib.import_module doesn't care what you call the module. as is not and cannot be part of importlib.import_module's functionality. It just gives you the module object; what you call that object is up to you.
As for your context, you've misunderstood what a relative import is. The second argument to importlib.import_module has nothing to do with from imports, and importlib.import_module('thing', 'whatever') is not supposed to be equivalent to from whatever import thing.
If you want to do
import scale.lib.hypervisor.esx65.pika_3_5 as pika
that's pika = importlib.import_module('scale.lib.hypervisor.esx65.pika_3_5'). The second argument doesn't enter the picture. You seem to think this is somehow not dynamic, but it's as dynamic as any other importlib call.
I wrote a Python module which is just two methods, help and check. Check just takes a string file name and does something with it. When I import the module, there are no methods in it. Only __name__ and the like, but neither check nor help appears in the dir.
I am just importing the file. lyricCheck.py
Here's my code in lyricCheck.py:
#!/usr/bin/python
#python lyric checker
#Jim Boulter
#January 19, 2015
#Copyright 2015
import sys
import urllib2
import fileinput
from decimal import *
from re import *
from pygoogle import pygoogle
def help():
print 'usage: python check.py filename.txt\n'
print 'input line structure: artist name; song title\n'
def check(filename):
if(str(filename).lower == "help" or str(filename).lower == "-h"):
help()
return
#do lots of other stuff
If you are creating your package like this, and I believe you are, you need to import your module from your package:
~/tmp$ mkdir lyricCheck
~/tmp$ cd lyricCheck/
~/tmp/lyricCheck$ touch __init__.py
~/tmp/lyricCheck$ cat > lyricCheck.py
#!/usr/bin/python
#python lyric checker
#Jim Boulter
#January 19, 2015
#Copyright 2015
import sys
import urllib2
import fileinput
from decimal import *
from re import *
from pygoogle import pygoogle
def help():
print 'usage: python check.py filename.txt\n'
print 'input line structure: artist name; song title\n'
def check(filename):
if(str(filename).lower == "help" or str(filename).lower == "-h"):
help()
return
#do lots of other stuff
~/tmp/lyricCheck$ cd ..
~/tmp$ python
Python 2.7.6 (default, Mar 22 2014, 22:59:56)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import lyricCheck
>>> dir(lyricCheck)
['__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__']
Here's my payoff, and this should have worked if I had pygoogle, so this is how I know I found the issue:
>>> from lyricCheck import lyricCheck
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "lyricCheck/lyricCheck.py", line 12, in <module>
from pygoogle import pygoogle
ImportError: No module named pygoogle
You can put this in your __init__.py file to import the functions from the lower level module and make them available right at the package level:
from lyricCheck import help, check
Also note, when you do this:
from decimal import *
from re import *
You dump all the names in those modules into your module's namespace. It's generally considered better to declare them individually.
I am trying to run dripls and when I install everything including fabric, apache2 and setup appropriate configuration files
I have the following problem
AttributeError: 'module' object has no attribute 'url'
its from the package cherrypy
On this line
app = {
'root_url': cherrypy.url()
}
Could anyone point out what might be the problem?
**Update: **
Problematic script (full)
import cherrypy
import urlparse
import uuid
import os
# Service
socket = '0.0.0.0'
dripls_main_site_port = 8080
thread_pool = 10
bin_path = os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(os.path.realpath(__file__)))), "bin")
pidfile = os.path.join(bin_path, "dripls.pid")
error_log = os.path.join(bin_path, "error_log.log")
access_log = os.path.join(bin_path, "access_log.log")
app = {
'root_url': cherrypy.url()
}
# Shaper path
shaper_path = os.path.join(bin_path, "set_ts_lo.sh")
# Shape port range
shape_start_port = 10000
shape_end_port = 11000
# Environment overrides
if os.path.exists(os.path.join(os.path.dirname(os.path.realpath(__file__)), "env.py")):
from env import *
else:
from local import *
port = int(dripls_main_site_port)
# Final url rewrite. Hack to battle the fact that cherrypy is behind a proxy on different port
def get_final_url(path, args):
cherrypy_url = cherrypy.url(path, args)
scheme, netloc, path, qs, anchor = urlparse.urlsplit(cherrypy_url)
return urlparse.urlunsplit( (scheme, urlparse.urlsplit(app['root_url'])[1], path, qs, anchor))
def get_seeded_cid(cid):
return "{0}_{1}".format(cid, uuid.uuid4().hex)"
Update :
Python 2.7.3 (default, Apr 10 2013, 05:46:21)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import cherrypy
>>> cherrypy.url
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'url'
>>> print cherrypy.__file__
/usr/lib/pymodules/python2.7/cherrypy/__init__.pyc
>>>
That should work:
>>> import cherrypy
>>> cherrypy.url
<function url at 0xb6d8f80c>
Make sure that you didn't named the script cherrypy.py. If you did, it will prevent importing of cherrypy package.
Rename the file, and make sure there's no cherrypy.pyc file remained.
Sorry but I found out the problem.
It did not work if you install cherrypy using
sudo apt-get install python-cherrypy
You have install pip using
sudo apt-get install python-pip
and then
sudo pip install CherryPy