I have a Python 3.5 script which essentially is a Twitter scraper that collects tweets using the tweepy package and its StreamListener function.
Now my script runs perfectly within the Command shell but when I try to run it on the Ubuntu environment of my server I receive a decoding error.
File "/usr/lib/python3/dist-packages/tweepy/streaming.py", line 445, in filter
self._start(async)
File "/usr/lib/python3/dist-packages/tweepy/streaming.py", line 361, in _start
self._run()
File "/usr/lib/python3/dist-packages/tweepy/streaming.py", line 294, in _run
raise exception
File "/usr/lib/python3/dist-packages/tweepy/streaming.py", line 247, in _run
verify=self.verify)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 454, in reque st
prep = self.prepare_request(req)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 388, in prepa
re_request
hooks=merge_hooks(request.hooks, self.hooks),
File "/usr/lib/python3/dist-packages/requests/models.py", line 297, in prepare
self.prepare_auth(auth, url)
File "/usr/lib/python3/dist-packages/requests/models.py", line 490, in prepare _auth
r = auth(self)
File "/usr/lib/python3/dist-packages/requests_oauthlib/oauth1_auth.py", line 7 1, in __call__
r.url = to_native_str(r.url)
File "/usr/lib/python3/dist-packages/requests_oauthlib/oauth1_auth.py", line 1 4, in to_native_str
return string.decode('utf-8')
AttributeError: 'str' object has no attribute 'decode'
This makes sense because you can't decode a string which already is a string. But my question is how I can solve this issue because this is an error occuring within a package?
It looks like the problem comes from requests-oauthlib, you are running an old version of it, and the problem seems to have been fixed since 4.0.1 .
Upgrade your system (the packaged version with Ubuntu 16.10 is 7.0):
sudo apt update && apt upgrade
Or install using pip:
sudo pip3 install requests-oauthlib
Related
I keep trying to install requests, but every time I do it starts, then gives me this error. Is there any way I can get around this or fix this error so that I can use requests?
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\pip\basecommand.py", line 122, in main
status = self.run(options, args)
File "C:\Python27\lib\site-packages\pip\commands\install.py", line 278, in run
requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
File "C:\Python27\lib\site-packages\pip\req.py", line 1265, in prepare_files
req_to_install.extras):
File "C:\Python27\lib\site-packages\pip\_vendor\pkg_resources.py", line 2291, in requires
dm = self._dep_map
File "C:\Python27\lib\site-packages\pip\_vendor\pkg_resources.py", line 2484, in _dep_map
self.__dep_map = self._compute_dependencies()
File "C:\Python27\lib\site-packages\pip\_vendor\pkg_resources.py", line 2508, in _compute_dependencies
parsed = next(parse_requirements(distvers))
File "C:\Python27\lib\site-packages\pip\_vendor\pkg_resources.py", line 45, in <lambda>
next = lambda o: o.next()
File "C:\Python27\lib\site-packages\pip\_vendor\pkg_resources.py", line 2605, in parse_requirements
line, p, specs = scan_list(VERSION,LINE_END,line,p,(1,2),"version spec")
File "C:\Python27\lib\site-packages\pip\_vendor\pkg_resources.py", line 2573, in scan_list
raise ValueError("Expected "+item_name+" in",line,"at",line[p:])
ValueError: ('Expected version spec in', 'charset-normalizer ~=2.0.0', 'at', ' ~=2.0.0')
Storing debug log for failure in C:\Users\l--l\pip\pip.log
I have gotten into the files correctly but I cannot seem to get it to download requests.
Re-Install pip
curl https://bootstrap.pypa.io/pip/2.7/get-pip.py --output get-pip.py
python2.7 get-pip.py
Install requests
python2.7 -m pip install requests
OR Manual install
git clone git://github.com/psf/requests.git
cd requests
python2.7 -m pip install .
I am using Python 2.7, If i try to install Matplotlib I am getting this error if i use "pip install matplotlib"
Exception:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/pip/basecommand.py", line 232, in main
status = self.run(options, args)
File "/usr/local/lib/python2.7/dist-packages/pip/commands/install.py", line 339, in run
requirement_set.prepare_files(finder)
File "/usr/local/lib/python2.7/dist-packages/pip/req/req_set.py", line 355, in prepare_files
do_download, session=self.session,
File "/usr/local/lib/python2.7/dist-packages/pip/download.py", line 782, in unpack_url
session,
File "/usr/local/lib/python2.7/dist-packages/pip/download.py", line 667, in unpack_http_url
from_path, content_type = _download_http_url(link, session, temp_dir)
File "/usr/local/lib/python2.7/dist-packages/pip/download.py", line 843, in _download_http_url
_download_url(resp, link, content_file)
File "/usr/local/lib/python2.7/dist-packages/pip/download.py", line 615, in _download_url
for chunk in progress_indicator(resp_read(4096), 4096):
File "/usr/local/lib/python2.7/dist-packages/pip/utils/ui.py", line 46, in iter
for x in it:
File "/usr/local/lib/python2.7/dist-packages/pip/download.py", line 580, in resp_read
decode_content=False):
File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/response.py", line 256, in stream
data = self.read(amt=amt, decode_content=decode_content)
File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/response.py", line 186, in read
data = self._fp.read(amt)
File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/cachecontrol/filewrapper.py", line 54, in read
self.__callback(self.__buf.getvalue())
File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/cachecontrol/controller.py", line 205, in cache_response
self.serializer.dumps(request, response, body=body),
File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/cachecontrol/serialize.py", line 81, in dumps
).encode("utf8"),
MemoryError"
What might the problem be?
I am using raspberry Pi 2 with a 16gb SD card. I still have 8gb data free but still getting this error.
Kindly help
This error is coming up because, it seems, pip's caching mechanism is trying to read the entire file into memory before caching it… which poses a problem in a limited-memory environment, as matplotlib is ~50mb.
A simpler solution, until pip is patched to use a constant-space caching algorithm, is to run pip with --no-cache-dir to avoid the cache:
$ pip --no-cache-dir install matplotlib
It seems that you have insufficient RAM to build matplotlib from scratch. To overcome that, either turn on swap:
# create swap file of 512 MB
dd if=/dev/zero of=/swapfile bs=1024 count=524288
# modify permissions
chown root:root /swapfile
chmod 0600 /swapfile
# setup swap area
mkswap /swapfile
# turn swap on
swapon /swapfile
Or, if you have raspbian installed on your SD card, you can install matplotlib from the repository:
apt-get install python-matplotlib
--no-cache-dir didn't work for me. I just closed all apps and only then I was able to finish installation.
I finished installing pip on linux, the pip list command works. But when using the pip install command it got the following error:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/pip-6.0.7-py2.7.egg/pip/basecommand.py", line 232, in main
status = self.run(options, args)
File "/usr/local/lib/python2.7/site-packages/pip-6.0.7-py2.7.egg/pip/commands/install.py", line 339, in run
requirement_set.prepare_files(finder)
File "/usr/local/lib/python2.7/site-packages/pip-6.0.7-py2.7.egg/pip/req/req_set.py", line 333, in prepare_files
upgrade=self.upgrade,
File "/usr/local/lib/python2.7/site-packages/pip-6.0.7-py2.7.egg/pip/index.py", line 305, in find_requirement
page = self._get_page(main_index_url, req)
File "/usr/local/lib/python2.7/site-packages/pip-6.0.7-py2.7.egg/pip/index.py", line 783, in _get_page
return HTMLPage.get_page(link, req, session=self.session)
File "/usr/local/lib/python2.7/site-packages/pip-6.0.7-py2.7.egg/pip/index.py", line 872, in get_page
"Cache-Control": "max-age=600",
File "/usr/local/lib/python2.7/site-packages/pip-6.0.7-py2.7.egg/pip/_vendor/requests/sessions.py", line 473, in get
return self.request('GET', url, **kwargs)
File "/usr/local/lib/python2.7/site-packages/pip-6.0.7-py2.7.egg/pip/download.py", line 365, in request
return super(PipSession, self).request(method, url, *args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/pip-6.0.7-py2.7.egg/pip/_vendor/requests/sessions.py", line 461, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python2.7/site-packages/pip-6.0.7-py2.7.egg/pip/_vendor/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python2.7/site-packages/pip-6.0.7-py2.7.egg/pip/_vendor/cachecontrol/adapter.py", line 43, in send
resp = super(CacheControlAdapter, self).send(request, **kw)
File "/usr/local/lib/python2.7/site-packages/pip-6.0.7-py2.7.egg/pip/_vendor/requests/adapters.py", line 370, in send
timeout=timeout
File "/usr/local/lib/python2.7/site-packages/pip-6.0.7-py2.7.egg/pip/_vendor/requests/packages/urllib3/connectionpool.py", line 518, in urlopen
body=body, headers=headers)
File "/usr/local/lib/python2.7/site-packages/pip-6.0.7-py2.7.egg/pip/_vendor/requests/packages/urllib3/connectionpool.py", line 322, in _make_request
self._validate_conn(conn)
File "/usr/local/lib/python2.7/site-packages/pip-6.0.7-py2.7.egg/pip/_vendor/requests/packages/urllib3/connectionpool.py", line 727, in _validate_conn
conn.connect()
File "/usr/local/lib/python2.7/site-packages/pip-6.0.7-py2.7.egg/pip/_vendor/requests/packages/urllib3/connection.py", line 238, in connect
ssl_version=resolved_ssl_version)
File "/usr/local/lib/python2.7/site-packages/pip-6.0.7-py2.7.egg/pip/_vendor/requests/packages/urllib3/util/ssl_.py", line 254, in ssl_wrap_socket
return context.wrap_socket(sock)
File "/usr/local/lib/python2.7/ssl.py", line 350, in wrap_socket
_context=self)
File "/usr/local/lib/python2.7/ssl.py", line 537, in __init__
raise ValueError("check_hostname requires server_hostname")
ValueError: check_hostname requires server_hostname
How can I fix this?
pip 6.1.0 has been released, fixing this issue. You can upgrade with:
pip --trusted-host pypi.python.org install -U pip
to self-upgrade.
Original answer:
This is caused by a change in Python 2.7.9, which urllib3 needs to account for. See issue #543 for that project. Your OpenSSL libraries do not support SNI, which means urllib3 won't pass in the host name to the SSL socket wrapper, but Python 2.7.9 expects the hostname to be passed in anyway for different purposes.
urllib3 is indirectly used by requests (see requests issue 2435), which in turn is being used by pip.
I've opened a ticket to track this from pip's perspective.
The underlying issues have been fixed by the project maintainers, and awaiting a new release. You could install the current development version of pip if you are impatient:
pip install --trusted-host=github.com -U https://github.com/pypa/pip/archive/develop.zip
This'll install pip-6.1.0.dev0, when 6.1.0 is fully released you can upgrade again with pip install -U pip to get the final release from PyPI.
I get the same issue, and find that it can be avoided (pip 6.0.8) in my case as follows
pip --trusted-host pypi.python.org install <thing>
It is related to urllib3.
You can resolve it with urllib3 version 1.25.8.
Download that version of urllib3 manually and install it.
Even though you install thia version, pip will still use its own version.So you have to remove it and replace it.
Usually, installed module is on PythonXX/Lib/site-packages
Delete urllib3 in PythonXX/Lib/site-packages/pip/_vendor
Move "PythonXX/Lib/site-packages/urllib3" to "PythonXX/Lib/site-packages/pip/_vendor".
I encountered this problem and tried the above methods, but not work. I finally find it is because I turn on the VPN. When I turn off the VPN, I can successfully download packages.
I am trying to install a bunch of python dependencies using a requirements.txt file with the following command:
pip install -r requirements.txt
The requirements.txt file has the following packages:
pep8
selenium
paramiko
soappy
nose
wmi
mock
python-keystoneclient
python-novaclient
python-cinderclient
python-swiftclient
python-glanceclient
python-heatclient
python-neutronclient
But when running the pip install command I am getting this error:
Downloading/unpacking PrettyTable>=0.7,<0.8 (from python-keystoneclient->-r requ
irements.txt (line 9))
Cleaning up...
Exception:
Traceback (most recent call last):
File "C:\Python27\VirtualEnvs\test\lib\site-packages\pip\basecommand.py", line
122, in main
status = self.run(options, args)
File "C:\Python27\VirtualEnvs\test\lib\site-packages\pip\commands\install.py",
line 278, in run
requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundl
e=self.bundle)
File "C:\Python27\VirtualEnvs\test\lib\site-packages\pip\req.py", line 1197, i
n prepare_files
do_download,
File "C:\Python27\VirtualEnvs\test\lib\site-packages\pip\req.py", line 1375, i
n unpack_url
self.session,
File "C:\Python27\VirtualEnvs\test\lib\site-packages\pip\download.py", line 54
6, in unpack_http_url
resp = session.get(target_url, stream=True)
File "C:\Python27\VirtualEnvs\test\lib\site-packages\pip\_vendor\requests\sess
ions.py", line 395, in get
return self.request('GET', url, **kwargs)
File "C:\Python27\VirtualEnvs\test\lib\site-packages\pip\download.py", line 23
7, in request
return super(PipSession, self).request(method, url, *args, **kwargs)
File "C:\Python27\VirtualEnvs\test\lib\site-packages\pip\_vendor\requests\sess
ions.py", line 383, in request
resp = self.send(prep, **send_kwargs)
File "C:\Python27\VirtualEnvs\test\lib\site-packages\pip\_vendor\requests\sess
ions.py", line 486, in send
r = adapter.send(request, **kwargs)
File "C:\Python27\VirtualEnvs\test\lib\site-packages\pip\_vendor\requests\adap
ters.py", line 381, in send
raise ProxyError(e)
ProxyError: Cannot connect to proxy. Socket error: [Errno 10054] An existing con
nection was forcibly closed by the remote host.
Storing debug log for failure in C:\Users\cjmarti2\pip\pip.log
But for some reason if I install all packages using pip install <package> they all get installed correctly.
A couple of things to consider;
1) since I am in my company's intranet I am setting a proxy using set https_proxy=https://company-proxy.com:port. 2) Installing the exact same packages using requirements.txt in a Linux machine works fine. Any ideas?
Update:
I was using Python 2.7 for 64 bits. I uninstalled it and installed the 32 bits, and I no longer had this problem. Looks like the python version was the problem, the 64 bits one for some reason cause problems.
On a MBP, following instructions (http://techblog.rosedu.org/python-environment.html), I installed Xcode 4.4.1, brew (brew doctor says all set), and then python. Next, I tried to setup virtualenv:
> $MYPYTHON/bin/python distribute_setup.py
> $MYPYTHON/bin/easy_install pip
> $MYPYTHON/bin/pip install virtualenv
I then created a virtual environment and tried to install django:
> $MYPYTHON/bin/virtualenv $MYENV
> $MYENV/bin/pip install Django
It downloads 98% and then hangs for a while, and finally I get a traceback indicating a timeout:
pcm#pcm-mac[302]% $MYENV/bin/pip install django
Downloading/unpacking django
Downloading Django-1.4.1.tar.gz (7.7Mb): 7.5Mb downloaded
Exception:
Traceback (most recent call last):
File "/Users/pcm/Documents/workspace/virtual/lib/python2.7/site-packages/pip-1.1-py2.7.egg/pip/basecommand.py", line 104, in main
status = self.run(options, args)
File "/Users/pcm/Documents/workspace/virtual/lib/python2.7/site-packages/pip-1.1-py2.7.egg/pip/commands/install.py", line 245, in run
requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
File "/Users/pcm/Documents/workspace/virtual/lib/python2.7/site-packages/pip-1.1-py2.7.egg/pip/req.py", line 985, in prepare_files
self.unpack_url(url, location, self.is_download)
File "/Users/pcm/Documents/workspace/virtual/lib/python2.7/site-packages/pip-1.1-py2.7.egg/pip/req.py", line 1109, in unpack_url
retval = unpack_http_url(link, location, self.download_cache, self.download_dir)
File "/Users/pcm/Documents/workspace/virtual/lib/python2.7/site-packages/pip-1.1-py2.7.egg/pip/download.py", line 451, in unpack_http_url
download_hash = _download_url(resp, link, temp_location)
File "/Users/pcm/Documents/workspace/virtual/lib/python2.7/site-packages/pip-1.1-py2.7.egg/pip/download.py", line 368, in _download_url
chunk = resp.read(4096)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 380, in read
data = self._sock.recv(left)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 553, in read
s = self.fp.read(amt)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 380, in read
data = self._sock.recv(left)
timeout: timed out
I see the same thing, if I just run the version of pip from /usr/local/bin/. I tried other packages (selenium, mock) and they downloaded and installed fine.
I'm at a loss as to what I may be doing wrong, or if there is some issue with django installs via pip. Advice?
Use:
pip --default-timeout=60 install django