Getting strange difference inside Enthought Canopy vs. command line when trying to load and utilize urllib and/or urllib.request
Here's what I mean. I'm running Python 3.5 on MacOS 10.11.3. But I've tried this on Windows 10 machine too, and I'm getting the same results. The difference appears to be between using Canopy and using command line.
I'm trying to do basic screen scraping. Based on reading, I think I should be doing:
from urllib.request import urlopen
html = urlopen("http://pythonscraping.com/pages/page1.html")
print(html.read())
This works at a command prompt.
BUT, inside canopy, this does not work. Inside canopy I get the error
ImportError: No module named request
When Canopy tries to execute the from urllib.request import urlopen
Inside Canopy, THIS is what works:
import urllib
html = urllib.urlopen("http://pythonscraping.com/pages/page1.html")
print(html.read())
I would really like to understand what is happening, as I don't want my Canopy python scripts to fail when I run them outside of Canopy. Also, the Canopy approach does not seem consistent with docs that I've read... I just got there by trial & error.
urllib.request is a module that only exists in Python 3. Enthought Canopy Distribution still ships with a version of Python 2.7 (2.7.10 as of the current version 1.6.2).
In Python 2.x, you have the choice of using either urllib or urllib2, which expose functions like urlopen at the top level (e.g. urllib.urlopen rather than urllib.request.urlopen).
If you want your scripts to be able to run through either Python 3.x or in Enthought Canopy's Python distribution, then there are two possible solutions:
Use requests - this is generally the recommended library to use for interacting with HTTP in Python. It's a third-party module which you can install using standard pip or easy_install, or from the Canopy Package Index.
Your equivalent code would look similar to:
# This allows you to use the print() function inside Python 2.x
from __future__ import print_function
import requests
response = requests.get("http://pythonscraping.com/pages/page1.html")
print(response.text)
Use conditional importing to bring in the current function you need regardless of version. This is just using built-in features of Python and will not require third-party libraries.
Your code would then look similar to:
# This allows you to use the print() function inside Python 2.x
from __future__ import print_function
import sys
try:
# Try importing Python 3's urllib.request first.
from urllib.request import urlopen
except ImportError:
# Looks like we're running Python 2.something.
from urllib import urlopen
response = urlopen("http://pythonscraping.com/pages/page1.html")
# urllib.urlopen's response object is different based
# on Python version.
if sys.version_info[0] < 3:
print(response.read())
else:
# Python 3's urllib responses return the
# stream as a byte-stream, and it's up to you
# to properly set the encoding of the stream. This
# block just checks if the stream has a content-type set
# and if not, it defaults to just using utf-8
encoding = response.headers.get_content_charset()
if not encoding:
encoding = 'utf-8'
print(response.read().decode(encoding))
Related
I read this in addition to MANY other things: Importing requests module does not work
I am using VSCode and python 3.8.
I am able to import it seems any library except "requests"
Given the ages of the previous posts I hope to know what a good current next step could be, please and thank you
import math
import asynchat
import signal
import importlib
import requests <-----Will NOT import
response = requests.get("http://api.open-notify.org/astros.json" [api.open-notify.org])
print(response.text)
print(response)
I think your requests is not well installed. Make sure it's installed with the python you are using with.
Try pip3 install requests.
My code is no longer working after moving from Python 2.7.9 to Python 3.4.3. I am trying to get my little program to run on my RPI and when I tried running with 2.7.9 via python server.py I get this error:
Code Issue:
def setVolume(volume):
volume = max(90, min(30, volume)) #Max Volume Spam Protection
sendPost(speakerAddress+"volume","<volume>"+str(volume)+"</volume>")
You are importing Request object and yet you are trying to
return request.urlopen(address).read()
As far as I know you can keep your old from urllib import request in the new code aswell. Here's what Python 3.4.3 docs say about it: https://docs.python.org/3/library/urllib.request.html
So you should be able to import request and parse from urllib in Python 3.4.3. If that doesn't work for you maybe consider reinstalling your Python 3.4.3 distribution.
I'm writing a sublime text 2 plugin that uses a module SEAPI.py which in itself imports the requests module.
Since sublime text 2 uses it's own embedded python interpreter, it doesn't see the requests module installed in my ubuntu machine (I get the following error: ImportError: No module named requests).
Best solution I could find so far was to copy the 'requests' module (the whole directory of files) from /usr/lib/python2.7/dist-packages/requests into my plugin directory in the sublime text packages dir.
But after that, it says that it can't find the 'urllib3' module.
Is there a better way to import the requests module so that I won't have to copy all the files into my plugin directory ?
The current code I'm using is as follows:
MyPlugin.py
import sublime
import sublime_plugin
import SEAPI
...
SEAPI.py
import requests
try:
import simplejson as json
except:
import json
from time import time, sleep
...
Edit:
The selected answer is correct and fixes my main question, but a different problem exists with using the current version of 'Requests' with the embedded sublime text 2 interpreter. ST2's python is missing various modules which exist in regular 2.7 python (such as 'fileio').
I've solved it with using the 'Requests' module from here:
https://github.com/bgreenlee/sublime-github
And I had to edit the 'urllib3/response.py' file to this:
try:
from cStringIO import StringIO as BytesIO
except ImportError:
pass # _fileio doesn't seem to exist in ST's python in Linux, but we don't need it
You need to bundle full requests distribution with your Python package and then modify Python's sys.path (where it looks for modules) to point to a folder containing requests folder.
Download Requests library from a PyPi and extract it manually under your plugin folder
Before importing requests in your plugin, append the corrcet folder to sys.path to point a folder where it can found requests import
The (untested) code should look like something like this:
import sys
import os
# request-dists is the folder in our plugin
sys.path.append(os.path.join(os.path.dirname(__file__), "requests-dist"))
import requests
This also assumes that requests setup.py does not do any hacks when you install the module using easy_install or pip.
You also could import requests zip directly as Python supports importing from ZIP files, assuming requests is distributed in compatible way. Example (advanced):
https://github.com/miohtama/ztanesh/blob/master/zsh-scripts/python-lib/zipimporter.py
More about sys.path trick (2004)
http://www.johnny-lin.com/cdat_tips/tips_pylang/path.html
Mikko's answer is good, but I may have found a slightly easier way:
import MyAwesomePlugin.requests
"MyAwesomePlugin" being the name of your plugin, of course.
I wanted to install urllib2 package from PyPI but it is not available.
It seems that it has been updated to urllib3, but is there any way to download urllib2 ?
import urllib2
Is that what you want?
If you find any library under http://docs.python.org/ you can always import without installing it.
Update 1:
If you need the source code...
The official Cpython code: http://hg.python.org/cpython/file/3b5fdb5bc597/Lib/urllib
Note The urllib2 module has been split across several modules in
Python 3 named urllib.request and urllib.error. The 2to3 tool will
automatically adapt imports when converting your sources to Python 3.
or try this? http://code.reddit.com/docs/urllib2-pysrc.html
I can't guarantee the integrity for the second alternative link.
I can use urllib.request module with Python 3.1. But when I execute the same program using Python 2.7, an error comes along the lines of;
AttributeError: 'module' object has no attribute 'request'.
I believe this error is because theres no request module in urllib for Python 2.7. Because I need to use tweepy i'll have to stick with Python 2.7 since tweepy does not support Python 3.
So how I can use urllib.request module in Python 2.7?
It is also possible to use six module to make code for both python2 & python3:
from six.moves import urllib
# ...
result = urllib.request.urlopen(url)
Use urllib2.urlopen for Python 2.
Take a look at http://docs.python.org/library/urllib2.html.
The urllib2 module is the predecessor to urllib.request/urllib.error (it has been split into those modules in Python 3.0).