We Keep Coding

What are the corresponding parameters of Requests "data" and "params" in urllib2?

I have been successfully implementing python Requests module to send out POST requests to server with specified
resp = requests.request("POST", url, proxies, data, headers, params, timeout)
However, for a certain reason, I now need to use python urllib2 module to query. For urllib2.urlopen's parameter "data," what I understand is that it helps to form the query string (which is the same as Requests "params"). requests.request's parameter "data," on the other hand, is used to fill the request body.
After searching and reading many posts, examples, and documentations, I still have not been able to figure out what is the corresponding parameter of requests.request's "data" in urllib2.
Any advice is much appreciated! Thanks.
-Janton

It doesn't matter what it is called - it is a matter of passing it in at the right place. For example in this example, the POST data is a dictionary (name can be anything).
The dictionary is urlencoded and the urlencoded name can again be anything but I've picked "postdata", which is the data that is POSTed
import urllib # for the urlencode
import urllib2
searchdict = {'q' : 'urllib2'}
url = 'https://duckduckgo.com/html'
postdata = urllib.urlencode(searchdict)
req = urllib2.Request(url, postdata)
response = urllib2.urlopen(req)
print response.read()
print response.getcode()
If your POST data is plain text (not a Python type such as a dictionary) it can work without urllib.urlencode:
import urllib2
searchstring = 'q=urllib2'
url = 'https://duckduckgo.com/html'
req = urllib2.Request(url, searchstring)
response = urllib2.urlopen(req)
print response.read()
print response.getcode()

Send a string of OCR text to resAPI

I am trying to work with RestfulAPI's on python.
After OCR a pdf, I want to send the text to an restfulAPI to get back retrieve specific words along with their position within the text. I have not manage to send the string of text to the API yet.
Code follows:
import requests
import PyPDF2
import json
url = "http://xxapi.xxapi.org/xxx.util.json"
pdfFileObj = open('/Users/xxx/pdftoOCR.pdf','rb')
pdfReader = PyPDF2.PdfFileReader(pdfFileObj)
pageObj = pdfReader.getPage(1) # To try with the text found in the first page
data = {"text": pageObj.extractText()}
data_json = json.dumps(data)
params = {'text':'string'}
r = requests.post(url, data=data_json, params=params)
r1 = json.loads(r.text)
Although I get a response 200 from the request, The data should come in Json format with the need to poll some token URL (Which I don`t know how to do it either) Also I don't think the request is correct as when I paste the token url to the browser I see an empty Json file (No words, no position) even if I know the piece of text I'm trying to send contains the desired words.
Thanks in advance! I work with OS X , python 3.5

Well, many thanks to #Jose.Cordova.Alvear for resolving this issue
import json
import requests
pdf= open('test.pdf','rb')
url = "http://xxapi.xxapi.org/xxx.util.json"
payload = {
'file' :pdf
}
response = requests.post(url, files=payload)
print response.json()

HTTPS authentication in Python 2.7

we are trying to pass the Virustotal API through our proxy which is getting denied. HTTP websites are accessible using the code but HTTPS is not going through. Request any of you to post few sample codes which would help us. 
import postfile
host = "www.virustotal.com"
selector = "https://www.virustotal.com/vtapi/v2/file/scan"
fields = [("apikey", "-- YOUR API KEY --")]
file_to_send = open("test.txt", "rb").read()
files = [("file", "test.txt", file_to_send)]
json = postfile.post_multipart(host, selector, fields, files)
print json

Two ideas to try
1) without HTTPS you can post to http://www.virustotal.com/vtapi/v2/file/scan
2) try the python requests library
import requests
params = {'apikey': '-YOUR API KEY HERE-'}
files = {'file': ('myfile.exe', open('myfile.exe', 'rb'))}
response = requests.post('https://www.virustotal.com/vtapi/v2/file/scan', files=files, params=params)
json_response = response.json()

How do I read a response from Python Requests?

I have two Python scripts. One uses the Urllib2 library and one uses the Requests library.
I have found Requests easier to implement, but I can't find an equivalent for urlib2's read() function. For example:
...
response = url.urlopen(req)
print response.geturl()
print response.getcode()
data = response.read()
print data
Once I have built up my post url, data = response.read() gives me the content - I am trying to connect to a vcloud director api instance and the response shows the endpoints that I have access to. However if I use the Requests library as follows.....
....
def post_call(username, org, password, key, secret):
endpoint = '<URL ENDPOINT>'
post_url = endpoint + 'sessions'
get_url = endpoint + 'org'
headers = {'Accept':'application/*+xml;version=5.1', \
'Authorization':'Basic '+ base64.b64encode(username + "#" + org + ":" + password), \
'x-id-sec':base64.b64encode(key + ":" + secret)}
print headers
post_call = requests.post(post_url, data=None, headers = headers)
print post_call, "POST call"
print post_call.text, "TEXT"
print post_call.content, "CONTENT"
post_call.status_code, "STATUS CODE"
....
....the print post_call.text and print post_call.content returns nothing, even though the status code equals 200 in the requests post call.
Why isn't my response from Requests returning any text or content?

Requests doesn't have an equivalent to Urlib2's read().
>>> import requests
>>> response = requests.get("http://www.google.com")
>>> print response.content
'<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage"><head>....'
>>> print response.content == response.text
True
It looks like the POST request you are making is returning no content. Which is often the case with a POST request. Perhaps it set a cookie? The status code is telling you that the POST succeeded after all.
Edit for Python 3:
Python now handles data types differently. response.content returns a sequence of bytes (integers that represent ASCII) while response.text is a string (sequence of chars).
Thus,
>>> print response.content == response.text
False
>>> print str(response.content) == response.text
True

If the response is in json you could do something like (python3):
import json
import requests as reqs
# Make the HTTP request.
response = reqs.get('http://demo.ckan.org/api/3/action/group_list')
# Use the json module to load CKAN's response into a dictionary.
response_dict = json.loads(response.text)
for i in response_dict:
print("key: ", i, "val: ", response_dict[i])
To see everything in the response you can use .__dict__:
print(response.__dict__)

If you push, for example image, to some API and want the result address(response) back you could do:
import requests
url = 'https://uguu.se/api.php?d=upload-tool'
data = {"name": filename}
files = {'file': open(full_file_path, 'rb')}
response = requests.post(url, data=data, files=files)
current_url = response.text
print(response.text)

If the Response is in Json you can directly use below method in Python3, no need for json import and json.loads() method:
response.json()

There are three different ways for you to get the contents of the response you have got.
Content - (response.content) - libraries like beautifulsoup accept input as binary
JSON (response.json()) - most of the API calls give response in this format only
Text (response.text) - serves any purpose including regex based search, or dumping data to a file etc.
Depending the type of webpage you are scraping, you can use the attribute accordingly.

Python - POSTing with a urllib2 opener

I have a urllib2 opener, and wish to use it for a POST request with some data.
I am looking to receive the content of the page that I am POSTing to, and also the URL of the page that is returned (I think this is just a 30x code; so something along those lines would be awesome!).
Think of this as the code:
anOpener = urllib2.build_opener(???,???)
anOpener.addheaders = [(???,???),(???,???),...,(???,???)]
# do some other stuff with the opener
data = urllib.urlencode(dictionaryWithPostValues)
pageContent = anOpener.THE_ANSWER_TO_THIS_QUESTION
pageURL = anOpener.THE_SECOND_PART_OF_THIS_QUESTION

This is such a silly question once one realizes the answer.
Just use:
open(URL,data)
for the first part, and like Rachel Sanders mentioned,
geturl()
for the second part.
I really can't figure out how the whole Request/opener thing works though; I couldn't find any nice documentation :/

This page should help you out:
http://www.voidspace.org.uk/python/articles/urllib2.shtml#data
import urllib
import urllib2
url = 'http://www.someserver.com/cgi-bin/register.cgi'
values = {'name' : 'Michael Foord',
'location' : 'Northampton',
'language' : 'Python' }
data = urllib.urlencode(values)
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)
the_page = response.read()
the_url = response.geturl() # <- doc claims this gets the redirected url
It looks like you can also use response.info() to get the Location header directly instead of using .geturl().
Hope that helps!

If you add data to the request the method gets automatically changed to POST. Check out the following example:
import urllib2
import json
url = "http://server.local/x/y"
data = {"name":"JackBauer"}
method = "PUT"
request = urllib2.Request(url)
request.add_header("Content-Type", "application/json")
request.get_method = lambda: method
if data: request.add_data(json.dumps(data))
response = urllib2.urlopen(request)
if response: print response.read()
As i mentioned the lambda is not needed if you use GET/POST.