We Keep Coding

How to add dictionary in url as query param python request

Actually i am calling 3rd party API and requirement in to add json dictionary as it is. refer below URL example
https://pguat.paytm.com/oltp/HANDLER_INTERNAL/getTxnStatus?JsonData={"MID":"MID117185435","ORDERID":"ORDR4o22310421111",
"CHECKSUMHASH":
"NU9wPEWxmbOTFL2%2FUKr3lk6fScfnLy8wORc3YRylsyEsr2MLRPn%2F3DRePtFEK55ZcfdTj7mY9vS2qh%2Bsm7oTRx%2Fx4BDlvZBj%2F8Sxw6s%2F9io%3D"}
The query param name in "JsonData" and data should be in {} brackets.
import requests
import json
from urllib.parse import urlencode, quote_plus
import urllib.request
import urllib
data = '{"MID":"MID117185435","ORDERID":"ORDR4o22310421111","CHECKSUMHASH":"omcrIRuqDP0v%2Fa2DXTlVI4XtzvmuIW56jlXtGEp3S%2B2b1h9nU9cfJx5ZO2Hp%2FAN%2F%2FyF%2F01DxmoV1VHJk%2B0ZKHrYxqvDMJa9IOcldrfZY1VI%3D"}'
jsonData = data
uri = 'https://pguat.paytm.com/oltp/HANDLER_INTERNAL/getTxnStatus?jsonData='+str(quote_plus(data))
r = requests.get(uri)
print(r.url)
print(r.json)
print(r.json())
print(r.url) output on console
https://pguat.paytm.com/oltp/HANDLER_INTERNAL/getTxnStatus?jsonData=%7B%22MID%22%3A%22MEDTPA37902117185435%22%2C%22ORDERID%22%3A%22medipta1521537969o72718962111%22%2C%22CHECKSUMHASH%22%3A%22omcrIRuqDP0v%252Fa2DXTlVI4XtzvmuIW56jlXtGEp3S%252B2b1h9nU9cfJx5ZO2Hp%252FAN%252F%252FyF%252F01DxmoV1VHJk%252B0ZKHrYxqvDMJa9IOcldrfZY1VI%253D%22%7D
It converts {} to %7B and i want {} as it is..
Plz help ...

You need to undo quote_plus by importing and using unquote_plus.
I didn’t test against your url, just against your string.
When I print your uri string I get this as my output:
https://pguat.paytm.com/oltp/HANDLER_INTERNAL/getTxnStatus?jsonData=%7B%22MID%22%3A%22MID117185435%22%2C%22ORDERID%22%3A%22ORDR4o22310421111%22%2C%22CHECKSUMHASH%22%3A%22omcrIRuqDP0v%252Fa2DXTlVI4XtzvmuIW56jlXtGEp3S%252B2b1h9nU9cfJx5ZO2Hp%252FAN%252F%252FyF%252F01DxmoV1VHJk%252B0ZKHrYxqvDMJa9IOcldrfZY1VI%253D%22%7D
If I surround it like this:
print(str(unquote_plus(uri)))
I get this as output:
https://pguat.paytm.com/oltp/HANDLER_INTERNAL/getTxnStatus?jsonData={"MID":"MID117185435","ORDERID":"ORDR4o22310421111","CHECKSUMHASH":"omcrIRuqDP0v%2Fa2DXTlVI4XtzvmuIW56jlXtGEp3S%2B2b1h9nU9cfJx5ZO2Hp%2FAN%2F%2FyF%2F01DxmoV1VHJk%2B0ZKHrYxqvDMJa9IOcldrfZY1VI%3D"}

Python Parse JSON Response from URL

I'm am wanting to get information about my Hue lights using a python program. I am ok with sorting the information once I get it, but I am struggling to load in the JSON info. It is sent as a JSON response. My code is as follows:
import requests
import json
response= requests.get('http://192.168.1.102/api/F5La7UpN6XueJZUts1QdyBBbIU8dEvaT1EZs1Ut0/lights')
data = json.load(response)
print(data)
When this is run, all I get is the error:
in load return loads(fp.read(),
Response' object has no attribute 'read'

The problem is you are passing in the actual response which consists of more than just the content. You need to pull the content out of the response:
import requests
r = requests.get('https://github.com/timeline.json')
print r.text
# The Requests library also comes with a built-in JSON decoder,
# just in case you have to deal with JSON data
import requests
r = requests.get('https://github.com/timeline.json')
print r.json
http://www.pythonforbeginners.com/requests/using-requests-in-python
Looks like it will parse the JSON for you already...

Use response.content to access response content and json.loads method instead of json.load:
data = json.loads(response.content)
print data

Using Urllib2 with 511 api that requires token

So, all I want to do is send a request to the 511 api and return the train times from the train station. I can do that using the full url request, but I would like to be able to set values without paste-ing together a string and then sending that string. I want to have the api return the train times for different stations. I see other requests that use headers, but I don't know how to use headers with a request and am confused by the documentation.
This works...
urllib2.Request("http://services.my511.org/Transit2.0/GetNextDeparturesByStopCode.aspx?token=xxxx-xxx&stopcode=70142")
response = urllib2.urlopen(request)
the_page = response.read()
I want to be able to set values like this...
token = xxx-xxxx
stopcode = 70142
url = "http://services.my511.org/Transit2.0/GetNextDeparturesByStopCode.aspx?"
... and then put them together like this...
urllib2.Request(url,token, stopcode)
and get the same result.

The string formatting documentation would be a good place to start to learn more about different ways to plug in values.
val1 = 'test'
val2 = 'test2'
url = "https://www.example.com/{0}/blah/{1}".format(val1, val2)
urllib2.Request(url)

The missing piece is "urllib" needs to be used along with "urllib2". Specifically, the function urllib.urlencode() returns the encoded versions of the values.
From the urllib documentation here
import urllib
query_args = { 'q':'query string', 'foo':'bar' }
encoded_args = urllib.urlencode(query_args)
print 'Encoded:', encoded_args
url = 'http://localhost:8080/?' + encoded_args
print urllib.urlopen(url).read()
So the corrected code is as follows:
import urllib
import urllib2
token = xxx-xxxx
stopcode = 70142
query_args = {"token":token, "stopcode":stopcode}
encoded_args = urllib.urlencode(query_args)
request = urllib2.Request(url+encoded_args)
response = urllib2.urlopen(request)
print(response.read())
Actually, it is a million times easier to use requests package and not urllib, urllib2. All that code above can be replaced with this:
import requests
token = xxx-xxxx
stopcode = 70142
query_args = {"token":token, "stopcode":stopcode}
r = request.get(url, params = query_args)
r.text

Parse XML from URL into python object

The goodreads website has this API for accessing a user's 'shelves:' https://www.goodreads.com/review/list/20990068.xml?key=nGvCqaQ6tn9w4HNpW8kquw&v=2&shelf=toread
It returns XML. I'm trying to create a django project that shows books on a shelf from this API. I'm looking to find out how (or if there is a better way than) to write my view so I can pass an object to my template. Currently, this is what I'm doing:
import urllib2
def homepage(request):
file = urllib2.urlopen('https://www.goodreads.com/review/list/20990068.xml?key=nGvCqaQ6tn9w4HNpW8kquw&v=2&shelf=toread')
data = file.read()
file.close()
dom = parseString(data)
I'm not entirely sure how to manipulate this object if I'm doing this correctly. I'm following this tutorial.

I'd use xmltodict to make a python dictionary out of the XML data structure and pass this dictionary to the template inside the context:
import urllib2
import xmltodict
def homepage(request):
file = urllib2.urlopen('https://www.goodreads.com/review/list/20990068.xml?key=nGvCqaQ6tn9w4HNpW8kquw&v=2&shelf=toread')
data = file.read()
file.close()
data = xmltodict.parse(data)
return render_to_response('my_template.html', {'data': data})

xmltodict using requests
import requests
import xmltodict
url = "https://yoursite/your.xml"
response = requests.get(url)
data = xmltodict.parse(response.content)

xmltodict using urllib3
import traceback
import urllib3
import xmltodict
def getxml():
url = "https://yoursite/your.xml"
http = urllib3.PoolManager()
response = http.request('GET', url)
try:
data = xmltodict.parse(response.data)
except:
print("Failed to parse xml from response (%s)" % traceback.format_exc())
return data

How do I read a response from Python Requests?

I have two Python scripts. One uses the Urllib2 library and one uses the Requests library.
I have found Requests easier to implement, but I can't find an equivalent for urlib2's read() function. For example:
...
response = url.urlopen(req)
print response.geturl()
print response.getcode()
data = response.read()
print data
Once I have built up my post url, data = response.read() gives me the content - I am trying to connect to a vcloud director api instance and the response shows the endpoints that I have access to. However if I use the Requests library as follows.....
....
def post_call(username, org, password, key, secret):
endpoint = '<URL ENDPOINT>'
post_url = endpoint + 'sessions'
get_url = endpoint + 'org'
headers = {'Accept':'application/*+xml;version=5.1', \
'Authorization':'Basic '+ base64.b64encode(username + "#" + org + ":" + password), \
'x-id-sec':base64.b64encode(key + ":" + secret)}
print headers
post_call = requests.post(post_url, data=None, headers = headers)
print post_call, "POST call"
print post_call.text, "TEXT"
print post_call.content, "CONTENT"
post_call.status_code, "STATUS CODE"
....
....the print post_call.text and print post_call.content returns nothing, even though the status code equals 200 in the requests post call.
Why isn't my response from Requests returning any text or content?

Requests doesn't have an equivalent to Urlib2's read().
>>> import requests
>>> response = requests.get("http://www.google.com")
>>> print response.content
'<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage"><head>....'
>>> print response.content == response.text
True
It looks like the POST request you are making is returning no content. Which is often the case with a POST request. Perhaps it set a cookie? The status code is telling you that the POST succeeded after all.
Edit for Python 3:
Python now handles data types differently. response.content returns a sequence of bytes (integers that represent ASCII) while response.text is a string (sequence of chars).
Thus,
>>> print response.content == response.text
False
>>> print str(response.content) == response.text
True

If the response is in json you could do something like (python3):
import json
import requests as reqs
# Make the HTTP request.
response = reqs.get('http://demo.ckan.org/api/3/action/group_list')
# Use the json module to load CKAN's response into a dictionary.
response_dict = json.loads(response.text)
for i in response_dict:
print("key: ", i, "val: ", response_dict[i])
To see everything in the response you can use .__dict__:
print(response.__dict__)

If you push, for example image, to some API and want the result address(response) back you could do:
import requests
url = 'https://uguu.se/api.php?d=upload-tool'
data = {"name": filename}
files = {'file': open(full_file_path, 'rb')}
response = requests.post(url, data=data, files=files)
current_url = response.text
print(response.text)

If the Response is in Json you can directly use below method in Python3, no need for json import and json.loads() method:
response.json()

There are three different ways for you to get the contents of the response you have got.
Content - (response.content) - libraries like beautifulsoup accept input as binary
JSON (response.json()) - most of the API calls give response in this format only
Text (response.text) - serves any purpose including regex based search, or dumping data to a file etc.
Depending the type of webpage you are scraping, you can use the attribute accordingly.