I'm working on my python project and I migrated from python2.6 to python 3.6. So I had to replace urllib2 with urllib.request ( and .error and .parse ).
But I'm facing an issue I can't solve, here it is...
I want to send a request written in JSON like below :
import json
import urllib2
data= json.dumps({
"jsonrpc":"2.0",
"method":"user.login",
"params":{
"user":"guest",
"password":"password"
}
"id":1,
"auth":None
})
with urllib2 I faced no issue, I just had to create the request with :
req=urllib2.Request("http://myurl/zabbix/api_jsonrpc.php",data,{'Content-type':'application/json})
send it with:
response=urllib2.urlopen(req)
and it was good but now with urllib.request, I have met many error raised by the library. check what I did ( the request is the same within 'data') :
import json
import urllib.request
data= json.dumps({
"jsonrpc":"2.0",
"method":"user.login",
"params":{
"user":"guest",
"password":"password"
}
"id":1,
"auth":None
})
req = urllib.request.Request("http://myurl/zabbix/api_jsonrpc.php",data,{'Content-type':'application/json})
response = urllib.request.urlopen(req)
and I get this error :
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "/tmp/Python-3.6.1/Lib/urllib/request.py", line 223, in urlopen
return opener.open(url, data, timeout)
File "/tmp/Python-3.6.1/Lib/urllib/request.py", line 524, in open
req = meth(req)
File "/tmp/Python-3.6.1/Lib/urllib/request.py", line 1248, in do_request_
raise TypeError(msg)
TypeError: POST data should be bytes, an iterable of bytes, or a file object. It cannot be of type str.
So I inquired about this and learned that I must use the function urllib.parse.urlencode() to convert my request into bytes, so I tried to use it on my request :
import urllib.parse
dataEnc=urllib.parse.urlencode(data)
another error occured :
Traceback (most recent call last):
File "/tmp/Python-3.6.1/Lib/urllib/parse.py", line 842, in urlencode
raise TypeError
TypeError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "/tmp/Python-3.6.1/Lib/urllib/parse.py", line 850, in urlencode
"or mapping object").with_traceback(tb)
File "/tmp/Python-3.6.1/Lib/urllib/parse.py", line 842, in urlencode
raise TypeError
TypeError: not a valid non-string sequence or mapping object
and I realized that json.dumps(data) just convert my array/dictionnary into a string, which is not valid for the urllib.parse.urlencode function, soooooo I retired the json.dumps from data and did this :
import json
import urllib.request
import urllib.parse
data= {
"jsonrpc":"2.0",
"method":"user.login",
"params":{
"user":"guest",
"password":"password"
}
"id":1,
"auth":None
}
dataEnc=urllib.parse.urlencode(data) #this one worked then
req=urllib.request.Request("http://myurl/zabbix/api_jsonrpc.php",data,{'Content-type':'application/json})
response = urllib.request.urlopen(req) #and this one too, but it was too beautiful
then I took a look in the response and got this :
b'{"jsonrpc":"2.0",
"error":{
"code":-32700,
"message":"Parse error",
"data":"Invalid JSON. An error occurred on the server while parsing the JSON text."}
,"id":1}
And I guess it's because the JSON message is not json.dumped !
There is always one element blocking me from doing the request correctly,
so I'm totally stuck with it, if any of you guys have an idea or an alternative I would be so happy.
best Regards
Gozu09
In fact you just need to pass your json data as a byte sequence like this:
data= {
"jsonrpc":"2.0",
"method":"user.login",
"params":{
"user":"guest",
"password":"password"
}
"id":1,
"auth":None
}
req = urllib.request.Request(
"http://myurl/zabbix/api_jsonrpc.php",
data=json.dumps(data).encode(), # Encode a string to a bytes sequence
headers={'Content-type':'application/json}
)
POST data should be bytes, an iterable of bytes, or a file object. It cannot be of type str
This error means that the data argument is expected to be an iterables of bytes.
st = "This is a string"
by = b"This is an iterable of bytes"
by2 = st.encode() # Convert my string to a bytes sequence
st2 = by.decode() # Convert my byte sequence into an UTF-8 string
json.dumps() returns a string, therefore you have to call json.dumps().encode() to convert it into a byte array.
By the way, urlencode is used when you want to convert a string that will be passed as an url argument (i.e: converting spaces characters to "%20"). The output of this method is a string, not a byte array
Related
my code us:
# init scrapy selector
response = Selector(text=content)
json_data = json.loads(script.get() for script in (re.findall(r'dataLayer\.push\(([^)]+)'),response.css('script::text'))).group(1)
print(json_data)
# debug data extraction logic
HummartScraper.parse_product(HummartScraper, '')
'
the output error is:
Traceback (most recent call last):
File "hummart2.py", line 86, in parse_product
json_data = json.loads(script.get() for script in (re.findall(r'dataLayer\.push\(([^)]+)'),response.css('script::text'))).group(1)
TypeError: findall() missing 1 required positional argument: 'string'
why is this error getting.
For a single dataLayer:
data_layer = response.css('script::text').re_first(r'dataLayer\.push\(([^)]+)')
data = json.loads(data_layer)
You can use response.css(...).re() to get a list of matches.
but this give me this type of error:
File "hummart2.py", line 88, in parse_product
data = json.loads(data_layer_raw)[1]
File "/home/danish-khan/miniconda3/lib/python3.7/json/__init__.py", line 341, in loads
raise TypeError(f'the JSON object must be str, bytes or bytearray, '
TypeError: the JSON object must be str, bytes or bytearray, not NoneType
I am trying to provide formdata to a scrapy.FormRequest object. The formdata is a dict of the following structure:
{
"param1": [
{
"paramA": "valueA",
"paramB": "valueB"
}
]
}
via equivalent to the following code, run in scrapy shell:
from scrapy import FormRequest
url = 'www.example.com'
method_post = 'POST'
formdata = <the above dict>
fr = FormRequest(url=url, method=method_post, formdata=formdata)
fetch(fr)
and in response I get the following error:
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/Users/chhk/.local/share/virtualenvs/project/lib/python3.6/site-packages/scrapy/http/request/form.py", line 31, in __init__
querystr = _urlencode(items, self.encoding)
File "/Users/chhk/.local/share/virtualenvs/project/lib/python3.6/site-packages/scrapy/http/request/form.py", line 66, in _urlencode
for k, vs in seq
File "/Users/chhk/.local/share/virtualenvs/project/lib/python3.6/site-packages/scrapy/http/request/form.py", line 67, in <listcomp>
for v in (vs if is_listlike(vs) else [vs])]
File "/Users/chhk/.local/share/virtualenvs/project/lib/python3.6/site-packages/scrapy/utils/python.py", line 119, in to_bytes
'object, got %s' % type(text).__name__)
TypeError: to_bytes must receive a unicode, str or bytes object, got dict
I have tried a variety of solutions, including the whole thing as a string, with various escape characters, and variations on the dict to make it more agreeable, but none of the solutions that remove this error work for the request (I get a 400 response).
I know that the formdata and that everything else I am doing is correct, in that I have replicated it successfully in curl (formdata was provided via -d formdata.txt).
Is there a way around FormRequest's inability to deal with complex dict structures? Or am I missing something?
Instead of formdata you can try to use body parameter. Example:
FormRequest(url=url, method=method_post, body=json.dumps(formdata))
I have an issue with json on Python3.
I try to get the "pairs" from string of URL: https://wex.nz/api/3/info
This is my code:
import urllib.request
import json
url= 'https://wex.nz/api/3/info'
content=urllib.request.urlopen(url)
for line in content:
liste=json.loads(line)
pairliste=liste['pairs']
print(pairliste)
This is my error:
Traceback (most recent call last):
File "/home/lennart/Documents/Test/Test.py", line 8, in <module>
liste=json.loads(line)
File "/usr/lib/python3.5/json/__init__.py", line 312, in loads
s.__class__.__name__))
TypeError: the JSON object must be str, not 'bytes'
Thanks to roganjosh.
This fixed my issue:
import urllib.request
import json
url= 'https://wex.nz/api/3/info'
content = urllib.request.urlopen(url)
data =json.loads(content.read().decode())
print(data)
I am partially able to work with json saved as file:
#! /usr/bin/python3
import json
from pprint import pprint
json_file='a.json'
json_data=open(json_file)
data = json.load(json_data)
json_data.close()
print(data[10])
But I am trying to achieve the same from data directly from web. I am trying with the accepted answer here:
#! /usr/bin/python3
from urllib.request import urlopen
import json
from pprint import pprint
jsonget=urlopen("http://api.crossref.org/works?query.author=Rudra+Banerjee")
data = json.load(jsonget)
pprint(data)
which is giving me error:
Traceback (most recent call last):
File "i.py", line 10, in <module>
data = json.load(jsonget)
File "/usr/lib64/python3.5/json/__init__.py", line 268, in load
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "/usr/lib64/python3.5/json/__init__.py", line 312, in loads
s.__class__.__name__))
TypeError: the JSON object must be str, not 'bytes'
What is going wrong here?
Changing the code as par Charlie's reply to:
jsonget=str(urlopen("http://api.crossref.org/works?query.author=Rudra+Banerjee"))
data = json.load(jsonget)
pprint(jsonget)
breaks at json.load:
Traceback (most recent call last):
File "i.py", line 9, in <module>
data = json.load(jsonget)
File "/usr/lib64/python3.5/json/__init__.py", line 265, in load
return loads(fp.read(),
AttributeError: 'str' object has no attribute 'read'
It's actually telling you the answer: you're getting back a byte array, where in Python 3 a string is different because of dealing with unicode. In Python 2.7, it would work. You should be able to fix it by converting your bytes explicitly to a string with
jsonget=str(urlopen("http://api.crossref.org/works?query.author=Rudra+Banerjee")_
I thought json.load() should be able to read objects exactly like http.client.HTTPResponse, but it seems to be tripping up on its read() being a bytes-like object. (I am using Python 3.3.) To my surprise, I found no resource directly addressing this use though I thought this was a primary use case.
import urllib.request, json
# Y!F url
yf = 'http://d.yimg.com/autoc.finance.yahoo.com/autoc'
# Mock lookup
data = urllib.parse.urlencode({'query': 'Ford', 'callback': 'YAHOO.Finance.SymbolSuggest.ssCallback'})
data = data.encode('utf-8')
request = urllib.request.Request(yf)
request.add_header('User-Agent','Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/536.26.14 (KHTML, like Gecko) Version/6.0.1 Safari/536.26.14')
request.add_header('Content-type','text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8;charset=utf-8')
request.add_header('Accept','text/plain')
mock = urllib.request.urlopen(request, data)
json.load(mock)
This results in the error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/json/__init__.py", line 264, in load
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/json/__init__.py", line 309, in loads
return _default_decoder.decode(s)
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/json/decoder.py", line 352, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
TypeError: can't use a string pattern on a bytes-like object
This has been solved in a previous thread: Python 3, let json object accept bytes or let urlopen output strings
(HT: Matthew Trevor)
Also, what Yahoo! returns here is not just to the JSON object but also a YAHOO.Finance.SymbolSuggest.ssCallback() wrapper. Stripping that fixes things. (Though still sad it's needed.)
This works:
import urllib.request, json, re
# Y!F url
yf = 'http://d.yimg.com/autoc.finance.yahoo.com/autoc'
# Mock lookup
data = urllib.parse.urlencode({'query': 'Ford', 'callback': 'YAHOO.Finance.SymbolSuggest.ssCallback'})
data = data.encode('utf-8')
request = urllib.request.Request(yf)
response = urllib.request.urlopen(request, data)
j = json.loads(re.search(r'{.*}',response.readall().decode('utf-8')).group())