I have been following this tutorial to download some page info from fb.
I am using Python 3.5 and the tutorial is using python2.
I was having some issues at first with a HTTP error code 400, basically saying I had to use the https protocol instead of http. So I have test in idle now that the data in coming and it looks to a novice like me as JSON. But when I try passing it to the json.loads it is giving this error
Traceback (most recent call last):
File "C:\Users\Levo\Desktop\facebookscrape.py", line 38, in <module>
testFacebookPageData(page_id, access_token)
File "C:\Users\Levo\Desktop\facebookscrape.py", line 34, in testFacebookPageData
data = json.loads(requests_until_succeed(url))
File "C:\Users\Levo\AppData\Local\Programs\Python\Python35\lib\json\__init__.py", line 312, in loads
s.__class__.__name__))
TypeError: the JSON object must be str, not 'bytes'
Apart from using the urllib library instead of the urllib2 library and https instead of http I don't what I am doing wrong? is the encryption of https the problem??
def requests_until_succeed(url):
req = urllib.request.Request(url)
success = False
while success is False:
try:
response = urllib.request.urlopen(req)
if response.getcode() == 200:
success = True
except Exception as e:
print(e)
time.sleep(5)
print ("Error for URL %s: %s" % (url, datetime.datetime.now()))
return response.read()
def testFacebookPageData(page_id, access_token):
base = "https://graph.facebook.com/v2.6"
node = "/" + page_id + "/feed"
parameters = "/?access_token=%s" % access_token
url = base + node + parameters
data = json.loads(requests_until_succeed(url))
print(json.dumps(data, indent = 4, sort_keys=True))
testFacebookPageData(page_id, access_token)
json.loads accepts python3 string, which is unicode, and responce.read() returns binary string.
Use data = json.loads(requests_until_succeed(url).decode('utf-8')) because responce is most likely utf-8.
Related
The Host value int he URL will be passed as arguments but when i print the URL , it is working fine but not executing properly as whole script.
Python script:
import requests
import json
import urllib
import sys
import os
host=(str(sys.argv[1]))
headers={
"accept": "application/json",
"content-type": "application/json"
}
test_urls = 'https://{host}/Thingworx/Things/PG.MonitorStats.Stream/Services/GetStreamData?maxItems=1&oldesFirst=false&appKey=0b858f3f-4ed0-499c-a4d2-9ad0fbc0da9b&method=post'.format(host=host)
print (test_urls)
def return_json(url):
try:
response = requests.get(url,headers=headers)
# Consider any status other than 2xx an error
if not response.status_code // 100 == 2:
return "Error: Unexpected response {}".format(response)
json_obj = response.json()
return json.dumps(json_obj)
except requests.exceptions.RequestException as e:
# A serious problem happened, like an SSLError or InvalidURL
return "Error: {}".format(e)
for url in test_urls:
print return_json(url).format(host=host)
Error Output:
Error: Invalid URL 'h': No schema supplied. Perhaps you meant http://h?
Error: No connection adapters were found for ':'
Error: Invalid URL '/': No schema supplied. Perhaps you meant http:///?
test_urls is a string (not an array of strings), so you're iterating on each character and trying to GET h, then t, then t, ...
And format on the output of return_json makes no sense. Nor does your print (assuming you're using Python 3, print is a function).
Also requests has shortcuts for "Consider any status other than 2xx an error": Response.ok and Response.raise_for_status().
And... why are you dump-ing the JSON response? If you want the textual version, just return Response.text there's no reason to decode then immediately re-encode the response.
I am using Python 2.7. My Rest server side API works fine and I am able to upload a zip file using Postman. I am trying to upload a zip file using Rest client api. I tried requests package, but it is unable to send the files. I am getting an error : missing file argument.
This is my python server side code :
#ns.route('/upload_file', strict_slashes=False)
class Upload(Resource):
#api.expect(upload_parser)
def post(self):
args = upload_parser.parse_args()
file_nameup = args.file.filename
Here is the rest api client code :
import requests
from requests.auth import HTTPBasicAuth
import json
headers={'Username': 'abc#gmail.com', 'apikey':'123-e01b', 'Content-Type':'application/zip'}
f = open('C:/Users/ADMIN/Downloads/abc.zip', 'rb')
files = {"file": f}
resp = requests.post("https://.../analytics/upload_file", files=files, headers=headers )
print resp.text
print "status code " + str(resp.status_code)
if resp.status_code == 200:
print ("Success")
print resp.json()
else:
print ("Failure")
This is my error :
{"message":"Input payload validation failed","errors":{"file":"Missing required
parameter in an uploaded file"}
status code 400
Failure
In postman, I passed a zip file with in body with "file" as key and value as abc.zip file. It worked fine. I tried to use httplib library, but it fails as my post url does not contain port number. This the error with httplib :
python HttpClientEx.py
Traceback (most recent call last):
File "HttpClientEx.py", line 4, in
h = http.client.HTTPConnection(url)
File "c:\python27\Lib\httplib.py", line 736, in init
(self.host, self.port) = self._get_hostport(host, port)
File "c:\python27\Lib\httplib.py", line 777, in _get_hostport
raise InvalidURL("nonnumeric port: '%s'" % host[i+1:])
httplib.InvalidURL: nonnumeric port: '// ....net/analytics/upload_file'
How to invoke rest url post and upload a file using urllib library. Please suggest any other ways to upload file in rest client. Thanks.
I found another duplicate post :
Python Requests - Post a zip file with multipart/form-data
The solution mentioned there did not work. I found that you need to provide the full path of the file, otherwise it will not work.
Use urllib3 module.
https://urllib3.readthedocs.io/en/latest/user-guide.html
Files & binary data
For uploading files using multipart/form-data encoding you can use the same approach as Form data and specify the file field as a tuple of (file_name, file_data):
with open('example.txt') as fp:
file_data = fp.read()
r = http.request(
'POST',
'http://httpbin.org/post',
fields={
'filefield': ('example.txt', file_data),
})
json.loads(r.data.decode('utf-8'))['files']
requests library worked with below changes in my code :
import requests
from requests.auth import HTTPBasicAuth
import json
from pathlib import Path
file_ids = ''
headers={'Username': 'abc#gmail.com', 'apikey':'123-456'}
# Upload file
f = open('C:/Users/ADMIN/Downloads/abc.zip', 'rb')
files = {"file": ("C:/Users/ADMIN/Downloads/abc.zip", f)}
resp = requests.post("https:// ../analytics/upload_file", files=files, headers=headers )
print resp.text
print "status code " + str(resp.status_code)
if resp.status_code == 201:
print ("Success")
data = json.loads(resp.text)
file_ids = data['file_ids']
print file_ids
else:
print ("Failure")
I am trying to create REstful Web Service using Flask. But i am having trouble in processing the xml data in the GET request.
uri="http://127.0.0.1:5000/test/api/getfamilyinfo"
request_body='''
<StayInfo>
<district>Khurda</district>
<village>BBSR</village>
<unit>Hogwarts</unit>
</StayInfo>
'''
body_format = {'Content-Type': 'application/xml'}
requests.get(uri, data = request_body, verify = False, headers = body_format)
I am getting error:
File "C:\Python27\lib\xml\etree\ElementTree.py", line 647, in parse
source = open(source, "rb")
TypeError: coercing to Unicode: need string or buffer, Response found</textarea>
My Code:
#app.route('/test/api/getfamilyinfo', methods=['GET'])
def getfamilyinfo():
errors = []
results = {}
if request.method == "GET":
try:
r=request.data
except Exception,e:
resp = jsonify({"error": str(e)})
return resp, status.HTTP_400_BAD_REQUEST
if r:
eTree = ElementTree.parse(r) ## The code is breaking here
Kindly help me to understand where i am going wrong. Thanks in advance.
ElementTree.parse() (docs) expects a filename (or file object).
You want ElementTree.fromstring() (docs).
IN the code below, I am trying to make a POST of data with urllib2. However, I am getting a HTTP 400 bad request error. Can anyone help me with why this might be the case? The URL is reachable from my computer and all relevant ports are open.
data = {'operation' : 'all'}
results = an.post(an.get_cookie(), 'http://{}:8080/api/v1/data/controller/core/action/switch/update-host-stats'.format(an.TARGET), data)
print results
def post(session_cookie, url, payload):
data = urllib.urlencode(payload)
req = urllib2.Request(url, data)
req.add_header('Cookie','session_cookie=' + session_cookie)
try:
returnedData = urllib2.urlopen(req, data, timeout = 30)
data = json.load(returnedData)
except urllib2.URLError, e:
print e.code
print 'URL ERROR'
return {}
return data
The following code works for me:
import json
import urllib2
import logging
def post_json_request(url, post_data, optional_headers = {}):
"""
HTTP POST to server with json as parameter
#param url: url to post the data to
#param post_data: JSON formatted data
#return: response as raw data
"""
response = ""
try:
req = urllib2.Request(url, post_data, optional_headers)
jsonDump = json.dumps(post_data)
response = urllib2.urlopen(req, jsonDump)
except Exception, e:
logging.fatal("Exception while trying to post data to server - %s", e)
return response
I'm using it in various stubborn platforms that insist to retrieve data on a specific method.
Hope it will help,
Liron
I'm trying to learn how urllib2 works and how it encapsulates its various components before sending out an actual request or response.
So far I have:
theurl = "www.example.com"
That obviously specifies the URL to look at.
req = urllib2.Request(theurl)
Don't know what this does, hence the question.
handle = urllib2.urlopen(req)
This one gets the page and does all the requests and responses required.
So my question is, what does urllib2.Request actually do?
To try and look at it to get an idea I tried
print req
and just got
<urllib2.Request instance at 0x123456789>
I also tried
print req.read()
and got:
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "/usr/lib64/python2.4/urllib2.py, line 207, in `__`getattr`__`
raise AttributeError, attr
AttributeError: read
So I'm obviously doing something wrong. If anyone can help in one of both my questions that would be great.
The class "Request" you're asking about:
http://docs.python.org/library/urllib2.html#urllib2.Request
class urllib2.Request(url[, data][,
headers][, origin_req_host][,
unverifiable])
This class is an abstraction of a URL
request.
The function you actually want to make a request (which can accept a Request object or wrap one around a URL string you provice) constructing a Request object): http://docs.python.org/library/urllib2.html#urllib2.urlopen
urllib2.urlopen(url[, data][,timeout])
Open the URL url, which can be either a string or a Request object.
Example:
theurl = "www.example.com"
try:
resp = urllib2.urlopen(theurl)
print resp.read()
except IOError as e:
print "Error: ", e
Example 2 (with Request):
theurl = "www.example.com"
try:
req = urllib2.Request(theurl)
print req.get_full_url()
print req.get_method()
print dir(req) # list lots of other stuff in Request
resp = urllib2.urlopen(req)
print resp.read()
except IOError as e:
print "Error: ", e
urllib2.Request() looks like a function call, but isn't - it's an object constructor. It creates an object of type Request from the urllib2 module, documented here.
As such, it probably doesn't do anything except initialise itself. You can verify this by looking at the source code, which should be in your Python installation's lib directory (urllib2.py, at least in Python 2.x).
If you want to have the constructed URL in the Request object use :
print(urllib2.Request.get_full_url())