(New to python)
I'm trying to make a simple authenticated put of a file... so I make two curls, the first one to authenticate (which prints the token out as expected) but when I use the same variable (token) to add it to the headers ("Authorization: Bearer %s" % str(token)) token is empty. What am I doing wrong here?
import urllib
import cStringIO
import pycurl
import requests
from urllib import urlencode
import os.path
# declarations
filename = "./profile.jpg"
response = cStringIO.StringIO()
c = pycurl.Curl()
# formdata
post_data = {'username': '...', 'password':'...'}
# Form data must be provided already urlencoded.
postfields = urlencode(post_data)
# Sets request method to POST,
# Content-Type header to application/x-www-form-urlencoded
# and data to send in request body.
print "*****************************************************"
# authenticate
c = pycurl.Curl()
c.setopt(c.POST, 1)
c.setopt(c.URL, "https://.../auth")
c.setopt(c.POSTFIELDS, postfields)
c.setopt(c.SSL_VERIFYPEER, 0)
c.setopt(c.SSL_VERIFYHOST, 0)
c.setopt(c.VERBOSE, 1)
c.perform()
c.close()
token = response.getvalue()
print token
print "*****************************************************"
# upload file
filesize = os.path.getsize(filename)
fin = open(filename, 'rb')
c = pycurl.Curl()
c.setopt(c.PUT, 1)
c.setopt(c.URL, "https://.../avatar")
c.setopt(c.HTTPPOST, [("file", (c.FORM_FILE, filename))])
c.setopt(c.HTTPHEADER, [
"Authorization: Bearer %s" % str(token),
"Content-Type: image/jpeg"
])
c.setopt(c.READFUNCTION, fin.read)
c.setopt(c.POSTFIELDSIZE, filesize)
c.setopt(c.SSL_VERIFYPEER, 0)
c.setopt(c.SSL_VERIFYHOST, 0)
c.setopt(c.VERBOSE, 1)
c.setopt(c.WRITEFUNCTION, response.write),
c.perform()
c.close()
print response.getvalue()
print "*****************************************************"
Request:
> PUT ../avatar HTTP/1.1
User-Agent: PycURL/7.19.3 libcurl/7.35.0 GnuTLS/2.12.23 zlib/1.2.8 libidn/1.28 librtmp/2.3
Host: 127.0.0.1:8080
Accept: */*
Transfer-Encoding: chunked
Authorization: Bearer
Expect: 100-continue
Response:
< HTTP/1.1 100 Continue
< HTTP/1.1 401 Unauthorized
< content-type: application/json; charset=utf-8
< cache-control: no-cache
< content-length: 86
< Date: Tue, 02 Jun 2015 19:09:29 GMT
< Connection: keep-alive
<
* Connection #1 to host 127.0.0.1 left intact
{"statusCode":401,"error":"Unauthorized","message":"Incorrect Token or Token Expired"}
I think there is an encoding problem. The print function is able to output something without caring about the encoding. Looking at the PycURL quickstart documentation it mentions this issue. I would try to manipulate the encoding on this line:
"Authorization: Bearer %s" % str(token)
and try to do something like this instead:
"Authorization: Bearer %s" % token.decode('iso-8859-1')
(I would try .decode("utf-8") also, depending on what the encoding is)
You might need to change response = cStringIO.StringIO() to response = BytesIO(). I cannot give a definitive answer because I'm unsure about your setup.
EDIT: My suspicions about encoding affirmed by this post about cStringIO where it says that Unicode is not supported.
Related
I have a HTTP/JSON Restful server implemented in Python thanks to the Bottle Web framework. I want to Gzip the data sent to the client.
#!/usr/bin/env python
# -*- coding: utf-8 -*-
#
# curl -H "Content-Type: application/json" -X POST -d '{"key1": 1, "key2": 2}' http://localhost:6789/post
#
from bottle import run, request, post, route, response
from zlib import compress
import json
data = {'my': 'json'}
#post('/post')
def api_post():
global data
data = json.loads(request.body.read())
return(data)
#route('/get')
def api_get():
global data
response.headers['Content-Encoding'] = 'identity'
return(json.dumps(data).encode('utf-8'))
#route('/getgzip')
def api_get_gzip():
global data
if 'gzip' in request.headers.get('Accept-Encoding', ''):
response.headers['Content-Encoding'] = 'gzip'
ret = compress(json.dumps(data).encode('utf-8'))
else:
response.headers['Content-Encoding'] = 'identity'
ret = json.dumps(data).encode('utf-8')
return(ret)
run(host='localhost', port=6789, debug=True)
When i test my server with Curl, the result is good (if i use the --compressed option tag):
$ curl -H "Accept-encoding: gzip, deflated" -v --compressed http://localhost:6789/getgzip
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 6789 (#0)
> GET /getgzip HTTP/1.1
> Host: localhost:6789
> User-Agent: curl/7.47.0
> Accept: */*
> Accept-encoding: gzip, deflated
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Date: Sun, 12 Nov 2017 09:09:09 GMT
< Server: WSGIServer/0.1 Python/2.7.12
< Content-Length: 22
< Content-Encoding: gzip
< Content-Type: text/html; charset=UTF-8
<
* Closing connection 0
{"my": "json"}
But not with HTTPie (or Firefox, or Chrome...):
$ http http://localhost:6789/getgzipHTTP/1.0 200 OK
Content-Encoding: gzip
Content-Length: 22
Content-Type: text/html; charset=UTF-8
Date: Sun, 12 Nov 2017 09:10:10 GMT
Server: WSGIServer/0.1 Python/2.7.12
http: error: ContentDecodingError: ('Received response with content-encoding: gzip, but failed to decode it.', error('Error -3 while decompressing: incorrect header check',))
Any idea ?
I Nicolargo,
According to the documentation of Httpie, default encoding is set to Accept-Encoding: gzip, deflate but your are using the compress Python function of the zlib module which implement a Lempel–Ziv–Welch Compression Algorithm (Gzip is based on DEFLATE Algorithm).
Or, according to the documentation of Bottle (https://bottlepy.org/docs/dev/recipes.html#gzip-compression-in-bottle) you will need a custom middleware to perform a gzip compression (see an example there: http://svn.cherrypy.org/tags/cherrypy-2.1.1/cherrypy/lib/filter/gzipfilter.py).
Edit:
The compress function of the zlib module do perform a gzip compatible compression.
I think it's more related to the the header of the data (as the Error mention). In http://svn.cherrypy.org/tags/cherrypy-2.1.1/cherrypy/lib/filter/gzipfilter.py there is a use of a write_gzip_header maybe you can try this.
Thanks to the edit section of Guillaume, it's now work with both Httpie and Curl.
Here is the complete code:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
#
# curl -H "Content-Type: application/json" -X POST -d '{"key1": 1, "key2": 2}' http://localhost:6789/post
#
from bottle import run, request, post, route, response
import zlib
import json
import struct
import time
data = {'my': 'json'}
#post('/post')
def api_post():
global data
data = json.loads(request.body.read())
return(data)
#route('/get')
def api_get():
global data
response.headers['Content-Encoding'] = 'identity'
return(json.dumps(data).encode('utf-8'))
#route('/getgzip')
def api_get_gzip():
global data
ret = json.dumps(data).encode('utf-8')
if 'gzip' in request.headers.get('Accept-Encoding', ''):
response.headers['Content-Encoding'] = 'gzip'
ret = gzip_body(ret)
else:
response.headers['Content-Encoding'] = 'identity'
return(ret)
def write_gzip_header():
header = '\037\213' # magic header
header += '\010' # compression method
header += '\0'
header += struct.pack("<L", long(time.time()))
header += '\002'
header += '\377'
return header
def write_gzip_trailer(crc, size):
footer = struct.pack("<l", crc)
footer += struct.pack("<L", size & 0xFFFFFFFFL)
return footer
def gzip_body(body, compress_level=6):
yield gzip_header()
crc = zlib.crc32("")
size = 0
zobj = zlib.compressobj(compress_level,
zlib.DEFLATED, -zlib.MAX_WBITS,
zlib.DEF_MEM_LEVEL, 0)
size += len(data)
crc = zlib.crc32(data, crc)
yield zobj.compress(data)
yield zobj.flush()
yield gzip_trailer(crc, size)
run(host='localhost', port=6789, debug=True)
It's a little complicated but it do the job...
I am trying to post data to my server from my microcontroller. I need to send raw http data from my controller and this is what I am sending below:
POST /postpage HTTP/1.1
Host: https://example.com
Accept: */*
Content-Length: 18
Content-Type: application/json
{"cage":"abcdefg"}
My server requires JSON encoding and not form encoded request.
For the above request sent, I get an 400 error from the server, HTTP/1.1 400 Bad Request
However, when I try to reach the post to my server via a python script via my laptop, I am able to get a proper response.
import requests
url='https://example.com'
mycode = 'abcdefg'
def enter():
value = requests.post('url/postpage',
params={'cage': mycode})
print vars(value)
enter()
Can anyone please let me know where I could be going wrong in the raw http data I'm sending above ?
HTTP specifies the separator between headers as a single newline, and requires a double newline before the content:
POST /postpage HTTP/1.1
Host: https://example.com
Accept: */*
Content-Length: 18
Content-Type: application/json
{"cage":"abcdefg"}
If you don’t think you’ve got all of the request right, try seeing what was sent by Python:
response = ...
request = response.request # request is a PreparedRequest.
headers = request.headers
url = request.url
Read the docs for PreparedRequest for more information.
To pass a parameter, use this Python:
REQUEST = 'POST /postpage%s HTTP/1.1\r\nHost: example.com\r\nContent-Length: 0\r\nConnection: keep-alive\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nUser-Agent: python-requests/2.4.3 CPython/2.7.9 Linux/4.4.11-v7+\r\n\r\n';
query = ''
for k, v in params.items():
query += '&' + k + '=' + v # URL-encode here if you want.
if len(query): query = '?' + query[1:]
return REQUEST % query
I'm trying to implement push notifications. I can trigger notifications with this call that I need to make from python:
curl -X POST -H "Content-Type: application/json" -H "X-Thunder-Secret-Key: secret2" --data-ascii "\"Hello World\"" http://localhost:8001/api/1.0.0/key2/channels/mychannel/
This works ok from the command line.
First I tried using the subprocess, but it gave me this strange error:
curl: (1) Protocol "http not supported or disabled in libcurl
So I gave up on that and I'm trying to use pycurl. But the problem is that I don't know what to do with -X and with --data-ascii options.
import pycurl
c = pycurl.Curl()
c.setopt(c.HTTPHEADER, ['Content-Type: application/json','X-Thunder-Secret-Key: secret2'])
c.setopt(c.URL, 'http://localhost:8001/api/1.0.0/key2/channels/mychannel/')
c.perform()
print("Done")
So how do I add -X option and how do I send the text message with the request?
If you need to do HTTP POST request, see documentation example.
I think something like this should work (I've used python 2):
import pycurl
c = pycurl.Curl()
postfields = '"Hello World"'
c.setopt(c.URL, 'http://pycurl.sourceforge.net/tests/testpostvars.php')
c.setopt(c.HTTPHEADER, ['Content-Type: application/json','X-Thunder-Secret-Key: secret2'])
# Here we set data for POST request
c.setopt(c.POSTFIELDS, postfields)
c.perform()
c.close()
This code produces following HTTP packet:
POST /tests/testpostvars.php HTTP/1.1
User-Agent: PycURL/7.19.5.1 libcurl/7.37.1 SecureTransport zlib/1.2.5
Host: pycurl.sourceforge.net
Accept: */*
Content-Type: application/json
X-Thunder-Secret-Key: secret2
Content-Length: 13
"Hello World"
I'm using Observium to pull Nginx stats on localhost however it returns '405 Not Allowed':
# curl -I localhost/nginx_status
HTTP/1.1 405 Not Allowed
Server: nginx
Date: Wed, 19 Jun 2013 22:12:37 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 166
Connection: keep-alive
Keep-Alive: timeout=5
# curl -I -H "Host: example.com" localhost/nginx_status
HTTP/1.1 200 OK
Server: nginx
Date: Wed, 19 Jun 2013 22:12:43 GMT
Content-Type: text/plain
Connection: keep-alive
Keep-Alive: timeout=5
Could you please advise how to add Host header with 'urllib2.urlopen' in Python (Python 2.6.6
):
Current script:
#!/usr/bin/env python
import urllib2
import re
data = urllib2.urlopen('http://localhost/nginx_status').read()
params = {}
for line in data.split("\n"):
smallstat = re.match(r"\s?Reading:\s(.*)\sWriting:\s(.*)\sWaiting:\s(.*)$", line)
req = re.match(r"\s+(\d+)\s+(\d+)\s+(\d+)", line)
if smallstat:
params["Reading"] = smallstat.group(1)
params["Writing"] = smallstat.group(2)
params["Waiting"] = smallstat.group(3)
elif req:
params["Requests"] = req.group(3)
else:
pass
dataorder = [
"Active",
"Reading",
"Writing",
"Waiting",
"Requests"
]
print "<<<nginx>>>\n";
for param in dataorder:
if param == "Active":
Active = int(params["Reading"]) + int(params["Writing"]) + int(params["Waiting"])
print Active
else:
print params[param]
You might want to check out the urllib2 missing manual for more information, but basically you create a dictionary of your header labels and values and pass it to the urllib2.Request method. A (slightly) modified version of the code from the linked manual:
from urllib import urlencode
from urllib2 import Request urlopen
# Define values that we'll pass to our urllib and urllib2 methods
url = 'http://www.something.com/blah'
user_host = 'example.com'
values = {'name' : 'Engineero', # dict of keys and values for our POST data
'location' : 'Interwebs',
'language' : 'Python' }
headers = { 'Host' : user_host } # dict of keys and values for our header
# Set up our request, execute, and read
data = urlencode(values) # encode for sending URL request
req = Request(url, data, headers) # make POST request to url with data and headers
response = urlopen(req) # get the response from the server
the_page = response.read() # read the response from the server
# Do other stuff with the response
I'm trying to send a POST message with a purely XML payload (I think) using urllib2 in IronPython. However, everytime I send it, it returns Error code 400 (Bad Request).
I'm actually trying to mimick a Boxee remove queue item call for which the actual data packets looks like this (from WireShark):
POST /action/add HTTP/1.1
User-Agent: curl/7.16.3 (Windows build 7600; en-US; beta) boxee/0.9.21.11487
Host: app.boxee.tv
Accept: */*
Accept-Encoding: deflate, gzip
Cookie: boxee_ping_version=9; X-Mapping-oompknoc=76D730BC9E858725098BF13AEFE32EB5; boxee_app=e01e36e85d368d4112fe4d1b6587b1fd
Connection: keep-alive
Content-Type: text/xml
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept-Language: en-us,en;q=0.5
Keep-Alive: 300
Connection: keep-alive
Content-Length: 53
<message type="dequeue" referral="3102296"></message>
I'm using the following python code to send the POST:
def PostProtectedPage(theurl, username, password, postdata):
req = urllib2.Request(theurl, data=postdata)
req.add_header('Content-Type', 'text/xml')
try:
handle = urllib2.urlopen(req)
except IOError, e: # here we are assuming we fail
pass
else: # If we don't fail then the page isn't protected
print "This page isn't protected by authentication."
sys.exit(1)
if not hasattr(e, 'code') or e.code != 401: # we got an error - but not a 401 error
print "This page isn't protected by authentication."
print 'But we failed for another reason.'
sys.exit(1)
authline = e.headers.get('www-authenticate', '') # this gets the www-authenticat line from the headers - which has the authentication scheme and realm in it
if not authline:
print 'A 401 error without an authentication response header - very weird.'
sys.exit(1)
authobj = re.compile(r'''(?:\s*www-authenticate\s*:)?\s*(\w*)\s+realm=['"](\w+)['"]''', re.IGNORECASE) # this regular expression is used to extract scheme and realm
matchobj = authobj.match(authline)
if not matchobj: # if the authline isn't matched by the regular expression then something is wrong
print 'The authentication line is badly formed.'
sys.exit(1)
scheme = matchobj.group(1)
realm = matchobj.group(2)
if scheme.lower() != 'basic':
print 'This example only works with BASIC authentication.'
sys.exit(1)
base64string = base64.encodestring('%s:%s' % (username, password))[:-1]
authheader = "Basic %s" % base64string
req.add_header("Authorization", authheader)
try:
handle = urllib2.urlopen(req)
except IOError, e: # here we shouldn't fail if the username/password is right
print "It looks like the username or password is wrong."
print e
sys.exit(1)
thepage = handle.read()
return thepage
However, whenever I run this, it returns Error 400 (Bad Request)
I know the authentication is correct because I use it elsewhere to fetch the queue (and I can't imagine it's not used, otherwise how would it now which account to apply the change to?)
Looking at the network capture, could I simply be missing adding some headers to the request? Probably something simple, but I just don't know enough about python or HTTP requests to know what's what.
Edit: BTW, I'm calling the code as follows (it's actually dynamic, but this is the basic idea):
PostProtectedPage("http://app.boxee.tv/action/add", "user", "pass", "<message type=\"dequeue\" referral=\"3102296\"></message>")
This worked fine for me:
curl -v -A 'curl/7.16.3 (Windows build 7600; en-US; beta) boxee/0.9.21.11487' \
-H 'Content-Type: text/xml' -u "USER:PASS" \
--data '<message type="dequeue" referral="12573293"></message>' \
'http://app.boxee.tv/action/add'
But I get 400 Bad Request if I try to remove a referral ID that isn't currently in the queue. If you're using the same referral ID as you detected from Wireshark, that's very likely what's happening for you, too. Use
wget -nv -m -nd --user=USER --password=PASS http://app.boxee.tv/api/get_queue
to make sure what you're trying to remove is actually in the queue.