I've hit something truly strange with a pyCurl script hitting a local Django-Tastypie REST webserver.
Issuing HTTP PUT requests to the server succeeds when I use everything but pycurl (including curl), and fails with error 400 in pycurl.
After much googling and experimentation, I'm stumped. What's wrong here?
Curl call that works:
curl --verbose -X PUT -H 'Content-Type: application/json' -d '{"first_name": "Gaius","id": 1,"last_name": "Balthazar","login": "gbalthazar"}' http://localhost:8000/api/person/1/
PyCurl call that DOESN'T work (error 400):
import pycurl
import StringIO
curl = pycurl.Curl()
url = 'http://localhost:8000/api/person/1/'
curl.setopt(pycurl.URL,url)
curl.setopt(pycurl.VERBOSE, 1)
body = '{"first_name": "Gaius","id": 1,"last_name": "Baltar","login": "gbaltar"}'
curl.setopt(pycurl.READFUNCTION, StringIO.StringIO(body).read)
curl.setopt(pycurl.UPLOAD, 1)
curl.setopt(pycurl.HTTPHEADER,['Content-Type: application/json','Expect:'])
curl.setopt(curl.TIMEOUT, 5)
curl.perform()
(I've tried removing the Expects header as well, I see the header set to 100-Continue in the pycurl call, but same result.)
Unfortunately this project really does need pycurl's low-level access to HTTP timing stats to measure performance, so I can't do it with another HTTP/REST library.
Output of Curl Call:
* About to connect() to localhost port 8000 (#0)
* Trying 127.0.0.1...
* connected
* Connected to localhost (127.0.0.1) port 8000 (#0)
> PUT /api/person/1/ HTTP/1.1
> User-Agent: curl/7.27.0
> Host: localhost:8000
> Accept: */*
> Content-Type: application/json
> Content-Length: 78
>
* upload completely sent off: 78 out of 78 bytes
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Date: Thu, 05 Jun 2014 23:45:26 GMT
< Server: WSGIServer/0.1 Python/2.7.3
< Vary: Accept
< X-Frame-Options: SAMEORIGIN
< Content-Type: application/json
<
* Closing connection #0
{"first_name": "Gaius", "id": 1, "last_name": "Balthazar", "login": "gbalthazar", "pk": "1", "resource_uri": "/api/person/1/"}
Output of PyCurl Verbose Call:
* About to connect() to localhost port 8000 (#0)
* Trying 127.0.0.1...
* connected
* Connected to localhost (127.0.0.1) port 8000 (#0)
> PUT /api/person/1/ HTTP/1.1
User-Agent: PycURL/7.27.0
Host: localhost:8000
Accept: */*
Transfer-Encoding: chunked
Content-Type: application/json
* HTTP 1.0, assume close after body
< HTTP/1.0 400 BAD REQUEST
< Date: Thu, 05 Jun 2014 23:44:25 GMT
< Server: WSGIServer/0.1 Python/2.7.3
< X-Frame-Options: SAMEORIGIN
< Content-Type: application/json
<
* Closing connection #0
{"error": ""}
What am I missing here?
Found the answer:
It needs length of the request body to handle correctly
For POST:
curl.setopt(pycurl.POSTFIELDSIZE, len(body))
For PUT:
curl.setopt(pycurl.INFILESIZE, len(body))
(Yes, it's a different option for different HTTP calls... that's libcurl for you)
Not completely sure what triggers this behaviour, but the above fixes it and the tests work now.
EDIT: Adding verbose pycurl output from this:
* About to connect() to localhost port 8000 (#0)
* Trying 127.0.0.1...
* connected
* Connected to localhost (127.0.0.1) port 8000 (#0)
> PUT /api/person/1/ HTTP/1.1
User-Agent: PycURL/7.27.0
Host: localhost:8000
Accept: */*
Content-Type: application/json
Content-Length: 72
* We are completely uploaded and fine
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Date: Fri, 06 Jun 2014 17:41:38 GMT
< Server: WSGIServer/0.1 Python/2.7.3
< Vary: Accept
< X-Frame-Options: SAMEORIGIN
< Content-Type: application/json
<
* Closing connection #0
{"first_name": "Gaius", "id": 1, "last_name": "Baltar", "login": "gbaltar", "pk": "1", "resource_uri": "/api/person/1/"}
Related
I'm trying to fetch a page made with react with Python's request.get that returns 404.
import requests
page=requests.get("https://example.com/foo", allow_redirects=True)
print(page.status_code)
results in 404. I see that requests support HTTP 1.1 only.
With curl the URL returns 404 but then the server responds with a different page anyways. The server is using HTTP2. Here are some hints from curl -vv that seem relevant:
$ curl -v https://example.com/foo
* Trying 10.0.0.1
* TCP_NODELAY set
* Connected to example.com (10.0.0.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
[snip]
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
> GET /foo HTTP/2
> Host: example.com
> User-Agent: curl/7.61.1
> Accept: */*
[snip]
< HTTP/2 404
< date: Fri, 08 Apr 2022 08:42:34 GMT
< content-type: text/html; charset=utf-8
< cache-control: public, max-age=0, s-maxage=300
< etag: W/"a568501bae2318d9d0ca13a89359638e"
< last-modified: Fri, 10 Sep 2021 17:30:40 UTC
< strict-transport-security: max-age=315360000; includeSubdomains; preload
< vary: Accept-Encoding
< x-content-type-options: nosniff
< cf-cache-status: MISS
[snip some cloudflare stuff]
Then finally followed by the content of https://example.com/bar,
The headers of the response seem to indicate it "offers" http 1.1, so how do I ask for it it with the request library?
Searching, I see httpx as a http2 compliant library but their examples have snippets of async but assuming background knowledge of the same. Do I have to use httpx for http2 on Python 3.8 or greater? Is there a way to do this without async/await?
You can use httpx for http/2, you have a specific part of the doc explaining how to activate it, you need at least python 3.6, the doc suggest the use of async by it's examples.
pip install httpx[http2]
From the doc:
client = httpx.AsyncClient(http2=True)
but you can use it with the classic Client:
import httpx
client = httpx.Client(http2=True)
if __name__ == "__main__":
resp = client.get('https://example.com/foo')
print(resp.content)
I am trying to use curl in python to push this command in Elastic search so that index patterns get created after the code finishes
import pycurl
import urllib
import urllib2
apiURL = 'http://localhost:9200/.kibana/index-pattern/james+_alerts* -d'
c = pycurl.Curl()
c.setopt(c.URL, apiURL)
c.setopt(c.POSTFIELDS, '{"title" : james+"_alerts*", "timeFieldName": "timeStamp"}')
c.setopt(c.VERBOSE, True)
c.perform()
c.close()
The output being returned is: -
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 9200 (#0)
> POST /.kibana/index-pattern/james+_alerts* -d HTTP/1.1
Host: localhost:9200
User-Agent: PycURL/7.43.0 libcurl/7.52.1 GnuTLS/3.5.6 zlib/1.2.11 libidn2/0.16 libpsl/0.17.0 (+libidn2/0.16) librtmp/2.3
Accept: */*
Content-Length: 56
Content-Type: application/x-www-form-urlencoded
* upload completely sent off: 56 out of 56 bytes
* HTTP 1.0, assume close after body
< HTTP/1.0 400 Bad Request
< content-type: application/json; charset=UTF-8
< content-length: 207
<
* Curl_http_done: called premature == 0
* Closing connection 0
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"invalid version format: -D HTTP/1.1"}],"type":"illegal_argument_exception","reason":"invalid version format: -D HTTP/1.1"},"status":400}
What could potentially be the issue and fix for this?
You have extra '-d' in apiURL. I guess that it came from copy&paste from command line. If not, then you must encode url (urllib.urlencode)
try:
request_url = http://sandbox.api.hmhco.com/v1/documents
with open("C:\\Users\\Animesh\\Downloads\\assignment_4.pdf", "rb") as f:
files = {
"file": f
}
r = requests.post(request_url, files=files)
r.raise_for_status()
return r.json()
except exceptions.RequestException as e:
print e
sys.exit(-1)
This gives a 500 status error. I used logging to get what is being sent:
send:
POST /v1/documents HTTP/1.1
Host: sandbox.api.hmhco.com
Vnd-HMH-Api-Key: some_api_key
Accept-Encoding: gzip, deflate
Content-Length: 56926
Accept: application/json
User-Agent: python-requests/2.9.1
Connection: keep-alive
Content-Type: multipart/form-data
Authorization: some_access_token
--e11c78c0aeeb4cafa9837388ab386660
Content-Disposition: form-data; name="file"; filename="assignment_4.pdf"
%PDF-1.5\n%\xc7\xec\x8f\xa2\n5.......afa9837388ab386660--'
reply: 'HTTP/1.1 500 Internal Server Error\r\n'
header: Access-Control-Allow-Headers: Authorization, Content-Type, Vnd-HMH-Api-Key
header: Access-Control-Allow-Methods: POST, GET, OPTIONS, DELETE, PUT, PATCH
header: Access-Control-Allow-Origin: *
header: Access-Control-Max-Age: 3600
header: Date: Tue, 29 Dec 2015 05:21:58 GMT
header: Server: nginx/1.4.6 (Ubuntu)
header: X-Application-Context: palantir:8070
header: Content-Length: 0
header: Connection: keep-alive
500 Server Error: Internal Server Error for url: http://sandbox.api.hmhco.com/v1/documents
But the corresponding cURL command works:
curl -v -X POST -H "Vnd-HMH-Api-Key:some_api_key" -H "accepts:application/json" -H "Content-Type:application/json" -H "Authorization:some_access_token" "http://sandbox.api.hmhco.com/v1/documents" -F file=#"C:\Users\Animesh\Downloads\assignment_4.pdf"
The verbose out gives:
> POST /v1/documents HTTP/1.1
> User-Agent: curl/7.37.0
> Host: sandbox.api.hmhco.com
> Accept: */*
> Vnd-HMH-Api-Key:some_api_key
> accepts:application/json
> Authorization:some_access_token
> Content-Length: 56982
> Expect: 100-continue
> Content-Type:multipart/form-data; boundary=------------------------20b5cc23cfa6d1d4
>
< HTTP/1.1 100 Continue
< HTTP/1.1 200 OK
< Access-Control-Allow-Headers: Authorization, Content-Type, Vnd-HMH-Api-Key
< Access-Control-Allow-Headers: accept, api_key, x-mashery-debug, Authorization, authCurrentDateTime, Content-Type
< Access-Control-Allow-Methods: POST, GET, OPTIONS, DELETE, PUT, PATCH
< Access-Control-Allow-Methods: POST, GET, OPTIONS, PUT, DELETE
< Access-Control-Allow-Origin: *
< Access-Control-Max-Age: 3600
< Access-Control-Request-Headers: accept, api_key, x-mashery-debug, Authorization, authCurrentDateTime, Content-Type
< Cache-Control: max-age=0, private, must-revalidate
< Content-Type: application/json;charset=UTF-8
< Date: Tue, 29 Dec 2015 05:28:40 GMT
< ETag: "e421273e09891349ef4808ca5677345c"
< P3P: CP="NOI ADM DEV PSAi COM NAV OUR OTRo STP IND DEM"
* Server nginx/1.4.6 (Ubuntu) is not blacklisted
< Server: nginx/1.4.6 (Ubuntu)
< Status: 200 OK
< User-Info: {"iss":"https://identity.api.hmhco.com","aud":"http://www.hmhco.com","iat":1451323840,"sub":"cn\u003dSauron Baraddur,uid\u003dsauron,uniqueIdentifier\u003dab1e436e-8177-4139-8e2d-52a6f7a8be27,dc\u0
3d1","http://www.imsglobal.org/imspurl/lis/v1/vocab/person":["Instructor"],"client_id":"ef5f7a03-58e8-48d7-a38a-abbd2696bdb6.hmhco.com","exp":1451409840}
< X-Application-Context: palantir:8070
< X-Rack-Cache: invalidate, pass
< X-Request-Id: a0078f7f5daf804affb98de6947170ca
< X-Runtime: 0.066617
< X-UA-Compatible: IE=Edge,chrome=1
< Content-Length: 452
< Connection: keep-alive
<
{"content_type":"application/pdf","created_at":"2015-12-29T00:28:40-05:00","file_tmp":"1451366920-13680-2938/assignment_4.pdf","original_filename":"assignment_4.pdf","secure_token":"7f395da4-5ccf-41a6-86a3-16a5
af0de79","size":56774,"updated_at":"2015-12-29T00:2 ......
This gives response code 200 and response body as a json:
{
"content_type": "application/pdf",
"created_at": "2015-12-28T14:13:25-05:00",
"file_tmp": "1451330005-13680-5506/assignment_4.pdf",
"original_filename": "assignment_4.pdf",
"secure_token": "9546cf79-0b00-4454-b577-6bdc8ddd7315",
"size": 56774,
"updated_at": "2015-12-28T14:13:25-05:00", ......
Change line which sends post request to:
r = requests.post(request_url, files=files, headers={'content-type': 'application/json'})
Because in sample above curl making post with 'content-type': 'application/json' and everything goes well and your python requests isn't. Setting content-type in python code should fix your issue.
I wrote the following POST method for my REST API, which was built using Flask. The method receives one parameter, which is a radio station url.
#app.route('/todo/api/v1.0/predvajaj', methods=['POST'])
def create_task():
print "Expression value: " + str(not request.json or not 'title' in request.json)
if not request.json or not 'title' in request.json:
abort(400)
link=request.json['title']
print "Link value: " + link
cmd = "pkill sox"
os.system(cmd)
time.sleep(2)
#link = "http://www.radiostationurl.m3u"
cmd = "sox -t mp3 " + link + " -t wav -r 22050 -c 1 - | sudo ../pifm - 90.5 &"
os.system(cmd)
return jsonify({'status': "ok"}), 201
The API runs on a Raspberry Pi with the ip address: 192.168.0.200. I tried testing the method locally (on the Pi), using the curl tool. This worked fine:
curl -i -H "Content-Type: application/json" -X POST -d '{"title":"http://www.radiostationurl.m3u"}' http://192.168.0.200:5000/todo/api/v1.0/predvajaj
Then I tried testing testing it with a computer (running Windows) in the same LAN with the same command and tool, but I get the following error:
HTTP/1.0 400 BAD REQUEST
Content-Type: text/html
Content-Length: 192
Server: Werkzeug/0.10.4 Python/2.7.3
Date: Wed, 05 Aug 2015 11:06:05 GMT
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>400 Bad Request</title>
<h1>Bad Request</h1>
<p>The browser (or proxy) sent a request that this server could not understand.</p>
Webserver output (two requests - one from the pi and from the other pc):
Expression value: False
192.168.0.200 - - [05/Aug/2015 11:05:56] "POST /todo/api/v1.0/predvajaj HTTP/1.1" 201 -
sox WARN wav: Length in output .wav header will be wrong since can't seek to fix it
192.168.0.103 - - [05/Aug/2015 11:06:05] "POST /todo/api/v1.0/predvajaj HTTP/1.1" 400
So the problem is in the evaluation of the if expression. Can anybody tell me why is it failing to evaluate?
EDIT: Tried running curl with the -v switch as #meuh suggested. The content-length is different.
Pi:
* About to connect() to 192.168.0.200 port 5000 (#0)
* Trying 192.168.0.200...
* connected
* Connected to 192.168.0.200 (192.168.0.200) port 5000 (#0)
> POST /todo/api/v1.0/predvajaj HTTP/1.1
> User-Agent: curl/7.26.0
> Host: 192.168.0.200:5000
> Accept: */*
> Content-Type: application/json
> Content-Length: 51
>
* upload completely sent off: 51 out of 51 bytes
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* HTTP 1.0, assume close after body
< HTTP/1.0 201 CREATED
< Content-Type: application/json
< Content-Length: 27
< Server: Werkzeug/0.10.4 Python/2.7.3
< Date: Wed, 05 Aug 2015 13:49:01 GMT
<
{
"status": "predvajam"
* Closing connection #0
}
Windows:
* About to connect() to 192.168.0.200 port 5000 (#0)
* Trying 192.168.0.200...
* Connected to 192.168.0.200 (192.168.0.200) port 5000 (#0)
> POST /todo/api/v1.0/predvajaj HTTP/1.1
> Host: 192.168.0.200:5000
> User-Agent: curl/7.43.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 49
>
* upload completely sent off: 49 out of 49 bytes
* HTTP 1.0, assume close after body
< HTTP/1.0 400 BAD REQUEST
< Content-Type: text/html
< Content-Length: 192
< Server: Werkzeug/0.10.4 Python/2.7.3
< Date: Wed, 05 Aug 2015 13:50:51 GMT
<
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>400 Bad Request</title>
<h1>Bad Request</h1>
<p>The browser (or proxy) sent a request that this server could not understand.</p>
* Closing connection 0
I overlooked a note about this issue, on the site with the tutorial, which says:
Note: if you are on Windows and use the Cygwin version of curl from bash then the above command will work just fine. However, if you are using the native version of curl from the regular command prompt there is a little dance that needs to be done to send double quotes inside the body of a request. Essentially on Windows you have to use double quotes to enclose the body of the request, and then inside it you escape a double quote by writing three of them in sequence.
The correct command, in my case, is:
curl -i -H "Content-Type: application/json" -X POST -d "{"""title""":"""http://www.radiostationurl.m3u"""}" http://192.168.0.200:5000/todo/api/v1.0/predvajaj
I am playing with HTTP transfers, just trying to make something work. I have a GAE server and I'm pretty sure it's working properly because it renders when I go to it with my browser, but here is the python code anyway:
import sys
print 'Content-Type: text/html'
print ''
print '<pre>'
number = -1
data = sys.stdin.read()
try:
number = int(data[data.find('=')+1:])
except:
number = -1
print 'If your number was', number, ', then you are awesome!!!'
print '</pre>'
I am just learning the whole HTTP POST vs GET vs Response process, but this is what I have been doing from the terminal:
$ telnet localhost 8080
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
GET http://localhost:8080/?number=28 HTTP/1.0
HTTP/1.0 200 Good to go
Server: Development/1.0
Date: Thu, 07 Jul 2011 21:29:28 GMT
Content-Type: text/html
Cache-Control: no-cache
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Content-Length: 61
<pre>
If your number was -1 , then you are awesome!!!
</pre>
Connection closed by foreign host.
I am using a GET here because I stumbled around for about 40 minutes trying to make a telnet POST work - with no success :(
I would appreciate any help on how to get this GET and/or the POST to work. Thanks in advance!!!!
when using GET, no data will be present in the request body, so sys.stdin.read() is bound to fail. instead, you might want to look at the environment, specifically os.environ['QUERY_STRING']
Another thing you're doing a little strangely is you are not using the correct request format. The second part of the request should not include the url scheme, host or port, it should look like:
GET /?number=28 HTTP/1.0
specify the host in a seperate Host: header; the server will determine the scheme on it's own.
When using POST, most servers won't read past the amount of data in the Content-Length header, which if you don't supply one, may be assumed to be zero bytes. The server may try to read any bytes after the point specified by the content-length to be the next request in a persistent connection, and when it doesn't begin with a valid request, it closes the connection. So basically:
POST / HTTP/1.0
Host: localhost: 8080
Content-Length: 2
Content-Type: text/plain
28
But why are you testing this in telnet? How about curl?
$ curl -vs -d'28' -H'Content-Type: text/plain' http://localhost:8004/
* About to connect() to localhost port 8004 (#0)
* Trying ::1... Connection refused
* Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 8004 (#0)
> POST / HTTP/1.1
> User-Agent: curl/7.20.1 (x86_64-redhat-linux-gnu) libcurl/7.20.1 NSS/3.12.6.2 zlib/1.2.3 libidn/1.16 libssh2/1.2.4
> Host: localhost:8004
> Accept: */*
> Content-Type: text/plain
> Content-Length: 2
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Date: Thu, 07 Jul 2011 22:09:17 GMT
< Server: WSGIServer/0.1 Python/2.6.4
< Content-Type: text/html; charset=UTF-8
< Content-Length: 45
<
* Closing connection #0
{'body': '28', 'method': 'POST', 'query': []}
or better yet, in python:
>>> import httplib
>>> headers = {"Content-type": "text/plain",
... "Accept": "text/plain"}
>>>
>>> conn = httplib.HTTPConnection("localhost:8004")
>>> conn.request("POST", "/", "28", headers)
>>> response = conn.getresponse()
>>> print response.read()
{'body': '28', 'method': 'POST', 'query': []}
>>>