I am capturing a curl output after spawning to a remote machine. The expect function keeps getting timed out , i tried different patterns still no luck. The curl request is of the form ,
hdl2.sendline("curl -v http://{0}/index.html -o /dev/null".format(host1))
The output received is
" > GET /index.html HTTP/1.1
> User-Agent: curl/7.35.0
> Host: 13.126.208.1
> Accept: */*
< HTTP/1.1 200 OK
< Date: Sun, 20 Aug 2017 12:32:54 GMT
* Server Apache/2.4.7 (Ubuntu) is not blacklisted
< Server: Apache/2.4.7 (Ubuntu)
< Last-Modified: Sun, 20 Aug 2017 09:56:44 GMT
< ETag: "2cf6-5572c61363668"
< Accept-Ranges: bytes
< Content-Length: 11510
< Vary: Accept-Encoding
< Content-Type: text/html
<
{ [data not shown]
100 11510 100 11510 0 0 3055k 0 --:--:-- --:--:-- --:--:-- 3746k
* Connection #0 to host 13.126.208.1 left intact
ubuntu#ip-172-31-28-48:~$ "
This is the end output , i have given expect as
hdl2.expect("\$ ")
But every time i get pexpect timeout. Any suggestions is appreciated.
This may happen because of line buffering: ubuntu#ip-172-31-28-48:~$ is not terminated with \n, so except may not consider this line. You can try this:
hdl2.sendline("curl -v http://{0}/index.html -o /dev/null; echo DONE".format(host1))
hdl2.expect("DONE")
(Use a string that will be unique for your data instead of DONE.)
Pexpect's default timeout is 30 seconds. If your curl command needs more time then you need to increase the timeout value. For example:
hdl2.expect("\$ ", timeout=600)
Related
I am trying to measure the respone time of a certain request using python-requests.
import requests
import time
start = time.time()
r = requests.get("https://www.dl.soc.i.kyoto-u.ac.jp/index.php/members/")
end = time.time()
print(end - start)
print(r.elapsed.seconds)
It gave me a result of
64.67747116088867
0.631163
Could anyone please explain the reason of this huge gap? Thanks.
By the way, when I was trying the same request on Google-Chrome, actually the first result is what I want.
I made some test with an artificially delaying webserver:
nc -l 8080
Then in another terminal in a Python session:
import time, requests
a=time.time()
r = requests.get("http://localhost:8080/")
b=time.time()
print r.elapsed, b-a
Pasting this issued this HTTP request on the server terminal:
GET / HTTP/1.1
Host: localhost:8080
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: python-requests/2.9.1
I waited for 5 seconds, then I pasted this reply in the server:
HTTP/1.1 200 OK
Content-Type: text/html
Content-Length: 12
Connection: keep-alive
Keep-Alive: timeout=15
Date: Thu, 04 Jan 2018 10:12:09 GMT
Server: Apache
Last-Modified: Wed, 09 Dec 2015 13:57:24 GMT
ETag: "28bd-52677784b6090"
Accept-Ranges: bytes
hello
I stated 12 bytes but only sent 6 (hello\n), so this was unfinished. I waited another five seconds, then pasted this text:
world
This finished the reply with the remaining six bytes (world\n). In the client terminal I saw the result appear:
0:00:05.185509 10.8904578686
So, obviously the r.elapsed is the Time-To-First-Byte (TTFB) while the call to requests.get() only terminates after the whole message has been received (Time-To-Last-Byte, TTLB).
I have the following piece of python 2.7 code:
import sys
from subprocess import Popen, PIPE
cmd = "curl -v http://172.23.85.34 2>&1"
p = Popen(cmd, shell=True, stderr=PIPE, stdout=PIPE)
ret = p.wait()
out, err = p.communicate()
sys.stdout.write("Command is:\n" + cmd + "\nOutput:\n" + out)
For some reason, the output that I get when I run the python script, is different from the output I get when I run curl -v http://172.23.85.34 2>&1 directly from bash. What is the reason for this?
Python output:
Command is:
curl -v http://172.23.85.34 2>&1
Output:
* Rebuilt URL to: http://172.23.85.34/
* Trying 172.23.85.34...
* TCP_NODELAY set
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 172.23.85.34 (172.23.85.34) port 80 (#0)
> GET / HTTP/1.1
> Host: 172.23.85.34
> User-Agent: curl/7.51.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Thu, 23 Mar 2017 13:35:06 GMT
< Server: Apache/2.4.23 (Unix) OpenSSL/1.0.2g
< Upgrade: h2
< Connection: Upgrade
< Last-Modified: Tue, 20 Dec 2016 09:33:57 GMT
< ETag: "61-54413bc4fb12a"
< Accept-Ranges: bytes
< Content-Length: 97
< Content-Type: text/html
<
{ [97 bytes data]
* Curl_http_done: called premature == 0
100 97 100 97 0 0 5307 0 --:--:-- --:--:-- --:--:-- 5388
* Connection #0 to host 172.23.85.34 left intact
<html>
<body>
<h1>It works!</h1>
<img src="Helium.jpg" alt="Helium">
</body>
</html>
cURL output:
$ curl -v http://172.23.85.34 2>&1
* Rebuilt URL to: http://172.23.85.34/
* Trying 172.23.85.34...
* TCP_NODELAY set
* Connected to 172.23.85.34 (172.23.85.34) port 80 (#0)
> GET / HTTP/1.1
> Host: 172.23.85.34
> User-Agent: curl/7.51.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Thu, 23 Mar 2017 13:37:01 GMT
< Server: Apache/2.4.23 (Unix) OpenSSL/1.0.2g
< Upgrade: h2
< Connection: Upgrade
< Last-Modified: Tue, 20 Dec 2016 09:33:57 GMT
< ETag: "61-54413bc4fb12a"
< Accept-Ranges: bytes
< Content-Length: 97
< Content-Type: text/html
<
<html>
<body>
<h1>It works!</h1>
<img src="Helium.jpg" alt="Helium">
</body>
</html>
* Curl_http_done: called premature == 0
* Connection #0 to host 172.23.85.34 left intact
There's PROGRESS METER info in the output of python script.
From man curl:
PROGRESS METER
curl normally displays a progress meter during operations, indicating the amount of transferred data, transfer speeds and estimated time left, etc.
curl displays this data to the terminal by default, so if you invoke curl to do an operation and it is about to write data to the terminal, it disables the progress
meter as otherwise it would mess up the output mixing progress meter and response data.
Simply speaking curl will display progress info by default, but disable it if the output is terminal, so when you run your python script, the output of curl is not terminal but piped to your python progress, makes the progress info in the output.
You'll get same output with option -s to disable it:
curl -v -s https://httpbin.org/get 2>&1
I wrote the following POST method for my REST API, which was built using Flask. The method receives one parameter, which is a radio station url.
#app.route('/todo/api/v1.0/predvajaj', methods=['POST'])
def create_task():
print "Expression value: " + str(not request.json or not 'title' in request.json)
if not request.json or not 'title' in request.json:
abort(400)
link=request.json['title']
print "Link value: " + link
cmd = "pkill sox"
os.system(cmd)
time.sleep(2)
#link = "http://www.radiostationurl.m3u"
cmd = "sox -t mp3 " + link + " -t wav -r 22050 -c 1 - | sudo ../pifm - 90.5 &"
os.system(cmd)
return jsonify({'status': "ok"}), 201
The API runs on a Raspberry Pi with the ip address: 192.168.0.200. I tried testing the method locally (on the Pi), using the curl tool. This worked fine:
curl -i -H "Content-Type: application/json" -X POST -d '{"title":"http://www.radiostationurl.m3u"}' http://192.168.0.200:5000/todo/api/v1.0/predvajaj
Then I tried testing testing it with a computer (running Windows) in the same LAN with the same command and tool, but I get the following error:
HTTP/1.0 400 BAD REQUEST
Content-Type: text/html
Content-Length: 192
Server: Werkzeug/0.10.4 Python/2.7.3
Date: Wed, 05 Aug 2015 11:06:05 GMT
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>400 Bad Request</title>
<h1>Bad Request</h1>
<p>The browser (or proxy) sent a request that this server could not understand.</p>
Webserver output (two requests - one from the pi and from the other pc):
Expression value: False
192.168.0.200 - - [05/Aug/2015 11:05:56] "POST /todo/api/v1.0/predvajaj HTTP/1.1" 201 -
sox WARN wav: Length in output .wav header will be wrong since can't seek to fix it
192.168.0.103 - - [05/Aug/2015 11:06:05] "POST /todo/api/v1.0/predvajaj HTTP/1.1" 400
So the problem is in the evaluation of the if expression. Can anybody tell me why is it failing to evaluate?
EDIT: Tried running curl with the -v switch as #meuh suggested. The content-length is different.
Pi:
* About to connect() to 192.168.0.200 port 5000 (#0)
* Trying 192.168.0.200...
* connected
* Connected to 192.168.0.200 (192.168.0.200) port 5000 (#0)
> POST /todo/api/v1.0/predvajaj HTTP/1.1
> User-Agent: curl/7.26.0
> Host: 192.168.0.200:5000
> Accept: */*
> Content-Type: application/json
> Content-Length: 51
>
* upload completely sent off: 51 out of 51 bytes
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* HTTP 1.0, assume close after body
< HTTP/1.0 201 CREATED
< Content-Type: application/json
< Content-Length: 27
< Server: Werkzeug/0.10.4 Python/2.7.3
< Date: Wed, 05 Aug 2015 13:49:01 GMT
<
{
"status": "predvajam"
* Closing connection #0
}
Windows:
* About to connect() to 192.168.0.200 port 5000 (#0)
* Trying 192.168.0.200...
* Connected to 192.168.0.200 (192.168.0.200) port 5000 (#0)
> POST /todo/api/v1.0/predvajaj HTTP/1.1
> Host: 192.168.0.200:5000
> User-Agent: curl/7.43.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 49
>
* upload completely sent off: 49 out of 49 bytes
* HTTP 1.0, assume close after body
< HTTP/1.0 400 BAD REQUEST
< Content-Type: text/html
< Content-Length: 192
< Server: Werkzeug/0.10.4 Python/2.7.3
< Date: Wed, 05 Aug 2015 13:50:51 GMT
<
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>400 Bad Request</title>
<h1>Bad Request</h1>
<p>The browser (or proxy) sent a request that this server could not understand.</p>
* Closing connection 0
I overlooked a note about this issue, on the site with the tutorial, which says:
Note: if you are on Windows and use the Cygwin version of curl from bash then the above command will work just fine. However, if you are using the native version of curl from the regular command prompt there is a little dance that needs to be done to send double quotes inside the body of a request. Essentially on Windows you have to use double quotes to enclose the body of the request, and then inside it you escape a double quote by writing three of them in sequence.
The correct command, in my case, is:
curl -i -H "Content-Type: application/json" -X POST -d "{"""title""":"""http://www.radiostationurl.m3u"""}" http://192.168.0.200:5000/todo/api/v1.0/predvajaj
I've hit something truly strange with a pyCurl script hitting a local Django-Tastypie REST webserver.
Issuing HTTP PUT requests to the server succeeds when I use everything but pycurl (including curl), and fails with error 400 in pycurl.
After much googling and experimentation, I'm stumped. What's wrong here?
Curl call that works:
curl --verbose -X PUT -H 'Content-Type: application/json' -d '{"first_name": "Gaius","id": 1,"last_name": "Balthazar","login": "gbalthazar"}' http://localhost:8000/api/person/1/
PyCurl call that DOESN'T work (error 400):
import pycurl
import StringIO
curl = pycurl.Curl()
url = 'http://localhost:8000/api/person/1/'
curl.setopt(pycurl.URL,url)
curl.setopt(pycurl.VERBOSE, 1)
body = '{"first_name": "Gaius","id": 1,"last_name": "Baltar","login": "gbaltar"}'
curl.setopt(pycurl.READFUNCTION, StringIO.StringIO(body).read)
curl.setopt(pycurl.UPLOAD, 1)
curl.setopt(pycurl.HTTPHEADER,['Content-Type: application/json','Expect:'])
curl.setopt(curl.TIMEOUT, 5)
curl.perform()
(I've tried removing the Expects header as well, I see the header set to 100-Continue in the pycurl call, but same result.)
Unfortunately this project really does need pycurl's low-level access to HTTP timing stats to measure performance, so I can't do it with another HTTP/REST library.
Output of Curl Call:
* About to connect() to localhost port 8000 (#0)
* Trying 127.0.0.1...
* connected
* Connected to localhost (127.0.0.1) port 8000 (#0)
> PUT /api/person/1/ HTTP/1.1
> User-Agent: curl/7.27.0
> Host: localhost:8000
> Accept: */*
> Content-Type: application/json
> Content-Length: 78
>
* upload completely sent off: 78 out of 78 bytes
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Date: Thu, 05 Jun 2014 23:45:26 GMT
< Server: WSGIServer/0.1 Python/2.7.3
< Vary: Accept
< X-Frame-Options: SAMEORIGIN
< Content-Type: application/json
<
* Closing connection #0
{"first_name": "Gaius", "id": 1, "last_name": "Balthazar", "login": "gbalthazar", "pk": "1", "resource_uri": "/api/person/1/"}
Output of PyCurl Verbose Call:
* About to connect() to localhost port 8000 (#0)
* Trying 127.0.0.1...
* connected
* Connected to localhost (127.0.0.1) port 8000 (#0)
> PUT /api/person/1/ HTTP/1.1
User-Agent: PycURL/7.27.0
Host: localhost:8000
Accept: */*
Transfer-Encoding: chunked
Content-Type: application/json
* HTTP 1.0, assume close after body
< HTTP/1.0 400 BAD REQUEST
< Date: Thu, 05 Jun 2014 23:44:25 GMT
< Server: WSGIServer/0.1 Python/2.7.3
< X-Frame-Options: SAMEORIGIN
< Content-Type: application/json
<
* Closing connection #0
{"error": ""}
What am I missing here?
Found the answer:
It needs length of the request body to handle correctly
For POST:
curl.setopt(pycurl.POSTFIELDSIZE, len(body))
For PUT:
curl.setopt(pycurl.INFILESIZE, len(body))
(Yes, it's a different option for different HTTP calls... that's libcurl for you)
Not completely sure what triggers this behaviour, but the above fixes it and the tests work now.
EDIT: Adding verbose pycurl output from this:
* About to connect() to localhost port 8000 (#0)
* Trying 127.0.0.1...
* connected
* Connected to localhost (127.0.0.1) port 8000 (#0)
> PUT /api/person/1/ HTTP/1.1
User-Agent: PycURL/7.27.0
Host: localhost:8000
Accept: */*
Content-Type: application/json
Content-Length: 72
* We are completely uploaded and fine
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Date: Fri, 06 Jun 2014 17:41:38 GMT
< Server: WSGIServer/0.1 Python/2.7.3
< Vary: Accept
< X-Frame-Options: SAMEORIGIN
< Content-Type: application/json
<
* Closing connection #0
{"first_name": "Gaius", "id": 1, "last_name": "Baltar", "login": "gbaltar", "pk": "1", "resource_uri": "/api/person/1/"}
I am playing with HTTP transfers, just trying to make something work. I have a GAE server and I'm pretty sure it's working properly because it renders when I go to it with my browser, but here is the python code anyway:
import sys
print 'Content-Type: text/html'
print ''
print '<pre>'
number = -1
data = sys.stdin.read()
try:
number = int(data[data.find('=')+1:])
except:
number = -1
print 'If your number was', number, ', then you are awesome!!!'
print '</pre>'
I am just learning the whole HTTP POST vs GET vs Response process, but this is what I have been doing from the terminal:
$ telnet localhost 8080
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
GET http://localhost:8080/?number=28 HTTP/1.0
HTTP/1.0 200 Good to go
Server: Development/1.0
Date: Thu, 07 Jul 2011 21:29:28 GMT
Content-Type: text/html
Cache-Control: no-cache
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Content-Length: 61
<pre>
If your number was -1 , then you are awesome!!!
</pre>
Connection closed by foreign host.
I am using a GET here because I stumbled around for about 40 minutes trying to make a telnet POST work - with no success :(
I would appreciate any help on how to get this GET and/or the POST to work. Thanks in advance!!!!
when using GET, no data will be present in the request body, so sys.stdin.read() is bound to fail. instead, you might want to look at the environment, specifically os.environ['QUERY_STRING']
Another thing you're doing a little strangely is you are not using the correct request format. The second part of the request should not include the url scheme, host or port, it should look like:
GET /?number=28 HTTP/1.0
specify the host in a seperate Host: header; the server will determine the scheme on it's own.
When using POST, most servers won't read past the amount of data in the Content-Length header, which if you don't supply one, may be assumed to be zero bytes. The server may try to read any bytes after the point specified by the content-length to be the next request in a persistent connection, and when it doesn't begin with a valid request, it closes the connection. So basically:
POST / HTTP/1.0
Host: localhost: 8080
Content-Length: 2
Content-Type: text/plain
28
But why are you testing this in telnet? How about curl?
$ curl -vs -d'28' -H'Content-Type: text/plain' http://localhost:8004/
* About to connect() to localhost port 8004 (#0)
* Trying ::1... Connection refused
* Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 8004 (#0)
> POST / HTTP/1.1
> User-Agent: curl/7.20.1 (x86_64-redhat-linux-gnu) libcurl/7.20.1 NSS/3.12.6.2 zlib/1.2.3 libidn/1.16 libssh2/1.2.4
> Host: localhost:8004
> Accept: */*
> Content-Type: text/plain
> Content-Length: 2
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Date: Thu, 07 Jul 2011 22:09:17 GMT
< Server: WSGIServer/0.1 Python/2.6.4
< Content-Type: text/html; charset=UTF-8
< Content-Length: 45
<
* Closing connection #0
{'body': '28', 'method': 'POST', 'query': []}
or better yet, in python:
>>> import httplib
>>> headers = {"Content-type": "text/plain",
... "Accept": "text/plain"}
>>>
>>> conn = httplib.HTTPConnection("localhost:8004")
>>> conn.request("POST", "/", "28", headers)
>>> response = conn.getresponse()
>>> print response.read()
{'body': '28', 'method': 'POST', 'query': []}
>>>