We Keep Coding

Issue with getting the response data using Locust

Im trying to see if I'm able to get the response data as I'm trying to learn how to use regex on Locust. I'm trying to reproduce my test script from JMeter using Locust.
This is the part of the code that I'm having problem with.
import time,csv,json
from locust import HttpUser, task,between,tag
class ResponseGet(HttpUser):
response_data= ""
wait_time= between (1,1.5)
host= "https://portal.com"
username= "NA"
password= "NA"
#task
def portal(self):
print("Portal Task")
response = self.client.post('/login', json={'username':'user','password':'123'})
print(response)
self.response_data = json.loads(response.text)
print(response_data)
I've tried this suggestion and I somehow can't make it work.
My idea is get response data > use regex to extract string > pass the string for the next task to use
For example:
Get login response data > use regex to extract token > use the token for the next task.
Is there any better way to do this?

The way you're doing it should work, but Locust's HttpUser's client is based on Requests so if you want to access the response data as a JSON you should be able to do that with just self.response_data = response.json(). But that will only work if the response body is valid JSON. Your code will also fail if the response body is not JSON.
If your problem is in parsing the response text as JSON, it's likely that the response just isn't JSON, possibly because you're getting an error or something. You could print the response body before your attempt to load it as JSON. But your current print(response) won't do that because it will just be printing the Response object returned by Requests. You'd need to print(response.text()) instead.
As far as whether a regex would be the right solution for getting at the token returned in the response, that will depend on how exactly the response is formatted.

Python, can someone explain this script to me?

I know nothing about Python but would like to clone this script with jquery using ajax post.
To do that i need to know what this script is doing in the first place.
import requests
import json
params = {'nearest': True, 'imageurl': img, 'timestamp':140000}
request = requests.post('http://example.com/api/upload/', data=params)
output = request.json()
print json.dumps(output['files'][0]['predicted_classes'])
Thanks. If something is unclear please comment and i'll clarify.

import requests
import json
above line imports two modules Request(contain methods for sending request to the server) and json(to serialise/deserialise data to json)
params = {'nearest': True, 'imageurl': img, 'timestamp':140000}
creating a dictionary with key value .here it is used to pass parameter
response= requests.post('http://example.com/api/upload/', data=params)
this is used to send Post resquest . here post is method in request module with parameters(Url,data_to_send)
output = response.json()
output has the response in json format
print json.dumps(output['files'][0]['predicted_classes'])
json dumps is used to convert to json format

This code does the following:
1) #First it imports the external modules.
2) #Next it defines params as a dictionary with 3 entries.
3) #Then it uses request libraries to get the file and transfigures "params" into a json object
4) #Lastly, the code prints the request.
To see the request you may need to use an image library to see what you gathered from the World Wide Web.

Python requests module: urlencoding json data

I'm working on an API wrapper. The spec I'm trying to build to has the following request in it:
curl -H "Content-type:application/json" -X POST -d data='{"name":"Partner13", "email":"example#example.com"}' http://localhost:5000/
This request produces the following response from a little test server I setup to see exatly what headers/params etc are sent as. This little script produces:
uri: http://localhost:5000/,
method: POST,
api_key: None,
content_type: application/json,
params: None,
data: data={"name":"Partner13", "email":"example#example.com"}
So that above is the result I want my python script to create when it hits the little test script.
I'm using the python requests module, which is the most beautiful HTTP lib I have ever used. So here is my python code:
uri = "http://localhost:5000/"
headers = {'content-type': 'application/json' }
params = {}
data = {"name":"Partner13", "email":"example#exmaple.com"}
params["data"] = json.dumps(data)
r = requests.post(uri, data=params, headers=headers)
So simple enough stuff. Set the headers, and create a dictionary for the POST parameters. That dictionary has one entry called "data" which is the JSON string of the data I want to send to the server. Then I call the post. However, the result my little test script gives back is:
uri: http://localhost:5000/,
method: POST,
api_key: None,
content_type: application/json,
params: None,
data: data=%7B%22name%22%3A+%22Partner13%22%2C+%22email%22%3A+%22example%40example.com%22%7D
So essentially the json data I wanted to send under the data parameter has been urlendcoded.
Does anyone know how to fix this? I have looked through the requests documentation and cannot seem to find a way to not auto urlencode the send data.
Thanks very much,
Kevin

When creating the object for the data keyword, simply assign a variable the result of json.dumps(data).
Also, because HTTP POST can accept both url parameters as well as data in the body of the request, and because the requests.post function has a keyword argument named "params", it might be better to use a different variable name for readability. The requests docs use the variable name "payload", so thats what I use.
data = {"name":"Partner13", "email":"example#exmaple.com"}
payload = json.dumps(data)
r = requests.post(uri, data=payload, headers=headers)

Requests automatically URL encodes dictionaries passed as data here. John_GG's solution works because rather than posting a dictionary containing the JSON encoded string in the 'data' field it simply passes the JSON encoded string directly: strings are not automatically encoded. I can't say I understand the reason for this behaviour in Requests but regardless, it is what it is. There is no way to toggle this behaviour off that I can find.
Best of luck with it, Kevin.

Why is my HTTP POST request data string (probably) being incorrectly encoded?

I'm having some trouble with twisted.web.client.Agent...
I think the string data in my post request isn't being formatted properly. I'm trying to do something analogous to this synchronous code:
from urllib import urlencode
import urllib2
page = 'http://example.com/'
id_string = 'this:is,my:id:string'
req = urllib2.Request(page, data=urlencode({'id': id_string})) # urlencode call returns 'id=this%3Ais%2Cmy%3Aid%3Astring'
resp = urllib2.urlopen(req)
Here's how I'm building my Agent request as of right now:
from urllib import urlencode
from StringIO import StringIO
page = 'http://example.com/'
id_string = 'my:id_string'
head = {'User-Agent': ['user agent goes here']}
data = urlencode({'id': id_string})
request = agent.request('POST', page, Headers(head), FileBodyProducer(StringIO(data)))
request.addCallback(foo)
Because of the HTTP response I'm getting (null JSON string), I'm beginning to suspect that the id is not being properly encoded in the POST request, but I'm not sure what I can be doing about it. Is using urlencode with the Agent.request call valid? Is there another way I should be encoding these things?
EDIT: Some kind IRC guys have suggested that the problem may stem from the fact that I didn't send the header information that indicates the data is encoded in a url string. I know remarkably little about this kind of stuff... Can anybody point me in the right direction?

As requested, here's my comment in the form of an answer:
HTTP requests with bodies should have the Content-Type header set (to tell the server how to interpret the bytes in the body); in this case, it seems the server is expecting URL-encoded data, like a web browser would send when a form is filled out.
urllib2.Request apparently defaults the content type for you, but the twisted library seems to need it to be set manually. In this case, you want a content type of application/x-www-form-urlencoded.

Making a POST call instead of GET using urllib2

There's a lot of stuff out there on urllib2 and POST calls, but I'm stuck on a problem.
I'm trying to do a simple POST call to a service:
url = 'http://myserver/post_service'
data = urllib.urlencode({'name' : 'joe',
'age' : '10'})
content = urllib2.urlopen(url=url, data=data).read()
print content
I can see the server logs and it says that I'm doing GET calls, when I'm sending the data
argument to urlopen.
The library is raising an 404 error (not found), which is correct for a GET call, POST calls are processed well (I'm also trying with a POST within a HTML form).

Do it in stages, and modify the object, like this:
# make a string with the request type in it:
method = "POST"
# create a handler. you can specify different handlers here (file uploads etc)
# but we go for the default
handler = urllib2.HTTPHandler()
# create an openerdirector instance
opener = urllib2.build_opener(handler)
# build a request
data = urllib.urlencode(dictionary_of_POST_fields_or_None)
request = urllib2.Request(url, data=data)
# add any other information you want
request.add_header("Content-Type",'application/json')
# overload the get method function with a small anonymous function...
request.get_method = lambda: method
# try it; don't forget to catch the result
try:
connection = opener.open(request)
except urllib2.HTTPError,e:
connection = e
# check. Substitute with appropriate HTTP code.
if connection.code == 200:
data = connection.read()
else:
# handle the error case. connection.read() will still contain data
# if any was returned, but it probably won't be of any use
This way allows you to extend to making PUT, DELETE, HEAD and OPTIONS requests too, simply by substituting the value of method or even wrapping it up in a function. Depending on what you're trying to do, you may also need a different HTTP handler, e.g. for multi file upload.

This may have been answered before: Python URLLib / URLLib2 POST.
Your server is likely performing a 302 redirect from http://myserver/post_service to http://myserver/post_service/. When the 302 redirect is performed, the request changes from POST to GET (see Issue 1401). Try changing url to http://myserver/post_service/.

Have a read of the urllib Missing Manual. Pulled from there is the following simple example of a POST request.
url = 'http://myserver/post_service'
data = urllib.urlencode({'name' : 'joe', 'age' : '10'})
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)
print response.read()
As suggested by #Michael Kent do consider requests, it's great.
EDIT: This said, I do not know why passing data to urlopen() does not result in a POST request; It should. I suspect your server is redirecting, or misbehaving.

The requests module may ease your pain.
url = 'http://myserver/post_service'
data = dict(name='joe', age='10')
r = requests.post(url, data=data, allow_redirects=True)
print r.content

it should be sending a POST if you provide a data parameter (like you are doing):
from the docs:
"the HTTP request will be a POST instead of a GET when the data parameter is provided"
so.. add some debug output to see what's up from the client side.
you can modify your code to this and try again:
import urllib
import urllib2
url = 'http://myserver/post_service'
opener = urllib2.build_opener(urllib2.HTTPHandler(debuglevel=1))
data = urllib.urlencode({'name' : 'joe',
'age' : '10'})
content = opener.open(url, data=data).read()

Try this instead:
url = 'http://myserver/post_service'
data = urllib.urlencode({'name' : 'joe',
'age' : '10'})
req = urllib2.Request(url=url,data=data)
content = urllib2.urlopen(req).read()
print content

url="https://myserver/post_service"
data["name"] = "joe"
data["age"] = "20"
data_encoded = urllib2.urlencode(data)
print urllib2.urlopen(url + "?" + data_encoded).read()
May be this can help