I'm starting to learn how to use the python requests module. For practicing I tried to manage a challenge/response problem: I want to access the data on http://lema.rae.es/drae/srv/search?val=hacer
With the "Tamper Data" plugin for Firefox I inspected the necessary HTTP requests:
GET http://lema.rae.es/drae/srv/search?val=hacer
POST http://lema.rae.es/drae/srv/search?val=hacer
I copied the exact headers that are sent by Firefox in the two HTTP requests and implemented the JavaScript "challenge" function in Python. Then I'm doing the following:
url = "http://lema.rae.es/drae/srv/search?val=hacer"
headers = { ... }
r1 = requests.get(url=url, headers=headers)
html = r1.content.decode("utf-8")
formdata = challenge(html)
headers = { ... }
r2 = requests.post(url=url, data=formdata, headers=headers)
Unfortunately, the server will not answer in the expected way. I checked all the headers I'm sending via "r.request.headers" and they agree perfectly with the headers that firefox sends (according to Tamper Data)
What am I doing wrong?
You can inspect my full code here: http://pastebin.com/7JAZ9B4s
This is the response header I should be getting:
Date[Tue, 10 Feb 2015 17:13:53 GMT]
Vary[Accept-Encoding]
Content-Encoding[gzip]
Cache-Control[max-age=0, no-cache]
Keep-Alive[timeout=5, max=100]
Connection[Keep-Alive]
Content-Type[text/html; charset=UTF-8]
Set-Cookie[TS014dfc77=017ccc203c29467c4d9b347fb56ea0e89a7182e52b9d7b4a1174efbf134768569a005c7c85; Path=/]
Transfer-Encoding[chunked]
And this is the response header I really get:
Content-Length[5798]
Content-Type[text/html]
Pragma[no-cache]
Cache-Control[no-cache]
I found the reason why my code doesn't work:
The server expects the POSTDATA in exactly the same order in which the entries appear as input-elements of the form. In my code the values of the input-elements were stored in a python dict. But this data type does not preserve the order in which values have been declared!
The ruby script (referred to in the comments) however does work because the ruby dict data type seems to preserve the order of declaration!
Furthermore, reimplementing the javascript challenge() function in python was not necessary at all, because the server will be happy to accept any response string (that worked in the past) over and over again!
Related
I have to send a POST request to the /batch endpoint of : 'https://www.google-analytics.com'.
As mentioned in the Documentation I have to send the request to /batch endpoint and specify each payload on its own line.
I was able to achieve this using POSTMAN as follows:
My query is to make a POST request using Python's requests library
I tried something like this :
import requests
text = '''v=1&cid=43223523&tid=UA-XXXXXX-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=bookmarks&ev=13
v=1&cid=43223523&tid=UA-XXXXXX-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=upvotes&ev=65
v=1&cid=43223523&tid=UA-XXXXXX-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=questions&ev=15
v=1&cid=43223523&tid=UA-XXXXXX-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=postviews&ev=95'''
response = requests.post('https://www.google-analytics.com/batch', data=text)
but it doesn't works.
UPDATE
I Tried this and it works !
import http.client
conn = http.client.HTTPSConnection("www.google-analytics.com")
payload = "v=1&cid=43223523&tid=UA-200248207-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=bookmarks&ev=13\r\nv=1&cid=43223523&tid=UA-200248207-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=upvotes&ev=63\r\nv=1&cid=43223523&tid=UA-200248207-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=questions&ev=11\r\nv=1&cid=43223523&tid=UA-200248207-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=postviews&ev=23"
headers = {
'Content-Type': 'text/plain'
}
conn.request("POST", "/batch", payload, headers)
res = conn.getresponse()
But the question remains open, what's the issue with requests here.
You don't need to double-escape the newline symbol.
Moreover, you don't need the newline symbol at all for the multi-line string.
And also the indentations you put in your multi-line string are counted:
test = '''abc
def
ghi'''
print(test)
Here's an SO answer that explains this with some additional ways to make long stings: https://stackoverflow.com/a/10660443/4570170
Now the request body.
The documentation says
payload_data – The BODY of the post request. The body must include exactly 1 URI encoded payload and must be no longer than 8192 bytes.
So try uri-encoding your payload:
text = '''v=1&cid=43223523&tid=UA-XXXXXX-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=bookmarks&ev=13
v=1&cid=43223523&tid=UA-XXXXXX-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=upvotes&ev=65
v=1&cid=43223523&tid=UA-XXXXXX-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=questions&ev=15
v=1&cid=43223523&tid=UA-XXXXXX-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=postviews&ev=95'''
text_final = requests.utils.quote(text)
response = requests.post('https://www.google-analytics.com/batch', data=text_final)
Finally , I figured out the solution myself.
Updating for others help.
The problem was I was working on AWS Cloud9 and as mentioned in the documentation
Some environments are not able to send hits to Google Analytics directly. Examples of this are older mobile phones that can't run JavaScript or corporate intranets behind a firewall.
So we just need to include the User Agent parameter
ua=Opera/9.80
in each of our payloads
It works !
I am trying to use AWS DynamoDB in a Flutter app, and given the lack of an official AWS SDK for Dart I am forced to use the low level HTTP REST API.
The method for signing an AWS HTTP request is quite tedious, but using an AWS supplied sample as a guide, I was able to convert the Python to Dart pretty much line-for-line relatively easily. The end result was both sets of code producing the same auth signatures.
My issue came when I actually went to sent the request. The Python works as expected but sending a POST with Dart's HTTP package gives the error
The request signature we calculated does not match the signature you
provided. Check your AWS Secret Access Key and signing method. Consult
the service documentation for details.
I'll spare you the actual code for generating the auth signature, as the issue can be replicated simply by sending the same request hard-coded. See the Python and Dart code below.
Note: A valid response will return
Signature expired: 20190307T214900Z is now earlier than
20190307T215809Z (20190307T221309Z - 15 min.)
as the request signature uses current date and is only valid for 15 mins.
*****PYTHON CODE*****
import requests
headers = {'Content-Type':'application/json',
'X-Amz-Date':'20190307T214900Z',
'X-Amz-Target':'DynamoDB_20120810.GetItem',
'Authorization':'AWS4-HMAC-SHA256 Credential=AKIAJFZWA7QQAQT474EQ/20190307/ap-southeast-2/dynamodb/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-target, Signature=297c5a03c59db6da45bfe2fda6017f89a0a1b2ab6da2bb6e0d838ca40be84320'}
endpoint = 'https://dynamodb.ap-southeast-2.amazonaws.com/'
request_parameters = '{"TableName": "player-exports","Key": {"exportId": {"S": "HG1T"}}}'
r = requests.post(endpoint, data=request_parameters, headers=headers)
print('Response status: %d\n' % r.status_code)
print('Response body: %s\n' % r.text)
*****DART CODE*****
import 'package:http/http.dart' as http;
void main(List<String> arguments) async {
var headers = {'Content-Type':'application/json',
'X-Amz-Date':'20190307T214900Z',
'X-Amz-Target':'DynamoDB_20120810.GetItem',
'Authorization':'AWS4-HMAC-SHA256 Credential=AKIAJFZWA7QQAQT474EQ/20190307/ap-southeast-2/dynamodb/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-target, Signature=297c5a03c59db6da45bfe2fda6017f89a0a1b2ab6da2bb6e0d838ca40be84320'};
var endpoint = 'https://dynamodb.ap-southeast-2.amazonaws.com/';
var request_parameters = '{"TableName": "player-exports","Key": {"exportId": {"S": "HG1T"}}}';
http.post(endpoint, body: request_parameters, headers: headers).then((response) {
print("Response status: ${response.statusCode}");
print("Response body: ${response.body}");
});
}
The endpoint, headers and body are literally copy and pasted between the two sets of code.
Is there some nuance to how Dart HTTP works that I am missing here? Is there some map/string/json conversion of the headers or request_paramaters happening?
One thing I did note is that in the AWS provided example it states
For DynamoDB, the request can include any headers, but MUST include
"host", "x-amz-date", "x-amz-target", "content-type", and
"Authorization". Except for the authorization header, the headers must
be included in the canonical_headers and signed_headers values, as
noted earlier. Order here is not significant. Python note: The 'host'
header is added automatically by the Python 'requests' library.
But
a) When I add 'Host':'dynamodb.ap-southeast-2.amazonaws.com' to the headers in the Dart code I get the same result
and
b) If I look at r.request.headers after the Python requests returns, I can see that it has added a few new headers (Content-Length etc) automatically, but "Host" isn't one of them.
Any ideas why the seemingly same HTTP request works for Python Requests but not Dart HTTP?
Ok this is resolved now. My issue was in part a massive user-error. I was using a new IDE and when I generated the hardcoded example I provided I was actually still executing the previous file. Stupid, stupid, stupid.
But...
I was able to sort out the actual issue that caused me raise the question in the first place. I found that if you set the content type to "application/json" in the headers, the dart HTTP package automatically appends "; charset=utf-8". Because this value is part of the auth signature, when AWS encodes the values from the header to compare to the user-generated signature, they don't match.
The fix is simply to ensure that when you are setting the header content-type, make sure that you manually set it to "application/json; charset=utf-8" and not "application/json".
Found a bit more discussion about this "bug" after the fact here.
i quite new to pyhton. I just try a simple way to get an HTTP response with python to a simple get from the sonar Web API
i use the request library and try a simple use :
project = requests.get(url=Sonar_Api_Projects_Search, params=param_Projects, verify=False, headers={'Authorization': 'token {}'.format(token)})
the request is well formatted and work fine when i use it in e web browser.
but as a response i get this strange output :
{"err_code":500,"err_msg":"undefined method empty?' for
nil:NilClass\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/lib/authenticated_system.rb:132:in
login_from_basic_auth'\n\torg/jruby/RubyProc.java:290:in
call'\n\torg/jruby/RubyProc.java:224:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/http_authentication.rb:126:in
authenticate'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/http_authentication.rb:116:in
authenticate_with_http_basic'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/lib/authenticated_system.rb:129:in
login_from_basic_auth'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/lib/authenticated_system.rb:11:in
current_user'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/app/controllers/application_controller.rb:102:in set_user_session'\n\torg/jruby/RubyKernel.java:2223:in
send'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/activesupport-2.3.15/lib/active_support/callbacks.rb:178:in
evaluate_method'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/activesupport-2.3.15/lib/active_support/callbacks.rb:166:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/filters.rb:225:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/filters.rb:629:in
run_before_filters'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/filters.rb:615:in
call_filters'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/filters.rb:610:in
perform_action_with_filters'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/benchmarking.rb:68:in
perform_action_with_benchmark'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/activesupport-2.3.15/lib/active_support/core_ext/benchmark.rb:17:in
ms'\n\tjar:file:/D:/sonarqube-5.6.6_20170214/lib/server/jruby-complete-1.7.9.jar!/META-INF/jruby.home/lib/ruby/1.8/benchmark.rb:308:in
realtime'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/activesupport-2.3.15/lib/active_support/core_ext/benchmark.rb:17:in
ms'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/benchmarking.rb:68:in
perform_action_with_benchmark'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/rescue.rb:160:in
perform_action_with_rescue'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/flash.rb:151:in perform_action_with_flash'\n\torg/jruby/RubyKernel.java:2223:in
send'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/base.rb:532:in
process'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/filters.rb:606:in
process_with_filters'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/base.rb:391:in
process'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/base.rb:386:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/routing/route_set.rb:450:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/dispatcher.rb:87:in
dispatch'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/dispatcher.rb:85:in
dispatch'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/dispatcher.rb:121:in
_call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/dispatcher.rb:130:in
build_middleware_stack'\n\torg/jruby/RubyProc.java:290:in
call'\n\torg/jruby/RubyProc.java:224:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/query_cache.rb:29:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/connection_adapters/abstract/query_cache.rb:34:in
cache'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/query_cache.rb:9:in
cache'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/query_cache.rb:28:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/activerecord-2.3.15/lib/active_record/connection_adapters/abstract/connection_pool.rb:361:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/config/environment.rb:67:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/string_coercion.rb:25:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/rack-1.1.6/lib/rack/head.rb:9:in call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/rack-1.1.6/lib/rack/methodoverride.rb:24:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/params_parser.rb:15:in
call'\n\tfile:/D:/sonarqube-5.6.6_20170214/lib/server/jruby-rack-1.1.13.2.jar!/jruby/rack/session_store.rb:70:in
context'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/rack-1.1.6/lib/rack/session/abstract/id.rb:58:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/failsafe.rb:26:in
call'\n\tD:/sonarqube-5.6.6_20170214/web/WEB-INF/gems/gems/actionpack-2.3.15/lib/action_controller/dispatcher.rb:106:in
call'\n\tfile:/D:/sonarqube-5.6.6_20170214/lib/server/jruby-rack-1.1.13.2.jar!/rack/adapter/rails.rb:34:in
serve_rails'\n\tfile:/D:/sonarqube-5.6.6_20170214/lib/server/jruby-rack-1.1.13.2.jar!/rack/adapter/rails.rb:39:in
call'\n\tfile:/D:/sonarqube-5.6.6_20170214/lib/server/jruby-rack-1.1.13.2.jar!/rack/handler/servlet.rb:22:in
call'\n"}
Can someone help me ?
Thanks a lot
Best regards
Arnaud
Direct use of requests never worked for me.
I do the following and it is working fine:
(below code is to list projects in Sonar)
import json , requests, pprint
url = 'http://sonar_url:9000/api/projects/search'
myToken = 'fa2377941a95125443f4efade615512jjkd221211a48'
session = requests.Session()
session.auth = myToken, ''
call = getattr(session, 'get')
res = call(url)
print(res.status_code)
binary = res.content
output = json.loads(binary)
pprint.pprint(output)
...
#Parse json result
In Sonarqube 8.9, requests is working for me.
First, you should should create an API token. Per the docs:
This is the recommended way. Benefits are described in the page User Token. The token is sent via the login field of HTTP basic authentication, without any password.
The docs go on to provide a weird curl usage example:
# note that the colon after the token is required in curl to set an empty password
curl -u THIS_IS_MY_TOKEN: https://sonarqube.com/api/user_tokens/search
In requests, this looks something like this:
response = requests.get(
"http://your-sonar-instance.com/api/blah",
auth=HTTPBasicAuth("Some Sonarqube API token", "")
)
return json.loads(response.text)
See https://docs.sonarqube.org/latest/extend/web-api/ for API details.
Also note that auth=HTTPBasicAuth("token", "") seems to behave differently from auth=HTTPBasicAuth("token", None).
I know its an old question. Thankfully there is a wrapper library available now - https://github.com/shijl0925/python-sonarqube-api. It works quite well and is easy to setup.
If possible people from Sonarsource could make it the official one so that more people start using it and it gets maintained in the future too.
Before downvoting/marking as duplicate, please note:
I have already tried out this, this, this, this,this, this - basically almost all the methods I could find pointed out by the Requests documentation but do not seem to find any solution.
Problem:
I want to make a POST request with a set of headers and form data.
There are no files to be uploaded. As per the request body in Postman, we set the parameters by selecting 'form-data' under the 'Body' section for the request.
Here is the code I have:
headers = {'authorization': token_string,
'content-type':'multipart/form-data; boundary=----WebKitFormBoundaryxxxxxXXXXX12345'} # I get 'unsupported application/x-www-form-url-encoded' error if I remove this line
body = {
'foo1':'bar1',
'foo2':'bar2',
#... and other form data, NO FILE UPLOADED
}
#I have also tried the below approach
payload = dict()
payload['foo1']='bar1'
payload['foo2']='bar2'
page = ''
page = requests.post(url, proxies=proxies, headers=headers,
json=body, files=json.dump(body)) # also tried data=body,data=payload,files={} when giving data values
Error
{"errorCode":404,"message":"Required String parameter 'foo1' is not
present"}
EDIT:
Adding a trace of the network console. I am defining it in the same way in the payload as mentioned on the request payload.
There isn't any gui at all? You could get the network data from chrome, although:
Try this:
headers = {'authorization': token_string}
Probably there is more authorization? Or smthng else?
You shouldn't add Content-Type as requests will handle it for you.
Important, you could see the content type as WebKitFormBoundary, so for the payload you must take, the data from the "name" variable.
Example:
(I know you won't upload any file, it just an example) -
So in this case, for my payload would look like this: payload = {'photo':'myphoto'} (yea there would be an open file etc etc, but I try to keep it simple)
So your payload would be this-> (So always use name from the WebKit)
payload = {'foo1':'foo1data',
'foo2':'foo2data'}
session.post(url,data = payload, proxies etc...)
Important! As I can see you use the method from requests library. Firstly you always should create a session like this
session = requests.session() -> it will handle cookies, headers, etc, and won't open a new session, or plain requests with every requests.get/post.
I'm trying to create a super-simplistic Virtual In / Out Board using wx/Python. I've got the following code in place for one of my requests to the server where I'll be storing the data:
data = urllib.urlencode({'q': 'Status'})
u = urllib2.urlopen('http://myserver/inout-tracker', data)
for line in u.readlines():
print line
Nothing special going on there. The problem I'm having is that, based on how I read the docs, this should perform a Post Request because I've provided the data parameter and that's not happening. I have this code in the index for that url:
if (!isset($_POST['q'])) { die ('No action specified'); }
echo $_POST['q'];
And every time I run my Python App I get the 'No action specified' text printed to my console. I'm going to try to implement it using the Request Objects as I've seen a few demos that include those, but I'm wondering if anyone can help me explain why I don't get a Post Request with this code. Thanks!
-- EDITED --
This code does work and Posts to my web page properly:
data = urllib.urlencode({'q': 'Status'})
h = httplib.HTTPConnection('myserver:8080')
headers = {"Content-type": "application/x-www-form-urlencoded",
"Accept": "text/plain"}
h.request('POST', '/inout-tracker/index.php', data, headers)
r = h.getresponse()
print r.read()
I am still unsure why the urllib2 library doesn't Post when I provide the data parameter - to me the docs indicate that it should.
u = urllib2.urlopen('http://myserver/inout-tracker', data)
h.request('POST', '/inout-tracker/index.php', data, headers)
Using the path /inout-tracker without a trailing / doesn't fetch index.php. Instead the server will issue a 302 redirect to the version with the trailing /.
Doing a 302 will typically cause clients to convert a POST to a GET request.