I'm trying to write Python code for Twitter OAuth authentication. I'm getting a "401 Unauthorized" error code when I attempt to request a token.
In the process of trying to diagnose my problem, I'm going thru each step of the authentication process and trying to undercover any errors I'm making. With regard to generating the "Signature Base String", I found an online tool that tries to help validate signature base strings: http://quonos.nl/oauthTester/
When I use this tool, it complains:
Bad URL encoding!
Both key and value in the POST body need to be URL encoded.
Here is an example Signature Base String that my Python code generates:
POST&https%3A%2F%2Fapi.twitter.com%2F1.1%2Foauth%2Frequest_token&oauth_callback%3Doob%26oauth_consumer_key%3DeXL46FKblmfiXHvmC3wcew%26oauth_nonce%3DTAHTO%2FmlyeJ1x9FrgFixosZPYVhvWLXmq%2BdKKTL1rTY%3D%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1391813822%26oauth_version%3D1.0
When I paste this string into the validator, it says:
Bad URL encoding!
Both key and value in the POST body need to be URL encoded.
In this case: "TAHTO/mlyeJ1x9FrgFixosZPYVhvWLXmq+dKKTL1rTY" is bad
I'm very confused because all key/value pairs in the URL are, in fact, URL encoded (I'm assuming "URL encoded" means "percent encoded" here.)
Is there anything wrong with my base string here?
Edit:
The actual HTTP request headers I'm sending to Twitter to request a token are:
POST /1.1/oauth/request_token HTTP/1.1
Accept-Encoding: identity
Content-Length: 0
Connection: close
Accept: */*
User-Agent: Python-urllib/3.2
Host: api.twitter.com
Content-Type: application/x-www-form-urlencoded format
Authorization: OAuth oauth_callback="oob", oauth_consumer_key="eXL46FKblmfiXHvmC3wcew", oauth_nonce="nBcVYSqv8FEi0d7MEs8%2BqtqvdYA9JcbnW%2BVqoP%2FGlrI%3D", oauth_signature="WT9c3U5Puam7dEnMt3DWDsyVAHw%3D", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1391815422", oauth_version="1.0"
Related
I am scraping search results from google using people_also_ask module. The module itself dont have method to use proxies but I manually added proxies in the module. When I got blocked from google I printed the status and it was printing my ip address was banned from sending requests. The code I added in people_also_ask module to use proxies is
proxies = {
'http' : "http://username:passward#ip:port"
}
response = SESSION.get(URL, params=params, headers=HEADERS, proxies=proxies)
.I know it is an illegal activity but I want to know why it happens for education purpose mainly. I think the code to extract the data is irrelevant so I am adding simple code to send request using people_also_ask module
import people_also_ask as paa
queries = ["how to boil eggs","how to make cake","price of poco f1","price of wooden table","best soap in us","how much tesla worth"]
for query in queries:
questions = paa.get_related_questions(query ,40)
Note: The changes are made in first function named search() of google.py of people_also_people module
Note: I am doing searchs from browser without any problem. why is google allowing me to use google but blocked from using the script
The answer is quite simple. Although it is a proxy service, it doesn't guarantee 100% anonymity. When you send the HTTP GET request via the proxy server, the request sent by your program to the proxy server is:
GET http://www.whatsmybrowser.org/ HTTP/1.1
Host: www.whatsmybrowser.org
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: python-requests/2.10.0
Now, when the proxy server sends this request to the actual destination, it sends:
GET http://www.whatsmybrowser.org/ HTTP/1.1
Host: www.whatsmybrowser.org
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: python-requests/2.10.0
Via: 1.1 naxserver (squid/3.1.8)
X-Forwarded-For: 122.126.64.43
Cache-Control: max-age=18000
Connection: keep-alive
As you can see, it throws your IP (in my case, 122.126.64.43) in the HTTP header: X-Forwarded-For and hence the website knows that the request was sent on behalf of 122.126.64.43
Read more about this header at: https://www.rfc-editor.org/rfc/rfc7239
If you want to host your own squid proxy server and want to disable setting X-Forwarded-For header, read: http://www.squid-cache.org/Doc/config/forwarded_for/
I dont get any credit for the answer I copied this answer from the following post I found Python Requests module - proxy not working
I m trying to learn django and I'm to implement csrf token for some senstive actions.
But when I intercept the request/response I get csrf_token in every request in cookie field and the webpages where I have actually implemented the csrf_token, in those request I get another csrf token as csrfMiddleware parameter in data.
So I want to know why do I get two csrf_tokens in my request and response.
POST /demo/login/ HTTP/1.1
Host: xx.xx.xx.xx:8000
User-Agent: xxxxxx
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Referer: http://xx.xx.xx.xx:8000/demo/login/
Content-Type: application/x-www-form-urlencoded
Content-Length: 125
Connection: close
Cookie: csrftoken=n4bSbWP8p9Uce3b5iAxI0UvDG0qQq7B3OwBXisww754LYztEm0wFf9ARLpVM2v7W
Upgrade-Insecure-Requests: 1
csrfmiddlewaretoken=VIqUszlij0OLyTgYEp7V2TRsNUtBqkISmaQZz52G1WYkipyxIP6Sh8WGSjYx2IeL&username=qwerty6&password=password%40123
According to the Django documentation:
For the value stored in the cookie:
In order to protect against BREACH attacks, the token is not simply the secret; a random salt is prepended to the secret and used to scramble it.
For the csrfmiddlewaretoken:
A hidden form field with the name csrfmiddlewaretoken present in all outgoing POST forms. The value of this field is, again, the value of the secret, with a salt which is both added to it and used to scramble it. The salt is regenerated on every call to get_token() so that the form field value is changed in every such response.
Therefore the values differ due to being scrambled in a different way.
Where is the csrftoken stored?
When I access an API endpoint (logout API, it do not need the params):
POST /rest-auth/logout/ HTTP/1.1
Host: 10.10.10.105:8001
Connection: keep-alive
Content-Length: 0
Accept: application/json, text/plain, */*
Origin: http://localhost:8080
Authorization: Token 0fe2977498e51ed12ddc93026b08ab0b1a06a434
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.146 Safari/537.36
Referer: http://localhost:8080/register
Accept-Encoding: gzip, deflate
Accept-Language: zh-CN,zh;q=0.9,en;q=0.8
Cookie: sessionid=b95zopro0qvkrexj8kq6mzo1d3z2hvbl; csrftoken=z53lKL0f7VHkilYS5Ax8FMaQCU2ceouje9OeTJOgTy4gH0UgHVltAlOe2KFNNNB6
the header is upper. In the Response I get an error:
{"detail":"CSRF Failed: CSRF token missing or incorrect."}
So, the backend must have verified the csrftoken.
In the backend database, I can not find the csrftoken field:
So I want to know where it is saved in the encrypted session_data?
Given this QA in the django docs, you can see that the framework by default uses the Double Submit Cookie approach (rather than the synchronizer pattern).
This approach does not require the server to store the CSRF token, as the only check it does is comparing the token within the cookie with the one in the header (or parameter) and verify that they are equal.
The synhronizer pattern, on the other hand, does store the CSRF token somewhere in the server, and for each request it verifies its validity by comparing it with the one sent over the header ( or as before, in a POST parameter ).
You can read more about the two approaches here.
I guess you are testing your API with a web service testing application, in which case you are missing the second token somewhere in your request.
This section explains how to place the token for AJAX calls:
AJAX
While the above method can be used for AJAX POST requests, it has some inconveniences: you have to remember to pass the CSRF token in as POST data with every POST request. For this reason, there is an alternative method: on each XMLHttpRequest, set a custom X-CSRFToken header to the value of the CSRF token. This is often easier, because many JavaScript frameworks provide hooks that allow headers to be set on every request.
Seeing your request above, therefore you should place this header (with the value of the current token, of course):
X-CSRFToken: z53lKL0f7VHkilYS5Ax8FMaQCU2ceouje9OeTJOgTy4gH0UgHVltAlOe2KFNNNB6
This is a follow up on Security Dialogflow fulfillment thread.
the answer there
explore the req.headers.authorization you will find an authentication variable
(concat these three things:
Your dialogflow username
The character ':'
Your dialogflow password
and encode it in base64)
makes sense but in my python implementation the
request headers I get is:
Accept: */*
Content-Type: application/json; charset=UTF-8
Content-Length: 571
Host: xxxxxxxx
User-Agent: Apache-HttpClient/4.5.4 (Java/1.8.0_151)
Accept-Encoding: gzip,deflate
X-Forwarded-Proto: https
X-Forwarded-For: xx.xxxx.xx..xx
PS: I tried both V1 and V2
not sure how to take care of authorization
You have to set the basic auth fields in the Fullfilment settings (the ones below the Fullfilment-URL). Only then you will receive the base64 encoded part in the Authorization Header.
This has nothing to do with your personal credentials you use to login to dialogflow! Do not use them for basic auth!
I am new to Web requests . I saw a piece of code that does HTML PDF conversion like this :
headers = {'content-type': 'text/html', 'accept': 'application/pdf'}
urllib2.Request(url, data=html, headers=headers) # html is a string and it works fine
The url does the pdf conversion and it needs html as input .
Why is 'data' keyword argument so important ? Why can't be clubbed as just another param ?
I would have thought that urllib2.Request(url, params = {'data': html}) where data is just one of the key value pairs . And server does it processing accordingly .
Why do we need 'data' as something seperate from other parameters ?
Is it because we specify 'content-type' in the header and it bound to the data keyword as a convention ?
I am writing an API that makes everything is request like a keyword arguement , for a simple library purpose . So I would like to know when is data required and when is not as well . I do understand params but 'data' is that mandatory or only for post requests where you have a specific content-type to sent to server? What if I have multiple content types now ?
When the data attribute is provided, the request is sent as POST. It is not mandatory, it can be None, if it is none (or not provided) it is sent as GET. This is all described here: http://docs.python.org/2/library/urllib2.html#urllib2.Request
Does request also have the same convention ? I ask so because in
request we have request.get . So request.get(url, data=something)
would be converted to a POST ? And how is this data seen at the server
side any idea ?
request.get(url, data="test") would be sent as a GET request with "test" as the body of the request. This is the raw HTTP request:
GET /headers HTTP/1.1\r\nHost: httpbin.org\r\nContent-Length: 4\r\nAccept-Encoding: gzip, deflate, compress\r\nAccept: */*\r\nUser-Agent: python-requests/2.2.1 CPython/2.7.5 Windows/7\r\n\r\ntest
Formatted:
GET /headers HTTP/1.1
Host: httpbin.org
Content-Length: 4
Accept-Encoding: gzip, deflate, compress
Accept: */*
User-Agent: python-requests/2.2.1 CPython/2.7.5 Windows/7
test
The server will in most cases just ignore it.