This is a follow up on Security Dialogflow fulfillment thread.
the answer there
explore the req.headers.authorization you will find an authentication variable
(concat these three things:
Your dialogflow username
The character ':'
Your dialogflow password
and encode it in base64)
makes sense but in my python implementation the
request headers I get is:
Accept: */*
Content-Type: application/json; charset=UTF-8
Content-Length: 571
Host: xxxxxxxx
User-Agent: Apache-HttpClient/4.5.4 (Java/1.8.0_151)
Accept-Encoding: gzip,deflate
X-Forwarded-Proto: https
X-Forwarded-For: xx.xxxx.xx..xx
PS: I tried both V1 and V2
not sure how to take care of authorization
You have to set the basic auth fields in the Fullfilment settings (the ones below the Fullfilment-URL). Only then you will receive the base64 encoded part in the Authorization Header.
This has nothing to do with your personal credentials you use to login to dialogflow! Do not use them for basic auth!
Related
I am scraping search results from google using people_also_ask module. The module itself dont have method to use proxies but I manually added proxies in the module. When I got blocked from google I printed the status and it was printing my ip address was banned from sending requests. The code I added in people_also_ask module to use proxies is
proxies = {
'http' : "http://username:passward#ip:port"
}
response = SESSION.get(URL, params=params, headers=HEADERS, proxies=proxies)
.I know it is an illegal activity but I want to know why it happens for education purpose mainly. I think the code to extract the data is irrelevant so I am adding simple code to send request using people_also_ask module
import people_also_ask as paa
queries = ["how to boil eggs","how to make cake","price of poco f1","price of wooden table","best soap in us","how much tesla worth"]
for query in queries:
questions = paa.get_related_questions(query ,40)
Note: The changes are made in first function named search() of google.py of people_also_people module
Note: I am doing searchs from browser without any problem. why is google allowing me to use google but blocked from using the script
The answer is quite simple. Although it is a proxy service, it doesn't guarantee 100% anonymity. When you send the HTTP GET request via the proxy server, the request sent by your program to the proxy server is:
GET http://www.whatsmybrowser.org/ HTTP/1.1
Host: www.whatsmybrowser.org
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: python-requests/2.10.0
Now, when the proxy server sends this request to the actual destination, it sends:
GET http://www.whatsmybrowser.org/ HTTP/1.1
Host: www.whatsmybrowser.org
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: python-requests/2.10.0
Via: 1.1 naxserver (squid/3.1.8)
X-Forwarded-For: 122.126.64.43
Cache-Control: max-age=18000
Connection: keep-alive
As you can see, it throws your IP (in my case, 122.126.64.43) in the HTTP header: X-Forwarded-For and hence the website knows that the request was sent on behalf of 122.126.64.43
Read more about this header at: https://www.rfc-editor.org/rfc/rfc7239
If you want to host your own squid proxy server and want to disable setting X-Forwarded-For header, read: http://www.squid-cache.org/Doc/config/forwarded_for/
I dont get any credit for the answer I copied this answer from the following post I found Python Requests module - proxy not working
I m trying to learn django and I'm to implement csrf token for some senstive actions.
But when I intercept the request/response I get csrf_token in every request in cookie field and the webpages where I have actually implemented the csrf_token, in those request I get another csrf token as csrfMiddleware parameter in data.
So I want to know why do I get two csrf_tokens in my request and response.
POST /demo/login/ HTTP/1.1
Host: xx.xx.xx.xx:8000
User-Agent: xxxxxx
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Referer: http://xx.xx.xx.xx:8000/demo/login/
Content-Type: application/x-www-form-urlencoded
Content-Length: 125
Connection: close
Cookie: csrftoken=n4bSbWP8p9Uce3b5iAxI0UvDG0qQq7B3OwBXisww754LYztEm0wFf9ARLpVM2v7W
Upgrade-Insecure-Requests: 1
csrfmiddlewaretoken=VIqUszlij0OLyTgYEp7V2TRsNUtBqkISmaQZz52G1WYkipyxIP6Sh8WGSjYx2IeL&username=qwerty6&password=password%40123
According to the Django documentation:
For the value stored in the cookie:
In order to protect against BREACH attacks, the token is not simply the secret; a random salt is prepended to the secret and used to scramble it.
For the csrfmiddlewaretoken:
A hidden form field with the name csrfmiddlewaretoken present in all outgoing POST forms. The value of this field is, again, the value of the secret, with a salt which is both added to it and used to scramble it. The salt is regenerated on every call to get_token() so that the form field value is changed in every such response.
Therefore the values differ due to being scrambled in a different way.
Where is the csrftoken stored?
When I access an API endpoint (logout API, it do not need the params):
POST /rest-auth/logout/ HTTP/1.1
Host: 10.10.10.105:8001
Connection: keep-alive
Content-Length: 0
Accept: application/json, text/plain, */*
Origin: http://localhost:8080
Authorization: Token 0fe2977498e51ed12ddc93026b08ab0b1a06a434
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.146 Safari/537.36
Referer: http://localhost:8080/register
Accept-Encoding: gzip, deflate
Accept-Language: zh-CN,zh;q=0.9,en;q=0.8
Cookie: sessionid=b95zopro0qvkrexj8kq6mzo1d3z2hvbl; csrftoken=z53lKL0f7VHkilYS5Ax8FMaQCU2ceouje9OeTJOgTy4gH0UgHVltAlOe2KFNNNB6
the header is upper. In the Response I get an error:
{"detail":"CSRF Failed: CSRF token missing or incorrect."}
So, the backend must have verified the csrftoken.
In the backend database, I can not find the csrftoken field:
So I want to know where it is saved in the encrypted session_data?
Given this QA in the django docs, you can see that the framework by default uses the Double Submit Cookie approach (rather than the synchronizer pattern).
This approach does not require the server to store the CSRF token, as the only check it does is comparing the token within the cookie with the one in the header (or parameter) and verify that they are equal.
The synhronizer pattern, on the other hand, does store the CSRF token somewhere in the server, and for each request it verifies its validity by comparing it with the one sent over the header ( or as before, in a POST parameter ).
You can read more about the two approaches here.
I guess you are testing your API with a web service testing application, in which case you are missing the second token somewhere in your request.
This section explains how to place the token for AJAX calls:
AJAX
While the above method can be used for AJAX POST requests, it has some inconveniences: you have to remember to pass the CSRF token in as POST data with every POST request. For this reason, there is an alternative method: on each XMLHttpRequest, set a custom X-CSRFToken header to the value of the CSRF token. This is often easier, because many JavaScript frameworks provide hooks that allow headers to be set on every request.
Seeing your request above, therefore you should place this header (with the value of the current token, of course):
X-CSRFToken: z53lKL0f7VHkilYS5Ax8FMaQCU2ceouje9OeTJOgTy4gH0UgHVltAlOe2KFNNNB6
I need to log into a website using python but the login page requires a sessionID cookie in the request header. Using Google developer tools along with a webclient(hurl.it), I was able to determine the required format of the request header that is acceptable by the webserver:
Accept: */*
Accept-Encoding: gzip, deflate
Content-Length: 85
Content-Type: application/x-www-form-urlencoded
Cookie: www_amsterdam-dance-event_nl_session=l9Abno8a1UyHPof%2BOyVqk8BxHjesGMi78z6Ot0ZXCCbI%2BxVKqjm30ALTfW%2FR7yKcDaqfEtFOyysTrjIeU8lU5ylv1TOlW6GLHY8jDfKKWSULKsUUJiTh92DbvkuYBuE6zt%2FeLs44lDna6Nz3uMCOaSARN7gCpoSz0TOcFaes8Hk9q6FikP1F9e%2B%2FsMwfUP0RTA0Rc5gJFyJPxHXNCdn%2BT49mhHYnzoIWVlxGHhlaEkZX1PPsYx1xq0BCgpb0WnPViuiZiBnQY2nz%2BBO4Uur0WPNfpSSWZg5Qxz79nYeChlRe16JhYjVOdaiUhnfEvp1jM7h%2BCdR6cUeatd7HGbftRCjINDrVuPeyB5ltVihStmzKEjOmWetI0xNuaNswsPIKKuo%2BV6JFNfdLcA6h3iy1K8o%2FA49tKGMP2rmGe4e5Jec%3Df395212364d1ffc80cf95ebf5abf3b40f9dc6441;
User-Agent: runscope/0.1
email=******%40beatswitch.com&login_token=545a46230b291&password=*****&submission=
I have produced the following request using Python requests module:
POST /my-ade/login/ HTTP/1.1
Host: www.amsterdam-dance-event.nl
Content-Length: 85
Accept-Encoding: gzip,deflate
Accept: */*
User-Agent: runscope/0.1
Connection: keep-alive
Cookie: www_amsterdam-dance-event_nl_session=l9Abno8a1UyHPof%2BOyVqk8BxHjesGMi78z6Ot0ZXCCbI%2BxVKqjm30ALTfW%2FR7yKcDaqfEtFOyysTrjIeU8lU5ylv1TOlW6GLHY8jDfKKWSULKsUUJiTh92DbvkuYBuE6zt%2FeLs44lDna6Nz3uMCOaSARN7gCpoSz0TOcFaes8Hk9q6FikP1F9e%2B%2FsMwfUP0RTA0Rc5gJFyJPxHXNCdn%2BT49mhHYnzoIWVlxGHhlaEkZX1PPsYx1xq0BCgpb0WnPViuiZiBnQY2nz%2BBO4Uur0WPNfpSSWZg5Qxz79nYeChlRe16JhYjVOdaiUhnfEvp1jM7h%2BCdR6cUeatd7HGbftRCjINDrVuPeyB5ltVihStmzKEjOmWetI0xNuaNswsPIKKuo%2BV6JFNfdLcA6h3iy1K8o%2FA49tKGMP2rmGe4e5Jec%3Df395212364d1ffc80cf95ebf5abf3b40f9dc6441;
Content-Type: application/x-www-form-urlencoded
login_token=545a46230b291&password=*****&email=******%40beatswitch.com&submission='
When I load the former request header with hurl.it, everything works perfectly and the webserver lets me log in but trying the almost-same request with the same parameters fails in python. While using python's request, the webserver presents an error page. Any help would be highly appreciated. I need a solution desperately.
EDIT:
Here is the code:
#Open the login page to get sessionID and login_token
loginURL = "https://www.amsterdam-dance-event.nl/my-ade/login/"
loginReq = session.get(loginURL)
loginSoup = BeautifulSoup(loginReq.text)
loginToken = loginSoup.find('input',attrs={'name':'login_token'})['value']
sessionID= loginReq.cookies['www_amsterdam-dance-event_nl_session']
cookie = 'www_amsterdam-dance-event_nl_session='+sessionID
#Construct the header and post it to the webserver
headers = {'Content-Length':'85','Accept':'*/*','User-Agent':' runscope/0.1','Content-Type':'application/x-www-form-urlencoded','Accept-Encoding':'gzip,deflate','Cookie':cookie}
payload = {'email':'*******#beatswitch.com','password':'********','login_token':loginToken,'submission':''}
loggedinReq = session.post(loginURL,headers=headers,data=payload)
I found the solution, thanks to Md. Mohsin. I was trying to handle the request headers and cookies manually while the requests module can handle them by itself. So I REMOVED the following line from the code and let requests take total control, and everything worked as intended:
headers = {'Content-Length':'85','Accept':'*/*','User-Agent':' runscope/0.1','Content-Type':'application/x-www-form-urlencoded','Accept-Encoding':'gzip,deflate','Cookie':cookie}
I'm trying to write Python code for Twitter OAuth authentication. I'm getting a "401 Unauthorized" error code when I attempt to request a token.
In the process of trying to diagnose my problem, I'm going thru each step of the authentication process and trying to undercover any errors I'm making. With regard to generating the "Signature Base String", I found an online tool that tries to help validate signature base strings: http://quonos.nl/oauthTester/
When I use this tool, it complains:
Bad URL encoding!
Both key and value in the POST body need to be URL encoded.
Here is an example Signature Base String that my Python code generates:
POST&https%3A%2F%2Fapi.twitter.com%2F1.1%2Foauth%2Frequest_token&oauth_callback%3Doob%26oauth_consumer_key%3DeXL46FKblmfiXHvmC3wcew%26oauth_nonce%3DTAHTO%2FmlyeJ1x9FrgFixosZPYVhvWLXmq%2BdKKTL1rTY%3D%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1391813822%26oauth_version%3D1.0
When I paste this string into the validator, it says:
Bad URL encoding!
Both key and value in the POST body need to be URL encoded.
In this case: "TAHTO/mlyeJ1x9FrgFixosZPYVhvWLXmq+dKKTL1rTY" is bad
I'm very confused because all key/value pairs in the URL are, in fact, URL encoded (I'm assuming "URL encoded" means "percent encoded" here.)
Is there anything wrong with my base string here?
Edit:
The actual HTTP request headers I'm sending to Twitter to request a token are:
POST /1.1/oauth/request_token HTTP/1.1
Accept-Encoding: identity
Content-Length: 0
Connection: close
Accept: */*
User-Agent: Python-urllib/3.2
Host: api.twitter.com
Content-Type: application/x-www-form-urlencoded format
Authorization: OAuth oauth_callback="oob", oauth_consumer_key="eXL46FKblmfiXHvmC3wcew", oauth_nonce="nBcVYSqv8FEi0d7MEs8%2BqtqvdYA9JcbnW%2BVqoP%2FGlrI%3D", oauth_signature="WT9c3U5Puam7dEnMt3DWDsyVAHw%3D", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1391815422", oauth_version="1.0"