I have this HTTPS call in curl below;
header1="projectName: zhikovapp"
header2="Authorization: Bearer HZCdsf="
bl_url="https://BlazerNpymh.com/api/documents?pdfDate=$today"
curl -s -k -H "$header1" -H "$header2" "$bl_url"
I would like to write an equivalent python call using requests module.
header ={
"projectName": "zhikovapp",
"Authorization": "Bearer HZCdsf="
}
response = requests.get(bl_url, headers = header)
However, the request was not valid. What is wrong?
The contents of the returned response is like this;
<Response [400]>
_content = '{"Message":"The request is invalid."}'
headers = {'Content-Length': '37', 'Access-Control-Allow-Headers': 'projectname, authorization, Content-Type', 'Expires': '-1', 'cacheControlHeader': 'max-age=604800', 'Connection': 'keep-alive', 'Pragma': 'no-cache', 'Cache-Control': 'no-cache', 'Date': 'Sat, 15 Oct 2016 02:41:13 GMT', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Methods': 'GET, POST, PUT, DELETE, OPTIONS', 'Content-Type': 'application/json; charset=utf-8'}
reason = 'Bad Request'
I am using python 2.7
EDIT: I corrected some syntex errors after Soviut pointed them out.
In request.get() the headers argument should be defined as a dictionary, a set of key/value pairs. You've defined a set (a unique list) of strings instead.
You should declare your headers like this:
headers = {
"projectName": "zhikovapp",
"Authorization": "Bearer HZCdsf="
}
response = requests.get(bl_url, headers=headers)
Note the "key": "value" format of each line inside the dictionary.
Edit: Your Access-Control-Allow-Headers say they'll accept projectname and authorization in lower case. You've named your header projectName and Authorization with upper case letters in them. If they don't match, they'll be rejected.
If you have $today defined in the shell you make curl call from, and you don't substitute it in the requests' call URL, then it's a likely reason for the 400 Bad Request.
Access-Control-* and other CORS headers have nothing to do with non-browser clients. Also HTTP headers are generally case insensitive.
Following #furas's advice here's the output:
$ curl -H "projectName: zhikovapp" -H "Authorization: Bearer HZCdsf=" \
http://httpbin.org/get
{
"args": {},
"headers": {
"Accept": "*/*",
"Authorization": "Bearer HZCdsf=",
"Host": "httpbin.org",
"Projectname": "zhikovapp",
"User-Agent": "curl/7.35.0"
},
"origin": "1.2.3.4",
"url": "http://httpbin.org/get"
}
And the same request with requests:
import requests
res = requests.get('http://httpbin.org/get', headers={
"projectName" : "zhikovapp",
"Authorization" : "Bearer HZCdsf="
})
print(res.json())
{
'args': {},
'headers': {
'Accept': '*/*',
'Accept-Encoding': 'gzip, deflate, compress',
'Authorization': 'Bearer HZCdsf=',
'Host': 'httpbin.org',
'Projectname': 'zhikovapp',
'User-Agent': 'python-requests/2.2.1 CPython/3.4.3 '
'Linux/3.16.0-38-generic'
},
'origin': '1.2.3.4',
'url': 'http://httpbin.org/get'
}
As you can see the only difference is User-Agent header. It's unlikely the cause but you can easily set it in headers to the value you like.
Related
I have the following Python function which sends a post request using the requests library:
def http_post(self, url: str, headers: dict, data: str, auth: AuthBase):
token = self._xsuaa.get_token(self._service)
headers.update({'Proxy-Authorization': f"Bearer {token}"})
res = requests.post(
url,
headers=headers,
data=data,
proxies={'http': self._proxy},
auth=auth,
verify=False,
timeout=100,
allow_redirects=True)
When printing the headers dict, it looks like this:
{
'Content-Type': 'multipart/mixed;boundary=batch_4724f345-bb46-437d-a970-197a7b82bf41',
'Content-Transfer-Encoding': 'binary',
'sap-cancel-on-close': 'true',
'sap-contextid-accept': 'header',
'Accept': 'application/json',
'Accept-Language': 'de-DE',
'DataServiceVersion': '2.0',
'MaxDataServiceVersion': '2.0',
'Proxy-Authorization': 'Bearer <token>'
}
However, when I take a look at res.request.headers, I get the following:
{
'User-Agent': 'python-requests/2.26.0',
'Accept-Encoding': 'gzip, deflate',
'Accept': 'application/json',
'Connection': 'keep-alive',
'Content-Type': 'multipart/mixed; boundary=batch_4724f345-bb46-437d-a970-197a7b82bf41',
'Content-Transfer-Encoding': 'binary',
'sap-cancel-on-close': 'true',
'sap-contextid-accept': 'header',
'Accept-Language': 'de-DE',
'DataServiceVersion': '2.0',
'MaxDataServiceVersion': '2.0',
'Content-Length': '659',
'Authorization': 'Basic <auth>'
}
For some reason, the proxy-authorization header field is gone and accordingly, I get a 407 error in the response. I have read in the documentation that proxy credentials provided in the URL overwrite proxy-authorization headers, but my URL contains none. I also tried removing the auth=auth line from the request, but the problem still persited. Can someone point me in the right direction as to why this field is seemingly ignored or overwritten by requests?
I inspected how a request was sent to a website in firefox:
(Unfortunately I had to change the website URL to a fake one to prevent the server form being requested too much).
I tried to do this request in python:
import requests
import json
seq = 'ATGGCAGACTCTATTGAGGTC'
url = 'http://www.test.com'
body = {'QUERY': seq}
headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
r = requests.post(url, data=json.dumps(body), headers=headers)
print(r.text)
However when doing this the website says: Empty gene sequence passed for blast analysis. Please enter a valid gene sequence. So that means that the sequence (i.e. QUERY) is not sent correctly to the server. What am I missing here?
(P.s. hopefully missing of the website is not a problem to answer this question, if it is please let me know maybe I can ask to mention their website)
I am guessing the string / sequence that you are submitting to that particular website is the problem. I ran your sample code against a POST accepting website:
import requests
import json
seq = 'ATGGCAGACTCTATTGAGGTC'
url = 'http://httpbin.org/post'
body = {'QUERY': seq}
headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
r = requests.post(url, data=json.dumps(body), headers=headers)
print(r.text)
And got this result, which shows your query properly formed:
{
"args": {},
"data": "{\"QUERY\": \"ATGGCAGACTCTATTGAGGTC\"}",
"files": {},
"form": {},
"headers": {
"Accept": "text/plain",
"Accept-Encoding": "gzip, deflate",
"Content-Length": "34",
"Content-Type": "application/json",
"Host": "httpbin.org",
"User-Agent": "python-requests/2.22.0"
},
"json": {
"QUERY": "ATGGCAGACTCTATTGAGGTC"
},
"origin": "2.122.222.8, 2.122.222.8",
"url": "https://httpbin.org/post"
}
Are you sure the it's supposed to be "QUERY=<>?" Cause it could be incorrect formatting of the body. Normally it's in JSON format as in "title: information." Note the ':' rather than the '='
I am using requests library to query a F5 Big IP. I get the list of Virtual Server. I need to do a loop to get each VS name (VS1, VS2, VS3) from the response to use in another request like
https://localhost/mgmt/tm/ltm/virtual/VS1
What code will get each name value from the response? I tried this but could not get it to work.
url = "https://bigipname.domain.local/mgmt/tm/ltm/virtual"
querystring = {"$select":"name"}
headers = {
'Content-Type': "application/json",
'Accept': "*/*",
'Cache-Control': "no-cache",
'Host': "bigipgname.domain.local",
'accept-encoding': "gzip, deflate",
'Connection': "keep-alive",
'cache-control': "no-cache"
}
response = requests.request("GET", url, headers=headers, params=querystring, verify=False)
I get the response in the following json format :
{'kind': 'tm:ltm:virtual:virtualcollectionstate', 'selfLink': 'https://localhost/mgmt/tm/ltm/virtual?$select=name&ver=13.1.1.2', 'items': [{'name': 'VS1'}, {'name': 'VS2'}, {'name': 'VS3'}]}
Any help is appreciated. Thanks
You can use a list comprehension to extract the "items".
new_list = [item["name"] for item in response["items"]]
I have a python 3.x application that uses requests to submit data to an API. The API requires an Authorization header. Relevant code looks something like this:
the_headers = {'Authorization': auth_header_token}
response = requests.post(
the_url,
json=the_data,
headers=the_headers
)
Works fine when I run it on a server or client machine.
BUT when I run it from a Docker container, I get a bad authorization header error.
In both scenarios, I get the following output:
print(res.request.headers)
{
'User-Agent': 'python-requests/2.18.4',
'Accept-Encoding': 'gzip, deflate',
'Accept': '*/*',
'Connection': 'keep-alive',
'Authorization': 'auth_header_token',
'Content-Length': '2984',
'Content-Type': 'application/json'
}
But the team that supports the API tells me my failing request does not contain the Auth header
Good request:
{
'x-ssl': 'on',
host: 'ip1:port1',
connection: 'close',
'content-length': '2984',
accept: '*/*',
'accept-encoding': 'gzip, deflate',
authorization: 'auth_header_token',
'content-type': 'application/json',
'user-agent': 'python-requests/2.18.4',
'x-forwarded-for': 'ip2',
'x-forwarded-port': 'port2',
'x-forwarded-proto': 'https'
}
Bad request:
{
'x-ssl': 'on',
host: 'ip1:port1',
connection: 'close',
'content-length': '2984',
accept: '*/*',
'accept-encoding': 'gzip, deflate',
'content-type': 'application/json',
'user-agent': 'python-requests/2.18.4',
'x-forwarded-for': 'ip3',
'x-forwarded-port': 'port2',
'x-forwarded-proto': 'https'
}
I had originally thought that this issue was perhaps related to an issue on Github Session's Authorization header isn't sent on redirect, but both requests have x-forwarded-for on the receiving end, so now I'm not so sure.
Same code base, only thing that changes is runtime environment (MacOS vs Docker container running on MacOS).
I am trying to hit the Bitbucket API for my account, and a successful attempt looks like:
curl --user screename:mypassword https://api.bitbucket.org/1.0/user/repositories
in the command line. In python, I try:
import requests
url = 'https://api.bitbucket.org/1.0/user/repositories'
then
r = requests.post(url, data={'username': myscreename, 'password':mypassword})
and
r = requests.post(url, data="myscreename:mypassword")
and
r = requests.post(url, data={"user": "myscreename:mypassword"})
all get 405 error. The API is https://confluence.atlassian.com/bitbucket/rest-apis-222724129.html.
I wonder:
What am I doing wrong in the requests version, they all look similar to my curl attempt
What is the difference between requesting with curl and python requests module? What general pattern can I recognize when reading an API with a curl example and then writing it in python?
Thank you
ANSWER:
it required the right headers
https://answers.atlassian.com/questions/18451025/answers/18451117?flashId=-982194107
UPDATE:
# ===============
# get user
# ===============
import requests
import json
# [BITBUCKET-BASE-URL], i.e.: https://bitbucket.org/
url = '[BITBUCKET-BASE-URL]/api/1.0/user/'
headers = {'Content-Type': 'application/json'}
# get user
# [USERNAME], i.e.: myuser
# [PASSWORD], i.e.: itspassword
r = requests.get(url, auth=('[USERNAME]', '[PASSWORD]'), headers=headers)
print(r.status_code)
print(r.text)
#print(r.content)
Here's a way to do basic HTTP auth with Python's requests module:
requests.post('https://api.bitbucket.org/1.0/user/repositories', auth=('user', 'pass'))
With the other way you're passing the user/pass through the request's payload, which is not desired since HTTP basic auth has its own place in the HTTP protocol.
If you want to "see" what's happening under the hood with your request I recommend using httpbin:
>>> url = "http://httpbin.org/post"
>>> r = requests.post(url, data="myscreename:mypassword")
>>> print r.text
{
"args": {},
"data": "myscreename:mypassword",
"files": {},
"form": {},
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate",
"Content-Length": "22",
"Host": "httpbin.org",
"User-Agent": "python-requests/2.5.1 CPython/2.7.6 Darwin/14.3.0"
},
"json": null,
"origin": "16.7.5.3",
"url": "http://httpbin.org/post"
}
>>> r = requests.post(url, auth=("myscreename", "mypassword"))
>>> print r.text
{
"args": {},
"data": "",
"files": {},
"form": {},
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate",
"Authorization": "Basic bXlzY3JlZW5hbWU6bXlwYXNzd29yZA==",
"Content-Length": "0",
"Host": "httpbin.org",
"User-Agent": "python-requests/2.5.1 CPython/2.7.6 Darwin/14.3.0"
},
"json": null,
"origin": "16.7.5.3",
"url": "http://httpbin.org/post"
}
And with curl:
curl -X POST --user myscreename:mypassword http://httpbin.org/post
{
"args": {},
"data": "",
"files": {},
"form": {},
"headers": {
"Accept": "*/*",
"Authorization": "Basic bXlzY3JlZW5hbWU6bXlwYXNzd29yZA==",
"Host": "httpbin.org",
"User-Agent": "curl/7.37.1"
},
"json": null,
"origin": "16.7.5.3",
"url": "http://httpbin.org/post"
}
Notice the resemblance between the last python example and the cURL one.
Now, getting right the API's format is another story, check out this link: https://answers.atlassian.com/questions/94245/can-i-create-a-bitbucket-repository-using-rest-api
The python way should be something like this:
requests.post('https://api.bitbucket.org/1.0/repositories', auth=('user', 'pass'), data = "name=repo_name")
With python3, you can use json={...} instead of data={...}, and it will set the header automatically to application/json:
import requests
url = 'https://api.bitbucket.org/1.0/user/repositories'
data = {
'data1': 'asd',
'data2': 'asd'
}
req = requests.post(url, auth=('user', 'password'), json = data)
data = req.json()
# data['index']