I'm using Requests to handle my post requests and ran into a situation where if I run the same exact code in Python 3 I get an invalid response, but if I run it in Python 2 it works!
import requests
url = "https://creator.zoho.com/api/xml/write"
querystring = {"authtoken":"token"}
payload = "------WebKitFormBoundary7MA4YWxkTrZu0gW\r\nContent-Disposition: form-data; name=\"XMLString\"\r\n\r\n\n
<ZohoCreator>
<applicationlist>
... content ...
</applicationlist>
</ZohoCreator>\r\n------WebKitFormBoundary7MA4YWxkTrZu0gW--"
headers = {
'content-type': "multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW",
'Content-Type': "application/xml",
'cache-control': "no-cache",
'Postman-Token': "03197e8c-2aef-4ac4-829d-f7dca06a14be",
'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0'
}
session = requests.Session()
response = session.request("POST", url, data=payload, headers=headers, params=querystring)
print(response.text)
Python 3 Response:
{"code":2945,"message":"LESS_THAN_MIN_OCCURANCE"}
Python 2 Response:
<response><result>
... content ...
<status>Success</status></add></form></result></response>
I'm positive the request is fine as it works in Postman and this is the code it generated. am I missing something when it comes to Python 3?
I dont really know how it works with python2.
But error show that it happen due to invalid ticket. Refer the link below to generate api and Post Url to insert data into zoho creator.
https://www.zoho.com/creator/help/api/prerequisites/generate-auth-token.html
https://www.zoho.com/creator/help/script/post-url.html#Example
I changed the structure and Content-Type. It now posts successfully with Python 3.
payload = "XMLString=<ZohoCreator> ... </ZohoCreator>"
headers = {
'Content-Type': "application/x-www-form-urlencoded",
'cache-control': "no-cache",
}
Related
I encounter an error "403 Client Error: Forbidden for url:" when running the following code.
import requests
url = "https://www.marinetraffic.com/map/gettrackjson/shipid:5630138/stdate:2022-05-18%2015:07/endate:2022-05-19%2015:07/trackorigin:livetrack"
headers = {
"accept": "application/json",
"accept-encoding": "gzip, deflate",
"user-agent": "Mozilla/5.0",
"x-requested-with": "XMLHttpRequest"
}
response = requests.get(url, headers=headers)
response.raise_for_status()
print(response.json())
On the other hand, replacing with following url works:
url = "https://www.marinetraffic.com/vesselDetails/latestPosition/shipid:5630138"
Would anyone know why the first url doesn't work and if there's a way to make it work? The original page is at https://www.marinetraffic.com/en/ais/home/centerx:79.3/centery:6.9/zoom:10.
Problem:
I tried to login with request.post for having the extra information in the part "preise", but soup.select("div[id*='preise']") always returns me the result which indicate login is not succeed.
payload = {
"username": "email",
"password": "password",
"rememberMe": True
}
with requests.Session() as s:
p = s.post("https://www.xxx", data=payload)
print(s.cookies)
response = s.get("https://www.xxx/yy")
soup = BeautifulSoup(response.content.decode("utf8"))
print(soup.select("div[id*='preise']"))
The cookieJar looks like below:
<RequestsCookieJar[<Cookie msid=5d7cbf9f-43b2-43dc-9f6a-c766c2f4e805 for www.xxx>]>
I tried:
adding some headers, but I didn't understand what I should add exactly, so it is not working
doubt if the login url is not the real login url, (reference: Why is my login not working with Python Requests?), but I couldn't find any other url possible.
tried to print cookiejar to check if login is succeed, which indeed returns some value. Thus I am really confused why I could not have the tag correctly.
I know that I could retrieve the correct result using selenium, but I still wonder if this issue could be solved with only request.
Tried with solution of #Tobias P. G
Tried with personal cookie
A couple of steps made this possible for me:
Copy headers from the login post on the website
Copy the encrypted password from the post on the website
Both done by using the developer tool and copying the information from the post request found under network. If you are unsure how to do that please ask and I will provide a brief introduction.
import requests
s = requests.Session()
headers = {
'Host': 'www.xxx',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:83.0) Gecko/20100101 Firefox/83.0',
'Accept': '*/*',
'Accept-Language': 'da,en;q=0.7,en-US;q=0.3',
'Accept-Encoding': 'gzip, deflate, br',
'Content-Type': 'application/x-www-form-urlencoded;',
'X-Requested-With': 'XMLHttpRequest',
'Content-Length': '70',
'Origin': 'https://www.xxx',
'Connection': 'keep-alive',
'Referer': 'https://www.xxx/yy',
'Cookie': 'cookie', # replace 'cookie' with those from the manual post request
}
payload = {
"username": "email",
"password": "password", # Copy the encrypted version from the manual post request
"rememberMe": "true"
}
url = 'https://www.xxx/yy'
r = s.post(url, headers=headers, data=payload)
print(r.reason)
p = s.get('https://www.xxx/yy')
print(p.content)
Hello I'm new to dealing with servers and requests so bear with me,
I am trying to send get requests to an API hosted on a different server than the site itself (soldhistory.ca) so that I can get a JSON response of a list of property ids and then manually send individual requests to the API sever to get the details of each property. The later part works fine with my authorization token (and doesn't when I use a random string) so I know the issue isn't my authorization token but getting the initial json of property ids doesn't work.
I keep getting an 'Access Denied' - 403 message even when I send the headers, authorization token, and user-agent. When I send the wrong token, I get a different message that says 'Access Denied: Token not found or Expired'. When I send no token, it says 'Signature Required'. I think I am missing something trivial here. I have also noticed that when I login, the json response of that has an access token and another token that is different from the access token in it called 'token' which I think may have something to do with the problem I am experiencing but I have no idea what do with it. All in all, how do I get authorized to be able to send requests to the API server?
I have included a dummy account I have made with fake credentials in the code below if anyone wants to send requests. If you visit the site ,zoom out of the maps entirely and filter to show any price and only show sold properties, you will see there is data on roughly 450 000 past properties sold in Canada that I would like to get. My end goal is to get this data. If anyone can help me out I would greatly appreciate it.
It is worth noting, I have also tried using selenium to initially go to the homepage and then transfer the cookies to the requests session but that didn't work either. I have also tried using selenium-requests with no luck either but maybe I did not implement those right.
Also, if you look at the XMLHttpRequests of the site you will that there is an initial request called properties that is made and then the subsequent get requests are generated from that json response. I am trying to get the JSON response for properties. It is a SEARCH method.
Code:
import requests
s = requests.Session()
s.get('https://www.soldhistory.ca')
headers = {
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:83.0) Gecko/20100101 Firefox/83.0',
'Accept': '*/*',
'Accept-Language': 'en-CA,en-US;q=0.7,en;q=0.3',
'Access-Control-Request-Method': 'SEARCH',
'Access-Control-Request-Headers': 'access-token,access_token,content-type',
'Referer': 'https://www.soldhistory.ca/mapsearchapp/search/eyJzZWFyY2giOnsic2VhcmNoVHlwZSI6InJlc2lkZW50aWFsIiwibGlzdGluZ1R5cGUiOlsiU29sZCJdLCJvcGVuSG91c2UiOnsiZnJvbSI6MCwidG8iOjB9LCJiZWQiOjAsIm1hcmtldGRheXMiOjAsImJhdGgiOjAsInNlYXJjaEJ5Ijoic2VhcmNoYWxsIiwic2VhcmNoQnlUZXh0IjoiIiwicHJpY2VSYW5nZSI6eyJtaW4iOjAsIm1heCI6MH0sImZlZXRSYW5nZSI6eyJtaW4iOjAsIm1heCI6MH0sInNob3dPbmx5IjpbXSwicHJpY2VEcm9wIjpmYWxzZSwic29ydGJ5IjoibmV3ZXN0IiwiY29uZG9UeXBlIjoiIiwiY29uZG9PY2N1cGFuY3kiOiIiLCJjb25kb1N0YXR1cyI6IiIsImNvbmRvQnVpbGRlciI6IiIsImtleXdvcmRzIjpbXSwiUG9zdGFsQ29kZSI6ZmFsc2UsIlByb3ZpbmNlIjpmYWxzZSwiQ2l0eSI6ZmFsc2UsImNpdHlOYW1lIjoiTWVuZXNldCJ9LCJsb2NhdGlvbiI6eyJMb25naXR1ZGUiOi04Ni43Njc2MTkyMDA0ODM5MiwiTGF0aXR1ZGUiOjUzLjIzNjIzOTgyNTA1NjUxLCJab29tIjoyLCJtYXBWaWV3VHlwZSI6InJvYWRtYXAiLCJtYXBJbmZvVHlwZSI6W10sInNlbGVjdGVkUGF0aElEIjoiIiwiQm91bmRzIjp7InNvdXRoIjotNC41NDgwMzU0MjY0NTgxNzQsIndlc3QiOi0xODAsIm5vcnRoIjo3OC4zNTI5NDI4MzEyNjQ2MywiZWFzdCI6MTgwfX0sImNvbnRyb2xTcGVjaWFsIjp7fX0=',
'Origin': 'https://www.soldhistory.ca',
'Connection': 'keep-alive',
}
response = s.options('https://api.mapsearch.vps-private.net/properties', headers=headers)
headers2 = {
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:83.0) Gecko/20100101 Firefox/83.0',
'Accept': 'application/json',
'Accept-Language': 'en-CA,en-US;q=0.7,en;q=0.3',
'Content-Type': 'application/json',
'Origin': 'https://www.soldhistory.ca',
'Connection': 'keep-alive',
'Referer': 'https://www.soldhistory.ca/',
}
data2 = '{"mail":"robbydummy123#gmail.com","pass":"helloworld"}'
response2 = s.post('https://www.soldhistory.ca/mapsearchapp/visitor/login', headers=headers2, data=data2, verify=True)
parsed = response2.json()
print(json.dumps(parsed, indent=1, sort_keys=True))
accessToken = parsed['accessToken']
headers3 = {
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:83.0) Gecko/20100101 Firefox/83.0',
'Accept': 'application/json',
'Accept-Language': 'en-CA,en-US;q=0.7,en;q=0.3',
'access_token': accessToken,
'Access-Token': accessToken,
'Content-Type': 'application/json',
'Content-Length': '317',
'Origin': 'https://www.soldhistory.ca',
'Connection': 'keep-alive',
'Referer': 'https://www.soldhistory.ca/mapsearchapp/search/eyJzZWFyY2giOnsic2VhcmNoVHlwZSI6InJlc2lkZW50aWFsIiwibGlzdGluZ1R5cGUiOlsiU29sZCJdLCJvcGVuSG91c2UiOnsiZnJvbSI6MCwidG8iOjB9LCJiZWQiOjAsIm1hcmtldGRheXMiOjAsImJhdGgiOjAsInNlYXJjaEJ5Ijoic2VhcmNoYWxsIiwic2VhcmNoQnlUZXh0IjoiIiwicHJpY2VSYW5nZSI6eyJtaW4iOjAsIm1heCI6MH0sImZlZXRSYW5nZSI6eyJtaW4iOjAsIm1heCI6MH0sInNob3dPbmx5IjpbXSwicHJpY2VEcm9wIjpmYWxzZSwic29ydGJ5IjoibmV3ZXN0IiwiY29uZG9UeXBlIjoiIiwiY29uZG9PY2N1cGFuY3kiOiIiLCJjb25kb1N0YXR1cyI6IiIsImNvbmRvQnVpbGRlciI6IiIsImtleXdvcmRzIjpbXSwiUG9zdGFsQ29kZSI6ZmFsc2UsIlByb3ZpbmNlIjpmYWxzZSwiQ2l0eSI6ZmFsc2UsImNpdHlOYW1lIjoiTWVuZXNldCJ9LCJsb2NhdGlvbiI6eyJMb25naXR1ZGUiOi04Ni43Njc2MTkyMDA0ODM5MiwiTGF0aXR1ZGUiOjUzLjIzNjIzOTgyNTA1NjUxLCJab29tIjoyLCJtYXBWaWV3VHlwZSI6InJvYWRtYXAiLCJtYXBJbmZvVHlwZSI6W10sInNlbGVjdGVkUGF0aElEIjoiIiwiQm91bmRzIjp7InNvdXRoIjotNC41NDgwMzU0MjY0NTgxNzQsIndlc3QiOi0xODAsIm5vcnRoIjo3OC4zNTI5NDI4MzEyNjQ2MywiZWFzdCI6MTgwfX0sImNvbnRyb2xTcGVjaWFsIjp7fX0=',
}
data3 = '{"query":{"coordinates":{"$geoWithin":{"$box":[[160.3305414532229,35.087235763335656],[2.6547602032228923,71.87799155489013]]}},"searchType":"residential","listingType":{"$in":["Sale","Sold","Rent"]}},"fields":["Latitude","Longitude","listingType","searchType","Price"],"sort":{"Price":1},"limit":20,"soldData":false}'
response3 = s.post('https://api.mapsearch.vps-private.net/properties', headers=headers3, data=data3)
parsed = response3.json()
print(json.dumps(parsed, indent=1, sort_keys=True))
print(response3.status_code)
I'm making a GET request to fetch JSON, which works absolutely fine from any browser on any device, but not by python requests:
url = 'https://angel.co/autocomplete/new_tags'
params = {'query': 'sci', 'tag_type': 'MarketTag'}
resp = requests.get(url,params=params)
resp.raise_for_status()
gives HTTPError: 403 Client Error: Forbidden for url: https://angel.co/autocomplete/new_tags?query=ab&tag_type=MarketTag
So I tried:
Python requests. 403 Forbidden - I not only tried using User-Agent in headers but also all other headers that I found in Request Headers section in firefox for JSON response, but still 403!
Python requests - 403 forbidden - despite setting `User-Agent` headers - By making request through Session object, I still get 403!
What can be the possible cause? Is there something else I could try using?
EDIT: Request Headers (inspecting headers section of JSON in firefox) that I used in headers attribute:
{'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Encoding': 'gzip, deflate, br',
'Accept-Language': 'en-US,en;q=0.5',
'Connection': 'keep-alive',
'Host': 'angel.co',
'If-None-Match: 'W/"5857a9eac987138be074e7bdd4537df8"',
'TE': 'Trailers',
'Upgrade-Insecure-Requests': 1,
'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:71.0) Gecko/20100101 Firefox/71.0'}
If a get request returns 403 Forbidden even after adding user-agent to headers, you may need to add more headers like this:
headers = {
'user-agent':"Mozilla/5.0 ...",
'accept': '"text/html,application...',
'referer': 'https://...',
}
r = requests.get(url, headers=headers)
In the chrome, Request headers can be found in the Network > Headers > Request-Headers of the Developer Tools. (Press F12 to toggle it.)
I assume you website detects, when a request isn' sent from a browser (made with javascript).
I had a similar issue recently, and this answer had worked for me.
https://open.spotify.com/search/results/cheval is the link that triggers various intermediary requests, one being the attempted request below.
When running the following request in Postman (Chrome plugin), response cookies (13) are shown but do not seem to exist when running this request in Python (response.cookies is empty). I have also tried using a session, but with the same result.
update: Although these cookies were retrieved after using Selenium (to login/solve captcha and transfer the login cookies to the session to use for the following request, it's still unknown what variable/s are required for the target cookies to be returned with that request).
How can those response cookies be retrieved (if at all) with Python?
url = "https://api.spotify.com/v1/search"
querystring = {"type":"album,artist,playlist,track","q":"cheval*","decorate_restrictions":"true","best_match":"true","limit":"50","anonymous":"false","market":"from_token"}
headers = {
'access-control-request-method': "GET",
'origin': "https://open.spotify.com",
'x-devtools-emulate-network-conditions-client-id': "0959BC056CD6303CAEC3E2E5D7796B72",
'user-agent': "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36",
'access-control-request-headers': "authorization",
'accept': "*/*",
'accept-encoding': "gzip, deflate, br",
'accept-language': "en-US,en;q=0.9",
'cache-control': "no-cache",
'postman-token': "253b0e50-7ef1-759a-f7f4-b09ede65e462"
}
response = requests.request("OPTIONS", url, headers=headers, params=querystring)
print(response.text)