Cant understand Request response json - python
I am sending requests to discord but the json response is encoded. I dont know how to convert it to the json which is shown in
cookies = {
'__dcfduid': '9cdb771aa91811ecbbb166a2644e1ebd',
'__sdcfduid': '9cdb771aa91811ecbbb166a2644e1ebd4139917683b03f98166d7f12ed245c6b4eec20a521219fa14bc3dcad9fc83f17',
'__cf_bm': 'XATJg9tmbXKxU0XYhhk_NEc7jJI3G9cezBLutkAPL14-1647867777-0-AVUnjtyleUf5uH4NZKnrirKJ67tGnkxs3rdmrUmPM7jxGtiu7AV0DEfzOThZyTXG+6WwhYAvb4vecRMRQLixl7sX5hKh05wldjuEukidOaruFgJ0EFkBwt9f3/fv678s1g==',
'locale': 'en-US',
}
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:98.0) Gecko/20100101 Firefox/98.0',
'Accept': '*/*',
'Accept-Language': 'en-US,en;q=0.5',
'Accept-Encoding': 'gzip, deflate, br',
'Content-Type': 'application/json',
'Authorization': 'Token here',
'X-Super-Properties': 'eyJvcyI6IldpbmRvd3MiLCJicm93c2VyIjoiRmlyZWZveCIsImRldmljZSI6IiIsInN5c3RlbV9sb2NhbGUiOiJlbi1VUyIsImJyb3dzZXJfdXNlcl9hZ2VudCI6Ik1vemlsbGEvNS4wIChXaW5kb3dzIE5UIDEwLjA7IFdpbjY0OyB4NjQ7IHJ2Ojk4LjApIEdlY2tvLzIwMTAwMTAxIEZpcmVmb3gvOTguMCIsImJyb3dzZXJfdmVyc2lvbiI6Ijk4LjAiLCJvc192ZXJzaW9uIjoiMTAiLCJyZWZlcnJlciI6IiIsInJlZmVycmluZ19kb21haW4iOiIiLCJyZWZlcnJlcl9jdXJyZW50IjoiIiwicmVmZXJyaW5nX2RvbWFpbl9jdXJyZW50IjoiIiwicmVsZWFzZV9jaGFubmVsIjoic3RhYmxlIiwiY2xpZW50X2J1aWxkX251bWJlciI6MTE5ODgwLCJjbGllbnRfZXZlbnRfc291cmNlIjpudWxsfQ==',
'X-Discord-Locale': 'en-US',
'X-Debug-Options': 'bugReporterEnabled',
'Origin': 'https://discord.com',
'DNT': '1',
'Connection': 'keep-alive',
'Referer': 'https://discord.com/channels/#me/948159391838904380',
# Requests sorts cookies= alphabetically
# 'Cookie': '__dcfduid=9cdb771aa91811ecbbb166a2644e1ebd; __sdcfduid=9cdb771aa91811ecbbb166a2644e1ebd4139917683b03f98166d7f12ed245c6b4eec20a521219fa14bc3dcad9fc83f17; __cf_bm=XATJg9tmbXKxU0XYhhk_NEc7jJI3G9cezBLutkAPL14-1647867777-0-AVUnjtyleUf5uH4NZKnrirKJ67tGnkxs3rdmrUmPM7jxGtiu7AV0DEfzOThZyTXG+6WwhYAvb4vecRMRQLixl7sX5hKh05wldjuEukidOaruFgJ0EFkBwt9f3/fv678s1g==; locale=en-US',
'Sec-Fetch-Dest': 'empty',
'Sec-Fetch-Mode': 'cors',
'Sec-Fetch-Site': 'same-origin',
'Pragma': 'no-cache',
'Cache-Control': 'no-cache',
# Requests doesn't support trailers
# 'TE': 'trailers',
}
data = {
'content':'hello',
'nonce':955456750633353216,
'tts':False}
response = requests.post('https://discord.com/api/v9/channels/948159391838904380/messages', headers=headers, cookies=cookies, json=data)
print(response.status_code)
print(response.content)
The response is
b'\x03\xfe\x00\x00dS\x97\xd3\xaa\xd3\xfa$\x03.s7vc}\x1bR\xf2D\xeb\x8e\xfc#\xda\xfa!\xe2\x96\x05Q\x8b\xc20\r0\x0b(\xf8!\x080\x93\x0b\xa6\xa7\xfa\
xf6\xecd\x08\xbd;\xf1\xcf\x03\xdb\x8a\x1909\x17bJ\xde\xc4\x98\x8c\xf3Q\x13.\x01\x8f\xb7\x991\x03-\x01G\xbf\xed3\xe0?\x0f\xc3dq\t\xc8\xdd\xa8\xbe
\xe30\xd6ze\xb5O\x9a\xbcsQ\x90\xe7\xd0fH\xc7\xd9`}H\xc6{\x1d\x83#-\xf2\xe9\xc0{)#c\x06\xdc\x16i[\x7f8\x97c\xb9\x90\xd30,\x01k{h\xf6\xed\xd8J9\x8
6]\xa1I1\xe1\x12pyzM\xf3\xbd\x1d\xca\xdf\x013\xd0k\t\xd8\xfc\xc3\x013\xbc\x7f.\x01y\xfc\xe5\xaa\xa1\xd9j\xc5\x13\xbe\xf7\xd3\xc0\x06\xe5\xdc\x8a
p\xc5\x0c\xdb2\x1c\xf8e\xb6D}?\t\x9b\xdd1\x1bi^\xb6#\x1f\x8ee\x9c1\x03j\xd2zEf\xa5\xf5\x9bRY\xc7Ln\xad\x14\x11\xd1\x82(S\xc7g\\\xdb#\xd7o_[;\xd5
f\xc0\x18\xc6y\x92\xebe\x92\x86\x81\x14\xa2w\xd6yR\xc1\x07R\x16(\xfcI>bi\xb8~\x13X\x12\xcc \xa7ax\x19\x03'
Does any one know how to get the json as shown in developers tool. I have never seen this type of response and know nothing about it. I want to know this for educational purpose. As message being sent without knowing the response.
You should use response.json() to get it as a JSON.
response.content returns a binary representation.
Related
Error 403: The supplied authentication is not authorized to access this resource
I want to send a POST request to the following URL and get the result.
https://www.snapptrip.com/flights/THR-tehran/KIH-kish-island?roundTrip=false&adultCount=1&childCount=0&inLapCount=0&date=2022-07-01&cabinType=E
To do this I have copied all headers and cookies from browser and set them like as follows:
import json
import requests
header = {
'accept': 'application/json, text/plain, */*',
'accept-encoding': 'gzip, deflate, br',
'accept-language': 'en-US,en;q=0.9',
'cache-control': 'no-cache',
'content-length': '147',
'content-type': 'application/json;charset=UTF-8',
'cookie': 'route=1656317791.775.282.960981; _gcl_au=1.1.2046258357.1656317792; analytics_campaign={%22source%22:%22google%22%2C%22medium%22:%22organic%22}; analytics_token=42b8404c-38bc-f45a-5da5-4b0a9d8d64d3; _yngt_iframe=1; _yngt=7cf79f92-2a8e5-5857f-fc332-20702cb6182a1; _gid=GA1.2.1347682050.1656317794; _hjSessionUser_1680486=eyJpZCI6IjI4YzJkNDgxLWQ5ZmMtNTE0ZC04ODZmLWY0YmU4YWUzZTY0MyIsImNyZWF0ZWQiOjE2NTYzMTgwMjQwOTgsImV4aXN0aW5nIjp0cnVlfQ==; CustomerKey=c6daab9d-76a3-318b-9dc4-a998b9aa3d77; _clck=11btp2n|1|f2q|0; analytics_session_token=6fd59170-a4ac-3fe4-48c2-6672506db682; yektanet_session_last_activity=6/29/2022; _6193bae56a6d2e30f5357d82=true; _hjIncludedInSessionSample=0; _hjSession_1680486=eyJpZCI6IjhlNGU0MWQ4LThmM2QtNGQyNy05YjY2LTRmMDViYTQzYjM3ZCIsImNyZWF0ZWQiOjE2NTY0NzgwOTM2NTUsImluU2FtcGxlIjpmYWxzZX0=; _hjAbsoluteSessionInProgress=0; crisp-client%2Fsession%2Fdb7f3f50-c4be-4ec3-85f9-24c3cff11750=session_8eb3f713-17e8-4046-ad90-ecec16421bed; marketingData={"utm_medium":"","utm_campaign":"","utm_source":"","utm_content":"","utm_term":"","referrer":"https://www.snapptrip.com/flights/THR-tehran/MHD-mashhad?date=2022-08-22&returnDate=&roundTrip=false&adultCount=1&childCount=0&inLapCount=0&cabinType=E"}; flightMarketingData={"utm_medium":"","utm_campaign":"","utm_source":"","utm_content":"","utm_term":"","referrer":"https://www.snapptrip.com/flights/THR-tehran/MHD-mashhad?date=2022-08-22&returnDate=&roundTrip=false&adultCount=1&childCount=0&inLapCount=0&cabinType=E"}; destinationCity=%D9%85%D8%B4%D9%87%D8%AF; _gat_UA-107687430-1=1; _ga_DKJN1PETDV=GS1.1.1656478091.3.1.1656480799.49; _ga=GA1.2.312049892.1656317793; _clsk=1bf2oby|1656480801280|29|1|h.clarity.ms/collect',
'origin': 'https://www.snapptrip.com',
'pragma': 'no-cache',
'referer': ' https://www.snapptrip.com/flights/THR-tehran/MHD-mashhad?roundTrip=false&adultCount=1&childCount=0&inLapCount=0&date=2022-06-30&cabinType=E',
'sec-ch-ua': '" Not A;Brand";v="99", "Chromium";v="100", "Google Chrome";v="100"',
'sec-ch-ua-mobile': '?0',
'sec-ch-ua-platform': 'Linux',
'sec-fetch-dest': 'empty',
'sec-fetch-mode': 'cors',
'sec-fetch-site': 'same-origin',
'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.88 Safari/537.36',
'x-access-token': '26a53f65-4702-4f99-a5d3-ed03e03ca6e6',
'x-customer-key': 'c6daab9d-76a3-318b-9dc4-a998b9aa3d77',
'x-request-id': 'Bearer db312b1422dc4ec4f657de5909612a238af4592d5c25828cccb5bb319090fbf8.e7cf77e7-1d36-489d-9df3-ada35364681e',
'x-session-id': '3cfbed40-7932-46ee-9513-f342813be7c3',
'x-user-id': '71d3efa6-6d61-4d3d-89b2-16f1e1a201a1',
}
data = {"origin": "THR", "destination": "MHD", "roundTrip": false,
"passengers": {"adults": 1, "children": 0, "infants": 0}, "cabinType": "E", "departDate": "2022-06-30"}
url = 'https://www.snapptrip.com/flights/THR-tehran/KIH-kish-island?roundTrip=false&adultCount=1&childCount=0&inLapCount=0&date=2022-07-01&cabinType=E'
res = requests.post(url=url, json=json.dumps(data), headers=header)
The header and cookie images in the browser are as follows:
After all, it throws following error:
403: The supplied authentication is not authorized to access this resource
Need help isolating results from xhr request
When I run the code below, it's giving me a lot of information I don't want. I only want to capture the data circled starting with 4. Does anyone know how to isolate the data in the request to get rid of everything but what is circled? Also, if anyone knows how to bring the results to a csv or xlsx file, that would be even better.
Working code:
import requests
url = "https://www.stockrover.com/stock_infos/grid?_dc=1644769629231"
data = {
"ticker": 4,
"rank": "5",
}
payload = "state=%7B%22sortInfo%22%3A%7B%7D%2C%22columns%22%3A%5B77%2C32%2C498%2C500%2C31%2C27%2C499%2C30%2C578%2C28%2C29%2C544%2C181%2C185%2C186%5D%2C%22view%22%3A281%2C%22priorPrimaryColumn%22%3A170%2C%22filterData%22%3A%5B%5D%2C%22name%22%3A%22New%201%22%2C%22cType%22%3A%22Screener%22%2C%22cNode%22%3A%22s_39%22%2C%22cIsFolder%22%3Afalse%2C%22gridSelection%22%3A%22BTU%22%2C%22lastActive%22%3A1396898415%2C%22primaryColumn%22%3A76%2C%22folderDisabledParams%22%3A%7B%22filterData%22%3A%5B%5D%7D%2C%22mainGridDateRange%22%3A%22ytd%22%2C%22groupState%22%3Anull%2C%22moversGridDateRange%22%3A%221_day%22%2C%22peersGridDateRange%22%3A%221_day%22%2C%22lastGridSelections%22%3A%5B%22BTU%22%5D%2C%22lastQuantNode%22%3A%5B%5D%2C%22includeQuotesInTable%22%3Afalse%2C%22includeAllQuotesLastValue%22%3Afalse%2C%22markets%22%3A%7B%22panel%22%3A%22summary%22%7D%2C%22researchPanel%22%3A%22tablePanel%22%2C%22recentSearchTickers%22%3A%5B%22SPY%22%2C%22AMZN%22%2C%22AAPL%22%2C%22s_32%22%2C%22%5ENDX%22%2C%22AXP%22%2C%22XOM%22%2C%22AFL%22%2C%22%5EDJX%22%2C%22AIT%22%2C%22ADVC%22%5D%2C%22quotesBoxTickers%22%3A%5B%22AMZN%22%2C%22AAPL%22%2C%22SPY%22%5D%2C%22checkedQuotesBoxTickers%22%3A%5B%22AMZN%22%2C%22AAPL%22%2C%22SPY%22%5D%2C%22dashboard%22%3A%7B%22buttonRef%22%3A%22272%22%7D%2C%22tickerSelectedFeeds%22%3A%5B%22Benzinga%20News%22%2C%22Yahoo%20News%22%5D%2C%22marketSelectedFeeds%22%3A%5B%22Google%20News%22%2C%22Stock%20Market%20News%20-%20Investing.com%22%5D%2C%22bondsSelectedFeeds%22%3A%5B%22Bonds%20Strategy%20-%20Investing.com%22%5D%2C%22commoditiesSelectedFeeds%22%3A%5B%22Commodities%20%26%20Futures%20News%20-%20Investing.com%22%2C%22Commodities%20Fundamental%20Analysis%20-%20Investing.com%22%2C%22Commodities%20Strategy%20Analysis%20-%20Investing.com%22%5D%2C%22stocksSelectedFeeds%22%3A%5B%22CNNMoney%20News%22%2C%22Google%20News%22%2C%22Seeking%20Alpha%20Top%20Stories%22%5D%2C%22etfsSelectedFeeds%22%3A%5B%22Economy%20News%20-%20Investing.com%22%2C%22ETF%20Analysis%20-%20Investing.com%22%2C%22Investing%20Ideas%20-%20Investing.com%22%5D%2C%22topPanel%22%3A%22researchPanel%22%2C%22maxRecordsNode%22%3Afalse%2C%22version%22%3A7%2C%22lastGridSelectionsRaw%22%3A%5B%22BTU%22%5D%2C%22lastSelectionScreeners%22%3A%22s_39%22%2C%22quotesDisabled%22%3Atrue%2C%22lastSelectionPortfolios%22%3A%22p_2%22%2C%22comparisonPanels%22%3A%7B%22Portfolio%22%3A%22p_2%22%2C%22Index%22%3A%22%5EDJX%22%2C%22Watchlist%22%3A%22Watchlists%22%2C%22Screener%22%3A%22s_39%22%7D%2C%22lastSelectionWatchlists%22%3A%22w_26%22%2C%22indicesSelectedFeeds%22%3A%5B%22Google%20News%22%2C%22Yahoo%20News%22%5D%2C%22newsActive%22%3A%22tickerNews%22%2C%22recentSearchMetrics%22%3A%5B%22Price%22%2C%22EPS%22%2C%22Sales%22%5D%2C%22editPanel%22%3A%22positionsPanel%22%2C%22newsType%22%3A%22marketNews%22%2C%22tableColumns%22%3A%5B%22ticker%22%2C%22rank%22%2C%22score_rank%22%2C%22filter_score%22%2C%22company%22%2C%22cash%22%2C%22currentassets%22%2C%22netppe%22%2C%22intangibles%22%2C%22totalassets%22%2C%22currentliabilities%22%2C%22longtermdebt%22%2C%22totaldebt%22%2C%22totalliabilities%22%2C%22equity%22%2C%22tangiblebookvalue%22%2C%22cash_short_term_p%22%2C%22net_ppe_p%22%2C%22intangibles_p%22%5D%2C%22last_save%22%3A1644769379%2C%22panels%22%3A%7B%22collapsed%22%3A%7B%22chp%22%3Atrue%2C%22ip%22%3Atrue%2C%22mp%22%3Afalse%2C%22qp%22%3Afalse%2C%22conp%22%3Atrue%2C%22fsp%22%3Afalse%7D%2C%22viewportWidth%22%3A%221920%22%2C%22viewportHeight%22%3A%221069%22%2C%22chartPanelHeight%22%3A483%2C%22controlPanelWidth%22%3A296%2C%22insightPanelWidth%22%3A%22485%22%2C%22quoteBoxHeight%22%3A200%2C%22navigationPanelWidth%22%3A277%7D%7D&updateMarket=true&page=1&start=0&limit=250"
headers = {
'authority': 'www.stockrover.com',
'sec-ch-ua': '" Not A;Brand";v="99", "Chromium";v="98", "Google Chrome";v="98"',
'x-csrf-token': '7yR4pfI0kAArtjJak535+NJrpB0L212PAbXCg0kbyE4SyjFaQ73sMHJLiqAkPb5nGzfC8KvAa3kTADLAEQXyOQ==',
'sec-ch-ua-mobile': '?0',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.82 Safari/537.36',
'content-type': 'application/x-www-form-urlencoded; charset=UTF-8',
'accept': 'application/json',
'x-requested-with': 'XMLHttpRequest',
'sec-ch-ua-platform': '"Windows"',
'origin': 'https://www.stockrover.com',
'sec-fetch-site': 'same-origin',
'sec-fetch-mode': 'cors',
'sec-fetch-dest': 'empty',
'referer': 'https://www.stockrover.com/research/table/281/s_39/BTU',
'accept-language': 'en-US,en;q=0.9',
'cookie': 'remember_me_pref=0; user_name=test11964; plan=3; premiumBraintreeKey=MIIBCgKCAQEAzM4LJfrNnBOgRFB1dDJkmqTFCWT2Y%2BksOydD8xDH4R033WUzxbffMZb%2B3dqEyQvOVjLcwFIHByDc4Xwej7enas2E%2FVRyh7Cvyadn7M5zQeRyLcI9Ys5KCozMwxJPc0x76FlXPwiAo1Qlz3RcLb9wGHBag2R51FuTie%2BhVDCgzWajqDCREzRhi%2Fqlt3D%2FxXNo%2FiwJlpOUr%2Fx1QnkkILxgKlq1dD7KJ767O5ojYKXsO%2BV2Bfu7sSD3djsOxQJ1%2FRbaDm2E96EDkWhhOeOpPndQ6IuSl4NmnJg%2Fcq6f8csW8M3Ys%2BMZPFkdxPC4%2FfRM1XC9o76PjpVNBIO%2ByJEELKZedwIDAQAB; lr=1644769628; _Ruby2_session=Q1drcmlhazYvUFZLd0NydnRXUGpoUzArZDlxYWRCcW9sRUx5VDBydWVWRHdGWDZlMnlESURzbldwbFV1L0drbUlKaWt5MXRtaS9iR0ZYZEpPVHQ1N25qRnR1d3FrY0tzQW1qQm9CdTZ3MSs0d2c3MlpuMjRiQWhCOHI1cGNWekZ4cUdJd0ZFcGtpeng3MFlqZjFDUW9RYmpFMU9DeGdGMVZKR1EwMjVhSE9yVHl4VXFtQm9aYVBtNHF5d0pwMjJ1aVlNMUVRUzdnVFZWZ1AxQkY5Q0p6a2RKay9QL05tOWk4cHZiSERtaGRxeTlxTWZnV3Q0cjdwR3RndUtmeUp3QThhMnJaV2dGZjlPUUtjcGRidDhiajRxK2g0RUZTMWNZUDBaeGNCcUVxSDJ1QnZVRlRkWk9tUExJNWN3TDN5T1BQcmhVVGsycStVTzJRaUwvSkk2TnNVZldTOGU3Tm5wQ3RUMy9nazFqbzdrUWtvYzRwQWRpV3dnTVB3YzhodFV2U0FRR3VKdllMY01NZmdOdGtmOEJ4UT09LS1nTXBrYldhQ0pEeWJ3ak9qQjcrTGV3PT0%3D--1439f36a7f9362aee4b5b666747a2d63d72e81bd'
}
response = requests.request("POST", url, headers=headers, data=payload)
print(response.json())
The data structure in this case is rather unusual but this looks like it might work for you:
import requests
url = "https://www.stockrover.com/stock_infos/grid?_dc=1644769629231"
payload = "state=%7B%22sortInfo%22%3A%7B%7D%2C%22columns%22%3A%5B77%2C32%2C498%2C500%2C31%2C27%2C499%2C30%2C578%2C28%2C29%2C544%2C181%2C185%2C186%5D%2C%22view%22%3A281%2C%22priorPrimaryColumn%22%3A170%2C%22filterData%22%3A%5B%5D%2C%22name%22%3A%22New%201%22%2C%22cType%22%3A%22Screener%22%2C%22cNode%22%3A%22s_39%22%2C%22cIsFolder%22%3Afalse%2C%22gridSelection%22%3A%22BTU%22%2C%22lastActive%22%3A1396898415%2C%22primaryColumn%22%3A76%2C%22folderDisabledParams%22%3A%7B%22filterData%22%3A%5B%5D%7D%2C%22mainGridDateRange%22%3A%22ytd%22%2C%22groupState%22%3Anull%2C%22moversGridDateRange%22%3A%221_day%22%2C%22peersGridDateRange%22%3A%221_day%22%2C%22lastGridSelections%22%3A%5B%22BTU%22%5D%2C%22lastQuantNode%22%3A%5B%5D%2C%22includeQuotesInTable%22%3Afalse%2C%22includeAllQuotesLastValue%22%3Afalse%2C%22markets%22%3A%7B%22panel%22%3A%22summary%22%7D%2C%22researchPanel%22%3A%22tablePanel%22%2C%22recentSearchTickers%22%3A%5B%22SPY%22%2C%22AMZN%22%2C%22AAPL%22%2C%22s_32%22%2C%22%5ENDX%22%2C%22AXP%22%2C%22XOM%22%2C%22AFL%22%2C%22%5EDJX%22%2C%22AIT%22%2C%22ADVC%22%5D%2C%22quotesBoxTickers%22%3A%5B%22AMZN%22%2C%22AAPL%22%2C%22SPY%22%5D%2C%22checkedQuotesBoxTickers%22%3A%5B%22AMZN%22%2C%22AAPL%22%2C%22SPY%22%5D%2C%22dashboard%22%3A%7B%22buttonRef%22%3A%22272%22%7D%2C%22tickerSelectedFeeds%22%3A%5B%22Benzinga%20News%22%2C%22Yahoo%20News%22%5D%2C%22marketSelectedFeeds%22%3A%5B%22Google%20News%22%2C%22Stock%20Market%20News%20-%20Investing.com%22%5D%2C%22bondsSelectedFeeds%22%3A%5B%22Bonds%20Strategy%20-%20Investing.com%22%5D%2C%22commoditiesSelectedFeeds%22%3A%5B%22Commodities%20%26%20Futures%20News%20-%20Investing.com%22%2C%22Commodities%20Fundamental%20Analysis%20-%20Investing.com%22%2C%22Commodities%20Strategy%20Analysis%20-%20Investing.com%22%5D%2C%22stocksSelectedFeeds%22%3A%5B%22CNNMoney%20News%22%2C%22Google%20News%22%2C%22Seeking%20Alpha%20Top%20Stories%22%5D%2C%22etfsSelectedFeeds%22%3A%5B%22Economy%20News%20-%20Investing.com%22%2C%22ETF%20Analysis%20-%20Investing.com%22%2C%22Investing%20Ideas%20-%20Investing.com%22%5D%2C%22topPanel%22%3A%22researchPanel%22%2C%22maxRecordsNode%22%3Afalse%2C%22version%22%3A7%2C%22lastGridSelectionsRaw%22%3A%5B%22BTU%22%5D%2C%22lastSelectionScreeners%22%3A%22s_39%22%2C%22quotesDisabled%22%3Atrue%2C%22lastSelectionPortfolios%22%3A%22p_2%22%2C%22comparisonPanels%22%3A%7B%22Portfolio%22%3A%22p_2%22%2C%22Index%22%3A%22%5EDJX%22%2C%22Watchlist%22%3A%22Watchlists%22%2C%22Screener%22%3A%22s_39%22%7D%2C%22lastSelectionWatchlists%22%3A%22w_26%22%2C%22indicesSelectedFeeds%22%3A%5B%22Google%20News%22%2C%22Yahoo%20News%22%5D%2C%22newsActive%22%3A%22tickerNews%22%2C%22recentSearchMetrics%22%3A%5B%22Price%22%2C%22EPS%22%2C%22Sales%22%5D%2C%22editPanel%22%3A%22positionsPanel%22%2C%22newsType%22%3A%22marketNews%22%2C%22tableColumns%22%3A%5B%22ticker%22%2C%22rank%22%2C%22score_rank%22%2C%22filter_score%22%2C%22company%22%2C%22cash%22%2C%22currentassets%22%2C%22netppe%22%2C%22intangibles%22%2C%22totalassets%22%2C%22currentliabilities%22%2C%22longtermdebt%22%2C%22totaldebt%22%2C%22totalliabilities%22%2C%22equity%22%2C%22tangiblebookvalue%22%2C%22cash_short_term_p%22%2C%22net_ppe_p%22%2C%22intangibles_p%22%5D%2C%22last_save%22%3A1644769379%2C%22panels%22%3A%7B%22collapsed%22%3A%7B%22chp%22%3Atrue%2C%22ip%22%3Atrue%2C%22mp%22%3Afalse%2C%22qp%22%3Afalse%2C%22conp%22%3Atrue%2C%22fsp%22%3Afalse%7D%2C%22viewportWidth%22%3A%221920%22%2C%22viewportHeight%22%3A%221069%22%2C%22chartPanelHeight%22%3A483%2C%22controlPanelWidth%22%3A296%2C%22insightPanelWidth%22%3A%22485%22%2C%22quoteBoxHeight%22%3A200%2C%22navigationPanelWidth%22%3A277%7D%7D&updateMarket=true&page=1&start=0&limit=250"
headers = {
'authority': 'www.stockrover.com',
'sec-ch-ua': '" Not A;Brand";v="99", "Chromium";v="98", "Google Chrome";v="98"',
'x-csrf-token': '7yR4pfI0kAArtjJak535+NJrpB0L212PAbXCg0kbyE4SyjFaQ73sMHJLiqAkPb5nGzfC8KvAa3kTADLAEQXyOQ==',
'sec-ch-ua-mobile': '?0',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.82 Safari/537.36',
'content-type': 'application/x-www-form-urlencoded; charset=UTF-8',
'accept': 'application/json',
'x-requested-with': 'XMLHttpRequest',
'sec-ch-ua-platform': '"Windows"',
'origin': 'https://www.stockrover.com',
'sec-fetch-site': 'same-origin',
'sec-fetch-mode': 'cors',
'sec-fetch-dest': 'empty',
'referer': 'https://www.stockrover.com/research/table/281/s_39/BTU',
'accept-language': 'en-US,en;q=0.9',
'cookie': 'remember_me_pref=0; user_name=test11964; plan=3; premiumBraintreeKey=MIIBCgKCAQEAzM4LJfrNnBOgRFB1dDJkmqTFCWT2Y%2BksOydD8xDH4R033WUzxbffMZb%2B3dqEyQvOVjLcwFIHByDc4Xwej7enas2E%2FVRyh7Cvyadn7M5zQeRyLcI9Ys5KCozMwxJPc0x76FlXPwiAo1Qlz3RcLb9wGHBag2R51FuTie%2BhVDCgzWajqDCREzRhi%2Fqlt3D%2FxXNo%2FiwJlpOUr%2Fx1QnkkILxgKlq1dD7KJ767O5ojYKXsO%2BV2Bfu7sSD3djsOxQJ1%2FRbaDm2E96EDkWhhOeOpPndQ6IuSl4NmnJg%2Fcq6f8csW8M3Ys%2BMZPFkdxPC4%2FfRM1XC9o76PjpVNBIO%2ByJEELKZedwIDAQAB; lr=1644769628; _Ruby2_session=Q1drcmlhazYvUFZLd0NydnRXUGpoUzArZDlxYWRCcW9sRUx5VDBydWVWRHdGWDZlMnlESURzbldwbFV1L0drbUlKaWt5MXRtaS9iR0ZYZEpPVHQ1N25qRnR1d3FrY0tzQW1qQm9CdTZ3MSs0d2c3MlpuMjRiQWhCOHI1cGNWekZ4cUdJd0ZFcGtpeng3MFlqZjFDUW9RYmpFMU9DeGdGMVZKR1EwMjVhSE9yVHl4VXFtQm9aYVBtNHF5d0pwMjJ1aVlNMUVRUzdnVFZWZ1AxQkY5Q0p6a2RKay9QL05tOWk4cHZiSERtaGRxeTlxTWZnV3Q0cjdwR3RndUtmeUp3QThhMnJaV2dGZjlPUUtjcGRidDhiajRxK2g0RUZTMWNZUDBaeGNCcUVxSDJ1QnZVRlRkWk9tUExJNWN3TDN5T1BQcmhVVGsycStVTzJRaUwvSkk2TnNVZldTOGU3Tm5wQ3RUMy9nazFqbzdrUWtvYzRwQWRpV3dnTVB3YzhodFV2U0FRR3VKdllMY01NZmdOdGtmOEJ4UT09LS1nTXBrYldhQ0pEeWJ3ak9qQjcrTGV3PT0%3D--1439f36a7f9362aee4b5b666747a2d63d72e81bd'
}
response = requests.request("POST", url, headers=headers, data=payload)
stock_info = response.json()['stock_infos']
for info in stock_info:
key = info[0]
for i, sub in enumerate(info[1:], 1):
if sub == key:
print(info[i:])
break
Output:
['AA', 1452.0, 4285.0, 6679.0, 0.0, 14197.0, 2929.0, 1724.0, 1725.0, 8736.0, 3878.0, 3878.0, 10.23, 47.05, None]
['ACH', 1773.84, 7909.0, 18758.17, 788.41, 30523.38, 9252.6, 8782.1, 14242.28, 18580.33, 9038.55, 8250.17, 5.81, 61.46, 2.58]
...and the values for all observed tickers
Post request returning empty or encoded
I am trying to make a post request to an api but I get an empty string as response or an encoded/encrypted one I dont understand well why(both respones i get a 200 OK repsonse). this is from the page I am trying to replicate the request: [Request Header from website].
thats the payload [the payload]
and this is my code:
headers = {
'authority': 'node1.web3api.com',
'method': 'POST',
'path' : '/',
'scheme': 'https',
'accept': '*/*',
'accept-encoding': 'gzip, deflate, br',
'accept-language': 'de,de-DE;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6',
'content-length': '252',
'content-type': 'application/json',
'origin' :'https://etherscan.io',
'referer': 'https://etherscan.io/',
'sec-ch-ua': '" Not A;Brand";v="99", "Chromium";v="96", "Microsoft Edge";v="96"',
'sec-ch-ua-mobile': '?0',
'sec-ch-ua-platform': '"Windows"',
'sec-fetch-dest': 'empty',
'sec-fetch-mode': 'cors',
'sec-fetch-site': 'cross-site',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36 Edg/96.0.1054.62'
}
payload = {"jsonrpc": "2.0", "id": 1, "method": "eth_call", "params": [
{"from": "0x0000000000000000000000000000000000000000",
"data": "0xc87b56dd0000000000000000000000000000000000000000000000000000000000000004",
"to": "0xd754937672300ae6708a51229112de4017810934"}, "latest"]}
with requests.session() as session:
session.get('https://etherscan.io/address/0xd754937672300ae6708a51229112de4017810934#readContract')
r = requests.post('https://node1.web3api.com/', headers=headers, data=json.dumps(payload))
print(r.text)
print(r.content)
print(r.status_code)
Not able to scrape a website, some issue with post request
I am trying to scrape some data from https://tradingtick.com/options/callvsput.php. I have used the header & form-data given in the network tab without much success.
Here is my code using python requests library.
url = 'https://tradingtick.com/options/callvsput.php'
headers = {'Accept': 'application/json, text/plain, */*', 'Accept-Encoding': 'gzip, deflate, br', 'Accept-Language': 'en-US,en;q=0.9,hi;q=0.8', 'Connection': 'keep-alive', 'Content-Length': '134', 'Content-Type': 'application/json;charset=UTF-8', 'DNT': '1', 'Host': 'tradingtick.com', 'Origin': 'https://tradingtick.com', 'Referer': 'https://tradingtick.com/options/callvsput.php', 'sec-ch-ua': '" Not;A Brand";v="99", "Microsoft Edge";v="91", "Chromium";v="91"', 'sec-ch-ua-mobile': '?0', 'Sec-Fetch-Dest': 'empty', 'Sec-Fetch-Mode': 'cors', 'Sec-Fetch-Site': 'same-origin', 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 Edg/91.0.864.64'}
payload = {
"dataType": "\"hist\"",
"endStrike": "\"34800\"",
"hDate": "\"2021-07-02\"",
"range": "true",
"startStrike": "\"34800\"",
"type": "\"data\"",
"underlying": "\"BANKNIFTY\""}
r = requests.post(url, headers=headers, data=payload)
OUTPUT:
In [72]: r.text
Out[72]: '\n'
While the response tab on browser shows that it is supposed to return a json object containing data displayed in the chart on the browser. C
Could someone please help me by pointing out my mistake ? Thank you !
To get data from server, use json= parameter:
import json
import requests
api_url = "https://tradingtick.com/options/callvsput.php"
payload = {
"dataType": "hist",
"endStrike": "37000",
"hDate": "2021-07-02",
"range": True,
"startStrike": "33000",
"type": "data",
"underlying": "BANKNIFTY",
}
data = requests.post(api_url, json=payload).json()
# pretty print the data:
print(json.dumps(data, indent=4))
Prints:
[
"[{\"x\":\"02-07 09:10\",\"y\":\"6704850\",\"type\":\"CE\"},{\"x\":\"02-07 15:30\",\"y\":\"9949875\",\"type\":\"CE\"},{\"x\":\"02-07 09:10\",\"y\":\"4627525\",\"type\":\"PE\"},{\"x\":\"02-07 15:30\",\"y\":\"7585225\",\"type\":\"PE\"}]",
"[{\"x\":\"2021-07-02 09:10\",\"y\":\"6704850\",\"type\":\"CE\"},{\"x\":\"2021-07-02 09:15\",\"y\":\"6704850\",\"type\":\"CE\"},{\"x\":\"2021-07-02 09:20\",\"y\":\"7428425\",\"type\":\"CE\"},{\"x\":\"2021-07-02 09:25\",\"y\":\"8827225\",\"type\":\"CE\"},{\"x\":\"2021-07-02 09:30\",\"y\":\"9545125\",\"type\":\"CE\"},{\"x\":\"2021-07-02 09:35\",\"y\":\"9843525\",\"type\":\"CE\"},{\"x\":\"2021-07-02 09:40\",\"y\":\"10247750\",\"type\":\"CE\"},{\"x\":\"2021-07-02 09:45\",\"y\":\"10584800\",\"type\":\"CE\"},{\"x\":\"2021-07-02 09:50\",\"y\":\"10812225\",\"type\":\"CE\"},{\"x\":\"2021-07-02 09:55\",\"y\":\"11055975\",\"type\":\"CE\"},{\"x\":\"2021-07-02 10:00\",\"y\":\"11339575\",\"type\":\"CE\"},{\"x\":\"2021-07-02 10:05\",\"y\":\"11464125\",\"type\":\"CE\"},{\"x\":\"2021-07-02 10:10\",\"y\":\"11660625\",\"type\":\"CE\"},{\"x\":\"2021-07-02 10:15\",\"y\":\"11624000\",\"type\":\"CE\"},{\"x\":\"2021-07-02 10:20\",\"y\":\"11721075\",\"type\":\"CE\"},{\"x\":\"2021-07-02 10:25\",\"y\":\"11738100\",\"type\":\"CE\"},{\"x\":\"2021-07-02 10:30\",\"y\":\"11822750\",\"type\":\"CE\"},{\"x\":\"2021-07-02 10:35\",\"y\":\"11937250\",\"type\":\"CE\"},{\"x\":\"2021-07-02 10:40\",\"y\":\"11875275\",\"type\":\"CE\"},{\"x\":\"2021-07-02 10:45\",\"y\":\"11905750\",\"type\":\"CE\"},{\"x\":\"2021-07-02 10:50\",\"y\":\"12055750\",\"type\":\"CE\"},{\"x\":\"2021-07-02 10:55\",\"y\":\"12181600\",\"type\":\"CE\"},{\"x\":\"2021-07-02 11:00\",\"y\":\"12256325\",\"type\":\"CE\"},{\"x\":\"2021-07-02 11:05\",\"y\":\"12317450\",\"type\":\"CE\"},{\"x\":\"2021-07-02 11:10\",\"y\":\"12355075\",\"type\":\"CE\"},{\"x\":\"2021-07-02 11:15\",\"y\":\"12480125\",\"type\":\"CE\"},{\"x\":\"2021-07-02 11:20\",\"y\":\"12533675\",\"type\":\"CE\"},{\"x\":\"2021-07-02 11:25\",\"y\":\"12496975\",\"type\":\"CE\"},{\"x\":\"2021-07-02 11:30\",\"y\":\"12508850\",\"type\":\"CE\"},{\"x\":\"2021-07-02 11:35\",\"y\":\"12546700\",\"type\":\"CE\"},{\"x\":\"2021-07-02 11:40\",\"y\":\"12453625\",\"type\":\"CE\"},{\"x\":\"2021-07-02 11:45\",\"y\":\"12408600\",\"type\":\"CE\"},{\"x\":\"2021-07-02 11:50\",\"y\":\"12445200\",\"type\":\"CE\"},{\"x\":\"2021-07-02 11:55\",\"y\":\"12473000\",\"type\":\"CE\"},{\"x\":\"2021-07-02 12:00\",\"y\":\"12450550\",\"type\":\"CE\"},{\"x\":\"2021-07-02 12:05\",\"y\":\"12394100\",\"type\":\"CE\"},{\"x\":\"2021-07-02 12:10\",\"y\":\"12301100\",\"type\":\"CE\"},{\"x\":\"2021-07-02 12:15\",\"y\":\"12299025\",\"type\":\"CE\"},{\"x\":\"2021-07-02 12:20\",\"y\":\"12327625\",\"type\":\"CE\"},{\"x\":\"2021-07-02 12:25\",\"y\":\"12296525\",\"type\":\"CE\"},{\"x\":\"2021-07-02 12:30\",\"y\":\"12357550\",\"type\":\"CE\"},{\"x\":\"2021-07-02 12:35\",\"y\":\"12264850\",\"type\":\"CE\"},{\"x\":\"2021-07-02 12:40\",\"y\":\"12358600\",\"type\":\"CE\"},{\"x\":\"2021-07-02 12:45\",\"y\":\"12402250\",\"type\":\"CE\"},{\"x\":\"2021-07-02 12:50\",\"y\":\"12418000\",\"type\":\"CE\"},{\"x\":\"2021-07-02 12:55\",\"y\":\"12429875\",\"type\":\"CE\"},{\"x\":\"2021-07-02 13:00\",\"y\":\"12285275\",\"type\":\"CE\"},{\"x\":\"2021-07-02 13:05\",\"y\":\"11514050\",\"type\":\"CE\"},{\"x\":\"2021-07-02 13:10\",\"y\":\"11464225\",\"type\":\"CE\"},{\"x\":\"2021-07-02 13:15\",\"y\":\"11515700\",\"type\":\"CE\"},{\"x\":\"2021-07-02 13:20\",\"y\":\"11886425\",\"type\":\"CE\"},{\"x\":\"2021-07-02 13:25\",\"y\":\"12082300\",\"type\":\"CE\"},{\"x\":\"2021-07-02 13:30\",\"y\":\"12119650\",\"type\":\"CE\"},{\"x\":\"2021-07-02 13:35\",\"y\":\"12191600\",\"type\":\"CE\"},{\"x\":\"2021-07-02 13:40\",\"y\":\"12256550\",\"type\":\"CE\"},{\"x\":\"2021-07-02 13:45\",\"y\":\"12203225\",\"type\":\"CE\"},{\"x\":\"2021-07-02 13:55\",\"y\":\"12267875\",\"type\":\"CE\"},{\"x\":\"2021-07-02 14:00\",\"y\":\"12321375\",\"type\":\"CE\"},{\"x\":\"2021-07-02 14:05\",\"y\":\"12404150\",\"type\":\"CE\"},{\"x\":\"2021-07-02 14:10\",\"y\":\"12480950\",\"type\":\"CE\"},{\"x\":\"2021-07-02 14:15\",\"y\":\"12460350\",\"type\":\"CE\"},{\"x\":\"2021-07-02 14:20\",\"y\":\"12448800\",\"type\":\"CE\"},{\"x\":\"2021-07-02 14:25\",\"y\":\"12346675\",\"type\":\"CE\"},{\"x\":\"2021-07-02 14:30\",\"y\":\"12308050\",\"type\":\"CE\"},{\"x\":\"2021-07-02 14:35\",\"y\":\"12231425\",\"type\":\"CE\"},{\"x\":\"2021-07-02 14:40\",\"y\":\"12196475\",\"type\":\"CE\"},{\"x\":\"2021-07-02 14:45\",\"y\":\"12134000\",\"type\":\"CE\"},{\"x\":\"2021-07-02 14:50\",\"y\":\"12148500\",\"type\":\"CE\"},{\"x\":\"2021-07-02 14:55\",\"y\":\"12104975\",\"type\":\"CE\"},{\"x\":\"2021-07-02 15:00\",\"y\":\"11983575\",\"type\":\"CE\"},{\"x\":\"2021-07-02 15:05\",\"y\":\"11855825\",\"type\":\"CE\"},{\"x\":\"2021-07-02 15:10\",\"y\":\"11654750\",\"type\":\"CE\"},{\"x\":\"2021-07-02 15:15\",\"y\":\"11509750\",\"type\":\"CE\"},{\"x\":\"2021-07-02 15:20\",\"y\":\"10947325\",\"type\":\"CE\"},{\"x\":\"2021-07-02 15:25\",\"y\":\"10242200\",\"type\":\"CE\"},{\"x\":\"2021-07-02 15:30\",\"y\":\"9949875\",\"type\":\"CE\"},{\"x\":\"2021-07-02 09:10\",\"y\":\"4627525\",\"type\":\"PE\"},{\"x\":\"2021-07-02 09:15\",\"y\":\"4627525\",\"type\":\"PE\"},{\"x\":\"2021-07-02 09:20\",\"y\":\"5146275\",\"type\":\"PE\"},{\"x\":\"2021-07-02 09:25\",\"y\":\"6343150\",\"type\":\"PE\"},{\"x\":\"2021-07-02 09:30\",\"y\":\"6871175\",\"type\":\"PE\"},{\"x\":\"2021-07-02 09:35\",\"y\":\"7231975\",\"type\":\"PE\"},{\"x\":\"2021-07-02 09:40\",\"y\":\"7842050\",\"type\":\"PE\"},{\"x\":\"2021-07-02 09:45\",\"y\":\"8126225\",\"type\":\"PE\"},{\"x\":\"2021-07-02 09:50\",\"y\":\"8282750\",\"type\":\"PE\"},{\"x\":\"2021-07-02 09:55\",\"y\":\"8533500\",\"type\":\"PE\"},{\"x\":\"2021-07-02 10:00\",\"y\":\"8762800\",\"type\":\"PE\"},{\"x\":\"2021-07-02 10:05\",\"y\":\"8932375\",\"type\":\"PE\"},{\"x\":\"2021-07-02 10:10\",\"y\":\"9097025\",\"type\":\"PE\"},{\"x\":\"2021-07-02 10:15\",\"y\":\"9227350\",\"type\":\"PE\"},{\"x\":\"2021-07-02 10:20\",\"y\":\"9223325\",\"type\":\"PE\"},{\"x\":\"2021-07-02 10:25\",\"y\":\"9279675\",\"type\":\"PE\"},{\"x\":\"2021-07-02 10:30\",\"y\":\"9439825\",\"type\":\"PE\"},{\"x\":\"2021-07-02 10:35\",\"y\":\"9592700\",\"type\":\"PE\"},{\"x\":\"2021-07-02 10:40\",\"y\":\"9704525\",\"type\":\"PE\"},{\"x\":\"2021-07-02 10:45\",\"y\":\"9732475\",\"type\":\"PE\"},{\"x\":\"2021-07-02 10:50\",\"y\":\"9624900\",\"type\":\"PE\"},{\"x\":\"2021-07-02 10:55\",\"y\":\"9613400\",\"type\":\"PE\"},{\"x\":\"2021-07-02 11:00\",\"y\":\"9706525\",\"type\":\"PE\"},{\"x\":\"2021-07-02 11:05\",\"y\":\"9722000\",\"type\":\"PE\"},{\"x\":\"2021-07-02 11:10\",\"y\":\"9742975\",\"type\":\"PE\"},{\"x\":\"2021-07-02 11:15\",\"y\":\"9828825\",\"type\":\"PE\"},{\"x\":\"2021-07-02 11:20\",\"y\":\"9831100\",\"type\":\"PE\"},{\"x\":\"2021-07-02 11:25\",\"y\":\"9747400\",\"type\":\"PE\"},{\"x\":\"2021-07-02 11:30\",\"y\":\"9778100\",\"type\":\"PE\"},{\"x\":\"2021-07-02 11:35\",\"y\":\"9774625\",\"type\":\"PE\"},{\"x\":\"2021-07-02 11:40\",\"y\":\"9760975\",\"type\":\"PE\"},{\"x\":\"2021-07-02 11:45\",\"y\":\"9781525\",\"type\":\"PE\"},{\"x\":\"2021-07-02 11:50\",\"y\":\"9793475\",\"type\":\"PE\"},{\"x\":\"2021-07-02 11:55\",\"y\":\"9817825\",\"type\":\"PE\"},{\"x\":\"2021-07-02 12:00\",\"y\":\"9868475\",\"type\":\"PE\"},{\"x\":\"2021-07-02 12:05\",\"y\":\"9931450\",\"type\":\"PE\"},{\"x\":\"2021-07-02 12:10\",\"y\":\"10016575\",\"type\":\"PE\"},{\"x\":\"2021-07-02 12:15\",\"y\":\"10061475\",\"type\":\"PE\"},{\"x\":\"2021-07-02 12:20\",\"y\":\"10086900\",\"type\":\"PE\"},{\"x\":\"2021-07-02 12:25\",\"y\":\"10143900\",\"type\":\"PE\"},{\"x\":\"2021-07-02 12:30\",\"y\":\"10177150\",\"type\":\"PE\"},{\"x\":\"2021-07-02 12:35\",\"y\":\"10260575\",\"type\":\"PE\"},{\"x\":\"2021-07-02 12:40\",\"y\":\"10318175\",\"type\":\"PE\"},{\"x\":\"2021-07-02 12:45\",\"y\":\"10362150\",\"type\":\"PE\"},{\"x\":\"2021-07-02 12:50\",\"y\":\"10451925\",\"type\":\"PE\"},{\"x\":\"2021-07-02 12:55\",\"y\":\"10455500\",\"type\":\"PE\"},{\"x\":\"2021-07-02 13:00\",\"y\":\"10498575\",\"type\":\"PE\"},{\"x\":\"2021-07-02 13:05\",\"y\":\"10551675\",\"type\":\"PE\"},{\"x\":\"2021-07-02 13:10\",\"y\":\"10612375\",\"type\":\"PE\"},{\"x\":\"2021-07-02 13:15\",\"y\":\"10623050\",\"type\":\"PE\"},{\"x\":\"2021-07-02 13:20\",\"y\":\"10714400\",\"type\":\"PE\"},{\"x\":\"2021-07-02 13:25\",\"y\":\"10740825\",\"type\":\"PE\"},{\"x\":\"2021-07-02 13:30\",\"y\":\"10702850\",\"type\":\"PE\"},{\"x\":\"2021-07-02 13:35\",\"y\":\"10688800\",\"type\":\"PE\"},{\"x\":\"2021-07-02 13:40\",\"y\":\"10662700\",\"type\":\"PE\"},{\"x\":\"2021-07-02 13:45\",\"y\":\"10532575\",\"type\":\"PE\"},{\"x\":\"2021-07-02 13:55\",\"y\":\"10395625\",\"type\":\"PE\"},{\"x\":\"2021-07-02 14:00\",\"y\":\"10358725\",\"type\":\"PE\"},{\"x\":\"2021-07-02 14:05\",\"y\":\"10226650\",\"type\":\"PE\"},{\"x\":\"2021-07-02 14:10\",\"y\":\"10135400\",\"type\":\"PE\"},{\"x\":\"2021-07-02 14:15\",\"y\":\"10184600\",\"type\":\"PE\"},{\"x\":\"2021-07-02 14:20\",\"y\":\"10249900\",\"type\":\"PE\"},{\"x\":\"2021-07-02 14:25\",\"y\":\"10159825\",\"type\":\"PE\"},{\"x\":\"2021-07-02 14:30\",\"y\":\"10197025\",\"type\":\"PE\"},{\"x\":\"2021-07-02 14:35\",\"y\":\"10271500\",\"type\":\"PE\"},{\"x\":\"2021-07-02 14:40\",\"y\":\"10214775\",\"type\":\"PE\"},{\"x\":\"2021-07-02 14:45\",\"y\":\"10143575\",\"type\":\"PE\"},{\"x\":\"2021-07-02 14:50\",\"y\":\"10039275\",\"type\":\"PE\"},{\"x\":\"2021-07-02 14:55\",\"y\":\"9860975\",\"type\":\"PE\"},{\"x\":\"2021-07-02 15:00\",\"y\":\"9807250\",\"type\":\"PE\"},{\"x\":\"2021-07-02 15:05\",\"y\":\"9557125\",\"type\":\"PE\"},{\"x\":\"2021-07-02 15:10\",\"y\":\"9286975\",\"type\":\"PE\"},{\"x\":\"2021-07-02 15:15\",\"y\":\"9161000\",\"type\":\"PE\"},{\"x\":\"2021-07-02 15:20\",\"y\":\"8598650\",\"type\":\"PE\"},{\"x\":\"2021-07-02 15:25\",\"y\":\"7964650\",\"type\":\"PE\"},{\"x\":\"2021-07-02 15:30\",\"y\":\"7585225\",\"type\":\"PE\"}]",
"Jul 02, 15:30",
"08-Jul",
"[{\"x\":\"2021-07-02 15:30\",\"y\":\"34971.05\"},{\"x\":\"2021-07-02 09:10\",\"y\":\"34853.05\"},{\"x\":\"2021-07-02 15:25\",\"y\":\"34968.00\"},{\"x\":\"2021-07-02 15:20\",\"y\":\"34943.95\"},{\"x\":\"2021-07-02 15:15\",\"y\":\"34934.95\"},{\"x\":\"2021-07-02 15:10\",\"y\":\"34919.00\"},{\"x\":\"2021-07-02 15:05\",\"y\":\"34934.10\"},{\"x\":\"2021-07-02 15:00\",\"y\":\"34884.25\"},{\"x\":\"2021-07-02 14:55\",\"y\":\"34887.05\"},{\"x\":\"2021-07-02 14:50\",\"y\":\"34904.35\"},{\"x\":\"2021-07-02 14:45\",\"y\":\"34905.15\"},{\"x\":\"2021-07-02 14:40\",\"y\":\"34911.45\"},{\"x\":\"2021-07-02 14:35\",\"y\":\"34950.00\"},{\"x\":\"2021-07-02 14:30\",\"y\":\"34955.00\"},{\"x\":\"2021-07-02 14:25\",\"y\":\"34936.00\"},{\"x\":\"2021-07-02 14:20\",\"y\":\"34841.00\"},{\"x\":\"2021-07-02 14:15\",\"y\":\"34862.00\"},{\"x\":\"2021-07-02 14:10\",\"y\":\"34865.00\"},{\"x\":\"2021-07-02 14:05\",\"y\":\"34812.35\"},{\"x\":\"2021-07-02 14:00\",\"y\":\"34812.65\"},{\"x\":\"2021-07-02 13:55\",\"y\":\"34853.45\"},{\"x\":\"2021-07-02 13:45\",\"y\":\"34919.85\"},{\"x\":\"2021-07-02 13:40\",\"y\":\"34893.55\"},{\"x\":\"2021-07-02 13:35\",\"y\":\"34925.00\"},{\"x\":\"2021-07-02 13:30\",\"y\":\"34940.00\"},{\"x\":\"2021-07-02 13:25\",\"y\":\"34933.85\"},{\"x\":\"2021-07-02 13:20\",\"y\":\"34925.45\"},{\"x\":\"2021-07-02 13:15\",\"y\":\"34989.00\"},{\"x\":\"2021-07-02 13:10\",\"y\":\"35000.50\"},{\"x\":\"2021-07-02 13:05\",\"y\":\"35023.00\"},{\"x\":\"2021-07-02 13:00\",\"y\":\"35023.05\"},{\"x\":\"2021-07-02 12:55\",\"y\":\"34945.20\"},{\"x\":\"2021-07-02 12:50\",\"y\":\"34944.90\"},{\"x\":\"2021-07-02 12:45\",\"y\":\"34944.00\"},{\"x\":\"2021-07-02 12:40\",\"y\":\"34924.70\"},{\"x\":\"2021-07-02 12:35\",\"y\":\"34944.55\"},{\"x\":\"2021-07-02 12:30\",\"y\":\"34938.45\"},{\"x\":\"2021-07-02 12:25\",\"y\":\"34910.55\"},{\"x\":\"2021-07-02 12:20\",\"y\":\"34943.80\"},{\"x\":\"2021-07-02 12:15\",\"y\":\"34910.00\"},{\"x\":\"2021-07-02 12:10\",\"y\":\"34887.00\"},{\"x\":\"2021-07-02 12:05\",\"y\":\"34921.60\"},{\"x\":\"2021-07-02 12:00\",\"y\":\"34883.50\"},{\"x\":\"2021-07-02 11:55\",\"y\":\"34860.00\"},{\"x\":\"2021-07-02 11:50\",\"y\":\"34850.60\"},{\"x\":\"2021-07-02 11:45\",\"y\":\"34871.00\"},{\"x\":\"2021-07-02 11:40\",\"y\":\"34842.45\"},{\"x\":\"2021-07-02 11:35\",\"y\":\"34857.65\"},{\"x\":\"2021-07-02 11:30\",\"y\":\"34863.55\"},{\"x\":\"2021-07-02 11:25\",\"y\":\"34862.85\"},{\"x\":\"2021-07-02 11:20\",\"y\":\"34845.55\"},{\"x\":\"2021-07-02 11:15\",\"y\":\"34845.20\"},{\"x\":\"2021-07-02 11:10\",\"y\":\"34832.10\"},{\"x\":\"2021-07-02 11:05\",\"y\":\"34830.25\"},{\"x\":\"2021-07-02 11:00\",\"y\":\"34828.00\"},{\"x\":\"2021-07-02 10:55\",\"y\":\"34825.00\"},{\"x\":\"2021-07-02 10:50\",\"y\":\"34816.70\"},{\"x\":\"2021-07-02 10:45\",\"y\":\"34852.95\"},{\"x\":\"2021-07-02 10:40\",\"y\":\"34924.35\"},{\"x\":\"2021-07-02 10:35\",\"y\":\"34897.30\"},{\"x\":\"2021-07-02 10:30\",\"y\":\"34923.10\"},{\"x\":\"2021-07-02 10:25\",\"y\":\"34889.90\"},{\"x\":\"2021-07-02 10:20\",\"y\":\"34871.00\"},{\"x\":\"2021-07-02 10:15\",\"y\":\"34906.35\"},{\"x\":\"2021-07-02 10:10\",\"y\":\"34854.05\"},{\"x\":\"2021-07-02 10:05\",\"y\":\"34855.65\"},{\"x\":\"2021-07-02 10:00\",\"y\":\"34847.90\"},{\"x\":\"2021-07-02 09:55\",\"y\":\"34841.30\"},{\"x\":\"2021-07-02 09:50\",\"y\":\"34773.60\"},{\"x\":\"2021-07-02 09:45\",\"y\":\"34820.35\"},{\"x\":\"2021-07-02 09:40\",\"y\":\"34805.00\"},{\"x\":\"2021-07-02 09:35\",\"y\":\"34840.70\"},{\"x\":\"2021-07-02 09:30\",\"y\":\"34812.35\"},{\"x\":\"2021-07-02 09:25\",\"y\":\"34775.65\"},{\"x\":\"2021-07-02 09:20\",\"y\":\"34857.25\"},{\"x\":\"2021-07-02 09:15\",\"y\":\"34868.45\"}]"
]
Fixing error 408 on first POST request when scraping data
I'm trying to scrape website with BS4. This is the website that I have:
https://www.wsl.ch/de/ueber-die-wsl/news/alle-news.html
I want to scrape all urls of the news articles that are on this page. If I just put url in request lib, I will not get URLs of the website. But If i go to inspect page -> network, there is one post request that returns HTML that has all urls ( href-s ).
I have to use post request In order to get all URLs on the website, but the problem is that Im always getting error 408.
url = 'https://www.wsl.ch/de/ueber-die-wsl/news/alle-news.filter.html?tx_wslfilter_filter%5Baction%5D=ajax&tx_wslfilter_filter%5Bcontroller%5D=Filter&cHash=88a50dfb12c7c7e03ce68f244dbfda20'
headers = {
'Accept-Encoding': 'gzip, deflate, br',
'Accept-Language': 'en-GB,en-US;q=0.9,en;q=0.8',
'Connection': 'keep-alive',
'Content-Length': '757',
'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',
'Host': 'www.wsl.ch',
'Origin': 'https://www.wsl.ch',
'Referer': 'https://www.wsl.ch/de/ueber-die-wsl/news/alle-news.html',
'Sec-Fetch-Dest': 'empty',
'Sec-Fetch-Mode': 'cors',
'Sec-Fetch-Site': 'same-origin',
'Server-Timing': 'miss, db;dur=63, app;dur=55.2'}
response = requests.post(url, headers = headers)
print(response)
soup = BeautifulSoup(response.content, 'html.parser')
print(soup)
I have tried with and without headers but its the same.
What should I do?
You are not sending body in your post request.
I have corrected your code, now you will not get 408 (timeout)
from bs4 import BeautifulSoup
import requests
url = 'https://www.wsl.ch/de/ueber-die-wsl/news/alle-news.filter.html?tx_wslfilter_filter%5Baction%5D=ajax&tx_wslfilter_filter%5Bcontroller%5D=Filter&cHash=88a50dfb12c7c7e03ce68f244dbfda20'
headers = {
'Accept-Encoding': 'gzip, deflate, br',
'Accept-Language': 'en-GB,en-US;q=0.9,en;q=0.8',
'Connection': 'keep-alive',
'Content-Length': '757',
'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',
'Host': 'www.wsl.ch',
'Origin': 'https://www.wsl.ch',
'Referer': 'https://www.wsl.ch/de/ueber-die-wsl/news/alle-news.html',
'Sec-Fetch-Dest': 'empty',
'Sec-Fetch-Mode': 'cors',
'Sec-Fetch-Site': 'same-origin',
'Server-Timing': 'miss, db;dur=63, app;dur=55.2'}
data='tx_wslfilter_filter%5Btype%5D=news&tx_wslfilter_filter%5Bslf%5D=0&tx_wslfilter_filter%5Blang%5D=0&tx_wslfilter_filter%5Bpage%5D=1&tx_wslfilter_filter%5Bperpage%5D=10&tx_wslfilter_filter%5Bkeyword%5D=&tx_wslfilter_filter%5Ball%5D=1&tx_wslfilter_filter%5Bcategory%5D%5B10%5D=10&tx_wslfilter_filter%5Bcategory%5D%5B11%5D=11&tx_wslfilter_filter%5Bcategory%5D%5B12%5D=12&tx_wslfilter_filter%5Bcategory%5D%5B13%5D=13&tx_wslfilter_filter%5Bcategory%5D%5B1%5D=1&tx_wslfilter_filter%5Btag%5D%5B76%5D=76&tx_wslfilter_filter%5Btag%5D%5B1%5D=1&tx_wslfilter_filter%5Btag%5D%5B11%5D=11&tx_wslfilter_filter%5Btag%5D%5B7%5D=7&tx_wslfilter_filter%5Btag%5D%5B9%5D=9&tx_wslfilter_filter%5Btag%5D%5B8%5D=8&tx_wslfilter_filter%5Btag%5D%5B52%5D=52&tx_wslfilter_filter%5Byear%5D=0'
response = requests.post(url,data=data, headers = headers)
print(response)
soup = BeautifulSoup(response.content, 'html.parser')
print(soup)