I am trying to do some scraping from websites using GET and POST methods, but now I am facing a new challenge.
I am trying to get data from a credit simulator, I found this portuguese site (https://www.bancomontepio.pt/particulares/credito/pessoal/credito-pessoal-online).
As far as I know, I need to use POST method, but I have to specify the data (the Amount value, the Term...). I usually do it by creating a dictionary structure but that is not working.
I'm kinda lost to be fair, maybe the problem is on the header...
Here is my code:
import requests
import warnings
warnings.filterwarnings("ignore")
term=24
amount=5000
url = 'https://simuladores.bancomontepio.pt/ITSCredit.External/Calculator/ITSCredit.Calculator.UI.External/gateway/Calculator/api/Calculator/Calculate?hash=-1359629931'
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.122 Safari/537.36',
'Accept-Language': 'pt-PT,pt;q=0.9,en-US;q=0.8,en;q=0.7'}
payload = {'Amount': amount,'Term': term,'ProductCode':"26B1129900X"}
response = requests.post(url, headers=headers, data=payload, verify=False).json()
If i take off the .json(), I get the error Response [410].
The goal is to get the TAN or TAEG that change when term ("Prazo") or amount ("Montante") values change.
Any ideias?
[EDIT]
headers = {'Accept': 'application/json, text/plain, */*' ,
'Accept-Encoding': 'gzip, deflate, br',
'Accept-Language': 'pt-PT,pt;q=0.9,en-US;q=0.8,en;q=0.7',
'Connection': 'keep-alive',
'Content-Length': '957',
'content-type': 'text/plain',
'Cookie': '_gcl_au=1.1.911606195.1646064658; OptanonAlertBoxClosed=2022-02-28T16:45:11.586Z; _ga=GA1.2.1147601977.1646064657; _ga_8WVEJF7X11=GS1.1.1646305654.3.1.1646309750.0; _ga_63QCVBV1V3=GS1.1.1646305679.1.1.1646309750.0; ASP.NET_SessionId=wlfbf2dx4oatlio0vl1ftinq; _gid=GA1.2.449121330.1646650093; calc-cookie=; OptanonConsent=isGpcEnabled=0&datestamp=Mon+Mar+07+2022+11%3A38%3A48+GMT%2B0000+(Hora+padr%C3%A3o+da+Europa+Ocidental)&version=6.30.0&isIABGlobal=false&consentId=6caccc97-6af1-4b55-9049-5694835d9f7a&interactionCount=2&landingPath=NotLandingPage&groups=C0001%3A1%2CC0002%3A1%2CC0003%3A1%2CC0004%3A1&hosts=H10%3A1%2CH20%3A1%2CH7%3A1%2CH8%3A1%2CH23%3A1%2CH11%3A1%2CH24%3A1%2CH13%3A1%2CH25%3A1&genVendors=&geolocation=ES%3B&AwaitingReconsent=false; _gali=slider-container; _gat_UA-186811106-6=1',
'Host': 'simuladores.bancomontepio.pt',
'Origin': 'https://simuladores.bancomontepio.pt',
'Referer': 'https://simuladores.bancomontepio.pt/',
'sec-ch-ua': '" Not A;Brand";v="99", "Chromium";v="90", "Google Chrome";v="90"',
'sec-ch-ua-mobile': '?0',
'Sec-Fetch-Dest': 'empty',
'Sec-Fetch-Mode': 'cors',
'Sec-Fetch-Site': 'same-origin',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36'}headers = {'Accept': 'application/json, text/plain, */*' ,
'Accept-Encoding': 'gzip, deflate, br',
'Accept-Language': 'pt-PT,pt;q=0.9,en-US;q=0.8,en;q=0.7',
'Connection': 'keep-alive',
'Content-Length': '957',
'content-type': 'text/plain',
'Cookie': '_gcl_au=1.1.911606195.1646064658; OptanonAlertBoxClosed=2022-02-28T16:45:11.586Z; _ga=GA1.2.1147601977.1646064657; _ga_8WVEJF7X11=GS1.1.1646305654.3.1.1646309750.0; _ga_63QCVBV1V3=GS1.1.1646305679.1.1.1646309750.0; ASP.NET_SessionId=wlfbf2dx4oatlio0vl1ftinq; _gid=GA1.2.449121330.1646650093; calc-cookie=; OptanonConsent=isGpcEnabled=0&datestamp=Mon+Mar+07+2022+11%3A38%3A48+GMT%2B0000+(Hora+padr%C3%A3o+da+Europa+Ocidental)&version=6.30.0&isIABGlobal=false&consentId=6caccc97-6af1-4b55-9049-5694835d9f7a&interactionCount=2&landingPath=NotLandingPage&groups=C0001%3A1%2CC0002%3A1%2CC0003%3A1%2CC0004%3A1&hosts=H10%3A1%2CH20%3A1%2CH7%3A1%2CH8%3A1%2CH23%3A1%2CH11%3A1%2CH24%3A1%2CH13%3A1%2CH25%3A1&genVendors=&geolocation=ES%3B&AwaitingReconsent=false; _gali=slider-container; _gat_UA-186811106-6=1',
'Host': 'simuladores.bancomontepio.pt',
'Origin': 'https://simuladores.bancomontepio.pt',
'Referer': 'https://simuladores.bancomontepio.pt/',
'sec-ch-ua': '" Not A;Brand";v="99", "Chromium";v="90", "Google Chrome";v="90"',
'sec-ch-ua-mobile': '?0',
'Sec-Fetch-Dest': 'empty',
'Sec-Fetch-Mode': 'cors',
'Sec-Fetch-Site': 'same-origin',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36'}
payload = {'CCRDCalculateInput':{},'MultifunctionsCalculateInput':{},'Device':{'Browser':'chrome','BrowserVersion':'90.0.4430.212','Device':'Desktop','Os':'windows','OsVersion':'windows-10','UserAgent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36'},'IsCustomer':'true','Amount':7500,'Term':60,'ConditionCode':'26B1129900X-01-I-129900-F','CreditDestinationCode':'129900','ProductCode':'26B1129900X','FinancedExpenses':'false','FrequencyTrancheCode':'null','GoalCode':'C006','GoalDescription':'PROJETOS PESSOAIS','FrequencyTypeCode':'M','FamilyCode':'CP','Proponents':[{'Position':1,
'Birthday':'1992-03-07T13:03:30.000Z','State':'true','EntityType':{'ID':1,'CompanyID':1,'Code':'P','Description':'Proponente','Value':'null','ValueString':'null','State':'true','Imported':'null'},'ExpenseCodes':['009']}],'Counterparts':0,'OptionalExpenses':[{'Code':'009','Factor':1}],'ResidualValue':0,'InterestOnly':0,'Deferment':0}
Now I'm getting a empty json()... Response 200 but I got this structure:
{'Status': 'Unknown',
'Error': {'VisibleToHuman': False, 'Code': '0', 'Message': ''},
'Result': None}
As far as I know, the status should be "OK" to get some info on the Result.
Cheers
Looks like you need to expand the payload to include more (all) of the parameters (including the cookies, specifically the ASP.NET_SessionId).
import requests
import warnings
warnings.filterwarnings("ignore")
term=24
amount=5000
url = 'https://simuladores.bancomontepio.pt/ITSCredit.External/Calculator/ITSCredit.Calculator.UI.External/gateway/Calculator/api/Calculator/Calculate?hash=-1359629931'
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.122 Safari/537.36',
'Accept-Language': 'pt-PT,pt;q=0.9,en-US;q=0.8,en;q=0.7',
'Cookie':'ASP.NET_SessionId=fhkyn1vn5knlw3uhdnh50nii;'}
payload = {
"CCRDCalculateInput":{},
"MultifunctionsCalculateInput":{},
"Device":{
"Browser":"chrome",
"BrowserVersion":"96.0.4664.110",
"Device":"Desktop",
"Os":"windows",
"OsVersion":"windows-10",
"UserAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"},
"IsCustomer":'true',
"Amount":amount,
"Term":term,
"ConditionCode":"26B1129900X-01-I-129900-F",
"CreditDestinationCode":"129900",
"ProductCode":"26B1129900X",
"FinancedExpenses":'false',
"FrequencyTrancheCode":'null',
"GoalCode":"C006",
"GoalDescription":"PROJETOS PESSOAIS",
"FrequencyTypeCode":"M",
"FamilyCode":"CP",
"Proponents":[{
"Position":'1',
"Birthday":"1992-03-10T13:03:24.000Z",
"State":'true',
"EntityType":{
"ID":'1',
"CompanyID":'1',
"Code":"P",
"Description":"Proponente",
"Value":'null',
"ValueString":'null',
"State":'true',
"Imported":'null'},
"ExpenseCodes":[
"009"]}],
"Counterparts":'0',
"OptionalExpenses":[{
"Code":"009",
"Factor":'1'}],
"ResidualValue":'0',
"InterestOnly":'0',
"Deferment":'0'}
jsonData = requests.post(url, headers=headers, json=payload, verify=False).json()
results = jsonData['Result']
mtic = results['MTIC']
installment = results['PeriodInstallment'][0]['Installment']
taeg = results['TAEG']
tan = results['PeriodInstallment'][0]['TAN']
print(f'Installment: {installment}\nTAEG: {taeg}\nTAN: {tan}\nMTIC: {mtic}')
Output:
Installment: 224.5
TAEG: 14.8
TAN: 7.0
MTIC: 5708.2
Related
this is what it looks like to me | I am working on this project to connect to zalando but I am encountering this error, I really don't understand how to fix it, would someone please tell me how I could fix it and successfully connect?
`def login():
headers = {
'authority': 'accounts.zalando.com',
'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
'accept-language': 'fr-FR,fr;q=0.9',
'referer': 'https://accounts.zalando.com/authenticate?sales_channel=733af55a-4133-4d7c-b5f3-d64d42c135fe&request=eyJjbGllbnRfaWQiOiJmYXNoaW9uLXN0b3JlLXdlYiIsInJlc3BvbnNlX3R5cGUiOiJjb2RlIiwic2NvcGVzIjpbIm9wZW5pZCJdLCJyZWRpcmVjdF91cmkiOiJodHRwczovL3d3dy56YWxhbmRvLmZyL3Nzby9jYWxsYmFjayIsInN0YXRlIjoiZXlKdmNtbG5hVzVoYkY5eVpYRjFaWE4wWDNWeWFTSTZJbWgwZEhCek9pOHZkM2QzTG5waGJHRnVaRzh1Wm5JdmJYbGhZMk52ZFc1MEx5SXNJblJ6SWpvaU1qQXlNeTB3TVMweU5GUXhNam96TlRveU1sb2lmUT09Iiwibm9uY2UiOiJjZmU4ZmJkOS00ZjYzLTQxMTgtOWYxNy0zYWU0OWNhNGRhN2MiLCJ1aV9sb2NhbGVzIjpbImZyLUZSIl0sInJlcXVlc3RfaWQiOiJ3ZURUYWIrSTctbk9WSG5COjc0ZDUzN2VkLTdkOGUtNGU0NS04MDRiLTNhMTdjYmMwZTdhMDpsamloMDh3WTVQWStsK3F5IiwiZiI6WyJQV0NfQSJdfQ==&ui_locales=fr-FR&passwordMeterFT=true',
'sec-ch-ua': '"Not_A Brand";v="99", "Google Chrome";v="109", "Chromium";v="109"',
'sec-ch-ua-mobile': '?0',
'sec-ch-ua-platform': '"Windows"',
'sec-fetch-dest': 'document',
'sec-fetch-mode': 'navigate',
'sec-fetch-site': 'same-origin',
'sec-fetch-user': '?1',
'upgrade-insecure-requests': '1',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36',
}
url = 'https://zalando.com/api'
r = requests.get(url, headers=headers)
#print(r.cookies.get_dict())
#print(r.headers)
headers = {
'authority': 'accounts.zalando.com',
'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
'accept-language': 'fr-FR,fr;q=0.9',
'cookie': r.headers['Set-Cookie'],
'referer': 'https://accounts.zalando.com/authenticate?sales_channel=733af55a-4133-4d7c-b5f3-d64d42c135fe&request=eyJjbGllbnRfaWQiOiJmYXNoaW9uLXN0b3JlLXdlYiIsInJlc3BvbnNlX3R5cGUiOiJjb2RlIiwic2NvcGVzIjpbIm9wZW5pZCJdLCJyZWRpcmVjdF91cmkiOiJodHRwczovL3d3dy56YWxhbmRvLmZyL3Nzby9jYWxsYmFjayIsInN0YXRlIjoiZXlKdmNtbG5hVzVoYkY5eVpYRjFaWE4wWDNWeWFTSTZJbWgwZEhCek9pOHZkM2QzTG5waGJHRnVaRzh1Wm5JdmJYbGhZMk52ZFc1MEx5SXNJblJ6SWpvaU1qQXlNeTB3TVMweU5GUXhNam96TlRveU1sb2lmUT09Iiwibm9uY2UiOiJjZmU4ZmJkOS00ZjYzLTQxMTgtOWYxNy0zYWU0OWNhNGRhN2MiLCJ1aV9sb2NhbGVzIjpbImZyLUZSIl0sInJlcXVlc3RfaWQiOiJ3ZURUYWIrSTctbk9WSG5COjc0ZDUzN2VkLTdkOGUtNGU0NS04MDRiLTNhMTdjYmMwZTdhMDpsamloMDh3WTVQWStsK3F5IiwiZiI6WyJQV0NfQSJdfQ==&ui_locales=fr-FR&passwordMeterFT=true',
'sec-ch-ua': '"Not_A Brand";v="99", "Google Chrome";v="109", "Chromium";v="109"',
'sec-ch-ua-mobile': '?0',
'sec-ch-ua-platform': '"Windows"',
'sec-fetch-dest': 'document',
'sec-fetch-mode': 'navigate',
'sec-fetch-site': 'same-origin',
'sec-fetch-user': '?1',
'upgrade-insecure-requests': '1',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36',
'x-csrf-token': r.cookies.get_dict()['Zalando-Client-Id'],
'x-flow-id': r.headers['X-Zalando-Child-Request-Id']
}
params = {
'email': 'talkaboutanthopls#gmail.com',
'secret': '#Sneakers974!!?',
}
response = requests.post('https://accounts.zalando.com/api/login', json=params, headers=headers)
print(response.content)
print(response.json())
print(response.cookies.get_dict())
print(response.headers)
login()`
i try to use the api-endpoint from this site:
https://horoguides.com/hk/watch_finder
I searched for the api-endpoint in the network-tab and try to rebuild this api-access with the following code:
import requests
url = "https://horoguides.com/hk/ajaj/watch/searchWatches"
payload = {
"addLimit": "LIMIT 0, 20",
"addOrder": "ORDER BY establish DESC",
}
headers = {
'Accept': "application/json, text/javascript, */*; q=0.01",
'Accept-Language': "de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7",
'Connection': "keep-alive",
'Content-Type': "multipart/form-data; boundary=---011000010111000001101001",
'Cookie': "PHPSESSID=siob5k70qu4gh8bkio07qtocv3; _gid=GA1.2.40295814.1663575664; __gads=ID=2fc582d62ff2a986-223e4e8c26ce00a9:T=1663575664:RT=1663575664:S=ALNI_MaTX_1U4CELXasmH0td3MvCRQ5S5Q; _gat_UA-90322481-1=1; _gat_gtag_UA_90322481_1=1; _ga_6Z9E9PKG02=GS1.1.1663594500.3.1.1663594710.0.0.0; _ga=GA1.1.699639573.1663575664",
'Origin': "https://horoguides.com",
'Referer': "https://horoguides.com/hk/watch_finder",
'Sec-Fetch-Dest': "empty",
'Sec-Fetch-Mode': "cors",
'Sec-Fetch-Site': "same-origin",
'User-Agent': "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36",
'X-Requested-With': "XMLHttpRequest",
'sec-ch-ua': "^\^Chromium^^;v=^\^104^^, ^\^"
}
resp = requests.request("POST", url, json=payload, headers=headers)
print(resp.status_code)
respJSON = resp.json()
print(respJSON)
But as response i only get:
200
{'status': 'invalid'}
Why is this reponse from the api-endpoint not working?
I also tried to run this in Insomnia and get the same result.
You need to fix the payload. The following code works:
import requests
url = "https://horoguides.com/hk/ajaj/watch/searchWatches"
payload = {
"addLimit": "LIMIT 0, 20",
"addOrder": "ORDER BY establish DESC",
'lang': 'hk',
'ajaxID': 'searchWatches'
}
headers = {
'Accept': "application/json, text/javascript, */*; q=0.01",
'Accept-Language': "de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7",
'Connection': "keep-alive",
'Origin': "https://horoguides.com",
'Referer': "https://horoguides.com/hk/watch_finder",
'Sec-Fetch-Dest': "empty",
'Sec-Fetch-Mode': "cors",
'User-Agent': "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36",
'X-Requested-With': "XMLHttpRequest"
}
resp = requests.request("POST", url, data=payload, headers=headers)
print(resp.status_code)
respJSON = resp.json()
print(respJSON)
Result in terminal:
200
{'act': 'watch/searchWatches', 'status': 'success', 'getData': {'a5124': {'id': '5124', 'name': '116610-LN-0001', 'url_name': '116610-ln-97200', 'establish': '2014', 'w_brand_id': '39', 'w_brand_abbr': '', 'w_brand_name': 'ROLEX', 'w_brand_urlname': 'rolex', 'w_brand_localname': '勞力士', 'hype_default_currency': 'NT$', 'w_series_name': 'SUBMARINER', 'w_series_urlname':[....]
For requests documentation, see https://requests.readthedocs.io/en/latest/
When I run the code below, it's giving me a lot of information I don't want. I only want to capture the data circled starting with 4. Does anyone know how to isolate the data in the request to get rid of everything but what is circled? Also, if anyone knows how to bring the results to a csv or xlsx file, that would be even better.
Working code:
import requests
url = "https://www.stockrover.com/stock_infos/grid?_dc=1644769629231"
data = {
"ticker": 4,
"rank": "5",
}
payload = "state=%7B%22sortInfo%22%3A%7B%7D%2C%22columns%22%3A%5B77%2C32%2C498%2C500%2C31%2C27%2C499%2C30%2C578%2C28%2C29%2C544%2C181%2C185%2C186%5D%2C%22view%22%3A281%2C%22priorPrimaryColumn%22%3A170%2C%22filterData%22%3A%5B%5D%2C%22name%22%3A%22New%201%22%2C%22cType%22%3A%22Screener%22%2C%22cNode%22%3A%22s_39%22%2C%22cIsFolder%22%3Afalse%2C%22gridSelection%22%3A%22BTU%22%2C%22lastActive%22%3A1396898415%2C%22primaryColumn%22%3A76%2C%22folderDisabledParams%22%3A%7B%22filterData%22%3A%5B%5D%7D%2C%22mainGridDateRange%22%3A%22ytd%22%2C%22groupState%22%3Anull%2C%22moversGridDateRange%22%3A%221_day%22%2C%22peersGridDateRange%22%3A%221_day%22%2C%22lastGridSelections%22%3A%5B%22BTU%22%5D%2C%22lastQuantNode%22%3A%5B%5D%2C%22includeQuotesInTable%22%3Afalse%2C%22includeAllQuotesLastValue%22%3Afalse%2C%22markets%22%3A%7B%22panel%22%3A%22summary%22%7D%2C%22researchPanel%22%3A%22tablePanel%22%2C%22recentSearchTickers%22%3A%5B%22SPY%22%2C%22AMZN%22%2C%22AAPL%22%2C%22s_32%22%2C%22%5ENDX%22%2C%22AXP%22%2C%22XOM%22%2C%22AFL%22%2C%22%5EDJX%22%2C%22AIT%22%2C%22ADVC%22%5D%2C%22quotesBoxTickers%22%3A%5B%22AMZN%22%2C%22AAPL%22%2C%22SPY%22%5D%2C%22checkedQuotesBoxTickers%22%3A%5B%22AMZN%22%2C%22AAPL%22%2C%22SPY%22%5D%2C%22dashboard%22%3A%7B%22buttonRef%22%3A%22272%22%7D%2C%22tickerSelectedFeeds%22%3A%5B%22Benzinga%20News%22%2C%22Yahoo%20News%22%5D%2C%22marketSelectedFeeds%22%3A%5B%22Google%20News%22%2C%22Stock%20Market%20News%20-%20Investing.com%22%5D%2C%22bondsSelectedFeeds%22%3A%5B%22Bonds%20Strategy%20-%20Investing.com%22%5D%2C%22commoditiesSelectedFeeds%22%3A%5B%22Commodities%20%26%20Futures%20News%20-%20Investing.com%22%2C%22Commodities%20Fundamental%20Analysis%20-%20Investing.com%22%2C%22Commodities%20Strategy%20Analysis%20-%20Investing.com%22%5D%2C%22stocksSelectedFeeds%22%3A%5B%22CNNMoney%20News%22%2C%22Google%20News%22%2C%22Seeking%20Alpha%20Top%20Stories%22%5D%2C%22etfsSelectedFeeds%22%3A%5B%22Economy%20News%20-%20Investing.com%22%2C%22ETF%20Analysis%20-%20Investing.com%22%2C%22Investing%20Ideas%20-%20Investing.com%22%5D%2C%22topPanel%22%3A%22researchPanel%22%2C%22maxRecordsNode%22%3Afalse%2C%22version%22%3A7%2C%22lastGridSelectionsRaw%22%3A%5B%22BTU%22%5D%2C%22lastSelectionScreeners%22%3A%22s_39%22%2C%22quotesDisabled%22%3Atrue%2C%22lastSelectionPortfolios%22%3A%22p_2%22%2C%22comparisonPanels%22%3A%7B%22Portfolio%22%3A%22p_2%22%2C%22Index%22%3A%22%5EDJX%22%2C%22Watchlist%22%3A%22Watchlists%22%2C%22Screener%22%3A%22s_39%22%7D%2C%22lastSelectionWatchlists%22%3A%22w_26%22%2C%22indicesSelectedFeeds%22%3A%5B%22Google%20News%22%2C%22Yahoo%20News%22%5D%2C%22newsActive%22%3A%22tickerNews%22%2C%22recentSearchMetrics%22%3A%5B%22Price%22%2C%22EPS%22%2C%22Sales%22%5D%2C%22editPanel%22%3A%22positionsPanel%22%2C%22newsType%22%3A%22marketNews%22%2C%22tableColumns%22%3A%5B%22ticker%22%2C%22rank%22%2C%22score_rank%22%2C%22filter_score%22%2C%22company%22%2C%22cash%22%2C%22currentassets%22%2C%22netppe%22%2C%22intangibles%22%2C%22totalassets%22%2C%22currentliabilities%22%2C%22longtermdebt%22%2C%22totaldebt%22%2C%22totalliabilities%22%2C%22equity%22%2C%22tangiblebookvalue%22%2C%22cash_short_term_p%22%2C%22net_ppe_p%22%2C%22intangibles_p%22%5D%2C%22last_save%22%3A1644769379%2C%22panels%22%3A%7B%22collapsed%22%3A%7B%22chp%22%3Atrue%2C%22ip%22%3Atrue%2C%22mp%22%3Afalse%2C%22qp%22%3Afalse%2C%22conp%22%3Atrue%2C%22fsp%22%3Afalse%7D%2C%22viewportWidth%22%3A%221920%22%2C%22viewportHeight%22%3A%221069%22%2C%22chartPanelHeight%22%3A483%2C%22controlPanelWidth%22%3A296%2C%22insightPanelWidth%22%3A%22485%22%2C%22quoteBoxHeight%22%3A200%2C%22navigationPanelWidth%22%3A277%7D%7D&updateMarket=true&page=1&start=0&limit=250"
headers = {
'authority': 'www.stockrover.com',
'sec-ch-ua': '" Not A;Brand";v="99", "Chromium";v="98", "Google Chrome";v="98"',
'x-csrf-token': '7yR4pfI0kAArtjJak535+NJrpB0L212PAbXCg0kbyE4SyjFaQ73sMHJLiqAkPb5nGzfC8KvAa3kTADLAEQXyOQ==',
'sec-ch-ua-mobile': '?0',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.82 Safari/537.36',
'content-type': 'application/x-www-form-urlencoded; charset=UTF-8',
'accept': 'application/json',
'x-requested-with': 'XMLHttpRequest',
'sec-ch-ua-platform': '"Windows"',
'origin': 'https://www.stockrover.com',
'sec-fetch-site': 'same-origin',
'sec-fetch-mode': 'cors',
'sec-fetch-dest': 'empty',
'referer': 'https://www.stockrover.com/research/table/281/s_39/BTU',
'accept-language': 'en-US,en;q=0.9',
'cookie': 'remember_me_pref=0; user_name=test11964; plan=3; premiumBraintreeKey=MIIBCgKCAQEAzM4LJfrNnBOgRFB1dDJkmqTFCWT2Y%2BksOydD8xDH4R033WUzxbffMZb%2B3dqEyQvOVjLcwFIHByDc4Xwej7enas2E%2FVRyh7Cvyadn7M5zQeRyLcI9Ys5KCozMwxJPc0x76FlXPwiAo1Qlz3RcLb9wGHBag2R51FuTie%2BhVDCgzWajqDCREzRhi%2Fqlt3D%2FxXNo%2FiwJlpOUr%2Fx1QnkkILxgKlq1dD7KJ767O5ojYKXsO%2BV2Bfu7sSD3djsOxQJ1%2FRbaDm2E96EDkWhhOeOpPndQ6IuSl4NmnJg%2Fcq6f8csW8M3Ys%2BMZPFkdxPC4%2FfRM1XC9o76PjpVNBIO%2ByJEELKZedwIDAQAB; lr=1644769628; _Ruby2_session=Q1drcmlhazYvUFZLd0NydnRXUGpoUzArZDlxYWRCcW9sRUx5VDBydWVWRHdGWDZlMnlESURzbldwbFV1L0drbUlKaWt5MXRtaS9iR0ZYZEpPVHQ1N25qRnR1d3FrY0tzQW1qQm9CdTZ3MSs0d2c3MlpuMjRiQWhCOHI1cGNWekZ4cUdJd0ZFcGtpeng3MFlqZjFDUW9RYmpFMU9DeGdGMVZKR1EwMjVhSE9yVHl4VXFtQm9aYVBtNHF5d0pwMjJ1aVlNMUVRUzdnVFZWZ1AxQkY5Q0p6a2RKay9QL05tOWk4cHZiSERtaGRxeTlxTWZnV3Q0cjdwR3RndUtmeUp3QThhMnJaV2dGZjlPUUtjcGRidDhiajRxK2g0RUZTMWNZUDBaeGNCcUVxSDJ1QnZVRlRkWk9tUExJNWN3TDN5T1BQcmhVVGsycStVTzJRaUwvSkk2TnNVZldTOGU3Tm5wQ3RUMy9nazFqbzdrUWtvYzRwQWRpV3dnTVB3YzhodFV2U0FRR3VKdllMY01NZmdOdGtmOEJ4UT09LS1nTXBrYldhQ0pEeWJ3ak9qQjcrTGV3PT0%3D--1439f36a7f9362aee4b5b666747a2d63d72e81bd'
}
response = requests.request("POST", url, headers=headers, data=payload)
print(response.json())
The data structure in this case is rather unusual but this looks like it might work for you:
import requests
url = "https://www.stockrover.com/stock_infos/grid?_dc=1644769629231"
payload = "state=%7B%22sortInfo%22%3A%7B%7D%2C%22columns%22%3A%5B77%2C32%2C498%2C500%2C31%2C27%2C499%2C30%2C578%2C28%2C29%2C544%2C181%2C185%2C186%5D%2C%22view%22%3A281%2C%22priorPrimaryColumn%22%3A170%2C%22filterData%22%3A%5B%5D%2C%22name%22%3A%22New%201%22%2C%22cType%22%3A%22Screener%22%2C%22cNode%22%3A%22s_39%22%2C%22cIsFolder%22%3Afalse%2C%22gridSelection%22%3A%22BTU%22%2C%22lastActive%22%3A1396898415%2C%22primaryColumn%22%3A76%2C%22folderDisabledParams%22%3A%7B%22filterData%22%3A%5B%5D%7D%2C%22mainGridDateRange%22%3A%22ytd%22%2C%22groupState%22%3Anull%2C%22moversGridDateRange%22%3A%221_day%22%2C%22peersGridDateRange%22%3A%221_day%22%2C%22lastGridSelections%22%3A%5B%22BTU%22%5D%2C%22lastQuantNode%22%3A%5B%5D%2C%22includeQuotesInTable%22%3Afalse%2C%22includeAllQuotesLastValue%22%3Afalse%2C%22markets%22%3A%7B%22panel%22%3A%22summary%22%7D%2C%22researchPanel%22%3A%22tablePanel%22%2C%22recentSearchTickers%22%3A%5B%22SPY%22%2C%22AMZN%22%2C%22AAPL%22%2C%22s_32%22%2C%22%5ENDX%22%2C%22AXP%22%2C%22XOM%22%2C%22AFL%22%2C%22%5EDJX%22%2C%22AIT%22%2C%22ADVC%22%5D%2C%22quotesBoxTickers%22%3A%5B%22AMZN%22%2C%22AAPL%22%2C%22SPY%22%5D%2C%22checkedQuotesBoxTickers%22%3A%5B%22AMZN%22%2C%22AAPL%22%2C%22SPY%22%5D%2C%22dashboard%22%3A%7B%22buttonRef%22%3A%22272%22%7D%2C%22tickerSelectedFeeds%22%3A%5B%22Benzinga%20News%22%2C%22Yahoo%20News%22%5D%2C%22marketSelectedFeeds%22%3A%5B%22Google%20News%22%2C%22Stock%20Market%20News%20-%20Investing.com%22%5D%2C%22bondsSelectedFeeds%22%3A%5B%22Bonds%20Strategy%20-%20Investing.com%22%5D%2C%22commoditiesSelectedFeeds%22%3A%5B%22Commodities%20%26%20Futures%20News%20-%20Investing.com%22%2C%22Commodities%20Fundamental%20Analysis%20-%20Investing.com%22%2C%22Commodities%20Strategy%20Analysis%20-%20Investing.com%22%5D%2C%22stocksSelectedFeeds%22%3A%5B%22CNNMoney%20News%22%2C%22Google%20News%22%2C%22Seeking%20Alpha%20Top%20Stories%22%5D%2C%22etfsSelectedFeeds%22%3A%5B%22Economy%20News%20-%20Investing.com%22%2C%22ETF%20Analysis%20-%20Investing.com%22%2C%22Investing%20Ideas%20-%20Investing.com%22%5D%2C%22topPanel%22%3A%22researchPanel%22%2C%22maxRecordsNode%22%3Afalse%2C%22version%22%3A7%2C%22lastGridSelectionsRaw%22%3A%5B%22BTU%22%5D%2C%22lastSelectionScreeners%22%3A%22s_39%22%2C%22quotesDisabled%22%3Atrue%2C%22lastSelectionPortfolios%22%3A%22p_2%22%2C%22comparisonPanels%22%3A%7B%22Portfolio%22%3A%22p_2%22%2C%22Index%22%3A%22%5EDJX%22%2C%22Watchlist%22%3A%22Watchlists%22%2C%22Screener%22%3A%22s_39%22%7D%2C%22lastSelectionWatchlists%22%3A%22w_26%22%2C%22indicesSelectedFeeds%22%3A%5B%22Google%20News%22%2C%22Yahoo%20News%22%5D%2C%22newsActive%22%3A%22tickerNews%22%2C%22recentSearchMetrics%22%3A%5B%22Price%22%2C%22EPS%22%2C%22Sales%22%5D%2C%22editPanel%22%3A%22positionsPanel%22%2C%22newsType%22%3A%22marketNews%22%2C%22tableColumns%22%3A%5B%22ticker%22%2C%22rank%22%2C%22score_rank%22%2C%22filter_score%22%2C%22company%22%2C%22cash%22%2C%22currentassets%22%2C%22netppe%22%2C%22intangibles%22%2C%22totalassets%22%2C%22currentliabilities%22%2C%22longtermdebt%22%2C%22totaldebt%22%2C%22totalliabilities%22%2C%22equity%22%2C%22tangiblebookvalue%22%2C%22cash_short_term_p%22%2C%22net_ppe_p%22%2C%22intangibles_p%22%5D%2C%22last_save%22%3A1644769379%2C%22panels%22%3A%7B%22collapsed%22%3A%7B%22chp%22%3Atrue%2C%22ip%22%3Atrue%2C%22mp%22%3Afalse%2C%22qp%22%3Afalse%2C%22conp%22%3Atrue%2C%22fsp%22%3Afalse%7D%2C%22viewportWidth%22%3A%221920%22%2C%22viewportHeight%22%3A%221069%22%2C%22chartPanelHeight%22%3A483%2C%22controlPanelWidth%22%3A296%2C%22insightPanelWidth%22%3A%22485%22%2C%22quoteBoxHeight%22%3A200%2C%22navigationPanelWidth%22%3A277%7D%7D&updateMarket=true&page=1&start=0&limit=250"
headers = {
'authority': 'www.stockrover.com',
'sec-ch-ua': '" Not A;Brand";v="99", "Chromium";v="98", "Google Chrome";v="98"',
'x-csrf-token': '7yR4pfI0kAArtjJak535+NJrpB0L212PAbXCg0kbyE4SyjFaQ73sMHJLiqAkPb5nGzfC8KvAa3kTADLAEQXyOQ==',
'sec-ch-ua-mobile': '?0',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.82 Safari/537.36',
'content-type': 'application/x-www-form-urlencoded; charset=UTF-8',
'accept': 'application/json',
'x-requested-with': 'XMLHttpRequest',
'sec-ch-ua-platform': '"Windows"',
'origin': 'https://www.stockrover.com',
'sec-fetch-site': 'same-origin',
'sec-fetch-mode': 'cors',
'sec-fetch-dest': 'empty',
'referer': 'https://www.stockrover.com/research/table/281/s_39/BTU',
'accept-language': 'en-US,en;q=0.9',
'cookie': 'remember_me_pref=0; user_name=test11964; plan=3; premiumBraintreeKey=MIIBCgKCAQEAzM4LJfrNnBOgRFB1dDJkmqTFCWT2Y%2BksOydD8xDH4R033WUzxbffMZb%2B3dqEyQvOVjLcwFIHByDc4Xwej7enas2E%2FVRyh7Cvyadn7M5zQeRyLcI9Ys5KCozMwxJPc0x76FlXPwiAo1Qlz3RcLb9wGHBag2R51FuTie%2BhVDCgzWajqDCREzRhi%2Fqlt3D%2FxXNo%2FiwJlpOUr%2Fx1QnkkILxgKlq1dD7KJ767O5ojYKXsO%2BV2Bfu7sSD3djsOxQJ1%2FRbaDm2E96EDkWhhOeOpPndQ6IuSl4NmnJg%2Fcq6f8csW8M3Ys%2BMZPFkdxPC4%2FfRM1XC9o76PjpVNBIO%2ByJEELKZedwIDAQAB; lr=1644769628; _Ruby2_session=Q1drcmlhazYvUFZLd0NydnRXUGpoUzArZDlxYWRCcW9sRUx5VDBydWVWRHdGWDZlMnlESURzbldwbFV1L0drbUlKaWt5MXRtaS9iR0ZYZEpPVHQ1N25qRnR1d3FrY0tzQW1qQm9CdTZ3MSs0d2c3MlpuMjRiQWhCOHI1cGNWekZ4cUdJd0ZFcGtpeng3MFlqZjFDUW9RYmpFMU9DeGdGMVZKR1EwMjVhSE9yVHl4VXFtQm9aYVBtNHF5d0pwMjJ1aVlNMUVRUzdnVFZWZ1AxQkY5Q0p6a2RKay9QL05tOWk4cHZiSERtaGRxeTlxTWZnV3Q0cjdwR3RndUtmeUp3QThhMnJaV2dGZjlPUUtjcGRidDhiajRxK2g0RUZTMWNZUDBaeGNCcUVxSDJ1QnZVRlRkWk9tUExJNWN3TDN5T1BQcmhVVGsycStVTzJRaUwvSkk2TnNVZldTOGU3Tm5wQ3RUMy9nazFqbzdrUWtvYzRwQWRpV3dnTVB3YzhodFV2U0FRR3VKdllMY01NZmdOdGtmOEJ4UT09LS1nTXBrYldhQ0pEeWJ3ak9qQjcrTGV3PT0%3D--1439f36a7f9362aee4b5b666747a2d63d72e81bd'
}
response = requests.request("POST", url, headers=headers, data=payload)
stock_info = response.json()['stock_infos']
for info in stock_info:
key = info[0]
for i, sub in enumerate(info[1:], 1):
if sub == key:
print(info[i:])
break
Output:
['AA', 1452.0, 4285.0, 6679.0, 0.0, 14197.0, 2929.0, 1724.0, 1725.0, 8736.0, 3878.0, 3878.0, 10.23, 47.05, None]
['ACH', 1773.84, 7909.0, 18758.17, 788.41, 30523.38, 9252.6, 8782.1, 14242.28, 18580.33, 9038.55, 8250.17, 5.81, 61.46, 2.58]
...and the values for all observed tickers
I would like to get the json data from for instance https://app.weathercloud.net/d0838117883#current using python requests module.
I tried:
import re
import requests
device='0838117883'
URL='https://app.weathercloud.net'
URL1=URL+'/d'+device
URL2=URL+'/device/stats'
headers={'Content-Type':'text/plain; charset=UTF-8',
'Referer':URL1,
'User-Agent':'Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/48.0.2564.82 Chrome/48.0.2564.82 Safari/537.36',
'Accept':'application/json, text/javascript,*/*'}
with requests.Session() as s:
#get html from URL1 in order to get the CSRF token
page = s.get(URL1)
CSRF=re.findall('WEATHERCLOUD_CSRF_TOKEN:"(.*)"},',page.text)[0]
#create parameters for URL2, in order to get the json file
params={'code':device,'WEATHERCLOUD_CSRF_TOKEN':CSRF}
page_stats=requests.get(URL2,params=params,headers=headers)
print(page_stats.url)
print(page_stats) #<Response [200]>
print(page_stats.text) #empty
print(page_stats.json()) #error
But the page_stats is empty.
How can I get the stats data from weathercloud?
Inspecting the page with DevTools, you'll find a useful endpoint:
https://app.weathercloud.net/device/stats
You can "replicate" the original web request made by your browser with requests library:
import requests
cookies = {
'PHPSESSID': '************************',
'WEATHERCLOUD_CSRF_TOKEN':'***********************',
'_ga': '**********',
'_gid': '**********',
'__gads': 'ID=**********',
'WeathercloudCookieAgreed': 'true',
'_gat': '1',
'WEATHERCLOUD_RECENT_ED3C8': '*****************',
}
headers = {
'Connection': 'keep-alive',
'sec-ch-ua': '^\\^Google',
'Accept': 'application/json, text/javascript, */*; q=0.01',
'X-Requested-With': 'XMLHttpRequest',
'sec-ch-ua-mobile': '?0',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36',
'sec-ch-ua-platform': '^\\^Windows^\\^',
'Sec-Fetch-Site': 'same-origin',
'Sec-Fetch-Mode': 'cors',
'Sec-Fetch-Dest': 'empty',
'Referer': 'https://app.weathercloud.net/d0838117883',
'Accept-Language': 'it-IT,it;q=0.9,en-US;q=0.8,en;q=0.7,es;q=0.6',
}
params = (
('code', '0838117883'),
('WEATHERCLOUD_CSRF_TOKEN', '****************'),
)
response = requests.get('https://app.weathercloud.net/device/stats', headers=headers, params=params, cookies=cookies)
# Serializing json
json_object = json.loads(response.text)
json Output:
{'last_update': 1632842172,
'bar_current': [1632842172, 1006.2],
'bar_day_max': [1632794772, 1013.4],
'bar_day_min': [1632845772, 1006.2],
'bar_month_max': [1632220572, 1028],
'bar_month_min': [1632715572, 997.3],
'bar_year_max': [1614418512, 1038.1],
'bar_year_min': [1615434432, 988.1],
'wdir_current': [1632842172, 180],
..............}
That's it.
Closed. This question is not reproducible or was caused by typos. It is not currently accepting answers.
This question was caused by a typo or a problem that can no longer be reproduced. While similar questions may be on-topic here, this one was resolved in a way less likely to help future readers.
Closed 2 years ago.
Improve this question
I have a HAR file which looks like this:
'Host': 'deal.ig.com', 'Accept-Encoding': 'gzip, deflate, br', 'x-device-user-agent': 'vendor=IG Group | applicationType=ig | platform=WTP | deviceType=Win32 | view=chart | version=0.3923.0+6a372d26', 'Referer': 'https://deal.ig.com/wtp', 'Connection': 'keep-alive', 'Sec-Fetch-Mode': 'cors', 'Accept-Language': 'en-GB', 'content-type': 'application/json', 'Sec-Fetch-Dest': 'empty', 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/80.0.3987.122 Safari/537.36', 'cst': '91a1d0c7ab1932e6997d6c15ff179715ac273ad1b5cd4e684d276cf9ba0e46df01111', 'Sec-Fetch-Site': 'same-origin', 'Cookie': 'optimizelyEndUserId=oeu1583408158372r0.31262903776855944; _ga=GA1.2.3213041.1583408160; _gid=GA1.2.77060031.1583408160; _gat=1; AMCVS_434717FE52A6476F0A490D4C%40AdobeOrg=1; s_ecid=MCMID%7C50579323134050101141167903551723140495; _gcl_au=1.1.105363083.1583408170; _gat_UA-53269626-3=1; _gali=loginbutton; adviggroupigcomuklive=1; s_cvp_mktch19=%5B%5B%27DIR%27%2C%271583408174937%27%5D%5D; s_nr2=1583408174936-New; gpv_pn=ex%20nav%3Alogin; s_cc=true; s_cvp_mktch28=%5B%5B%27DIR%27%2C%271583408174934%27%5D%2C%5B%27DIR%27%2C%271583408174947%27%5D%5D; ACCESS-TOKEN=eyJraWQiOiJDQVE4QU1JSUJDZ0tDQVFFQXFKdiIsInR5cCI6IkNMSUVOVF9BQ0NFU1MiLCJhbGciOiJSUzI1NiIsInppcCI6IkdaSVAifQ.H4sIAAAAAAAAAFXOvQrCMBQF4He5c4fcNMlNuomL0kFoXYo4pPkpgRrFtiCI724mwTN-HA7nDda5-5bXBZoLtAO1HVwrcHMKeT16aAAZcpRGc6hgW8Iz21sofNh13aB1zQrv-3MRgxY9c2RHNDUPyhjyyqGMEckQSus41dbjKJ0XQWnhOSkXzWhZEMpHhiVlLdkVGpS6FkwjyQJLOQd9ytMc-jTlUy6tPMb_Vng9fiBIfr50IW8W2gAAAA.qGsWbnN0LzmeCKkepN0vWY4z1sOYO6SgvNoPcQR3o6_6h8-5obpfQ_O1UQT1EOsPCxHJgibmX0_eHI8ll4gkY5oq75692T-1UU4nYJoXXW-iP1kNX-bV1Kba-sJybFWOywO_hp9wqKdko7jDGg7Ax8Q6JWb478enYZ8Y3VB2tzLMNMpK-J0RJyVt9_DRzkhvlguB7z1WdZluk5AD-z6mnX4SeUmzY2VjaOlRybp2oudUbGzwft5PiVlyVs8-O8oBr_EwKdG8O5Hnxzfg5GRThfTPzjuha02CgXi1M892AlBqnU7RCc39I8vNH5XPEcyow_RghDjb4k1VSR-mZtiMCw; CST=91a1d0c7ab1932e6997d6c15ff179715ac273ad1b5cd4e684d276cf9ba0e46df01111; ID=TD=DB400FA09853FBF32BD51785C67BDB2F2EEB2D75:CS=2:A=1; IG-ENVIRONMENT=PROD; REFRESH-TOKEN=eyJraWQiOiJDQVE4QU1JSUJDZ0tDQVFFQXFKdiIsInR5cCI6IkNMSUVOVF9SRUZSRVNIIiwiYWxnIjoiUlMyNTYiLCJ6aXAiOiJHWklQIn0.H4sIAAAAAAAAAFXMuwoCMRCF4XdJbZFJMrlsJzZaCbs2lrlMlsAaxF1BEN_dKWw85cfhf4u8NOrbqYhBgAQFGLwSO_Fc6dHjjZiP-3G8eq8l82G6sASIUGR2MUHQimwIrtgMWCu44ABjVk7HAglzMWS9KcrZXEOKkowtVQKPay1uYgD02kgPDhnWlfNT6_NCU5v7ufOrp_r_otf9Bwja4ecLU4FikcUAAAA.CfStuGzJg4E4cDWuroNg5rj1vMsfmYjbkJfexIVfylwhBO_0WRiDQlyZ0G3RULW_6VpsQsR3qXj3UslUh79XY96fuCfLjFO5mP9EPO32j2Q72_WaT7NTb5HnTHv1xbblYwSOQwPraMaYwYkrDys14p5EDlPGNm9p3WXrk1s7-GZrvwryqcTgUfltvha2A_CG8odvzUqwP5ZTvs6Ua9fGDc7XTCAbmXq8IlutVCSNeR-8N10dfGVwzAsDe-wzEsGG3lDPuGbZuc29GgQIFpjEsVZuL0bw_JhiV57eknUBOC3NdsDSWk8p765Ex3V3SxT72Y0OUcZelteFLmWnUZNAgA; callerReqId=a122ee92484fefdd; client_id=a20fa8511e2a302574dddc5533444d0b; defaultDealingPlatform=PUREDEAL; deviceOs=Other; deviceType=Desktop; exitPath=uk; exitUrl=https://www.ig.com/uk; preferredAccountId=KY7KR; sessionOpen=true; AMCV_434717FE52A6476F0A490D4C%40AdobeOrg=-330454231%7CMCIDTS%7C18327%7CMCMID%7C50579323134050101141167903551723140495%7CMCAAMLH-1584012984%7C6%7CMCAAMB-1584012984%7C6G1ynYcLPuiQxYZrsz_pkqfLG9yMXBpb2zX5dvJdYQJzPXImdj0y%7CMCOPTOUT-1583415384s%7CNONE%7CMCAID%7CNONE%7CMCSYNCSOP%7C411-18334%7CvVersion%7C3.1.2%7CMCCIDH%7C-1029498915; s_sq=%5B%5BB%5D%5D; cpaEnabled=true; AAMC_iggroup_0=REGION%7C6; aamoptsegs=aam%3D10776225%2Caam%3D10776328%2Caam%3D10873328%2Caam%3D10873753%2Caam%3D10554416%2Caam%3D10800273%2Caam%3D10800207%2Caam%3D12237189%2Caam%3D14258131%2Caam%3D16895981%2Caam%3D11348465%2Caam%3D17467451; linkedinexcl=seg%3D10554416; aam_uuid=54084551990277770212092742170574063792; X-SECURITY-TOKEN=283f5a46528c57fffa30c7bfb7dfa50fed42d92c1316723dd9844aa0e8dec2f801111; session_start_time=1583408194012; wtp:session=%7B%22authenticated%22%3A%7B%22authenticator%22%3A%22authenticator%3Aplatform%22%2C%22cst%22%3A%2291a1d0c7ab1932e6997d6c15ff179715ac273ad1b5cd4e684d276cf9ba0e46df01111%22%2C%22xst%22%3A%22283f5a46528c57fffa30c7bfb7dfa50fed42d92c1316723dd9844aa0e8dec2f801111%22%7D%7D; lang=en-GB', 'ig-account-id': 'KY7KR', 'x-security-token': '283f5a46528c57fffa30c7bfb7dfa50fed42d92c1316723dd9844aa0e8dec2f801111', 'ADRUM': 'isAjax:true', 'Accept': '*/*'}
{'Host': 'deal.ig.com', 'Accept-Encoding': 'gzip, deflate, br', 'x-device-user-agent': 'vendor=IG Group | applicationType=ig | platform=WTP | deviceType=Win32 | view=chart | version=0.3923.0+6a372d26', 'Referer': 'https://deal.ig.com/wtp', 'Connection': 'keep-alive', 'Sec-Fetch-Mode': 'cors', 'CST': '91a1d0c7ab1932e6997d6c15ff179715ac273ad1b5cd4e684d276cf9ba0e46df01111', 'Accept-Language': 'en-GB', 'X-Transaction-Id': 'htmlcharts_66dc4433-97a1-4fe8-a87a-22b8a71e1c35', 'Sec-Fetch-Dest': 'empty', 'X-SECURITY-TOKEN': '283f5a46528c57fffa30c7bfb7dfa50fed42d92c1316723dd9844aa0e8dec2f801111', 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/80.0.3987.122 Safari/537.36', 'Sec-Fetch-Site': 'same-origin', 'Cookie': 'optimizelyEndUserId=oeu1583408158372r0.31262903776855944; _ga=GA1.2.3213041.1583408160; _gid=GA1.2.77060031.1583408160; _gat=1; AMCVS_434717FE52A6476F0A490D4C%40AdobeOrg=1; s_ecid=MCMID%7C50579323134050101141167903551723140495; _gcl_au=1.1.105363083.1583408170; _gat_UA-53269626-3=1; _gali=loginbutton; adviggroupigcomuklive=1; s_cvp_mktch19=%5B%5B%27DIR%27%2C%271583408174937%27%5D%5D; s_nr2=1583408174936-New; gpv_pn=ex%20nav%3Alogin; s_cc=true; s_cvp_mktch28=%5B%5B%27DIR%27%2C%271583408174934%27%5D%2C%5B%27DIR%27%2C%271583408174947%27%5D%5D; ACCESS-TOKEN=eyJraWQiOiJDQVE4QU1JSUJDZ0tDQVFFQXFKdiIsInR5cCI6IkNMSUVOVF9BQ0NFU1MiLCJhbGciOiJSUzI1NiIsInppcCI6IkdaSVAifQ.H4sIAAAAAAAAAFXOvQrCMBQF4He5c4fcNMlNuomL0kFoXYo4pPkpgRrFtiCI724mwTN-HA7nDda5-5bXBZoLtAO1HVwrcHMKeT16aAAZcpRGc6hgW8Iz21sofNh13aB1zQrv-3MRgxY9c2RHNDUPyhjyyqGMEckQSus41dbjKJ0XQWnhOSkXzWhZEMpHhiVlLdkVGpS6FkwjyQJLOQd9ytMc-jTlUy6tPMb_Vng9fiBIfr50IW8W2gAAAA.qGsWbnN0LzmeCKkepN0vWY4z1sOYO6SgvNoPcQR3o6_6h8-5obpfQ_O1UQT1EOsPCxHJgibmX0_eHI8ll4gkY5oq75692T-1UU4nYJoXXW-iP1kNX-bV1Kba-sJybFWOywO_hp9wqKdko7jDGg7Ax8Q6JWb478enYZ8Y3VB2tzLMNMpK-J0RJyVt9_DRzkhvlguB7z1WdZluk5AD-z6mnX4SeUmzY2VjaOlRybp2oudUbGzwft5PiVlyVs8-O8oBr_EwKdG8O5Hnxzfg5GRThfTPzjuha02CgXi1M892AlBqnU7RCc39I8vNH5XPEcyow_RghDjb4k1VSR-mZtiMCw; CST=91a1d0c7ab1932e6997d6c15ff179715ac273ad1b5cd4e684d276cf9ba0e46df01111; ID=TD=DB400FA09853FBF32BD51785C67BDB2F2EEB2D75:CS=2:A=1; IG-ENVIRONMENT=PROD; REFRESH-TOKEN=eyJraWQiOiJDQVE4QU1JSUJDZ0tDQVFFQXFKdiIsInR5cCI6IkNMSUVOVF9SRUZSRVNIIiwiYWxnIjoiUlMyNTYiLCJ6aXAiOiJHWklQIn0.H4sIAAAAAAAAAFXMuwoCMRCF4XdJbZFJMrlsJzZaCbs2lrlMlsAaxF1BEN_dKWw85cfhf4u8NOrbqYhBgAQFGLwSO_Fc6dHjjZiP-3G8eq8l82G6sASIUGR2MUHQimwIrtgMWCu44ABjVk7HAglzMWS9KcrZXEOKkowtVQKPay1uYgD02kgPDhnWlfNT6_NCU5v7ufOrp_r_otf9Bwja4ecLU4FikcUAAAA.CfStuGzJg4E4cDWuroNg5rj1vMsfmYjbkJfexIVfylwhBO_0WRiDQlyZ0G3RULW_6VpsQsR3qXj3UslUh79XY96fuCfLjFO5mP9EPO32j2Q72_WaT7NTb5HnTHv1xbblYwSOQwPraMaYwYkrDys14p5EDlPGNm9p3WXrk1s7-GZrvwryqcTgUfltvha2A_CG8odvzUqwP5ZTvs6Ua9fGDc7XTCAbmXq8IlutVCSNeR-8N10dfGVwzAsDe-wzEsGG3lDPuGbZuc29GgQIFpjEsVZuL0bw_JhiV57eknUBOC3NdsDSWk8p765Ex3V3SxT72Y0OUcZelteFLmWnUZNAgA; callerReqId=a122ee92484fefdd; client_id=a20fa8511e2a302574dddc5533444d0b; defaultDealingPlatform=PUREDEAL; deviceOs=Other; deviceType=Desktop; exitPath=uk; exitUrl=https://www.ig.com/uk; preferredAccountId=KY7KR; sessionOpen=true; AMCV_434717FE52A6476F0A490D4C%40AdobeOrg=-330454231%7CMCIDTS%7C18327%7CMCMID%7C50579323134050101141167903551723140495%7CMCAAMLH-1584012984%7C6%7CMCAAMB-1584012984%7C6G1ynYcLPuiQxYZrsz_pkqfLG9yMXBpb2zX5dvJdYQJzPXImdj0y%7CMCOPTOUT-1583415384s%7CNONE%7CMCAID%7CNONE%7CMCSYNCSOP%7C411-18334%7CvVersion%7C3.1.2%7CMCCIDH%7C-1029498915; s_sq=%5B%5BB%5D%5D; cpaEnabled=true; AAMC_iggroup_0=REGION%7C6; aamoptsegs=aam%3D10776225%2Caam%3D10776328%2Caam%3D10873328%2Caam%3D10873753%2Caam%3D10554416%2Caam%3D10800273%2Caam%3D10800207%2Caam%3D12237189%2Caam%3D14258131%2Caam%3D16895981%2Caam%3D11348465%2Caam%3D17467451; linkedinexcl=seg%3D10554416; aam_uuid=54084551990277770212092742170574063792; X-SECURITY-TOKEN=283f5a46528c57fffa30c7bfb7dfa50fed42d92c1316723dd9844aa0e8dec2f801111; session_start_time=1583408194012; wtp:session=%7B%22authenticated%22%3A%7B%22authenticator%22%3A%22authenticator%3Aplatform%22%2C%22cst%22%3A%2291a1d0c7ab1932e6997d6c15ff179715ac273ad1b5cd4e684d276cf9ba0e46df01111%22%2C%22xst%22%3A%22283f5a46528c57fffa30c7bfb7dfa50fed42d92c1316723dd9844aa0e8dec2f801111%22%7D%7D; lang=en-GB', 'ADRUM': 'isAjax:true', 'Accept': '*/*'}
{'Referer': 'https://deal.ig.com/wtp', 'Host': 'cdn.walkme.com', 'Sec-Fetch-Dest': 'script', 'Accept-Encoding': 'gzip, deflate, br', 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/80.0.3987.122 Safari/537.36', 'Sec-Fetch-Site': 'cross-site', 'Connection': 'keep-alive', 'Sec-Fetch-Mode': 'no-cors', 'Accept-Language': 'en-GB', 'Accept': '*/*'}
{}
{}
{'Host': 'ec.walkme.com', 'Accept-Encoding': 'gzip, deflate, br', 'Referer': 'https://deal.ig.com/wtp', 'Connection': 'keep-alive', 'Sec-Fetch-Mode': 'cors', 'Content-Type': 'text/plain', 'Origin': 'https://deal.ig.com', 'Content-Length': '1068', 'Accept-Language': 'en-GB', 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/80.0.3987.122 Safari/537.36', 'Sec-Fetch-Site': 'cross-site', 'Sec-Fetch-Dest': 'empty', 'Accept': 'text/html, */*; q=0.01'}
{'Host': 'ec.walkme.com', 'Accept-Encoding': 'gzip, deflate, br', 'Referer': 'https://deal.ig.com/wtp', 'Connection': 'keep-alive', 'Sec-Fetch-Mode': 'no-cors', 'Content-Type': 'text/plain;charset=UTF-8', 'Origin': 'https://deal.ig.com', 'Content-Length': '399', 'Accept-Language': 'en-GB', 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/80.0.3987.122 Safari/537.36', 'Sec-Fetch-Site': 'cross-site', 'Sec-Fetch-Dest': 'empty', 'Accept': '*/*'}
{'Host': 'deal.ig.com', 'Accept-Encoding': 'gzip, deflate, br', 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/80.0.3987.122 Safari/537.36', 'Referer': 'https://deal.ig.com/wtp', 'Connection': 'keep-alive', 'Sec-Fetch-Mode': 'cors', 'Content-Type': 'application/json', 'Origin': 'https://deal.ig.com', 'Content-Length': '1417', 'Accept-Language': 'en-GB', 'Sec-Fetch-Dest': 'empty', 'Sec-Fetch-Site': 'same-origin', 'Cookie': 'optimizelyEndUserId=oeu1583408158372r0.31262903776855944; _ga=GA1.2.3213041.1583408160; _gid=GA1.2.77060031.1583408160; _gat=1; AMCVS_434717FE52A6476F0A490D4C%40AdobeOrg=1; s_ecid=MCMID%7C50579323134050101141167903551723140495; _gcl_au=1.1.105363083.1583408170; _gat_UA-53269626-3=1; adviggroupigcomuklive=1; s_cvp_mktch19=%5B%5B%27DIR%27%2C%271583408174937%27%5D%5D; s_nr2=1583408174936-New; gpv_pn=ex%20nav%3Alogin; s_cc=true; s_cvp_mktch28=%5B%5B%27DIR%27%2C%271583408174934%27%5D%2C%5B%27DIR%27%2C%271583408174947%27%5D%5D; ACCESS-TOKEN=eyJraWQiOiJDQVE4QU1JSUJDZ0tDQVFFQXFKdiIsInR5cCI6IkNMSUVOVF9BQ0NFU1MiLCJhbGciOiJSUzI1NiIsInppcCI6IkdaSVAifQ.H4sIAAAAAAAAAFXOvQrCMBQF4He5c4fcNMlNuomL0kFoXYo4pPkpgRrFtiCI724mwTN-HA7nDda5-5bXBZoLtAO1HVwrcHMKeT16aAAZcpRGc6hgW8Iz21sofNh13aB1zQrv-3MRgxY9c2RHNDUPyhjyyqGMEckQSus41dbjKJ0XQWnhOSkXzWhZEMpHhiVlLdkVGpS6FkwjyQJLOQd9ytMc-jTlUy6tPMb_Vng9fiBIfr50IW8W2gAAAA.qGsWbnN0LzmeCKkepN0vWY4z1sOYO6SgvNoPcQR3o6_6h8-5obpfQ_O1UQT1EOsPCxHJgibmX0_eHI8ll4gkY5oq75692T-1UU4nYJoXXW-iP1kNX-bV1Kba-sJybFWOywO_hp9wqKdko7jDGg7Ax8Q6JWb478enYZ8Y3VB2tzLMNMpK-J0RJyVt9_DRzkhvlguB7z1WdZluk5AD-z6mnX4SeUmzY2VjaOlRybp2oudUbGzwft5PiVlyVs8-O8oBr_EwKdG8O5Hnxzfg5GRThfTPzjuha02CgXi1M892AlBqnU7RCc39I8vNH5XPEcyow_RghDjb4k1VSR-mZtiMCw; CST=91a1d0c7ab1932e6997d6c15ff179715ac273ad1b5cd4e684d276cf9ba0e46df01111; ID=TD=DB400FA09853FBF32BD51785C67BDB2F2EEB2D75:CS=2:A=1; IG-ENVIRONMENT=PROD; REFRESH-TOKEN=eyJraWQiOiJDQVE4QU1JSUJDZ0tDQVFFQXFKdiIsInR5cCI6IkNMSUVOVF9SRUZSRVNIIiwiYWxnIjoiUlMyNTYiLCJ6aXAiOiJHWklQIn0.H4sIAAAAAAAAAFXMuwoCMRCF4XdJbZFJMrlsJzZaCbs2lrlMlsAaxF1BEN_dKWw85cfhf4u8NOrbqYhBgAQFGLwSO_Fc6dHjjZiP-3G8eq8l82G6sASIUGR2MUHQimwIrtgMWCu44ABjVk7HAglzMWS9KcrZXEOKkowtVQKPay1uYgD02kgPDhnWlfNT6_NCU5v7ufOrp_r_otf9Bwja4ecLU4FikcUAAAA.CfStuGzJg4E4cDWuroNg5rj1vMsfmYjbkJfexIVfylwhBO_0WRiDQlyZ0G3RULW_6VpsQsR3qXj3UslUh79XY96fuCfLjFO5mP9EPO32j2Q72_WaT7NTb5HnTHv1xbblYwSOQwPraMaYwYkrDys14p5EDlPGNm9p3WXrk1s7-GZrvwryqcTgUfltvha2A_CG8odvzUqwP5ZTvs6Ua9fGDc7XTCAbmXq8IlutVCSNeR-8N10dfGVwzAsDe-wzEsGG3lDPuGbZuc29GgQIFpjEsVZuL0bw_JhiV57eknUBOC3NdsDSWk8p765Ex3V3SxT72Y0OUcZelteFLmWnUZNAgA; callerReqId=a122ee92484fefdd; client_id=a20fa8511e2a302574dddc5533444d0b; defaultDealingPlatform=PUREDEAL; deviceOs=Other; deviceType=Desktop; exitPath=uk; exitUrl=https://www.ig.com/uk; preferredAccountId=KY7KR; sessionOpen=true; AMCV_434717FE52A6476F0A490D4C%40AdobeOrg=-330454231%7CMCIDTS%7C18327%7CMCMID%7C50579323134050101141167903551723140495%7CMCAAMLH-1584012984%7C6%7CMCAAMB-1584012984%7C6G1ynYcLPuiQxYZrsz_pkqfLG9yMXBpb2zX5dvJdYQJzPXImdj0y%7CMCOPTOUT-1583415384s%7CNONE%7CMCAID%7CNONE%7CMCSYNCSOP%7C411-18334%7CvVersion%7C3.1.2%7CMCCIDH%7C-1029498915; s_sq=%5B%5BB%5D%5D; cpaEnabled=true; linkedinexcl=seg%3D10554416; aam_uuid=54084551990277770212092742170574063792; AAMC_iggroup_0=REGION%7C6; aamoptsegs=aam%3D10776225%2Caam%3D10776328%2Caam%3D10873328%2Caam%3D10873753%2Caam%3D10554416%2Caam%3D10800273%2Caam%3D10800207%2Caam%3D12237189%2Caam%3D14258131%2Caam%3D16895981%2Caam%3D11348465%2Caam%3D17467451; X-SECURITY-TOKEN=283f5a46528c57fffa30c7bfb7dfa50fed42d92c1316723dd9844aa0e8dec2f801111; session_start_time=1583408194012; wtp:session=%7B%22authenticated%22%3A%7B%22authenticator%22%3A%22authenticator%3Aplatform%22%2C%22cst%22%3A%2291a1d0c7ab1932e6997d6c15ff179715ac273ad1b5cd4e684d276cf9ba0e46df01111%22%2C%22xst%22%3A%22283f5a46528c57fffa30c7bfb7dfa50fed42d92c1316723dd9844aa0e8dec2f801111%22%7D%7D; lang=en-GB', 'ADRUM': 'isAjax:true', 'Accept': '*/*'}
{'Host': 'ec.walkme.com', 'Accept-Encoding': 'gzip, deflate, br', 'Referer': 'https://deal.ig.com/wtp', 'Connection': 'keep-alive', 'Sec-Fetch-Mode': 'cors', 'Content-Type': 'text/plain', 'Origin': 'https://deal.ig.com', 'Content-Length': '955', 'Accept-Language': 'en-GB', 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/80.0.3987.122 Safari/537.36', 'Sec-Fetch-Site': 'cross-site', 'Sec-Fetch-Dest': 'empty', 'Accept': 'text/html, */*; q=0.01'}
{'Host': 'col.eum-appdynamics.com', 'Accept-Encoding': 'gzip, deflate, br', 'Referer': 'https://deal.ig.com/wtp', 'Connection': 'keep-alive', 'Sec-Fetch-Mode': 'cors', 'Origin': 'https://deal.ig.com', 'Content-Length': '2142', 'Accept-Language': 'en-GB', 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/80.0.3987.122 Safari/537.36', 'Sec-Fetch-Site': 'cross-site', 'Content-type': 'text/plain', 'Sec-Fetch-Dest': 'empty', 'Accept': '*/*'}
I want to save it to a file, here is my code:
for ent in proxy.har['log']['entries']:
url = ent['request']['headers']
response = ent['response']
#print(ent['request']['headers'])
headers = {e['name']: e['value'] for e in ent['request']['headers']}
print(headers)
#time.sleep(20)
with open('har.txt', 'w') as har:
json.dump(headers, har)
The har.txt file for some reason only saves the last row from the har file, not sure why thats happening
with open('har.txt', 'a') as har:
use 'a' to append.. not 'w' to write
With open('...','w') you are erasing the content of the file on every loop. Open it in append mode so id doesn't get erased:
open('har.txt', 'a')