I download options chain data from this page: https://opstra.definedge.com/optionchain
I found an api to download the json data with python here at https://opstra.definedge.com/api/openinterest/optionchain/free/NIFTY&26MAY2022
All was working fine for few months. Suddenly I started getting a 401 error. How can I bypass the 401 error?
I have a account to authenticate myself against the server.
My previous python code that is not working anymore
import requests
sym = "NIFTY"
exp_date = '26MAY2022'
# def oc(sym,exp_date ):
url = "https://opstra.definedge.com/api/openinterest/optionchain/"+sym+'&'+exp_date
headers = { "accept-encoding" : "gzip, deflate, br",
"accept-language" : "en-US,en;q=0.9",
"referer": "https://opstra.definedge.com/optionchain",
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.136 Safari/537.36"}
response = requests.get(url, headers=headers)
print(response)
Can anyone help me in getting the data again with python?
Related
Currently I have this code to send the request:
import requests
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.1 Safari/605.1.15','Accept-Encoding': 'gzip, deflate, br','Accept-Language': 'en-US'}
url = "https://www.nseindia.com/api/option-chain-indices?symbol=NIFTY"
res = requests.get(url, headers = headers)
But when I execute the last line, it either hangs for a long time, or just gives 401 error.
I tried other variations of headers, but still not getting the correct response.
Just pasting the above url in the browser, gives the json object painlessly.
Also if I try the above code from an online python tool, I'm sometimes getting the needed o/p. Does this mean I'll be able to get this working only when I host it on a trusted domain or something?
This should work:
import requests
headers = {
'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
'Accept-Encoding':'gzip, deflate, br',
'Connection':'keep-alive',
'Cookie':'AKA_A2=A; ak_bmsc=2C30C88FD1C6BEED087CCD02E7643772~000000000000000000000000000000~YAAQPvASAr+cI2F6AQAAk6+tdQxncocfEFby+qeRnNu3MgRblj1MWVtVy+W1Stx/CNaRaf9PhfVoT568zV8qztByVrxV+WfdrCN2nXU0nToPdEoaZFeZ7irUu8aSUXcln/sou0taKkr1gjmS3f6faZs+Rv8LA32eUAtlTD+GgYL0OKTJ44qVVinDxeeaVOiLxzQaiv0YjRCLcovFhO7jVBCJhNeXzgOeUYCLjkOg+2DEnRaF1Cd85f83pkjjieOFpjvywz20ImVWy1fr+S2nEDqmcgKZdhjHPfJ76+Z3bvVB/Kyv2dH7J8BMjlVf7kxyGbmot54yxchJNEMs0A/QTkeow2Xa54IcGZo/RUxGRu90SFu6VpfcxLaVOdN9EbvhcNs//OPA1jhDm9Nf4A==; bm_sv=BB4B29FC4D88791AABD65B43FACB0AF7~ObLG1UzBN4vOInl5m0vWqjOpZUXtLDHJDxr92uXdHHp5bjKjrEMMJcJRzS5VY5lkIs3N7JH+gZtoTnYIWKFqPZFhFC8Oo+sjmZLrin4taKkPfpvp7RdbqySQh6BLQwbWg3UgQJUQN29H0q9MJN6FuaW2b2i13zn5CmZUSDSpJVo=',
'Host':'www.nseindia.com',
'Sec-Fetch-Dest':'document',
'Sec-Fetch-Mode':'navigate',
'Sec-Fetch-Site':'none',
'Sec-Fetch-User':'?1',
'TE':'trailers',
'Upgrade-Insecure-Requests':'1',
'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:90.0) Gecko/20100101 Firefox/90.0',
}
r = requests.get('https://www.nseindia.com/api/option-chain-indices?symbol=NIFTY', headers=headers)
print(r.json())
I'm trying to login to a specific game site using python, this is my code:
import requests
import webbrowser
s = requests.session()
url = "https://www.novaragnarok.com/"
myheaders = {'user_agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like
Gecko) Chrome/77.0.3865.120 Safari/537.36 OPR/64.0.3417.146'}
mydata = {'server':'NovaRO','username':'MyUsername','password':'MyPassword'}
r = s.post(url, data=mydata, headers=myheaders)
print(r.content)
Replacing MyUsername and MyPassword with my account doesn't return the page after Login, any hints on what I'm doing wrong?
There are two issues with your code.
First of all, you are sending the POST request to the wrong URL, try using this URL:
https://www.novaragnarok.com:443/?module=account&action=login&call=ajax
Secondly, I would recommend replacing your current headers with the following ones:
myheaders = {
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:70.0) Gecko/20100101 Firefox/70.0",
"Origin": "https://www.novaragnarok.com",
"DNT": "1",
"Referer": "https://www.novaragnarok.com/",
"Upgrade-Insecure-Requests": "1",
}
hope this helps
so ive been trying to figure out how to do the 'follow' thing using python codes on imvu.com, but it always returns the message "invalid arguments" error in $: failed reading: not a valid json value"
import requests
headers = {
"Origin": "https://secure.imvu.com/",
"Referer": "https://secure.imvu.com/next/av/Sammy165/",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36",
"X-IMVU-SAUCE": "" #removed sauce for account safety
}
url = "https://api.imvu.com/profile/profile-user-696969696/subscriptions"
data = {"id": "https://api.imvu.com/profile/profile-user-175389029"}
req = requests.post(url=url, headers=headers, data=data)
print(req.text)
Have you tried
requests.post(url=url, headers=headers, json=data)
?
You have to do json.dumps(data). See code below
import requests
import json
headers = {
"Origin": "https://secure.imvu.com/",
"Referer": "https://secure.imvu.com/next/av/Sammy165/",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36",
"X-IMVU-SAUCE": "" #removed sauce for account safety
}
url = "https://api.imvu.com/profile/profile-user-696969696/subscriptions"
data = {"id": "https://api.imvu.com/profile/profile-user-175389029"}
req = requests.post(url=url, headers=headers, data=json.dumps(data))
print(req.text)
Output:
{"status":"failure","error":"ERROR-GENERIC-001","message":"Permission Denied: You are not allowed to modify this subscription set."}
I'm trying to scraping an AJAX loaded part on a webpage without executing the javascript. By using Chrome dev tool, I found that the AJAX container is pulling the content from a URL through a POST request, so I want to duplicate the request with python requests package. But strangely, by using the Headers information given from Chrome, I always get 400 error, and the same happens with the curl command copied from Chrome. So I'm wondering whether someone could kindly share some insights.
The website I'm interested in is here. Using Chrome: ctrl-shift-I, network, XHR, and the part I want is 'content'. The script I'm using is:
headers = {"authority": "cafe.bithumb.com",
"path": "/boards/43/contents",
"method": "POST",
"origin":"https://cafe.bithumb.com",
"accept-language": "zh-CN,zh;q=0.9,en;q=0.8",
"user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36",
"accept-encoding":"gzip, deflate, br",
"content-type": "application/x-www-form-urlencoded; charset=UTF-8",
"accept":"application/json, text/javascript, */*; q=0.01",
"referer":"https://cafe.bithumb.com/view/boards/43",
"x-requested-with":"XMLHttpRequest",
"scheme": "https",
"content-length":"1107"}
s=requests.Session()
s.headers.update(headers)
r = s.post('https://cafe.bithumb.com/boards/43/contents')
You just need to compare two post data, then you will find they have almost same except the a few parameter(draw=page...start=xx). That means you can scrape Ajax data by modifying draw and start.
Edit: Data was transformed to dictionary so we do not need urlencode, also we don't need cookie(i tested).
import requests
import json
headers = {
"Accept": "application/json, text/javascript, */*; q=0.01",
"Origin": "https://cafe.bithumb.com",
"X-Requested-With": "XMLHttpRequest",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.92 Safari/537.36",
"DNT": "1",
"Content-Type": "application/x-www-form-urlencoded; charset=UTF-8",
"Referer": "https://cafe.bithumb.com/view/boards/43",
"Accept-Encoding": "gzip, deflate, br"
}
string = """columns[0][data]=0&columns[0][name]=&columns[0][searchable]=true&columns[0][orderable]=false&columns[0][search][value]=&columns[0][search][regex]=false&columns[1][data]=1&columns[1][name]=&columns[1][searchable]=true&columns[1][orderable]=false&columns[1][search][value]=&columns[1][search][regex]=false&columns[2][data]=2&columns[2][name]=&columns[2][searchable]=true&columns[2][orderable]=false&columns[2][search][value]=&columns[2][search][regex]=false&columns[3][data]=3&columns[3][name]=&columns[3][searchable]=true&columns[3][orderable]=false&columns[3][search][value]=&columns[3][search][regex]=false&columns[4][data]=4&columns[4][name]=&columns[4][searchable]=true&columns[4][orderable]=false&columns[4][search][value]=&columns[4][search][regex]=false&start=30&length=30&search[value]=&search[regex]=false"""
article_root = "https://cafe.bithumb.com/view/board-contents/{}"
for page in range(1,4):
with requests.Session() as s:
s.headers.update(headers)
data = {"draw":page}
data.update( { ele[:ele.find("=")]:ele[ele.find("=")+1:] for ele in string.split("&") } )
data["start"] = 30 * (page - 1)
r = s.post('https://cafe.bithumb.com/boards/43/contents', data = data, verify = False) # set verify = False while you are using fiddler
json_data = json.loads(r.text).get("data") # transform string to dict then we can extract data easier
for each in json_data:
url = article_root.format(each[0])
print(url)
I am trying to access this URL by using requests.get() but it gives 405. No such issue when accessing via browser.
I have added the proper headers required, try this
import requests
headers = {
"Host": "www.propertyshark.com",
"Connection": "keep-alive",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36",
"Upgrade-Insecure-Requests": "1",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
"Accept-Encoding": "gzip, deflate, br",
"Accept-Language": "en-US,en;q=0.9",
}
url = "https://www.propertyshark.com/mason/Property/53047525/1302-Richland-Ave-Santa-Ana-CA-92703/"
req = requests.get(url, headers=headers)
print(req.status_code)
Just a guess,probably because you are trying to access the page directly.
Can you try adding a referral url
Where my_referer is the url of the website home page
requests.get(url, headers={'referer': my_referer})