I am using the following code:
import requests
from requests.structures import CaseInsensitiveDict
url='https://public-api.solscan.io/account/transactions?account=24jvtWN7qCf5GQ5MaE7V2R4SUgtRxND1w7hyvYa2PXG6'
headers = CaseInsensitiveDict()
headers["accept"] = "application/json"
resp = requests.get(url, headers=headers)
print(resp.json())
I get an error: JSONDecodeError: Expecting value: line 1 column 1 (char 0)
I expect to get an output with in the form of key:value and then covnvert that to a DataFrame. Why does my error occur?
Use headers that actually work.
Try this:
import requests
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36'
}
url = 'https://public-api.solscan.io/account/transactions?account=24jvtWN7qCf5GQ5MaE7V2R4SUgtRxND1w7hyvYa2PXG6'
resp = requests.get(url, headers=headers)
print(resp.json()[0]['signer'])
Sample output:
['7BjLjdJEGLaLscYkpw57YKzYqRY1i3ypnfLK8R2bgDrC', '7qfzWZmyYU1PBYJG5Y2ksSbaf6xHc77Tx47urzySFins']
Related
To send the message on my Telegram channel, I use the following template that works perfectly:
import requests
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36"
}
chat_telegram = ['XXXXXXXXXXXXX']
textalert = f'Test'
botalert = 'YYYYYYYYYYYYYYYYYYYYY'
urlalert = f'https://api.telegram.org/bot{botalert}/sendMessage'
params = {'text':textalert, 'chat_id':chat_telegram, 'parse_mode':'HTML'}
return_request = requests.get(urlalert, headers=headers, params=params)
I've tried using all four of the following ways to try to retrieve the message ID that was sent:
print(return_request['update']['message']['message_id'])
print(return_request['message']['message_id'])
print(return_request.update.message.message_id)
print(return_request.message.message_id)
But none returned positively, how should I proceed to recover the value I need?
Here is the API map
The message id is stored in the response.text part of the response, you can retrieve it via the json method:
import requests
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36"
}
chat_id = ['chat id']
textalert = f'Test'
botalert = 'some token'
urlalert = f'https://api.telegram.org/bot{botalert}/sendMessage'
params = {'text':textalert, 'chat_id':chat_id, 'parse_mode':'HTML'}
response = requests.get(urlalert, headers=headers, params=params)
json_data = response.json()
# get the message id
print('message id', json_data['result']['message_id'])
alternatively, you can retrieve it with the json module:
import json
json_data = json.loads(response.text)
# json_data['result']['message_id']
For a small project I tried to get past the google consent page to webscrape the prices found by it. However i could not get the to the good stuff. Sofar i tried the following code, as also proposed by Using python requests with google search.
First try:
import requests
import pandas as pd
from bs4 import BeautifulSoup as bs
#%% get acces
headers = {
'user-agent': 'Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36',
"cookie": "CONSENT=YES"}
cookie = {
'set-cookie': 'CONSENT=YES+cb.20211212-16-p1.nl+FX+671'
}
url = 'https://www.google.com/search?q=kitesurf+bar&rlz=1C1CAFA_enNL603NL674&sxsrf=AOaemvJF3BPrjeczkCI1e1YotCSJYcKjug:1641909060258&source=lnms&tbm=shop&sa=X&ved=2ahUKEwiBwqjy66n1AhXO-KQKHUKVBSoQ_AUoAXoECAEQAw'
s = requests.Session()
s.get(url,headers=headers)
purl = 'https://consent.google.com/s'
payload = {
'gl': 'NL',
'm': 'false',
'pc': 'srp',
'continue': 'https://www.google.com/search?q=kitesurf+bar&rlz=1C1CAFA_enNL603NL674&sxsrf=AOaemvJF3BPrjeczkCI1e1YotCSJYcKjug:1641909060258&source=lnms&tbm=shop&sa=X&ved=2ahUKEwiBwqjy66n1AhXO-KQKHUKVBSoQ_AUoAXoECAEQAw',
'ca': 'r',
'x': '6',
'v': 'cb.20211212-16-p1.nl+FX+092',
't': 'ADw3F8jHa3HqOqq133-wOkXCYf4K_r-AIA:1641909071268',
'hl': 'nl',
'src': '1'
}
s.post(purl,params=payload,headers=headers)
page = s.get(url, headers=headers, cookies=cookie)
Second try:
import requests
from bs4 import BeautifulSoup
with requests.Session() as s:
# url = f"https://www.google.com/search?q=fitness+wear"
headers = {
"referer":"referer: https://www.google.com/",
"user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36"
}
s.post(url, headers=headers)
response = s.get(url, headers=headers)
soup = BeautifulSoup(response.text, 'html.parser')
print(soup)```
I'm trying to scrape all the follower names from a profile page using requests module. The problem is when I run the script below, I get the first 20 names over and over again.
The parameters used in post requests only have two keys and values like size:20 and continuation:timestamp. I tried to use the parameters in the right way but still I get the same results repeatedly.
import time
import requests
link = 'https://api-mainnet.rarible.com/marketplace/api/v4/followers'
params = {'user': '0xe744d23107c9c98df5311ff8c1c8637ec3ecf9f3'}
payload = {"size": 20}
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36',
'origin': 'https://rarible.com',
'referer': 'https://rarible.com/'
}
with requests.Session() as s:
s.headers.update(headers)
while True:
res = s.post(link,params=params,json=payload)
print(s.headers)
for item in res.json():
print(item['owner'].get('name',''))
payload['continuation'] = f"{int(time.time() * 1000)}"
time.sleep(2)
How can I parse all the follower names from that page using requests?
Your next continuation value is in X-CONTINUATION response header, so this will work when increasing size in payload doesn't:
import requests
link = 'https://api-mainnet.rarible.com/marketplace/api/v4/followers'
params = {'user': '0xe744d23107c9c98df5311ff8c1c8637ec3ecf9f3'}
payload = {"size": 20}
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36',
'origin': 'https://rarible.com',
'referer': 'https://rarible.com/'
}
res = requests.post(link, headers=headers, params=params, json=payload)
print(res.headers["X-CONTINUATION"])
while True:
for item in res.json():
print(item['owner'].get('name',))
if not res.headers["X-CONTINUATION"]:
break
payload['continuation'] = res.headers["X-CONTINUATION"]
res = requests.post(link, headers=headers, params=params, json=payload)
some api may block you from extracting values more than certain limit and also may show in pages with limits.
For me just increasing the size payload worked with your code.
import time
import requests
link = 'https://api-mainnet.rarible.com/marketplace/api/v4/followers'
params = {'user': '0xe744d23107c9c98df5311ff8c1c8637ec3ecf9f3'}
payload = {"size": 10000}
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36',
'origin': 'https://rarible.com',
'referer': 'https://rarible.com/'
}
with requests.Session() as s:
s.headers.update(headers)
res = s.post(link,params=params,json=payload)
print(len(res.json()))
for item in res.json():
print(item['owner'].get('name',''))
im trying to get specific data in json response from spotify. My current code is:
with requests.Session()as(c):
url = 'https://accounts.spotify.com/en/login?continue=https:%2F%2Fwww.spotify.com%2Fint%2Faccount%2Foverview%2F'
headers = {'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36'}
page = c.get(url, headers=headers)
CSRF = page.cookies['csrf_token']
headers = {'Accept': '*/*', 'User-Agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 10_0_1 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) Version/10.0 Mobile/14A403 Safari/602.1',
'Referer': 'https://accounts.spotify.com/en/login/?continue=https:%2F%2Fwww.spotify.com%2Fus%2Fgooglehome%2Fregister%2F&_locale=en-US'}
url = 'https://accounts.spotify.com/api/login'
login_data = {
'remember': 'true',
'username': USER,
'password': PASS,
'csrf_token': CSRF
}
cookies = dict(__bon='MHwwfC0xNDAxNTMwNDkzfC01ODg2NDI4MDcwNnwxfDF8MXwx')
login = c.post(url, headers=headers, data=login_data, cookies=cookies)
if '{"displayName":"' in login.text:
url = 'https://www.spotify.com/us/account/overview/'
capture = c.get(url, headers=headers)
csr = capture.headers['X-Csrf-Token']
json_data = json.loads(login.text)
result = json_data['displayName']
print(result)
when I run this
{"displayName":"John Doe",
"smallImageUrl":"https://scontent.xx.fbcdn.net/v/t1.0-1/p50x50/10407982_10104564418730171_2968639978505808989_n.jpg?_nc_cat=110\u0026_nc_oc=AQmoXCg0tfbf9LuxGWAbpEv-96K57xmie4S3avDrYh3l90g8W-ParNV5mNK0oPU6ERk\u0026_nc_ht=scontent.xx\u0026oh=2fe2149364f012a3c5e43e6d999375ab\u0026oe=5DBD8940",
"largeImageUrl":"https://scontent.xx.fbcdn.net/v/t1.0-1/p200x200/10407982_10104564418730171_2968639978505808989_n.jpg?_nc_cat=110\u0026_nc_oc=AQmoXCg0tfbf9LuxGWAbpEv-96K57xmie4S3avDrYh3l90g8W-ParNV5mNK0oPU6ERk\u0026_nc_ht=scontent.xx\u0026oh=3aec23a8c56be536739ba4cca4e1cc6e\u0026oe=5DBC409D"}
I would like to only print the value: John Doe (SOLVED). but how to get John Doe value in field for discord bot ?
embed=discord.Embed()
embed.add_field(name=undefined, value=undefined, inline=False)
await self.bot.say(embed=embed)
you can use the json method of the requests response object.
login = c.post(url, headers=headers, data=login_data, cookies=cookies)
login_json = login.json()
if "displayName" in login_json:
print(login_json["displayName"])
Use json library
import json
#Your response, probably login.text
s = """{"displayName":"John Doe","smallImageUrl":"https://scontent.xx.fbcdn.net/v/t1.0-1/p50x50/10407982_10104564418730171_2968639978505808989_n.jpg?_nc_cat=110\u0026_nc_oc=AQmoXCg0tfbf9LuxGWAbpEv-96K57xmie4S3avDrYh3l90g8W-ParNV5mNK0oPU6ERk\u0026_nc_ht=scontent.xx\u0026oh=2fe2149364f012a3c5e43e6d999375ab\u0026oe=5DBD8940","largeImageUrl":"https://scontent.xx.fbcdn.net/v/t1.0-1/p200x200/10407982_10104564418730171_2968639978505808989_n.jpg?_nc_cat=110\u0026_nc_oc=AQmoXCg0tfbf9LuxGWAbpEv-96K57xmie4S3avDrYh3l90g8W-ParNV5mNK0oPU6ERk\u0026_nc_ht=scontent.xx\u0026oh=3aec23a8c56be536739ba4cca4e1cc6e\u0026oe=5DBC409D"}"""
dj = json.loads(s)
print(dj["displayName"])
OUTPUT
John Doe
from fake_useragent import UserAgent
import requests
ua = UserAgent()
header = {'User-Agent':str(ua.chrome)}
d = {"query": "/api/v2/details/ip/", "query_entry": "41.219.127.69"}
r = requests.get("https://talosintelligence.com/sb_api/query_lookup/",
data = d, headers=header)
When I run the same result from the main site "talosintelligence.com" and look at the network counsel, that exact URL is responds with a JSON file but a get request from python returns None.
I got it to work by setting the referer header..
import requests
sess = requests.session()
ip_addr = "41.219.127.69"
ret = sess.get('https://talosintelligence.com/sb_api/query_lookup', data={"query": "/api/v2/details/ip/", "query_entry": ip_addr, "offset": 0, "order": "ip asc"}, headers={'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.31 Safari/537.36', 'referer': 'https://talosintelligence.com/reputation_center/lookup?search=' + ip_addr})