I am trying to get a location from an ip address, the code for that is:
import urllib.request
import json
with urllib.request.urlopen("https://geolocation-db.com/json") as url:
data = json.loads(url.read().decode())
print(data)
This does produce a result but it is the wrong result and gives me the wrong address (although the right countr. However, when I use a website like https://iplocation.com/ I do get my proper address.
I am quite confused why there are different results, could someone help?
Nevermind, I realised that the website itself is producing weird and inaccurate results. Didn't even think that could be the case.
Related
I am not experienced in web development, and trying to use requests.get to get some authenticated data. So far the internet appears to tell me to just do it, and i think i am formatting it wrong, but unsure how. After some trial and error, i was able to grab my cookie for the website. The following is some a made up version of what i grabbed with similar formating.
cookie = "s:abcDEfGHIJ12k34LMNopqRst5UvW-6xy.ZAbCd/eFGhi7j8KlmnoPqrstUvWXYZ90a1BCDE2fGH3"
Then, in python, i am trying to send a request. Following is a bit more pseudo code for what i am doing
r = requests.get('https://www.website.com/api/getData', cookies={"connect.sid": cookie})
After all this, the site keeps sending me a 400 error. Wondering if you guys had any idea if I am putting in the wrong cookie/part of cookie. If everything looks right and it is probably the site at fault, or what.
Grabbed a wireshark capture, and found there were other fields in the cookie that were sent that i had not filled out.
_ga
_gid
___gads
Filled those out with the relevant values, and it works.
hope you are all doing well. This question is a bit more random than others I have asked. I am making a bot that extracts every username of the first 600,000,000 accounts on the platform Roblox, and loads it into a list.
This is my problem. I am using requests to get to the account page, but I can't find out how to extract the username from that page. I have tried using headers and inspect element but they don't work. If anyone has some suggestions on how to complete this, please help. Also, I am extraordinarily bad at network programming, so I may have made a noob mistake somewhere. Code is attached below.
import requests
users = []
for i in range(1, 600000001):
r = requests.get("https://web.roblox.com/users/{i}/profile".format(i=i))
print(r.status_code)
if r.status_code == 404:
users.append('Deleted')
continue
print(r.headers.get('username'))
You have to know that before working on the scraping, you have some errors in the code:
First of all in the 4th line if you want to use the .format command to insert values in a string you only have to insert the {}; so you should write:
r = requests.get("https://web.roblox.com/users/{}/profile".format(i))
And later you should remove continue from your code
But before doing anything you have to try the link be sure it's working, so copy the link, past it on your browser and remove i and add a number.
If it works you can go on with the code, if not you have to find another link to access to the page you want.
Eventually, to take the elements from the html page you have to use r.content.
But before continuing with coding you have to print(r.content).
You will see a long dict full of elements but you don't have to be afraid of it:
you have to search the value that interest you, and see how it's called, and you will be able to call that value writing
`<name_of_variable> = r.content['<name_of_the_value>']`
Ok so I'm trying to get my likes in python in the same format as I use " get('e1/me/stream.json) ". My understanding is that I can only use v1 with get in python. Listening the requests I tried this url :
https://api.soundcloud.com/e1/me/track_likes/ids?app_version=c53f4bf&client_id=02gUJC0hH2ct1EGOcYXQIzRFU91c72Ea&cursor=1426363878000000&limit=5000&linked_partitioning=1&page_number=0&page_size=200
I get 401 Unauthorized everytime, but what's weird is that if I go to another url and then hit previous all the ids are displayed.
It used to be as simple as get(e1/me/likes.json) but now it doesn't work anymore.
Thank you so much and happy holidays to you guys!
Alex
Ok so the answer is
me/favorites?oauth_token=your_auth_token
Weird that it's perfectly fine to get your stream (https://api.soundcloud.com/e1/me/stream.json?) but need authentification for your public likes.
I'm trying to read the following URL into Python:
http://www.google.com/trends/fetchComponent?q=nepal&cid=TIMESERIES_GRAPH_0&export=3
with the code:
trend_url = 'http://www.google.com/trends/fetchComponent?q=nepal&cid=TIMESERIES_GRAPH_0&export=3'
response = urllib2.urlopen(trend_url)
the_page = response.read()
The resulting value of the_page, for reasons that I don't understand, is an error page.
UPDATE: I think that the problem is related to some authentication issue: when I try to open the link in the browser's incognito window, it also returns an error page.
use requests
import requests
a = requests.get('http://www.google.com/trends/fetchComponent?q=nepal&cid=TIMESERIES_GRAPH_0&export=3')
a.text
u'// Data table response\ngoogle.visualization.Query.setResponse({"version":" ....
I tested your example and it is works.
I think is kinda late, but I think that Google does that in order to protect their data. You have to create a web-scraping that will go to the interface put the word you want, and it will generate the page/url. That is not the same as going at first sight for the URL generated.
I'm trying to connect to a torrent tracker to receive a list of peers to play bit torrent with, however I am having trouble forming the proper GET request.
As far as I understand, I must obtain the 20 byte SHA1 hash of the bencoded 'info' section from the .torrent file. I use the following code:
h = hashlib.new('sha1')
h.update(bencode.bencode(meta_dict['info']))
info_hash = h.digest()
This is where I am stuck. I can not figure out how to create the proper url-encoded info_hash to stick into a URL string as a parameter.
I believe it involves some combination of urllib.urlencode and urllib.quote, however my attempts have not worked so far.
well a bit late but might help someone.
Using module requests encodes the url by it's self. First you need to create a dictionary with the parameters (info_hash, peer_id etc). Then you only have to do a get request
response = requests.get(tracker_url, params=params)
I think that urllib.quote_plus() is all you need.