Disclaimer: I am very new to python, and have no idea what i am doing, i am teaching myself from the web.
I have some code that looks like this
Code:
from request import get # Ed: added for clarity
myurl = URLBASE + args.key
response = get(myurl)
# check key is valid
json = response.text # Ed: this is a requests.Response object
print(json)
if json is None:
sys.exit("Problem getting API data, check your key")
print("how did i get here")
Output:
null
how did i get here
But I have no idea how that is possible ... it literally says it is null in the print, but then doesn't match in the 'if'. Any help would be appreciated.
thx
So I am sure I still don't fully understand, but this "fixes" my problem.
The requests.Response object has Property/Method json - so i should have been using that, thanks wim, instead of text. So changing the code to this (below), as suggested, makes the code work.
from request import get
myurl = URLBASE + args.key
response = get(myurl)
# check key is valid
json = response.json()
if json is None:
sys.exit("Problem getting API data, check your key")
print("how did i get here")
The question (for me inquisitively) remains, how would I do an if statement to determine if a string is null?
Thanks to Ry and wim, for their help.
Related
Hey guys I was wondering if there was a way to make my bot skip invalid urls after 1 try to continue with the for loop but continue doesn't seem to work
def check_valid(stripped_results):
global vstripped_results
vstripped_results = []
for tag in stripped_results:
conn = requests.head("https://" + tag)
conn2 = requests.head("http://" + tag)
status_code = conn.status_code
website_is_up = status_code == 200
if website_is_up:
vstripped_results.append(tag)
else:
continue
stripped results is an array of an unknown amount of domains and Subdomains which is why I have the 'https://' part and tbh I'm not even sure whether my if statement is effective or not.
Any help would be greatly appreciated I don't want to get rate limited by discord anymore from sending so many invalid domains through. :(
This is easy. To check the validity of a URL there exist a python library, namely Validators. This library can be used to validate any URL for if it exist or not. Let's take it step by step.
Firstly,
Here is the documentation link for validators:
https://validators.readthedocs.io/en/latest/
How do you validate a link using validators?
It is simple. Let's work on command line for a moment.
This image shows it. This module gives out boolean result on if it is a valid link or not.
Here for the link of this question it gave out True and when it would be false then it would give you the error.
You can validate it using this syntax:
validators.url('Add your URL variable here')
Remember that this gives boolean value so code for it that way.
So you can use it this way...
I wouldn't be implementing it in your code as I want you to try it yourself once. I would help you with this if you are unable to do it.
Thank You! :)
Try this?
def check_valid(stripped_results):
global vstripped_results
vstripped_results = []
for tag in stripped_results:
conn = requests.head("https://" + tag)
conn2 = requests.head("http://" + tag)
status_code = conn.status_code
website_is_up = status_code == 200
if website_is_up:
vstripped_results.append(tag)
else:
#Do the thing here
I'm trying to get an output url from a json (not a file) and store it in a variable, but, i'm always get a KeyError, and i don't know why, already tried everything, i think.
Code:
import requests
start_url = 'url_here'
r = requests.post(
"https://api.deepai.org/api/deepdream",
data={'image': start_url},
headers={'api-key': 'xxx-xxxx-xxxx-xxxx'}
)
final_url = r.json()['output_url']
i get this message in the terminal:
final_url = r.json()['output_url']
KeyError: 'output_url'
it should just store it in the final_url variable, but it's not happening.
ps: already tried print(final_url), i get nothing.
can someone help me? Or clarify why that's happening? Thank you!
I am using the code below which I found here, but it always returns the same quote. I can see that data itself has more quotes, but not sure how to parse them out one at a time. Thank you.
import requests
url = "http://www.forbes.com/forbesapi/thought/uri.json?enrich=true&query=1&relatedlimit=5"
response = requests.get(url)
data = response.json()
quote=data['thought']['quote'].strip()
Returns:
u'Teach self-denial and make its practice pleasure, and you can create for the world a destiny more sublime that ever issued from the brain of the wildest dreamer.'
type(data) returns a dict. Then data.keys() returns [u'thought']. data.items() returns a bunch of text. Why only one key, if are seemingly more than one quote inside? And why does data = response.json() return disc with type(data), rather than a json object?
I copied your json file at http://jsonviewer.stack.hu/ and saw the structure of data. Once you understand the structure.
import requests
url = "http://www.forbes.com/forbesapi/thought/uri.json?enrich=true&query=1&relatedlimit=5"
response = requests.get(url)
data = response.json()
#print recent quote from Thought
print '*'*10
quote=data['thought']["quote"]
print quote
#print quotes from related authors
print '*'*10
quotes_of_related_authors = data['thought']['relatedAuthorThoughts']
for i in quotes_of_related_authors:
print i.get('quote')
# print quotes from related theme thoughts
print '*'*10
quotes_of_related_theme_thoughts = data['thought']['relatedThemeThoughts']
for i in quotes_of_related_theme_thoughts:
print i.get('quote')
So I'm experimenting with json abit and this is the code I've got this far,
import json
from utorrent.client import UTorrentClient
uTorrent = UTorrentClient("xxxx", "xxxx", "xxxx")
data = uTorrent.list()
torrents = json.loads(data)["torrents"]
for torrent in torrents:
print item[0] # hash
print item[2] # name
print item[21] # status
print item[26] # folder
The typical json output can be viewed here. But im getting an "expected string or buffer" error. Anyone with any pointers?
The point with above code is to print out each hash/name.. for each torrent found in the list provided by uTorrent
Did you try using load instead of loads? I was having the same problem and I realized there's a difference.
I'm looking at scraping some data from Facebook using Python 2.7. My code basically augments by 1 changing the Facebook profile ID to then capture details returned by the page.
An example of the page I'm looking to capture the data from is graph.facebook.com/4.
Here's my code below:
import scraperwiki
import urlparse
import simplejson
source_url = "http://graph.facebook.com/"
profile_id = 1
while True:
try:
profile_id +=1
profile_url = urlparse.urljoin(source_url, str(profile_id))
results_json = simplejson.loads(scraperwiki.scrape(profile_url))
for result in results_json['results']:
print result
data = {}
data['id'] = result['id']
data['name'] = result['name']
data['first_name'] = result['first_name']
data['last_name'] = result['last_name']
data['link'] = result['link']
data['username'] = result['username']
data['gender'] = result['gender']
data['locale'] = result['locale']
print data['id'], data['name']
scraperwiki.sqlite.save(unique_keys=['id'], data=data)
#time.sleep(3)
except:
continue
profile_id +=1
I am using the scraperwiki site to carry out this check but no data is printed back to console despite the line 'print data['id'], data['name'] used just to check the code is working
Any suggestions on what is wrong with this code? As said, for each returned profile, the unique data should be captured and printed to screen as well as populated into the sqlite database.
Thanks
Any suggestions on what is wrong with this code?
Yes. You are swallowing all of your errors. There could be a huge number of things going wrong in the block under try. If anything goes wrong in that block, you move on without printing anything.
You should only ever use a try / except block when you are looking to handle a specific error.
modify your code so that it looks like this:
while True:
profile_id +=1
profile_url = urlparse.urljoin(source_url, str(profile_id))
results_json = simplejson.loads(scraperwiki.scrape(profile_url))
for result in results_json['results']:
print result
data = {}
# ... more ...
and then you will get detailed error messages when specific things go wrong.
As for your concern in the comments:
The reason I have the error handling is because, if you look for
example at graph.facebook.com/3, this page contains no user data and
so I don't want to collate this info and skip to the next user, ie. no
4 etc
If you want to handle the case where there is no data, then find a way to handle that case specifically. It is bad practice to swallow all errors.