Unfortunately i can provide only the output of the request and not the full code since it contains quite private infos, basically when printing the request as text file i get a json one, something like that:
{"paymentResource":{"paymentToken":"PAYID-MEJ------","intent":"authorize","redirectUrl":"https://www.paypal.com/checkoutnow?nolegacy=1\u0026token=EC-5JS-----2S","authenticateUrl":null}}
How can i scrape that Paypal url? tried by doing this but it didn't worked (ppstep2 is the name of the request):
content = ppstep2.json()
pp = content["redirectUrl"]
I only get this error while doing it:
pp = content["redirectUrl"]
KeyError: 'redirectUrl'
Your variable content is a dictionary.
To get the value for "redirectUrl" you can do this:
pp = content['paymentResource']['redirectUrl']
The key error was caused by not including ['paymentResource']
I would recommend reviewing python dictionaries and the .get() method as well.
https://docs.python.org/3/library/stdtypes.html?highlight=dictionary%20get#dict.get
Try adding a print(json.dumps(content, indent=4)) before you try to access it and look at the output. You might spot why then.
redirectUrl isn't part of content. It's in the content['paymentResource'] dictionary in that response content.
using content['paymentResource']['redirectUrl'] should work.
Edit: If you want to try and get a value without ending up with an exception, try using .get():
# This will result in a KeyError as you experienced:
pp = content["redirectUrl"]
# This will instead set pp to None if 'redirectUrl' doesn't exist as a Key
pp = content.get("redirectUrl", None)
import requests
nexmokey = 'mykey'
nexmosec = 'mysecretkey'
nexmoBal = 'https://rest.nexmo.com/account/get-balance?api_key={}&api_secret={}'.format(nexmokey,nexmosec)
rr = requests.get(nexmoBal)
print(rr.url)
I would like to send a request to post at
https://rest.nexmo.com/account/get-balance?api_key=mykey&api_secret=mysecretkey
but why does %0D appear?
https://rest.nexmo.com/account/get-balance?api_key=mykey%0D&api_secret=mysecretkey%0D
requests.get expects parameters like api_secret=my_secret to be provided through the params argument, not as part of the URL, which is URL-encoded for you.
Use this:
nexmoBal = 'https://rest.nexmo.com/account/get-balance'
rr = requests.get(nexmoBal, params={'api_key': nexmokey, 'api_secret': nexmosec})
The fact that %0D ends up in there, indicates you have a character #13 (0D hexadecimal) in there, which is a carriage return (part of the end of line on Windows systems) - probably because you are reading the key and secret from some file and didn't include them in the example code.
Also, note that you mention you want to post, but you're calling .get().
This question already has answers here:
What does "SyntaxError: Missing parentheses in call to 'print'" mean in Python?
(11 answers)
Closed 5 years ago.
I've been trying to scrape some twitter's data, but when ever I run this code I get the error SyntaxError: Missing parentheses in call to 'print'.
Can someone please help me out with this one?
Thanks for your time :)
"""
Use Twitter API to grab user information from list of organizations;
export text file
Uses Twython module to access Twitter API
"""
import sys
import string
import simplejson
from twython import Twython
#WE WILL USE THE VARIABLES DAY, MONTH, AND YEAR FOR OUR OUTPUT FILE NAME
import datetime
now = datetime.datetime.now()
day=int(now.day)
month=int(now.month)
year=int(now.year)
#FOR OAUTH AUTHENTICATION -- NEEDED TO ACCESS THE TWITTER API
t = Twython(app_key='APP_KEY', #REPLACE 'APP_KEY' WITH YOUR APP KEY, ETC., IN THE NEXT 4 LINES
app_secret='APP_SECRET',
oauth_token='OAUTH_TOKEN',
oauth_token_secret='OAUTH_TOKEN_SECRET')
#REPLACE WITH YOUR LIST OF TWITTER USER IDS
ids = "4816,9715012,13023422, 13393052, 14226882, 14235041, 14292458, 14335586, 14730894,\
15029174, 15474846, 15634728, 15689319, 15782399, 15946841, 16116519, 16148677, 16223542,\
16315120, 16566133, 16686673, 16801671, 41900627, 42645839, 42731742, 44157002, 44988185,\
48073289, 48827616, 49702654, 50310311, 50361094,"
#ACCESS THE LOOKUP_USER METHOD OF THE TWITTER API -- GRAB INFO ON UP TO 100 IDS WITH EACH API CALL
#THE VARIABLE USERS IS A JSON FILE WITH DATA ON THE 32 TWITTER USERS LISTED ABOVE
users = t.lookup_user(user_id = ids)
#NAME OUR OUTPUT FILE - %i WILL BE REPLACED BY CURRENT MONTH, DAY, AND YEAR
outfn = "twitter_user_data_%i.%i.%i.txt" % (now.month, now.day, now.year)
#NAMES FOR HEADER ROW IN OUTPUT FILE
fields = "id screen_name name created_at url followers_count friends_count statuses_count \
favourites_count listed_count \
contributors_enabled description protected location lang expanded_url".split()
#INITIALIZE OUTPUT FILE AND WRITE HEADER ROW
outfp = open(outfn, "w")
outfp.write(string.join(fields, "\t") + "\n") # header
#THE VARIABLE 'USERS' CONTAINS INFORMATION OF THE 32 TWITTER USER IDS LISTED ABOVE
#THIS BLOCK WILL LOOP OVER EACH OF THESE IDS, CREATE VARIABLES, AND OUTPUT TO FILE
for entry in users:
#CREATE EMPTY DICTIONARY
r = {}
for f in fields:
r[f] = ""
#ASSIGN VALUE OF 'ID' FIELD IN JSON TO 'ID' FIELD IN OUR DICTIONARY
r['id'] = entry['id']
#SAME WITH 'SCREEN_NAME' HERE, AND FOR REST OF THE VARIABLES
r['screen_name'] = entry['screen_name']
r['name'] = entry['name']
r['created_at'] = entry['created_at']
r['url'] = entry['url']
r['followers_count'] = entry['followers_count']
r['friends_count'] = entry['friends_count']
r['statuses_count'] = entry['statuses_count']
r['favourites_count'] = entry['favourites_count']
r['listed_count'] = entry['listed_count']
r['contributors_enabled'] = entry['contributors_enabled']
r['description'] = entry['description']
r['protected'] = entry['protected']
r['location'] = entry['location']
r['lang'] = entry['lang']
#NOT EVERY ID WILL HAVE A 'URL' KEY, SO CHECK FOR ITS EXISTENCE WITH IF CLAUSE
if 'url' in entry['entities']:
r['expanded_url'] = entry['entities']['url']['urls'][0]['expanded_url']
else:
r['expanded_url'] = ''
print r
#CREATE EMPTY LIST
lst = []
#ADD DATA FOR EACH VARIABLE
for f in fields:
lst.append(unicode(r[f]).replace("\/", "/"))
#WRITE ROW WITH DATA IN LIST
outfp.write(string.join(lst, "\t").encode("utf-8") + "\n")
outfp.close()
It seems like you are using python 3.x, however the code you are running here is python 2.x code. Two ways to solve this:
Download python 2.x on Python's website and use it to run your script
Add parentheses around your print call at the end by replacing print r by print(r) at the end (and keep using python 3)
But today, a growing majority of python programmers are using python 3, and the official python wiki states the following:
Python 2.x is legacy, Python 3.x is the present and future of the
language
If I were you, I'd go with the second option and keep using python 3.
Looks like you trying to run Python 2 code in Python 3, where print is function and required parentheses:
print(foo)
You just need to add pranethesis to your print statmetnt to convert it to a function, like the error says:
print expression -> print(expression)
In Python 2, print is a statement, but in Python 3, print is a function. So you could alternatively just run your code with Python 2. print(expression) is backwards compatible with Python 2.
Also, why are you capitalizing all your comments? It's annoying. Your code also violates PEP 8 in several ways. Get an editor like PyCharm (it's free) that can automatically detect errors like this.
You didn't leave a space between # and your comment
You didn't leave spaces between = and other tokens
Within python 2, print has been a statement, not a function. That means you can use it without parentheses. In python 3, that has changed. It is a function there and you need to use print(foo) instead of print foo.
I am using Python3 and the package requests to fetch HTML data.
I have tried running the line
r = requests.get('https://github.com/timeline.json')
, which is the example on their tutorial, to no avail. However, when I run
request = requests.get('http://www.math.ksu.edu/events/grad_conf_2013/')
it works fine. I am getting errors such as
AttributeError: 'MockRequest' object has no attribute 'unverifiable'
Error in sys.excepthook:
I am thinking the errors have something to do with the type of webpage I am attempting to get, since the html page that is working is just basic html that I wrote.
I am very new to requests and Python in general. I am also new to stackoverflow.
As a little example, here is a little tool which I developed in order to fetch data from a website, in this case IP and show it:
# Import the requests module
# TODO: Make sure to install it first
import requests
# Get the raw information from the website
r = requests.get('http://whatismyipaddress.com')
raw_page_source_list = r.text
text = ''
# Join the whole list into a single string in order
# to simplify things
text = text.join(raw_page_source_list)
# Get the exact starting position of the IP address string
ip_text_pos = text.find('IP Information') + 62
# Now extract the IP address and store it
ip_address = text[ip_text_pos : ip_text_pos + 12]
# print 'Your IP address is: %s' % ip_address
# or, for Python 3 ... #
# print('Your IP address is: %s' % ip_address)
I don't know a single thing of perl but from a big perl script, I managed to get the relevant parts and make a HTTP request. So, this perl code works perfectly.
#!/usr/bin/perl -w
use strict;
use LWP::UserAgent;
use HTTP::Request::Common;
my $ua = new LWP::UserAgent;
my $request = "X-CTCH-PVer: 0000001\r\n";
my $method_url = "http://localhost:8088/ctasd/GetStatus";
my $response = $ua->request (POST $method_url,Content => $request);
my $data = $response->status_line . "\n";
print $data;
print $response->content;
The above code outputs:
200 OK
X-CTCH-PVer: 0000001
From my understanding, it's doing a POST to a URL with the specified data. With that base, my python code looks like:
#!/usr/bin/python
import urllib
url = "http://localhost:8088/ctasd/GetStatus"
data = urllib.urlencode([("X-CTCH-PVer", "0000001")])
print urllib.urlopen(url, data).read()
But, this returns response as:
X-CTCH-Error: Missing protocol header "X-CTCH-PVer"
Please help me in making a Python equivalent of perl code.
So, the actual thing was, the $request in Perl was literally been sent as POST data without any change. Now I get why is the name content in Perl.
#!/usr/bin/python
import urllib
url = "http://localhost:8088/ctasd/GetStatus"
print urllib.urlopen(url, "X-CTCH-PVer: 0000001").read()
Worked. I actually found out about this after capturing the traffic in both cases and analysing it in wireshark.
The error is because you are not sending the header, you are making/sending a urlencoded string, hence the function urllib.urlencode
Try setting the request with actual headers:
#!/usr/bin/python
import urllib2
request = urllib2.Request("http://localhost:8088/ctasd/GetStatus", headers={"X-CTCH-PVer" : "0000001"})
contents = urllib2.urlopen(request).read()