I've just discovered something strange. When downloading data from facebook with GET using the requests 2.18.4 library, I get error when I just use
requests.get('https://.../{}/likes?acces_token={}'.format(userID,token))
into which I parse the user ID and access - the API does not read the access token correctly.
But, it works fine as
requests.get('https://../{}'.format(userID), params={"access_token":token})
Or it works when I copy paste the values in the appropriate fields by hand in the python console.
So my hypothesis is that it has something to with how the token string got parsed using the params vs the string. But what I don't understand at all, why would that be the case? Or is ? character somehow strange in this case?
Double check if both the URLs are the same (in your post they differ by the /likes substring).
Then you can check how the library requests concatenated parameters from the params argument:
url = 'https://facebook.com/.../{}'.format(userID)
r = requests.Request('GET', url, params={"access_token":token})
pr = r.prepare()
print pr.url
Related
I would like to get the COMPLETE URL using os.environ.
I am rendering notebooks with voila and I would like to open url from a dashboard using parameters in the URL.
So far I have:
URL_BASE = os.environ.get('SCRIPT_NAME')
PARAMETERS = os.environ.get("QUERY_STRING")
print(f'{URL_BASE=}')
print(f'{PARAMETERS=}')
assuming this is the url:
https://flyingcar.org/john/voila/render/shared/users/j/john/learn_url.ipynb?redirects=2&name=john&dossier=SA123445#{whatever=no/f=5}
URL_BASE="flyingcar.org/john/voila/render/shared/users/j/john/learn_url.ipynb"
&
PARAMETERS="redirects=2&name=john&dossier=SA123445"
Having a look at the whole collection of vars in os.environ I dont see any that would include the whole url (including what is there after #) in order to parse that part as well as with parameters.
captured_values = parse_qs(PARAMETERS)
print('here parse_qs of query:',captured_values)
>>> here parse_qs of query: {'d': ['34'], 'f': ['56']}
Some ideas?
I tried to print all the os.environ variables with:
for k,v in os.environ.items():
print(k,v)
but it nothing seems to contain what is beyond the # symbol in the URL
Any ideas?
Thanks
RELATED:
Get current URL in Python
The fragment (#) part of the URI never leaves the user agent, so it can't be picked up by the python web server. See: https://datatracker.ietf.org/doc/html/rfc3986#page-24
... instead, the fragment identifier is separated
from the rest of the URI prior to a dereference, and thus the
identifying information within the fragment itself is dereferenced
solely by the user agent, regardless of the URI scheme.
I am trying to have a request.get statement with two urls in it. What I am aiming to do is have requests (Python Module) make two requests based on list or two strings I provide. How can I pass multiple strings from a list into a request.get statement, and have requests go to each url (string) and have do something?
Thanks
Typically if we talking python requests library it only runs one url get request at a time. If what you are trying to do is perform multiple requests with a list of known urls then it's quite easy.
import requests
my_links = ['www.google.com', 'www.yahoo.com']
my_responses = []
for link in my_links:
payload = requests.get(link).json()
print('got response from {}'.format(link))
my_response.append(payload)
print(payload)
my_responses now has all the content from the pages.
You don't. The requests.get() method (or any other method, really) takes single URL and makes a single HTTP request because that is what most humans want it to do.
If you need to make two requests, you must call that method twice.
requests.get(url)
requests.get(another_url)
Of course, these calls are synchronous, the second will only begin once the first response is received.
I am having an issue with a http request I make using the python requests lib. When I make this query using the browsers, this is how the request looks like:
mysite/test?query=name%3A"My+Name+Is"*
Now, when I try to use the python lib, I use this dictionary as my params:
{'query' : 'name:My Name Is*'}
When I do this query using the browser, it works however, when I do this using the python lib, it does not work. Now my question is, should I be escaping the '*'? or are the spaces messing this up? the python lib should already be encoding this string so I would think I don’t need to do that prior to passing it. Any advice or comments are appreciated!
Here is a sample of the code making the request:
url = 'https://rink.hockeyapp.net/api/2/apps/app_id/app_versions/version_id/crash_reasons/search'
params={'per_page': 100, 'page': 1, 'query': 'reason:"Some String With Spaces*"'}
requests.get(url, params=params)
I am trying to scrape some data (reproduce a POST operation I did in a browser) by using Python Requests library. Expecting it will return the content I saw while using browser by copying the request header and post form.
However, I am not quite sure what is the correct way to send cookies using Python Requests.
Here is a screen shot how it looks like in Chrome.
It looks like I can either use cookie as a key in the request header or use the cookie argument in the post command.
(1). If I want to use cookie in the request header, should I treat the whole cookie content as a long string and set the key to be cookie and set the value to be that string?
(2). If I want to use the cookie argument in the request.post command, should I manually translate that long string into a dictionary first, and then pass to cookie argument?. Something like this?
mycookie = {'firsttimevisitor':'N'; 'cmTPSet':'Y'; 'viewType':'List'... }
# Then
r = requests.post(myurl, data=myformdata, cookies=mycookie, headers=myheaders)
Thanks!
Just follow the documentation:
>>> url = 'http://httpbin.org/cookies'
>>> cookies = dict(cookies_are='working')
>>> r = requests.get(url, cookies=cookies)
>>> r.text
'{"cookies": {"cookies_are": "working"}}'
So use a dict for cookie, and note it's cookies not cookie.
Yes. But make sure you call it "Cookie" (With capital C)
I always did it with a dict. Requests expects you to give a dict.
A string will give the following
cookiejar.set_cookie(create_cookie(name, cookie_dict[name]))
TypeError: string indices must be integers, not str
I am using Python and Google App Engine.
I need to get access to certain webpage by adding some elements to the url.
What is the syntax for adding a GET parameter to a URL?
You put ? in the end of the url. After the ? you put var1=val1&var2=val2& ....
For example, if your raw url (without the get parameters) is http://www.example.com/ and you have two parameters, param1=7 and param2=seven, then the full url should be:
http://www.example.com/?param1=7¶m2=seven.
If you want to generate the param1=7¶m2=seven in python, you can use a parameter dictionary, like this:
import urllib
parameters = urllib.urlencode({'param1':'7', 'param2':'seven'})
The value of parameters is 'param1=7¶m2=seven' so you can append the parameters string to a url like this:
raw_url = 'http://www.example.com/'
url = raw_url + '?' + params
Now the value of url is 'http://www.example.com/?param1=7¶m2=seven'.
I am fairly certain this has been asked many times before, but the query parameters start with ? and are separated by & like so:
http://www.site.com/script.php?key=value&var=num
http://www.foo.com/somepath?keyword1=value1&keyword2=value2&keyword3=value3
The requests module handles this pretty cute:
>>> import requests
>>> reply = requests.get('https://example.com', {'abc':1, 'def':'<?>'})
>>> reply.url
'https://example.com/?abc=1&def=%3C%3F%3E'