I'm looking for a way to build urls in python3 without having to do string concatenation. I get that I can
import requests
url_endpoint = 'https://www.duckduckgo.com'
mydict = {'q': 'whee! Stanford!!!', 'something': 'else'}
resp = requests.get(url_endpoint, params=mydict)
print(resp.url) # THIS IS EXACTLY WHAT I WANT
or
from requests import Request, Session
s = Session()
req = Request('GET', url, params={'q': 'blah'})
print(req.url)
# I didn't get this to work, but from the docs
# it should build the url without making the call
or
url = baseurl + "?" + urllib.urlencode(params)
I like that the request library intelligently decides to drop ? if it isn't needed, but that code actually makes a full GET request so instead of just building a full text url (which I plan to dump to an html tag). I am using django, but I didn't see anything to help with that in the core library.
Django comes with QueryDicts which basically do everything you want.
def make_url(url, args=None):
query = QueryDict(mutable=True)
query.update(args or {})
return '{}{}{}'.format(url, '?' if query else '', query.urlencode())
It supports multiple values per argument just like you can encounter in a url: example.com/foo?a=1&a=2&a=3.
Related
I am making my first API; any advice to improve my process is much appreciated.
I plan on passing JSON-like strings into the HTML request to this FastAPI microservice down there
#app.get("/create/{value}")
def createJSON(value:str):
person_json = value.strip()
fileName = person_json['Value']['0'] + person_json['Value']['1']
with open('%s.JSON','w') as writeFile:
writeFile.write(string)
return "Person has been created"
My HTTP request would look like this:
http://127.0.0.1:8000/create/{"Key":{"0":"name","1":"grad_year","2":"major","3":"quarter","4":"pronoun","5":"hobbies","6":"fun_fact","7":"food","8":"clubs","9":"res"},"Value":{"0":"adfasdfa","1":"adf'asd","2":"asd","3":"fads","4":"fa","5":"sdfa","6":"df","7":"asd","8":"fa","9":"df"}}
However, when doing this. The values passed are strings. Thus rendering the fileName portion of the code useless. How can I convert it to a Python dict? I have tried to use .strip(), but it did not help.
You're on the wrong track, Such a request should be essentially modeled as POST or a PUT request. That would allow you to send JSON in the body of the request and obtain it as a dict in python. You can see here
And even if you want to pass data in a GET request, there are query params
Coming back to the original doubt, you would have to use json.loads() to parse the json data and load it in a python dict then you can dump whatever file you like after that.
I'd recommend using the requests library
import requests
url = 'http://127.0.0.1:8000/create/'
params = dict(
name = 'Josh',
grad_year = '1987',
major = 'computer science',
quarter = '3'
)
resp = requests.get(url=url, params=params)
data = resp.json()
Then see here how to handle the JSON Response Content:
https://requests.readthedocs.io/en/master/user/quickstart/#json-response-content
The dict in the code I posted is different than the JSON you're trying to send through though. I assume you have a specific reason for having a "Key" array with the names than a "Value" array for the values of those specific names. But if not I'd recommend using a dictionary instead that way you can do things like:
fileName = person_json['name'] + person_json['grad-year']
Running a server on
> http://127.0.0.1:5000/
and trying to do a post request (the actual code is a bit more complicated but this is the part I cannot get working). Basically, trying to get something like the following to work but this returns and error saying its not found.
r = requests.post('http://127.0.0.1:5000/api/user')
I also tried something like the following but it returns:
url = url_for('api.userlistapi')
payload = {'email':email, 'password':password,'profile_verified':False}
r = requests.post(url)
self.prepare_url(url, params)
File "....appp/flask/lib/python2.7/site-packages/requests/models.py", line 360, in prepare_url
"Perhaps you meant http://{0}?".format(url))
MissingSchema: Invalid URL u'/api/user': No schema supplied. Perhaps you meant http:///api/user?
Any help would be appreciated! It might just be something dumb with routing or that a post request should be done differntly? I am also using angular and those requests to the same domain work.
You were on the right tracks with the second solution you tested.
You just have to add _external=True to url_for arguments:
url = url_for('api.userlistapi', _external=True)
payload = {'email':email, 'password':password,'profile_verified':False}
r = requests.post(url)
This way, Flask is able to construct a full url with domain included. Otherwise, url_for only builds a relative url meant to be called from within your domain.
--
Also, as a side note, you can pass your parameters directly with requests the following way:
r = requests.post(url, params=payload)
But it depends on other factors in the rest of your code, based on what you want to do.
I need to PUT data from a string (no dict) as the body of the call to a REST API.
When I call
r = requests.put(url, data = string)
then I can see in r.request.body after this call that it is None. Also, the server responds with a "411 Length Required" error.
However, when I try it with a dict instead of a string then it works, as the server responds with the correct JSON data. Moreover, in that case I can see in r.request.body the correct data.
Any ideas?
PS: I am using Python 2.7.3 and Python-requests 1.2.0
Even after three attempts to clarify the question, it still isn't clear what you're asking here, but I can try to throw out enough info that you can figure out the answer.
First, r = requests.put(url, data = string) returns a Response, which doesn't have a body, but it does have a request, and a history of 0 or more redirect requests, all of which are PreparedRequest objects, which do have body attributes.
On the other hand, if you did r.requests.Request(method='PUT', url=url, data=string), that would return a Request, which has to be prepare()d before it has a body.
Either way, if I do a simple test and look at the results, I find that the body is always correct:
>>> resp = requests.put('http://localhost/nosuchurl', data='abc')
>>> resp.request.body
'abc'
>>> resp = requests.put('http://localhost/redirect_to_https', data='abc')
>>> resp.history[-1].request.body
'abc'
>>> req = requests.Request(method='PUT', url='http://localhost/nosuchurl', data='abc')
>>> preq = req.prepare()
>>> preq.body
'abc'
My best guess is that you need to be looking at resp.history[0].request.body, but you're looking at resp.request.body, or something similar.
If Redirection and History in the quickstart tutorial doesn't help, read the detailed API docs, or just experiment with all of them until you figure it out.
Or do this:
resp = request.put('http://localhost/nosuchurl', data='abc', allow_redirects=False)
And then do the redirect-handling manually.
I created a chatbot that connects to a server and can read messages, now I'm at the point where I need to send messages, requiring request payload (according to the Network tab in Developer tools on google chrome). My opener consists of nothing but the following:
import urllib
import urllib2
from cookielib import CookieJar
self.cj = CookieJar()
self.opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(self.cj))
To stay and connected and read messages, I do the following, I do the following:
def connect(self,settings,xhr):
xhr_polling = self.get_code(xhr)
data = self.opener.open("http://chat2-1.wikia.com:80/socket.io/1/xhr-polling/" + xhr_polling + "?name=HairyBot&key=" +
settings['chatkey'] + "&roomId=" + str(settings['room']) + "&t=" + timestamp())
return data.read()
Settings consisting of the roomId and chatkey. The timestamp function creates a timestamp in accordance to what the servers needs (which isn't necessary to know for this question). Back to the question though, how can a payload be added to the opener to send a message to the chat?
As a suggestion, I recommend use the Requests library. It makes this stuff really simple:
import requests
session = requests.session() # For connection pooling
def connect(self,settings,xhr):
xhr_polling = self.get_code(xhr)
request = session.get('http://chat2-1.wikia.com:80/socket.io/1/xhr-polling/' + xhr_polling, params={
'name': 'HairyBot',
'key': settings['chatkey'],
'roomId': settings['room'],
't': timestamp()
})
return request.text
If you want to send a POST request instead, just change get to post and add some data:
def connect(self,settings,xhr):
xhr_polling = self.get_code(xhr)
request = session.post('http://chat2-1.wikia.com:80/socket.io/1/xhr-polling/' + xhr_polling, params={
'name': 'HairyBot',
'key': settings['chatkey'],
'roomId': settings['room'],
't': timestamp()
}, data={
'key': 'value'
})
return request.text
I'm not sure what you mean by "a payload", but presumably it's just another form variable named payload. If so, you send it the same way you do any other form variable, and you're already sending a bunch—roomId, t, etc.
One way of sending form variables is by URL-encoding them, tacking them onto the query string, and sending a GET request. That's what you're doing now. (It would be better to use proper urllib methods instead of hacking it together with string concatenation, but the end result is the same.)
The other way is by sending a POST body. The urllib2 documentation explains how to do this, and there are plenty of good examples online, but basically all you have to do is call urllib.urlencode() on your name-value pairs, then pass the result as the second argument (or as a keyword argument named data) to the open call.
In other words, something like this:
data = self.opener.open("http://chat2-1.wikia.com:80/socket.io/1/xhr-polling/" + xhr_polling,
urllib.urlencode(("name", "HairyBot"),
("key", settings['chatkey']),
("roomId", str(settings['room']),
("key", settings['chatkey']),
("t", timestamp()),
("payload", payload)))
Or, if you prefer, most servers will allow you to send some parameters on the query string and others in the POST data, so you can leave your existing code alone and just make one change:
data = self.opener.open("http://chat2-1.wikia.com:80/socket.io/1/xhr-polling/" + xhr_polling + "?name=HairyBot&key=" +
settings['chatkey'] + "&roomId=" + str(settings['room']) + "&t=" + timestamp(),
urllib.urlencode(("payload", payload)))
There's a lot of stuff out there on urllib2 and POST calls, but I'm stuck on a problem.
I'm trying to do a simple POST call to a service:
url = 'http://myserver/post_service'
data = urllib.urlencode({'name' : 'joe',
'age' : '10'})
content = urllib2.urlopen(url=url, data=data).read()
print content
I can see the server logs and it says that I'm doing GET calls, when I'm sending the data
argument to urlopen.
The library is raising an 404 error (not found), which is correct for a GET call, POST calls are processed well (I'm also trying with a POST within a HTML form).
Do it in stages, and modify the object, like this:
# make a string with the request type in it:
method = "POST"
# create a handler. you can specify different handlers here (file uploads etc)
# but we go for the default
handler = urllib2.HTTPHandler()
# create an openerdirector instance
opener = urllib2.build_opener(handler)
# build a request
data = urllib.urlencode(dictionary_of_POST_fields_or_None)
request = urllib2.Request(url, data=data)
# add any other information you want
request.add_header("Content-Type",'application/json')
# overload the get method function with a small anonymous function...
request.get_method = lambda: method
# try it; don't forget to catch the result
try:
connection = opener.open(request)
except urllib2.HTTPError,e:
connection = e
# check. Substitute with appropriate HTTP code.
if connection.code == 200:
data = connection.read()
else:
# handle the error case. connection.read() will still contain data
# if any was returned, but it probably won't be of any use
This way allows you to extend to making PUT, DELETE, HEAD and OPTIONS requests too, simply by substituting the value of method or even wrapping it up in a function. Depending on what you're trying to do, you may also need a different HTTP handler, e.g. for multi file upload.
This may have been answered before: Python URLLib / URLLib2 POST.
Your server is likely performing a 302 redirect from http://myserver/post_service to http://myserver/post_service/. When the 302 redirect is performed, the request changes from POST to GET (see Issue 1401). Try changing url to http://myserver/post_service/.
Have a read of the urllib Missing Manual. Pulled from there is the following simple example of a POST request.
url = 'http://myserver/post_service'
data = urllib.urlencode({'name' : 'joe', 'age' : '10'})
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)
print response.read()
As suggested by #Michael Kent do consider requests, it's great.
EDIT: This said, I do not know why passing data to urlopen() does not result in a POST request; It should. I suspect your server is redirecting, or misbehaving.
The requests module may ease your pain.
url = 'http://myserver/post_service'
data = dict(name='joe', age='10')
r = requests.post(url, data=data, allow_redirects=True)
print r.content
it should be sending a POST if you provide a data parameter (like you are doing):
from the docs:
"the HTTP request will be a POST instead of a GET when the data parameter is provided"
so.. add some debug output to see what's up from the client side.
you can modify your code to this and try again:
import urllib
import urllib2
url = 'http://myserver/post_service'
opener = urllib2.build_opener(urllib2.HTTPHandler(debuglevel=1))
data = urllib.urlencode({'name' : 'joe',
'age' : '10'})
content = opener.open(url, data=data).read()
Try this instead:
url = 'http://myserver/post_service'
data = urllib.urlencode({'name' : 'joe',
'age' : '10'})
req = urllib2.Request(url=url,data=data)
content = urllib2.urlopen(req).read()
print content
url="https://myserver/post_service"
data["name"] = "joe"
data["age"] = "20"
data_encoded = urllib2.urlencode(data)
print urllib2.urlopen(url + "?" + data_encoded).read()
May be this can help