I'm trying to download data from an external API. There's going to be a lot of downloads, so I want to use pipelines for easier parallelization. The way the API is set up, I can make a request to start a download job, and pass a postback url in that request. When the download job finishes, their API sends a POST to the given url. I want to do the following:
class DownloadPipeline(pipeline.Pipeline):
async = True
public_callbacks = True
def run(self, filename):
postback = self.get_callback_url()
# make API request with postback as a param
def callback(self):
# Read data from the POST
However, all the docs I've read online only have examples of GET requests on the callback url, where data is passed through a query string on the URL. Is there a way to read POST data instead?
Looks like both the POST and GET both call over to run_callback() ... so you should be able to do either
Related
I have a Flask app that generates video stream links. It connects to a server using login credentials and grabs a one time use link (that expires when a new link is generated using the same credentials). Using a list of credentials I am able to stream to as many devices as I like, so long as I have enough accounts.
The issue I am having is that one of the clients doesn't like the way the stream is returned.
#app.route("/play", methods=["GET"])
def play():
def streamData():
try:
useAccount(<credentials>)
with requests.get(link, stream=True) as r:
for chunk in r.iter_content(chunk_size=1024):
yield chunk
except:
pass
finally:
freeAccount(<credentials>)
...
# return redirect(link)
return Response(streamData())
If I return a redirect then there are no playback issues at all. The problem with a redirect is I don't have a way of marking the credentials as in use, then freeing them after.
The problem client is TVHeadend. I am able to get it to work by enabling the additional avlib inside of TVHeadend... But I shouldn't have to do that. I don't have to when I return a redirect.
What could be the cause of this?
Is it possible to make my app respond in the same way as the links server does?
My guess is that TVHeadend is very strict on if something complies to whatever standards... and I am guessing my app doesn't?
I'd like to do a load test over my NLP web app using Locust. The website is simple where users only need to pass in their text and its language type, and then the results will show on the next page.
Therefore, I want to make my "locusts" to first pass in two values on the index page and then go to the corresponding page, which is supposed to be quick and easy. But my code doesn't work and the error message wrote there was something wrong with GET POST method (HTTP 405 & HTTP 500 error). Can anyone help me check my code?
from locust import HttpUser, TaskSet, between, task, SequentialTaskSet
class UserBehavior(SequentialTaskSet):
#task
def submit(self):
self.client.post('/', {'text': 'Kobe Bryant is the best NBA player.', 'language': 'en'})
#task
def get_boto(self):
self.client.get('/boto')
class WebsiteUser(HttpUser):
tasks = [UserBehavior]
wait_time = between(1, 2)
May be before post request with data user must login?
class UserBehavior(SequentialTaskSet):
def on_start(self):
self.client.post("/login", {
"username": "test_user",
"password": ""
})
A 405 sounds like you might not be sending your requests in the right way, maybe using the wrong verb. I would use a debug proxy (like Charles Proxy) to record the traffic to your site while you go through your flow so you know how the requests are structured and expected to be. Then I would create a new separate Python file and try to use Requests to get something working successfully (check out Python : Trying to POST form using requests for a POST example). Locust's HttpUser.client is based on Requests so once you get the separate Python file working and doing what you want, you should be able to drop in the same calls and replace request or session with self.client and it should work.
I have a Django project that contains a second, third-party Django app. In one of my views, I need to query a view from the other app (within the same Django project).
Currently, it works something like this:
import requests
def my_view(request):
data = requests.get('http://localhost/foo/bar').json()
# mangle data
return some_response
Is there a clean way to send the request without going all the way through DNS and the webserver and just go directly to the other app's view (ideally going through the middleware as well of course)
A Django view function accepts a request and returns a response object. There is no reason that a view cannot invoke another view by constructing a request (or cloning or passing its own) and interpreting the response. (c.f. the testing framework).
Of course, if the other view has undesirable side-effects, then the controlling view will have to unwind them. Working within a transaction should allow it to delve in the results of the view it invoked, and then abort the transaction and perform its own.
You can use urllib as shown below
import urllib, urllib3, urllib.request
url = "http://abcd.com" # API URL
postdata = urllib.parse.urlencode(values).encode("utf-8")
req = urllib.request.Request(url)
with urllib.request.urlopen(req, data=postdata) as response:
resp = response.read()
print(resp)
this is a two-part question: I have seen individual pieces discussed, but can't seem to get the recommended suggestions to work together. I want to create a web service to store images and their metadata passed from a caller and run a test call from Postman to make sure it is working. So to pass an image (Drew16.jpg) to the web service via Postman, it appears I need something like this:
For the web service, I have some python/flask code to read the request (one of many variations I have tried):
from flask import Flask, jsonify, request, render_template
from flask_restful import Resource, Api, reqparse
...
def post(self, name):
request_data = request.get_json()
userId = request_data['UserId']
type = request_data['ImageType']
image = request.files['Image']
Had no problem with the data portion and straight JSON but adding the image has been a bugger. Where am I going wrong on my Postman config? What is the actual set of Python commands for reading the metadata and the file from the post? TIA
Pardon the almost blog post. I am posting this because while you can find partial answers in various places, I haven't run across a complete post anywhere, which would have saved me a ton of time. The problem is you need both sides to the story in order to verify either.
So I want to send a request using Postman to a Python/Flask web service. It has to have an image along with some metadata.
Here are the settings for Postman (URL, Headers):
And Body:
Now on to the web service. Here is a bare bones service which will take the request, print the metadata and save the file:
from flask import Flask, request
app = Flask(__name__)
# POST - just get the image and metadata
#app.route('/RequestImageWithMetadata', methods=['POST'])
def post():
request_data = request.form['some_text']
print(request_data)
imagefile = request.files.get('imagefile', '')
imagefile.save('D:/temp/test_image.jpg')
return "OK", 200
app.run(port=5000)
Enjoy!
Make sure `request.files['Image'] contains the image you are sending and follow http://flask.pocoo.org/docs/1.0/patterns/fileuploads/ to save the file to your file system. Something like
file = request.files['Image']
file.save('./test_image.jpg')
might do what you want, while you will have to work out the details of how the file should be named and where it should be placed.
I'm building a small tool (no UI) that will accept webhooks from several services, reformat the content, and then send standardized content to another webhook.
Example: Stripes webhook has lots of raw data, but I want to ping another webhook with a summary. So I want to take the raw data stripe sends me, reformat it to a simple string, and then send to another webhook.
I imagine the script doing this:
# 1. receive webhook sent to URL with ?service=stripe param
# 2. parse URL to get service param
# 3. parse data received in payload
# 4. use some of the data received in new string
# 5. send string to new webhook
I'd love to host this on GAE. I've built lots of projects with Django, but since this doesn't need a UI or database that seems heavy. I'd love any help. I've got the GAE project set up, and this going:
import web #using web.py
import logging
urls = (
"/.*", "hooks",
)
app = web.application(urls, globals())
class hooks:
# 1. DONE receive webhook sent to URL with ?service=stripe param
def POST(self):
# 2. parse URL to get service param
# ...
# service_name = [parsed service name]
# 3. DONE parse data received in payload
data = web.data()
logging.info(data)
# 4. DONE use some of the data received in new string
# (I've got the reformatting scripts already written)
# data_to_send = reformat(data, service_name)
# 5. send data_to_send as payload to new webhook
# new_webhook_url = 'http://example.com/1823123/'
# CURL for new webhook is: curl -X POST -H 'Content-Type: application/json' 'http://example.com/1823123/' -d '{"text": data_to_send}'
return 'OK'
app = app.gaerun()
So on GAE, is there a preferred method for (2) parsing incoming URL and (5) sending a webhook?
I'm not familiar with web.py. Many GAE apps are based on webapp2.
For parsing URLs with webapp2, you create routes. Here is a simple route that I created for processing PayPal IPNs:
(r'/ipn/(\S+)/(\w+)', website.ProcessIPN)
I then have a handler that processes this route:
class ProcessIPN(webapp2.RequestHandler):
def post(self, user, secret):
payload = json.loads(self.request.body)
...
The route is a regular expression that captures two parts of the URL and these are passed as the two parameters of the handler (user and secret). Assuming your payload is JSON so you can easily get it with webapp2 as indicated above.
For sending a webhook, you need to use urlfetch.
Given how simple your use case is, I recommend not using web.py and instead using webapp2.