We Keep Coding

Python Flask: How to wait for webhook to be executed?

I am working on a Python flask app, and the main method start() calls an external API (third_party_api_wrapper()). That external API has an associated webhook (webhook()) that receives the output of that external API call (note that the output that webhook() receives is actually different from the response returned in the third_party_wrapper())
The main method start() needs the result of webhook(). How do I make start() wait for webhook() to be executed? And how do wo pass the returned value of webhook() back to start()?
Here's is a minimal code snippet to capture the scenario.
#app.route('/webhook', methods=['POST'])
def webhook():
return "webhook method has executed"
# this method has a webhook that calls webhook() after this method has executed
def third_party_api_wrapper():
url = 'https://api.thirdparty.com'
response = requests.post(url)
return response
# this is the main entry point
#app.route('/start', methods=['POST'])
def start():
third_party_api_wrapper()
# The rest of this code depends on the output of webhook().
# How do we wait until webhook() is called, and how do we access the returned value?

The answer to this question really depends on how you plan on running your app in production. It's much simpler if we make the assumption that you only plan to have a single instance of your app running at once (as opposed to multiple behind a load balancer, for example), so I'll make that assumption first to give you a place to start, and comment on a more "production-ready" solution afterwards.
A big thing to keep in mind when writing a web application is that you have to understand how you want the outside world to interact with your app. Do you expect to have the /start endpoint called only once at the beginning of your app's lifetime, or is this a generic endpoint that may start any number of background processes that you want the caller of each to wait for? Or, do you want the behavior where any caller after the first one will wait for the same process to complete as the first one? I can't answer these questions for you, it depends on the use-case you're trying to implement. I'll give you a relatively simple solution that you should be able to modify to fulfill any of the ones I mentioned though.
This solution will use the Event class from the threading standard library module; I added some comments to clarify which parts you may have to change depending on the specifics of the API you're calling and stuff like that.
import threading
import uuid
from typing import Any
import requests
from flask import Flask, Response, request
# The base URL for your app, if you're running it locally this should be fine
# however external providers can't communicate with your `localhost` so you'll
# need to change this for your app to work end-to-end.
BASE_URL = "http://localhost:5000"
app = Flask(__name__)
class ThirdPartyProcessManager:
def __init__(self) -> None:
self.events = {}
self.values = {}
def wait_for_request(self, request_id: str) -> None:
event = threading.Event()
actual_event = self.events.setdefault(request_id, event)
if actual_event is not event:
raise ValueError(f"Request {request_id} already exists.")
event.wait()
return self.values.pop(request_id)
def finish_request(self, request_id: str, value: Any) -> None:
event = self.events.pop(request_id, None)
if event is None:
raise ValueError(f"Request {request_id} does not exist.")
self.values[request_id] = value
event.set()
MANAGER = ThirdPartyProcessManager()
# This is assuming that you can specify the callback URL per-request, otherwise
# you may have to get the request ID from the body of the request or something
#app.route('/webhook/<request_id>', methods=['POST'])
def webhook(request_id: str) -> Response:
MANAGER.finish_request(request_id, request.json)
return "webhook method has executed"
# Somehow in here you need to create or generate a unique identifier for this
# request--this may come from the third-party provider, or you can generate one
# yourself. There are three main paths I see here:
# - If you can specify the callback/webhook URL in each call, you can just pass them
# <base>/webhook/<request_id> and use that to identify which request is being
# responded to in the webhook.
# - If the provider gives you a request ID, you can return it from this function
# then retrieve it from the request body in the webhook route
# For now, I'll assume the first situation but you should be able to implement the second
# with minimal changes
def third_party_api_wrapper() -> str:
request_id = uuid.uuid4().hex
url = 'https://api.thirdparty.com'
# Just an example, I don't know how the third party API you're working with works
response = requests.post(
url,
json={"callback_url": f"{BASE_URL}/webhook/{request_id}"}
)
# NOTE: unrelated to the problem at hand, you should always check for errors
# in HTTP responses. This method is an easy way provided by requests to raise
# for non-success status codes.
response.raise_for_status()
return request_id
#app.route('/start', methods=['POST'])
def start() -> Response:
request_id = third_party_api_wrapper()
result = MANAGER.wait_for_request(request_id)
return result
If you want to run the example fully locally to test it, do the following:
Comment out lines 62-71, which actually make the external API call
Add a print statement after line 77, so that you can get the ID of the "in flight" request. E.g. print("Request ID", request_id)
In one terminal, run the app by pasting the above code into an app.py file and running flask run in that directory.
In another terminal, start the process via:
curl -XPOST http://localhost:5000/start
Copy the request ID that will be logged in the first terminal that's running the server.
In a third terminal, complete the process by calling the webhook:
curl -XPOST http://localhost:5000/webhook/<your_request_id> -H Content-Type:application/json -d '{"foo":"bar"}'
You should see {"foo":"bar"} as the response in the second terminal that made the /start request.
I hope that's enough to help you get started w/ whatever problem you're trying to solve.
There are a couple of design-y comments I have based on the information provided as well:
As I mentioned before, this will not work if you have more than one instance of the app running at once. This works by storing the state of in-flight requests in a global state inside your python process, so if you have more than one process, they won't all be working and modifying the same state. If you need to run more than one instance of your process, I would use a similar approach with some database backend to store the shared state (assuming your requests are pretty short-lived, Redis might be a good choice here, but once again it'll depend on exactly what you're trying to do).
Even if you do only have one instance of the app running, flask is capable of being run in a variety of different server contexts--for example, the server might be using threads (the default), greenlets via gevent or a similar library, or multiple processes, or maybe some other approach entirely in order to handle multiple requests concurrently. If you're using an approach that creates multiple processes, you should be able to use the utilities provided by the multiprocessing module to implement the same approach as I've given above.
This approach probably will work just fine for something where the difference in time between the API call and the webhook response is small (on the order of a couple of seconds at most I'd say), but you should be wary of using this approach for something where the difference in time can be quite large. If the connection between the client and your server fails, they'll have to make another request and run the long-running process that your third party is completing for you again. Some proxies and load balancers may also have time out behavior that could terminate the request after a certain amount of time even if nothing goes wrong in the connection between your server and the client making a request to it. An alternative approach would be for your /start endpoint to return quickly and give the client a request_id that they could poll for updates. As an example, AWS Athena's API is structured like this--there is a StartQueryExecution method, and separate GetQueryExecution and GetQueryResults methods that the client makes requests to check the status of a query and retrieve the results respectively (there are also other methods like StopQueryExecution and GetQueryRuntimeStatistics available as well). You can check out the documentation here.
I know that's a lot of info, but I hope it helps. Happy to update the answer w/ more specific info if you'll provide some more details about your use-case.

Triggering Zap A by passing a POST function from Zap B

Using Zapier, I'm trying to set up a Zap that will run once a day and go through a spreadsheet to pick up anything with a specific keyword and add it to another sheet, remove it from the original sheet, and rinse and repeat. I'm using Daily Schedule trigger to run some Python code with a Post function to send the webhook. The URL links to another Zap with a Catch Hook trigger, which should execute when that webhook comes in to run through the steps to adjust the spreadsheets, and then concludes with an A/B path which will repeat the webhook if there are still more instances of the keyword to move/delete. However, my Zap is not triggering to the webhook, and I'm not sure why. I'm very new to webhooks and been unable to find an answer on my own.
import requests
hookUrl = 'https://hooks.zapier.com/hooks/catch/123456/1abcd2/'
id = input.get('value')
response = requests.post(hookUrl, id)
id = int(id)+1
return {'value': id}
I would expect the code to execute a post to the webhook URL and trigger the Zap, but the Zap does not react. The code executes successfully otherwise with returning the incremented ID.
Any insight?

Here's what worked:
import requests
hookUrl = 'https://hooks.zapier.com/hooks/catch/123456/1abcd2/'
payload = {'id': input.get('value')}
r = requests.post(hookUrl, data=payload)
id = int(input.get('value'))+1
return {'value': id}
The error was in not defining a key for the variable now called payload. Once the key was added to keep the JSON formatting (thank you, xavdid!), it worked like a charm. Now, I have a Zap that sends out this webhook every day at midnight, which runs a DIFFERENT Zap with a final pathing step which either a.) terminates the Zap or b.) sends a webhook and loops the Zap again, depending on whether any entries in a Spreadsheet with a given variable still exist.
Thanks for the help!

How to delete context/session_id at end of conversation in Wit.ai bot

I've been having issues with Wit.ai where my Python bot will retain the context after ending a conversation. This behaviour is the same in the Facebook client and the pywit interactive client.
The conversation starts with a simple 'Hi' and can end at different points within different branches if a user taps a 'Thanks, bye' quick reply after a successful query.
If the conversation is then started with 'Hi' once again, the session state is saved from before which leads to wrong responses. What is the best way to delete the context after the user has said goodbye?
I tried creating a goodbye function that triggers after the bot has sent its final message but it didn't work e.g.
def goodbye(request):
del request['context'] # or request.clear()
return request
The documentation (https://wit.ai/docs/http/20160526#post--converse-link) suggests you clear the session_id and generate a new one but gives no hints as to how.

You can generate new Session Ids using uuid. Session ID has to be any text that is unique, it can even be system date. I suggest you use uuid
Check here as to how to generate it.

I was confronted with the same issue and I solved it in the following way.
I first created a simple end_session action, to be called at the end of each conversation path:
def end_session(request):
return {'end_session': True}
Then I inserted the following code just after returning from run_actions:
if 'end_session' in context:
context = {}
session_hash = uuid.uuid1().hex
As you see, in addition to clearing the context, as you do, I also recreate a new session id (as per Swapnesh Khare's suggestion).
I'm not sure this is the best solution, but it works for me.

A way to view non-standard message headers after email sent with Mailgun?

I'm using Mailgun's RESTful API. I need a way to view non-standard headers of messages that have already been sent to connect them with a certain ID I have for database purposes. The main purpose of this is to log what emails are failing to be sent or not, along with the reasons and such.
My python code keeps returning a 405 response everytime I attempt to call something from "https://api.mailgun.net/v3/MY_DOMAIN_NAME/messages" when it is GET instead of POST. I am able to make other GET calls with events and stats as opposed to messages, so I do not think its the way I am writing the code exactly.
By calling on events I am able to view message status information, error codes, and ID's different from the ones I need. If I can just get this one non-standard header I will be set since it will give me the ID I need.
So, to the point, my question is if anyone who is familiar with this API knows if what I am trying to do is possible, and if so what should I look to be doing?
Here is an example of a call I make in case that helps:
This doesn't work:
def get_failed_events(start_date, end_date):
return requests.get(api_url.format('messages'), auth=('api', key), params={ 'begin': start_date, 'end': end_date}, verify=False)
This does:
def get_failed_events(start_date, end_date):
return requests.get(api_url.format('events'), auth=('api', key), params={ 'begin': start_date, 'end': end_date}, verify=False)

I found out that even though I was passing the information, I wasn't exactly passing it the way mailgun wants. It needs to be something like this c# example:
var forMessageLog = "{\"x-header-id\": 1232}";
message.Headers.Add("X-Mailgun-Variables", forMessageLog);
Otherwise it will not appear in user variables.

Jabber bot - how to get the availability of contacts?

I need to set up a jabber bot, using python, that will send messages based on the online/offline availability of several contacts.
I've been looking into pyxmpp and xmpppy, but couldn't find any way (at least nothing straightforward) to check the status of a given contact.
Any pointers on how to achieve this?
Ideally I would like something like e.g. bot.status_of("contact1#gmail.com") returning "online"

I don't think it is possible in the way you want it because the presence of contacts (which contains the information about their availability) is received asynchronously by the bot.
You will have to write a presence handler function and registered it with the connection. This function will get called whenever a presence is received from a contact. The parameter of the call will tell you if the contact is online or not. Depending upon it you can send the message to the contact.
Using xmpppy you do it something like this:
def connect(jid, password, res, server, proxy, use_srv):
conn = xmpp.Client(jid.getDomain())
if not conn.connect(server=server, proxy=proxy, use_srv=use_srv):
log( 'unable to connect to server.')
return None
if not conn.auth(jid.getNode(), password, res):
log( 'unable to authorize with server.')
return None
conn.RegisterHandler( 'presence', callback_presence)
return conn
conn = connect(...)
def callback_presence(sess, pres):
if pres.getStatus() == "online":
msg = xmpp.Message(pres.getFrom(), "Hi!")
conn.send(msg)
PS: I have not tested the code but it should be something very similar to this.

What you want is done via a <presence type="probe"/>. This is done on behalf of the client, and SHOULD not be done by them (as per the RFC for XMPP IM). Since this is a bot, you could implement the presence probe, and receive the current presence of a given entity. Remember to send the probe to the bare JID (sans resource), because the server responds on behalf of clients for presence probes. This means your workflow will look like:
<presence/> // I'm online! BOT
<presence from="juliet#capulet.org/balcony"/> RESPONSE
<presence from="romeo#montague.net/hallway"/> // and so on... RESPONSE
<presence type="probe" to="benvolio#montague.net"/> BOT
<presence from="benvoio#montague.net/hallway"> RESPONSE
<status>Huzzah!</status>
<priority>3</priority>
</presence>
Take a look at that portion of the RFC for more in depth information on how your call flow should behave.

What you need to do is:
Connect.
Declare a presence handler. That handler maintains a cache of every contact's presence (see details below)
Send initial presence to the server, which will provoke the reception of presence statuses from all of your online contacts, which will in turn trigger the handler.
The status_of() method reads the cache and deduces the contact's presence status immediately.
Now, it depends on what presence information you need. For the sake of simplicity, let's pretend you just need an "online"/"offline" value. The cache would be a dictionary whose keys are the bare (no resource) JIDs, and the values are a set of connected resources for this JID. For example:
{'foo#bar.com': set(['work', 'notebook']), 'bob#example.net': set(['gtalk'])}
Now, when you receive an "available" presence from a certain JID/resource:
if jid not in cache:
cache[jid] = set()
cache[jid].add(resource)
Reciprocally, when you receive an "unavailable" presence:
if jid in cache: # bad people send "unavailable" just to make your app crash
cache[jid].discard(resource)
if 0 == len(cache[jid]):
del cache[jid]
And now:
def is_online(jid):
return jid in cache
Of course, if you want more detailed information, you could maintain not only the list of available resources for a contact but also the status, status message, priority, etc. of each resource.