Using Zapier, I'm trying to set up a Zap that will run once a day and go through a spreadsheet to pick up anything with a specific keyword and add it to another sheet, remove it from the original sheet, and rinse and repeat. I'm using Daily Schedule trigger to run some Python code with a Post function to send the webhook. The URL links to another Zap with a Catch Hook trigger, which should execute when that webhook comes in to run through the steps to adjust the spreadsheets, and then concludes with an A/B path which will repeat the webhook if there are still more instances of the keyword to move/delete. However, my Zap is not triggering to the webhook, and I'm not sure why. I'm very new to webhooks and been unable to find an answer on my own.
import requests
hookUrl = 'https://hooks.zapier.com/hooks/catch/123456/1abcd2/'
id = input.get('value')
response = requests.post(hookUrl, id)
id = int(id)+1
return {'value': id}
I would expect the code to execute a post to the webhook URL and trigger the Zap, but the Zap does not react. The code executes successfully otherwise with returning the incremented ID.
Any insight?
Here's what worked:
import requests
hookUrl = 'https://hooks.zapier.com/hooks/catch/123456/1abcd2/'
payload = {'id': input.get('value')}
r = requests.post(hookUrl, data=payload)
id = int(input.get('value'))+1
return {'value': id}
The error was in not defining a key for the variable now called payload. Once the key was added to keep the JSON formatting (thank you, xavdid!), it worked like a charm. Now, I have a Zap that sends out this webhook every day at midnight, which runs a DIFFERENT Zap with a final pathing step which either a.) terminates the Zap or b.) sends a webhook and loops the Zap again, depending on whether any entries in a Spreadsheet with a given variable still exist.
Thanks for the help!
Related
I am working on a Python flask app, and the main method start() calls an external API (third_party_api_wrapper()). That external API has an associated webhook (webhook()) that receives the output of that external API call (note that the output that webhook() receives is actually different from the response returned in the third_party_wrapper())
The main method start() needs the result of webhook(). How do I make start() wait for webhook() to be executed? And how do wo pass the returned value of webhook() back to start()?
Here's is a minimal code snippet to capture the scenario.
#app.route('/webhook', methods=['POST'])
def webhook():
return "webhook method has executed"
# this method has a webhook that calls webhook() after this method has executed
def third_party_api_wrapper():
url = 'https://api.thirdparty.com'
response = requests.post(url)
return response
# this is the main entry point
#app.route('/start', methods=['POST'])
def start():
third_party_api_wrapper()
# The rest of this code depends on the output of webhook().
# How do we wait until webhook() is called, and how do we access the returned value?
The answer to this question really depends on how you plan on running your app in production. It's much simpler if we make the assumption that you only plan to have a single instance of your app running at once (as opposed to multiple behind a load balancer, for example), so I'll make that assumption first to give you a place to start, and comment on a more "production-ready" solution afterwards.
A big thing to keep in mind when writing a web application is that you have to understand how you want the outside world to interact with your app. Do you expect to have the /start endpoint called only once at the beginning of your app's lifetime, or is this a generic endpoint that may start any number of background processes that you want the caller of each to wait for? Or, do you want the behavior where any caller after the first one will wait for the same process to complete as the first one? I can't answer these questions for you, it depends on the use-case you're trying to implement. I'll give you a relatively simple solution that you should be able to modify to fulfill any of the ones I mentioned though.
This solution will use the Event class from the threading standard library module; I added some comments to clarify which parts you may have to change depending on the specifics of the API you're calling and stuff like that.
import threading
import uuid
from typing import Any
import requests
from flask import Flask, Response, request
# The base URL for your app, if you're running it locally this should be fine
# however external providers can't communicate with your `localhost` so you'll
# need to change this for your app to work end-to-end.
BASE_URL = "http://localhost:5000"
app = Flask(__name__)
class ThirdPartyProcessManager:
def __init__(self) -> None:
self.events = {}
self.values = {}
def wait_for_request(self, request_id: str) -> None:
event = threading.Event()
actual_event = self.events.setdefault(request_id, event)
if actual_event is not event:
raise ValueError(f"Request {request_id} already exists.")
event.wait()
return self.values.pop(request_id)
def finish_request(self, request_id: str, value: Any) -> None:
event = self.events.pop(request_id, None)
if event is None:
raise ValueError(f"Request {request_id} does not exist.")
self.values[request_id] = value
event.set()
MANAGER = ThirdPartyProcessManager()
# This is assuming that you can specify the callback URL per-request, otherwise
# you may have to get the request ID from the body of the request or something
#app.route('/webhook/<request_id>', methods=['POST'])
def webhook(request_id: str) -> Response:
MANAGER.finish_request(request_id, request.json)
return "webhook method has executed"
# Somehow in here you need to create or generate a unique identifier for this
# request--this may come from the third-party provider, or you can generate one
# yourself. There are three main paths I see here:
# - If you can specify the callback/webhook URL in each call, you can just pass them
# <base>/webhook/<request_id> and use that to identify which request is being
# responded to in the webhook.
# - If the provider gives you a request ID, you can return it from this function
# then retrieve it from the request body in the webhook route
# For now, I'll assume the first situation but you should be able to implement the second
# with minimal changes
def third_party_api_wrapper() -> str:
request_id = uuid.uuid4().hex
url = 'https://api.thirdparty.com'
# Just an example, I don't know how the third party API you're working with works
response = requests.post(
url,
json={"callback_url": f"{BASE_URL}/webhook/{request_id}"}
)
# NOTE: unrelated to the problem at hand, you should always check for errors
# in HTTP responses. This method is an easy way provided by requests to raise
# for non-success status codes.
response.raise_for_status()
return request_id
#app.route('/start', methods=['POST'])
def start() -> Response:
request_id = third_party_api_wrapper()
result = MANAGER.wait_for_request(request_id)
return result
If you want to run the example fully locally to test it, do the following:
Comment out lines 62-71, which actually make the external API call
Add a print statement after line 77, so that you can get the ID of the "in flight" request. E.g. print("Request ID", request_id)
In one terminal, run the app by pasting the above code into an app.py file and running flask run in that directory.
In another terminal, start the process via:
curl -XPOST http://localhost:5000/start
Copy the request ID that will be logged in the first terminal that's running the server.
In a third terminal, complete the process by calling the webhook:
curl -XPOST http://localhost:5000/webhook/<your_request_id> -H Content-Type:application/json -d '{"foo":"bar"}'
You should see {"foo":"bar"} as the response in the second terminal that made the /start request.
I hope that's enough to help you get started w/ whatever problem you're trying to solve.
There are a couple of design-y comments I have based on the information provided as well:
As I mentioned before, this will not work if you have more than one instance of the app running at once. This works by storing the state of in-flight requests in a global state inside your python process, so if you have more than one process, they won't all be working and modifying the same state. If you need to run more than one instance of your process, I would use a similar approach with some database backend to store the shared state (assuming your requests are pretty short-lived, Redis might be a good choice here, but once again it'll depend on exactly what you're trying to do).
Even if you do only have one instance of the app running, flask is capable of being run in a variety of different server contexts--for example, the server might be using threads (the default), greenlets via gevent or a similar library, or multiple processes, or maybe some other approach entirely in order to handle multiple requests concurrently. If you're using an approach that creates multiple processes, you should be able to use the utilities provided by the multiprocessing module to implement the same approach as I've given above.
This approach probably will work just fine for something where the difference in time between the API call and the webhook response is small (on the order of a couple of seconds at most I'd say), but you should be wary of using this approach for something where the difference in time can be quite large. If the connection between the client and your server fails, they'll have to make another request and run the long-running process that your third party is completing for you again. Some proxies and load balancers may also have time out behavior that could terminate the request after a certain amount of time even if nothing goes wrong in the connection between your server and the client making a request to it. An alternative approach would be for your /start endpoint to return quickly and give the client a request_id that they could poll for updates. As an example, AWS Athena's API is structured like this--there is a StartQueryExecution method, and separate GetQueryExecution and GetQueryResults methods that the client makes requests to check the status of a query and retrieve the results respectively (there are also other methods like StopQueryExecution and GetQueryRuntimeStatistics available as well). You can check out the documentation here.
I know that's a lot of info, but I hope it helps. Happy to update the answer w/ more specific info if you'll provide some more details about your use-case.
I'm working on a django application which reads csv file from dropbox, parse data and store it in database. For this purpose I need background task which checks if the file is modified or changed(updated) and then updates database.
I've tried 'Celery' but failed to configure it with django. Then I find django-background-tasks which is quite simpler than celery to configure.
My question here is how to initialize repeating tasks?
It is described in documentation
but I'm unable to find any example which explains how to use repeat, repeat_until or other constants mentioned in documentation.
can anyone explain the following with examples please?
notify_user(user.id, repeat=<number of seconds>, repeat_until=<datetime or None>)
repeat is given in seconds. The following constants are provided:
Task.NEVER (default), Task.HOURLY, Task.DAILY, Task.WEEKLY,
Task.EVERY_2_WEEKS, Task.EVERY_4_WEEKS.
You have to call the particular function (notify_user()) when you really need to execute it.
Suppose you need to execute the task while a request comes to the server, then it would be like this,
#background(schedule=60)
def get_csv(creds):
#read csv from drop box with credentials, "creds"
#then update the DB
def myview(request):
# do something with my view
get_csv(creds, repeat=100)
return SomeHttpResponse
Excecution Procedure
1. Request comes to the url hence it would dispatch to the corresponding view, here myview()
2. Excetes the line get_csv(creds, repeat=100) and then creates a async task in DB (it wont excetute the function now)
3. Returning the HTTP response to the user.
After 60 seconds from the time which the task creation, get_csv(creds) will excecutes repeatedly in every 100 seconds
For example, suppose you have the function from the documentation
#background(schedule=60)
def notify_user(user_id):
# lookup user by id and send them a message
user = User.objects.get(pk=user_id)
user.email_user('Here is a notification', 'You have been notified')
Suppose you want to repeat this task daily until New Years day of 2019 you would do the following
import datetime
new_years_2019 = datetime.datetime(2019, 01, 01)
notify_user(some_id, repeat=task.DAILY, repeat_until=new_years_2019)
I've been having issues with Wit.ai where my Python bot will retain the context after ending a conversation. This behaviour is the same in the Facebook client and the pywit interactive client.
The conversation starts with a simple 'Hi' and can end at different points within different branches if a user taps a 'Thanks, bye' quick reply after a successful query.
If the conversation is then started with 'Hi' once again, the session state is saved from before which leads to wrong responses. What is the best way to delete the context after the user has said goodbye?
I tried creating a goodbye function that triggers after the bot has sent its final message but it didn't work e.g.
def goodbye(request):
del request['context'] # or request.clear()
return request
The documentation (https://wit.ai/docs/http/20160526#post--converse-link) suggests you clear the session_id and generate a new one but gives no hints as to how.
You can generate new Session Ids using uuid. Session ID has to be any text that is unique, it can even be system date. I suggest you use uuid
Check here as to how to generate it.
I was confronted with the same issue and I solved it in the following way.
I first created a simple end_session action, to be called at the end of each conversation path:
def end_session(request):
return {'end_session': True}
Then I inserted the following code just after returning from run_actions:
if 'end_session' in context:
context = {}
session_hash = uuid.uuid1().hex
As you see, in addition to clearing the context, as you do, I also recreate a new session id (as per Swapnesh Khare's suggestion).
I'm not sure this is the best solution, but it works for me.
Using the code below I'm sending an email on error. I'm trying to include a link to the Cloud Console logs in the email but the request ID seems to be wrong about 30% of the time.
If I find the request with the wrong ID it's always almost a perfect match except the last three characters are 0 (in the Stackdriver console) instead of 101 (returned from the env variable), always the same substitution - is this a bug with cloud console or am I trying to use these IDs wrong?
The code (stripped down version):
class ErrorAlertMiddleware(object):
def process_response(self, request, response):
if response.status_code == 500:
logger.info(os.environ.get('REQUEST_LOG_ID'))
msg = 'Link to logs: https://console.cloud.google.com/logs/viewer?' + '&'.join((
'project=%s' % MY_APP_ID,
'expandAll=true',
'filters=request_id:%s' % os.environ.get('REQUEST_LOG_ID'),
'resource=gae_app',
))
# this is a utility func that simply sends email
sendemail(ERROR_RECIPIENT, msg)
return response
Note I've also logged the REQUEST_LOG_ID to ensure it's not being encoded or something and the log output matches what shows in the link
Instead of os.environ.get('REQUEST_LOG_ID'), use request.environ.get('REQUEST_LOG_ID').
It may be possible that os.environ['REQUEST_LOG_ID'] changes between the start of the current request and the time you access it, but request.environ['REQUEST_LOG_ID'] should not change once the request is initialized. The docs state that if one request ID is greater than another, than it occurred later than the other. This implies that the requestID in the Stackdriver console was generated before the one in your email link. This makes me think that somewhere along the line, os.environ['REQUEST_LOG_ID'] is being updated from '....000' to '....101' before you access it, while the copy in request.environ['REQUEST_LOG_ID'] should remain unchanged.
For more info on the request.environ, take a look at the source code in google.appengine.runtime.request_environment.py. I haven't really found documentation on that, but that code led me to believe that the os.environ is not as safe to access as the request.environ.
I'm using Mailgun's RESTful API. I need a way to view non-standard headers of messages that have already been sent to connect them with a certain ID I have for database purposes. The main purpose of this is to log what emails are failing to be sent or not, along with the reasons and such.
My python code keeps returning a 405 response everytime I attempt to call something from "https://api.mailgun.net/v3/MY_DOMAIN_NAME/messages" when it is GET instead of POST. I am able to make other GET calls with events and stats as opposed to messages, so I do not think its the way I am writing the code exactly.
By calling on events I am able to view message status information, error codes, and ID's different from the ones I need. If I can just get this one non-standard header I will be set since it will give me the ID I need.
So, to the point, my question is if anyone who is familiar with this API knows if what I am trying to do is possible, and if so what should I look to be doing?
Here is an example of a call I make in case that helps:
This doesn't work:
def get_failed_events(start_date, end_date):
return requests.get(api_url.format('messages'), auth=('api', key), params={ 'begin': start_date, 'end': end_date}, verify=False)
This does:
def get_failed_events(start_date, end_date):
return requests.get(api_url.format('events'), auth=('api', key), params={ 'begin': start_date, 'end': end_date}, verify=False)
I found out that even though I was passing the information, I wasn't exactly passing it the way mailgun wants. It needs to be something like this c# example:
var forMessageLog = "{\"x-header-id\": 1232}";
message.Headers.Add("X-Mailgun-Variables", forMessageLog);
Otherwise it will not appear in user variables.