I'm trying to use cloud functions to update data by calling an external API once a day.
So far I have:
Cloud Schedule set to invoke Function 1
Function 1 - loop over items and create a task for each item
Task - invoke Function 2 with data provided by function 1
Function 2 - call external API to get data and update our db
The issue is that there are ~2k items to update daily and a cloud function times out before it can do that, hence why I put them in a queue. But even placing the items in the queue takes too long for the cloud function so that is timing out before it can add them all.
Is there a simple way to bulk add multiple tasks to a queue at once?
Failing that, a better solution to all of this?
All written in python
Code for function 1:
def refresh(request):
for i in items:
# Create a client.
client = tasks_v2.CloudTasksClient()
# TODO(developer): Uncomment these lines and replace with your values.
project = 'my-project'
queue = 'refresh-queue'
location = 'europe-west2'
name = i['name'].replace(' ','')
url = f"https://europe-west2-my-project.cloudfunctions.net/endpoint?name={name}"
# Construct the fully qualified queue name.
parent = client.queue_path(project, location, queue)
# Construct the request body.
task = {
"http_request": { # Specify the type of request.
"http_method": tasks_v2.HttpMethod.GET,
"url": url, # The full url path that the task will be sent to.
}
}
# Use the client to build and send the task.
response = client.create_task(request={"parent": parent, "task": task})
Answering your question “Is there a simple way to bulk add multiple tasks to a queue at once?” As per the public documentation The best approach is to implement a double-injection pattern.
For this you will have a new queue where you are going to add a single task that contains multiple tasks of the original queue, then on the receiving end of this queue, you will have a service that will get the data of this task and create one task per entry on a second queue.
Additionally, I will suggest you use the 500/50/5 pattern to a cold queue. This will help both the task queue and the Cloud Function service to ramp up on a safe ratio.
Chris32's answer is correct, but one thing i noticed in your code snippet is you should create the client outside the for loop.
def refresh(request):
# Create a client.
client = tasks_v2.CloudTasksClient()
# TODO(developer): Uncomment these lines and replace with your values.
project = 'my-project'
queue = 'refresh-queue'
location = 'europe-west2'
for i in items:
name = i['name'].replace(' ','')
url = f"https://europe-west2-my-project.cloudfunctions.net/endpoint?name={name}"
# Construct the fully qualified queue name.
parent = client.queue_path(project, location, queue)
# Construct the request body.
task = {
"http_request": { # Specify the type of request.
"http_method": tasks_v2.HttpMethod.GET,
"url": url, # The full url path that the task will be sent to.
}
}
# Use the client to build and send the task.
response = client.create_task(request={"parent": parent, "task": task})
In app engine i would do client = tasks_v2.CloudTasksClient() outside of def refresh, at the file level, but i dont know if that matters for cloud functions.
Second thing,
Modify "Function 2" to take multiple 'names' instead of just one. Then in "Function 1" you can send 10 names to "Function 2" at a time
BATCH_SIZE = 10 # send 10 names to Function 2
def refresh(request):
# Create a client.
client = tasks_v2.CloudTasksClient()
# ...
for i in range(0, len(items), BATCH_SIZE)]:
items_batch = items[i:i + BATCH_SIZE]
names = ','.join([i['name'].replace(' ','') for i in items_batch])
url = f"https://europe-west2-my-project.cloudfunctions.net/endpoint?names={names}"
# Construct the fully qualified queue name.
# ...
If those 2 quick-fixes don't do it, then you'll have to split "Function 1" into Function 1A" and "Function 1B"
Function 1A:
BATCH_SIZE = 100 # send 100 names to Function 1B
def refresh(request):
client = tasks_v2.CloudTasksClient()
for i in range(0, len(items), BATCH_SIZE)]:
items_batch = items[i:i + BATCH_SIZE]
names = ','.join([i['name'].replace(' ','') for i in items_batch])
url = f"https://europe-west2-my-project.cloudfunctions.net/endpoint-for-function-1b?names={names}"
# send the task.
response = client.create_task(request={
"parent": client.queue_path('my-project', 'europe-west2', 'refresh-queue'),
"task": {
"http_request": {"http_method": tasks_v2.HttpMethod.GET, "url": url}
}})
Function 1B:
BATCH_SIZE = 10 # send 10 names to Function 2
def refresh(request):
# set `names` equal to the query param `names`
client = tasks_v2.CloudTasksClient()
for i in range(0, len(names), BATCH_SIZE)]:
names_batch = ','.join(names[i:i + BATCH_SIZE])
url = f"https://europe-west2-my-project.cloudfunctions.net/endpoint-for-function-2?names={names_batch}"
# send the task.
response = client.create_task(request={
"parent": client.queue_path('my-project', 'europe-west2', 'refresh-queue'),
"task": {
"http_request": {"http_method": tasks_v2.HttpMethod.GET, "url": url}
}})
Related
I'm fairly new to Python and I'm trying to migrate subscriptions from an older youtube account to a newer one that I'll use going forward. I pulled my subscriptions export from the old one and have around 470+ subs that I'll need to migrate over.
I found this article which absolutely works with automatically subscribing to a youtube channel via their channel_id but it seems like in the key value pair I can only run the .py script once per value.
I tried all sorts of googling to see how I can include multiple values in the key (channelId) but it always only auto subs to the last one in the dictionary.
Can someone please help show me what I'm missing? I feel like there has to be a way to add multiple channelId values in there key dictionary, right?!
Here's what my code looks like > screenshot
import os
import google.oauth2.credentials
import google_auth_oauthlib.flow
from googleapiclient.discovery import build
from googleapiclient.errors import HttpError
from google_auth_oauthlib.flow import InstalledAppFlow
# The CLIENT_SECRETS_FILE variable specifies
# the name of a file that contains
# client_id and client_secret.
CLIENT_SECRETS_FILE = "client_secret.json"
# This scope allows for full read/write access
# to the authenticated user's account and
# requires requests to use an SSL connection.
SCOPES = ['https://www.googleapis.com/auth/youtube.force-ssl']
API_SERVICE_NAME = 'youtube'
API_VERSION = 'v3'
def get_authenticated_service():
flow = InstalledAppFlow.from_client_secrets_file(CLIENT_SECRETS_FILE, SCOPES)
credentials = flow.run_console()
return build(API_SERVICE_NAME, API_VERSION, credentials = credentials)
def print_response(response):
print(response)
# Build a resource based on a list of
# properties given as key-value pairs.
# Leave properties with empty values out
# of the inserted resource.
def build_resource(properties):
resource = {}
for p in properties:
# Given a key like "snippet.title", split into
# "snippet" and "title", where "snippet" will be
# an object and "title" will be a property in that object.
prop_array = p.split('.')
ref = resource
for pa in range(0, len(prop_array)):
is_array = False
key = prop_array[pa]
# For properties that have array values, convert a name like
# "snippet.tags[]" to snippet.tags, and set a flag to handle
# the value as an array.
if key[-2:] == '[]':
key = key[0:len(key)-2:]
is_array = True
if pa == (len(prop_array) - 1):
# Leave properties without values
# out of inserted resource.
if properties[p]:
if is_array:
ref[key] = properties[p].split(', ')
else:
ref[key] = properties[p]
elif key not in ref:
# For example, the property is "snippet.title",
# but the resource does not yet have a "snippet"
# object. Create the snippet object here.
# Setting "ref = ref[key]" means that in the
# next time through the "for pa in range ..." loop,
# we will be setting a property in the
# resource's "snippet" object.
ref[key] = {}
ref = ref[key]
else:
# For example, the property is "snippet.description",
# and the resource already has a "snippet" object.
ref = ref[key]
return resource
# Remove keyword arguments that are not set
def remove_empty_kwargs(**kwargs):
good_kwargs = {}
if kwargs is not None:
for key, value in kwargs.items():
if value:
good_kwargs[key] = value
return good_kwargs
def subscriptions_insert(client, properties, **kwargs):
resource = build_resource(properties)
kwargs = remove_empty_kwargs(**kwargs)
response = client.subscriptions().insert(
body = resource,**kwargs).execute()
return print_response(response)
if __name__ == '__main__':
# When running locally, disable OAuthlib's
# HTTPs verification. When running in production
# * do not * leave this option enabled.
os.environ['OAUTHLIB_INSECURE_TRANSPORT'] = '1'
client = get_authenticated_service()
subscriptions_insert(client,
{'snippet.resourceId.kind': 'youtube# channel',
'snippet.resourceId.channelId': 'UC09fL42MpkktKZWmWxYiDhw', 'UC0Q7Hlz75NYhYAuq6O0fqHw'},
part ='snippet')```
According to YouTube Data API v3 documentation (Subscriptions: insert endpoint and Subscriptions resource), it seems that you can only subscribe a channel at a time. As you have by default 10,000 of quota per day, except if you request extended quota, because Subscriptions: insert costs 50 of quota, then for 470+ subscriptions, you would need 3 days to proceed.
Otherwise you can proceed as follows, it seems that the first time I tried with ~500 channels I have been subscribed to ~290 of them but now I mostly only receive (when removing -H 'Accept-Encoding: gzip, deflate, br' from the cURL request):
{
"error": {
"code": 429,
"message": "Resource has been exhausted (e.g. check quota).",
"errors": [
{
"message": "Resource has been exhausted (e.g. check quota).",
"domain": "global",
"reason": "rateLimitExceeded"
}
],
"status": "RESOURCE_EXHAUSTED"
}
}
So it's an unsure method that you can try to deepen.
Ever wondered how to do that in a single request without using any quota?
Go on an ad hoc YouTube channel YOUR_CHANNEL that you want to subscribed to: https://www.youtube.com/channel/YOUR_CHANNEL_ID
Open the Network tab of your web-browser by using Ctrl + Shift + E (on Firefox) and filter XHR requests.
Now click on Subscribe.
You should see a request to subscribe, copy it as cURL (by right-clicking).
Change at the end
"channelIds":["YOUR_CHANNEL_ID"]
to:
"channelIds":["YOUR_CHANNEL_ID_0, YOUR_CHANNEL_ID_1, ..., YOUR_CHANNEL_ID_499"]
Where YOUR_CHANNEL_ID_0 is your YOUR_CHANNEL_ID and YOUR_CHANNEL_ID_1 the second channel you want to subscribe to and so forth.
Execute the modified cURL request in a terminal and that's it!
Note that this webpage contains a subscriptions count and this one contains all your subscriptions.
To get more than 249 different channels, I used:
import requests, json
channelIds = set()
pageToken = ''
API_KEY = 'AIzaSy...'
i = 0
while len(channelIds) < 250:
url = f'https://www.googleapis.com/youtube/v3/search?q={i}&type=channel&maxResults=50&key={API_KEY}'
if pageToken != '':
url += f"&pageToken={pageToken}"
content = requests.get(url).text
data = json.loads(content)
for item in data['items']:
channelIds.add(item['id']['channelId'])
print(len(channelIds))
if 'nextPageToken' in data:
pageToken = data['nextPageToken']
else:
break
i += 1
print('["' + '","'.join(channelIds) + '"]', len(channelIds))
As #Benjamin Loison has mentioned, there is a quota on the limit on the usage of the API. If you'd like to raise the limit, I think there is a form you can fill out to request more. However, I don't recommend you do so since the form is applicable mainly for a large application that will be used for a long time and involve a long process of human inspection on what you're trying to build (This is based on my personal experience, might not be entirely accurate).
My suggestion would be to use the script you have to print out a list of channel links, and you can click into each of them and press the subscribe button. 470-ish channels should not take you a long time.
Is have code
class Tasks(SequentialTaskSet):
#task
def shopping(self):
with self.parent.environment.parsed_options.shopping.request_airshopping(
client=self.client, catch_response=True) as response:
self.run_req(response, self.parent.environment.parsed_options.rsa)
class WebUser(HttpUser):
tasks = [Tasks]
min_wait = 0
max_wait = 0
#staticmethod
#events.test_start.add_listener
def on_test_start(environment: Environment, **kwargs):
os.environ["ENABLE_SLEEP"] = "False"
credentials = {
"login": environment.parsed_options.login,
"password": environment.parsed_options.password,
"structure_unit_code": environment.parsed_options.suid,
}
login = Login(base_url=environment.host)
response = login.request_login(credentials)
parse_xml = parsing_xml_response(response.text)
headers = {
"Content-Type": "application/xml",
"Authorization": f'Bearer {parse_xml.xpath("//Token")[0].text}'}
shopping = Shopping(
base_url=environment.parsed_options.host,
headers=headers
)
set_paxs(environment, shopping)
set_flight(environment, shopping)
environment.parsed_options.__dict__.update({"shopping": shopping})
If start without worker - success working
If start workers - errors: TypeError: can not serialize 'Shopping' object
How to send object Shopping in worker from master?
Try https://docs.locust.io/en/stable/running-distributed.html
You can do this by sending a message from worker to master or master to worker. The message is just a string but you can include a data payload with it. You need a function that registers what your message is and what to do when it receives that message, which many times will be a function that does something with the data payload. The docs use this example:
from locust import events
from locust.runners import MasterRunner, WorkerRunner
# Fired when the worker receives a message of type 'test_users'
def setup_test_users(environment, msg, **kwargs):
for user in msg.data:
print(f"User {user['name']} received")
environment.runner.send_message('acknowledge_users', f"Thanks for the {len(msg.data)} users!")
# Fired when the master receives a message of type 'acknowledge_users'
def on_acknowledge(msg, **kwargs):
print(msg.data)
#events.init.add_listener
def on_locust_init(environment, **_kwargs):
if not isinstance(environment.runner, MasterRunner):
environment.runner.register_message('test_users', setup_test_users)
if not isinstance(environment.runner, WorkerRunner):
environment.runner.register_message('acknowledge_users', on_acknowledge)
The #events.init.add_lisenter means Locust will run that function when it starts up and the key part is that register_message call. The first argument is an arbitrary string, whatever you want it to be. The second argument is the function to call when that message is received.
Note the arguments the function to be called has defined. One of them is msg, which you need to use to get the data that is sent.
Then you can send the message and data payload any time you want. The docs use this example:
users = [
{"name": "User1"},
{"name": "User2"},
{"name": "User3"},
]
environment.runner.send_message('test_users', users)
The string is the message you want the receiver to match and the second argument users in this case is the actual data you want to send to the other instance(s).
I recently encountered an issue that I have not been able to solve, despite calling the Bloomberg helpdesk and researching thoroughly the internet for similar cases.
In short, I am using the official Python blpapi from Bloomberg (https://github.com/msitt/blpapi-python) and now am experiencing some connectivity issue: I cannot leave a session opened.
Here is the code I am running: https://github.com/msitt/blpapi-python/blob/master/examples/SimpleHistoryExample.py
I simply added a "while True loop" and a "time.sleep" in it so that I can keep the session open and refresh my data every 30 seconds (this is my use case).
This use to run perfectly fine for days, however, since last Friday, I am now getting those log messages:
22FEB2021_08:54:18.870 29336:26880 WARN blpapi_subscriptionmanager.cpp:7437 blpapi.session.subscriptionmanager.{1} Could not find a service for serviceCode: 90.
22FEB2021_08:54:23.755 29336:26880 WARN blpapi_platformcontroller.cpp:377 blpapi.session.platformcontroller.{1} Connectivity lost, no connected endpoints.
22FEB2021_08:54:31.867 29336:26880 WARN blpapi_platformcontroller.cpp:344 blpapi.session.platformcontroller.{1} Connectivity restored.
22FEB2021_08:54:32.731 29336:26880 WARN blpapi_subscriptionmanager.cpp:7437 blpapi.session.subscriptionmanager.{1} Could not find a service for serviceCode: 90.
which goes on and on and on, along with those responses as well:
SessionConnectionDown = {
server = "localhost:8194"
}
ServiceDown = {
serviceName = "//blp/refdata"
servicePart = {
publishing = {
}
}
}
SessionConnectionUp = {
server = "localhost:8194"
encryptionStatus = "Clear"
compressionStatus = "Uncompressed"
}
ServiceUp = {
serviceName = "//blp/refdata"
servicePart = {
publishing = {
}
}
}
I still can pull the data from the bloomberg API: I see the historical data request results just fine. However:
Those service/session status messages messes up my code (I could still ignore them)
For some reason the connect/reconnect also messes my Excel BBG in the background and prevent me from using the BBG excel add-in at all! I now have those "#N/A Connection" outputs in all of my workbooks using bloomberg formulas.
screenshot from excel
Has anyone ever encountered such cases? If yes, please do not hesitate to share your experience, any help is more than appreciated!
Wishing you all a great day,
Adrien
I cannot comment yet so I will try to "answer" it. I use blpapi everyday and pull data all day. I am a BBG Anywhere user and never have any session issues. If you log in from a different device it will kill your session for the Python app. Once you log back in where the python app is running it will connect again.
Why do you have another while loop and sleep to keep the session alive? You should create a separate session and always call it to run your request. You should not need any "keep alive" code inside the request. Just don't call session.stop(). This is what I ended up doing after much trial and error from not knowing what to do.
I run my model using Excel, trying to move away from any substantial code in Excel and use it as a GUI until I can migrate to a custom GUI. I also have BDP functions in my Excel and they work fine.
import blpapi
# removed optparse because it is deprecated.
from argparse import ArgumentParser
SERVICES = {}
def parseCmdLine():
parser = ArgumentParser(description='Retrieve reference data.')
parser.add_argument('-a',
'--ip',
dest='host',
help='server name or IP (default: %(default)s)',
metavar='ipAddress',
default='localhost')
parser.add_argument('-p',
dest='port',
type=int,
help='server port (default: %(default)s)',
metavar='tcpPort',
default=8194)
args = parser.parse_args()
return args
def start_session():
"""Standard session for synchronous refdata requests. Upon creation
the obj is held in SERVICES['session'].
Returns:
obj: A session object.
"""
args = parseCmdLine()
# Fill SessionOptions
sessionOptions = blpapi.SessionOptions()
sessionOptions.setServerHost(args.host)
sessionOptions.setServerPort(args.port)
# Create a Session
session = blpapi.Session(sessionOptions)
# Start a Session
session.start()
SERVICES['session'] = session
return SERVICES['session']
def get_refDataService():
"""Create a refDataService object for functions to use. Upon creation
it is held in SERVICES['refDataService'].
Returns:
obj: refDataService object.
"""
# return the session and request because requests need session.send()
global SERVICES
if 'session' not in SERVICES:
start_session()
session = SERVICES['session']
# Check for SERVICES['refdata'] not needed because start_session()
# is called by this function and start_session() is never called on its own.
session.openService("//blp/refdata")
refDataService = session.getService("//blp/refdata")
SERVICES['refDataService'] = refDataService
session = SERVICES['session']
refDataService = SERVICES['refDataService']
print('get_refDataService called. Curious when this is called.')
return session, refDataService
# sample override request
def ytw_oride_muni(cusip_dict):
"""Requests the Price To Worst for a dict of cusips and Yield To Worst values.
The dict must be {'cusip' : YTW}. Overrides apply to each request, so this
function is designed for different overrides for each cusip. Although they could
all be the same as that is not a restriction.
Returns: Single level nested dict
{'cusip Muni': {'ID_BB_SEC_NUM_DES': 'val', 'PX_ASK': 'val1', 'YLD_CNV_ASK': 'val2'}}
"""
session, refDataService = get_refDataService()
fields1 = ["ID_BB_SEC_NUM_DES", "PX_ASK", "YLD_CNV_ASK"]
try:
values_dict = {}
# For different overrides you must send separate requests.
# This loops and creates separate messages.
for cusip, value in cusip_dict.items():
request = refDataService.createRequest("ReferenceDataRequest")
# append security to request
request.getElement("securities").appendValue(f"{cusip} Muni")
# append fields to request
request.getElement("fields").appendValue(fields1[0])
request.getElement("fields").appendValue(fields1[1])
request.getElement("fields").appendValue(fields1[2])
# add overrides
overrides = request.getElement("overrides")
override1 = overrides.appendElement()
override1.setElement("fieldId", "YLD_CNV_ASK")
override1.setElement("value", f"{value}")
session.sendRequest(request)
# Process received events
while(True):
# We provide timeout to give the chance to Ctrl+C handling:
ev = session.nextEvent(500)
# below msg.messageType == ReferenceDataResponse
for msg in ev:
if msg.messageType() == "ReferenceDataResponse":
if msg.hasElement("responseError"):
print(msg)
if msg.hasElement("securityData"):
data = msg.getElement("securityData")
num_cusips = data.numValues()
for i in range(num_cusips):
sec = data.getValue(i).getElement("security").getValue()
try:
des = data.getValue(i).getElement("fieldData").getElement("ID_BB_SEC_NUM_DES").getValue()
except:
des = None
try:
ptw = data.getValue(i).getElement("fieldData").getElement("PX_ASK").getValue()
except:
ptw = None
try:
ytw = data.getValue(i).getElement("fieldData").getElement("YLD_CNV_ASK").getValue()
except:
ytw = None
values = {'des': des, 'ptw': ptw, 'ytw': ytw}
# Response completly received, so we could exit
if ev.eventType() == blpapi.Event.RESPONSE:
values_dict.update({sec: values})
break
finally:
# Stop the session
# session.stop()
return values_dict
I am using slack webhooks to send message to my slack channel.
The problem it keeps posting messages every few mins.
Here what I did...
I created a simple function under util folder.
def send_to_slack(text):
conn_id = "https://hooks.slack.com/services/your/slack/URL"
task_slack_alert(text, url, is_error=False, args=None)
def task_slack_alert(msg, url, is_error=False, args=None):
slack_msg = ":red_circle: Task Failed" if is_error else ":green_heart: Task Message"
"""*Task*: {task}
*Dag*: {dag}
*Execution Time*: {exec_ts}""".format(
task=args["task"],
dag=args["dag"],
exec_ts=args["ts"],
) if args else ""
message = {'text': + msg}
response = requests.post(url=url, data=json.dumps(message))
time.sleep(1)
print(f"Slack response {response}")
if response.status_code != 200:
print(f"Error sending chat message. Got: {response.status_code}")
In my dag (which is under another folder) I call the function
The dag copy data from oracle to snowflake db and this works without slack part.
Inside my dag i do the following:
x = {‘key1’: [‘value1’, ‘value 2’, … ‘value10]}
send_to_slack('My test message from python')
default_args = {...
'on_failure_callback': send_to_slack, }
with DAG(‘my_dag’,
default_args=default_args,
catchup=False) as dag:
parallel = 4
start = DummyOperator(task_id='start')
tasks = []
i = 0
for s in x.keys():
for t in x.get(s):
task = OracleToSnowflakeOperator(
task_id=s + '_' + t,
source_oracle_conn_id=source_oracle_conn_id,
source_schema=schema,
source_table=table,…
)
if i <= parallel:
task.set_upstream(start)
else:
task.set_upstream(tasks[i - (parallel + 1)])
i = i + 1
tasks.append(task)
I know if I define the function inside the same dag, it will be called every time the dag is parsed.
not my case, so What's wrong?
Thanks
You're calling the function send_to_slack inside your DAG file, this means it will run every time the scheduler evaluates your DAG (every few minutes).
You should either:
Use the slack operator that comes with Airflow and put it downstream from your OracleToSnowflakeOperator and treat it like any other operator
Edit your OracleToSnowflakeOperator, which I assume is a custom one, and put the logic to call Slack in there (use the slack hook)
Basically you should put be encapsulating the call to Slack inside a custom operator or use the standard Slack operator provided, don't put it inside your DAG definition.
I'm trying to wrap my head around the channel features of Google App Engine since they don't (easily) provide websockets.
My current situation is that I have a long work (file processing) that is being executed asynchronously via a worker.
This worker update the state of the file processing in the database at every lines in order to inform the customer.
From that current perspective, a F5 will indicate the evolution of the processing.
Now I'd like to implement a live update system. Of course I could do an XHR request every 5 seconds but a live connection seems better... introducing Channels since Websockets seems out of the possibilities.
From what I understood, I can channel.send_message to one client only, not to a "room". The issue here, is that the worker that process the file does not have any information which customer is currently connected (could be one, could be ten).
I could loop over all the customer and post to each client_id, suspecting that at least one of them will get the message, but this is awfully useless and too resourceful.
I was hoping there was a better way to achieve this ? Maybe a nice alternative to Google Channels feature without having to reconfigure my whole App Engine system (like for Websockets)?
One solution I can think of, which is not the absolute ideal but would be more suited, is to manage dedicated database tables (could also be implemented in Memcache) with :
A table that contains a list of rooms
A table that contains a list of client_id connected to the room
e.g. :
Rooms (id, name)
Clients (id, room_id, client_id)
Now, instead of posting to channel.send_message(client_id, Message), one would make a wrapper like this :
def send_to_room(room, message):
# Queries are SQLAlchemy like :
room = Rooms.query.filter(Rooms.name === room).first()
if not room:
raise Something
clients = Clients.query.filter(Rooms.room_id === room.id).all()
for client in clients:
channel.send_message(client.client_id, message)
And voilà, you have a Room like implementation in Google App Engine.
The drawback of this solution is to add two tables (or equivalent) in your database.
Does someone has better?
I am assuming that the long running task is being kicked off by the client.
So before you kick off the task make a ajax request from the client to a handler similar to this one. This handler has two things returned to the client. The token param which is used by the javascript api to create a channel, and a cid param which is used to determine which client created the channel.
from google.appengine.api import channel
#ae.route("/channel")
class CreateChannel(webapp2.RequestHandler):
def get(self):
cid = str(uuid.uuid4())
token = channel.create_channel(cid)
data = {
"cid":cid,
"token":token
}
self.response.write(json.dumps(data))
Now use the channel javascript api to create a new channel
https://cloud.google.com/appengine/docs/python/channel/javascript
var onClosed = function(resp){
console.log("channel closed");
};
var onOpened = function(resp){
console.log("channel created");
};
var onmessage = function(resp){
console.log("The client received a message from the backend task");
console.log(resp);
};
var channel_promise = $.ajax({
url: "/channel",
method: "GET"
});
channel_promise.then(function(resp){
//this channel id is important you need to get it to the backend process so it knows which client to send the message to.
var client_id = resp.data.cid;
var channel = new goog.appengine.Channel(resp.data.token);
handler = {
'onopen': $scope.onOpened,
'onmessage': $scope.onMessage,
'onerror': function () {
},
'onclose': function () {
alert("channel closed.")
}
};
socket = channel.open(handler);
//onOpened is the callback function to call after channel has been created
socket.onopen = onOpened;
//onClose is the callback function to call after channel has been closed
socket.onclose = onClosed;
//onmessage is the callback function to call when receiving messages from your task queue
socket.onmessage = onMessage;
});
Now we are all set up to listen for channel messages.
So when the user clicks the button we need to kickoff the backend task.
var letsDoSomeWorkOnClick = function(){
//make sure you pass the client id with every ajax request
$.ajax({
url: "/kickoff",
method: "POST",
params: {"cid":client_id}
});
}
Now the app engine handler to start the backend task queue. I use the deffered library to do this. https://cloud.google.com/appengine/articles/deferred
#ae.route("/kickoff")
KickOffHandler(webapp2.RequestHandler):
def post(self):
cid = self.request.get("cid")
req = {}
req['cid'] = cid
task = MyLongRunningTask()
deferred.defer(task.long_runner_1, req, _queue="my-queue")
example task:
class MyLongRunningTask:
def long_runner_1(self,req):
# do a whole bunch of stuff
channel.send_message(req["cid"], json.dumps({"test":"letting client know task is done"})