We Keep Coding

Exchanging data between Python Telethon library and Node.js

I faced such a problem: in my small test app I have simple node.js (express) server and python script, which allows me to interact with Telegram API using Telethon library. In my scenario I have to provide my python script a phone number and a password. These data is asked in the input mode, so I can't figure out, how am I able to:
Accept input request from python script on node.js side;
Provide these credentials by node.js and pass them back via python's input;
Repeat this actions several times to gain all info I need.
These are my test files:
file.py
import os
from telethon import TelegramClient
api_id = 12345
api_hash = 'hash'
session = 'testing'
proxy = None
client = TelegramClient(session, api_id, api_hash, proxy=proxy).start()
def get_ids_list():
ids_dict = {}
async def do_fetch():
async for dialog in client.iter_dialogs():
ids_dict[dialog.name] = dialog.id
with client:
client.loop.run_until_complete(do_fetch())
return ids_dict
def print_ids_list():
async def do_print():
async for dialog in client.iter_dialogs():
print(dialog.name, 'has ID', dialog.id)
with client:
client.loop.run_until_complete(do_print())
print_ids_list()
When this script is run, I'm prompted the following input:
Please enter your phone (or bot token):
And this is my index.js, in which I want to pass prepared data to this input:
import express from "express";
import { spawn } from "child_process";
const app = express();
const port = 3000;
app.get("/", (req, res) => {
var myPythonScript = "path/to/file.py";
var pythonExecutable = "python";
var uint8arrayToString = function (data) {
return String.fromCharCode.apply(null, data);
};
const scriptExecution = spawn(pythonExecutable, [myPythonScript]);
scriptExecution.stdout.on("data", (data) => {
console.log(uint8arrayToString(data));
});
scriptExecution.stderr.on("data", (data) => {
console.log(uint8arrayToString(data));
});
scriptExecution.on("exit", (code) => {
console.log("Process quit with code : " + code);
});
});
app.listen(port, () =>
console.log(`Example app listening on port ${port}!`)
);
So, is there a way to solve this case?

Using with client is equivalent to client.start(), and as stated:
By default, this method will be interactive (asking for user input if needed), and will handle 2FA if enabled too.
You need to instead do what it does manually, remove with block, and make a function to authenticate (or confirm if already authorized).
for a minimal func example:
....
if not client.is_connected():
await client.connect()
if not await client.is_user_authorized():
await client.send_code_request(phone)
# get the code somehow, and in a reasonably fast way
try:
await client.sign_in(phone, code)
except telethon.errors.SessionPasswordNeededError:
'''
Retry with password.
note: it's necessary to make it error once
even if you know you have a pass, iow don't pass password the first time.
'''
await client.sign_in(phone, code, password=password)
return client
else:
return client
Dealing with the steps sequentially and interactivly while waiting for needed params to login successfully while also keeping in mind you have time limit until code expires is your task to handle any of their undefined behavior dependant on your usecase.

Websockets messages only sent at the end and not in instances using async / await, yield in nested for loops

I have a computationally heavy process that takes several minutes to complete in the server. So I want to send the results of every iteration to the client via websockets.
The overall application works but my problem is that all the messages are arriving at the client in one big chunk after the entire simulation finishes. I must be missing something here as I expect the await websocket.send_json() to send the message during the process and not all of them at the end.
Server python (FastAPI)
# A very simplified abstraction of the actual app.
def simulate_intervals(data):
for t in range(data.n_intervals):
state = interval(data) # returns a JAX NumPy array
yield state
def simulate(data):
for key in range(data.n_trials):
trial = simulate_intervals(data)
yield trial
#app.websocket("/ws")
async def socket(websocket: WebSocket):
await websocket.accept()
while True:
# Get model inputs from client
data = await websocket.receive_text()
# Minimal computation
nodes = distributions(data)
nodosJson = json.dumps(nodes, cls=NumpyEncoder)
# I expect this message to be sent early on,
# but the client gets it at the end with all the other messages.
await websocket.send_json({"tipo": "nodos", "datos": json.loads(nodosJson)})
# Heavy computation
trials = simulate(data)
for trialI, trial in enumerate(trials):
for stateI, state in enumerate(trial):
stateString = json.dumps(state, cls=NumpyEncoder)
await websocket.send_json(
{
"tipo": "estado",
"datos": json.loads(stateString),
"trialI": trialI,
"stateI": stateI,
}
)
await websocket.send_json({"tipo": "estado", "msg": "fin"})
For completeness, here is the basic client code.
Client
const ws = new WebSocket('ws://localhost:8000/ws');
ws.onopen = () => {
console.log('Conexión exitosa');
};
ws.onmessage = (e) => {
const mensaje = JSON.parse(e.data);
console.log(mensaje);
};
botonEnviarDatos.onclick = () => {
ws.send(JSON.stringify({...}));
}

I was not able to make it work as posted in my question, still interested in hearing from anyone who understands why it is not possible to send multiple async messages without them getting blocked.
For anyone interested, here is my current solution:
Ping pong messages from client and server
I changed the logic so the server and client are constantly sending each other messages and not trying to stream the data in a single request from the client.
This actually works much better than my original attempt because I can detect when a sockets gets disconnected and stop processing in the server. Basically, if the client disconnects, no new requests for data are sent from that client and the server never continues the heavy computation.
Server
# A very simplified abstraction of the actual app.
def simulate_intervals(data):
for t in range(data.n_intervals):
state = interval(data) # returns a JAX NumPy array
yield state
def simulate(data):
for key in range(data.n_trials):
trial = simulate_intervals(data)
yield trial
#app.websocket("/ws")
async def socket(websocket: WebSocket):
await websocket.accept()
while True:
# Get messages from client
data = await websocket.receive_text()
# "tipo" is basically the type of data being sent from client or server to the other one.
# In this case, "tipo": "inicio" is the client sending inputs and requesting for a certain data in response.
if data["tipo"] == "inicio":
# Minimal computation
nodes = distributions(data)
nodosJson = json.dumps(nodes, cls=NumpyEncoder)
# In this first interaction, the client gets the first message without delay.
await websocket.send_json({"tipo": "nodos", "datos": json.loads(nodosJson)})
# Since this is a generator (def returns yield) it does not actually
# trigger that actual computationally heavy process.
trials = simulate(data)
# define some initial variables to count the iterations
trialI = 0
stateI = 0
trialsLen = args.number_trials
statesLen = 600
# load the first trial (also a generator)
# without the for loop used before, the counters and next()
# allow us to do the same as being done before in the for loop
trial = next(trials)
# With the use of generators and next() it is possible to keep
# this first message light on the server and send the first response
# as quickly as possible.
# This type of message asks for the next instance of the simluation
# without processing the entire model.
elif data["tipo"] == "sim":
# check if we are within the limits (before this was a nested for loop)
if trialI < trialsLen and stateI < statesLen:
# Trigger the next instance of the simulation
state = next(trial)
# update counter
stateI = stateI + 1
# Send the message with 1 instance of the simulation.
#
stateString = json.dumps(state, cls=NumpyEncoder)
await websocket.send_json(
{
"tipo": "estado",
"datos": json.loads(stateString),
"trialI": trialI,
"stateI": stateI,
}
)
# Check if the second loop is done
if stateI == statesLen:
# update counter of first loop
trialI = trialI + 1
# update counter of second loop
stateI = 0
# Check if there are more pending trials,
# otherwise stop and notify the client we are done.
try:
trial = next(trials)
except StopIteration:
await websocket.send_json({"tipo": "fin"})
Client
Just the part that actually changed:
ws.onmessage = (e) => {
const mensaje = JSON.parse(e.data);
// Simply check the type of incoming message so it can be processed
if (mensaje.tipo === 'fin') {
viz.calcularResultados();
} else if (mensaje.tipo === 'nodos') {
viz.pintarNodos(mensaje.datos);
} else if (mensaje.tipo === 'estado') {
viz.sumarEstado(mensaje.datos);
}
// After receiving a message, ping the server for the next one
ws.send(
JSON.stringify({
tipo: 'sim',
})
);
};
This seems like reasonable solution to keep the server and client working together. I am able to show in the client the progress of a long simulation and the user experience is much better than having to wait for a long time for the server to respond. Hope it helps other with a similar problem.

I got a similar issue, and was able to resolve it by adding a small await asyncio.sleep(0.1) after sending json messages. I have not dived into asyncios internals yet, but my guess is that websocker.send shedules a message to be sent, but since the async function continues to run it never has a chance to do it in the background. Sleeping the async function makes asyncio pick up other tasks while it is waiting.

Why is the python client not receiving SSE events?

I am have a python client listening to SSE events from a server with node.js API
The flow is I sent an event to the node.js API through call_notification.py and run seevents.py in loop using run.sh(see below)
However I don't see that python client is receiving this SSE event? any guidance on why is that?
call_notification.py
import requests
input_json = {'BATS':'678910','root_version':'12A12'}
url = 'http://company.com/api/root_event_notification?params=%s'%input_json
response = requests.get(url)
print response.text
node.js API
app.get("/api/root_event_notification", (req, res, next) => {
console.log(req.query.params)
var events = require('events');
var eventEmitter = new events.EventEmitter();
//Create an event handler:
var myEventHandler = function () {
console.log('new_root_announced!');
res.status(200).json({
message: "New root build released!",
posts: req.query.params
});
}
seevents.py (python client listening to SSE events)
import json
import pprint
import sseclient
def with_urllib3(url):
"""Get a streaming response for the given event feed using urllib3."""
import urllib3
http = urllib3.PoolManager()
return http.request('GET', url, preload_content=False)
def with_requests(url):
"""Get a streaming response for the given event feed using requests."""
import requests
return requests.get(url, stream=True)
url = 'http://company.com/api/root_event_notification'
response = with_urllib3(url) # or with_requests(url)
client = sseclient.SSEClient(response)
#print client.events()
for event in client.events():
print "inside"
pprint.pprint(json.loads(event.data))
run.sh
#!/bin/sh
while [ /usr/bin/true ]
do
echo "Running sseevents.py"
python sseevents.py 2>&1 | tee -a sseevents.log.txt
echo "sleeping for 30 sec"
sleep 30
done
OUTPUT:-
Run call_notification.py on Terminal
node.js API OUTPUT
new_root_announced!
{'root_version': 'ABCD', 'BATS': '143'}
./run.sh --> DON'T SEE ABOVE EVENT below
Running sseevents.py
sleeping for 30 sec
Running sseevents.py
sleeping for 30 sec
Running sseevents.py
sleeping for 30 sec

Very short answer to you question:
The server code is not sending a SSE message back to the client.
Why? Because you need to follow the SSE format.
According to JASON BUTZ in Server-Sent Events With Node
You should send a Connection: keep-alive header to ensure the client keeps the connection open as well. A Cache-Control header should be sent with the value no-cache to discourage the data being cached. Finally, the Content-Type needs to be set to text/event-stream.
With all of that done a newline (\n) should be sent to the client and then the events can be sent. Events must be sent as strings, but what is in that string doesn’t matter. JSON strings are perfectly fine.
Event data must be sent in the format "data: <DATA TO SEND HERE>\n".
It’s important to note that at the end of each line should be a newline character. To signify the end of an event an extra newline character needs to be added as well.
Multiple data lines are perfectly fine.
Long answer to your question:
According to Eric Bidelman in html5rocks.com:
When communicating using SSEs, a server can push data to your app whenever it wants, without the need to make an initial request. In other words, updates can be streamed from server to client as they happen.
But, in order for this to happen, the client has to "start" by asking for it AND prepare to receive a stream of messages (when they happen).
The "start" is done by calling a SSE API endpoint (in your case, calling the Node.js API code).
The preparation is done by preparing to handle a stream of asynchronous messages.
SSEs open a single unidirectional channel between server and client.*
* The emphasis is mine
This means that the server has a "direct" channel to the client. It is not intended to be "started" (opened) by some other process/code that is not "the client" code.
Assuming from OP comments...
Expected behavior (verbose)
A client Alice calls the API endpoint with params {name: "Alice"}, nothing (visible) happens.
...then a client Bob calls the API endpoint with params {name: "Bob"}, client Alice receives a SSE with payload {name: "Bob", says: "Hi"}.
...then a client Carol calls the API endpoint with params {name: "Carol"}, clients Alice AND Bob each one receives a SSE with payload {name: "Carol", says: "Hi"}.
...and so on. Every time a new client calls the API endpoint with params, every other client who has a channel "open" will receive a SSE with the new "Hi" payload.
...and then client Bob "disconnects" from the server, client Alice, client Carol and all the clients that have a channel "open" will receive a SSE with payload {name: "Bob", says: "Bye"}.
...and so on. Every time an old client "disconnects" from the server, every other client who has a channel "open" will receive a SSE with the new "Bye" payload.
Abstracted behavior
Each new client that asks to "open" a channel sending some params or an old client "disconnects" from the server, they cause and event in the server.
Every time such an event happens in the server, the server sends a SSE message with the params and a message as payload to all the "open" channels.
Note on blocking Each client with an "open" channel will be "stuck" in an infinite waiting loop for events to happen. It is client design responsibility to use "threading" code techniques to avoid blocking.
Code
Your Python client should "ask" to start the single unidirectional channel AND keep waiting UNTIL the channel is closed. Should not end and start all over again with a different channel. It should keep the same channel open.
From the network perspective, it will be like a "long" response that does not end (until the SSE messaging is over). The response just "keeps coming and coming".
Your Python client code does that. I noted it is the exact sample code used from sseclient-py library.
Client code for Python 3.4
To include the parameters you want to send to the server, use some code from the Requests library docs/#passing-parameters-in-urls.
So, mixing those samples we end up with the following code as your Python 3.4 client:
import json
import pprint
import requests
import sseclient # sseclient-py
# change the name for each client
input_json = {'name':'Alice'}
#input_json = {'name':'Bob'}
#input_json = {'name':'Carol'}
url = 'http://company.com/api/root_event_notification'
stream_response = requests.get(url, params=input_json, stream=True)
client = sseclient.SSEClient(stream_response)
# Loop forever (while connection "open")
for event in client.events():
print ("got a new event from server")
pprint.pprint(event.data)
Client code for Python 2.7
To include the parameters you want to send to the server, encode them in the URL as query parameters using urllib.urlencode() library.
Make the http request with urllib3.PoolManager().request() so you will end up with a stream response.
Note that the sseclient library returns event data as unicode string. To convert back the JSON object to python object (with python strings) use byteify, a recursive custom function ( thanks to Mark Amery ).
Use the following code as your Python 2.7 client:
import json
import pprint
import urllib
import urllib3
import sseclient # sseclient-py
# Function that returns byte strings instead of unicode strings
# Thanks to:
# [Mark Amery](https://stackoverflow.com/users/1709587/mark-amery)
def byteify(input):
if isinstance(input, dict):
return {byteify(key): byteify(value)
for key, value in input.iteritems()}
elif isinstance(input, list):
return [byteify(element) for element in input]
elif isinstance(input, unicode):
return input.encode('utf-8')
else:
return input
# change the name for each client
input_json = {'name':'Alice'}
#input_json = {'name':'Bob'}
#input_json = {'name':'Carol'}
base_url = 'http://localhost:3000/api/root_event_notification'
url = base_url + '?' + urllib.urlencode(input_json)
http = urllib3.PoolManager()
stream_response = http.request('GET', url, preload_content=False)
client = sseclient.SSEClient(stream_response)
# Loop forever (while connection "open")
for event in client.events():
print ("got a new event from server")
pprint.pprint(byteify(json.loads(event.data)))
Now, the server code should:
emit an inside-server 'hello' event so other clients listen to the event
"open" the channel
Register to listen for all possible inside-server events to happen (this means, keeping the channel "open" and not sending anything between messages, just keeping the channel "open").
This includes to emit an inside-server 'goodbye' event so other clients listen to the event WHEN channel is closed by the client/network (and finally "wrap up").
Use the following Node.js API code:
var EventEmitter = require('events').EventEmitter;
var myEmitter = new EventEmitter;
function registerEventHandlers(req, res) {
// Save received parameters
const myParams = req.query;
// Define function that adds "Hi" and send a SSE formated message
const sayHi = function(params) {
params['says'] = "Hi";
let payloadString = JSON.stringify(params);
res.write(`data: ${payloadString}\n\n`);
}
// Define function that adds "Bye" and send a SSE formated message
const sayBye = function(params) {
params['says'] = "Bye";
let payloadString = JSON.stringify(params);
res.write(`data: ${payloadString}\n\n`);
}
// Register what to do when inside-server 'hello' event happens
myEmitter.on('hello', sayHi);
// Register what to do when inside-server 'goodbye' event happens
myEmitter.on('goodbye', sayBye);
// Register what to do when this channel closes
req.on('close', () => {
// Emit a server 'goodbye' event with "saved" params
myEmitter.emit('goodbye', myParams);
// Unregister this particular client listener functions
myEmitter.off('hello', sayHi);
myEmitter.off('goodbye', sayBye);
console.log("<- close ", req.query);
});
}
app.get("/api/root_event_notification", (req, res, next) => {
console.log("open -> ", req.query);
// Emit a inside-server 'hello' event with the received params
myEmitter.emit('hello', req.query);
// SSE Setup
res.writeHead(200, {
'Content-Type': 'text/event-stream',
'Cache-Control': 'no-cache',
'Connection': 'keep-alive',
});
res.write('\n');
// Register what to do when possible inside-server events happen
registerEventHandlers(req, res);
// Code execution ends here but channel stays open
// Event handlers will use the open channel when inside-server events happen
})
...continue quoting Eric Bidelman in html5rocks.com:
Sending an event stream from the source is a matter of constructing a plaintext response, served with a text/event-stream Content-Type, that follows the SSE format. In its basic form, the response should contain a "data:" line, followed by your message, followed by two "\n" characters to end the stream
In the client code, the sseclient-py library takes care of interpreting the SSE format so every time the two "\n" characters arrive, the library "iterates" a new "iterable" object (a new event) that has the data property with the message sent from the server.
This is how I tested the code
Started server with Node.js API code
Run a client with only the "Alice" line uncommented (Nothing is seen on this client console yet).
Run a second client with only "Bob" line uncommented. The console of the first client "Alice" shows: Bob saying "Hi" (Nothing is seen on Bob's client console yet).
Run a third client with only "Carol" line uncommented. Alice's and Bob's consoles show: Carol saying "Hi" (Nothing is seen on Carol's client console yet).
Stop/kill Bob's client. Alice's and Carol's consoles show: Bob saying "Bye".
So, code works OK :)

GAE Channels to multiple clients?

I'm trying to wrap my head around the channel features of Google App Engine since they don't (easily) provide websockets.
My current situation is that I have a long work (file processing) that is being executed asynchronously via a worker.
This worker update the state of the file processing in the database at every lines in order to inform the customer.
From that current perspective, a F5 will indicate the evolution of the processing.
Now I'd like to implement a live update system. Of course I could do an XHR request every 5 seconds but a live connection seems better... introducing Channels since Websockets seems out of the possibilities.
From what I understood, I can channel.send_message to one client only, not to a "room". The issue here, is that the worker that process the file does not have any information which customer is currently connected (could be one, could be ten).
I could loop over all the customer and post to each client_id, suspecting that at least one of them will get the message, but this is awfully useless and too resourceful.
I was hoping there was a better way to achieve this ? Maybe a nice alternative to Google Channels feature without having to reconfigure my whole App Engine system (like for Websockets)?

One solution I can think of, which is not the absolute ideal but would be more suited, is to manage dedicated database tables (could also be implemented in Memcache) with :
A table that contains a list of rooms
A table that contains a list of client_id connected to the room
e.g. :
Rooms (id, name)
Clients (id, room_id, client_id)
Now, instead of posting to channel.send_message(client_id, Message), one would make a wrapper like this :
def send_to_room(room, message):
# Queries are SQLAlchemy like :
room = Rooms.query.filter(Rooms.name === room).first()
if not room:
raise Something
clients = Clients.query.filter(Rooms.room_id === room.id).all()
for client in clients:
channel.send_message(client.client_id, message)
And voilà, you have a Room like implementation in Google App Engine.
The drawback of this solution is to add two tables (or equivalent) in your database.
Does someone has better?

I am assuming that the long running task is being kicked off by the client.
So before you kick off the task make a ajax request from the client to a handler similar to this one. This handler has two things returned to the client. The token param which is used by the javascript api to create a channel, and a cid param which is used to determine which client created the channel.
from google.appengine.api import channel
#ae.route("/channel")
class CreateChannel(webapp2.RequestHandler):
def get(self):
cid = str(uuid.uuid4())
token = channel.create_channel(cid)
data = {
"cid":cid,
"token":token
}
self.response.write(json.dumps(data))
Now use the channel javascript api to create a new channel
https://cloud.google.com/appengine/docs/python/channel/javascript
var onClosed = function(resp){
console.log("channel closed");
};
var onOpened = function(resp){
console.log("channel created");
};
var onmessage = function(resp){
console.log("The client received a message from the backend task");
console.log(resp);
};
var channel_promise = $.ajax({
url: "/channel",
method: "GET"
});
channel_promise.then(function(resp){
//this channel id is important you need to get it to the backend process so it knows which client to send the message to.
var client_id = resp.data.cid;
var channel = new goog.appengine.Channel(resp.data.token);
handler = {
'onopen': $scope.onOpened,
'onmessage': $scope.onMessage,
'onerror': function () {
},
'onclose': function () {
alert("channel closed.")
}
};
socket = channel.open(handler);
//onOpened is the callback function to call after channel has been created
socket.onopen = onOpened;
//onClose is the callback function to call after channel has been closed
socket.onclose = onClosed;
//onmessage is the callback function to call when receiving messages from your task queue
socket.onmessage = onMessage;
});
Now we are all set up to listen for channel messages.
So when the user clicks the button we need to kickoff the backend task.
var letsDoSomeWorkOnClick = function(){
//make sure you pass the client id with every ajax request
$.ajax({
url: "/kickoff",
method: "POST",
params: {"cid":client_id}
});
}
Now the app engine handler to start the backend task queue. I use the deffered library to do this. https://cloud.google.com/appengine/articles/deferred
#ae.route("/kickoff")
KickOffHandler(webapp2.RequestHandler):
def post(self):
cid = self.request.get("cid")
req = {}
req['cid'] = cid
task = MyLongRunningTask()
deferred.defer(task.long_runner_1, req, _queue="my-queue")
example task:
class MyLongRunningTask:
def long_runner_1(self,req):
# do a whole bunch of stuff
channel.send_message(req["cid"], json.dumps({"test":"letting client know task is done"})

Can't make GAE Channel API work on local computer

I am creating a small application to test how GAE Channel API works. I think I have done all as it's described in the documentation but when I launch it, it shows an error in FireFox error log about syntax in the beginning and then another repeating error that an element wasn't found that.
Here is the first error info:
Source: http://127.0.0.1:8080/_ah/channel/dev?command=connect&channel=channel-773698929-185804764220139124118
Line 1, symbol 1
Here is the url where my javascript code tries to connect repeatedly and it raises the second error:
http://127.0.0.1:8080/_ah/channel/dev?command=poll&channel=channel-2071442473-185804764220139124118&client=1
I get the token through a JSON request with jQuery $.get. Then I run this code to get the token and open the channel. The error begins to show just when I run socket = channel.open(handler):
var response = JSON.parse(data);
var token = response.token.toString();
channel = new goog.appengine.Channel(token);
var handler = {
'onopen': onOpened,
'onmessage': onMessage,
'onerror': function() {
},
'onclose': function() {
}
};
socket = channel.open(handler);
Here is the server side code in Python to open the channel:
class OpenChannel(webapp.RequestHandler):
def get(self):
user = users.get_current_user()
token = channel.create_channel(user.user_id())
serialized = json.dumps({'token': token})
self.response.headers['Content-Type'] = "application/json"
self.response.out.write(serialized)
What's my error and what can I do? Thanks!

It seems that Channel API works on localhost different way than on GAE hosting. I uploaded it to the cloud and it works well now. Though it looks like it working fine on the local computer, it shows permanent JS error repeating in the error log.

You could try removing the handler argument and adding the handlers as methods of the socket object i.e. socket.onopen = function() {}; etc. That worked for me. But you are right. According to this, you should be able to get this working by using the handler argument. Hmm.