We Keep Coding

Python/Quart: how to send while awaiting a response?

Using Python 3.9 and Quart 0.15.1, I'm trying to create a websocket route that will listen on a websocket for incoming request data, parse it, and send outbound response data to a client, over and over in a loop - until and unless the client sends a JSON struct with a given key "key" in the payload, at which point we can move on to further processing.
I can receive the initial inbound request from the client, parse it, and send outbound responses in a loop, but when I try to gather the second payload to parse for the presence of "key", things fall apart. It seems I can either await websocket.send_json() or await websocket.receive(), but not both at the same time.
The Quart docs suggest using async-timeout to (https://pgjones.gitlab.io/quart/how_to_guides/request_body.html?highlight=timeout) to timeout if the body of a request isn't received in the desired amount of time, so I thought I'd try to send messages in a while loop, with a brief period of time spent in await websocket.receive() before timing out if a response wasn't receive()'d:
#app.websocket('/listen')
async def listen():
payload_requested = await websocket.receive()
parsed_payload_from_request = json.loads(payload_requested)
while "key" not in parsed_payload_from_request:
response = "response string"
await websocket.send_json(response)
async with timeout (1):
payload_requested = await websocket.receive()
parsed_payload_from_request = json.loads(payload_requested)
if "key" == "present":
do_stuff()
...but that doesn't seem to work, an asyncio.exceptions.CancelledError is thrown by the timeout.
I suspect there's a better way to accomplish this using futures and asyncio, but it's not clear to me from the docs.

I think your code is timing out waiting for a message from the client. You may not need it in this case.
I've tried to write out code as you've described your needs and got this,
#app.websocket('/listen')
async def listen():
while True:
data = await websocket.receive_json()
if "key" in data:
await websocket.send_json({"key": "response"})
else:
do_stuff()
return # websocket closes
does it do what you want, if not what goes wrong?

How to decode Arrow Flight `FlightData` with a pure gRPC client

I came across a situation where we need to use a plain gRPC client (through the grpc.aio API) to talk to an Arrow Flight gRPC server.
The DoGet call did make it to the server, and we have received a FlightData in response. If our understanding of the Flight gRPC definition is correct, the response contains a flatbuffers message that can somehow be decoded into a RecordBatch.
Following, is the client-side code,
import asyncio
import pathlib
import grpc
import pyarrow as pa
import pyarrow.flight as pf
import flight_pb2, flight_pb2_grpc
async def main():
ticket = pf.Ticket("tick")
sock_file = pathlib.Path.cwd().joinpath("arena.sock").resolve()
async with grpc.aio.insecure_channel(f"unix://{sock_file}") as channel:
stub = flight_pb2_grpc.FlightServiceStub(channel)
async for data in stub.DoGet(flight_pb2.Ticket(ticket=ticket.ticket)):
assert type(data) is flight_pb2.FlightData
print(data)
# How to convert data into a RecordBatch?
asyncio.run(main())
Currently we stuck on this last step of decoding the FlightData response.
The question is two fold,
are there some existing facilities form pyarrow.flight that we can use to decode a python grpc object of the FlightData type;
if #1 is not possible, what are some other options to decode the content of the FlightData and reconstruct a RecordBatch from scratch?
The main interest here is to use the AsyncIO of plain gRPC client. Supposedly, this is not feasible with the current version of Arrow Flight gRPC client.

There is indeed no utility exposed in pyarrow.flight for this.
ArrowData contains, among other things, the Arrow IPC header and body. So you can instead decode it using pyarrow.ipc. Here's an example:
import asyncio
import pathlib
import struct
import grpc
import pyarrow as pa
import pyarrow.flight as pf
import Flight_pb2, Flight_pb2_grpc
async def main():
ticket = pf.Ticket("tick")
async with grpc.aio.insecure_channel("localhost:1234") as channel:
stub = Flight_pb2_grpc.FlightServiceStub(channel)
schema = None
async for data in stub.DoGet(Flight_pb2.Ticket(ticket=ticket.ticket)):
# 4 bytes: Need IPC continuation token
token = b'\xff\xff\xff\xff'
# 4 bytes: message length (little-endian)
length = struct.pack('<I', len(data.data_header))
buf = pa.py_buffer(token + length + data.data_header + data.data_body)
message = pa.ipc.read_message(buf)
print(message)
if schema is None:
# This should work but is unimplemented
# print(pa.ipc.read_schema(message))
schema = pa.ipc.read_schema(buf)
print(schema)
else:
batch = pa.ipc.read_record_batch(message, schema)
print(batch)
print(batch.to_pydict())
asyncio.run(main())
Server:
import pyarrow.flight as flight
import pyarrow as pa
class TestServer(flight.FlightServerBase):
def do_get(self, context, ticket):
table = pa.table([[1,2,3,4]], names=["a"])
return flight.RecordBatchStream(table)
TestServer("grpc://localhost:1234").serve()
There's some discussion about async Flight APIs, please join the dev# mailing list if you would like to chime in.

Webhook status code 400: {"embeds": ["Must be 10 or fewer in length."]}

I´m developing a bot that sends messages to the discord channel every second, however when I run the program and only sent 10 messages and after only print [1/1] Webhook status code 400: {"embeds": ["Must be 10 or fewer in length."]}. I don't find why is this happening, I'm using discord_webhook library and python to do this. Here is my code
async def __scrape_api(session, pid):
async with session.get(API_URL + pid) as response:
data = await response.json()
print(f"scrape {data}")
if not __search_product(pid):
name = data["name"]
image_url = data['skus'][0]['image']
for size in data['skus']:
if size['available']:
print("sent")
message = DiscordEmbed(title=f'{name}', url=f'{PRODUCT_URL_0}{size["sku"]}{PRODUCT_URL_1}',
description=f'talla: {size["dimensions"]["Tallas Calzado"]}\nPrecio: {size["bestPriceFormated"]}')
message.set_thumbnail(url=image_url)
message.set_timestamp()
webhook.add_embed(message)
response = webhook.execute()
time.sleep(1)
# save to database
__insert_new_product(pid, name, image_url, data['available'])

I found the solution for this after few minutes of taking a deeper look inside it's code. Basically each embed you send over is being held in the webhook object, once you made 10 requests with embeds then it gives you that error. All you have to do is:
webhook.execute(remove_embeds=True)

Threads can only be started once in Django Channels

I created a simple Django Channels consumer that should connects to an external source, retrieve data and send it to the client. So, the user opens the page > the consumer connects to the external service and gets the data > the data is sent to the websocket.
Here is my code:
import json
from channels.generic.websocket import WebsocketConsumer, AsyncConsumer, AsyncJsonWebsocketConsumer
from binance.client import Client
import json
from binance.websockets import BinanceSocketManager
import time
import asyncio
client = Client('', '')
trades = client.get_recent_trades(symbol='BNBBTC')
bm = BinanceSocketManager(client)
class EchoConsumer(AsyncJsonWebsocketConsumer):
async def connect(self):
await self.accept()
await self.send_json('test')
bm.start_trade_socket('BNBBTC', self.process_message)
bm.start()
def process_message(self, message):
JSON1 = json.dumps(message)
JSON2 = json.loads(JSON1)
#define variables
Rate = JSON2['p']
Quantity = JSON2['q']
Symbol = JSON2['s']
Order = JSON2['m']
asyncio.create_task(self.send_json(Rate))
print(Rate)
This code works when i open one page; if i try to open a new window with a new account, though, it will throw the following error:
File "C:\Users\User\Desktop\Heroku\github\master\myapp\consumers.py", line 54, in connect
bm.start()
File "C:\Users\User\lib\threading.py", line 843, in start
raise RuntimeError("threads can only be started once")
threads can only be started once
I'm new to Channels, so this is a noob question, but how can i fix this problem? What i wanted to do was: user opens the page and gets the data, another user opens the page and gets the data; is there no way to do that? Or am i simply misunderstanding how Django Channels and websockets works?

Do you really need a secondary thread ?
class EchoConsumer(AsyncJsonWebsocketConsumer):
symbol = ''
async def connect(self):
self.symbol = 'BNBBTC'
# or, more probably, retrieve the value for "symbol" from query_string
# so the client can specify which symbol he's interested into:
# socket = new WebSocket("ws://.../?symbol=BNBBTC");
await self.accept()
def process_message(self, message):
# PSEUDO-CODE BELOW !
if self.symbol == message['symbol']:
await self.send({
'type': 'websocket.send',
'text': json.dumps(message),
})
For extra flexibility, you might also accept al list of symbols from the client, instead:
//HTML
socket = new WebSocket("ws://.../?symbols=XXX,YYY,ZZZ");
then (in the consumer):
class EchoConsumer(AsyncJsonWebsocketConsumer):
symbols = []
async def connect(self):
# here we need to parse "?symbols=XXX,YYY,ZZZ" ...
# the code below has been stolen from another project of mine and should be suitably adapted
params = urllib.parse.parse_qs(self.scope.get('query_string', b'').decode('utf-8'))
try:
self.symbols = json.loads(params.get('symbols', ['[]'])[0])
except:
self.symbols = []
def process_message(self, message):
if message['symbol'] in self.symbols:
...

I'm no Django developer, but if I understand correctly, the function connect is being called more than once-- and bm.start references the same thread most likely made in bm.start_trade_socket (or somewhere else in connect). In conclusion, when bm.start is called, a thread is started, and when it is done again, you get that error.

Here start() Start the thread’s activity.
It should be called at most once per thread object. You have made a global object of BinanceSocketManager as "bm".
It will always raise a RuntimeError if called more than once on the same thread object.
Please refer the below mentioned code, it may help you
from channels.generic.websocket import WebsocketConsumer, AsyncConsumer, AsyncJsonWebsocketConsumer
from binance.client import Client
import json
from binance.websockets import BinanceSocketManager
import time
import asyncio
class EchoConsumer(AsyncJsonWebsocketConsumer):
client = Client('', '')
trades = client.get_recent_trades(symbol='BNBBTC')
bm = BinanceSocketManager(client)
async def connect(self):
await self.accept()
await self.send_json('test')
self.bm.start_trade_socket('BNBBTC', self.process_message)
self.bm.start()
def process_message(self, message):
JSON1 = json.dumps(message)
JSON2 = json.loads(JSON1)
#define variables
Rate = JSON2['p']
Quantity = JSON2['q']
Symbol = JSON2['s']
Order = JSON2['m']
asyncio.create_task(self.send_json(Rate))
print(Rate)

How can I download the chat history of a group in Telegram?

I would like to download the chat history (all messages) that were posted in a public group on Telegram. How can I do this with python?
I've found this method in the API https://core.telegram.org/method/messages.getHistory which I think looks like what I'm trying to do. But how do I actually call it? It seems there's no python examples for the MTproto protocol they use.
I also looked at the Bot API, but it doesn't seem to have a method to download messages.

You can use Telethon. Telegram API is fairly complicated and with the telethon, you can start using telegram API in a very short time without any pre-knowledge about the API.
pip install telethon
Then register your app (taken from telethon):
the link is: https://my.telegram.org/
Then to obtain message history of a group (assuming you have the group id):
chat_id = YOUR_CHAT_ID
api_id=YOUR_API_ID
api_hash = 'YOUR_API_HASH'
from telethon import TelegramClient
from telethon.tl.types.input_peer_chat import InputPeerChat
client = TelegramClient('session_id', api_id=api_id, api_hash=api_hash)
client.connect()
chat = InputPeerChat(chat_id)
total_count, messages, senders = client.get_message_history(
chat, limit=10)
for msg in reversed(messages):
# Format the message content
if getattr(msg, 'media', None):
content = '<{}> {}'.format( # The media may or may not have a caption
msg.media.__class__.__name__,
getattr(msg.media, 'caption', ''))
elif hasattr(msg, 'message'):
content = msg.message
elif hasattr(msg, 'action'):
content = str(msg.action)
else:
# Unknown message, simply print its class name
content = msg.__class__.__name__
text = '[{}:{}] (ID={}) {}: {} type: {}'.format(
msg.date.hour, msg.date.minute, msg.id, "no name",
content)
print (text)
The example is taken and simplified from telethon example.

With an update (August 2018) now Telegram Desktop application supports saving chat history very conveniently.
You can store it as json or html formatted.
To use this feature, make sure you have the latest version of Telegram Desktop installed on your computer, then click Settings > Export Telegram data.
https://telegram.org/blog/export-and-more

The currently accepted answer is for very old versions of Telethon. With Telethon 1.0, the code can and should be simplified to the following:
# chat can be:
# * int id (-12345)
# * str username (#chat)
# * str phone number (+12 3456)
# * Peer (types.PeerChat(12345))
# * InputPeer (types.InputPeerChat(12345))
# * Chat object (types.Chat)
# * ...and many more types
chat = ...
api_id = ...
api_hash = ...
from telethon.sync import TelegramClient
client = TelegramClient('session_id', api_id, api_hash)
with client:
# 10 is the limit on how many messages to fetch. Remove or change for more.
for msg in client.iter_messages(chat, 10):
print(msg.sender.first_name, ':', msg.text)
Applying any formatting is still possible but hasattr is no longer needed. if msg.media for example would be enough to check if the message has media.
A note, if you're using Jupyter, you need to use async directly:
from telethon import TelegramClient
client = TelegramClient('session_id', api_id, api_hash)
# Note `async with` and `async for`
async with client:
async for msg in client.iter_messages(chat, 10):
print(msg.sender.first_name, ':', msg.text)

Now, you can use TDesktop to export chats.
Here is the blog post about Aug 2018 update.
Original Answer:
Telegram MTProto is hard to use to newbies, so I recommend telegram-cli.
You can use third-party tg-export script, but still not easy to newbies too.

You can use the Telethon library. for this you need to register your app and connect your client code to it (look at this).
Then to obtain message history of a entry (such as channel, group or chat):
from telethon.sync import TelegramClient
from telethon.errors import SessionPasswordNeededError
client = TelegramClient(username, api_id, api_hash, proxy=("socks5", proxy_ip, proxy_port)) # if in your country telegram is banned, you can use the proxy, otherwise remove it.
client.start()
# for login
if not client.is_user_authorized():
client.send_code_request(phone)
try:
client.sign_in(phone, input('Enter the code: '))
except SessionPasswordNeededError:
client.sign_in(password=input('Password: '))
async for message in client.iter_messages(chat_id, wait_time=0):
messages.append(Message(message))
# write your code