We Keep Coding

How to split array and use several requests.get in parallel with python?

My originale requeste is:
def get_foo_by_bars(authorisation_token: str, bar_ids: list):
r = requests.get(BASE_URL + "/api/v1/foo/bar",
params={"bar_ids": bar_ids, "data_type": "Float"},
headers={"Authorization": authorisation_token})
if r.status_code == 200:
return r.json()["data"]["data"]
My problem is bar_ids size contain more 80 element so my url size is more 2048 char. I want to be able to launch several requests in parallel with for example 10 bar_id then do a merge of the x responses at the end before the return.

That might be possible via asyncio + aiohttp. Unfortunatly I have no API to test this against right now, so the following code might have some issues, but should at least give you an idea:
import asyncio
import aiohttp
# async function to get json result for subset of bar_ids
async def get(session, **kwargs):
try:
async with session.get(**kwargs) as response:
await response.read()
if response.status == 200:
return await response.json()
return {}
except Exception as exc:
print(f"ERROR:\n{kwargs}\n{exc}")
return {}
# async function to split bar_ids into subsets, get their result and join them to the final result
async def main(bar_ids, package_size):
async with aiohttp.ClientSession() as session:
packaged_kwargs = [{
"url": BASE_URL + "/api/v1/foo/bar",
"params": {"bar_ids": bar_ids[i:i + package_size], "data_type": "Float"},
"headers": {"Authorization": AUTHORIZATION_TOKEN},
} for i in range(0, len(bar_ids), package_size)]
json_list = await asyncio.gather(*[get(session, **kwargs) for kwargs in packaged_kwargs])
result = {key: value for json_dict in json_list for key, value in json_dict.items()}
print(result)
# parameters
BASE_URL = "https://www.google.com"
AUTHORIZATION_TOKEN = "823ljf9823puj8ö3"
bar_ids = list(range(100))
package_size = 10
# run
asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy()) # only required for windows
asyncio.run(main(bar_ids,package_size))

Editing an ephemeral message posted by a slack bot

I've been working on creating a slack bot. I have been working on a command that can set the status of all users in a channel by recording some data into a database. For larger channels, this can take a while. My goal is to post a message that says "Processing" and then every second, replacing it with a variation of the word with various amounts of periods to indicate to users that the command indeed went through but was just computing. However, I have been encountering 2 issues.
Here is what my code currently looks like:
#app.route('/set', methods=['POST'])
def set():
data = request.form
user_id = data.get('user_id')
response_url = data.get("response_url")
start_time = time.time()
try:
x = threading.Thread(target=set_main, args=(data,))
x.start()
except Exception:
exception = traceback.format_exc()
dealWithUnknownErrors(user_id=user_id, exception=exception)
requests.post(response_url, json={})
return ""
def set_main(data):
user_id = data.get('user_id')
text = data.get('text')
channel_id = data.get('channel_id')
response_url = data.get("response_url")
text_list = text.split()
memberData = getMemberData()
x = None
if not memberData.get(user_id).get("leadership").get("isLeadership"):
whisperAndCatchErrors(text=ACCESS_DENIED_TEXT, user_id=user_id, channel_id=channel_id)
elif len(text_list) == 0:
whisperAndCatchErrors(text=SET_UNKNOWN_COMMAND_TEXT, user_id=user_id, channel_id=channel_id)
elif len(text_list) == 1 and text_list[0].lower() == "help":
whisperAndCatchErrors(text=SET_HELP_TEXT, user_id=user_id, channel_id=channel_id)
elif text_list[0].lower() == "leadershipstatus":
x = threading.Thread(target=set_leadershipStatus_main, args=(user_id, channel_id, text_list,))
elif text_list[0].lower() == "position":
x = threading.Thread(target=set_position_main, args=(user_id, channel_id, text_list,))
elif text_list[0].lower() == "instrument":
x = threading.Thread(target=set_instrument_main, args=(user_id, channel_id, text_list,))
else:
whisperAndCatchErrors(text=SET_UNKNOWN_COMMAND_TEXT, user_id=user_id, channel_id=channel_id)
x.start() if x else None
if x:
while x.is_alive():
message = {
"replace_original": True,
"text": "Processing"
}
requests.post(response_url, json=message)
time.sleep(1)
message = {
"replace_original": True,
"text": "Processing."
}
requests.post(response_url, json=message)
time.sleep(1)
message = {
"replace_original": True,
"text": "Processing.."
}
requests.post(response_url, json=message)
time.sleep(1)
message = {
"replace_original": True,
"text": "Processing..."
}
requests.post(response_url, json=message)
time.sleep(1)
message = {
"delete_original": True
}
requests.post(response_url, json=message)
The first issue that I have been running into is that the messages are not replacing the previous messages. They instead just post the messages into the channel, one after the other. I am unsure why they do not edit the preceding message.
The second issue is that I cannot use requests.post(response_url, json=message) more than 5 times before it stops allowing responses to be posted to the server (source), but ephemeral messages cannot be edited using the slack API (source).
Is there a way to add this "animation"? Also, is there a reason that my responses are not editing the previous messages? If I can fix that, then I could at least ditch the "animation" stuff and instead just post a single message to not go over the response limit.
Thanks for the help!

Pytest always fails the second time

Hi thank u for your time,
My PyTest always fail the second time on put request.
But when I try testing put using Postman, the issue didn't occur, I manage to put several requests consecutively.
But on PyTest testing, it is successful the first time but always fail the second time. I need to do modifications on JSON request every time for the put test to be successful. am really confuse, I don't know how to debug this.
I am using :
python 3.10
flask 2.1.2
sqlalchemy 1.4.39
#pytest.mark.asyncio
async def test_put_student():
randomFavSub = ""
randomFavSub = randomFavSub.join(random.choice(letters) for i in range(10))
request_dict = {
"user_id": "0e4c1d44-04f6-4a26-a02d-8e67a91b00f1",
"fav_sub": "Subject" + randomFavSub
}
headers = {
'content-type': 'application/json',
'Accepts': 'application/json'
}
async with aiohttp.ClientSession(headers=headers) as session:
await asyncio.sleep(5)
async with session.put(URL + GLOBAL_ID, json=request_dict) as response:
if response.status == 200:
data = await response.json()
data_student = data['student']
data_student_first = data_student[0]
else:
data = await response.text()
assert False, "modify Failure response is text " + str(response.status)
# TODO #38 Generalize the assert to all conditions
for key, value in request_dict.items():
assert (value == data_student_first[key]), "modify FAILURE " + key
The GLOBAL_ID is retrieved from here
#pytest.mark.asyncio
async def test_post_student():
request_dict = {
"user_id": "0e4c1d44-04f6-4a26-a02d-8e67a91b00f1",
"fav_sub": "Science"
}
global GLOBAL_ID
async with aiohttp.ClientSession() as session:
async with session.post(URL, json=request_dict) as response:
if response.status == 200:
data = await response.json()
data_student = data['student']
data_student_first = data_student[0]
GLOBAL_ID = data_student_first['id']
assert GLOBAL_ID, "GLOBAL_ID couldn't be created"
else:
data = await response.text()
assert False, 'retrieve Failure response is text'
for key, value in request_dict.items():
assert value == data_student_first[key], "create FAILURE key"
I run post, get, update, delete request in that order.But only the update seems to fail.

Actually its a mistake from my part, in the put I have two sessions running causing the conflict. If others want details. I can explain further more.

How to reconnect after a ServerDisconnectedError or ServerTimeoutError in "await asyncio.gather(*tasks)" when using aiohttp.ClientSession()?

So, I'm trying to repeat a failed loop, because of errors such as ServerDisconnectedError and ServerTimeoutError, but I not getting it. Basically, what I'm doing is sending product codes to an endpoint that will return me some information about each one. I'm using asyncio.ensure_future to create the tasks and await asyncio.gather(*tasks) do the tasks (I'm not really familiar with asyncio and aiohttp functions, so I might be wrong in this explanation).
The ETL is running perfectly, the problem is the bigger is the chunk_size (amount of product code to be sent to the endpoint) the higher is the probability to get a ServerDisconnectedError or ServerTimeoutError during the loop. I can catch both of these two exceptions, the problem is, once I repeat the loop with "continue" it looks like the repeated loop doesnt work the properly (taking just a few time to send the same chunk of codes that took a good time running the first time), continually showing me the same excepetions I got the first time of the round. I don't know what to do, if I must reconnect the server once disconnected or anything else. I need to repeat the loop that failed and send again the chunk of product codes to the endpoint, as it did for the first time, in an attempt to acquire the information from this endpoint. Could anybody help with that? I hope I have been clear in the explanation of the problem.
So, this is the code:
async def get_all_balance(session, url, headers, product_code, company_code, balance_code):
async with session.post(url,
data=json.dumps(
{'products': [product_code], 'companies': company_code, 'cdBalance': balance_code, 'inStock': 1, 'inSalesOrder': 1}),
headers=headers) as resp:
if resp.status == 200:
data = await resp.read()
product_balance = json.loads(data.decode("utf-8"))
for data in product_balance['saldos']:
lst_balance.append({
"cd_SKU": data.get('cdSKU'),
"cd_product": data.get('cdProduct'),
"cd_reference": data.get('cdReference'),
"cd_color": data.get('cdColor'),
"ds_size": data.get('dsSize'),
"cd_company": data.get('cdCompany'),
"nr_CNPJ_company": data.get('nrCNPJCompany'),
"qt_stock": data.get('qtStock'),
"qt_sales_order": data.get('qtSalesOrder'),
"cd_balance": codigo_balance
})
else:
print("Resp.Status: {}".format(resp.status))
async def main():
token = get_token() # Get a token to access the endpoint
headers = {
'Authorization': 'Bearer ' + token,
'Content-Type': 'application/json'
}
async with aiohttp.ClientSession() as session:
lst_dict_cd_product = get_product_balance() # Get a list of dicts containing the product codes such as [{"cdProduto": "13889"}, {"cdProduto": "059788"}, ...]
print("Qt products: " + str(len(lst_dict_cd_product)))
tasks_balance = []
lst_companies = [{"cdCompany": 1}, {"cdCompany": 2}, ...]
lst_codigo_balance = [1, 11, 14] # A balance code to be sent to the endpoint as a part of the payload
chunk_size = 30000
count = 0
num_chunks = len(lst_dict_cd_product) / chunk_size
mod_chunks = len(lst_dict_cd_product) % chunk_size
# print(num_chunks)
# print(mod_chunks)
if mod_chunks != 0:
print(int(num_chunks) + 1)
else:
print(int(num_chunks))
for balance_code in lst_balance_code:
if balance_code == 1:
company_code= lst_companies[0:]
print(company_code)
else:
company_code= lst_companies[0:1]
print(company_code)
for start in range(0, len(lst_dict_cd_product), chunk_size):
end = start + chunk_size
# print(lst_dict_cd_product[start:end]) # item chunks
lst_product_code = lst_dict_cd_product[start:end]
print("Chunks : {} of {} round {} s:e{}".format(
start, end, count, str(lst_codigo_product[0]) + ":" + str(lst_codigo_product[-1])))
if count != 0:
time.sleep(60)
k=0
while k < 1:
for product_code in lst_codigo_product:
url = 'https://www30.bhan.com.br:9443/api/v1/produto/saldoproduto'
balance_tasks.append(asyncio.ensure_future(
get_all_balance(session, url, headers, product_code, company_code, balance_code))) # emp_all[0:]
try:
await asyncio.gather(*balance_tasks)
count += 1
k += 1
except Exception as e:
continue
# Save a json file
asyncio.run(main())
```python

Make an infinite loop:
import time
connection = None
while(True):
connection = try_connect()
if connection:
break # Get out of the loop.
time.sleep(2) # Seconds.
do_someting(connection)
EDIT:
def try_connect():
try:
connection = ... # get connection.
except [failed_connection_exception_type]:
return False
else:
return connection
EDIT 2:
Short version
import time
connection = None
while(True):
try:
connection = ... # get connection.
except [exception_type_to_catch]:
time.sleep(2) # Seconds.
continue
else:
break
do_someting(connection)

Why is my websocket request "unauthorized" only when written in OOP, when the same works perfectly when written with just functions?

I'm writing a Python program that does some trading automation. The API I work with is from Deribit, whose preferred transport mechanism is Websocket. I'm a complete newbie to Python's websockets and asyncio modules.
Here's the code I first wrote for authenticating my client and then sending a separate private message to get an order position from the account, written only with functions and no classes:
import asyncio
import websockets
import json
CL_ID = 'qxv0EeAu'
CL_SECRET = 't24F49ocH1_qFawiKnEyqlWF5D-haABb31O8xCQhySg'
REQ_URL = 'wss://test.deribit.com/ws/api/v2'
acc_token = ''
msg = {
"jsonrpc": "2.0",
"id": 1,
"params": {}
}
async def auth_api():
global msg
global acc_token
msg["method"] = "public/auth"
msg["params"] = {
"grant_type": "client_credentials",
"client_id": CL_ID,
"client_secret": CL_SECRET,
"scope": "session:test"
}
async with websockets.connect(REQ_URL) as websocket:
await websocket.send(json.dumps(msg))
while websocket.open:
response = await websocket.recv()
response_json = json.loads(response)
acc_token = response_json["result"]["access_token"]
return
async def get_position(websocket, instrument):
global msg
global acc_token
msg["id"] += 1
msg["method"] = "private/get_position"
msg["params"] = {
"access_token": acc_token,
"instrument_name": instrument
}
await websocket.send(json.dumps(msg))
while websocket.open:
response = await websocket.recv()
return response
async def main():
global msg
await auth_api()
async with websockets.connect(REQ_URL) as websocket:
response = await get_position(websocket, "BTC-PERPETUAL")
print(response)
asyncio.get_event_loop().run_until_complete(main())
It works perfectly fine. Here's my result:
{"jsonrpc":"2.0","id":2,"result":{"total_profit_loss":0.000209124,"size_currency":-0.017402402,"size":-150.0,"settlement_price":8649.9,"realized_profit_loss":2.67e-7,"open_orders_margin":0.0,"mark_price":8619.5,"maintenance_margin":0.000100079,"leverage":100,"kind":"future","instrument_name":"BTC-PERPETUAL","initial_margin":0.000174039,"index_price":8619.45,"floating_profit_loss":0.000061161,"estimated_liquidation_price":-14.95,"direction":"sell","delta":-0.017402402,"average_price":8724.34},"usIn":1573756522511975,"usOut":1573756522512240,"usDiff":265,"testnet":true}
I decided to rewrite it the OOP way, and here's the class I created (the file is named "Call_Deribit"):
import asyncio, websockets, json
class WSClient():
def __init__(self, key=None, secret=None, url=None):
self.api_key = key
self.api_secret = secret
self.msg = {
"jsonrpc": "2.0",
"id": 0
}
if url:
self.host = url
else:
self.host = 'wss://test.deribit.com/ws/api/v2'
async def call_api(self, msg):
async with websockets.connect(self.host) as websocket:
print("Connected to URL:", self.host)
try:
await websocket.send(msg)
while websocket.open:
response = await websocket.recv()
response_json = json.loads(response)
return response_json
except Exception as e:
return e
def request(self, method, params, session=None):
msg = self.msg
msg["id"] += 1
msg["method"] = method
msg["params"] = params
if session != None:
msg["params"]["scope": "session:{}".format(session)]
return asyncio.get_event_loop().run_until_complete(self.call_api(json.dumps(msg)))
def get_order_book(self, instrument):
method = "public/get_order_book"
params = {
"instrument_name": instrument
}
return self.request(method, params)
And here's the main file I'm accessing the class from and where I make all the requests:
import json, asyncio, websockets
from Call_Deribit import WSClient
CL_ID = 'qxv0EeAu'
CL_SECRET = 't24F49ocH1_qFawiKnEyqlWF5D-haABb31O8xCQhySg'
REQ_URL = 'wss://test.deribit.com/ws/api/v2'
method_auth = "public/auth"
params_auth = {
"grant_type": "client_credentials",
"client_id": CL_ID,
"client_secret": CL_SECRET
}
main_client = WSClient(key=CL_ID, secret=CL_SECRET, url=REQ_URL)
auth_response = main_client.request(method_auth, params_auth)
acc_token = auth_response["result"]["access_token"]
method_pos = "private/get_position"
params_pos = {
"access_token": acc_token,
"instrument_name": "BTC-PERPETUAL"
}
position = main_client.request(method_pos, params_pos)
print(position)
The first request for authentication is working this time, and I'm able to extract the access token as well, but the second private/get_position message is, for whatever reason, returning an unauthorized error.
{'jsonrpc': '2.0', 'id': 1, 'error': {'message': 'unauthorized', 'code': 13009}, 'testnet': True, 'usIn': 1573756936534405, 'usOut': 1573756936534629, 'usDiff': 224}
I've spent hours on it, and I seem to be doing exactly the same thing in the OOP version as I did on the original one. My familiarity with OOP and its concepts (such as inheritance) is limited, so I'd like to know what I'm missing here, and why my code isn't working in the OOP version, despite following the same exact workflow as in the original version.
Here's the documentation for the Deribit API: https://docs.deribit.com/v2/?python#json-rpc
Any help would be greatly appreciated.

Adding the scope under params_auth in the main file works:
params_auth = {
"grant_type": "client_credentials",
"client_id": CL_ID,
"client_secret": CL_SECRET,
"scope": "session:test"
}