We Keep Coding

How can I actually grab the data out of a running stream/websocket within a function?

I am quite confused how to handle websockets/streams data within a function in Python.
I have a Python script that triggers a ws/stream which runs 24/7:
async def quote_callback(q):
print('quote', q)
return q
# Initiate Class Instance
stream = Stream(public_key,
secret_key,
base_url=base_url,
data_feed='sip')
# Subscribing to all relevant symbols
for symbol in traded_symbols:
stream.subscribe_quotes(quote_callback, symbol)
# run stream
stream.run()
This returns an infinite stream of quotes for all symbols in traded_symbols:
quote Quote({ 'ask_exchange': 'N',
'ask_price': 79.0,
'ask_size': 5,
'bid_exchange': 'U',
'bid_price': 78.99,
'bid_size': 3,
'conditions': ['R'],
'symbol': 'MRK',
'tape': 'A',
'timestamp': 1641496334124695884})
quote Quote({ 'ask_exchange': 'H',
'ask_price': 334.64,
'ask_size': 1,
'bid_exchange': 'A',
'bid_price': 334.48,
'bid_size': 1,
'conditions': ['R'],
'symbol': 'FB',
'tape': 'C',
'timestamp': 1641496334125929728})
... etc.
Now I have a function within this script where I want to grab the (last) next value out of the stream for any given symbol like so:
def get_quote(symbol):
ask_price = # stream to catch the (last) occurring ask_price of symbol
return ask_price
get_quote("FB")
# should return --> 334.64
How to accomplish this?
It seems that all of the code after stream.run isn't executed when triggered via main.py while loop?
# trader.py
# Stream logic
quotes = {}
async def quote_callback(q):
quotes[q.symbol] = q
print(q)
return q
# Initiate Class Instance
stream = Stream(public_key,
secret_key,
base_url=base_url,
data_feed='sip') # <- replace to SIP if you have PRO subscription
# Subscribing to all symbols
for symbol in traded_symbols:
stream.subscribe_quotes(quote_callback, symbol)
# run stream
stream.run()
def run_rufus():
...
stream_price = quotes[symbol].ask_price
# main.py
from trader import run_rufus
while True:
run_rufus()
When I remove all of the stream part run_rufus() is continuously executed. I guess there is some bug with my code structure.
Update
import multiprocessing
import os
from alpaca_trade_api.stream import Stream
# Grab required env variables
secret_key = os.environ['SECRET_KEY']
public_key = os.environ['PUB_KEY']
base_url = os.environ['APCA_API_BASE_URL']
traded_symbols = [
"ADBE", # Adobe
"ABNB", # Airbnb
"AMZN", # Amazon
"FB"
]
quotes = {}
async def quote_callback(q):
quotes[q.symbol] = q
print(q)
print(F'-----------------------------')
print(F'Quotes: {quotes}')
return q
# Initiate Class Instance
stream = Stream(public_key,
secret_key,
base_url=base_url,
data_feed="sip")
# Subscribing to all relevant symbols
for symbol in traded_symbols:
stream.subscribe_quotes(quote_callback, symbol)
# run stream
p = multiprocessing.Process(target=stream.run)
p.start()
def get_quote(symbol):
if symbol in quotes: # if the dictionary quotes has a key with the value of symbol
return quotes[symbol].ask_price
return -1 # a return value of -1 indicates that there was no quote with the given symbol received yet
price = get_quote("FB")
while price == -1:
price = get_quote("FB")
print(price)
returns
Quotes: {'ABNB': Quote({ 'ask_exchange': 'M',
'ask_price': 156.7,
'ask_size': 1,
'bid_exchange': 'Q',
'bid_price': 156.57,
'bid_size': 2,
'conditions': ['R'],
'symbol': 'ABNB',
'tape': 'C',
'timestamp': 1641840163168672614}), 'ADBE': Quote({ 'ask_exchange': 'P',
'ask_price': 517.0,
'ask_size': 2,
'bid_exchange': 'P',
'bid_price': 516.79,
'bid_size': 1,
'conditions': ['R'],
'symbol': 'ADBE',
'tape': 'C',
'timestamp': 1641840163466840647}), 'FB': Quote({ 'ask_exchange': 'Q',
'ask_price': 323.97,
'ask_size': 3,
'bid_exchange': 'V',
'bid_price': 323.88,
'bid_size': 3,
'conditions': ['R'],
'symbol': 'FB',
'tape': 'C',
'timestamp': 1641840160572326189}), 'AMZN': Quote({ 'ask_exchange': 'M',
'ask_price': 3178.2,
'ask_size': 2,
'bid_exchange': 'M',
'bid_price': 3175.81,
'bid_size': 2,
'conditions': ['R'],
'symbol': 'AMZN',
'tape': 'C',
'timestamp': 1641840162908516864})
}

First, you can directly access the dictionary fields as members of the Quote objects - i.e. you can just do q.ask_price to get the price from the received Quotes.
In your quote_callback function you get every Quote object as an argument. So, if you always only want to access the last received Quote object which has symbol = "HB", you could define a global dictionary named e.g. quotes which will contain one key == symbol and value == last Quote with that symbol per symbol type.
Now, in get_quote you can use a simple dictionary lookup for the desired key in the quotes dictionary. In the end, you need to repeatedly call get_quote until a quote object with the desired symbol value is received. The whole code could look like this:
import multiprocessing
quotes = {}
async def quote_callback(q):
quotes[q.symbol] = q
return q
# Initiate Class Instance
stream = Stream(public_key,
secret_key,
base_url=base_url,
data_feed="sip")
# Subscribing to all relevant symbols
for symbol in traded_symbols:
stream.subscribe_quotes(quote_callback, symbol)
# run stream
p = multiprocessing.Process(target=stream.run)
p.start()
def get_quote(symbol):
if symbol in quotes.keys():
return quotes[symbol].ask_price
print(quotes)
return -1
price = get_quote("FB")
while price == -1:
price = get_quote("FB")
print(price)
If there are any errors, please add them to your question, then I can take a look.

Python only encryption/obfuscation

I'm looking for simple password-based obfuscation/security of strings.
I've pretty much gone over each example of > Simple way to encode a string according to a password?
And none of them work with my python 3.7.
I got the error with ord() so I updated the code, but even after, its still broken. For examle:
from itertools import cycle
def encode_zip_cycle(key, clear):
enc = [chr((ord(clear_char) + ord(key_char)) % 256)
for clear_char, key_char in zip(clear, cycle(key))]
return base64.urlsafe_b64encode("".join(enc).encode())
def decode_zip_cycle(key, enc):
enc = base64.urlsafe_b64decode(enc)
dec = [chr((256 + enc_char - ord(key_char)) % 256)
for enc_char, key_char in zip(enc, cycle(key))]
print(dec)
return "".join(dec)
text = "ATTACKATONCEfor Live 2154125-21-512^!££613-123!"
s = "1235348udgfjff"
print("Text : " + text)
print("Shift : " + str(s))
print("Cipher: ", encode_zip_cycle(s, text)) # , type(encode(s, text)))
print("Original text: ", decode_zip_cycle(s, encode_zip_cycle(s, text)))
Gives me
Text : ATTACKATONCEfor Live 2154125-21-512^!££613-123!
Shift : 1235348udgfjff
Cipher: b'csKGwod2dn95w4nCs8K1wqnCr8OMw5XCo1J_wp7CqcKZWMKVwoTCmcKXwp_CmsKXY2dgZ2RhbcKmwpbDhcKHDQnCnGJlYGZlZ1k='
['A', '\x90', 'S', '\x8d', 'T', 'B', '>', '\n', '\x15', '\\', '#', 'X', 'M', '\\', '\x84', '\x90', 'v', '\x8d', '|', '\x8f', 'T', 'N', '1', '[', '=', 'è', '\x19', '\\', 'm', '\x90', 'v', '\x8d', 'f', '$', '\x8a', ' ', '^', '\x1d', '\\', '/', '\\', '1', '\x91', 'm', '\x8f', 'e', '\x8f', 'c', '+', 'ò', 'ü', '\x00', 'þ', '÷', '\x07', '\\', 'u', '\x90', 'c', '\x8e', 'R', '\x8e', 'O', '\x98', '¥', '[', '6', 'ø', 'ÿ', 'ú', '5', '3', '4', '$']
Original text: ASTB>
\#XM\v|TN1[=è\mvf$ ^\/\1mec+òü þ÷\ucRO¥[6øÿú534$

In encode_zip_cycle you encode the "encrypted" string into utf-8 before doing the second encoding into base64. Yet, you don't revert this operation later in decode_zip_cycle.
This is the correct decode_zip_cycle function:
def decode_zip_cycle(key, enc):
enc = base64.urlsafe_b64decode(enc).decode()
dec = [chr((256 + ord(enc_char) - ord(key_char)) % 256)
for enc_char, key_char in zip(enc, cycle(key))]
print(dec)
return "".join(dec)

How to manually select elements from a list encoded in utf-8?

I have a list which looks like this:
['क', ',', 'म', '-', 'ह', 'औ', "'", ')', '(', 'स', '.', 'ए', 'प', 'श',
'भ', 'ल', 'य', 'न', 'इ', '}', 'ज', 'र', 'उ', 'ग', 'द', 'त', 't', 'थ',
'ब', 'अ', 'ई', 'o', '%', 'व', 'a', 'आ', '#', '–', 'q', 'i', '।', '/',
'ओ', 'फ', 'f', 's', 'u', '!', '?', 'ध', 'ऐ', '१', '+', '२', 'p', 'd',
'j', 'च', 'ऑ', 'b', 'छ', 'ऊ', 'l', 'e', 'w', 'ख', 'घ', 'c', 'r', 'y',
'g', 'n', 'ट', 'ड', 'x', '५', '"', '३', 'm', 'ठ', 'h', '४', '•', '$',
'>', 'v', 'z', 'झ', '७', '—', '६', 'k', 'ढ', '८', '&', 'ऋ', '\', '९',
'✉', '०', '॥', '°', '^', '~', '−', '·', 'ॐ', '×', '_', '→', '☆', '£',
'€', 'α', '‘', 'ष', '±', '†', 'β', '#', '\u200e', '░', '¬', '₹', 'π',
'½', '…', 'ऍ', 'º', 'σ', 'γ', 'δ', 'ऽ', '0', '²', 'ङ', 'ॠ', 'à', '≥',
'ः', 'ऎ', 'ω', 'μ', '{', 'ण', 'ं', '≈', 'ε', 'λ', 'θ', '्', '<', '↑',
'\uf0a7', 'φ', '\u200b', '📝', 'ञ', 'о', 'ƒ', '©', '←', 'ळ', 'ा', '■',
'¢', 'ρ', '∞', 'î', '⁄', '√', 'ব', '§', '¾', '≤', '॰', '্', 'é', 'و',
'`', '¥', '♂', '₩', 'å', '´', 'ü', 'á', 'ó', 'ভ', 'в', '¼', '़', 'è',
'ʁ', 'े', 'и', '≡', 'ζ', 'í', '↙', '″', '\u200d', '₫', 'م', '»', 'ː',
'‡', 'ö', 'ँ', 'د', 'η', 'ð', '♦', 'শ', 'প', 'ी', 'ú', '⅓', 'ب', '≠',
'κ', '∈', 'ç', '�', 'এ', 'উ', 'র', 'ν', 'â', 'ê', 'ū', 'к', 'ø', 'ù',
'ā', 'ä', 'æ', '↓', 'ô', 'স', 'ो', 'ō', '●', 'ē', '₨', 'ि', '„', 'ī',
'฿', 'ò', 'τ', 'ਸ', 'ऩ', 'ª', 'р', 'ত', 'н', 'с', 'χ', 'ಕ', 'ë', 'ॉ',
'خ', 'ا', 'ψ', 'а', '¿', 'ì', 'ý', 'µ', 'ौ', 'š', '‰', '÷', 'ি', 'আ',
'ə', 'у', '★', 'ل', 'ॡ', 'č', '⊕', 'ृ', 'ñ', 'ै', 'û', 'ु', 'ू', 'м',
'þ', 'г', 'ι', 'മ', 'پ', '☉', 'த', 'ш', '¤', 'е', '📥', 'ş', 'ã', 'ž',
'খ', 'ع', '水', 'õ', 'ক', 'д', 'ï', 'ج', 'ধ', 'ж', 'ऒ', 'ଓ', '¹', 'ॅ',
'ħ', 'ন', 'ও', 'ʃ', '\u200c', 'ز', 'х', 'מ', '⊂', 'ф', 'য', 'എ',
'\u202a', 'ষ', '九', '♣', 'ł', 'ऌ', 'া', 'త', 'з', 'ß', 'ف', 'ר', '―',
'п', 'غ', 'ऱ', 'ر', 'ŋ', 'ϕ', 'ऴ', 'ክ', 'ğ', 'ą', 'ś', 'ę', '¨', 'ч',
'ż', '№', 'س', 'œ', 'ă', '♠', 'ش', '◾', '\uf0fc', 'ÿ', 'ש', '\ufeff',
'ಜ', 'ن', 'ʊ', '³', 'ć', 'آ', 'চ', 'ɛ', 'я', '이', 'đ', 'জ', 'ġ', '⅜',
'ɑ', '˚', 'ξ', 'л', 'б', 'т', 'ц', '∟', 'й', 'ಭ', 'സ', 'ɒ', 'అ', 'న',
'⍨', '⌛', '⌣', 'ĝ', 'ő', 'ĉ', '،', 'ċ', 'ĵ', 'ژ', 'হ', 'ŝ', 'ণ', '沖',
'⅞', 'ت', '١', 'ɸ', 'ɪ', 'ʌ', 'ě', 'ע', '¸', 'দ', 'ŭ', '∧', 'أ', 'અ',
'ɨ', 'ĥ', '∀', 'ગ', 'ű', 'ʒ', 'ح', 'ث', '█', '∩', 'ق', '↔', '®', 'ਪ',
'⇒', '⅔', '∑', '⇔', 'ழ', 'ю', '月', 'ذ', 'ǻ', 'ń', '∪', 'ك', 'ʰ', 'ё',
'э', '™', 'ض', 'ਦ', 'ɹ', '☞', 'ঞ', 'ম', 'ু', '②', '道', 'ӏ', '大', 'щ',
'א', 'ʔ', 'ǁ', 'ה', '∂', 'ţ', 'പ', '∨', '성', 'ỳ', 'ബ', '∇', 'ظ', 'ط',
'ẽ', 'ص', 'ಶ', 'υ', 'ഞ', 'қ', 'ỹ', 'ź']
I need to select only the Devanagari characters (i.e. 'अ', 'आ'...'क','ख','घ'..) out of above characters. Is there any other way to do this in python except that of manually selecting the indices?

You can iterate through each item in the list, convert each character to a unicode number using ord, then test whether it lies within the Devanagari unicode character range (see: https://en.wikipedia.org/wiki/Devanagari_(Unicode_block)). If it does, add it to the output list.
It will end up something like this:
newlist = []
for c in list:
if ord(c) >= 2304 and ord(c) <= 2431:
newlist.append(c)

Scrape several instances of a webpage the fastest way possible

So, after a lot of attempts, search and research I give up.
I have a webpage where all employees name, phone, mail and userid can be query. The way you do that is that the request to the server needs to have at least 4 digits, with all 26 ascll character + 0-9 numbers. I was able to do it with Selenium in Python...but it whould take 20 days to go through - see code.
from selenium import webdriver
import csv
alphanum = ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm',
'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'x', 'y', 'z', '1',
'2', '3', '4', '5', '6', '7', '8', '9', '0']
driver = webdriver.Firefox()
driver.get('http://brnet.intra.corpintra.net/Quem/pessoas2/Default.asp')
list_base = dict()
i = 0
data_str = []
found = False
for first_chr in alphanum:
for second_chr in alphanum:
for third_chr in alphanum:
text = first_chr + second_chr + third_chr
element_name = driver.find_element_by_name('nome').clear()
element_name = driver.find_element_by_name('nome')
element_name.send_keys(text)
element_search = driver.find_element_by_name('B1')
element_search.click()
if driver.find_elements_by_class_name('dados'):
for table_data in driver.find_elements_by_class_name('dados'):
cells_table = table_data.find_elements_by_tag_name('td')
for cell_data in cells_table:
data_str.append(cell_data.text.strip())
if list_base:
for key, value in list_base.items():
for data in data_str:
if data in value:
found = False
else:
found = True
else:
found = False
if found is False:
list_base[i] = data_str
i = i+1
data_str = []
found = False
driver.back()
w = csv.writer(open("output.csv", "w"))
for key, value in list_base.items():
w.writerow([key, value])
driver.quit()
Is there a way to reduce the time?

Is the cc recipients in a received email a Python list? (Google App Engine)

I am trying to pull the cc'ed email addresses from received email. I am working in the development server.
The tutorial says that "cc contains a list of the cc recipients." But it seems that message.cc returns a string. I am just using the code I copied from the cookbook:
class ReceiveEmail(InboundMailHandler):
def receive(self, message):
logging.info("Received email from %s" % message.sender)
plaintext = message.bodies(content_type='text/plain')
for text in plaintext:
txtmsg = ""
txtmsg = text[1].decode()
logging.info("Body is %s" % txtmsg)
logging.info("CC email is %s" % message.cc)
So if I have 1 cc, the log shows:
CC email is cc12#example.com
If there are more than 1:
CC email is cc12#example.com, cc13#example.com
To get the first email "cc12#example.com", I tried:
logging.info("CC email is %s" % message.cc[0])
but this gives:
CC email is c
so the result is treated as a string.
When I try
logging.info("CC email is %s" % list(message.cc)
I get
['c', 'c', '1', '2', '#', 'e', 'x', 'a', 'm', 'p', 'l', 'e', '.', 'c', 'o', 'm', ',', ' ', 'c', 'c', '1', '3', '#', 'e', 'x', 'a', 'm', 'p', 'l', 'e', '.', 'c', 'o', 'm', ',', ' ', 'c', 'c', '1', '4', '#', 'e', 'x', 'a', 'm', 'p', 'l', 'e', '.', 'c', 'o', 'm'
Again, it appears that message.cc returns string not list.
Do I need to use regex to get the emails? Any suggestions about what I am doing wrong? Thanks!

Try:
cc_list = message.cc.split(',')

cc
A recipient's email address (a string) or a list of email addresses to appear on the Cc: line in the message header.
Message Fields
cc is a string
message.cc.split(", ")[0] is "cc12#example.com" that you want.