We Keep Coding

Python D-Bus: Subscribe to signal and read property with dasbus

How to monitor and read Ubuntu's "Night Light" status via D-Bus using Python with dasbus? I can't figure out the API docs on how to read a property or subscribe to a signal.
Likely candidates:
dasbus.client.property.get()
GLibClient.subscribe()
The following is adapted from the basic examples and prints the interfaces and properties/signals of the object:
#!/usr/bin/env python3
from dasbus.connection import SessionMessageBus
bus = SessionMessageBus()
# dasbus.client.proxy.ObjectProxy
proxy = bus.get_proxy(
"org.gnome.SettingsDaemon.Color", # bus name
"/org/gnome/SettingsDaemon/Color", # object path
)
print(proxy.Introspect())
# read and print properties "NightLightActive" and "Temperature" from interface "org.gnome.SettingsDaemon.Color" in (callback) function
# subscribe to signal "PropertiesChanged" in interface "org.freedesktop.DBus.Properties" / register callback function
Resources
https://pypi.org/project/dbus-python/
What is recommended to use pydbus or dbus-python and what are the differences?
https://wiki.python.org/moin/DbusExamples
Migration from dbus to GDbus in Python 3

Looking at the dasbus examples and the Introspection data it looks like to get the property the dasbus is pythonic so proxy.<property name> works. For your example of NightLightActive it would be:
print("Night light active?", proxy.NightLightActive)
For the signal you need to connect to the signal on the proxy so that seems to take the form of proxy.<signal name>.connect so for example:
proxy.PropertiesChanged.connect(callback)
And this will need to have an EventLoop running.
My entire test was:
from dasbus.connection import SessionMessageBus
from dasbus.loop import EventLoop
bus = SessionMessageBus()
loop = EventLoop()
# dasbus.client.proxy.ObjectProxy
proxy = bus.get_proxy(
"org.gnome.SettingsDaemon.Color", # bus name
"/org/gnome/SettingsDaemon/Color", # object path
)
print("Night light active?", proxy.NightLightActive)
print("Temperature is set to:", proxy.Temperature)
def callback(iface, prop_changed, prop_invalidated):
print("The notification:",
iface, prop_changed, prop_invalidated)
proxy.PropertiesChanged.connect(callback)
loop.run()

Ram usage is constantly increasing on my azure server running python scripts

Issue in Brief
I have recently started using an azure server running ubuntu 20.04. My workflow includes running around 50 python scripts 24/7 and they are operationally very important to my team. But the issue when I just start those python scripts my ram usage is nominal like 12/16 GB remains free in my system after running all my scripts.
But slowly RAM usage by those scripts starts increasing to the point where the system starts to kill them to free up some main memory.
I have no idea what the issue is over here. My scripts are pretty simple and I really don't know where and how do I resolve this issue. Can anyone please show/tell me some guidelines on how do I approach solving this issue?
Comments
I am using python 3.10. Script function is to download the data from some server and upload it to my MySQL database. I can provide the code if anyone asks for it.
Let me know if I can provide anything else to make this easier for you.
Code files
I am uploading the code which is taking up the maximum memory according to htop.
dcx_trades.py
import json
import time
import datetime
from mysql_connector import SQLConnector
import pandas as pd
import sys
import os
import signal
from contextlib import contextmanager
def raise_timeout(signum, frame):
print("timeout")
raise Exception("timouttt")
#contextmanager
def timeout(time):
# Register a function to raise a TimeoutError on the signal.
signal.signal(signal.SIGALRM, raise_timeout)
# Schedule the signal to be sent after ``time``.
signal.alarm(time)
try:
yield
except TimeoutError:
# exit()
pass
finally:
# Unregister the signal so it won't be triggered
# if the timeout is not reached.
signal.signal(signal.SIGALRM, signal.SIG_IGN)
from_db = {'user': 'db_user_name', 'password': 'password', 'host': 'host_url', 'database': 'crypto'}
s = SQLConnector('crypto', from_db)
dict_ = {'timestamp': '', "exchange": "coindcx", "symbol":"", 'error_msg':''}
df = pd.DataFrame(columns = ["exchange_id","timestamp","symbol","price","quantity","exchange","turnover"])
df.set_index('symbol')
while True:
try:
data = pd.read_csv('dcx_trades.csv')
trades = data.to_dict(orient='records')
data = data.iloc[0:0]
if len(trades):
for trade in trades:
utc_time = datetime.datetime.fromtimestamp(trade['T']/1000, datetime.timezone.utc)
local_time = utc_time.astimezone()
datetime_formatted = local_time.strftime("%Y-%m-%d %H:%M:%S")
dict_['timestamp'] = datetime_formatted
dict_["exchange_id"] = 12345
dict_["symbol"] = trade['s']
dict_['price'] = trade['p']
dict_['quantity'] = trade['q']
dict_['turnover'] = float(trade['p'])*float(trade['q'])
dict_['error'] = '0'
df = df.append(dict_, ignore_index=True)
print(df)
df_new = df
df_new= df_new.to_dict(orient='records')
df = df.iloc[0:0]
data.to_csv('dcx_trades.csv', mode='w', index=False)
if len(df_new):
with timeout(60):
try:
print(datetime.datetime.now())
s.add_multipletrades(df_new)
print(datetime.datetime.now())
except Exception as e:
print(e)
os.execv(sys.executable, ['python'] + sys.argv)
print("error_time:", datetime.datetime.now())
except Exception as e:
data = pd.read_csv('dcx_trades.csv')
data = data.loc[1:]
data.to_csv('dcx_trades.csv',index=False)
pass
Objective of the file:
Firstly s = SQLConnector('crypto', from_db) this lines makes the connection with the DB. All the database-related functions have been defined in another file named mysql_connector.py which I have imported in the beginning.
Then the code reads from the CSV file named dcx_trades.csv and preprocesses the data as per the database table. Before uploading the data into DB it clears the CSV file so as to remove duplicates. timeout(60) function is used because sometimes the file get stuck while writing into the DB and thus it needs to get restarted which is what timeout() function does.

All of those transforms can easily be done in SQL --
LOAD DATA into a temp table with whatever columns match the columns and datatypes in the file
Run a single INSERT .. SELECT .. to copy the values over, doing whatever expressions are needed (such as p * q).

Post beanstalk queue data to Mssql using python script

Guys I'm currently developing an offline ALPR solution.
So far I've used OpenAlpr software running on Ubuntu. By using a python script I found on StackOverlFlow I'm able to read the beanstalk queue data (plate number & meta data) of the ALPR but I need to send this data from the beanstalk queue to a mssql database. Does anyone know how to export beanstalk queue data or JSON data to the database? The code below is for local-host, how do i modify it to automatically post data to the mssql database? The data in the beanstalk queue is in JSON format [key=value].
The read & write csv was my addition to see if it can save the json data as csv on localdisk
import beanstalkc
import json
from pprint import pprint
beanstalk = beanstalkc.Connection(host='localhost', port=11300)
TUBE_NAME='alprd'
text_file = open('output.csv', 'w')
# For diagnostics, print out a list of all the tubes available in Beanstalk.
print beanstalk.tubes()
# For diagnostics, print the number of items on the current alprd queue.
try:
pprint(beanstalk.stats_tube(TUBE_NAME))
except beanstalkc.CommandFailed:
print "Tube doesn't exist"
# Watch the "alprd" tube; this is where the plate data is.
beanstalk.watch(TUBE_NAME)
while True:
# Wait for a second to get a job. If there is a job, process it and delete it from the queue.
# If not, return to sleep.
job = beanstalk.reserve(timeout=5000)
if job is None:
print "No plates yet"
else:
plates_info = json.loads(job.body)
# Do something with this data (e.g., match a list, open a gate, etc.).
# if 'data_type' not in plates_info:
# print "This shouldn't be here... all OpenALPR data should have a data_type"
# if plates_info['data_type'] == 'alpr_results':
# print "Found an individual plate result!"
if plates_info['data_type'] == 'alpr_group':
print "Found a group result!"
print '\tBest plate: {} ({:.2f}% confidence)'.format(
plates_info['candidates'][0]['plate'],
plates_info['candidates'][0]['confidence'])
make_model = plates_info['vehicle']['make_model'][0]['name']
print '\tVehicle information: {} {} {}'.format(
plates_info['vehicle']['year'][0]['name'],
plates_info['vehicle']['color'][0]['name'],
' '.join([word.capitalize() for word in make_model.split('_')]))
elif plates_info['data_type'] == 'heartbeat':
print "Received a heartbeat"
text_file.write('Best plate')
# Delete the job from the queue when it is processed.
job.delete()
text_file.close()

AFAIK there is no way to directly export data from beanstalkd.
What you have makes sense, that is streaming data out of a tube into a flat file or performing a insert into the DB directly
Given the IOPS beanstalkd can be produced, it might still be a reasonable solution (depends on what performance you are expecting)
Try asking https://groups.google.com/forum/#!forum/beanstalk-talk as well

NFQueue/Scapy Man in the Middle

I'm trying to construct a man in the middle attack on a webpage (i.e. HTTP traffic). I'm doing this by using a Linux machine attached to Ethernet and a client attached to the Linux box via its WiFi hotspot.
What I've done so far is use NFQueue from within the IPTables Linux firewall to route all TCP packets on the FORWARD chain to the NFQueue queue, which a Python script is picking up and then processing those rules. I'm able to read the data off of the HTTP response packets, but whenever I try to modify them and pass them back (accept the packets), I'm getting an error regarding the strings:
Exception AttributeError: "'str' object has no attribute 'build_padding'" in 'netfilterqueue.global_callback' ignored
My code is here, which includes things that I've tried that didn't work. Notably, I'm using a third-party extension for scapy called scapy_http that may be interfering with things, and I'm using a webpage that is not being compressed by gzip because that was messing with things as well. The test webpage that I'm using is here.
#scapy
from scapy.all import *
#nfqueue import
from netfilterqueue import NetfilterQueue
#scapy http extension, not really needed
import scapy_http.http
#failed gzip decoding, also tried some other stuff
#import gzip
def print_and_accept(packet):
#convert nfqueue datatype to scapy-compatible
pkt = IP(packet.get_payload())
#is this an HTTP response?
if pkt[TCP].sport == 80:
#legacy trial that doesn't work
#data = packet.get_data()
print('HTTP Packet Found')
#check what's in the payload
stringLoad = str(pkt[TCP].payload)
#deleted because printing stuff out clogs output
#print(stringLoad)
#we only want to modify a specific packet:
if "<title>Acids and Bases: Use of the pKa Table</title>" in stringLoad:
print('Target Found')
#strings kind of don't work, I think this is a me problem
#stringLoad.replace('>Acids and Bases: Use of the pK<sub>a</sub>', 'This page has been modified: a random ')
#pkt[TCP].payload = stringLoad
#https://stackoverflow.com/questions/27293924/change-tcp-payload-with-nfqueue-scapy
payload_before = len(pkt[TCP].payload)
# I suspect this line is a problem: the string assigns,
# but maybe under the hood scapy doesn't like that very much
pkt[TCP].payload = str(pkt[TCP].payload).replace("Discussion", "This page has been modified")
#recalculate length
payload_after = len(pkt[TCP].payload)
payload_dif = payload_after - payload_before
pkt[IP].len = pkt[IP].len + payload_dif
#recalculate checksum
del pkt[TCP].chksum
del pkt[IP].chksum
del pkt.chksum
print('Packet Modified')
#redudant
#print(stringLoad)
#this throws an error (I think)
print(str(pkt[TCP].payload))
#no clue if this works or not yet
#goal here is to reassign modified packet to original parameter
packet.set_payload(str(pkt))
#this was also throwing the error, so tried to move away from it
#print(pkt.show2())
#bunch of legacy code that didn't work
#print(GET_print(pkt))
#print(pkt.show())
#decompressed_data = zlib.decompress(str(pkt[TCP].payload), 16 + zlib.MAX_WBITS)
#print(decompressed_data)
#print(str(gzip.decompress(pkt[TCP].payload)))
# print(pkt.getlayer(Raw).load)
#print('HTTP Contents Shown')
packet.accept()
def GET_print(packet1):
ret = "***************************************GET PACKET****************************************************\n"
ret += "\n".join(packet1.sprintf("{Raw:%Raw.load%}\n").split(r"\r\n"))
ret += "*****************************************************************************************************\n"
return ret
print('Test: Modify a very specific target')
print('Program Starting')
nfqueue = NetfilterQueue()
nfqueue.bind(1, print_and_accept)
try:
print('Packet Interface Starting')
nfqueue.run()
except KeyboardInterrupt:
print('\nProgram Ending')
nfqueue.unbind()
Apologies in advance if this is hard to read or badly formatted code; Python isn't a language that I write in often. Any help is greatly appreciated!

Pipsta Printer and Printing a list

I'm trying to modify the simple python script provided with my Pipsta Printer so that instead of printing a single line of text, it prints out a list of things.
I know there are probably better ways to do this, but using the script below, could somebody please tell me what changes I need to make around the "txt = " part of the script, so that I can print out a list of items and not just one item?
Thanks.
# BasicPrint.py
# Copyright (c) 2014 Able Systems Limited. All rights reserved.
'''This simple code example is provided as-is, and is for demonstration
purposes only. Able Systems takes no responsibility for any system
implementations based on this code.
This very simple python script establishes USB communication with the Pipsta
printer sends a simple text string to the printer.
Copyright (c) 2014 Able Systems Limited. All rights reserved.
'''
import argparse
import platform
import sys
import time
import usb.core
import usb.util
FEED_PAST_CUTTER = b'\n' * 5
USB_BUSY = 66
# NOTE: The following section establishes communication to the Pipsta printer
# via USB. YOU DO NOT NEED TO UNDERSTAND THIS SECTION TO PROGRESS WITH THE
# TUTORIALS! ALTERING THIS SECTION IN ANY WAY CAN CAUSE A FAILURE TO COMMUNICATE
# WITH THE PIPSTA. If you are interested in learning about what is happening
# herein, please look at the following references:
#
# PyUSB: http://sourceforge.net/apps/trac/pyusb/
# ...which is a wrapper for...
# LibUSB: http://www.libusb.org/
#
# For full help on PyUSB, at the IDLE prompt, type:
# >>> import usb
# >>> help(usb)
# 'Deeper' help can be trawled by (e.g.):
# >>> help(usb.core)
#
# or at the Linux prompt, type:
# pydoc usb
# pydoc usb.core
PIPSTA_USB_VENDOR_ID = 0x0483
PIPSTA_USB_PRODUCT_ID = 0xA053
def parse_arguments():
'''Parse the arguments passed to the script looking for a font file name
and a text string to print. If either are mssing defaults are used.
'''
txt = 'Hello World from Pipsta!'
parser = argparse.ArgumentParser()
parser.add_argument('text', help='the text to print',
nargs='*', default=txt.split())
args = parser.parse_args()
return ' '.join(args.text)
def main():
"""The main loop of the application. Wrapping the code in a function
prevents it being executed when various tools import the code.
"""
if platform.system() != 'Linux':
sys.exit('This script has only been written for Linux')
# Find the Pipsta's specific Vendor ID and Product ID
dev = usb.core.find(idVendor=PIPSTA_USB_VENDOR_ID,
idProduct=PIPSTA_USB_PRODUCT_ID)
if dev is None: # if no such device is connected...
raise IOError('Printer not found') # ...report error
try:
# Linux requires USB devices to be reset before configuring, may not be
# required on other operating systems.
dev.reset()
# Initialisation. Passing no arguments sets the configuration to the
# currently active configuration.
dev.set_configuration()
except usb.core.USBError as ex:
raise IOError('Failed to configure the printer', ex)
# The following steps get an 'Endpoint instance'. It uses
# PyUSB's versatile find_descriptor functionality to claim
# the interface and get a handle to the endpoint
# An introduction to this (forming the basis of the code below)
# can be found at:
cfg = dev.get_active_configuration() # Get a handle to the active interface
interface_number = cfg[(0, 0)].bInterfaceNumber
# added to silence Linux complaint about unclaimed interface, it should be
# release automatically
usb.util.claim_interface(dev, interface_number)
alternate_setting = usb.control.get_interface(dev, interface_number)
interface = usb.util.find_descriptor(
cfg, bInterfaceNumber=interface_number,
bAlternateSetting=alternate_setting)
usb_endpoint = usb.util.find_descriptor(
interface,
custom_match=lambda e:
usb.util.endpoint_direction(e.bEndpointAddress) ==
usb.util.ENDPOINT_OUT
)
if usb_endpoint is None: # check we have a real endpoint handle
raise IOError("Could not find an endpoint to print to")
# Now that the USB endpoint is open, we can start to send data to the
# printer.
# The following opens the text_file, by using the 'with' statemnent there is
# no need to close the text_file manually. This method ensures that the
# close is called in all situation (including unhandled exceptions).
txt = parse_arguments()
usb_endpoint.write(b'\x1b!\x00')
# Print a char at a time and check the printers buffer isn't full
for x in txt:
usb_endpoint.write(x) # write all the data to the USB OUT endpoint
res = dev.ctrl_transfer(0xC0, 0x0E, 0x020E, 0, 2)
while res[0] == USB_BUSY:
time.sleep(0.01)
res = dev.ctrl_transfer(0xC0, 0x0E, 0x020E, 0, 2)
usb_endpoint.write(FEED_PAST_CUTTER)
usb.util.dispose_resources(dev)
# Ensure that BasicPrint is ran in a stand-alone fashion (as intended) and not
# imported as a module. Prevents accidental execution of code.
if __name__ == '__main__':
main()

Pipsta uses linux line-feed ('\n', 0x0A, 10 in decimal) to mark a new line. For a quick test change BasicPrint.py as follows:
#txt = parse_arguments()
txt = 'Recipt:\n========\n1. food 1 - 100$\n2. drink 1 - 200$\n\n\n\n\n'
usb_endpoint.write(b'\x1b!\x00')
for x in txt:
usb_endpoint.write(x)
res = dev.ctrl_transfer(0xC0, 0x0E, 0x020E, 0, 2)
while res[0] == USB_BUSY:
time.sleep(0.01)
res = dev.ctrl_transfer(0xC0, 0x0E, 0x020E, 0, 2)
I commented out parameter parsing and injected a test string with multi line content.