We Keep Coding

How to log production database changes made via the Django shell

I would like to automatically generate some sort of log of all the database changes that are made via the Django shell in the production environment.
We use schema and data migration scripts to alter the production database and they are version controlled. Therefore if we introduce a bug, it's easy to track it back. But if a developer in the team changes the database via the Django shell which then introduces an issue, at the moment we can only hope that they remember what they did or/and we can find their commands in the Python shell history.
Example. Let's imagine that the following code was executed by a developer in the team via the Python shell:
>>> tm = TeamMembership.objects.get(person=alice)
>>> tm.end_date = date(2022,1,1)
>>> tm.save()
It changes a team membership object in the database. I would like to log this somehow.
I'm aware that there are a bunch of Django packages related to audit logging, but I'm only interested in the changes that are triggered from the Django shell, and I want to log the Python code that updated the data.
So the questions I have in mind:
I can log the statements from IPython but how do I know which one touched the database?
I can listen to the pre_save signal for all model to know if data changes, but how do I know if the source was from the Python shell? How do I know what was the original Python statement?

This solution logs all commands in the session if any database changes were made.
How to detect database changes
Wrap execute_sql of SQLInsertCompiler, SQLUpdateCompiler and SQLDeleteCompiler.
SQLDeleteCompiler.execute_sql returns a cursor wrapper.
from django.db.models.sql.compiler import SQLInsertCompiler, SQLUpdateCompiler, SQLDeleteCompiler
changed = False
def check_changed(func):
def _func(*args, **kwargs):
nonlocal changed
result = func(*args, **kwargs)
if not changed and result:
changed = not hasattr(result, 'cursor') or bool(result.cursor.rowcount)
return result
return _func
SQLInsertCompiler.execute_sql = check_changed(SQLInsertCompiler.execute_sql)
SQLUpdateCompiler.execute_sql = check_changed(SQLUpdateCompiler.execute_sql)
SQLDeleteCompiler.execute_sql = check_changed(SQLDeleteCompiler.execute_sql)
How to log commands made via the Django shell
atexit.register() an exit handler that does readline.write_history_file().
import atexit
import readline
def exit_handler():
filename = 'history.py'
readline.write_history_file(filename)
atexit.register(exit_handler)
IPython
Check whether IPython was used by comparing HistoryAccessor.get_last_session_id().
import atexit
import io
import readline
ipython_last_session_id = None
try:
from IPython.core.history import HistoryAccessor
except ImportError:
pass
else:
ha = HistoryAccessor()
ipython_last_session_id = ha.get_last_session_id()
def exit_handler():
filename = 'history.py'
if ipython_last_session_id and ipython_last_session_id != ha.get_last_session_id():
cmds = '\n'.join(cmd for _, _, cmd in ha.get_range(ha.get_last_session_id()))
with io.open(filename, 'a', encoding='utf-8') as f:
f.write(cmds)
f.write('\n')
else:
readline.write_history_file(filename)
atexit.register(exit_handler)
Put it all together
Add the following in manage.py before execute_from_command_line(sys.argv).
if sys.argv[1] == 'shell':
import atexit
import io
import readline
from django.db.models.sql.compiler import SQLInsertCompiler, SQLUpdateCompiler, SQLDeleteCompiler
changed = False
def check_changed(func):
def _func(*args, **kwargs):
nonlocal changed
result = func(*args, **kwargs)
if not changed and result:
changed = not hasattr(result, 'cursor') or bool(result.cursor.rowcount)
return result
return _func
SQLInsertCompiler.execute_sql = check_changed(SQLInsertCompiler.execute_sql)
SQLUpdateCompiler.execute_sql = check_changed(SQLUpdateCompiler.execute_sql)
SQLDeleteCompiler.execute_sql = check_changed(SQLDeleteCompiler.execute_sql)
ipython_last_session_id = None
try:
from IPython.core.history import HistoryAccessor
except ImportError:
pass
else:
ha = HistoryAccessor()
ipython_last_session_id = ha.get_last_session_id()
def exit_handler():
if changed:
filename = 'history.py'
if ipython_last_session_id and ipython_last_session_id != ha.get_last_session_id():
cmds = '\n'.join(cmd for _, _, cmd in ha.get_range(ha.get_last_session_id()))
with io.open(filename, 'a', encoding='utf-8') as f:
f.write(cmds)
f.write('\n')
else:
readline.write_history_file(filename)
atexit.register(exit_handler)

I would consider something like this:
Wrapping each python session with some sort initialisation code using e.g.
PYTHONSTARTUP environment variable
https://docs.python.org/3/using/cmdline.html#envvar-PYTHONSTARTUP
In the file where PYTHONSTARTUP points to registering Exit handler using atexit
https://docs.python.org/3/library/atexit.html
These two things should allow you to use some lower level APIs of
django-reversion to wrap the whole terminal session with
https://django-reversion.readthedocs.io/en/stable/api.html#creating-revisions (something like this but calling __enter__ and __exit__ of that context manager directly in your startup and atexit code). Unfortunately I don't know the details but it should be doable.
In atexit / revision end calling the code to list the additional lines
of the terminal session and storing them somewhere else in the database with a reference to the specific revision.
See:
https://docs.python.org/3/library/readline.html#readline.get_history_length
https://docs.python.org/3/library/readline.html#readline.get_history_item
Basically, the idea is that you could call get_history_length twice: at the beginning and end of the terminal session. That will allow you to get relevant lines of where the change took place using get_history_item. You may end up having more lines of history than what you actually need but at least there is enough context to see what's going on.

Based on the answer of aaron and the implementation of the built-in IPython magic %logstart, this is the solution we came up with in the end.
All commands of the last IPython session are logged in a history file if any of the commands triggered a database write through the Django ORM.
Here's an excerpt of the generated history file:
❯ cat ~/.python_shell_write_history
# Thu, 27 Jan 2022 16:20:28
#
# New Django shell session started
#
# Thu, 27 Jan 2022 16:20:28
from people.models import *
# Thu, 27 Jan 2022 16:20:28
p = Person.objects.first()
# Thu, 27 Jan 2022 16:20:28
p
#[Out]# <Person: Test Albero Jose Maria>
# Thu, 27 Jan 2022 16:20:28
p.email
#[Out]# 'test-albero-jose-maria#gmail.com'
# Thu, 27 Jan 2022 16:20:28
p.save()
Here's our manage.py now:
#!/usr/bin/env python
import os
import sys
def shell_audit(logfname: str) -> None:
"""If any of the Python shell commands changed the Django database during the
session, capture all commands in a logfile for future analysis."""
import atexit
from django.db.models.sql.compiler import (
SQLDeleteCompiler,
SQLInsertCompiler,
SQLUpdateCompiler,
)
changed = False
def check_changed(func):
def _func(*args, **kwargs):
nonlocal changed
result = func(*args, **kwargs)
if not changed and result:
changed = not hasattr(result, "cursor") or bool(result.cursor.rowcount)
return result
return _func
SQLInsertCompiler.execute_sql = check_changed(SQLInsertCompiler.execute_sql)
SQLUpdateCompiler.execute_sql = check_changed(SQLUpdateCompiler.execute_sql)
SQLDeleteCompiler.execute_sql = check_changed(SQLDeleteCompiler.execute_sql)
def exit_handler():
if not changed:
return None
from IPython.core import getipython
shell = getipython.get_ipython()
if not shell:
return None
logger = shell.logger
# Logic borrowed from %logstart (IPython.core.magics.logging)
loghead = ""
log_session_head = "#\n# New Django shell session started\n#\n"
logmode = "append"
log_output = True
timestamp = True
log_raw_input = False
logger.logstart(logfname, loghead, logmode, log_output, timestamp, log_raw_input)
log_write = logger.log_write
input_hist = shell.history_manager.input_hist_parsed
output_hist = shell.history_manager.output_hist_reprs
log_write(log_session_head)
for n in range(1, len(input_hist)):
log_write(input_hist[n].rstrip() + "\n")
if n in output_hist:
log_write(output_hist[n], kind="output")
atexit.register(exit_handler)
if __name__ == "__main__":
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "project.settings")
try:
from django.core.management import execute_from_command_line
except ImportError:
# The above import may fail for some other reason. Ensure that the
# issue is really that Django is missing to avoid masking other
# exceptions on Python 2.
try:
import django # noqa: F401
except ImportError:
raise ImportError(
"Couldn't import Django. Are you sure it's installed and "
"available on your PYTHONPATH environment variable? Did you "
"forget to activate a virtual environment?"
)
raise
if sys.argv[1] == "shell":
logfname = os.path.expanduser("~/.python_shell_write_history")
shell_audit(logfname)
execute_from_command_line(sys.argv)

You could use django's receiver annotation.
For example, if you want to detect any call of the save method, you could do:
from django.db.models.signals import post_save
from django.dispatch import receiver
import logging
#receiver(post_save)
def logg_save(sender, instance, **kwargs):
logging.debug("whatever you want to log")
some more documentation for the signals

How can I launch an Android app on a device through Python?

I have consulted several topics on the subject, but I didn't see any related to launching an app on a device directly using a ppadb command.
I managed to do this code:
import ppadb
import subprocess
from ppadb.client import Client as AdbClient
# Create the connect functiun
def connect():
client = AdbClient(host='localhost', port=5037)
devices = client.devices()
for device in devices:
print (device.serial)
if len(devices) == 0:
print('no device connected')
quit()
phone = devices[0]
print (f'connected to {phone.serial}')
return phone, client
if __name__ == '__main__':
phone, client = connect()
import time
time.sleep(5)
# How to print each app on the emulator
list = phone.list_packages()
for truc in list:
print(truc)
# Launch the desired app through phone.shell using the package name
phone.shell(????????????????)
From there, I have access to each app package (com.package.name). I would like to launch it through a phone.shell() command but I can't access the correct syntax.
I can execute a tap or a keyevent and it's perfectly working, but I want to be sure my code won't be disturbed by any change in position.

From How to start an application using Android ADB tools, the shell command to launch an app is
am start -n com.package.name/com.package.name.ActivityName
Hence you would call
phone.shell("am start -n com.package.name/com.package.name.ActivityName")
A given package may have multiple activities. To find out what they are, you can use dumpsys package as follows:
def parse_activities(package, connection, retval):
out = ""
while True:
data = connection.read(1024)
if not data: break
out += data.decode('utf-8')
retval.clear()
retval += [l.split()[-1] for l in out.splitlines() if package in l and "Activity" in l]
connection.close()
activities = []
phone.shell("dumpsys package", handler=lambda c: parse_activities("com.package.name", c, activities))
print(activities)

Here is the correct and easiest answer:
phone.shell('monkey -p com.package.name 1')
This method will launch the app without needing to have acces to the ActivityName

Using AndroidViewClient/cluebra, you can launch the MAIN Activity of a package as follows:
#! /usr/bin/env python3
# -*- coding: utf-8 -*-
from com.dtmilano.android.viewclient import ViewClient
ViewClient.connectToDeviceOrExit()[0].startActivity(package='com.example.package')
This connects to the device (waiting if necessary) and then invokes startActivity() just using the package name.
startActivity() can also receive a component which is used when you know the package and the activity.

A Problem on getting docker stats by Python

I tried to use Python to get the docker stats, by using Python's docker module.
the code is:
import docker
cli = docker.from_env()
for container in cli.containers.list():
stream = container.stats()
print(next(stream))
I run 6 docker containers, but when I run the code, It needs a few second to get all containers' stats, so is there have some good methods to get the stats immediately?

Docker stats inherently takes a little while, a large part of this is waiting for the next value to come through the stream
$ time docker stats 1339f13154aa --no-stream
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
...
real 0m1.556s
user 0m0.020s
sys 0m0.015s
You could reduce the time it takes to execute by running the commands in parralell, as opposed to one at a time.
To achieve this, you could use the wonderful threading or multiprocessing library.
Digital Ocean provides a good tutorial on how to accomplish this with a ThreadPoolExecutor:
import requests
import concurrent.futures
def get_wiki_page_existence(wiki_page_url, timeout=10):
response = requests.get(url=wiki_page_url, timeout=timeout)
page_status = "unknown"
if response.status_code == 200:
page_status = "exists"
elif response.status_code == 404:
page_status = "does not exist"
return wiki_page_url + " - " + page_status
wiki_page_urls = [
"https://en.wikipedia.org/wiki/Ocean",
"https://en.wikipedia.org/wiki/Island",
"https://en.wikipedia.org/wiki/this_page_does_not_exist",
"https://en.wikipedia.org/wiki/Shark",
]
with concurrent.futures.ThreadPoolExecutor() as executor:
futures = []
for url in wiki_page_urls:
futures.append(executor.submit(get_wiki_page_existence, wiki_page_url=url))
for future in concurrent.futures.as_completed(futures):
print(future.result())

This is what I use to get the stats directly for each running docker (still takes like 1 second per container, so I don't think it can be helped). Besides this if it helps https://docker-py.readthedocs.io/en/stable/containers.html documentation for the arguments. Hope it helps.
import docker
client = docker.from_env()
containers = client.containers.list()
for x in containers:
print(x.stats(decode=None, stream=False))
As the TheQueenIsDead suggested, it might require threading if you want to get it faster.

Executing program via pyvmomi creates a process, but nothing happens after that

I'm studying vCenter 6.5 and community samples help a lot, but in this particular situation I can't figure out, what's going on. The script:
from __future__ import with_statement
import atexit
from tools import cli
from pyVim import connect
from pyVmomi import vim, vmodl
def get_args():
*Boring args parsing works*
return args
def main():
args = get_args()
try:
service_instance = connect.SmartConnectNoSSL(host=args.host,
user=args.user,
pwd=args.password,
port=int(args.port))
atexit.register(connect.Disconnect, service_instance)
content = service_instance.RetrieveContent()
vm = content.searchIndex.FindByUuid(None, args.vm_uuid, True)
creds = vim.vm.guest.NamePasswordAuthentication(
username=args.vm_user, password=args.vm_pwd
)
try:
pm = content.guestOperationsManager.processManager
ps = vim.vm.guest.ProcessManager.ProgramSpec(
programPath=args.path_to_program,
arguments=args.program_arguments
)
res = pm.StartProgramInGuest(vm, creds, ps)
if res > 0:
print "Program executed, PID is %d" % res
except IOError, e:
print e
except vmodl.MethodFault as error:
print "Caught vmodl fault : " + error.msg
return -1
return 0
# Start program
if __name__ == "__main__":
main()
When I execute it in console, it successfully connects to the target virtual machine and prints
Program executed, PID is 2036
In task manager I see process with mentioned PID, it was created by the correct user, but there is no GUI of the process (calc.exe). RMB click does not allow to "Expand" the process.
I suppose, that this process was created with special parameters, maybe in different session.
In addition, I tried to run batch file to check if it actually executes, but the answer is no, batch file does not execute.
Any help, advices, clues would be awesome.
P.S. I tried other scripts and successfully transferred a file to the VM.
P.P.S. Sorry for my English.
Update: All such processes start in session 0.

Have you tried interactiveSession ?
https://github.com/vmware/pyvmomi/blob/master/docs/vim/vm/guest/GuestAuthentication.rst
This boolean argument passed to NamePasswordAuthentication and means the following:
This is set to true if the client wants an interactive session in the guest.

Python PowerPoint SaveAs Task Scheduler

I have automated the creation of a PowerPoint slide-deck via Python and have set-up a trigger in Task Scheduler for a daily report to be generated.
This is all fine while my computer is logged in, but the script fails when the setting: "Run whether is logged in or not".
I checked to see what line was at fault and it turns out it is this one:
Presentation.SaveAs('C:\\Users\\me\\Desktop\\test.pptx')
I am running Task Scheduler with highest priority but it only runs this task with the "User is logged in" state.
Below is the entire basic code segment for reference:
import win32com.client, MSO, MSPPT, sys, os
g = globals()
for c in dir(MSO.constants): g[c] = getattr(MSO.constants, c)
for c in dir(MSPPT.constants): g[c] = getattr(MSPPT.constants, c)
error_file = open('C:\\Users\\me\\Desktop\\error_file.txt', 'wb')
run = False
try:
Application = win32com.client.Dispatch("PowerPoint.Application")
Application.Visible = True
Presentation = Application.Presentations.Add()
Slide = Presentation.Slides.Add(1, ppLayoutBlank)
Presentation.SaveAs('C:\\Users\\me\\Desktop\\test.pptx')
Presentation.Close()
Application.Quit()
run = True
except:
run = False
if run == True:
error_file.write('ok')
else:
error_file.write('fail')
Any help on this would be much appreciated.
Thanks,
JP