We Keep Coding

crontab some commands works, others do not

Summary
I've got a python script that writes a row to a sqlite3 database. I would like to run it with crontab, but can't get it working.
What I've tried so far
crontab -l
* * * * * /opt/homebrew/bin/python3 /Users/natemcintosh/dev/battery_condition/get_battery_condition.py
* * * * * /opt/homebrew/bin/python3 /Users/natemcintosh/dev/battery_condition/testing_cron.py
The first command is the one that appends to the database row. I can copy and paste that command at the command line, run it, and it adds the row. It does not run every minute. The contents of /Users/natemcintosh/dev/battery_condition/get_battery_condition.py are
# /Users/natemcintosh/dev/battery_condition/get_battery_condition.py
import subprocess
import re
import sqlite3
import datetime
def get_conds():
# Run the command to get battery info
command = "system_profiler SPPowerDataType".split()
response = subprocess.run(command, check=True, capture_output=True)
response = str(response.stdout)
# Get just the parts of interest
cycle_count = re.findall(r"Cycle Count: (\d+)", response)[0]
condition = re.findall(r"Condition: (\w+)", response)[0]
max_capacity = re.findall(r"Maximum Capacity: (\d+)", response)[0]
now = str(datetime.datetime.now().isoformat())
return [now, cycle_count, condition, max_capacity]
def append_row_to_db(db_name: str, items: list):
conn = sqlite3.connect(db_name)
with conn:
conn.execute("INSERT INTO battery_condition VALUES (?,?,?,?)", items)
conn.close()
if __name__ == "__main__":
# Get the condition
battery_condition = get_conds()
# Append to the file
filename = "/Users/natemcintosh/dev/battery_condition/battery_condition.db"
append_row_to_db(filename, battery_condition)
The second command is a test script that does work.
#/Users/natemcintosh/dev/battery_condition/testing_cron.py
import datetime
if __name__ == "__main__":
with open("/Users/natemcintosh/Desktop/crontest.txt", "a+") as f:
now = str(datetime.datetime.now().isoformat())
f.write(f"{now}\n")
Every minute, a new line appears in /Users/natemcintosh/Desktop/crontest.txt with the current date and time. I've also tried having the test script write to other locations on disk, and they all seem to work.

With the help of Gordon Davisson, I've figure out the issue. When I call subprocess.run(command, check=True, capture_output=True), the command did not have the full path to the executable.
It used to be
command = "system_profiler SPPowerDataType".split()
I got the full path to the system_profiler command with
$ which system_profiler
/usr/sbin/system_profiler
And fixed the line in my python script to be
command = "/usr/sbin/system_profiler SPPowerDataType".split()
The tool now runs successfully!

How to execute copy command from Psycopg2 on remote database connection? [duplicate]

I need to export some rows from a table in a PostgreSQL database to a .csv file using a Python script:
#!/usr/bin/python
# -*- coding: utf-8 -*-
import sys, psycopg2
...
conn = psycopg2.connect("dbname=dbname user=user password=password")
cur = conn.cursor()
sql = "\copy (SELECT * FROM table WHERE month=6) TO '/mnt/results/month/table.csv' WITH CSV DELIMITER ';';"
cur.execute(sql)
cur.close()
...
But when I run the script I get this:
Syntax error at or near «\»
LINE 1: \copy (SELECT * FROM TABLE WHERE month=6) TO '...
Does anyone know what can be wrong or give me a tip about?

The \copy is not an SQL command, it is a command specific for the Postgres terminal client psql and cannot be used in this context.
Use copy_expert(sql, file, size=8192) instead, e.g.:
sql = "COPY (SELECT * FROM a_table WHERE month=6) TO STDOUT WITH CSV DELIMITER ';'"
with open("/mnt/results/month/table.csv", "w") as file:
cur.copy_expert(sql, file)
Read more about the function in the documentation.

\COPY is a psql command (client based) and therefore not a valid SQL command. Try the same simply using COPY (without the backslash\).
If you want the output file directly at the client, you might wanna use the STDOUT of COPY, otherwise the it will be created at the database server, which is a luxury not everyone has.

Yes, all statements are correct related to \COPY as SQL command.
If you want to try the same as you stated in the question then you can try it as a shell command in python script.
Eg:
import os
cmd1 = " Your SQL to csv Command"
os.system(cmd1)

You can use this syntax:
query = """ select * from instalacje WHERE date > '2022-02-20'"""
outputquery = "COPY ({0}) TO STDOUT WITH CSV HEADER".format(query)
with open("C:/1/Wojciech_CCC.csv", "w") as f:
cur.copy_expert(outputquery, f)

Command line argument with python

I am trying to execute below python script. But getting below issue after execution.
Code
#!/usr/bin/python
def Student(Student_Id):
msg = Student_Id + "." +
return msg
Error
C:\Users\Desktop>Test.py 2asd
After an investigation found that argument which I am passing through the command line is considered as Null.

You need to call like
Test.py -n 2asd

python script raises an error depending on how it was called from the shell

I'm making this small tool to convert a skype's chat database into a somewhat more readable representation under the form of an IRC-like chat export. I'm making this because I saved some of my old skypes chats' .db files, and now I'm trying to extract the content from them. That part I got working, now however there's something I just cannot figure out why it happens.
If I invoke my script as ./skype2text.py file.db chat_partner_id it works fine, and prints the chat with the specified user id to stdout
Having that working I wanted to save the output to a file instead of printing it to stdout, so I just ran it as echo $(./skype2text.py file.db chat_partner_id) first just to see how it went, so I could redirect it to a file, and that's when the weird stuff happens. It prints the first chat line and crashes. (And also completely ignores newlines after that).
$ echo $(./skype2text.py "main 1.db" miya)
Traceback (most recent call last):
File "./skype2text.py", line 62, in <module>
print(u"<" + row[0] + u"> " + unicode(parser.unescape(unicode(row[1]))))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe0' in position 47: ordinal not in range(128)
<Luda> C'est moi <ss type="wink">;)</ss> <MiYa> None
Here is the code
#!/usr/bin/env python2
# charset=utf-8
from __future__ import print_function
import sys
import sqlite3
import os.path
import HTMLParser
def eprint(*args, **kwargs):
print(*args, file=sys.stderr, **kwargs)
def eprint_use():
eprint("usage : " + sys.argv[0] + " <file.db> <partner's skype ID> [output file]")
# actual code here
# first of all check argv and print the help message if it's wrong
if len(sys.argv) < 3 or len(sys.argv) > 4:
eprint_use()
else:
database_path = sys.argv[1]
partner_id = sys.argv[2]
output_path = sys.argv[3] if len(sys.argv) == 4 else partner_id + '.txt'
if not os.path.isfile(database_path):
sys.exit('the file %s does not exist' % (database_path))
connection = sqlite3.connect(database_path)
cursor = connection.cursor()
parser = HTMLParser.HTMLParser()
cursor.execute("SELECT from_dispname,body_xml FROM Messages WHERE dialog_partner='" + partner_id + "' ORDER BY timestamp")
for row in cursor.fetchall():
print(u"<" + row[0] + u"> " + unicode(parser.unescape(unicode(row[1]))))
I've ommitted most of the comments at the top that serve no purpose, so line 62 here refers to the very last line.
I may be doing something wrong with the SQL queries at some point. I don't really check if the input is valid either, but that's not really the point. Why does this happen ? Why does calling the script differently causes it to crash although it works perfectly on its own ? I've checked sys.argv as well and it contains the same thing in both cases. Also yes, I have an output_path variable which is not used, I'll adapt the output depending on the third parameter, if it contains a filename I'll output to the file instead, for now. The weirdest is why does it cause an unicode exception ?
$ bash --version
GNU bash, version 4.3.48(1)-release (x86_64-pc-linux-gnu)
$ python2 --version
Python 2.7.10

probably the default encoding is utf-8 in first case (when it works), and it is ascii in second case (when UnicodeEncodeError happens)
maybe try:
for row in cursor.fetchall():
res = u"<" + row[0] + u"> " + unicode(parser.unescape(unicode(row[1])))
print(res.encode('utf-8'))

How can I execute "source FileName.sql" in a python script?

I would like to execute the MySQL query source FileName.sql in a Python script on Linux.
I am able to execute other queries like SELECT * FROM table_name but this one is giving an error. I am executing this on a Linux server with a MySQL database using Python. The frontend I am using is Putty.
The Python script I have used is:
import MySQLdb
db = MySQLdb.connect("hostname","username","pswrd","dbname")
cursor = db.cursor()
cursor.execute("source FileName.sql")
db.close()
How can I execute the query source FileName.sql on the location where this file-> FileName.sql is located?

source is not a SQL command. It's a MySQL CLI command, it only exists in the console application mysql (and wherever else implemented). All it does is to read the contents of FileName.sql and issue the SQL commands inside.
To do this in python, you can use something like
Edit: This assumes you have 1 query per line! If you have multi-line queries, you'll have to find the means to extract each query from the file.
import MySQLdb
db = MySQLdb.connect("hostname","user","pass","db")
cursor = db.cursor()
for line in open("FileName.sql"):
cursor.execute(line)
db.close()

You can execute a bash command with Python and import your SQL file.
This exemple is for MySQL
import subprocess
command = "mysql -u username --password=p#55W0rD database_name < file.sql".split()
p = subprocess.Popen(command, stdout=subprocess.PIPE)
p.communicate() # you can see if errors are returned
if your SQL file creates a database, remove database_name.
sources:
https://docs.python.org/3/library/subprocess.html#popen-constructor
https://dev.mysql.com/doc/refman/8.0/en/mysql-batch-commands.html

Separate the scripts in SQL file in python using ";" as delimiter
Execute each command iteratively.
awesome5team had developed a nice solution in https://github.com/awesome5team/General-Resources-Box/issues/7
Code snippet from the same:
import mysql.connector
cnx = mysql.connector.connect(user='root',
password='YOUR-PASSWORD-FOR-MYSQL',
host='localhost',
database='YOUR-DATABASE-NAME')
cursor =cnx.cursor()
def executeScriptsFromFile(filename):
fd = open(filename, 'r')
sqlFile = fd.read()
fd.close()
sqlCommands = sqlFile.split(';')
for command in sqlCommands:
try:
if command.strip() != '':
cursor.execute(command)
except IOError, msg:
print "Command skipped: ", msg
executeScriptsFromFile('SQL-FILE-LOCATION')
cnx.commit()