Python script not executed from Cron - python

I am trying to run a Python script from Cron but it does not work. I have already tried everything I have seen in multiple Stackoverflow questions.The machine is a Raspberry running Raspbian. The following piece of code is the edition of crontab:
PATH=/usr/sbin:/usr/bin:/sbin/bin:/sbin:/bin:/home/pi/miniconda/bin:/usr/local/bin:/usr/local/sbin
*/5 * * * * rsync -az --timeout=10 --progress pritms#bigdata.trainhealthmanagement.com:Upload/*.csv /home/pi/PAD-S100/PAD-S100-Bloque_Motor/from_repo/ | /bin/sh /home/pi/PAD-S100/PAD-S100-Bloque_Motor/adddate_to_logs.sh >> /home/pi/PAD-S100/PAD-S100-Bloque_Motor/log.log 2>&1
*/5 * * * * /bin/sh /home/pi/PAD-S100/PAD-S100-Bloque_Motor/from_repo/launcher.sh | /bin/sh /home/pi/PAD-S100/PAD-S100-Bloque_Motor/adddate_to_logs.sh >> /home/pi/PAD-S100/PAD-S100-Bloque_Motor/log.log 2>&1
*/30 * * * * rm /home/pi/PAD-S100/PAD-S100-Bloque_Motor/log.log
* * * * * /usr/bin/python /home/pi/PAD-S100/PAD-S100-Bloque_Motor/from_repo/event_management.py | /bin/sh /home/pi/PAD-S100/PAD-S100-Bloque_Motor/adddate_to_logs.sh >> home/pi/PAD-S100/PAD-S100-Bloque_Motor/log.log 2>&1
0 0 * * * rm /home/pi/PAD-S100/PAD-S100-Bloque_Motor/from_repo/*.csv | /bin/sh /home/pi/PAD-S100/PAD-S100-Bloque_Motor/adddate_to_logs.sh >> /home/pi/PAD-S100/PAD-S100-Bloque_Motor/log.log 2>&1
Crontab Observations:
The Path obtained from echo $PATH is included.
launcher.sh, addddate_to_logs.sh and
event_management are executables using the command sudo chmod a+x <file_name>.
The log.log file does not show anything strange.
The system log file /var/log/syslog has the following logs:
Feb 27 15:11:08 raspberrypi cron[21814]: sendmail: Cannot open :25
Feb 27 15:12:01 raspberrypi rsyslogd-2007: action 'action 17' suspended, next retry is Mon Feb 27 15:13:31 2017 [try http://www.rsyslog.com/e/2007 ]
Feb 27 15:12:01 raspberrypi CRON[22209]: (pi) CMD (/usr/bin/python /home/pi/PAD-S100/PAD-S100-Bloque_Motor/from_repo/event_management.py | /bin/sh /home/pi/PAD-S100/PAD-S100-Bloque_Motor/adddate_to_logs.sh >> home/pi/PAD-S100/PAD-S100-Bloque_Motor/log.log 2>&1)
Feb 27 15:12:09 raspberrypi sSMTP[22212]: Unable to set UsesSTARTTILS=""
Feb 27 15:12:09 raspberrypi sSMTP[22212]: Unable to locate
Feb 27 15:12:09 raspberrypi cron[21814]: sendmail: Cannot open :25
Feb 27 15:12:09 raspberrypi sSMTP[22212]: Cannot open :25
Feb 27 15:12:09 raspberrypi CRON[22205]: (pi) MAIL (mailed 178 bytes of output but got status 0x0001 from MTA#012)
We can observe that it is probable that the failing crontab line is the one of the python script. As I am not an expert in Linux I believe it may be something related to the sSMTP. The same kind of error log appears after every call of the cron python script. But I have no idea of how to fix it or configure the local email.
Here is the piece of code of event_management.py file :
#!/usr/bin/python
# -*- coding: utf-8 -*-
import imaplib
import email
import csv
import datetime
EMAIL = <email_user>
FROM_PWD = <password>
SMTP_SERVER = 'mail.o365.alstom.com'
datum = dict()
translate = {'#09': 1, '#0A': 2, '#0B': 3, '#0C': 4}
def connect_imap():
mail = imaplib.IMAP4_SSL(SMTP_SERVER)
mail.login(EMAIL, FROM_PWD)
return mail
def read_email_from_gmail(writer, mail):
mail.select('BRMS')
kind, data = mail.search(None, 'ALL')
mail_ids = data[0]
id_list = mail_ids.split()
first_email_id = int(id_list[0])
latest_email_id = int(id_list[-1])
for i in range(latest_email_id, first_email_id, -1):
typ, data = mail.fetch(i, '(RFC822)')
for response_part in data:
if isinstance(response_part, tuple):
msg = email.message_from_string(response_part[1])
for part in msg.walk():
if part.get_content_type() == 'text/html':
content = part.get_payload()
manage_email_content(content, writer)
return 0
def manage_email_content(content, writer):
content = content.split('\n')
for i, line in enumerate(content):
if 'Alert description' in line:
line = line.split()
datum['Event code'] = line[-1][4:]
if line[-1][:3] in translate:
datum['Motor block num'] = translate[line[-1][:3]]
else:
datum['Motor block num'] = 'Defecto ajeno al bloque motor'
elif 'Alert condition' in line:
line = line.split()
datum['Code description'] = ' '.join(line[4:])
elif 'Unit id' in line:
line = line.split()
datum['Train num'] = line[3][3:]
elif 'Alert raised' in line:
line = line.split()
datum['Date'] = line[4][:10]
datum['Time'] = line[4][11:]
writer.writerow(datum)
print datum
return 0
def move_to_trash_before_date(mail, folder, days_before):
# required to perform search, m.list() for all lables, '[Gmail]/Sent Mail'
no_of_msgs = int(mail.select(folder)[1][0])
print("- Found a total of {1} messages in '{0}'.".format(folder, no_of_msgs))
before_date = (datetime.date.today() - datetime.timedelta(days_before)).strftime("%d-%b-%Y")
typ, data = mail.search(None, '(BEFORE {0})'.format(before_date)) # search pointer for msgs before before_date
if data != ['']: # if not empty list means messages exist
no_msgs_del = data[0].split()[-1] # last msg id in the list
print("- Marked {0} messages for removal with dates before {1} in '{2}'.".format(no_msgs_del, before_date, folder))
mail.store("1:{0}".format(no_msgs_del), '+X-GM-LABELS', '\\Trash') # move to trash
empty_folder(mail, 'Elementos eliminados', do_expunge=True) # can send do_expunge=False, default True
else:
print("- Nothing to remove.")
return 0
def empty_folder(mail, folder, do_expunge=True):
mail.select(folder) # select all trash
mail.store("1:*", '+FLAGS', '\\Deleted') # Flag all Trash as Deleted
if do_expunge: # See Gmail Settings -> Forwarding and POP/IMAP -> Auto-Expunge
mail.expunge() # not need if auto-expunge enabled
else:
print("Expunge was skipped.")
return 0
def disconnect_imap(mail):
mail.close()
mail.logout()
return 0
def main():
with open('email_data.csv', 'w') as f:
writer = csv.DictWriter(f, fieldnames=['Time', 'Date', 'Train num', 'Motor block num',
'Event code', 'Code description'], delimiter=';')
try:
m = connect_imap()
writer.writeheader()
read_email_from_gmail(writer, m)
move_to_trash_before_date(m, 'BRMS', 15) # inbox cleanup, before 15 days
disconnect_imap(m)
except Exception, e:
print str(e)
if __name__ == "__main__":
main()
event_management file connects to an Outlook email folder, reads the emails and builds a CSV file with data extracted from the the emails' contents. This file works properly, it is already tested; and it works fine when executed manually (not using Cron). So I not sure it is related to the sSMTP issue appearing in the system log.
I will apreciate every kind of help or suggestions!

After some test and reading other users' answers, I have found the problem. It is a combination of two different issues that are not directly related, but together made this problem a pain in the ass to debug.
First Problem:
log.log file contains logs and errors from three different executables, hence I did not notice that evet_management file did not have the correct permissions. I did not apply chmod command well, and I have not notice it as it contained a lot of data.
Conclussion 1: One cronjob, one log file.
Conclussion 2: /var/log/syslog contains a lot of data, from various resources, hence it may confuse you when trying to debug. Better to produce log files apart.
Second Problem:
I have two Python distributions installed in my machine. When I execute manually the script, one is used. When Cron executes the script the other one is used. Furthermore I noticed it when first problem was fixed. I got an error of module not found when running the Python script by Cron in the log file, but perfectly worked when manually executed. Hence I have seen that when using pip install <module-name>, it is just for one distro. To check wich version of Python I was using:
which Python
Conclussion: Be smart, don't be like me, don't mess with multiple Python distributions.
Bonus: Always use full paths to be clear. Cron has different env than yours.

Related

Cron SyntaxError with Selenium Python Script

I'm a beginner trying to write a small Python script that I will run through Cron, but I'm getting a Syntax Error from my Cron output log about the program, although the script works fine running through Terminal.
File "/home/pi/Selenium.py", line 19
clickinput = driver.find_element_by_link_text(f"{date}- File")
^
SyntaxError: invalid syntax
I tried looking up how to do this, and one person with a similar issue changed Cron to run as Bash instead of sh, but when I added SHELL = /bin/bash to the Cron file, nothing happened, and I got the same error. Additionally, when I change the f-string into a non f-string, a "selenium module not found" error gets thrown instead.
Any help figuring out the cause of the error would be appreciated!
Cron command:
PATH = /home/pi/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/s$
SHELL = /bin/bash
20 18 * * * /home/pi/Selenium.py > /home/pi/Desktop/backdown.log 2>&1
20 18 * * * env > /home/pi/Desktop/env.output 2>&1
Here is my Python script (minus private info)
#!/usr/bin/env python
import time
from datetime import datetime
from selenium import webdriver
driver = webdriver.Chrome("/usr/lib/chromium-browser/chromedriver")
driver.get("website");
searchinput = driver.find_element_by_name('mail')
searchinput.send_keys("email")
searchinput = driver.find_element_by_name("pass")
searchinput.send_keys("password")
searchinput.submit()
date = datetime.today().strftime("%m/%d")
if date.find("0") == 0:
date = date.lstrip("0")
clickinput = driver.find_element_by_link_text(f"{date}- File")
clickinput.click()
(This is my first time posting, so sorry if it's bad)

Python Script doesn´t work when started via other script

I´m currently working on a raspberry pi 4 and wrote a script in python that send a mail with a picture and then rename the file and puts it in another folder.
The script works fine when I start with command
sudo python script.py
but when start it with another script it won´t execute the part with the renaming
Now the question what is my mistake ?
import os
import time
from sendmail import mail
from sendmail import file_rename
from time import sleep
pic = '/home/pi/Monitor/Bewegung.jpg'
movie= '/home/pi/Monitor/Aufnahme.avi'
archiv = '/home/pi/Archiv/'
time = time.strftime('%d.%m.%Y %H:%M')
mail(filename = pic )
file_rename(oldname = pic ,name = 'Serverraum Bild' + time ,format = '.jpg' ,place = archiv )
file_rename(oldname = movie ,name = 'Serverraum Video' + time ,format = '.avi' ,place = archiv )
I see that you are starting the script as a user with sudo privileges.
but when start it with another script it won´t execute the part with the renaming
This makes me suspicious that the caller script does not have the correct permissions to rename/move a file. You can view the permissions of the script with the following command
ls -la callerscript.py

What does "GetPassWarning: Can not control echo on the terminal" mean? "Inappropriate ioctl for device"?

I wrote a script that sends an email with the daily holiday. It works when run independently. I also wrote a script that sets a cronjob for it to run automatically, which correctly sets a cronjob, but the emails never send. The cronjob does work for other test scripts though. So both pieces work independently, just not for the email script. I checked the terminal mail and here's the error that arises which I don't know how to fix:
/anaconda3/lib/python3.7/getpass.py:91: GetPassWarning: Can not control echo on the terminal.
passwd = fallback_getpass(prompt, stream)
Warning: Password input may be echoed.
Password for <bggoldberg33#gmail.com>: Traceback (most recent call last):
File "/anaconda3/lib/python3.7/getpass.py", line 69, in unix_getpass
old = termios.tcgetattr(fd) # a copy to save
termios.error: (25, 'Inappropriate ioctl for device')
I'm fairly new to Python so can anyone help explain what this means and how to resolve it?
Python version 3.7.3 and using command line to run on MacOS 10.13.4
Here are the email and cron scripts for reference:
import yagmail
import pandas as pd
from datetime import datetime
# create variables
df = pd.read_excel('/Users/bgoldberg/PythonScripts/FunStuff/holidays.xlsx')
df['DateCheck'] = df['Date'] == datetime.today().strftime('%m/%d')
df = df.loc[df['DateCheck'] == True]
df.reset_index(inplace=True)
if len(df) == 1:
holiday1 = df.at[0, 'Holiday']
link1 = df.at[0, 'Link']
content = f'Happy {holiday1}! {link1}'
elif len(df) > 1:
holiday1 = df.at[0, 'Holiday']
link1 = df.at[0, 'Link']
holiday2 = df.at[1, 'Holiday']
link2 = df.at[1, 'Link']
content = f'Happy {holiday1}! {link1} \nHappy {holiday2}! {link2}'
else:
pass
weekday = datetime.today().strftime('%A')
# create email
receiver = 'bggoldberg33#gmail.com'
body = f'{content}'
yag = yagmail.SMTP('bggoldberg33#gmail.com')
yag.send(
to = receiver,
subject = 'Fun Daily Holidays',
contents = body
)
Cronjob:
from crontab import CronTab
my_cron = CronTab(user='bgoldberg')
job = my_cron.new(command='/anaconda3/bin/python3 /Users/bgoldberg/PythonScripts/FunStuff/emailTest.py')
job.minute.every(1)
my_cron.write()
I've just come across this problem myself and managed to find the answer - hopefully it's still of use to you.
The problem is that the yagmail keychain entry you've created is in the login keychain, but cron cannot access the login keychain because it is not run from an interactive session. Cron can only access entries from the System keychain.
What you need to do is open the Keychain Access app and copy the yagmail entry from login to System. If you now try to run your cron job it should work.
This answer was of use in finding the solution.
Note that in theory you should also be able to do this from the command line using the macOS security command to export and import keychain entries, but I haven't tested this.

Sqlite python - attempt to write a read only database

I have a simple python script that puts data in a database. Both the script and
the database have owner www-data. When I run sudo python and write the
commands one by one it works, but if I run python monitor.py or sudo python monitor.py it doesn't work; it says, "attempt to write a read only database".
This is my script: (it receives data from arduino)
from serial import Serial
from time import sleep
import sqlite3
serial_port = '/dev/ttyACM0';
serial_bauds = 9600;
# store the temperature in the database
def log_light(value):
conn=sqlite3.connect('/var/db/arduino.db')
curs=conn.cursor()
curs.execute("UPDATE sensor1 set status = (?)", (value,))
# commit the changes
conn.commit()
conn.close()
def main():
s = Serial(serial_port, serial_bauds);
s.write('T');
sleep(0.05);
line = s.readline();
temperature = line;
s.write('H');
sleep(0.05);
line = s.readline();
humidity = line;
s.write('L');
sleep(0.05);
line = s.readline();
light = line;
log_light(light);
if __name__=="__main__":
main()
It sounds like a permission problem. Write access is granted only to the user, which is root. You need to change the user to be directly yourself, not root. You can do this using chmod on many *nix systems.
You could also gove write access to anyone in the group.

Python script failing from crontab

I've gone through multiple threads, but I still can't seem to find my problem.
I'm building a really simple Twitter bot that I'd like to fire every hour, on the hour with a cron job from a Raspberry Pi. Here's my crontab:
PYTHONPATH=/usr/bin/python
MAILTO=*myemail*
00 * * * * /home/username/directory/my_script.py >> /var/log/cron.log
Then the script:
#! /usr/bin/env python
import sys
from twython import Twython, TwythonError
from pymarkovchain import MarkovChain
#TWITTER ACCESS
apiKey = KEY
apiSecret = SECRET
accessToken = TOKEN
accessKey = KEY
#text to pull
text = open('/home/username/directory/text.txt').read()
#Generate database and frequency table
mc = MarkovChain('/home/username/directory/markov')
mc.generateDatabase(text)
tweet = mc.generateString()
api = Twython(apiKey,apiSecret,accessToken,accessKey)
try:
api.update_status(status=tweet)
except TwythonError as e:
print e
The first thing I checked was all of my referenced files to make sure they were absolute references. Then, I checked to make sure my file paths were correct. I'm really stumped here. Running the script from command line with the full path works as expected. Any thoughts are appreciated.
After trying the suggestions above and reading countless articles, I learned that cron has to run as root, not as the user. I checked the logs and saw that the user calling the script was the owner of the file, not root. So, running chmod a+x my_script.py took care of it.
Thanks for all the suggestions - especially those getting the errors to the correct log file.
To debug better, you might want to redirect stderr:
00 * * * * /home/username/directory/my_script.py >> /tmp/cron.log 2>&1
# or
00 * * * * /home/username/directory/my_script.py >> /tmp/cron.log 2>/tmp/cron-error.log
(I also changed the path there to make sure your cron user has permission to write output.)
Another thing you could try is run the script with Python in cron:
00 * * * * python /home/username/directory/my_script.py >> /tmp/cron.log 2>&1

Categories