We Keep Coding

Automatic script doesn't output data - python

I am just for fun collecting weather data with my Raspberry Pi.
If I execute my python script in the console everything is working fine.But if I add the python-file to crontab to start it after rebooting, it isn't working. (crontab-entry: #reboot python3 /home/pi/Documents/PythonProgramme/WeatherData/weatherdata.py &)
#! /usr/bin/python3
from pyowm import OWM
import csv
import schedule
from datetime import datetime
import time
key = 'XXXXXX'
def weather_request(text):
owm = OWM(key)
mgr = owm.weather_manager()
karlsruhe = mgr.weather_at_place('Karlsruhe, DE').weather
hamburg = mgr.weather_at_place('Hamburg, DE').weather
cities = (karlsruhe, hamburg)
with open('weatherdata.csv', 'a') as file:
writer = csv.writer(file)
row = [datetime.now().strftime("%Y-%m-%d %H:%M:%S")]
for city in cities:
row.append(city.temperature('celsius')['temp'])
row.append(round(row[1] - row[2], 2))
row.append(text)
writer.writerow(row)
schedule.every().day.at("08:00").do(weather_request, 'morgens')
schedule.every().day.at("13:00").do(weather_request, 'mittags')
schedule.every().day.at("18:00").do(weather_request, 'abends')
while 1:
schedule.run_pending()
time.sleep(1)
If I run ps -aef | grep python it is showing, that my script is running: pi 337 1 21 10:32 ? 00:00:10 python3 /home/pi/Documents/PythonProgramme/WeatherData/weatherdata.py
But I never get any data. What am I missing?
Thanks in advance!

where are you checking the output file?
Have tried to open the file with full path?
with open('***<fullPath>***weatherdata.csv', 'a') as

Related

I'm currently working on a script for my sensor on my Raspberry Pi. The code underneath should get the values of my sensor and write it into a the data.json file. My problem is, if I run the scipt with my the Thonny editor everything works but if I add the script to my crontab menu the data does not get written to the data.json file.
The Code:
import time
import board
import adafruit_dht
import psutil
import io
import json
import os
from gpiozero import LED
from datetime import date
from datetime import datetime
# We first check if a libgpiod process is running. If yes, we kill it!
for proc in psutil.process_iter():
if proc.name() == "libgpiod_pulsein" or proc.name() == "libgpiod_pulsei":
proc.kill()
sensor = adafruit_dht.DHT11(board.D23)
# init
temp_values = [10]
hum_values = [10]
counter = 0
dataLED = LED(13)
dataList = []
def errSignal():
for i in range(0,3):
dataLED.on()
time.sleep(0.1)
dataLED.off()
time.sleep(0.1)
#on startup
def runSignal():
for i in range(0,5):
dataLED.on()
time.sleep(0.2)
dataLED.off()
time.sleep(0.2)
def getExistingData():
with open('data.json') as fp:
dataList = json.load(fp)
print(dataList)
def startupCheck():
if os.path.isfile("data.json") and os.access("data.json", os.R_OK):
# checks if file exists
print("File exists and is readable.")
# get json data an push into arr on startup
getExistingData()
else:
print("Either file is missing or is not readable, creating file...")
# create json file
with open("data.json", "w") as f:
print("The json file is created.")#
def calc_avgValue(values):
sum = 0
for iterator in values:
sum += iterator
return sum / len(values)
def onOFF():
dataLED.on()
time.sleep(0.7)
dataLED.off()
# data led blinking on startup
runSignal()
# checks if file exists
startupCheck()
while True:
try:
temp_values.insert(counter, sensor.temperature)
hum_values.insert(counter, sensor.humidity)
counter += 1
time.sleep(6)
if counter >= 10:
print(
"Temperature: {}*C Humidity: {}% ".format(
round(calc_avgValue(temp_values), 2),
round(calc_avgValue(hum_values), 2)
)
)
# get time
today = date.today()
now = datetime.now()
# create json obj
data = {
"temperature": round(calc_avgValue(temp_values), 2),
"humidity": round(calc_avgValue(hum_values), 2),
"fullDate": str(today),
"fullDate2": str(today.strftime("%d/%m/%Y")),
"fullDate3": str(today.strftime("%B %d, %Y")),
"fullDate4": str(today.strftime("%b-%d-%Y")),
"date_time": str(now.strftime("%d/%m/%Y %H:%M:%S"))
}
# push data into list
dataList.append(data)
# writing to data.json
with open("data.json", "w") as f:
json.dump(dataList, f, indent=4, separators=(',',': '))
# if data is written signal appears
onOFF()
print("Data has been written to data.json...")
counter = 0
except RuntimeError as error:
continue
except Exception as error:
sensor.exit()
while True:
errSignal()
raise error
time.sleep(0.2)
Crontab Menu:
The line in the center is the script.

Investigation areas:
Do not put & in crontab, it serves no purpose.
You should capture the output of your scripts to see what is going on. You do this by adding >/tmp/stats.out 2>/tmp/stats.err (and similar for the other 2 lines). You will see what output and errors your scripts encounter.
cron does not run your scripts in the same environment, and from the same directory you are running them. Load what you require in the script.
cron might not have permissions to write into data.yml in the directory it is running from. Specify a full path, and ensure cron can write in that directory.
Look at https://unix.stackexchange.com/questions/109804/crontabs-reboot-only-works-for-root for usage of #reboot. Things that should occur at startup should be configured through systemd or init.d (I do not know what Rasperry Pie uses vs distro). Cron is to schedule jobs, not run things at startup.
It could be as simple as not having python3 in the PATH configured in cron.

I want to use Cron to execute my python script every hour of the day. Therefore I created a cronjob that looks like: #hourly /home/pi/Desktop/repository/auslastung_download/auslastung.py
The cronjob should execute the following script:
from bs4 import BeautifulSoup
from selenium.webdriver.firefox.options import Options as FirefoxOptions
from selenium import webdriver
from datetime import datetime, date
def get_auslastung_lichtenberg():
try:
url = "https://www.mcfit.com/de/fitnessstudios/studiosuche/studiodetails/studio/berlin-lichtenberg/"
options = FirefoxOptions()
options.add_argument("--headless")
driver = webdriver.Firefox(options=options)
driver.get(url)
html_content = driver.page_source
soup = BeautifulSoup(html_content, 'html.parser')
elems = soup.find_all('div', {'class': 'sc-iJCRLp eDJvQP'})
#print(elems)
auslastung = str(elems).split("<span>")[1]
#print(auslastung)
auslastung = auslastung[:auslastung.rfind('</span>')]
#print(auslastung)
auslastung = str(auslastung).split("Auslastung ")[1]
#print(auslastung)
auslastung = auslastung[:auslastung.rfind('%')]
print(auslastung)
now = datetime.now()
current_time = now.strftime("%H:%M:%S")
#print("Current Time =", current_time)
today = date.today()
print(today)
ergebnis = {'date': today, 'time': current_time,'studio': "Berlin Lichtenberg", 'auslastung': auslastung}
return ergebnis
finally:
try:
driver.close()
except:
pass
"""
import json
with open('database.json', 'w') as f:
json.dump(get_auslastung_lichtenberg(), f)
"""
import csv
with open('/home/pi/Desktop/repository/auslastung_download/data.csv', mode='a') as file:
fieldnames = ['date', 'time', 'studio', 'auslastung']
writer = csv.DictWriter(file, fieldnames=fieldnames)
writer.writerow(get_auslastung_lichtenberg())
When executed via python3 auslastung.pyeverything works fine and the script writes into the data.csv file.
Maybe someone can help me :)

First of all you must ensure that your script runs.
If interactively you run python3 auslastung.py why do you invoke your python script differently on your cron.
have you tried to run just /home/pi/Desktop/repository/auslastung_download/auslastung.py interactively? without initial python3, does it run?
If your script runs with python3 auslastung.py on your crontab you should include full path to both interpreter and script:
#hourly /paht/to/python3 /full/path/to/script.py
If you made your script to run directly without need to indicate interpreter, just /full/path/to/script.py then on your crontab you should include full path to script:
#hourly /full/path/to/script.py
You may include a shebang: a very first line of your script indicate which interpreter is used to execute it. So your first line should be #!/path/to/your/interpreter
An you have to ensure then that script has execute permision with chmod +x auslastung.py.

I´m currently working on a raspberry pi 4 and wrote a script in python that send a mail with a picture and then rename the file and puts it in another folder.
The script works fine when I start with command
sudo python script.py
but when start it with another script it won´t execute the part with the renaming
Now the question what is my mistake ?
import os
import time
from sendmail import mail
from sendmail import file_rename
from time import sleep
pic = '/home/pi/Monitor/Bewegung.jpg'
movie= '/home/pi/Monitor/Aufnahme.avi'
archiv = '/home/pi/Archiv/'
time = time.strftime('%d.%m.%Y %H:%M')
mail(filename = pic )
file_rename(oldname = pic ,name = 'Serverraum Bild' + time ,format = '.jpg' ,place = archiv )
file_rename(oldname = movie ,name = 'Serverraum Video' + time ,format = '.avi' ,place = archiv )

I see that you are starting the script as a user with sudo privileges.
but when start it with another script it won´t execute the part with the renaming
This makes me suspicious that the caller script does not have the correct permissions to rename/move a file. You can view the permissions of the script with the following command
ls -la callerscript.py

I'm trying to change delimiter on CSV and write to a new file, is just a simple modification but isn't it.
#!/usr/bin/python
#-*- econde: utf-8 -*-
import sys
import csv
def main():
r = open(sys.argv[1],"r")
wr = open(sys.argv[2],"a+")
rea = csv.reader(r, delimiter=',')
writer = csv.writer(wr,delimiter="|", quotechar="'")
for row in rea:
#line = str(row).replace(",","|")
#writer.writerow("".join(line))
writer.writerow(row)
print type(row)
print row
r.close()
wr.close()
if __name__ == '__main__':
main()
Update:
The output in console looks so so like:
./csv_read.py fim.csv salida.csv
<type 'list'>
['9/17/18 22:29', 'any', 'la_cuerda.net', 'Sep 17 22:29:29 running yum[37144]: Installed: ImageMagic-toolkit-2.1.7-1.noarch']
but in the file is writing 3 times the same string but on three different ways
the first way is still the same: 1 char per field (wrong format and brackets are included)
the second way is inserting all in one cell without split it like the original
This is the content of the Input File and the Output file
$ cat Input.csv
Time(GMT),Host,dest,Alert
9/17/18 22:34,any,google.com.mx,monitor: Agent started: 'discovery.channel.org->any'.
9/17/18 22:29,any,la_cuerda.net,Sep 17 22:29:29 running yum[37144]: Installed: ImageMagic-toolkit-2.1.7-1.noarch
$ cat Output.csv
[,'''',T,i,m,e,(,G,M,T,),'''',|, ,'''',H,o,s,t,'''',|, ,'''',d,e,s,t,'''',|, ,'-''',A,l,e,r,t,'''',]
[,'''',9,/,1,7,/,1,8, ,2,2,:,3,4,'''',|, ,'''',a,n,y,'''',|, ,'''',g,o,o,g,l,e,.,c,o,m,.,m,x,'''',|, ,",m,o,n,i,t,o,r,:, ,A,g,e,n,t, ,s,t,a,r,t,e,d,:, ,'''',d,i,s,c,o,v,e,r,y,.,c,h,a,n,n,e,l,.,o,r,g,-,>,a,n,y,'''',.,",]
[,'''',9,/,1,7,/,1,8, ,2,2,:,2,9,'''',|, ,'''',a,n,y,'''',|, ,'''',l,a,_,c,u,e,r,d,a,.,n,e,t,'''',|, ,'''',S,e,p, ,1,7, ,2,2,:,2,9,:,2,9, ,r,u,n,n,i,n,g, ,y,u,m,[,3,7,1,4,4,],:, ,I,n,s,t,a,l,l,e,d,:, ,I,m,a,g,e,M,a,g,i,c,-,t,o,o,l,k,i,t,-,2,.,1,.,7,-,1,.,n,o,a,r,c,h,'''',]
Time(GMT)|Host|dest|Alert
9/17/18 22:34|any|google.com.mx|'monitor: Agent started: ''discovery.channel.org->any''.'
9/17/18 22:29|any|la_cuerda.net|Sep 17 22:29:29 running yum[37144]: Installed: ImageMagic-toolkit-2.1.7-1.noarch
Time(GMT)|Host|dest|Alert
9/17/18 22:34|any|google.com.mx|'monitor: Agent started: ''discovery.channel.org->any''.'
9/17/18 22:29|any|la_cuerda.net|Sep 17 22:29:29 running yum[37144]: Installed: ImageMagic-toolkit-2.1.7-1.noarch
Time(GMT)|Host|dest|Alert
9/17/18 22:34|any|google.com.mx|'monitor: Agent started: ''discovery.channel.org->any''.'
9/17/18 22:29|any|la_cuerda.net|Sep 17 22:29:29 running yum[37144]: Installed: ImageMagic-toolkit-2.1.7-1.noarch

wr = open(sys.argv[2],"a+") is the cause. Each time you run your program, it appends its output to the file. The bogus data you see is from previous runs.
Unless your program is really supposed to append to the output file rather than overwrite it, open the file in wb mode.
Also note that csv.reader and csv.writer docs mandate opening the file in binary mode (because the module is supposed to do its own transcoding).

I set up a crontab file to execute a python file every minute however it states it executed but no file is generated.
crontest1.py file contains:
#!/usr/local/bin/python
# -*- coding: utf-8 -*-
if __name__ == '__main__':
f = open('/APPS/CronRun/crontest/dummy1.txt','w')
f.write('hello world it is a file')
f.close()
Crontab file:
# /etc/crontab - root's crontab for FreeBSD
#
# $FreeBSD: release/10.0.0/etc/crontab 194170 2009-06-14 06:37:19Z brian $
#
SHELL=/bin/sh
PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin
#
#minute hour mday month wday who command
#
* * * * * root /APPS/CronRun/crontest/crontest1.py
11 11 * * * root /usr/bin/find /APPS/* | grep python | grep core | xargs rm
14 14 17 * * root /APPS/CronRun/Report/report.py
The crontab executes but no file appears.
I tried a simple date >> datelog.txt
which works via crontab but the python file does not seem to execute.
The python file works if executed manually from the shell but not via crontab
I have tried explicitly stating: python /APPS/CronRun/crontest/crontest1.py in the crontab file, but this does not work
I have previously added cron jobs that work fine and still execute daily without any issues
e.g
31,01 * * * * root /APPS/CronRun/makelist/list.py

Try the code without :
if __name__ == '__main__':
Like this:
#!/usr/local/bin/python
# -*- coding: utf-8 -*-
f = open('/APPS/CronRun/crontest/dummy1.txt','w')
f.write('hello world it is a file')
f.close()
Maybe " crontest1.py " is executed by crontab as a module - not "main".
If that doesn't help try to update the Python.
Python 3.1.x and lower
may be the reason.