We Keep Coding

Cron job with django application

I would like to use a cron task in order to delete media files if the condition is True.
Users generate export files stored in the Media folder. In order to clean export files in the background, I have a Cron task which loops over each file and looks if the expiry delay is passed or not.
I used django-cron library
Example:
File in Media Folder : Final_Products___2019-04-01_17:50:43.487845.xlsx
My Cron task looks like this :
class MyCronExportJob(CronJobBase):
""" Cron job which removes expired files at 18:30 """
RUN_AT_TIMES = ['18:30']
schedule = Schedule(run_at_times=RUN_AT_TIMES)
code = 'app.export_cron_job'
def do(self):
now = datetime.datetime.now()
media_folder = os.listdir(os.path.join(settings.MEDIA_ROOT, 'exports'))
for files in media_folder:
file = os.path.splitext(files.split(settings.EXPORT_TITLE_SEPARATOR, 1)[1])[0]
if datetime.datetime.strptime(file, '%Y-%m-%d_%H:%M:%S.%f') + timedelta(minutes=settings.EXPORT_TOKEN_DELAY) < now:
os.remove(os.path.join(os.path.join(settings.MEDIA_ROOT, 'exports'), files))
# settings.EXPORT_TOKEN_DELAY = 60 * 24
I edited my crontab -e :
30 18 * * * source /home/user/Bureau/Projets/app/venv/bin/activate.csh && python /home/user/Bureau/Projets/app/src/manage.py runcrons --force app.cron.MyCronExportJob
Then I launched service cron restart
But nothing as changed. My file is still there. However, it should be removed because his date is greater than now + settings.EXPORT_TOKEN_DELAY
I'm using Ubuntu to local dev and FreeBSD as a production server environment.
EDIT:
I tried some things but crontab doesn't work for the moment.
1) * * * * * /bin/date >> /home/user/Bureau/Projets/app/cron_output
==> It works, so crontab works
2) I ran : python manage.py runcrons in my console
==> It works
3) I ran this script (cron.sh):
source /home/user/.bashrc
cd /home/user/Bureau/Projets/app
pyenv activate app
python src/manage.py runcrons --force
deactivate
==> It works
4) I ran this crontab line :
35 10 * * * /home/user/Bureau/Projets/app/utility/cron.sh
==> Service restarted at 10h32, I waited until 10h38 : nothing !

Jinja not imported when executing script from another one

I have a web server with CGI script calling python scripts.
When i try to execute in a main file (test1.py) another script called via
os.system('/var/www/cgi-bin/readIRtemp.py '+arg1+' '+arg2+' '+arg3)
I get his error message in /var/log/apache2/error.log :
import: not found
from: can't read /var/mail/jinja2
this is understandable for me since when called directly from the python console my script works !
its content is:
import sys, os
from jinja2 import Environment, FileSystemLoader, select_autoescape
last20values=sys.argv[1]
currTempInDegreesCelcius=sys.argv[2]
print('test '+last20values+' '+currTempInDegreesCelcius)
env = Environment(
loader=FileSystemLoader('/var/www/html/templates'),
autoescape=select_autoescape(['html', 'xml'])
)
template = env.get_template('IR.html')
updatedTemplate=template.render( arrayOfTemp = last20values, currTemp=currTempInDegreesCelcius)
Html_file=open("/var/www/html/IR.html","w")
Html_file.write(updatedTemplate)
Html_file.close()
I read somewhere something like maybe when calling os.system() the script is running with a different user account or some crazy things like that ... please help!
of course i chmod 777 * everything but that doesnt help ...

How to access an AWS Lambda environment variable from Python

Using the new environment variable support in AWS Lambda, I've added an env var via the webui for my function.
How do I access this from Python? I tried:
import os
MY_ENV_VAR = os.environ['MY_ENV_VAR']
but my function stopped working (if I hard code the relevant value for MY_ENV_VAR it works fine).

AWS Lambda environment variables can be defined using the AWS Console, CLI, or SDKs. This is how you would define an AWS Lambda that uses an LD_LIBRARY_PATH environment variable using AWS CLI:
aws lambda create-function \
--region us-east-1
--function-name myTestFunction
--zip-file fileb://path/package.zip
--role role-arn
--environment Variables={LD_LIBRARY_PATH=/usr/bin/test/lib64}
--handler index.handler
--runtime nodejs4.3
--profile default
Once created, environment variables can be read using the support your language provides for accessing the environment, e.g. using process.env for Node.js. When using Python, you would need to import the os library, like in the following example:
...
import os
...
print("environment variable: " + os.environ['variable'])
Resource Link:
AWS Lambda Now Supports Environment Variables
Assuming you have created the .env file along-side your settings module.
.
├── .env
└── settings.py
Add the following code to your settings.py
# settings.py
from os.path import join, dirname
from dotenv import load_dotenv
dotenv_path = join(dirname(__file__), '.env')
load_dotenv(dotenv_path)
Alternatively, you can use find_dotenv() method that will try to find a .env file by (a) guessing where to start using file or the working directory -- allowing this to work in non-file contexts such as IPython notebooks and the REPL, and then (b) walking up the directory tree looking for the specified file -- called .env by default.
from dotenv import load_dotenv, find_dotenv
load_dotenv(find_dotenv())
Now, you can access the variables either from system environment variable or loaded from .env file.
Resource Link:
https://github.com/theskumar/python-dotenv
gepoggio answered in this post: https://github.com/serverless/serverless/issues/577#issuecomment-192781002
A workaround is to use python-dotenv:
https://github.com/theskumar/python-dotenv
import os
import dotenv
dotenv.load_dotenv(os.path.join(here, "../.env"))
dotenv.load_dotenv(os.path.join(here, "../../.env"))
It tries to load it twice because when ran locally it's in
project/.env and when running un Lambda the .env is located in
project/component/.env

Both
import os
os.getenv('MY_ENV_VAR')
And
os.environ['MY_ENV_VAR']
are feasible solutions, just make sure in the lambda GUI that the ENV variables are actually there.

I used this code; it includes both cases, setting the variable from the handler and setting it from outside the handler.
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""Trying new lambda stuff"""
import os
import configparser
class BqEnv(object):
"""Env and self variables settings"""
def __init__(self, service_account, configfile=None):
config = self.parseconfig(configfile)
self.env = config
self.service_account = service_account
#staticmethod
def parseconfig(configfile):
"""Connection and conf parser"""
config = configparser.ConfigParser()
config.read(configfile)
env = config.get('BigQuery', 'env')
return env
def variable_tests(self):
"""Trying conf as a lambda variable"""
my_env_var = os.environ['MY_ENV_VAR']
print my_env_var
print self.env
return True
def lambda_handler(event, context):
"""Trying env variables."""
print event
configfile = os.environ['CONFIG_FILE']
print configfile
print type(str(configfile))
bqm = BqEnv('some-json.json', configfile)
bqm.variable_tests()
return True
I tried this with a demo config file that has this:
[BigQuery]
env = prod
And the setting on lambda was the following:
Hope this can help!

os.environ["variable_name"]
In the configuration section of AWS lambda, make sure you declare the variable with the same name that you're trying to access here. For this example, it should be variable_name

Running xlwings on a server

Solved the problem by making sure that python27.dll and win32api into the PYTHON_WIN directory. The two files with an arrow were missing in the server folder (P:/)
I am using udf module of xlwings and setting xlwings.bas configuration into a server. (I have my personal disk C:/ however other users need to run this macro and the file must be stored in P:/)
I have python, xlwings and other modules installed in P:/, but I get some strange error about importing os lib. Does any one have ever passed through this ?
Thanks in advance.
I got this from python:
"Python process exited before it was possible to create the interface object.
Command: P:\Python 27\python.exe -B -c ""import sys, os;sys.path.extend(os.path.normcase(os.path.expandvars(r'P:\Risco\Python Scripts')).split(';'));import xlwings.server; xlwings.server.serve('{2bb649ad-487e-48d5-ab31-24adaeefca59}')""
Working Dir: "
Xlwings.bas is set like this:
PYTHON_WIN = "P:\Python 27\python.exe"
PYTHON_MAC = ""
PYTHON_FROZEN = ThisWorkbook.Path & "\build\exe.win32-2.7"
PYTHONPATH = "P:\Risco\Python Scripts"
UDF_MODULES = "FetchPL"
UDF_DEBUG_SERVER = False
LOG_FILE = ""
SHOW_LOG = True
OPTIMIZED_CONNECTION = False

Activate Virtual Environment and Install Packages via Script

I often find myself recreating file structures for Flask apps so I have decided to make a script to do all that for me. I would like the script to create all the folders I need as well as the files with some basic boilerplate, which it does, that part is working fine. However I would also like to create a virtual environment and install Flask to that environment. That is where I am encountering the problem. The script runs but it installs Flask to my system installation of Python.
I followed the advice in this question here but it's not working. I am running Ubuntu 12.04.4 LTS via crouton on a Chromebook.
#!/usr/bin/python
from os import mkdir, chdir, getcwd, system
import sys
APP_NAME = sys.argv[1]
ROOT = getcwd()
PROJECT_ROOT = ROOT + '/' + APP_NAME
# dictionary represents folder structure. Key is the folder name and the value is it's contents
folders = {APP_NAME : {'app' : {'static': {'css' : '', 'img' : '', 'js' : ''}, 'templates' : ''} } }
def create_folders(dic):
for key in dic:
if isinstance(dic[key], dict):
mkdir(key)
prev = getcwd() + '/' + key
chdir(prev)
create_folders(dic[key])
else:
mkdir(key)
create_folders(folders)
chdir(PROJECT_ROOT)
open('config.py', 'a').close()
with open('run.py', 'a') as run:
run.write("""stuff""")
with open('app/__init__.py', 'a') as init:
init.write("""stuff""")
with open('app/views.py', 'a') as views:
views.write("""stuff""")
open('app/models.py', 'a').close()
open('app/forms.py', 'a').close()
with open('app/templates/layout.html', 'a') as layout:
layout.write("""stuff""")
system('chmod a+x run.py')
system('virtualenv venv')
system('. venv/bin/activate;sudo pip install flask') # this does not seem to be working the way I am expecting it to

I suppose your calls are not within the same console session and therefore the console environment is not as expected. I suggest to concatenate the related commands in one system call using subprocess.Popen like this (including suggestions by limasxgoesto0):
subprocess.Popen('virtualenv venv;source venv/bin/activate;pip install flask')
You should probably be using subprocess; os.system is deprecated.