We Keep Coding

zc.buildout variable substitution prepends a "="

When using collective.recipe.template, my template is generated fine, except that variables are inserted with a extra = in front of it.
So in the config and template below, the generated bin/npm script, on the third line ends up reading:
cd = /home/andre/dev/myapp/webapp/frontend/static
... when if fact it should be:
cd /home/andre/dev/myapp/webapp/frontend/static
buildout.cfg
[baseconfig]
webapp_dir= = ${buildout:directory}/webapp
resources_dir = ${buildout:directory}/resources
playbooks_dir = ${buildout:directory}/playbooks
npm_root = ${:resources_dir}/static/
[config]
<= baseconfig
npm_root = ${:webapp_dir}/frontend/static
[npm]
recipe = collective.recipe.template
input = templates/npm.sh.in
output = ${buildout:bin-directory}/npm
mode = 744
templates/npm.sh.in
#!/bin/bash
cd ${config:npm_root}
case "$1" in
install)
npm install
;;
workbox)
npm run workbox
;;
copy-workbox)
npm run copy-workbox
;;
build-bower)
npm run build-bower
;;
build-sw)
npm run build-sw
;;
build-all)
npm run build-all
;;
*)
echo $"Usage: $0 {install|workbox|copy-workbox|build-bower|build-sw|build-all}"
exit 1
esac

Woops, discovered my error. Typo in webapp_dir which I never saw for some reason even after reading and re-reading multiple times.

My daemon die directly after starting

I have to develop a program with Python3 which executes automatically programs at specific times. I must use a daemon and I can't use external libraries.
That's why I create an install program which configure the daemon (I followed this tutorial https://openclassrooms.com/courses/faire-un-demon-sous-linux).
The file gobatch is the program which executes automatically programs.
#! /usr/bin/python3
# -*- coding: utf8 -*-
import os
import sys
import time
import shutil
# Check if script is start with root rights
if os.geteuid() != 0:
exit('You need to have root privileges to run this script. Use \'sudo ./install.py\'.')
# Title
print('----- ProcessManager install ----- \n')
time.sleep(1)
# Get path of gobatch file
gobatchPath = input("Please enter the path (absolute) where the ProcessManager gobatch program is located: ")
# Check if file exists
try:
with open(gobatchPath):
pass
except IOError:
exit('Error : file does not exists.')
# Copy gobatch file into /usr/bin/
shutil.copy(gobatchPath, '/usr/bin/gobatch')
# Create and fill the automatic launch program
description = 'Deamon that allows you to run cyclicaly at a date or a specific time a program'
fileContent = '#### BEGIN INIT INFO \n' \
'# Provides: chillispot et freeradius dans le chroot \n' \
'# Required-Start: $local_fs $network \n' \
'# Required-Stop: $local_fs $remote_fs _\n' \
'# Default-Start: 2 3 4 5 \n' \
'# Default-Stop: 0 1 6 \n' \
'# Short-Description: Wireless & LAN Access Point Controller \n' \
'# Description: ChilliSpot is an open source captive portal \n' \
'# or wireless LAN access point controller. \n' \
'### END INIT INFO \n\n\n' \
'DESC="' + description + '"\n' \
'DEAMON=/usr/bin/gobatch'
file = open('/etc/init.d/gobatch', 'w')
file.write(fileContent)
file.close()
# Give the rights
os.chmod('/etc/init.d/gobatch', 0o755)
# Save gobatch file to be active
os.system('update-rc.d gobatch defaults')
However, when I start the service with : /etc/init.d/gobatch start and displaying the status with service gobatch status I get
gobatch.service - SYSV: ChilliSpot is an open source captive portal
Loaded: loaded (/etc/init.d/gobatch; bad; vendor preset: enabled)
Active: inactive (dead)
Docs: man:systemd-sysv-generator(8)
Can you tell me if my way is the right method or it exists other solutions?
And do you have any idea why my daemon die directly after starting?
Thank's for help!
UPDATE
Currently, my gobatch file is not a real program because I prefer create the deamon before.
gobatch file
#! /bin/bash
echo "it works !"
My work is to create the installer of the daemon, not the gobatch program. It's another person who must do that. That's why I'm using a minmial content.
UPDATE 2:
My new /etc/init.d/gobatch :
#!/bin/sh
#### BEGIN INIT INFO
# Provides: chillispot et freeradius dans le chroot
# Required-Start: $local_fs $network
# Required-Stop: $local_fs $remote_fs _
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Wireless & LAN Access Point Controller
# Description: ChilliSpot is an open source captive portal
# or wireless LAN access point controller.
### END INIT INFO
PATH=/bin:/usr/bin:/sbin:/usr/sbin
DESC="Deamon that allows you to run cyclicaly at a date or a specific time a program"
NAME=gobatch
DEAMON=/usr/sbin/gobatch
PIDFILE=/var/run/$NAME.pid
SCRIPTNAME=/etc/init.d/"$NAME"
. /lib/lsb/init-functions
case "$1" in
start) log_daemon_msg "Starting gobatch"
start_daemon -p $PIDFILE $DAEMON
log_end_msg $?
;;
stop) log_daemon_msg "Stopping gobatch"
killproc -p $PIDFILE $DAEMON
RETVAL=$?
[ $RETVAL -eq 0 ] && [ -e "$PIDFILE" ] && rm -f $PIDFILE
log_end_msg $RETVAL
;;
restart) log_daemon_msg "Restarting gobatch"
$0 stop
$0 start
;;
esac
exit 0
With this code I get this error :
gobatch.service: Failed at step EXEC spawning /etc/init.d/gobatch: Exec format error
UPDATE 3:
The last error into logs file is :
janv. 08 15:33:07 ubuntu systemd[1]: Failed to start SYSV: ChilliSpot is an open source captive portal.
I hope after that, my program will works.

Your script echoes and exits immediately, so it is not a deamon. A daemon needs to continue running after invocation.
You need to have a program that behaves like a deamon in order to test its installation. The installation will not create a deamon out of a regular program. It will only prepare the environment for running it.
The question How do you create a daemon in Python? will give you more information how to write a proper daemon in Python.

I suspect that once your gobatch prints the executions is done.
Try
#! /bin/bash
for i in $(seq 1 10)
do
echo "it works !"
sleep 1
done

Python startup script using STDOUT for outputs, using init.d method

My environment is Debian Jesse on a embedded arm display.
I am using a bash script to automatically launch my app using the init.d method. It launches a second python script as a daemon that handles my application on startup and reboot.
Because it is run this way to the best of my knowledge this is now a daemon background process, and it disconnects STDOUT and STDIN from my python script.
The system and application is for a single purpose, so spamming the console with outputs from a background process is not only not a problem, but it is desired. With the outputs I can easily ssh or serial console into the display and see all the live debug outputs or exceptions.
I have looked into possible ways to force the process to the foreground, or redirect the outputs to STDOUT but have not found any definite answer when the script is run at startup.
My logging to a file is working perfect and otherwise the app works well in the state it is in. Currently when I need to debug I stop the application and run it manually to get all the outputs.
I have considered using sockets to redirect the outputs from the app, and then running a separate script that is listening will print to console... but that seems less than ideal and that a better solution might exist.
Is there methods to achieve this or should I just accept this.
EDIT 1 (additional details)
Because I am using multiple logs for multiple processes I have created a logger class. The stream handler uses the default which should be sys.stderr.
import logging
import logging.handlers
class LoggerSetup(object):
def __init__(self, logName, logFileNameAndPath, logSizeBytes, logFileCount):
log = logging.getLogger(logName)
log.setLevel(logging.INFO)
formatter = logging.Formatter('%(asctime)s - %(levelname)s - ' + logName + ' - %(message)s',datefmt="%m-%d %H:%M:%S")
# Add the log message handler to the logger
if(logFileCount > 0):
fileHandler = logging.handlers.RotatingFileHandler(logFileNameAndPath, maxBytes=logSizeBytes, backupCount=logFileCount)
log.addHandler(fileHandler)
fileHandler.setFormatter(formatter)
consoleHandler = logging.StreamHandler()
log.addHandler(consoleHandler)
consoleHandler.setFormatter(formatter)
log.info(logName + ' initialized')
For more reference here is the startup script launch on boot. It then runs my python run.py which handles the rest of the startup process.
#!/bin/sh
### BEGIN INIT INFO
# Provides: ArcimotoStart
# Required-Start: $remote_fs $syslog
# Required-Stop: $remote_fs $syslog
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Startup script
# Description: startip script that points to run.py
### END INIT INFO
# Change the next 3 lines to suit where you install your script and what you want to call it
DAEMON=/app/run.py
DAEMON_NAME=app
# Add any command line options for your daemon here
DAEMON_OPTS=""
# This next line determines what user the script runs as.
# Root generally not recommended but necessary if you are using certain features in Python.
DAEMON_USER=root
# The process ID of the script when it runs is stored here:
PIDFILE=/var/run/$DAEMON_NAME.pid
. /lib/lsb/init-functions
do_start () {
log_daemon_msg "Starting system $DAEMON_NAME daemon"
start-stop-daemon --start --background --pidfile $PIDFILE --make-pidfile --user $DAEMON_USER --chuid $DAEMON_USER --startas $DAEMON -- $DAEMON_OPTS
log_end_msg $?
}
do_stop () {
log_daemon_msg "Stopping system $DAEMON_NAME daemon"
start-stop-daemon --stop --pidfile $PIDFILE --retry 10
log_end_msg $?
}
case "$1" in
start|stop)
do_${1}
;;
restart|reload|force-reload)
do_stop
do_start
;;
status)
status_of_proc "$DAEMON_NAME" "$DAEMON" && exit 0 || exit $?
;;
*)
echo "Usage: /etc/init.d/$DAEMON_NAME {start|stop|restart|status}"
exit 1
;;
esac
exit 0

How to source script via python

I can source bash script (without shebang) easy as bash command in terminal but trying to do the same via python command
sourcevars = "cd /etc/openvpn/easy-rsa && . ./vars"
runSourcevars = subprocess.Popen(sourcevars, shell = True)
or
sourcevars = [". /etc/openvpn/easy-rsa/vars"]
runSourcevars = subprocess.Popen(sourcevars, shell = True)
I receive :
Please source the vars script first (i.e. "source ./vars")
Make sure you have edited it to reflect your configuration.
What's the matter, how to do it correctly?I've read some topics here,e.g here but could not solve my problem using given advices. Please explain with examples.
UPDATED:
# os.chdir = ('/etc/openvpn/easy-rsa')
initvars = "cd /etc/openvpn/easy-rsa && . ./vars && ./easy-rsa ..."
# initvars = "cd /etc/openvpn/easy-rsa && . ./vars"
# initvars = [". /etc/openvpn/easy-rsa/vars"]
cleanall = ["/etc/openvpn/easy-rsa/clean-all"]
# buildca = ["printf '\n\n\n\n\n\n\n\n\n' | /etc/openvpn/easy-rsa/build-ca"]
# buildkey = ["printf '\n\n\n\n\n\n\n\n\n\nyes\n ' | /etc/openvpn/easy-rsa/build-key AAAAAA"]
# buildca = "cd /etc/openvpn/easy-rsa && printf '\n\n\n\n\n\n\n\n\n' | ./build-ca"
runInitvars = subprocess.Popen(cmd, shell = True)
# runInitvars = subprocess.Popen(initvars,stdout=subprocess.PIPE, shell = True, executable="/bin/bash")
runCleanall = subprocess.Popen(cleanall , shell=True)
# runBuildca = subprocess.Popen(buildca , shell=True)
# runBuildca.communicate()
# runBuildKey = subprocess.Popen(buildkey, shell=True )
UPDATE 2
buildca = ["printf '\n\n\n\n\n\n\n\n\n' | /etc/openvpn/easy-rsa/build-ca"]
runcommands = subprocess.Popen(initvars+cleanall+buildca, shell = True)

There's absolutely nothing wrong with this in and of itself:
# What you're already doing -- this is actually fine!
sourcevars = "cd /etc/openvpn/easy-rsa && . ./vars"
runSourcevars = subprocess.Popen(sourcevars, shell=True)
# ...*however*, it won't have any effect at all on this:
runOther = subprocess.Popen('./easy-rsa build-key yadda yadda', shell=True)
However, if you subsequently try to run a second subprocess.Popen(..., shell=True) command, you'll see that it doesn't have any of the variables set by sourcing that configuration.
This is entirely normal and expected behavior: The entire point of using source is to modify the state of the active shell; each time you create a new Popen object with shell=True, it's starting a new shell -- their state isn't carried over.
Thus, combine into a single call:
prefix = "cd /etc/openvpn/easy-rsa && . ./vars && "
cmd = "/etc/openvpn/easy-rsa/clean-all"
runCmd = subprocess.Popen(prefix + cmd, shell=True)
...such that you're using the results of sourcing the script in the same shell invocation as that in which you actually source the script.
Alternately (and this is what I'd do), require your Python script to be invoked by a shell which already has the necessary variables in its environment. Thus:
# ask your users to do this
set -a; . ./vars; ./yourPythonScript
...and you can error out if people don't do so very easy:
import os, sys
if not 'EASY_RSA' in os.environ:
print >>sys.stderr, "ERROR: Source vars before running this script"
sys.exit(1)

Two bash jobs executed at the same time

I am using Python to execute an external program by simply using Python's "os" library:
os.system("./script1") # script1 generates several log files and the most important for me is "info.log" !
os.system("./script2") # script2 uses the output from script1
The problem is that those scripts are in 50000 element "for loop" and "script1" needs at least 2 min to finish its job (it has fixed time duration)
In the first 1-2 sec I can find out whether I need the output data or not by looking in to the "info.log" file. However, as "script1" is an already compiled program and I can't modify it, I have to wait until it finishes.
I was thinking about a method in Bash that allows me to run two processes at the same time:
one is to start ./script1 and the other is to monitor any change in the "info.log" file...
If "info.log" has been updated or changed in size, then the second script should terminate both processes.
Something like:
os.system("./mercury6 1 > ./energy.log 2>/dev/null & sleep 2 & if [ $(stat -c %s ./info.log) -ge 1371]; then kill %1; else echo 'OK'; fi")
– which does not work...
Please, if someone knows a method, let me know!

I would suggest using subprocess, with a combination of this answer describing how to kill a bash script along with some method of monitoring the file, something like watchdog, or a simple polling loop.

import os
import time
os.system("script1")
while True: # Now this would run in 10 sec cycles and checks the output of the logs
try: # And you don't have the wait the whole 2 mins to script1 to be ready
o = open("info.log", "r") # or add os.system("script2") here
# Add here more logs
break
except:
print "waiting script1 to be ready.."
time.sleep(10) # Modify (seconds) to be sufficient to script1 to be ready
continue

You can try this one. Tell me if script1 pauses on runtime so we could try to configure it further.
#!/bin/bash
[ -n "$BASH_VERSION" ] || {
echo "You need Bash to run this script."
exit 1
}
set +o monitor ## Disable job control.
INFO_LOG='./info.log'
if [[ ! -f $INFO_LOG ]]; then
# We must create the file.
: > "$INFO_LOG" || {
echo "Unable to create $INFO_LOG."
exit 1
}
fi
read FIRSTTIMESTAMP < <(stat -c '%s' "$INFO_LOG") && [[ $FIRSTTIMESTAMP == +([[:digit:]]) ]] || {
echo "Unable to get stat of $INFO_LOG."
exit 1
}
# Run the first process with the script.
./script1 &
SCRIPT1_PID=$!
disown "$SCRIPT1_PID"
# Run the second process.
(
function kill_processes {
echo "Killing both processes."
kill "$SCRIPT1_PID"
exit 1
}
[[ -f $INFO_LOG ]] || {
echo "File has been deleted."
kill_processes
}
read NEWTIMESTAMP < <(stat -c '%s' "$INFO_LOG") && [[ $NEWTIMESTAMP == +([[:digit:]]) ]] || {
echo "Unable to get new timestamp of $INFO_LOG."
kill_processes
}
[[ NEWTIMESTAMP -ne FIRSTTIMESTAMP ]] || {
echo "$INFO_LOG has changed."
kill_processes
}
sleep 4s
) &
disown "$!"
You might also want to check some other similar solutions here: linux script with netcat stops working after x hours
(
function kill_processes {
echo "Killing both processes."
kill "$mercury6_PID"
exit 1
}
while IFS= read -r LINE; do
[[ "${LINE}" == "ejected" ]] && { ## Perhaps it should be == *ejected* ?
echo "$INFO_LOG has changed."
kill_processes
}
done < <(tail -f "$INFO_LOG")
) &
disown "$!"

It has been done ! not so elegant but it works! this is the code:
#!/bin/bash
[ -n "$BASH_VERSION" ] || {
echo "You need Bash to run this script."
exit 1
}
# set +o monitor ## Disable job control.
INFO_LOG="./info.out"
# Run the first process with the script.
./mercury6 1 > ./energy.log 2>/dev/null &
mercury6_PID=$!
disown "$mercury6_PID"
# Run the second process.
(
sleep 5
function kill_processes {
echo "Killing both processes."
kill "$mercury6_PID"
exit 1
}
while IFS= read -r LINE; do
[[ "${LINE}" == "ejected" ]] || [[ "${LINE}" == "complete." ]] && {
echo "$INFO_LOG has changed."
killall tail
kill_processes
}
done < <(tail -f "$INFO_LOG")
)
#&
#disown "$!"
If you have any recommendations you are very welcome! BTW I will try to make it with os.fork() in python as well.
Thanks again!

See here (Google is your friend). You can make the processes background tasks with the & sign, then use wait to have the parent process wait for its children.