We Keep Coding

Anacron will not run notifications from python/bash scripts

The basics
I am trying to add desktop notifications to some fairly simple scripts running with anacron just to let me know when they are running and when they have finished. For some reason, the scripts DO run, but the notifications never get sent. If I run the scripts manually (that is to say, using ./bash_test.sh instead of sudo anacron -fdn testing), the notifications send just fine.
The scripts
The bash script I am trying to run looks like this:
#!/bin/bash
python -c 'import notifications; notifications.clamstart()'
#some clamav scanning stuff happens in here, this bit runs fine
python -c 'import notifications; notifications.clamfinish()'
and the corresponding notifications.py file looks like:
from plyer import notification
def clamstart():
notification.notify(
message="Security script has started running.",
app_name="Clam Scan Daily",
hints={"desktop-entry":"clamtk"}
)
def clamfinish():
notification.notify(
message="Security script has finished running.",
app_name="Clam Scan Daily",
hints={"desktop-entry":"clamtk"}
)
Supplemental info
These two files are in the same directory, so as far as I'm aware the import statements should work fine (and they do, when I run it with ./bash_test.sh)
I have already tried using notify-send, that was what I had set up initially and it was running into the same problem, which is why I decided to try and switch to python's plyer notify() and see if that worked.
ALL of these components work fine individually, they only stop working when I try to run them using anacron with sudo anacron -fdn testing
I believe the anacrontab is set up properly since it runs except for the notifications bit, but just in case I'll add it here too:
SHELL=/bin/sh
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
# the maximal random delay added to the base delay of the jobs
RANDOM_DELAY=45
# the jobs will be started during the following hours only
START_HOURS_RANGE=3-22
#period in days delay in minutes job-identifier command
1 5 cron.daily nice run-parts /etc/cron.daily
7 25 cron.weekly nice run-parts /etc/cron.weekly
#monthly 45 cron.monthly nice run-parts /etc/cron.monthly
1 10 backup-script /home/glottophilos/backup_daily.sh
7 15 virus-scan /home/glottophilos/clamscan_daily.sh
1 10 testing /home/glottophilos/testscript.sh
I should also note that I am pretty opposed to the idea of using cron instead of anacron because this is a setup for a personal rig that is not on all the time. If there is another way of handling the scheduling though that doesn't require anacron at all, I'm happy to explore that option!
This is NOT a duplicate of Using notify-send with cron They are similar, but the answer that I posted has some structural differences and works where the other does not for reasons I'm not entirely sure of.

Solution
OK, so the solution, as per #Nick ODell's direction in the comments on the original, appears to have been doing this in the etc/anacrontab file:
SHELL=/bin/sh
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
# the maximal random delay added to the base delay of the jobs
RANDOM_DELAY=45
# the jobs will be started during the following hours only
START_HOURS_RANGE=3-22
#period in days delay in minutes job-identifier command
1 5 cron.daily nice run-parts /etc/cron.daily
7 25 cron.weekly nice run-parts /etc/cron.weekly
#monthly 45 cron.monthly nice run-parts /etc/cron.monthly
1 10 backup-script sudo -u glottophilos DISPLAY=:0 DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus /home/glottophilos/backup.sh>
7 15 virus-scan sudo -u glottophilos DISPLAY=:0 DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus /home/glottophilos/clamscan-daily.sh>
1 10 testing sudo -u glottophilos DISPLAY=:0 DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus /home/glottophilos/testscript.sh>
and then using this format in the bash script (avoiding python altogether):
notify-send --app-name="Clam Scan Daily" --hint=string:desktop-entry:clamtk "Security script is running."

subprocess.check_output(..) process returning the same output when run in loop

I am a novice in python. So please bear with me :)
I want to write a script which will check the CPU utilization of the system and in case it crosses a threshold, it should take thread dumps along with other details.
I have written a script in python as below and using "vmstat" command to see the user cycle and system cycle.
In case these 2 combined exceeds the threshold, I will take thread dumps.
while True:
output = subprocess.check_output("vmstat|tail -1", shell=True).decode('utf-8');
m = re.search(r"(\d+)\s+(\d+)\s+(\d+)\s+\d+\s+\d+$", output)
print('US', int(m.group(1)))
print('SY', int(m.group(2)))
# us: Time spent running non-kernel code. (user time, including nice time)
# sy: Time spent running kernel code. (system time)
usage=int(m.group(1))+int(m.group(2))
if usage>CPU:
...
else
print "cpu usage is below threshold"
But issue is I am getting the same values
US 1
SY 0
cpu usage is below threshold
US 1
SY 0
cpu usage is below threshold
US 1
SY 0
cpu usage is below threshold
US 1
SY 0
cpu usage is below threshold
Simultaneously if I run "vmstat" command in a different console, then I can see different values for 'US' and 'SY' parameters -
I think the subprocess.check_output(..) is running the "vmstat" process only once.
Can we somehow enable it to run every time?

Your Python is OK, except for a missing semicolon after the else.
To see what's going wrong, start by making sure that the system is going to be busy. Launch a cpu-intensive command in some other terminal window, something like:
yes | wc -c
If you don't have a spare terminal window, start it in the background like this:
yes | wc -c &
While that command is running, execute vmstat 1 5 and see that it shows that the system is using CPU.
Now run the command that your program tells Python to execute:
vmstat|tail -1
Do that a few times. That will tell you why your program always shows the same numbers. It's because vmstat with no arguments always shows the same numbers.
Well, technically that's not quite true, it doesn't always show the same values. Those values can slowly change over time. What's really happening is this (copied from the vmstat man page):
vmstat reports information about processes, memory, paging, block IO, traps, disks and cpu activity.
The first report produced gives averages since the last reboot.
If your system has been up for a while then it's going to take a lot of activity to change the averages that are shown on that first line.
To fix, change your Python program to run vmstat 1 2|tail -1. (Run that yourself a couple of times to check that it shows what you want it to show.) Obviously it will take a second to accumulate and emit the up-to-date data, but for the purposes of this program that's probably acceptable. Depending on how sensitive you want your monitoring to be you might even want to tell vmstat to sample over a longer period.
Don't forget to kill the CPU-waster that's running in the background or in some other terminal.

How to run an optimal number of instances of a single python script on linux

I have a script that's performing an independent task on about 1200 different files. It loops through each file and checks if it has already been completed or is in progress, if it hasn't been done and isn't being actively worked on (which it wouldn't be if it's not being run in parallel) then it performs a task with the file. This follows the general outline below:
myScript.py:
for file in directory:
fileStatus = getFileStatus(file)
if fileStatus != 'Complete' and fileStatus != 'inProgress':
setFileStatus(file, 'inProgress')
doTask(file)
setFileStatus(file, 'Complete')
doTask() takes 20-40 minutes on my machine and will arc from minimal RAM requirements at the beginning to about 8GB toward the middle, and back down to minimal requirements at the end. Depending on the file this will occur over a variable amount of time.
I would like to run this script in parallel with itself so that all tasks are completed in the least amount of time possible, using the maximum amount of my machine's resources. Assuming (in ignorance) the limiting resource is RAM (of which my machine has 64GB), and that the scripts will all have peak RAM consumption at the same time, I could mimic the response to this question in a manner such as:
python myScript.py &
python myScript.py &
python myScript.py &
python myScript.py &
python myScript.py &
python myScript.py &
python myScript.py &
python myScript.py &
However, I imagine I could fit more in depending on where each process is in its execution.
Is there a way to dynamically determine how many resources I have available and accordingly create, destroy or pause instances of this script so that the machine is working at maximum efficiency with respect to time? I would like to avoid making changes to myScript and instead call it from another which would handle the creating, destroying and pausing.

GNU Parallel is built for doing stuff like:
python myScript.py &
python myScript.py &
python myScript.py &
python myScript.py &
python myScript.py &
python myScript.py &
python myScript.py &
python myScript.py &
It also has some features to do resource limitation. Finding the optimal number is, however, really hard given that:
Each job runs for 20-40 minutes (if this was fixed, it would be easier)
Has a RAM usage envelope like a mountain (if it stayed at the same level all through the run, it would be easier)
If the 64 GB RAM is the limiting resource, then it is always safe to run 8 jobs:
cat filelist | parallel -j8 python myScript.py
If you have plenty of CPU power and is willing to risk wasting some, then you can run start a job if there is 8 GB memory free and if the last job was started more than 3 minutes ago (assuming jobs reach their peak memory usage within 3-5 minutes). GNU Parallel will kill the newest job and put it back on the queue, if the free memory goes below 4 GB:
cat filelist | parallel -j0 --memlimit 8G --delay 300 python myScript.py

Update:
Thanks for clarifying it further. However, with the requirements and approach you just mentioned, you are going to end up reinventing multi-threading. I suggest you avoid the multiple script calls and have all control inside your loop(s) (like the one in my original response). You are probably looking for querying the memory usage of the processes (like this).One particular component that might help you here is setting priority of the individual tasks (mentioned here).You may find this link particularly useful for scheduling priority of tasks.Infact, I recommend using threading2 package here, since it has inbuilt features on priority control.
Original Response:
Since you have roughly identified which parts require how much memory, you may employ multithreading pretty easily.
import threading
thread1 = threading.Thread(target=process1 , args=(yourArg1,)) # process1 takes 1 GB
thread2 = threading.Thread(target=process2 , args=(yourArg1,)) # process2 takes 1 GB
threadList1 = [thread1,thread2]
thread3 = threading.Thread(target=process3 , args=(yourArg1,)) # process3 takes 0.5 GB
thread4 = threading.Thread(target=process4 , args=(yourArg1,)) # process4 takes 0.5 GB
threadList2 = [thread3,thread4]
# Batch1 :
for thread in threadList1:
thread.start()
for thread in threadList1:
thread.join()
# Batch2 :
for thread in threadList2:
thread.start()
for thread in threadList2:
thread.join()

schedule automate shell script running not as ROOT

I have a shell script that I want to run automatically every day at 08 AM, and I am not authorised to use the crontab because I don't have root permission
My home directory is /home/user1/.
Any suggestions?

Ideally you should have your system administrator add your user account to /etc/cron.allow - without that you do not have permission to use the crontab command, as you probably discovered.
If that is not possible, then you could use a wrapper shell script along these lines (note: untested):
#!/bin/bash
TIME="tomorrow 8am"
while :; do
# Get the current time in seconds since the Epoch
CURRENT=$(date +%s)
# Get the next start time in seconds
NEXT=$(date +%s -d "$TIME")
# Sleep for the intervening time
sleep $((NEXT-CURRENT))
# Execute the command to repeat
/home/user1/mycommand.sh
done
You start the wrapper script in the background, or e.g. in a screen session, and as long as it's active it will regularly execute your script. Keep in mind that:
Much like cron there is no real accuracy w.r.t. the start time. If you care about timing to the second, then this is not the way to go.
If the script is interrupted for whatever reason, such as a server reboot, you will have to somehow restart. Normally I'd suggest an #reboot entry in crontab, but that seems to not be an option for you.
If there is some sort of process-cleaning mechanism that kills long-term user processed you are probably out of luck.
Your system administrator may have simply neglected to allow users access to cron - or it may have been an explicit decision. In the second case they might not take too well to you leaving a couple of processes overnight in order to bypass that restriction.

Even if you dont have root permission you can set cron job. Chcek these 2 commands as user1, if you can modify it or its throwing any error.
crontab -l
If you can see then try this as well:
crontab -e
If you can open and edit, then you can run that script with cron.
by adding this line:
* 08 * * * /path/to/your/script

I don't think root permission is required to create a cron job. Editing a cronjob that's not owned by you - there's where you'd need root.

In a pinch, you can use at(1). Make sure the program you run reschedules the at job. Warning: this goes to heck if the machine is down for any length of time.

Running a job on multiple nodes of a GridEngine cluster

I have access to a 128-core cluster on which I would like to run a parallelised job. The cluster uses Sun GridEngine and my program is written to run using Parallel Python, numpy, scipy on Python 2.5.8. Running the job on a single node (4-cores) yields an ~3.5x improvement over a single core. I would now like to take this to the next level and split the job across ~4 nodes. My qsub script looks something like this:
#!/bin/bash
# The name of the job, can be whatever makes sense to you
#$ -N jobname
# The job should be placed into the queue 'all.q'.
#$ -q all.q
# Redirect output stream to this file.
#$ -o jobname_output.dat
# Redirect error stream to this file.
#$ -e jobname_error.dat
# The batchsystem should use the current directory as working directory.
# Both files will be placed in the current
# directory. The batchsystem assumes to find the executable in this directory.
#$ -cwd
# request Bourne shell as shell for job.
#$ -S /bin/sh
# print date and time
date
# spython is the server's version of Python 2.5. Using python instead of spython causes the program to run in python 2.3
spython programname.py
# print date and time again
date
Does anyone have any idea of how to do this?

Yes, you need to include the Grid Engine option -np 16 either in your script like this:
# Use 16 processors
#$ -np 16
or on the command line when you submit the script. Or, for more permanent arrangements, use an .sge_request file.
On all the GE installations I've ever used this will give you 16 processors (or processor cores these days) on as few nodes as necessary, so if your nodes have 4 cores you'll get 4 nodes, if they have 8 2 and so on. To place the job on, say 2 cores on 8 nodes (which you might want to do if you need a lot of memory for each process) is a little more complicated and you should consult your support team.