I have a python script that I am executing with cron job. This script generates some output while executing and I wish to post it to Slack channel in real time.
Here is a bash script that I have:
#!/bin/bash
log_file=logs_$(date '+\%Y-\%m-\%d_\%H:\%M').txt
cd /path/to/script/run.py > /path/to/logs/${log_file} 2>&1
cat /path/to/logs/${log_file} | while read LINE; do
(echo "$LINE" | grep -e "Message" ) && curl -X POST --silent --data-urlencode \
"payload={\"text\": \"$(echo $LINE | sed "s/\"/'/g")\"}" "https://hooks.slack.com/services/xxxxxx";
done
This scrip works but it of course posts all messages to slack once the python script has already been executed. Is there any way I could configure it so that messages would be sent to Slack in real time while the python script is still being executed?
You may be able to read the output from your run.py script via process substitution:
#!/bin/bash
log_file=logs_$(date '+\%Y-\%m-\%d_\%H:\%M').txt
while read -r line ; do
echo "$line"
(echo "$line" | grep -e "Message" ) && curl -X POST --silent --data-urlencode \
"payload={\"text\": \"$(echo $line | sed "s/\"/'/g")\"}" "https://hooks.slack.com/services/xxxxxx";
done < <(/path/to/script/run.py 2>&1) >> "$log_file"
It may also prove useful to paste your code into shellcheck.net and have a look at the suggested changes.
Your script shouldn't work at all as you're not executing run.py but you're changing your working directory into it, so unless run.py is a directory, your script should fail.
Also, commands in bash scripting are executed sequentially, so if you launch your python command and then read the log, no wonder that you're not getting log lines in real time.
What i would do is using some pipes and xargs:
#!/bin/bash
/path/to/script/run.py | grep -e "Message" | sed "s/\"/'/g" | xargs -I{} curl -L -X POST --silent --data-urlencode 'payload={"text":"{}"}' https://hooks.slack.com/services/xxx
I've added -L to the curl command because hooks.slack.com makes a redirect to api.slack.com and without that flag curl will stop after the 302 instead of following the Location header in the response.
Related
Current Situation
I created a php script, to start the python script.
Following is the script:
$python_file = "/var/www/web/test.py 2>&1 | tee -a /tmp/mylog 2>/dev/null >/dev/null &";
$command = "nohup python3 ".$python_file;
exec($command);
Problem:
After triggering the php script, the script keeps on running and finally it returns 504 error page.
Expected Solution
After triggering the above script, it needs to return immediately after the exec statement. is it possible?
add & to run in the background
$python_file = "/var/www/web/test.py 2>&1 | tee -a /tmp/mylog 2>/dev/null >/dev/null &";
$command = "nohup python3 ".$python_file . " &";
exec($command);
I am running Ubuntu 22.04 with xorg.
I need to find a way to compile microbit python code locally to a firmware hex file. Firstly, I followed the guide here https://microbit-micropython.readthedocs.io/en/latest/devguide/flashfirmware.html.
After a lot of debugging, I got to this point: https://pastebin.com/MGShD31N
However, the file platform.h does exist.
sawntoe#uwubuntu:~/Documents/Assignments/2022/TVP/micropython$ ls /home/sawntoe/Documents/Assignments/2022/TVP/micropython/yotta_modules/mbed-classic/api/platform.h
/home/sawntoe/Documents/Assignments/2022/TVP/micropython/yotta_modules/mbed-classic/api/platform.h
sawntoe#uwubuntu:~/Documents/Assignments/2022/TVP/micropython$
At this point, I gave up on this and tried using Mu editor with the AppImage. However, Mu requires wayland, and I am on xorg.
Does anyone have any idea if this is possible? Thanks.
Mu and the uflash command are able to retrieve your Python code from .hex files. Using uflash you can do the following for example:
uflash my_script.py
I think that you want is somehow possible to do, but its harder than just using their web python editor: https://python.microbit.org/v/2
Peter Till answers the original question. The additional below adds to this answer by showing how to automate the build and load process. I use Debian. The original question states that Ubuntu is used, which is built on Debian.
A script to find and mount the micro:bit
When code is loaded to the micro:bit, the board is dismounted from the system. So each time you have new code to load, you have to remount the board.
I modified a script to find and mount the micro:bit.
#!/bin/bash
BASEPATH="/media/$(whoami)/"
MICRO="MICROBIT"
if [ $# -eq 0 ]
then
echo "no argument supplied, use 'mount' or 'unmount'"
exit 1
fi
if [ $1 == "--help" ]
then
echo "mounts or unmounts a BBC micro:bit"
echo "args: mount - mount the microbit, unmout - unmount the microbit"
fi
# how many MICRO found in udiksctl dump
RESULTS=$(udisksctl dump | grep IdLabel | grep -c -i $MICRO)
case "$RESULTS" in
0 ) echo "no $MICRO found in 'udkisksctl dump'"
exit 0
;;
1 ) DEVICELABEL=$(udisksctl dump | grep IdLabel | grep -i $MICRO | cut -d ":" -f 2 | sed 's/^[ \t]*//')
DEVICE=$(udisksctl dump | grep -i "IdLabel: \+$DEVICELABEL" -B 12 | grep " Device:" | cut -d ":" -f 2 | sed 's/^[ \t]*//')
DEVICEPATH="$BASEPATH""$DEVICELABEL"
echo "found one $MICRO, device: $DEVICE"
if [[ -z $(mount | grep "$DEVICE") ]]
then
echo "$DEVICELABEL was unmounted"
if [ $1 == "mount" ]
then
udisksctl mount -b "$DEVICE"
exit 0
fi
else
echo "$DEVICELABEL was mounted"
if [ $1 == "unmount" ]
then
udisksctl unmount -b "$DEVICE"
exit 0
fi
fi
;;
* ) echo "more than one $MICRO found"
;;
esac
echo "exiting without doing anything"
I alias this script to mm in my .bashrc file.
Automate mounting the micro:bit and flashing the python file
I use the inotifywait command to run mm and to then run uflash to load the .py file I am working on. Each time that the python file is saved, the aliased command mm is run followed by the uflash command.
while inotifywait -e modify <your_file>.py ; do mm && uflash <your_file>.py ; done
Okay, so elaborating on Peter Till's answer.
Firstly, you can use uflash:
uflash path/to/your/code .
Or, you can use microfs:
ufs put path/to/main.py
Working Ubuntu 22.04 host CLI setup with Carlos Atencio's Docker to build your own firmware
After trying to setup the toolchain for a while, I finally decided to Google for a Docker image with the toolchain, and found https://github.com/carlosperate/docker-microbit-toolchain at this commit from Carlos Atencio, a Micro:Bit foundation employee, and that just absolutely worked:
# Get examples.
git clone https://github.com/bbcmicrobit/micropython
cd micropython
git checkout 7fc33d13b31a915cbe90dc5d515c6337b5fa1660
# Get Docker image.
docker pull ghcr.io/carlosperate/microbit-toolchain:latest
# Build setup to be run once.
docker run -v $(pwd):/home --rm ghcr.io/carlosperate/microbit-toolchain:latest yt target bbc-microbit-classic-gcc-nosd#https://github.com/lancaster-university/yotta-target-bbc-microbit-classic-gcc-nosd
docker run -v $(pwd):/home --rm ghcr.io/carlosperate/microbit-toolchain:latest make all
# Build one example.
docker run -v $(pwd):/home --rm ghcr.io/carlosperate/microbit-toolchain:latest \
tools/makecombinedhex.py build/firmware.hex examples/counter.py -o build/counter.hex
# Build all examples.
docker run -v $(pwd):/home --rm ghcr.io/carlosperate/microbit-toolchain:latest \
bash -c 'for f in examples/*; do b="$(basename "$f")"; echo $b; tools/makecombinedhex.py build/firmware.hex "$f" -o "build/${b%.py}.hex"; done'
And you can then flash the example you want to run with:
cp build/counter.hex "/media/$USER/MICROBIT/"
Some further comments at: Generating micropython + python code `.hex` file from the command line for the BBC micro:bit
I have a python script that calls a bash script, which calls another bash script that hangs only when called from python.
test.py
#!/usr/bin/python
import subprocess
print("Python")
subprocess.call(["/home/user/test/bash1.sh"])
bash1.sh
#!/bin/bash
echo "Bash 1"
var=$(echo "Bash 1 var")
echo $var
/home/user/test/bash2.sh
bash2.sh
#!/bin/bash
echo "Bash 2"
var=$(echo "Bash 2 var")
echo $var
randomkey=$(cat /dev/urandom | tr -dc 'a-z' | fold -w 8 | head -n 1)
echo $randomkey
When I run ./bash1.sh everything works just fine. When I run test.py bash2.sh hangs at:
randomkey=$(cat /dev/urandom | tr -dc 'a-z' | fold -w 8 | head -n 1)
I have a stinking feeling the pipes (|) aren't reaching their destinations. Any ideas how to make this work from test.py?
EDIT: Ubuntu VM with Python2.7
I solved this by upgrading to Python3.7 from 2.7
I've got the following bash script to parse endpoints JSON:
echo curl -s -H "X-Auth-Token: my_access_token" -X GET "https://api.selvpc.ru/identity/v3/endpoints?interface=public" | python -mjson.tool | grep -Pi '^\s*"url":\s*".*",?$' | awk '{print $2}' | tr -d '"' | sed "s/[%\\\$](tenant_id)s/my_project_id/g")
But bash says:
-bash: syntax error near unexpected token `)'
My hoster says, this script works well on Linux-based OS, but no guarantee to work on OS X. What can be the syntax issue?
EDIT:
If i use the following:
curl -s -H "X-Auth-Token: my_access_token" -X GET "https://api.selvpc.ru/identity/v3/endpoints?interface=public" | python -mjson.tool
JSON parses as expected. But with grep -Pi '^\s*"url":\s*".*",?$' I guess it causes grep warning
usage: grep [-abcDEFGHhIiJLlmnOoqRSsUVvwxZ] [-A num] [-B num] [-C[num]]
[-e pattern] [-f file] [--binary-files=value] [--color=when]
[--context[=num]] [--directories=action] [--label] [--line-buffered]
[--null] [pattern] [file ...]
I guess the first problem is grep error?
As #4ae1e1 suggested, please use a JSON processor for the job. jq is great and it's worthwhile investing your time to learn it.
wget https://github.com/stedolan/jq/releases/download/jq-1.5/jq-osx-amd64
mv jq-osx-amd64 jq
chmod u+x jq
curl -s -H "X-Auth-Token: $TOKEN" https://api.selvpc.ru/identity/v3/endpoints?interface=public | \
./jq -r .endpoints[].url
That will get you a list of OpenStack API endpoints.
I think a python script using python-keystoneclient can be easier to understand and maintain
I've got a python program which runs via crontab and that works perfectly. However, I decided to add the ability to notify me of what it's doing, and suddenly it's failing. It runs from the command line, however, running it as a crontab program causes it to fail
libnotify-Message: Unable to get session bus: /bin/dbus-launch terminated abnormally with the following error: Autolaunch error: X11 initialization failed.
What am I doing wrong?
Edit
I would like this program to still run from cron and be able to take advantage of notifying the user of it's work. Is there any way to do this?
Edit 2
I've tried using root's crontab and sudo -u esr python script.py yet this also fails, silently at that.
Edit 3
It is possible! Here's the code.
* * * * * su $user -c "DBUS_SESSION_BUS_ADDRESS=$(grep -z DBUS_SESSION_BUS_ADDRESS /proc/$(ps -au esr | grep -i "gnome-session" | awk '{ print $1 }')/environ | sed -e 's/DBUS_SESSION_BUS_ADDRESS=//') $(whereis notify-send | awk '{ print $2 }') -u normal -t 20000 \"Hello\" "
* * * * * su esr -c "DBUS_SESSION_BUS_ADDRESS=$(grep -z DBUS_SESSION_BUS_ADDRESS /proc/$(ps -au esr | grep -i "gnome-session" | awk '{ print $1 }')/environ | sed -e 's/DBUS_SESSION_BUS_ADDRESS=//') $(whereis notify-send | awk '{ print $2 }') -u normal -t 20000 \"Hello\" "
As per a suggestion, an explanation, unfortunately not mine
You're trying to run a script that requires user resources in an environment where said resources are not available. You will have to strip the script of all references to PyGTK and to the session bus if you want this to work.
I just wanted to mention that the following recipe works for users of the awesome window manager:
*/1 * * * * DBUS_SESSION_BUS_ADDRESS=$(grep -zi DBUS /proc/$(pgrep awesome)/environ | sed -r -e 's/^DBUS_SESSION_BUS_ADDRESS=//') DISPLAY=":0.0" notify-send -t 0 blah blah
You're trying to use GUI (GTK+ library calls) for cron program that has no access to graphical terminal. You need to avoid creating dialogs and windows when you run it from cron.