Running this on osx...
cd ${BUILD_DIR}/mydir && for DIR in $(find ./ '.*[^_].py' | sed 's/\/\//\//g' | awk -F "/" '{print $2}' | sort |uniq | grep -v .py); do
if [ -f $i/requirements.txt ]; then
pip install -r $i/requirements.txt -t $i/
fi
cd ${DIR} && zip -r ${DIR}.zip * > /dev/null && mv ${DIR}.zip ../../ && cd ../
done
cd ../
error:
(env) ➜ sh package_lambdas.sh find: .*[^_].py: No such file or directory
why?
find takes as an argument a list of directories to search. You provided what appears to be regular expression. Because there is no directory named (literally) .*[^_].py, find returns an error.
Below I have revised your script to correct that mistake (if I understand your intention). Because I see so many ill-written shell scripts these days, I've taken the liberty of "traditionalizing" it. Please see if you don't also find it more readable.
Changes:
use #!/bin/sh, guaranteed to be on an Unix-like system. Faster than bash, unless (like OS X) it is bash.
use lower case for variable names to distinguish from system variables (and not hide them).
eschew braces for variables (${var}); they're not needed in the simple case
do not pipe output to /usr/bin/true; route it to dev/null if that's what you mean
rm -f by definition cannot fail; if you meant || true, it's superfluous
put then and do on separate lines, easier to read, and that's how the Bourne shell language was meant to be used
Let && and || serve as line-continuation, so you can see what's happening step by step
Other changes I would suggest:
Use a subshell when changing the working directory temporarily. When it terminates, the working directory is restored automatically (retained by the parent), saving you the cd .. step, and errors.
Use set -e to cause the script to terminate on error. For expected errors, use || true explicitly.
Change grep .py to grep '\.py$', just for good measure.
To avoid Tilting Matchstick Syndrome, use something other than / as a sed substitute delimiter, e.g., sed 's://:/:g'. But sed could be avoided altogether with awk -F '/+' '{print $2}'.
Revised version:
#! /bin/sh
src_dir=lambdas
build_dir=bin
mkdir -p $build_dir/lambdas
rm -rf $build_dir/*.zip
cp -r $src_dir/* $build_dir/lambdas
#
# The sed is a bit complicated to be osx / linux cross compatible :
# ( .//run.sh vs ./run.sh
#
cd $build_dir/lambdas &&
for L in $(find . -exec grep -l '.*[^_].py' {} + |
sed 's/\/\//\//g' |
awk -F "/" '{print $2}' |
sort |
uniq |
grep -v .py)
do
if [ -f $i/requirements.txt ]
then
echo "Installing requirements"
pip install -r $i/requirements.txt -t $i/
fi
cd $L &&
zip -r $L.zip * > /dev/null &&
mv $L.zip ../../ &&
cd ../
done
cd ../
The find(1) manpage says its args are [path ...] [expression], where "expression" consists of "primaries" and "operands" (-flags). '.*[^-].py' doesn't look like any expression, so it's being interpreted as a path, and it's reporting that there is no file named '.*[^-].py' in the working directory.
Perhaps you meant:
find ./ -regex '.*[^-].py'
Related
I am running Ubuntu 22.04 with xorg.
I need to find a way to compile microbit python code locally to a firmware hex file. Firstly, I followed the guide here https://microbit-micropython.readthedocs.io/en/latest/devguide/flashfirmware.html.
After a lot of debugging, I got to this point: https://pastebin.com/MGShD31N
However, the file platform.h does exist.
sawntoe#uwubuntu:~/Documents/Assignments/2022/TVP/micropython$ ls /home/sawntoe/Documents/Assignments/2022/TVP/micropython/yotta_modules/mbed-classic/api/platform.h
/home/sawntoe/Documents/Assignments/2022/TVP/micropython/yotta_modules/mbed-classic/api/platform.h
sawntoe#uwubuntu:~/Documents/Assignments/2022/TVP/micropython$
At this point, I gave up on this and tried using Mu editor with the AppImage. However, Mu requires wayland, and I am on xorg.
Does anyone have any idea if this is possible? Thanks.
Mu and the uflash command are able to retrieve your Python code from .hex files. Using uflash you can do the following for example:
uflash my_script.py
I think that you want is somehow possible to do, but its harder than just using their web python editor: https://python.microbit.org/v/2
Peter Till answers the original question. The additional below adds to this answer by showing how to automate the build and load process. I use Debian. The original question states that Ubuntu is used, which is built on Debian.
A script to find and mount the micro:bit
When code is loaded to the micro:bit, the board is dismounted from the system. So each time you have new code to load, you have to remount the board.
I modified a script to find and mount the micro:bit.
#!/bin/bash
BASEPATH="/media/$(whoami)/"
MICRO="MICROBIT"
if [ $# -eq 0 ]
then
echo "no argument supplied, use 'mount' or 'unmount'"
exit 1
fi
if [ $1 == "--help" ]
then
echo "mounts or unmounts a BBC micro:bit"
echo "args: mount - mount the microbit, unmout - unmount the microbit"
fi
# how many MICRO found in udiksctl dump
RESULTS=$(udisksctl dump | grep IdLabel | grep -c -i $MICRO)
case "$RESULTS" in
0 ) echo "no $MICRO found in 'udkisksctl dump'"
exit 0
;;
1 ) DEVICELABEL=$(udisksctl dump | grep IdLabel | grep -i $MICRO | cut -d ":" -f 2 | sed 's/^[ \t]*//')
DEVICE=$(udisksctl dump | grep -i "IdLabel: \+$DEVICELABEL" -B 12 | grep " Device:" | cut -d ":" -f 2 | sed 's/^[ \t]*//')
DEVICEPATH="$BASEPATH""$DEVICELABEL"
echo "found one $MICRO, device: $DEVICE"
if [[ -z $(mount | grep "$DEVICE") ]]
then
echo "$DEVICELABEL was unmounted"
if [ $1 == "mount" ]
then
udisksctl mount -b "$DEVICE"
exit 0
fi
else
echo "$DEVICELABEL was mounted"
if [ $1 == "unmount" ]
then
udisksctl unmount -b "$DEVICE"
exit 0
fi
fi
;;
* ) echo "more than one $MICRO found"
;;
esac
echo "exiting without doing anything"
I alias this script to mm in my .bashrc file.
Automate mounting the micro:bit and flashing the python file
I use the inotifywait command to run mm and to then run uflash to load the .py file I am working on. Each time that the python file is saved, the aliased command mm is run followed by the uflash command.
while inotifywait -e modify <your_file>.py ; do mm && uflash <your_file>.py ; done
Okay, so elaborating on Peter Till's answer.
Firstly, you can use uflash:
uflash path/to/your/code .
Or, you can use microfs:
ufs put path/to/main.py
Working Ubuntu 22.04 host CLI setup with Carlos Atencio's Docker to build your own firmware
After trying to setup the toolchain for a while, I finally decided to Google for a Docker image with the toolchain, and found https://github.com/carlosperate/docker-microbit-toolchain at this commit from Carlos Atencio, a Micro:Bit foundation employee, and that just absolutely worked:
# Get examples.
git clone https://github.com/bbcmicrobit/micropython
cd micropython
git checkout 7fc33d13b31a915cbe90dc5d515c6337b5fa1660
# Get Docker image.
docker pull ghcr.io/carlosperate/microbit-toolchain:latest
# Build setup to be run once.
docker run -v $(pwd):/home --rm ghcr.io/carlosperate/microbit-toolchain:latest yt target bbc-microbit-classic-gcc-nosd#https://github.com/lancaster-university/yotta-target-bbc-microbit-classic-gcc-nosd
docker run -v $(pwd):/home --rm ghcr.io/carlosperate/microbit-toolchain:latest make all
# Build one example.
docker run -v $(pwd):/home --rm ghcr.io/carlosperate/microbit-toolchain:latest \
tools/makecombinedhex.py build/firmware.hex examples/counter.py -o build/counter.hex
# Build all examples.
docker run -v $(pwd):/home --rm ghcr.io/carlosperate/microbit-toolchain:latest \
bash -c 'for f in examples/*; do b="$(basename "$f")"; echo $b; tools/makecombinedhex.py build/firmware.hex "$f" -o "build/${b%.py}.hex"; done'
And you can then flash the example you want to run with:
cp build/counter.hex "/media/$USER/MICROBIT/"
Some further comments at: Generating micropython + python code `.hex` file from the command line for the BBC micro:bit
I am trying to run xargs on multiple files at once:
sh -c 'find . -name "*.py" | xargs pylint'
This will give me a single pylint score for all py files in a repo. However when I try to modify it to do both black and pylint, it loops through each file individually and gives me the pylint score and black diff on each file:
find . -name "*.py" | xargs -I % sh -c 'pylint %; black --check --diff %;'
Any way to pass in the py files in batch rather than each individually?
If you use -I option it runs given command once for each entry in input.
You can do this instead:
find . -name "*.py" |
xargs sh -c 'pylint "$#"; black --check --diff "$#"'
Is there a way to create a Makefile/script that will fail if a file from coverage.py library has values below a certain threshold? Say 80%.
For coverage.py you can use fail_under option
First, gather test coverage by using the coverage run command (here), then run the coverage report command
Option 1.
Run via coverage report command directly
coverage report --fail-under=80
Option 2.
Use configuration file by defining value in the report section of .coveragerc
[report]
fail_under = 80
Then run the coverage report command
coverage report
It will return non zero exit code if the coverage is below the value of fail_under
If you want something more fine-grained than an overall percentage threshold, you can try this coverage goals program I threw together: https://nedbatchelder.com/blog/202111/coverage_goals.html
I figured it out with the following. Feel free to copy if it fits your needs!
The Makefile goes as follows:
test: $(TEST_DIRECTORY)/*/*
rm -f $(TEST_OUTPUT_DIRECTORY)/coverage_tests.log
cd $(TEST_DIRECTORY)/inference && coverage run -m unittest discover
for file in $^ ; do \
TEST_FILE=$$(echo $${file} | grep -E -o test_.*) ; \
FILE=$$(echo $${TEST_FILE} | sed 's/test_//') ; \
if [[ ! -z $${FILE} ]] ; then \
cd $(TEST_DIRECTORY)/inference && coverage report --include $(INFERENCE_DIRECTORY)/$${FILE} > $(TEST_OUTPUT_DIRECTORY)/coverage_temp.log ; \
cd $(TEST_DIRECTORY) && ./coverage_check.sh $${FILE} < $(TEST_OUTPUT_DIRECTORY)/coverage_temp.log >> $(TEST_OUTPUT_DIRECTORY)/coverage_tests.log ; \
rm -f $(TEST_OUTPUT_DIRECTORY)/coverage_temp.log ; \
fi ; \
done
rm -f $(TEST_DIRECTORY)/inference/.coverage
and it utilizes a coverage check script which is here:
#!/bin/sh
while read line; do
for word in $line; do
if [[ "$word" == *"%"* ]]; then
COVERAGE=`echo $word | sed 's/%//g'`
if [ $COVERAGE -gt 80 ]
then
echo "$1: PASSED $word"
break 2
else
echo "$1: FAILED $word"
break 2
fi
fi
done
done
Could you please show me how to implement git hook?
Before committing, the hook should run a python script. Something like this:
cd c:\my_framework & run_tests.py --project Proxy-Tests\Aeries \
--client Aeries --suite <Commit_file_Name> --dryrun
If the dry run fails then commit should be stopped.
You need to tell us in what way the dry run will fail. Will there be an output .txt with errors? Will there be an error displayed on terminal?
In any case you must name the pre-commit script as pre-commit and save it in .git/hooks/ directory.
Since your dry run script seems to be in a different path than the pre-commit script, here's an example that finds and runs your script.
I assume from the backslash in your path that you are on a windows machine and I also assume that your dry-run script is contained in the same project where you have git installed and in a folder called tools (of course you can change this to your actual folder).
#!/bin/sh
#Path of your python script
FILE_PATH=tools/run_tests.py/
#Get relative path of the root directory of the project
rdir=`git rev-parse --git-dir`
rel_path="$(dirname "$rdir")"
#Cd to that path and run the file.
cd $rel_path/$FILE_PATH
echo "Running dryrun script..."
python run_tests.py
#From that point on you need to handle the dry run error/s.
#For demonstrating purproses I'll asume that an output.txt file that holds
#the result is produced.
#Extract the result from the output file
final_res="tac output | grep -m 1 . | grep 'error'"
echo -e "--------Dry run result---------\n"${final_res}
#If a warning and/or error exists abort the commit
eval "$final_res" | while read -r line; do
if [ $line != "0" ]; then
echo -e "Dry run failed.\nAborting commit..."
exit 1
fi
done
Now every time you fire git commit -m the pre-commit script will run the dry run file and abort the commit if any errors have occured, keeping your files in the stagin area.
I have implemented this in my hook. Here is the code snippet.
#!/bin/sh
#Path of your python script
RUN_TESTS="run_tests.py"
FRAMEWORK_DIR="/my-framework/"
CUR_DIR=`echo ${PWD##*/}`
`$`#Get full path of the root directory of the project under RUN_TESTS_PY_FILE
rDIR=`git rev-parse --git-dir --show-toplevel | head -2 | tail -1`
OneStepBack=/../
CD_FRAMEWORK_DIR="$rDIR$OneStepBack$FRAMEWORK_DIR"
#Find list of modified files - to be committed
LIST_OF_FILES=`git status --porcelain | awk -F" " '{print $2}' | grep ".txt" `
for FILE in $LIST_OF_FILES; do
cd $CD_FRAMEWORK_DIR
python $RUN_TESTS --dryrun --project $CUR_DIR/$FILE
OUT=$?
if [ $OUT -eq 0 ];then
continue
else
return 1
fi
done
I have been following this tutorial to install opencv and python:
https://www.pyimagesearch.com/2015/06/22/install-opencv-3-0-and-python-2-7-on-ubuntu/#comment-441393
The only difference is that I am trying to install opencv 3.3.1 instead of 3.0.0
I'm running on a laptop with Ubuntu 14.04 an i7 and NVIDIA GTX950M
The problem is that when I execute the command ldconfig
$ sudo make install
$ sudo ldconfig
I get the following message:
/sbin/ldconfig.real: /usr/lib/nvidia-384/libEGL.so.1 is not a symbolic link
/sbin/ldconfig.real: /usr/lib32/nvidia-384/libEGL.so.1 is not a symbolic link
So I found a solution to the problem:
Source: https://askubuntu.com/questions/900285/libegl-so-1-is-not-a-symbolic-link #muru #Gerard Tromp
The following is an easy-to-use version of Noisy_Botnet's solution. It
facilitates repeating the process for any update.
Create a shell script, i.e., paste the code bellow a text file, and save it with the .sh extension.
Change the executing permissions of the file i.e., go to the location of the file in the terminal and execute the following command $sudo chmod 744 nameofthefieleyoucreated.sh
execute the following command $sudo ./nameofthefileyoucreated.sh
#! /bin/sh
#
# find the file in /usr/lib
LIBEGL=`find /usr/lib/nvidia* -name libEGL.so.\* | egrep "[0-9][0-9]*\.[0-9][0-9]*$"`
LIBEGL_LINK=`echo $LIBEGL | sed 's/[0-9][0-9]*\.[0-9][0-9]*$/1/'`
printf "\n\nThe following commands will be executed:\n+++++++++++++++++++++++++++++++++++++++\n"
printf "mv $LIBEGL_LINK ${LIBEGL_LINK}.orig\nln -s $LIBEGL $LIBEGL_LINK\n\n"
while true; do
read -p "Do you wish to perform these commands? " yn
case $yn in
[Yy]* ) mv $LIBEGL_LINK ${LIBEGL_LINK}.orig; ln -s $LIBEGL $LIBEGL_LINK ; break;;
[Nn]* ) break;;
* ) echo "Please answer yes or no.";;
esac
done
# find the file in /usr/lib32
LIBEGL=`find /usr/lib32/nvidia* -name libEGL.so.\* | egrep "[0-9][0-9]*\.[0-9][0-9]*$"`
LIBEGL_LINK=`echo $LIBEGL | sed 's/[0-9][0-9]*\.[0-9][0-9]*$/1/'`
printf "\n\nThe following commands will be executed:\n+++++++++++++++++++++++++++++++++++++++\n"
printf "mv $LIBEGL_LINK ${LIBEGL_LINK}.orig\nln -s $LIBEGL $LIBEGL_LINK\n\n"
while true; do
read -p "Do you wish to perform these commands? " yn
case $yn in
[Yy]* ) mv $LIBEGL_LINK ${LIBEGL_LINK}.orig; ln -s $LIBEGL $LIBEGL_LINK ; break;;
[Nn]* ) break;;
* ) echo "Please answer yes or no.";;
esac
done