We Keep Coding

How to embed Python code in a fish script?

I'm trying to convert over a Bash script that includes the following commands:
PYCODE=$(cat << EOF
#INSERT_PYTHON_CODE_HERE
EOF
)
RESPONSE=$(COLUMNS=999 /usr/bin/env python3 -c "$PYCODE" $#)
The idea being that a sed find/replace is then used to inject an arbitrary Python script where #INSERT_PYTHON_CODE_HERE is, creating the script that is then ran.
The corresponding Fish command would seem to be something like this
set PYCODE "
#INSERT_PYTHON_CODE_HERE
"
set RESPONSE (COLUMNS=999 /usr/bin/env python3 -c "$PYCODE" $argv)
but this falls apart when you have a Python script that can include both ' and " (and any other valid) characters.
What is the correct way to handle translate this use of EOF?
As a side note, I would prefer not to modify the sed command that is injecting the python code, but for reference here it is:
set FISH_SCRIPT (sed -e "/#INSERT_PYTHON_CODE_HERE/r $BASE_DIR/test_sh.py" $BASE_DIR/../src/mfa.fish)

RISC-V Toolchain Makefile error problem with separators and code. How could I do it correct?

Im having a problem with a Makefile Im trying to create. I just want to create one by one the .elf file then the dump and lastly the bin file and then with a python script convert it to .hex file.
My goal is to actually create the .elf and .hex files with just using the make command.
Despite that because Im very new to Makefiles, Im getting an error after executing the make command:
make
test
make: *** [copy.hex] Error 1
Thank you in advance!
XLEN ?= 32
RISCV_PREFIX ?= riscv$(XLEN)-unknown-elf-
RISCV_GCC ?= $(RISCV_PREFIX)gcc
CFLAGS = -O2
./configure --prefix=/home/opt/Risc_V/my_tests --with-arch=rv32if --with-abi=ilp32d
SCRIPTDIR=/home/opt/Risc_V/tools/
RISCV_TEST_DIR=/home/opt/Risc_V/my_tests/my_software
PROGRAMS ?= copy.c
ALL_HEX = $(PROGRAMS:%.c=%.hex)
PWD := $(shell pwd)
all: $(ALL_HEX) $(PROGRAMS)
RISCV_OPTIONS ?= -o $(PROGRAMS).elf $(CFLAGS)
RISCV_LINK ?= $(RISCV_GCC) $(PROGRAMS) $(RISCV_OPTIONS)#produces .elf file!
RISCV_OBJDUMP ?= $(RISCV_PREFIX)objdump -D $(PROGRAMS).elf > $(PROGRAMS).dump#produces a dump file to see the assembly code!
RISCV_OBJCOPY ?= $(RISCV_PREFIX)objcopy -O binary $(PROGRAMS).elf $(PROGRAMS).bin#produces a bin file!
test: all
%.elf: %.c
$(RISCV_LINK)
%.bin: %.elf
$(RISCV_OBJDUMP)
%.dump: %.elf
$(RISV_OBJCOPY)
%.hex: %.bin $(SCRIPTDIR)/bin2hex.py
test
test: $(SCRIPTDIR)/bin2hex.py
python $(SCRIPTDIR)/bin2hex.py $(PROGRAMS).bin -a 0x0 > $(PROGRAMS).hex || exit -1
clean:
rm -rf *.elf *.hex *.map *.objdump *.i *.s *.bin *.dump

How do you know you "used tabs on each rule"? It's not enough to press the TAB key on your keyboard: you have to be sure that your editor actually inserts a TAB character when you press the TAB key, and that when your editor writes out your file it preserves the TAB character in the written file (some editors will "helpfully" reset whitespace when saving a file).
However, it's quite possible that this is not the problem because, as noted above, you haven't actually shown us the full makefile and we don't know what is on line 30 of the makefile which is where the error happens. It could be some completely different syntax error. It doesn't make sense to ask a question on SO and include part of the file, but elide the actual lines where the error occurs. You say "they are mostly paths etc.", but clearly make thinks there's something wrong with them, so you have to show them to us or we can't help you.
I will say there are various other errors in your makefile (but these are not the cause of your problem you quote here):
%.elf: %.c $(ALL_HEX)
$$(RISCV_LINK)
%.bin: %.elf
$$(RISCV_OBJDUMP)
%.dump: %.elf
$$(RISV_OBJCOPY)
These should not be using $$. You want to expand these variables, they should just be $(RISCV_LINK), $(RISCV_OBJDUMP), and $(RISCV_OBJCOPY) respectively.
test: $SCRIPTDIR/bin2hex.py
You are missing the parens here: this should be $(SCRIPTDIR). What you have is equivalent to this: $(S)CRIPTDIR which is not what you want.

"gsutil rm" command using STDIN

I use gsutil in a Linux environment for managing files in GCS. I enjoy being able to use the command
gsutil -m cp -I gs://...
preceded by some other command to pass the STDIN to gsutil for uploading files; in doing so, I can maintain a local list of files that have been uploaded or generate specific patterns to upload and hand them off.
I would like to be able to do a similar command like
gsutil -m rm -I gs://...
to scrub files similarly. Presently, I build a big list of files to remove and run it with the following code:
while read line
do
gsutil rm gs://...
done < "$myfile.txt"
This is extraordinarily slow compared to the multithreaded "gsutil -m rm..." command, and enabling the -m flag has no effect when you have to process files one at a time from a list. I also experimented with just running
gsutil -m rm gs://.../* # remove everything
<my command> | gsutil -m cp -I gs://.../ # put back the pieces that I want
but this involves recopying a lot of a data and wastes a lot of time; the data is already there and just needs to have some removed. Any thoughts would be appreciated. Also, I don't have a lot of flexibility on either end with renaming files; otherwise, a quick rename before uploading would handle all of this.

As an interim solution, since we don't have a -I option for rm right now, how about just creating a string of all the objects you want to delete in your loop and then using gsutil -m rm to delete it? You could also do this with a simple python script that invokes the gsutil command from within python as a separate process.
Expanding on your earlier example, maybe something like the following (disclaimer: my bash-fu isn't the greatest, and I haven't tested this):
objects=''
while read line
do
objects="$objects gs://$line"
done
gsutil -m rm $objects

For anyone wondering, I wound up doing like Zach Wilt indicated above. For reference, I was removing on the order of a couple thousand files from a span of 5 directories, so roughly 10,000 files. Doing this without the "-m" switch was taking upwards of 30 minutes; with the "-m" switch, it takes less than 30 seconds. Zoom!
For a robust example: I am using this to update Google Cloud Storage files to match local files. On the current day, I have a program that dumps lots of files that are incremental, and also a handful that are "rolled up". After a week, the incremental files get scrubbed locally automatically, but the same should happen in GCS to save the space. Here's how to do this:
#!/bin/bash
# get the full date strings for touch
start=`date --date='-9 days' +%x`
end=`date --date='-8 days' +%x`
# other vars
mon=`date --date='-9 days' +%b | tr [A-Z] [a-z]`
day=`date --date='-9 days' +%d`
# display start and finish times
echo "Cleaning files from $start"
# update start and finish times
touch --date="$start" /tmp/start1
touch --date="$end" /tmp/end1
# repeat for all servers
for dr in "dir1" "dir2" "dir3" ...
do
# list files in range and build retention file
find /local/path/$dr/ -newer /tmp/start1 ! -newer /tmp/end1 > "$dr-local.txt"
# get list of all files from appropriate folder on GCS
gsutil ls gs://gcs_path/$mon/$dr/$day/ > "$dr-gcs.txt"
# formatting the host list file
sed -i "s|gs://gcs_path/$mon/$dr/$day/|/local/path/$dr/|" "$dr-gcs.txt"
# build sed command file to delete matches
while read line
do
echo "\|$line|d" >> "$dr-del.txt"
done < "$dr-local.txt"
# run command file to strip lines for files that need to remain
sed -f "$dr-del.txt" <"$dr-gcs.txt" >"$dr-out.txt"
# convert local names to GCS names
sed -i "s|/local/path/$dr/|gs://gcs_path/$mon/$dr/$day/|" "$dr-out.txt"
# new variable to hold string
del=""
# convert newline separated file to one long string
while read line
do
del="$del$line "
done < "$dr-out.txt"
# remove all files matching the final output
gsutil -m rm $del
# cleanup files
rm $dr-local.txt
rm $dr-gcs.txt
rm $dr-del.txt
rm $dr-out.txt
done
You'll need to modify to fit your needs, but this is a concrete and working method for deleting files locally, and then synchronizing the change to Google Cloud Storage. Obviously, modify to fit your needs. Thanks again to #Zach Wilt.

write to file with tee command in python

I have a python script with variable names e.g V and It. I make a filename based on the parameter as following:
file_out=io.open("file_Iter" + str(It) + "_V_" +str(V)+".txt", 'w')
Then I'd like to redirect all my terminal output to a this file, so I use this command:
os.system("echo - START RUN $(LANG=en_US date +%b_%d_%Y_%k_%M)- | tee -a $file_out")
The file_out is created and also the echo command is shown correctly on the terminal but it is not written to file_out. If in tee command, I put e.g tee testfile.txt, then this file is created and also echo command writes in it.
Q: How should I change tee in order to write to the file that is created by variable names?

I'm not sure I understood it correctly, but I guess that what you want to do is the following:
fileName = "file_Iter" + str(It) + "_V_" +str(V)+".txt"
file_out=io.open(fileName, 'w')
os.system("echo - START RUN $(LANG=en_US date +%b_%d_%Y_%k_%M)- | tee -a " + fileName)
Pay attention to the end of the command, where fileName is concatenated.

$file_out refers to a shell variable -- a variable in the context of the shell that you're executing this command. However, file_out is a Python variable in the context of your Python script.
So, we need to pass this from Python into the shell. Luckily, Python provides an easy way to do this with string formatting commands. This will do what you want, I think:
os.system("echo - START RUN $(LANG=en_US date +%b_%d_%Y_%k_%M)- | " +
"tee -a {}".format(file_out))
However, there are a couple concerns with this. First, you should be using subprocess.Popen instead of os.system -- it's a newer module that's intended to replace os.system.
Also, you should be aware that, depending on the content of V and It, it may be possible for this shell to run unexpected commands. Imagine that V has the value '.txt ; sudo rm -rf / #'. Now your file name is 'file_IterIt_V_.txt ; sudo rm -rf / #.txt', and your shell command is 'echo - START RUN $(LANG=en_US date +%b_%d_%Y_%k_%M)- | tee -a file_IterIt_V_.txt ; sudo rm -rf / #.txt', which will happily remove many files from your computer. If you're going to do this, you should make sure to escape the filename in your shell command.

Rejecting files with Windows line endings using Perforce triggers

Using Perforce, I'd like to be able to reject submits which contain files with Windows line endings (\r\n IIRC, maybe just \r anywhere as really we only want files with Unix line endings).
Rather than dos2unix incoming files or similar, to help track down instances where users attempt to submit files with Windows line endings, I'd like to add a trigger to reject text submissions which contain files with non-Unix line endings.
Could someone demonstrate how the trigger itself could be written, perhaps with bash or python?
Thanks

Here's the minimal edit I can thing of for the bash example found in the p4 docs:
#!/bin/sh
# Set target string, files to search, location of p4 executable...
TARGET='\r\n'
DEPOT_PATH="//depot/src/..."
CHANGE=$1
P4CMD="/usr/local/bin/p4 -p 1666 -c copychecker"
XIT=0
echo ""
# For each file, strip off #version and other non-filename info
# Use sed to swap spaces w/"%" to obtain single arguments for "for"
for FILE in `$P4CMD files $DEPOT_PATH#=$CHANGE | \
sed -e 's/\(.*\)\#[0-9]* - .*$/\1/' -e 's/ /%/g'`
do
# Undo the replacement to obtain filename...
FILE="`echo $FILE | sed -e 's/%/ /g'`"
# ...and use #= specifier to access file contents:
# p4 print -q //depot/src/file.c#=12345
if $P4CMD print -q "$FILE#=$CHANGE" | fgrep "$TARGET" > /dev/null
then
echo "Submit fails: '$TARGET' not found in $FILE"
XIT=1
else
echo ""
fi
done
exit $XIT
The original example fails if the target is missing, this one fails if it's present -- just switching the then and else branches of the if. You could edit it further of course (e.g. giving grep, or fgrep, the -q flag to suppress output, if your grep supports it as e.g. GNU's does).