This is my use case: I have a python script that returns the name of a remote server and a file path from a config file. I need a way to use those two or more parameterized variables and their values as input for my shell script which will then sync files by connecting to the remote server. Any tips appreciated.
See if this is what you were looking for. I have a python script that prints out 3 lines, each a different value. For you, a[0] would be the server, a[1] the first file, and a[2] the second. The number of lines is arbitrary and this would work for any number, one per line, and allow spaces in the file names.
The "<(" executes what is inside and creates something like a pipe, which the readarray command reads (it takes standard input, namely "<")
> readarray a < <(python -c 'print "myserver:8080"; print "file1 which may have spaces"; print "another file"')
> echo ${a[0]}
myserver:8080
> echo ${a[1]}
file1 which may have spaces
> echo ${a[2]}
another file
Related
I've bash script doing some tasks but I need to manipulate on string obtained from configuration (for simplification in this test it's hardcoded). This manipulation can be done easily in python but is not simple in bash, so I've written a script in python doing this tasks and returning a string (or ideally an array of strings).
I'm calling this python script in my bash script. Both scripts are in the same directory and this directory is added to environment variables. I'm testing it on Ubuntu 22.04.
My python script below:
#!/usr/bin/python
def Get(input: str) -> list:
#Doing tasks - arr is an output array
return ' '.join(arr) #or ideally return arr
My bash script used to call the above python script
#!/bin/bash
ARR=("$(python -c "from test import Get; Get('val1, val2,val3')")")
echo $ARR
for ELEMENT in "${ARR[#]}"; do
echo "$ELEMENT"
done
When I added print in python script for test purposes I got proper results, so the python script works correctly. But in the bash script I got simply empty line. I've tried also something like that: ARR=("$(python -c "from test import Get; RES=Get('val1, val2,val3')")") and the iterate over res and got the same response.
It seems like the bash script cannot handle the data returned by python.
How can I rewrite this scripts to properly get python script response in bash?
Is it possible to get the whole array or only the string?
How can I rewrite this scripts to properly get python script response in bash?
Serialize the data from python side and deserialize on bash. Decide on proper protocol between the processes that would preserve any characters.
The best looks like it is to use newline or zero separated strings (protocol). Output delimiter separated elements from python (serialize) and read them properly on with readarray on bash side (deserialize).
$ tmp=$(python -c 'arr=[1,2,3]; print(*arr)')
$ readarray -t array <<<"$tmp"
$ declare -p array
declare -a array=([0]="1" [1]="2" [2]="3")
Or with zero separated stream. Note that Bash can't store zero bytes in variables, so we use redirection with process subtitution:
$ readarray -d '' -t array < <(python -c 'arr=[1,2,3]; print(*arr, sep="\0", end="")')
$ declare -p array
declare -a array=([0]="1" [1]="2" [2]="3")
I've solved my problem by exporting a string with elements separated by space.
I've also rewritten python code not to be a function but a script.
import sys
if len(sys.argv) > 1:
input = sys.argv[1]
#Doing tasks - arr is an output array
for element in arr:
print(element)
ARRAY=$(python script.py 'val1, val2,val3')
for ELEMENT in $ARRAY; do
echo "$ELEMENT"
done
The picture above shows the list of all kubernetes pods I need to save to a text file (or multiple text files).
I need a command which:
stores multiple pod logs into text files (or on single text file) - so far I have this command which stores one pod into one text file but this is not enough since I will have to spell out each pod name individually for every pod:
$ kubectl logs ipt-prodcat-db-kp-kkng2 -n ho-it-sst4-i-ie-enf > latest.txt
I then need the command to send these files into a python script where it will check for various strings - so far this works but if this could be included with the above command then that would be extremely useful:
python CheckLogs.py latest.txt latest2.txt
Is it possible to do either (1) or both (1) and (2) in a single command?
The simplest solution is to create a shell script that does exactly what you are looking for:
#!/bin/sh
FILE="text1.txt"
for p in $(kubectl get pods -o jsonpath="{.items[*].metadata.name}"); do
kubectl logs $p >> $FILE
done
With this script you will get the logs of all the pods in your namespace in a FILE.
You can even add python CheckLogs.py latest.txt
There are various tools that could help here. Some of these are commonly available, and some of these are shortcuts that I create my own scripts for.
xargs: This is used to run multiple command lines in various combinations, based on the input. For instance, if you piped text output containing three lines, you could potentially execute three commands using the content of those three lines. There are many possible variations
arg1: This is a shortcut that I wrote that simply takes stdin and produces the first argument. The simplest form of this would just be "awk '{print $1}'", but I designed mine to take optional parameters, for instance, to override the argument number, separator, and to take a filename instead. I often use "-i{}" to specify a substitution marker for the value.
skipfirstline: Another shortcut I wrote, that simply takes some multiline text input and omits the first line. It is just "sed -n '1!p'".
head/tail: These print some of the first or last lines of stdin. Interesting forms of this take negative numbers. Read the man page and experiment.
sed: Often a part of my pipelines, for making inline replacements of text.
I'd like to execute the following UNIX command in Python:
cd 2017-02-10; pwd; echo missing > 123.txt
The date directory DATE = 2017-02-10 and OUT = 123.txt are already variables in Python so I have tried variations of
call("cd", DATE, "; pwd; echo missing > ", OUT)
using the subprocess.call function, but I’m struggling to find documentation for multiple UNIX commands at once, which are normally separated by ; or piping with >
Doing the commands on separate lines in Python doesn’t work either because it “forgets” what was executed on the previous line and essentiality resets.
You can pass a shell script as a single argument, with strings to be substituted as out-of-band arguments, as follows:
date='2017-02-10'
out='123.txt'
subprocess.call(
['cd "$1"; pwd; echo missing >"$2"', # shell script to run
'_', # $0 for that script
date, # $1 for that script
out, # $2 for that script
], shell=True)
This is much more secure than substituting your date and out values into a string which is evaluated by the shell as code, because these values are treated as literals: A date of $(rm -rf ~) will not in fact try to delete your home directory. :)
Doing the commands on separate lines in Python doesn’t work either
because it “forgets” what was executed on the previous line and
essentiality resets.
This is because if you have separate calls to subprocess.call it will run each command in its own shell, and the cd call has no effect on the later shells.
One way around that would be to change the directory in the Python script itself before doing the rest. Whether or not this is a good idea depends on what the rest of the script does. Do you really need to change directory? Why not just write "missing" to 2017-02-10/123.txt from Python directly? Why do you need the pwd call?
Assuming you're looping through a list of directories and want to output the full path of each and also create files with "missing" in them, you could perhaps do this instead:
import os
base = "/path/to/parent"
for DATE, OUT in [["2017-02-10", "123.txt"], ["2017-02-11", "456.txt"]]:
date_dir = os.path.join(base, DATE)
print(date_dir)
out_path = os.path.join(date_dir, OUT)
out = open(out_path, "w")
out.write("missing\n")
out.flush()
out.close()
The above could use some error handling in case you don't have permission to write to the file or the directory doesn't exist, but your shell commands don't have any error handling either.
>>> date = "2017-02-10"
>>> command = "cd " + date + "; pwd; echo missing > 123.txt"
>>> import os
>>> os.system(command)
I'm trying to make some functions in python so that I can connect to a linux terminal and do stuff (like in this case, create a file). The code I have, works partially. The only thing that doesn't work is if you want to do something after you have entered the code. Like for instance you create the file and then want to navigate somewhere else (cd /tmp) for instance. Instead of doing the next command, it will just add to the file created.
def create_file(self, name, contents, location):
try:
log.info("Creating a file...")
self.device.execute("mkdir -p {}".format(location))
self.cd_path(location)
self.device.sendline("cat > {}".format(name))
self.device.sendline("{}".format(contents))
self.device.sendline("EOF") # send the CTRL + D command to save and exit I tried here with ^D as well
except:
log.info("Failed to create the file!")
The contents of the file is:
cat test.txt
#!/bin/bash
echo "Fail Method Requested"
exit 1
EOF
ls -d /tmp/asdasd
The order of commands executed is:
execute.create_file(test.txt, the_message, the_location)
execute.check_path("/tmp/adsasd") #this function just checks with ls -d if the directory exists.
I have tried with sendline the following combinations:
^D, EOF, <<EOF
I don't really understand how I could make this happen. I just want to create a file with a specific message. (When researching on how to do this with VI I got the same problem, but there the command I needed was the one for ESC)
If anyone could help with some input that would be great!!
Edit: As Rob mentioned below, sending the character "\x04" actually works. For anyone else having this issue, you can also consult this chart for other combinations if needed:
http://donsnotes.com/tech/charsets/ascii.html
You probably need to send the EOF character, which is typically CONTROL-D, not the three characters E, O, and F.
self.device.sendline("\x04")
http://wiki.bash-hackers.org/syntax/redirection#here_documents
Here docs allow you to use any file input termination string you like to represent end of file ( such as the literal EOF you're attempting to use now). Quoting that string tells the shell not to interpret expansions inside the heredoc content, ensuring that said content is treated as literal.
Using pipes.quote() here ensures that filenames with literal quotes, $s, spaces, or other surprising characters won't break your script. (Of course, you'll need to import pipes; on Python 3, by contrast, this has moved to shlex.quote()).
self.device.sendline("cat > {} <<'EOF'".format(pipes.quote(name)))
Then you can write the EOF as is, having told bash to interpret it as the end of file input.
(on os x 10.10.1)I am trying to use a paired-end merger (Casper) within a python script. i'm using os.system (don't want to use subprocess or pexpect modules). In my script here is the line that doesn't work:
os.system("casper %s %s -o %s"%(filein[0],filein[1],fileout))
#filein[0]: input file 1
#filein[1]: input file 2
#fileout: output prefix (default==casper)
Once my script is launched only the 2 first string parameters of this command are interpreted but not the third one, causing an output file with the default prefix name. Since my function is iterating through a lot of fastq files, they are all merged in a single "casper.fastq" file.
I tried to mess up with the part of the command that doesn't work (right after -o), putting meaningless string and still it is executed with no error and the default output, here is the "messed up line":
os.system("casper %s %s -ldkfnlqdskgfno %s"%(filein[0],filein[1],fileout))
Could anybody help in understanding what the heck is going on?
Print the command before execute it to check if your command wrapped correctly(like file name need to be quoted)
Execute your assumed output command directly to see if it is misinterpreted.