I have a bash script which extracts a variable (an integer) calculated in a Python script. I then try to do an arithmetic operation on this variable, but Bash seems to have trouble with this.
Python script:
import sys
foo = 278736 # This number is obtained by reading a file using this Python script
print(foo)
sys.exit(0)
Bash script:
#!/bin/bash
var=`python python_script.py`
var2=$((var/100)) # Line which causes the problem
This produces this error:
-bash: 278736: syntax error: operand expected (error token is "278736")
It seems that somehow, bash doesn't know how to convert the string "278736", extracted from the Python script, to an integer. If I do an arithmetic operation with a normal Bash variable, it works fine:
var_test=278736
var2=$((var_test/100)) # Works fine
I have searched, but can't find anybody who has asked the same question. Does anyone know how to make this work?
Related
I've bash script doing some tasks but I need to manipulate on string obtained from configuration (for simplification in this test it's hardcoded). This manipulation can be done easily in python but is not simple in bash, so I've written a script in python doing this tasks and returning a string (or ideally an array of strings).
I'm calling this python script in my bash script. Both scripts are in the same directory and this directory is added to environment variables. I'm testing it on Ubuntu 22.04.
My python script below:
#!/usr/bin/python
def Get(input: str) -> list:
#Doing tasks - arr is an output array
return ' '.join(arr) #or ideally return arr
My bash script used to call the above python script
#!/bin/bash
ARR=("$(python -c "from test import Get; Get('val1, val2,val3')")")
echo $ARR
for ELEMENT in "${ARR[#]}"; do
echo "$ELEMENT"
done
When I added print in python script for test purposes I got proper results, so the python script works correctly. But in the bash script I got simply empty line. I've tried also something like that: ARR=("$(python -c "from test import Get; RES=Get('val1, val2,val3')")") and the iterate over res and got the same response.
It seems like the bash script cannot handle the data returned by python.
How can I rewrite this scripts to properly get python script response in bash?
Is it possible to get the whole array or only the string?
How can I rewrite this scripts to properly get python script response in bash?
Serialize the data from python side and deserialize on bash. Decide on proper protocol between the processes that would preserve any characters.
The best looks like it is to use newline or zero separated strings (protocol). Output delimiter separated elements from python (serialize) and read them properly on with readarray on bash side (deserialize).
$ tmp=$(python -c 'arr=[1,2,3]; print(*arr)')
$ readarray -t array <<<"$tmp"
$ declare -p array
declare -a array=([0]="1" [1]="2" [2]="3")
Or with zero separated stream. Note that Bash can't store zero bytes in variables, so we use redirection with process subtitution:
$ readarray -d '' -t array < <(python -c 'arr=[1,2,3]; print(*arr, sep="\0", end="")')
$ declare -p array
declare -a array=([0]="1" [1]="2" [2]="3")
I've solved my problem by exporting a string with elements separated by space.
I've also rewritten python code not to be a function but a script.
import sys
if len(sys.argv) > 1:
input = sys.argv[1]
#Doing tasks - arr is an output array
for element in arr:
print(element)
ARRAY=$(python script.py 'val1, val2,val3')
for ELEMENT in $ARRAY; do
echo "$ELEMENT"
done
I have tried to execute a simple python command from cmd like C:\Users> stat.py < swagger.yaml > output.html, which executes stat.py by taking swagger.yaml as input argument and generates output.html file and it worked fine in cmd. But now i want to execute my stat.py file through another python file demo.py by passing the values swagger.yaml and output.html as sys.argv[0] and sys.argv[1] inside demo.py.
my command from cmd C:\Users> demo.py swagger.yaml output.html and my demo.py file is as follows..
# my demo.py file ....
import os
import sys
os.system('stat.py < sys.argv[1] > sys.argv[2]')
error - the system can not find the file specified.
Why i am getting this error and please any help to resolve it ..
Inside a normal string, no variable interpretation is applied. So you literally asked to read from a file named sys.argv[1] (possibly sys.argv1 if the file exists, thanks to shell globbing), and write to a file named sys.argv[2].
If you want to use the values sys.argv in your script, you need to format them into the string, e.g. with f-strings (modern Python 3.6 or so only):
os.system(f'stat.py < {sys.argv[1]} > {sys.argv[2]}') # Note f at beginning of literal
or on older Python 2.7, with str.format:
os.system('stat.py < {} > {}'.format(sys.argv[1], sys.argv[2]))
Note that however you slice it, this is dangerous; os.system is launching this in a shell, and arguments that contain shell metacharacters will be interpreted as such. It can't do anything the user didn't already have permission to do, but small mistakes by the user could dramatically change the behavior of the program. If you want to do this properly/safely, use subprocess, open the files yourself, and pass them in explicitly as stdin/stdout:
with open(sys.argv[1], 'rb') as infile, open(sys.argv[2], 'wb') as outfile:
subprocess.run(['stat.py'], stdin=infile, stdout=outfile)
This ensures the files can be opened in the first place before launching the process, doesn't allow the shell to interpret anything, and avoids the (minor) expense of launching a shell at all. It's also going to give you more useful errors if opening the files fails.
I need to extract text from a PDF. I tried the PyPDF2, but the textExtract method returned an encrypted text, even though the pdf is not encrypted acoording to the isEncrypted method.
So I moved on to trying accessing a program that does the job from the command prompt, so I could call it from python with the subprocess module. I found this program called textExtract, which did the job I wanted with the following command line on cmd:
"textextract.exe" "download.pdf" /to "download.txt"
However, when I tried running it with subprocess I couldn't get a 0 return code.
Here is the code I tried:
textextract = shlex.split(r'"textextract.exe" "download.pdf" /to "download.txt"')
subprocess.run(textextract)
I already tried it with shell=True, but it didn't work.
Can anyone help me?
I was able to get the following script to work from the command line after installing the PDF2Text Pilot application you're trying to use:
import shlex
import subprocess
args = shlex.split(r'"textextract.exe" "download.pdf" /to "download.txt"')
print('args:', args)
subprocess.run(args)
Sample screen output of running it from a command line session:
> C:\Python3\python run-textextract.py
args: ['textextract.exe', 'download.pdf', '/to', 'download.txt']
Progress:
Text from "download.pdf" has been successfully extracted...
Text extraction has been completed!
The above output was generated using Python 3.7.0.
I don't know if your use of spyder on anaconda affects things or not since I'm not familiar with it/them. If you continue to have problems with this, then, if it's possible, I suggest you see if you can get things working directly—i.e. running the the Python interpreter on the script manually from the command line similar to what's shown above. If that works, but using spyder doesn't, then you'll at least know the cause of the problem.
There's no need to build a string of quoted strings and then parse that back out to a list of strings. Just create a list and pass that:
command=["textextract.exe", "download.pdf", "/to", "download.txt"]
subprocess.run(command)
All that shlex.split is doing is creating a list by removing all of the quotes you had to add when creating the string in the first place. That's an extra step that provides no value over just creating the list yourself.
Suppose I have a python file main.py, and it has some optional parameters, --learning-rate, --batch-size, and etc.
If I want to run this file, I can input the following into the terminal(Ubuntu Linux for example).
python3 main.py --learning-rate 0.1 --batch-size 100
And now, I want to write some code in main.py, in order that after I enter the command above, I can get this command in a string by executing those code. The following is the string I want to get:
"python3 main.py --learning-rate 0.1 --batch-size 100"
The reason I want to do this is that I want to write this string into my recording file so that I can know better what command I have run.
Could anyone tell me what package should I import and what code should I write to get that command information during running the python file?
Thanks!
You cannot always get precisely what you typed, because the shell will have first done substitutions and expanded filenames before starting your script. For example, if you type python "foo.py" *.txt, your script won't see *.txt, it will see the list of files, and it won't see the quotes around foo.py.
With that caveat out of the way, the sys module has a variable named argv that contains all of the arguments. argv[0] is the name of the script.
To get the name of the python executable you can use sys.executable.
To tie it all together, you can do something like this:
print(sys.executable + " " + " ".join(sys.argv))
Why not just remake the command using the arguments you parsed? It won't be exactly what you typed, but that might be nice as it will be in a common format.
Ex (assuming the learning rate and batch size are stored in similarly named variables):
command = "python3 main.py --learning-rate {} --batch-size {}".format(learning_rate, batch_size)
Of course it will be a little more complicated if some commands are optional, but I assume in that case there would be a default value for these parameters, since your network would need those parameters every time.
i need some help with this...
I have a program installed on my computer that i want to call to calculate some things and give me an output-file...
in Matlab the command "dos()" does the job giving me also the cmd screen output in matlab.
I need this to work in python but i am making something wrong.
data='file.csv -v'
db=' -d D:\directory\bla\something.db'
anw='"D:\Program Files\bla\path\to\anw.exe"' + db + ' -i' + data
"anw" output is this one:
>>> anw
'"D:\\Program Files\\bla\\path\\to\\anw.exe" -d D:\\directory\\bla\\something.db -i file.csv -v'
## without the "" it does not work either
import subprocess as sb
p= sb.Popen('cmd','/K', anw) ## '/C' does not work either
i get the following error message from cmd.exe inside the python shell
Windows cannot find "\"D:\Program Files\bla\path\to\anw.exe"" Make sure you typed the name correctly, and then try again.
this line runs when i make a bat. file out of it.
it runs in matlab via "dos(anw)" so what is wrong here?
ps: i have blanks in my command... could this be the problem? i do not know where the first "\" comes from in the cmd. exe error message
for now i created a bat. file with all the stuff cmx.de should do in the specific directory where the input file lies...
i just had to tell python to change directory with
import os
os.chdir("D:\working\directory")
os.system(r'D:\working\directory\commands.bat')
it works good and gives me the output of cmd directly in the python shell