We Keep Coding

Can a variable expansion in python work like shell expansion of variable

I have a legacy python code that reads from a shell script for variable values like in a properties file. For example, say, I have a shell program x.sh for variable declaration as:
Y_HOME=/utils
Y_SPCL=$Y_HOME/spcl
UTIL1=$Y_SPCL/u1
Y_LIB=$Y_HOME/lib
Now, from within python program abc.py, I read x.sh file line by line and use line.split("=")[1] to get the value of the variable, say, UTIL1 as $Y_SPCL/u1 in non-expanded form and not in expanded form as /utils/spcl/u1.
Can I have sone mechanism in python to have vafiable expandion like in a shell program execution. I think, since I am using x.sh not as a shell program, rather as a configuration file like properties, there should be all variables in expanded form to let the python program run properly, such as:
Y_HOME=/utils
Y_SPCL=/utils/spcl
UTIL1=/utils/spcl/u1
Y_LIB=/utils/lib
This will have no change on the legacy python part of code and changing the configuration file as an external properties data.
Please pass your opinions.

There is a package dotenv that can do that.
pip install --user python-dotenv
Then, in Python:
import dotenv, os
dotenv.load_env("x.sh")
print(os.environ["Y_LIB"])
Important: Make sure your variable substitutions read like ${VAR}. So your x.sh would look like this:
Y_HOME=/utils
Y_SPCL=${Y_HOME}/spcl
UTIL1=${Y_SPCL}/u1
Y_LIB=${Y_HOME}/lib

Assuming variables must be declared before use and they form a correct path when expanded, you could do something like this.
There is the file fileName that contains the variables:
Y_HOME=/utils
Y_SPCL=$Y_HOME/spcl
UTIL1=$Y_SPCL/u1
Y_LIB=$Y_HOME/lib
So, For each "variable", you search for it in the next variables "values" and replace it for the proper "value". You could have a .py like this:
variables = []
with open("fileName", 'r') as f:
while True:
line = f.readline()[:-1] # Get the line minus '\n'
if not line:
break
variables.append(line.split('='))
for i in range(len(variables)):
current = "$" + variables[i][0]
for j in range(len(variables)):
replaced = None
while replaced != variables[j][1]:
# We replace until no further replaces happen
replaced = variables[j][1].replace(current, variables[i][1])
variables[j][1] = replaced
for var in variables:
print(var[0] + "=" + var[1])
Output:
Y_HOME=/utils
Y_SPCL=/utils/spcl
UTIL1=/utils/spcl/u1
Y_LIB=/utils/li

Chain of UNIX commands within Python

I'd like to execute the following UNIX command in Python:
cd 2017-02-10; pwd; echo missing > 123.txt
The date directory DATE = 2017-02-10 and OUT = 123.txt are already variables in Python so I have tried variations of
call("cd", DATE, "; pwd; echo missing > ", OUT)
using the subprocess.call function, but I’m struggling to find documentation for multiple UNIX commands at once, which are normally separated by ; or piping with >
Doing the commands on separate lines in Python doesn’t work either because it “forgets” what was executed on the previous line and essentiality resets.

You can pass a shell script as a single argument, with strings to be substituted as out-of-band arguments, as follows:
date='2017-02-10'
out='123.txt'
subprocess.call(
['cd "$1"; pwd; echo missing >"$2"', # shell script to run
'_', # $0 for that script
date, # $1 for that script
out, # $2 for that script
], shell=True)
This is much more secure than substituting your date and out values into a string which is evaluated by the shell as code, because these values are treated as literals: A date of $(rm -rf ~) will not in fact try to delete your home directory. :)

Doing the commands on separate lines in Python doesn’t work either
because it “forgets” what was executed on the previous line and
essentiality resets.
This is because if you have separate calls to subprocess.call it will run each command in its own shell, and the cd call has no effect on the later shells.
One way around that would be to change the directory in the Python script itself before doing the rest. Whether or not this is a good idea depends on what the rest of the script does. Do you really need to change directory? Why not just write "missing" to 2017-02-10/123.txt from Python directly? Why do you need the pwd call?
Assuming you're looping through a list of directories and want to output the full path of each and also create files with "missing" in them, you could perhaps do this instead:
import os
base = "/path/to/parent"
for DATE, OUT in [["2017-02-10", "123.txt"], ["2017-02-11", "456.txt"]]:
date_dir = os.path.join(base, DATE)
print(date_dir)
out_path = os.path.join(date_dir, OUT)
out = open(out_path, "w")
out.write("missing\n")
out.flush()
out.close()
The above could use some error handling in case you don't have permission to write to the file or the directory doesn't exist, but your shell commands don't have any error handling either.

>>> date = "2017-02-10"
>>> command = "cd " + date + "; pwd; echo missing > 123.txt"
>>> import os
>>> os.system(command)

How can I get the variable that a python file with a path prints when I call that python file from a shell script?

For some reason when I run this in my shell script only $exec_file and $_dir get passed into module_test.py, but neither $filename nor $modified_file get passed.
mod_test=$( /path/module_test.py $modified_file $_dir $filename )
(path is just a normal path that I decided to shorten for the sake of this example)
Am I typing this wrong? I am trying to get the output (an integer) of my module_test.py to be put into the variable mod_test.
My variables are:
modified_file = _File
_dir = /path to directory/
file = _File.py

Based on your example, you need to surround $_dir with quotes because it contains spaces, i.e.:
mod_test=$( /path/module_test.py $modified_file '$_dir' $filename )

Python call to external software command

I have a slight problem. I have a software which has a command with two inputs. The command is: maf2hal inputfile outputfile.
I need to call this command from a Python script. The Python script asks the user for path of input file and path of output file and stores them in two variables.The problem is that when I call the command maf2hal giving the two variable names as the arguments, the error I get is cannot locate file.
Is there a way around this? Here's my code:
folderfound = "n" # looping condition
while (folderfound == "n"):
path = raw_input("Enter path of file to convert (with the extension) > ")
if not os.path.exists(path):
print "\tERROR! file not found. Maybe file doesn't exist or no extension was provided. Try again!\n"
else:
print "\tFile found\n"
folderfound = "y"
folderfound = "y" # looping condition
while (folderfound == "y"):
outName = raw_input("Enter path of output file to be created > ")
if os.path.exists(outName):
print "\tERROR! File already exists \n\tEither delete the existing file or enter a new file name\n\n"
else:
print "Creating output file....\n"
outputName = outName + ".maf"
print "Done\n"
folderfound = "n"
hal_input = outputName #inputfilename, 1st argument
hal_output = outName + ".hal" #outputfilename, 2nd argument
call("maf2hal hal_input hal_output", shell=True)

This is wrong:
call("maf2hal hal_input hal_output", shell=True)
It should be:
call(["maf2hal", hal_input, hal_output])
Otherwise you're giving "hal_input" as the actual file name, rather than using the variable.
You should not use shell=True unless absolutely necessary, and in this case it is not only unnecessary, it is pointlessly inefficient. Just call the executable directly, as above.
For bonus points, use check_call() instead of call(), because the former will actually check the return value and raise an exception if the program failed. Using call() doesn't, so errors may go unnoticed.

There are a few problems. Your first reported error was that the call to the shell can't find the maf2hal program - that sounds like a path issue. You need to verify that the command is in the path of the shell that is being created.
Second, your call is passing the words "hal_input" and "hal_output". You'll need to build that command up first to pass the values of those variables;
cmd = "maf2hal {0} {1}".format(hal_input, hal_output)
call(cmd, shell=True)

Your code is literally trying to open a file called hal_input, not using the contents of your variable with the same name. It looks like you're using the subprocess module to execute, so you can just change it to call(["maf2hal", hal_input, hal_output], shell=True) to use the contents.

at the end of your code:
call("maf2hal hal_input hal_output", shell=True)
you are literally calling that string, not the executable and then those paths, you need to concatenate your strings together first, either by adding them of using .join
eg:
call("maf2hal " + hal_input + " " + hal_output", shell=True)
or
call("maf2hal ".join(hal_input, " ", hal_output), shell=True)

running a system command in a python script

I have been going through "A byte of Python" to learn the syntax and methods etc...
I have just started with a simple backup script (straight from the book):
#!/usr/bin/python
# Filename: backup_ver1.py
import os
import time
# 1. The files and directories to be backed up are specified in a list.
source = ['"C:\\My Documents"', 'C:\\Code']
# Notice we had to use double quotes inside the string for names with spaces in it.
# 2. The backup must be stored in a main backup directory
target_dir = 'E:\\Backup' # Remember to change this to what you will be using
# 3. The files are backed up into a zip file.
# 4. The name of the zip archive is the current date and time
target = target_dir + os.sep + time.strftime('%Y%m%d%H%M%S') + '.zip'
# 5. We use the zip command to put the files in a zip archive
zip_command = "zip -qr {0} {1}".format(target, ' '.join(source))
# Run the backup
if os.system(zip_command) == 0:
print('Successful backup to', target)
else:
print('Backup FAILED')
Right, it fails. If I run the zip command in the terminal it works fine. I think it fails because the zip_command is never actually run. And I don't know how to run it.
Simply typing out zip_command does not work. (I am using python 3.1)

Are you sure that the Python script is seeing the same environment you have access to when you enter the command manually in the shell? It could be that zip isn't on the path when Python launches the command.

It would help us if you could format your code as code; select the code parts, and click on the "Code Sample" button in the editor toolbar. The icon looks like "101/010" and if you hold the mouse pointer over it, the yellow "tool tip" box says "Code Sample <pre></pre> Ctrl+K"
I just tried it, and if you paste code in to the StackOverflow editor, lines with '#' will be bold. So the bold lines are comments. So far so good.
Your strings seem to contain backslash characters. You will need to double each backslash, like so:
target_dir = 'E:\\Backup'
This is because Python treats the backslash specially. It introduces a "backslash escape", which lets you put a quote inside a quoted string:
single_quote = '\''
You could also use a Python "raw string", which has much simpler rules for a backslash. A raw string is introduced by r" or r' and terminated by " or ' respectively. examples:
# both of these are legal
target_dir = r"E:\Backup"
target_dir = r'E:\Backup'

The next step I recommend is to modify your script to print the command string, and just look at the string and see if it seems correct.
Another thing you can try is to make a batch file that prints out the environment variables, and have Python run that, and see what the environment looks like. Especially PATH.
Here is a suggested example:
set
echo Trying to run zip...
zip
Put those in a batch file called C:\mytest.cmd, and then have your Python code run it:
result_code = os.system("C:\\mytest.cmd")
print('Result of running mytest was code', result_code)
If it works, you will see the environment variables printed out, then it will echo "Trying to run zip...", then if zip runs it will print a message with the version number of zip and how to run it.

zip command only work in linux not for windows.. thats why it make an error..