We Keep Coding

How to source a file having environment variables

I have a source file having environment variables (few environmental variables are conditional based as well).
source file is like (filename: global_setup)
if( -f /tools/tool/bin) then
setenv SHELL /bin/csh
endif
setenv LICENCE_FILE 2457#abc
setenv ...........
I want to source this file using python script. Code is like this:
import subprocess
import os
if os.path.isfile('global_setup'):
subprocess.call(['/bin/csh', '-c', 'source ~/global_setup'], shell=True)
else:
print("file not found");
But this is not working. Showing error as:
/home/global_setup: line 53: syntax error: unexpected end of file
Also, I tried to remove this EOF error by changing file format to Unix.
Even it works, i doubt setup will be done in some other terminal as i am using subprocess.call() in python.

Sourcing a file with subprocess.call() won't have an effect on the environment variables in Python, since subprocess runs, well, a subprocess.
You will need to parse the file yourself if you wish to affect os.environ within your Python interpreter.
Since your file is csh-style, the otherwise very nice envparse module won't do here.
Something you could do, though, is invoke
env = subprocess.check_output(['/bin/csh', '-c', 'source ~/global_setup; printenv'], shell=True)
and parse the resulting env variable line by line and add the contents into os.environ. This way the conditionals within global_setup would also work as expected.

Instead of doing it this way, you should be doing one of two things:
Source your files in the linux shell before invoking your script.
Convert your file into a dotenv file that Python can read using the Python-dotenv module.
Sourcing it before is easier IMO, instead of sourcing it during runtime using subprocess.

File paths within my Python script fail when launching via double-click (Windows 10)

I have a python script that reads and writes to files that are located relative to it, in directories above and beside it. When I run my script via Cygwin using
python script.py
The program works perfectly. However, when I run it by navigating through the windows GUI to my file and double clicking, I get a blank cmd prompt and then my program runs fine until I reach the point where I need to access the other files, at which point it fails and gives me this message in the cmd prompt that opens itself:
../FFPRM.TXT
../2025510296/FFPRM_000.TXT
Exception in Tkinter callback
Traceback (most recent call last):
File "C:\Users\rbanks\AppData\Local\Programs\Python\Python35-32\lib\tkinter\__init__.py", line 1549, in __call__
return self.func(*args)
File "C:\Users\rbanks\Desktop\TSAC\EXECUTABLE\T-SAC_GUI.py", line 705, in run_exe
invalid_entry, output_text = self.apply()
File "C:\Users\rbanks\Desktop\TSAC\EXECUTABLE\T-SAC_GUI.py", line 694, in apply
p = subprocess.Popen(['cp', output_file_path, output_file_path_id])
File "C:\Users\rbanks\AppData\Local\Programs\Python\Python35-32\lib\subprocess.py", line 950, in __init__
restore_signals, start_new_session)
File "C:\Users\rbanks\AppData\Local\Programs\Python\Python35-32\lib\subprocess.py", line 1220, in _execute_child startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified
I am deploying this script as well as the directory structure as a zip for users to be able to unzip and use anywhere on their PC, so it is important for me to be able to run it with a simple double click and my relative file paths.
My first thought was the cmd prompt that was opening and executing my script was in a different environment, but when I run:
cd
pause
in a .cmd script, I get:
C:\Users\rbanks\Desktop\TSAC\EXECUTABLE>pause
Which is the correct location.
I am not having any luck with Google, I assume because I can't seem to construct a sufficient search query. Could someone point me in the right direction please?

[edit] the other answer is correct(at least i suspect) but i will leave this here in the hopes that it helps the op in the future with path problems ... and doing something like this is just generally good practice
use
BASEPATH = os.path.abspath(os.path.dirname(__file__))
at the top of your script
the later
txt_file = os.path.join(BASEPATH,"my_file.txt")
or even
txt_file = os.path.join(BASEPATH,"..","my_file.txt")
this gives you the benefit of being able to do things like
if not os.path.exists(txt_file):
print "Cannot find file: %r"%txt_file
which will likely give you a better idea about what your problem might actually be (if its simply path related at least)

The problem is not the current directory. It is correct when double clicking on the icon
The problem is: Cygwin commands are not in the windows path
You are using python, so don't run simple copy commands like this, which make your script non-portable and subject to variations, requiring installation of cygwin, etc...
p = subprocess.Popen(['cp', output_file_path, output_file_path_id])
can be replaced by
import shutil
shutil.copyfile(output_file_path, output_file_path_id)
shutil module complements os module for file manipulation functions that aren't native in the operating system.
now you have a 100% pythonic solution, native, which will throw exceptions if cannot read/write files, so fully integrated in the rest of your program.
Before running an external command from Python make sure that no python way exists. There are so many useful modules out there.
Other examples of how to avoid running basic commands from python (of course if you need to run a C compilation it's different!):
zipfile package: much better than running zip.exe
gzip package: can open gzipped files natively from python
os.listdir() instead of parsing the output of cmd /c dir /B
os.remove() instead of calling rm or del
etc... python rules!

Explorer does not show folder if started with path dirname of file location

With Python 3.4.2 on Windows I want to open the explorer with the folder of the currently running script like this:
import os, subprocess
subprocess.check_call(['explorer', os.path.dirname(__file__)])
Instead I see that the explorer is opened by with a default folder
"C:\Users\XX\Documents" and an error is thrown:
Traceback (most recent call last):
File "C:/XXX/YYY.py", line 9, in <module>
subprocess.check_call(['explorer', os.path.dirname(__file__)])
File "C:\Python34\lib\subprocess.py", line 561, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['explorer', 'C:/XXX']' returned non-zero exit status 1
and although os.path.dirname returns paths with slashes on Windows (backslashes are path seperators) you can copy and paste it into the explorer and it will open the location just fine. (XXX is the path part, YYY the file name part)
Furthermore, if you manually write the path as it is in the explorer (with backslashes), even then the command fails.
subprocess.check_call(['explorer', r'C:\Users'])
But at least the explorer will open the right directory despite of throwing the same error (so maybe using call instead of check_call).
What's going on here? How can I show the folder of the running Python script file in the explorer?

You need to send the absolute path to the directory by calling os.path.abspath, and switch check_call with call (as you suspected).
import os
import subprocess
dirname = (os.path.dirname(os.path.abspath(__file__)))
subprocess.call(['explorer', dirname])
subprocess.check_call raises an exception if the return code of the process is nonzero. Even when explorer doesn't actually have an error, it has a non-zero return status, so you get an exception.
That's actually unrelated to the problem, though. The problem is that you were passing a string with forward slashes to explorer. This would have happened if you called your script like python path/to/script.py instead of python path\to\script.py. If you pass the absolute path to subprocess.call, explorer will open in the right directory. You would get the same error if you passed an empty string as the second arg to subprocess.call.

You need to use backslashes in your paths for windows explorer to undestand them. Testing from the command line, this works:
> explorer C:\Users
But this doesn't
> explorer C:/Users
Use os.path.normpath to convert the forward slashes into back slashes
Note that abspath, according to the docs, is implemented as normpath(join(os.getcwd(), path)), so the other solution also addresses this issue.

Starting module shell command from python subprocess module

I'm trying to run vnc server, but in order to do it first I need to run 'module load vnc'.
If I call which module in loaded bash shell then the command in not found is the PATH but in the same time it's available. It looks like the command is built-in.
In other words it looks like I need to execute two commands at once module load vnc;vncserver :8080 -localhost and I'm writing script to start it from python.
I have tried different variants with subprocess.Popen like
subprocess.Popen('module load vnc;vncserver :8080 -localhost', shell=True)
which returns 127 exit code or command not found.
subprocess.Popen('module load vnc;vncserver :8080 -localhost', shell=False)
showing
File <path>/subprocess.py line 621, in \__init__
errread, errwrite)
OSError: [Errno 2] No such file or directory.
If I specify shell=True, it executes from /bin/sh but I need it from /bin/bash.
Specifying executable='/bin/bash' doesn't help as it loads new bash shell but it starts as string but not as process, i.e. I see in ps list exactly the same command I would like to start.
Would you please advise how to do start this command from subprocess module? Is it possible to have it started with shell=False?

Environment Modules usually just modifies a couple environment variables for you. It's usually possible to skip the module load whatever step altogether and just not depend on those modules. I recommend
subprocess.Popen(['/possibly/path/to/vncserver', ':8080', '-localhost'],
env={'WHATEVER': 'you', 'MAY': 'need'})
instead of loading the module at all.
If you do insist on using this basic method, then you want to start bash yourself with Popen(['bash',....

If you want to do it with shell=False, just split this into two Popen calls.
subprocess.check_call('module load vnc'.split())
subprocess.Popen('vncserver :8080 -localhost'.split())

You can call module from a Python script. The module command is provided by the environment-modules software, which also provides a python.py initialization script.
Evaluating this script in a Python script enables the module python function. If environment-modules is installed in /usr/share/Modules, you can find this script at /usr/share/Modules/init/python.py.
Following code enables module python function:
import os
exec(open('/usr/share/Modules/init/python.py').read())
Thereafter you can load your module and start your application:
module('load', 'vnc')
subprocess.Popen(['vncserver', ':8080', '-localhost'])

Using subprocess to run Python script on Windows

Is there a simple way to run a Python script on Windows/Linux/OS X?
On the latter two, subprocess.Popen("/the/script.py") works, but on Windows I get the following error:
Traceback (most recent call last):
File "test_functional.py", line 91, in test_functional
log = tvnamerifiy(tmp)
File "test_functional.py", line 49, in tvnamerifiy
stdout = PIPE
File "C:\Python26\lib\subprocess.py", line 595, in __init__
errread, errwrite)
File "C:\Python26\lib\subprocess.py", line 804, in _execute_child
startupinfo)
WindowsError: [Error 193] %1 is not a valid Win32 application
monkut's comment: The use case isn't clear. Why use subprocess to run a python script? Is there something preventing you from importing the script and calling the necessary function?
I was writing a quick script to test the overall functionality of a Python-command-line tool (to test it on various platforms). Basically it had to create a bunch of files in a temp folder, run the script on this and check the files were renamed correctly.
I could have imported the script and called the function, but since it relies on sys.argv and uses sys.exit(), I would have needed to do something like..
import sys
import tvnamer
sys.argv.append("-b", "/the/folder")
try:
tvnamer.main()
except BaseException, errormsg:
print type(errormsg)
Also, I wanted to capture the stdout and stderr for debugging incase something went wrong.
Of course a better way would be to write the script in more unit-testable way, but the script is basically "done" and I'm doing a final batch of testing before doing a "1.0" release (after which I'm going to do a rewrite/restructure, which will be far tidier and more testable)
Basically, it was much easier to simply run the script as a process, after finding the sys.executable variable. I would have written it as a shell-script, but that wouldn't have been cross-platform. The final script can be found here

Just found sys.executable - the full path to the current Python executable, which can be used to run the script (instead of relying on the shbang, which obviously doesn't work on Windows)
import sys
import subprocess
theproc = subprocess.Popen([sys.executable, "myscript.py"])
theproc.communicate()

How about this:
import sys
import subprocess
theproc = subprocess.Popen("myscript.py", shell = True)
theproc.communicate() # ^^^^^^^^^^^^
This tells subprocess to use the OS shell to open your script, and works on anything that you can just run in cmd.exe.
Additionally, this will search the PATH for "myscript.py" - which could be desirable.

Yes subprocess.Popen(cmd, ..., shell=True) works like a charm. On Windows the .py file extension is recognized, so Python is invoked to process it (on *NIX just the usual shebang). The path environment controls whether things are seen. So the first arg to Popen is just the name of the script.
subprocess.Popen(['myscript.py', 'arg1', ...], ..., shell=True)

It looks like windows tries to run the script using its own EXE framework rather than call it like
python /the/script.py
Try,
subprocess.Popen(["python", "/the/script.py"])
Edit: "python" would need to be on your path.

For example, to execute following with command prompt or BATCH file we can use this:
C:\Python27\python.exe "C:\Program files(x86)\dev_appserver.py" --host 0.0.0.0 --post 8080 "C:\blabla\"
Same thing to do with Python, we can do this:
subprocess.Popen(['C:/Python27/python.exe', 'C:\\Program files(x86)\\dev_appserver.py', '--host', '0.0.0.0', '--port', '8080', 'C:\\blabla'], shell=True)
or
subprocess.Popen(['C:/Python27/python.exe', 'C:/Program files(x86)/dev_appserver.py', '--host', '0.0.0.0', '--port', '8080', 'C:/blabla'], shell=True)

You are using a pathname separator which is platform dependent. Windows uses "\" and Unix uses "/".

When you are running a python script on windows in subprocess you should use python in front of the script name. Try:
process = subprocess.Popen("python /the/script.py")

Supplemental info: It is worth noting that the documentation states that you need to use shell=True if you are using a dos shell command like dir
without it you get something like this.
>>> import subprocess
>>> subprocess.run(['dir'])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\foo\AppData\Local\Programs\Python\Python38\lib\subprocess.py", line 489, in run
with Popen(*popenargs, **kwargs) as process:
File "C:\Users\foo\AppData\Local\Programs\Python\Python38\lib\subprocess.py", line 854, in __ini
t__
self._execute_child(args, executable, preexec_fn, close_fds,
File "C:\Users\foo\AppData\Local\Programs\Python\Python38\lib\subprocess.py", line 1307, in _exe
cute_child
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] The system cannot find the file specified
>>> subprocess.run(['dir'], shell=True)
Volume in drive J is garbage
Volume Serial Number is 5EE7-B084
Also you can use path like objects for the args which is recent addition.
from pathlib import Path
subprocess.run(Path('c:/proj/myfile.bat'))
Also worth noting there is a whole set of windows specific controls that allow you to control how a process is spawned which concurrent operations can use.
So controlling subprocesses on windows is not as simple as posix style.