We Keep Coding

Logging printout of an executed python file within another file and printing out the result in terminal simultaneously

I have two Python files (main.py and main_test.py). The file main_test.py is executed within main.py. When I do not use a log file this is what gets printed out:
Main file: 17:41:18
Executed file: 17:41:18
Executed file: 17:41:19
Executed file: 17:41:20
When I use a log file and execute main.py>log, then I get the following:
Executed file: 17:41:18
Executed file: 17:41:19
Executed file: 17:41:20
Main file: 17:41:18
Also, when I use python3 main.py | tee log to print out and log the output, it waits and prints out after finishing everything. In addition, the problem of reversing remains.
Questions
How can I fix the reversed print out?
How can I print out results simultaneously in terminal and log them in a correct order?
Python files for replication
main.py
import os
import time
import datetime
import pytz
python_file_name = 'main_test'+'.py'
time_zone = pytz.timezone('US/Eastern') # Eastern-Time-Zone
curr_time = datetime.datetime.now().replace(microsecond=0).astimezone(time_zone).time()
print(f'Main file: {curr_time}')
cwd = os.path.join(os.getcwd(), python_file_name)
os.system(f'python3 {cwd}')
main_test.py
import pytz
import datetime
import time
time_zone = pytz.timezone('US/Eastern') # Eastern-Time-Zone
for i in range(3):
curr_time = datetime.datetime.now().replace(microsecond=0).astimezone(time_zone).time()
print(f'Executed file: {curr_time}')
time.sleep(1)

When you run a script like this:
python main.py>log
The shell redirects output from the script to a file called log. However, if the script launches other scripts in their own subshell (which is what os.system() does), the output of that does not get captured.
What is surprising about your example is that you'd see anything at all when redirecting, since the output should have been redirected and no longer echo - so perhaps there's something you're leaving out here.
Also, tee waits for EOF on standard in, or for some error to occur, so the behaviour you're seeing there makes sense. This is intended behaviour.
Why bother with shells at all though? Why not write a few functions to call, and import the other Python module to call its functions? Or, if you need things to run in parallel (which they didn't in your example), look at multiprocessing.
In direct response to your questions:
"How can I fix the reversed print out?"
Don't use redirection, and write to file directly from the script, or ensure you use the same redirection when calling other scripts from the first (that will get messy), or capture the output from the subprocesses in the subshell and pipe it to the standard out of your main script.
"How can I print out results simultaneously in terminal and log them in a correct order?"
You should probably just do it in the script, otherwise this is not a really a Python question and you should try SuperUser or similar sites to see if there's some way to have tee or similar tools write through live.
In general though, unless you have really strong reasons to have the other functionality running in other shells, you should look at solving your problems in the Python script. And if you can't, use you can use something like Popen or derivatives to capture the subscript's output and do what you need instead of relying on tools that may or may not be available on the host OS running your script.

How to run a python file from another securely?

I try to make my code modular because it's too long, the problem is I don't know whether i'm doing it safely. I segmented my code into different files, so 1 python file runs the others, sometimes I have to call 1 file that will run another file that will run another file, so multiple chained commands.
The issue is that some of the files will process sensitive information like passwords, so I don't know whether I do it safely. Ideally after 1 file is executed, it should close itself, and delete all variables from it's memory, and free that space, like it normally would as if I were to just execute 1 file, the problem is that I don't know whether if I call multiple files nested into one another, this applies. Obviously only the file that is executed should clear itself, not the one that is active, but I don't know if this is the case.
I have been calling my modules like this
os.system('python3 ' + filename)
And in each file subsequently the same code calling another file with os.system, forming a nested or chained call system.
For example if I call the first file from shell:
python3 file1.py
and then file1 calls:
os.system('python3 file2.py')
and then file2 calls:
os.system('python3 file3.py')
I would want file3 cleaned from the memory and closed entirely after it runs, whereas file2 and file1 might still be active. I don't want file3 to be still inside the memory after it executed itself. So if file3 works with passwords, it should obviously clean them from the memory after it runs.
How to do this?
I have read about multiple options:
from subprocess import call
call(["python3", "file2.py"])
import subprocess
subprocess.call("file2.py", shell=True)
execfile('file2.py')
import subprocess
subprocess.Popen("file2.py", shell=True)
Which one is safer?

Python is heavily relying on the notion of importation. You should not try to reinvent the wheel on this one. Just import your scripts from the main script and use functions to trigger them. If you want to be sure variables are discarded you should include a del statement at the end of the functions or as soon as the variable is no longer in use.
On another hand, your problem with password is flawed from the start. If a .py file contains a password in plain text it's not, it will never be, in no scenario, secured. You should implement a secret : see this topic : I need to securely store a username and password in Python, what are my options?

Import results of a c++-Programm to python

I'm currently dealing with some python based squish gui tests. Some of these tests call another tool, written in c++ and build as an executable. I have full access to that tool and I'm able to modify it. The tests call it via command line and currently evaluate the error code and create a passed or failed depending on the error codes value.
I think there is a better way to do it or? One Problem is, that the error code is limited to uint8 on unix systems and I would like to be able to share more than just an error code with my python script.
My first idea was printing everything in a file in json or xml and read that file. But this somehow sounds wrong for me. Has anybody a better idea?

When I first read the question, I immediately thought piping the output would work. Check this link out to get a better idea:
Linux Questions Piping
If this doesn't work, I do think writing your output to a file and reading it with your python script would get the job done.

You can capture the output of the external process via Python and process it as you see fit.
Here is a very simple variant:
import os
import subprocess
def main():
s = os_capture(["ls"])
if "ERROR" in s:
test.fail("Executing 'ls' failed.")
def os_capture(args, cwd=None):
if cwd is None:
cwd = os.getcwd()
stdout = subprocess.Popen(
args=args,
cwd=cwd,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT).communicate()[0]
return stdout

using mkdir and touch sub-processes sequentially doesn't work

I have an error that I keep encountering repeatedly, sadly without being able to find solution to at the site.
try:
#create working dir if it doens't exist already
if not os.path.isdir(WORKINGDIR):
print '>>>mdkir ',WORKINGDIR
subprocess.Popen(['mkdir',WORKINGDIR]).wait()
print os.path.isdir(WORKINGDIR)
#create output csv file
outputCSVFile = WORKINGDIR+ '/'+'results.csv'
if not os.path.isfile(outputCSVFile):
print '>>> touch',outputCSVFile
subprocess.check_output(['touch',outputCSVFile])
Although the line: print os.path.isdir(WORKINGDIR) always prints True, subprocess returns this error:
touch: cannot touch
`/nfs/iil/proj/mpgarch/archive_06/CommandsProfiling/fastScriptsOutput190916/results.csv':
No such file or directory
The same error doesn't appear when I use subprocess.checkoutput instead of subprocess.Popen().wait().
I know that this issue can be solved in many ways (such as using os methods to creates directories and files), but I am interested on why my way isn't working.
Thanks in advance.
EDIT: as some suggested, the problem probably lies with the fact that the program continues too fast after the subprocess.Popen, and hence the issue is solved using subprocess.checkoutput which is probably slower (since it has to wait for output). But still - I don't understand exactly what is happening, since os.path.istdir shows that the dir was created, before continuing to the line that performs touch

I suppose you have file permission problems.
In your path it appears that you are using NFS. Did you already try it on the local file system?
Anyway, you should avoid to use sub processes for simple file operations.
To create a directory:
if not os.path.exists(WORKINGDIR):
os.makedirs(WORKINGDIR)
For touch:
import os
def touch(fname, times=None):
with open(fname, 'a'):
os.utime(fname, times)
touch(WORKINGDIR+ '/'+'results.csv')

Python doesn't delete temporary file if the program fails

I'm writing a program, which, inter alia, works with temporary file, created using tempfile library.
The temporary file creates and fills in function:
def func():
mod_script = tempfile.NamedTemporaryFile(dir='special')
dest = open(mod_script, 'w')
# filling dest
return mod_script
(I use open() and not with open() because I execute the temporary file after calling func())
After some operations with mod_script outside func(), I call mod_script.close(). And all works fine.
But I have one problem. If my program fails (or if I interrupt it), the temporary file doesn't remove.
How do I fix it ?
I really don't want to write try...except...finally clauses because I'll have to write it so many times (there are many points, where my program can fail).

First, use a with statement, and pass delete=False to the constructor.
Then you need to put the necessary error handling in your program. Catch exceptions (see try..finally) and clean up during program exit whether it is successful or crashes.
Alternatively, keep the file open while executing it to prevent the automatic deletion-on-close from deleting it before you have executed it. This may have issues on Windows where it tends to have conflicts using files that are open.