We Keep Coding

how to I redirect both code file and terminal output to the same output file (simulating python repl)?

I would like to save the input code and the output result into a file. For example the following python code code.py:
print(2+2)
print(3+2)
to create a code-and-output.txt:
>>> print(2+2)
4
>>> print(3+2)
5
But I can not get it working. Basically, I want to code-and-output.txt to capture what would happen if I run interpreted python and run statements in python interactive environment (code + output).
Ways that I have tried so far:
Redirect stdout:
python code.py > code-and-output.txt
It only saves the output.
Redirect stdout and stdin:
python < code.py > code-and-output.txt
It does the same (only output).
nohup
nohup python code.py
The same problem: only output.
Script
script -q code-and-output.txt
python
print(2+2)
print(2+3)
ctr+d
ctr+d
It works but I need to do it manually. Moreover, it saves some garbage that I can not make them quiet with -q.
Bash Script
# bash-file.sh
python &
print(2+2)
print(2+3)
Does not work: commands run in console bash, not python. It does not work with & either: never ends python repl.
Using tty
open another terminal like /dev/pts/2 and send above bash-file.sh
cat bash-file.sh > /dev/pts/2
It just copies but does not run.
I am not interested in solutions like Jupyter and iPython. They have their own problems that does not address my requirement.
Any solution through linux commands (preferably) or python? Thank you.

Save this is as repl_sim.py in the same directory as your code.py:
with open("code.py", 'r') as input_file:
for line in input_file:
print(f">>> {line.strip()}")
eval(line)
Then run in your terminal with the following if you want to redirect the output to a text file named code-and-output.txt:
python repl_sim.py > code-and-output.txt
-OR-
Then run in your terminal with the following if you want to see the output as well as the make the text file matching:
python repl_sim.py | tee code-and-output.txt
It at least works for the example you provided as code.py.
Pure Python version of first option above so that you don't need shell redirect.
Save this code as repl_sim.py:
import contextlib
with open('code-and-output.txt', 'w') as f:
with contextlib.redirect_stdout(f):
with open("code.py", 'r') as input_file:
for line in input_file:
print(f">>> {line.strip()}")
eval(line)
Then run in your terminal with:
python repl_sim.py
That will result in code-and-output.txt with your desired content.
Contextlib use based on Raymond Hettinger's August 17th 2018 Tweet and contextlib.redirect_stdout() documentation
.

Running bash on python to read from S3 Bucket and saving output

I'm trying to run the following bash command on python and save that output into a variable. I'm new to using bash so any help will be appreciated.
Here's my usecase I have data stored in an S3 bucket (let's say the path is s3://test-bucket/folder1/subd1/datafiles/)
in the datafiles folder there are multiple data files:
a1_03_27_2020_N.csv
a1_04_05_2021_O.csv
a1_07_16_2021_N.csv
I'm trying to select the latest file (in this case a1_07_16_2021_N) and then read that data file using pandas
Here's what I have so far
The command to select the latest file
ls -t a1*|head -1
but then I'm not sure how to
1- run that command on python
2- how to save that the output as a variable
(I know this is not correct but something like
latest_file = os.environ['ls -t a1*|head -1'])
Then read the file:
df = pd.read_csv(latest_file)
Thank you in advance again!

Python replaces most shell functionality. You can do the search and filtering in python itself. No need for a callout.
from pathlib import Path
dir_to_search = Path("test-bucket/folder1/subd1/datafiles/")
try:
latest = max(dir_to_search.glob("a1*.csv"), key=lambda path: path.stat().st_mtime)
print(latest)
except ValueError:
print("no csv here")
But if you want to run the shell, several functions in subprocess will do it. For instance,
import subprocess as subp
result = subp.run("ls -t test-bucket/folder1/subd1/datafiles/a1* | head -1",
shell=True,
capture_output=True, text=True).stdout.strip()

PYTHON AND BATCH SCRIPT: Run file if it exists and create if it doesn't

Full Disclaimer: I DO NOT KNOW PYTHON.
Hi Guys,
I have made an AutoHotKey Script for my volume keys. I would like to create a batch file which runs a python file (so if I change computers, I can easily create this scripts) which would do the following
Check if volume_keys.ahk exists in the D Drive
If it exists, run that;
If it doesn't exist, then create a file named volume_keys.ahk and add my script to it.
My script is:
^!NumpadMult::Send {Volume_Mute}
^!NumpadAdd::Send {Volume_Up}
^!NumpadSub::Send {Volume_Down}
I know how to code the .bat file and just need help for the python point-of-view, but I request the community to check it:
#ECHO OFF
ECHO This script will run an AHK Script. If you want to stop this process from happening, then cross this window off.If you want to continye:
pause
cd d:
D:\run_volume_keys_ahk_script.py
I really appreciate any help by the community.
Thanks in advance

You can use the os library for this. Here's what the python program would look like.
import os
if os.path.isfile('D:\\volume_keys.ahk'): # check if it exists
os.system('D:\\volume_keys.ahk') # execute it
else:
with open('D:\\volume_keys.ahk', 'w') as f: # open it in w (write) mode
f.write('^!NumpadMult::Send {Volume_Mute} \
^!NumpadAdd::Send {Volume_Up} \
^!NumpadSub::Send {Volume_Down}') # Write to file
os.system('D:\\volume_keys.ahk') # execute

To activate the ahk script, you might want to use the subprocess module, of which I took the example from here
import subprocess
subprocess.call(["path/to/ahk.exe", "script.ahk"])
Note that you'll have to find the ahk executable on a computer before you can use the script, maybe you want to automatically check that too.
You can set the path you want to check for scripts in one string, and then add the filenames of your scripts as strings to a list. You can use listdir() from the os module to see any files and directories at a given path, then iterate over your scriptnames and check if it exists in that list of files. If it does, run it.
In this example I copy-pasted your script into a string as value for the key 'scriptname' in a dictionary, so that python can actually create the script file. This isn't really a neat way to do it though, you might want to have your scripts prepared in a directory next to your python script and copy them from there. See an example of how here
from os import listdir
from os.path import isfile, join
CHECK_PATH = "D:"
AHK_EXECUTABLE_PATH = "path/to/ahk.exe"
SCRIPTS_TO_CHECK = {'script1.ahk':"""^!NumpadMult::Send {Volume_Mute}
^!NumpadAdd::Send {Volume_Up}
^!NumpadSub::Send {Volume_Down} """, 'script2.ahk':" some other script here"}
files_to_check = set(listdir(CHECK_PATH)) # using a set for fast lookup later
for scriptname, script in SCRIPTS_TO_CHECK.items():
if not scriptname in files_to_check:
print(f"script {scriptname} not found, creating it.")
with open(scriptname, 'w') as file:
file.write(script)
# else
subprocess.call(AHK_EXECUTABLE_PATH, scriptname)

How to fix "'bool' object is not iterable" error when running a UNIX command on a directory of files in python

I am trying to run a list of files in a directory through a UNIX executable using a python. I would the output of the executable for each file written to a different directory but retaining the original filename.
I am using python 2.7 so using the subprocess.call method. I am getting an error that says "'bool' object is not iterable" which I am guessing is due to the part where I am trying to write the output files as when I run the following script through the console I get an expected output specific to the executable within the console window:
import subprocess
import os
for inp in os.listdir('/path/to/input/directory/'):
subprocess.call(['/path/to/UNIX/executable', inp])
My code is currently this:
import subprocess
import os
for inp in os.listdir('/path/to/input/directory/'):
out = ['/path/to/output/directory/%s' % inp]
subprocess.call(['/path/to/UNIX/executable', inp] > out)
However, this second lot of code returns the "'bool' is not iterable" error.
I'm guessing the solution is pretty trivial as it is not a complicated task however, as a beginner, I do not know where to start!
SOLVED: following #barak-itkin's answer, for those who may stumble across this issue in the future, the code ran successfully using the following:
import subprocess
import os
for inp in os.listdir('/path/to/input/directory/'):
with open('/path/to/output/directory/%s' % inp, 'w') as out_file:
subprocess.call(['/path/to/UNIX/executable', inp], stdout=out_file)

To write the output of a subprocess.call to a file, you would need to either use the > path/to/out as part of the command itself, or to do it "properly" by specifying the file to which the output should go:
# Option 1:
# Specify that the command is using a "shell" syntax, meaning that
# things like output redirection (such as with ">") should be handled
# by the shell that will evaluate the command
subprocess.call('my_command arg1 arg2 > /path/to/out', shell=True)
# Option 2:
# Open the file to which you want to write the output, and then specify
# the `stdout` parameter to be that file
with open('/path/to/out', 'w') as out_file:
subprocess.call(['my_command', 'arg1', 'arg2'], stdout=out_file)
Does this work for you?

Python: system command

I spend a few hours writing a little script.
Basically what it does is create a new text file and fills it up with whatever.
I zip the text file --using zipfile-- and here's where my problem lies.
I want to run the Windows system command:
copy /b "imgFile.jpg" + "zipFile.zip" newImage.jpg
To merge the image "imgFile.jpg" and the zip "zipFile.zip".
So:
os.system("copy /b \"imgFile.jpg\" + \"zipFile.zip\" newImage.jpg")
When I run my script, it all seems to go fine.
But when it's done and I try to extract the 'newImage.jpg' file, it gives me:
The archive is either in unknown format or damaged
This ONLY happens when I run the system command within the script.
It works fine when I use the shell. It even works if I use a separate script.
I've double checked my zip file. Everything is in good shape.
Is there something I'm doing wrong? Something I'm not seeing?

Have you tried using shutil?
import shutil
shutil.copy(src, dst)

There may be a problem with the way Python is passing the arguments to the shell command. Try using subprocess.call. This method takes arguments as an array and passes them that way to the command:
import subprocess
subprocess.call(["copy", "/b", '"imgFile.jpg" + "zipFile.zip"', "newImage.jpg"])