We Keep Coding

How to print to terminal like git log?

I'm writing a simple CLI application using python.
I have a list of records that I want to print in the terminal and I would like to output them just like git log does. So as a partial list you can load more using the down arrow, from which you quit by typing "q" (basically like the output of less, but without opening and closing a file).
How does git log do that?

You can pipe directly to a pager like this answer should work.
Alternatively, you can use a temporary file:
import os
import tempfile
import subprocess
# File contents for testing, replace with whatever
file = '\n'.join(f"{i} abc 123"*15 for i in range(400))
# Save the file to the disk
with tempfile.NamedTemporaryFile('w+', delete=False) as f:
f.write(file)
# Run `less` on the saved file
subprocess.check_call(["less", f.name])
# Delete the temporary file now that we are done with it.
os.unlink(f.name)

Device you are looking for is called pager, there exists pipepager function inside pydoc, which is not documented in linked pydoc docs, but using interactive python console you might learn that
>>> help(pydoc.pipepager)
Help on function pipepager in module pydoc:
pipepager(text, cmd)
Page through text by feeding it to another program.
therefore it seems that you should use this as follows
import pydoc
pydoc.pipepager("your text here", "less")
with limitation that it does depends on availability of less command.

How does git log do that?
git log invokes less when the output will not fit on the terminal. You can check that by running git log (if the repo doesn't have a lot of commits you can just resize the terminal before running the command) and then checking the running processes like so ps aux | grep less

call an executable string from python

I'm trying to find a way to run an executable script that can be downloaded from the web from Python, without saving it as a file. The script can be python code or bash or whatever - it should execute appropriately based on the shebang. I.e. if the following were saved in a file called script, then I want something that will run ./script without needing to save the file:
#!/usr/bin/env python3
import sys
from my_module import *
scoped_hash = sys.argv[1]
print(scoped_hash)
I have a function that reads such a file from the web and attempts to execute it:
def execute_artifact(command_string):
os.system('sh | ' + command_string)
Here's what happens when I call it:
>>> print(string)
'#!/usr/bin/env python3\nimport sys\nfrom my_module import *\n\nscoped_hash = sys.argv[1]\n\nobject_string = read_artifact(scoped_hash)\nparsed_object = parse_object(object_string)\nprint(parsed_object)\n'
>>> execute_artifact(string)
sh-3.2$ Version: ImageMagick 7.0.10-57 Q16 x86_64 2021-01-10 https://imagemagick.org
Copyright: © 1999-2021 ImageMagick Studio LLC
License: https://imagemagick.org/script/license.php
Features: Cipher DPC HDRI Modules OpenMP(4.5)
Delegates (built-in): bzlib freetype gslib heic jng jp2 jpeg lcms lqr ltdl lzma openexr png ps tiff webp xml zlib
Usage: import [options ...] [ file ]
Bizarrely, ImageMagick is called. I'm not sure what's going on, but I'm sure there's a better way to do this. Can anyone help me?

EDIT: This answer was added before OP updated requirements to include:
The script can be python code or bash or whatever - it should execute appropriately based on the shebang.
Some may still find the below helpful if they decided to try to parse the shebang themselves:
Probably, the sanest way to do this is to pass the string to the python interpreter as standard input:
import subprocess
p = subprocess.Popen(["python"], stdin=subprocess.PIPE)
p.communicate(command_string.encode())
My instinct tells me this entire thing is fraught with pitfalls. Perhaps, at least, you want to launch it using the same executable that launched your current process, so:
import subprocess
import sys
p = subprocess.Popen([sys.executable], stdin=subprocess.PIPE)
p.communicate(command_string.encode())
If you want to use arguments, I think using the -c option to pass in code as a string as an argument works, then you have access to the rest, so:
import subprocess
import sys
command_string = """
import sys
print(f"{sys.argv=}")
"""
completed_process = subprocess.run([sys.executable, "-c", command_string, "foo", "bar", "baz"])
The above prints:
sys.argv=['-c', 'foo', 'bar', 'baz']

This cannot be done in full generality.
If you want the shebang line to be interpreted as usual, you must write the script to a file. This is a hard requirement of the protocol that makes shebangs work. When a script with a shebang line is executed by the operating system, the kernel (and yes, it’s not the shell which does it, unlike what the question implies) reads the script and invokes the interpreter specified in the shebang, passing the pathname of the script as a command line argument. For that mechanism to work, the script must exist in the file system where the interpreter can find it. (It’s a rather fragile design, leading to some security issues, but it is what it is.)
Many interpreters will allow you to specify the program text on standard input or on the command line, but it is nowhere guaranteed that it will work for any interpreter. If you know you are working with an interpreter which can do it, you can simply try to parse the shebang line yourself and invoke the interpreter manually:
import io
import subprocess
import re
_RE_SHBANG = re.compile(br'^#!\s*(\S+)(?:\s+(.*))?\s*\n$')
def execute(script_body):
stream = io.BytesIO(script_body)
shebang = stream.readline()
m = _RE_SHBANG.match(shebang)
if not m:
# not a shebang
raise ValueError(shebang)
interp, arg = m.groups()
arg = (arg,) if arg is not None else ()
return subprocess.call([interp, *arg, '-c', script_body])
The above will work for POSIX shell and Python scripts, but not e.g. for Perl, node.js or standalone Lua scripts, as the respective interpreters take the -e option instead of -c (and the latter doesn’t even ignore shebangs in code given on the command line, so that needs to be separately stripped too). Feeding the script to the interpreter through standard input is also possible, but considerably more involved, and will prevent the script itself from using the standard input stream. That is also possible to overcome, but it doesn’t change the fact that it’s just a makeshift workaround that isn’t anywhere guaranteed to work in the first place. Better to simply write the script to a file anyway.

Git commit Vim in personal project

I am making a small python script that should prompt the user for input. I like how git commit prompts the user with a vim prompt, and then uses this prompt to get the commit message.
Is it possible to implement this behavior in python?
I cannot use input(or stdin in general)

Very simple: put initial text into a temporary file, start an editor (determined by the well-known environment variables with fallback to vi), wait the editor to finish and get the content of the temp file.
See an example at https://chase-seibert.github.io/blog/2012/10/31/python-fork-exec-vim-raw-input.html
import tempfile
import subprocess
import os
def raw_input_editor(default=None, editor=None):
''' like the built-in raw_input(), except that it uses a visual
text editor for ease of editing. Unline raw_input() it can also
take a default value. '''
with tempfile.NamedTemporaryFile(mode='r+') as tmpfile:
if default:
tmpfile.write(default)
tmpfile.flush()
subprocess.check_call([editor or get_editor(), tmpfile.name])
tmpfile.seek(0)
return tmpfile.read().strip()
def get_editor():
return (os.environ.get('VISUAL')
or os.environ.get('EDITOR')
or 'vi')

How to determine if Python script was run via command line?

Background
I would like my Python script to pause before exiting using something similar to:
raw_input("Press enter to close.")
but only if it is NOT run via command line. Command line programs shouldn't behave this way.
Question
Is there a way to determine if my Python script was invoked from the command line:
$ python myscript.py
verses double-clicking myscript.py to open it with the default interpreter in the OS?

If you're running it without a terminal, as when you click on "Run" in Nautilus, you can just check if it's attached to a tty:
import sys
if sys.stdin and sys.stdin.isatty():
# running interactively
print("running interactively")
else:
with open('output','w') as f:
f.write("running in the background!\n")
But, as ThomasK points out, you seem to be referring to running it in a terminal that closes just after the program finishes. I think there's no way to do what you want without a workaround; the program is running in a regular shell and attached to a terminal. The decision of exiting immediately is done just after it finishes with information it doesn't have readily available (the parameters passed to the executing shell or terminal).
You could go about examining the parent process information and detecting differences between the two kinds of invocations, but it's probably not worth it in most cases. Have you considered adding a command line parameter to your script (think --interactive)?

What I wanted was answered here: Determine if the program is called from a script in Python
You can just determine between "python" and "bash". This was already answered I think, but you can keep it short as well.
#!/usr/bin/python
# -*- coding: utf-8 -*-
import psutil
import os
ppid = os.getppid() # Get parent process id
print(psutil.Process(ppid).name())

I don't think there's any reliable way to detect this (especially in a cross-platform manner). For example on OS X, when you double-click a .py file and it tuns with "Python Launcher", it runs in a terminal, identically to if you execute it manually.
Although it may have other issues, you could package the script up with something like py2exe or Platypus, then you can have the double-clickable icon run a specific bit of code to differentiate (import mycode; mycode.main(gui = True) for example)

If you run python IDLE then "pythonw.exe" is being used to run coding while when you run the command line "python.exe" is used to run coding. The python folder path can vary so you have to revert the path to the python folder. m = '\\' and m = m[0] is to get m to be '\' because of escaping.
import sys
a = sys.executable
m = '\\'
m = m[0]
while True:
b = len(a)
c = a[(b - 1)]
if c == m:
break
a = a[:(b - 1)]
if sys.executable == a + 'pythonw.exe':
print('Running in Python IDLE')
else:
print('Running in Command line')

Update for later versions (e.g. Python 3.6 on Ubuntu 16.04): The statement to get the name has changed to psutil.Process(os.getpid()).parent().name()
I believe this CAN be done. At least, here is how I got it working in Python 2.7 under Ubuntu 14.04:
#!/usr/bin/env python
import os, psutil
# do stuff here
if psutil.Process(os.getpid()).parent.name == 'gnome-terminal':
raw_input("Press enter to close...")
Note that -- in Ubuntu 14 with the Gnome desktop (aka Nautilus) -- you might need to do this:
from a Nautilus window (the file browser), select Edit(menu)->Preferences(item) then Behavior(tab)->Executable Text Files(section)->Ask Each Time(radio).
chmod your script to be executable, or -- from a Nautilus window (the file browser) -- right click on the file->Properties(item) then Permissions(tab)->Execute:Allow executing file as program(checkbox)
double-click your file. If you select "Run in Terminal", you should see the "Type enter to close..." prompt.
now try from a bash prompt; you should NOT see the prompt.
To see how this works, you can fiddle with this (based on the answer by from #EduardoIvanec):
#!/usr/bin/env python
import os
import sys
import psutil
def parent_list(proc=None, indent=0):
if not proc:
proc = psutil.Process(os.getpid())
pid = proc.pid
name = proc.name
pad = " " * indent
s = "{0}{1:5d} {2:s}".format(pad, pid, name)
parent = proc.parent
if parent:
s += "\n" + parent_list(parent, indent+1)
return s
def invoked_from_bash_cmdline():
return psutil.Process(os.getpid()).parent.name == "bash"
def invoked_as_run_in_terminal():
return psutil.Process(os.getpid()).parent.name == "gnome-terminal"
def invoked_as_run():
return psutil.Process(os.getpid()).parent.name == "init"
if sys.stdin.isatty():
print "running interactively"
print parent_list()
if invoked_as_run_in_terminal():
raw_input("Type enter to close...")
else:
with open('output','w') as f:
f.write("running in the background!\n")
f.write("parent list:\n")
f.write(parent_list())

From the idea behind this answer, adding for Win10 compatibility (Ripped from Python 2.7 script; modify as needed):
import os, psutil
status = 1
if __name__ =="__main__":
status = MainFunc(args)
args = sys.argv
running_windowed = False
running_from = psutil.Process(os.getpid()).parent().name()
if running_from == 'explorer.exe':
args.append([DEFAULT OR DOUBLE CLICK ARGS HERE])
running_windowed = True
if running_windowed:
print('Completed. Exit status of {}'.format(status))
ready = raw_input('Press Enter To Close')
sys.exit(status)
There is a number of switch like statements you could add to be more universal or handle different defaults.

This is typically done manually/, I don't think there is an automatic way to do it that works for every case.
You should add a --pause argument to your script that does the prompt for a key at the end.
When the script is invoked from a command line by hand, then the user can add --pause if desired, but by default there won't be any wait.
When the script is launched from an icon, the arguments in the icon should include the --pause, so that there is a wait. Unfortunately you will need to either document the use of this option so that the user knows that it needs to be added when creating an icon, or else, provide an icon creation function in your script that works for your target OS.

My solution was to create command line scripts using setuptools. Here are a the relevant parts of myScript.py:
def main(pause_on_error=False):
if run():
print("we're good!")
else:
print("an error occurred!")
if pause_on_error:
raw_input("\nPress Enter to close.")
sys.exit(1)
def run():
pass # run the program here
return False # or True if program runs successfully
if __name__ == '__main__':
main(pause_on_error=True)
And the relevant parts of setup.py:
setup(
entry_points={
'console_scripts': [
'myScript = main:main',
]
},
)
Now if I open myScript.py with the Python interpreter (on Windows), the console window waits for the user to press enter if an error occurs. On the command line, if I run 'myScript', the program will never wait for user input before closing.

Although this isn't a very good solution, it does work (in windows at least).
You could create a batch file with the following contents:
#echo off
for %%x in (%cmdcmdline%) do if /i "%%~x"=="/c" set DOUBLECLICKED=1
start <location of python script>
if defined DOUBLECLICKED pause
If you want to be able to do this with a single file, you could try the following:
#echo off
setlocal EnableDelayedExpansion
set LF=^
:: The 2 empty lines are necessary
for %%x in (%cmdcmdline%) do if /i "%%~x"=="/c" set DOUBLECLICKED=1
echo print("first line of python script") %LF% print("second and so on") > %temp%/pyscript.py
start /wait console_title pyscript.py
del %temp%/pyscript.py
if defined DOUBLECLICKED pause
Batch code from: Pausing a batch file when double-clicked but not when run from a console window?
Multi-line in batch from: DOS: Working with multi-line strings

Okay, the easiest way I found and made was to simply run the program in the command line, even if it was ran in the Python IDLE.
exist = lambda x: os.path.exists(x) ## Doesn't matter
if __name__ == '__main__':
fname = "SomeRandomFileName" ## Random default file name
if exist(fname)==False: ## exist() is a pre-defined lambda function
jot(fname) ## jot() is a function that creates a blank file
os.system('start YourProgram.py') ## << Insert your program name here
os.system('exit'); sys.exit() ## Exits current shell (Either IDLE or CMD)
os.system('color a') ## Makes it look cool! :p
main() ## Runs your code
os.system("del %s" % fname) ## Deletes file name for next time
Add this to the bottom of your script and once ran from either IDLE or Command Prompt, it will create a file, re-run the program in the CMD, and exits the first instance.
Hope that helps! :)

I also had that question and, for me, the best solution is to set an environment variable in my IDE (PyCharm) and check if that variable exists to know if the script is being executed either via the command line or via the IDE.
To set an environment variable in PyCharm check:
How to set environment variables in PyCharm?
Example code (environment variable: RUNNING_PYCHARM = True):
import os
# The script is being executed via the command line
if not("RUNNING_PYCHARM" in os.environ):
raw_input("Press enter to close.")
I hope it works for you.

Based on existing solutions and using sets:
import psutil
def running_interactively():
"""Return True if any of our parent processes is a known shell."""
shells = {"cmd.exe", "bash.exe", "powershell.exe", "WindowsTerminal.exe"}
parent_names = {parent.name() for parent in psutil.Process().parents()}
# print(parent_names)
# print(f"Shell in parents? {shells & parent_names}")
return bool(shells & parent_names)
if not running_interactively():
input("\nPress ENTER to continue.")

This answer is currently specific to Windows, but it can be reconfigured to work with other operating systems in theory. Rather than installing psutil module like most of these answers recommend, you can make use of the subprocess module and the Windows tasklist command to explicitly get the name of the parent process of your Python program.
import os
import subprocess
shells = {"bash.exe", "cmd.exe", "powershell.exe", "WindowsTerminal.exe"}
# These are standard examples, but it can also be used to detect:
# - Nested python.exe processes (IDLE, etc.)
# - IDEs used to develop your program (IPython, Eclipse, PyCharm, etc.)
# - Other operating system dependent shells
s = subprocess.check_output(["tasklist", "/v", "/fo", "csv", "/nh", "/fi", f"PID eq {os.getppid()}"])
# Execute tasklist command to get the verbose info without the header (/nh) of a single process in CSV format (/fo csv)
# Such that its PID is equal to os.getppid()
entry = s.decode("utf-8").strip().strip('"').split('","')
# Decode from bytes to str, remove end whitespace and quotations from CSV format
# And split along the quote delimited commas
# This process may differ and require adjustment when used for an OS other than Windows
condition = entry and entry[0] in shells
# Check first that entry is not an empty sequence, meaning the process has already ended
# If it still exists, check if the first element -- the executable -- exists as an element of the set of executables you're looking for
I hope this is helpful for anyone looking for an answer to this problem while minimizing the number of dependencies you'd need.
This was tested in Python 3.8 and uses an f-string in the subprocess.check_output line of the code, so please be sure to convert the f-string to a compatible syntax if you're working with a version of Python before f-strings were introduced.

run python command line interpreter with imports loaded automatically

I would like to play around in the python interpreter but with a bunch of imports and object setup completed. Right now I'm launching the interpreter on the command line and doing the setup work every time. Is there any way to launch the command line interpreter with all the initialization work done?
Ex:
# Done automatically.
import foo
import baz
l = [1,2,3,4]
# Launch the interpreter.
launch_interpreter()
>> print l
>> [1,2,3,4]

You can create a script with the code you wish to run automatically, then use python -i to run it. For example, create a script (let's call it script.py) with this:
import foo
import baz
l = [1,2,3,4]
Then run the script
$ python -i script.py
>>> print l
[1, 2, 3, 4]
After the script has completed running, python leaves you in an interactive session with the results of the script still around.
If you really want some things done every time you run python, you can set the environment variable PYTHONSTARTUP to a script which will be run every time you start python. See the documentation on the interactive startup file.

I use PYTHONSTARTUP.
My .bash_profile has a path to my home folder .pyrc, which as the import statements in it.
https://docs.python.org/3/using/cmdline.html#envvar-PYTHONSTARTUP

I came across this question when trying to configure a new desk for my research and found that the answers above didn't quite suit my desire: to contain the entire desk configuration within one file (meaning I wouldn't create a separate script.py as suggested by #srgerg).
This is how I ended up achieving my goal:
export PYTHONPATH=$READ_GEN_PATH:$PYTHONPATH
alias prepy="python3 -i -c \"
from naive_short_read_gen import ReadGen
from neblue import neblue\""
In this case neblue is in the CWD (so no path extension is required there), whereas naive_short_read_gen is in an arbitrary directory on my system, which is specified via $READ_GEN_PATH.
You could do this in a single line if necessary: alias prepy=PYTHONPATH=$EXTRA_PATH:$PYTHONPATH python3 -i -c ....

You can use the -s option while starting the command line. The details are given in the documentation here

I think I know what you want to do. You might want to check IPython, because you cannot start the python interpreter without giving the -i option (at least not directly).
This is what I did in my project:
def ipShell():
'''Starts the interactive IPython shell'''
import IPython
from IPython.config.loader import Config
cfg = Config()
cfg.TerminalInteractiveShell.confirm_exit = False
IPython.embed(config=cfg, display_banner=False)
# Then add the following line to start the shell
ipShell()
You need to be careful, though, because the shell will have the namespace of the module that the function ipShell() is defined. If you put the definition in the file you run, then you will be able to access the globals() you want. There could be other workarounds to inject the namespace you want, b̶u̶t̶ ̶y̶o̶u̶ ̶w̶o̶u̶l̶d̶ ̶h̶a̶v̶e̶ ̶t̶o̶ ̶g̶i̶v̶e̶ ̶a̶r̶g̶u̶m̶e̶n̶t̶s̶ ̶t̶o̶ ̶t̶h̶e̶ ̶f̶u̶n̶c̶t̶i̶o̶n̶ ̶i̶n̶ ̶t̶h̶a̶t̶ ̶c̶a̶s̶e̶.
EDIT
The following function defaults to caller's namespace (__main__.__dict__).
def ipShell():
'''Starts the interactive IPython shell
with the namespace __main__.__dict__'''
import IPython
from __main__ import __dict__ as ns
from IPython.config.loader import Config
cfg = Config()
cfg.TerminalInteractiveShell.confirm_exit = False
IPython.embed(config=cfg, user_ns=ns, display_banner=False)
without any extra arguments.