We Keep Coding

How to emulate backtick subshells in plumbum

I want to write this command in Plumbum:
foo `date +%H%M%S`
or, equivalently:
foo $(date +%H%M%S)
(think of 'foo' as a command like 'mkdir')
How can I write this in Plumbum? I want to have this as a Plumbum object that I can reuse - ie each time I call it I call 'foo' with the current time, being different each call.
I tried using a subshell. This example:
#!/usr/bin/env python3
from plumbum import local
import time
s = local["sh"]
f = s["-c", "mkdir", "$(date +%H%M%S)"]
for i in range(0,10):
f()
time.sleep(1)
doesn't work because I get:
plumbum.commands.processes.ProcessExecutionError: Unexpected exit code: 1
Command line: | /usr/bin/sh -c mkdir '$(date +%H%M%S)'
Stderr: | mkdir: missing operand
| Try 'mkdir --help' for more information.
I could use time.localtime() to compute the time in Python, but that would be the same for every evaluation. I could also generate a shell script file with this in, but then I'd have to clean that up afterwards.
How can I write this in a Plumbum-native way?

You came close to having something that would work with sh -c. The trick is to understand that only the one argument directly after the -c is parsed as code by the shell; subsequent arguments go into $0, $1, etc. in the context in which that code is executed.
s = local['sh']
f = s['-c', 'mkdir "$(date +%H%M%S)"']
That said, I'd strongly argue that plumbum is being more of a hinderance than a help, and suggest getting rid of it outright.

I haven't find a way to do it using Plumbum API, but introducing a wrapper function works:
import time
from plumbum.cmd import mkdir, date
def mkdate():
return mkdir[date['+%H%M%S']()]
for i in range(0, 3):
mkdate()
time.sleep(1)
mkdate is a regular function, unfortunately, not a Plumbum object, but I haven't found anything in the docs right away =)

Start Python REPL and execute command [duplicate]

I would like to play around in the python interpreter but with a bunch of imports and object setup completed. Right now I'm launching the interpreter on the command line and doing the setup work every time. Is there any way to launch the command line interpreter with all the initialization work done?
Ex:
# Done automatically.
import foo
import baz
l = [1,2,3,4]
# Launch the interpreter.
launch_interpreter()
>> print l
>> [1,2,3,4]

You can create a script with the code you wish to run automatically, then use python -i to run it. For example, create a script (let's call it script.py) with this:
import foo
import baz
l = [1,2,3,4]
Then run the script
$ python -i script.py
>>> print l
[1, 2, 3, 4]
After the script has completed running, python leaves you in an interactive session with the results of the script still around.
If you really want some things done every time you run python, you can set the environment variable PYTHONSTARTUP to a script which will be run every time you start python. See the documentation on the interactive startup file.

I use PYTHONSTARTUP.
My .bash_profile has a path to my home folder .pyrc, which as the import statements in it.
https://docs.python.org/3/using/cmdline.html#envvar-PYTHONSTARTUP

I came across this question when trying to configure a new desk for my research and found that the answers above didn't quite suit my desire: to contain the entire desk configuration within one file (meaning I wouldn't create a separate script.py as suggested by #srgerg).
This is how I ended up achieving my goal:
export PYTHONPATH=$READ_GEN_PATH:$PYTHONPATH
alias prepy="python3 -i -c \"
from naive_short_read_gen import ReadGen
from neblue import neblue\""
In this case neblue is in the CWD (so no path extension is required there), whereas naive_short_read_gen is in an arbitrary directory on my system, which is specified via $READ_GEN_PATH.
You could do this in a single line if necessary: alias prepy=PYTHONPATH=$EXTRA_PATH:$PYTHONPATH python3 -i -c ....

You can use the -s option while starting the command line. The details are given in the documentation here

I think I know what you want to do. You might want to check IPython, because you cannot start the python interpreter without giving the -i option (at least not directly).
This is what I did in my project:
def ipShell():
'''Starts the interactive IPython shell'''
import IPython
from IPython.config.loader import Config
cfg = Config()
cfg.TerminalInteractiveShell.confirm_exit = False
IPython.embed(config=cfg, display_banner=False)
# Then add the following line to start the shell
ipShell()
You need to be careful, though, because the shell will have the namespace of the module that the function ipShell() is defined. If you put the definition in the file you run, then you will be able to access the globals() you want. There could be other workarounds to inject the namespace you want, b̶u̶t̶ ̶y̶o̶u̶ ̶w̶o̶u̶l̶d̶ ̶h̶a̶v̶e̶ ̶t̶o̶ ̶g̶i̶v̶e̶ ̶a̶r̶g̶u̶m̶e̶n̶t̶s̶ ̶t̶o̶ ̶t̶h̶e̶ ̶f̶u̶n̶c̶t̶i̶o̶n̶ ̶i̶n̶ ̶t̶h̶a̶t̶ ̶c̶a̶s̶e̶.
EDIT
The following function defaults to caller's namespace (__main__.__dict__).
def ipShell():
'''Starts the interactive IPython shell
with the namespace __main__.__dict__'''
import IPython
from __main__ import __dict__ as ns
from IPython.config.loader import Config
cfg = Config()
cfg.TerminalInteractiveShell.confirm_exit = False
IPython.embed(config=cfg, user_ns=ns, display_banner=False)
without any extra arguments.

Pass a code of Python in a command line

Is it possible to somehow convert string which I pass via command line e.g
python myfile.py "s = list(range(1000))"
into module? And then...
#myfile.py
def printlist(s):
print(s)
?
Something like a timeit module, but with my own code.
python -m timeit -s "s = list(range(1000))" "sorted(s)"

Save the code
import sys
print(sys.argv)
as filename.py, then run python filename.py Hi there! and see what happens ;)

I'm not sure if I understand correctly, but the sys module has the attribute argv that is a list of all the arguments passed in the command line.
test.py:
#!/usr/bin/env python
import sys
for argument in sys.argv:
print(argument)
This example will print every argument passed to the script, take in mind that the first element sys.argv[0] will always be the name of the script.
$ ./test.py hello there
./test.py
hello
there

There are two options to send short code snippets similar to what you can do via the timeit module or certain other -m module option functionality and these are:
You can use either the -c option of Python e.g.:
python -c "<code here>"
Or you could use piping such as:
echo <code here> | python
You can combine multiple statements using the ; semicolon statement separator. However if you use a : such as with while or for or def or if then you cannot use the ; so there may be limited options. Possibly some clever ways around this limitation but I have yet to see them.

How does argparse (and the deprecated optparse) respond to 'tab' keypress after python program name, in bash?

I have tested optcomplete working with the optparse module. Its example is a simple file so I could get that working. I also tested it using the argparse module as the prior one is deprecated. But I really do not understand how and by whom the python program gets called on tab presses. I suspect bash together with the shebang line and the argparse (or optparse) module are involved in some way. I have been trying to figure this out (now gonna read the source code).
I have a little more complex program structure, which includes a wrapper around the piece of code which handles the arguments. Its argparse.ArgumentParser() instantiation and calls to add_argument() - which are superclassed into another intermediate module to avoid duplicating code, and wrapper around that is being called - are inside a function.
I want to understand the way this tab completion works between bash and python (or for that matter any other interpretor like perl).
NOTE: I have a fair understanding of bash completion (which I learned just now), and I think I understand the bash(only) custom completion.
NOTE: I have read other similar SO questions, and none really answer this Q.
Edit: Here is the bash function.
I already understood how the python module gets to know about words typed in the command line, by reading os.environ values of variables
$COMP_WORDS
$COMP_CWORD
$COMP_LINE
$COMP_POINT
$COMPREPLY
These variables have values only on tab press.
My question is how does the python module gets triggered?

To understand what's happening here, let's check what that bash function actually does:
COMPREPLY=( $( \
COMP_LINE=$COMP_LINE COMP_POINT=$COMP_POINT \
COMP_WORDS="${COMP_WORDS[*]}" COMP_CWORD=$COMP_CWORD \
OPTPARSE_AUTO_COMPLETE=1 $1 ) )
See the $1 at the end? That means that it actually calls the Python file we want to execute with special environment variables set! To trace what's happening, let's prepare a little script to intercept what optcomplete.autocomplete does:
#!/usr/bin/env python2
import os, sys
import optparse, optcomplete
from cStringIO import StringIO
if __name__ == '__main__':
parser = optparse.OptionParser()
parser.add_option('-s', '--simple', action='store_true',
help="Simple really simple option without argument.")
parser.add_option('-o', '--output', action='store',
help="Option that requires an argument.")
opt = parser.add_option('-p', '--script', action='store',
help="Option that takes python scripts args only.")
opt.completer = optcomplete.RegexCompleter('.*\.py')
# debug env variables
sys.stderr.write("\ncalled with args: %s\n" % repr(sys.argv))
for k, v in sorted(os.environ.iteritems()):
sys.stderr.write(" %s: %s\n" % (k, v))
# setup capturing the actions of `optcomplete.autocomplete`
def fake_exit(i):
sys.stderr.write("autocomplete tried to exit with status %d\n" % i)
sys.stdout = StringIO()
sys.exit = fake_exit
# Support completion for the command-line of this script.
optcomplete.autocomplete(parser, ['.*\.tar.*'])
sys.stderr.write("autocomplete tried to write to STDOUT:\n")
sys.stderr.write(sys.stdout.getvalue())
sys.stderr.write("\n")
opts, args = parser.parse_args()
This gives us the following when we try to autocomplete it:
$ ./test.py [tab]
called with args: ['./test.py']
...
COMP_CWORD: 1
COMP_LINE: ./test.py
COMP_POINT: 10
COMP_WORDS: ./test.py
...
OPTPARSE_AUTO_COMPLETE: 1
...
autocomplete tried to exit with status 1
autocomplete tried to write to STDOUT:
-o -h -s -p --script --simple --help --output
So optcomplete.autocomplete just reads the environment, prepares the matches, writes them to STDOUT and exits. The result -o -h -s -p --script --simple --help --output is then put into a bash array (COMPREPLY=( ... )) and returned to bash to present the choices to the user. No magic involved :)

run python command line interpreter with imports loaded automatically

I would like to play around in the python interpreter but with a bunch of imports and object setup completed. Right now I'm launching the interpreter on the command line and doing the setup work every time. Is there any way to launch the command line interpreter with all the initialization work done?
Ex:
# Done automatically.
import foo
import baz
l = [1,2,3,4]
# Launch the interpreter.
launch_interpreter()
>> print l
>> [1,2,3,4]

You can create a script with the code you wish to run automatically, then use python -i to run it. For example, create a script (let's call it script.py) with this:
import foo
import baz
l = [1,2,3,4]
Then run the script
$ python -i script.py
>>> print l
[1, 2, 3, 4]
After the script has completed running, python leaves you in an interactive session with the results of the script still around.
If you really want some things done every time you run python, you can set the environment variable PYTHONSTARTUP to a script which will be run every time you start python. See the documentation on the interactive startup file.

I use PYTHONSTARTUP.
My .bash_profile has a path to my home folder .pyrc, which as the import statements in it.
https://docs.python.org/3/using/cmdline.html#envvar-PYTHONSTARTUP

I came across this question when trying to configure a new desk for my research and found that the answers above didn't quite suit my desire: to contain the entire desk configuration within one file (meaning I wouldn't create a separate script.py as suggested by #srgerg).
This is how I ended up achieving my goal:
export PYTHONPATH=$READ_GEN_PATH:$PYTHONPATH
alias prepy="python3 -i -c \"
from naive_short_read_gen import ReadGen
from neblue import neblue\""
In this case neblue is in the CWD (so no path extension is required there), whereas naive_short_read_gen is in an arbitrary directory on my system, which is specified via $READ_GEN_PATH.
You could do this in a single line if necessary: alias prepy=PYTHONPATH=$EXTRA_PATH:$PYTHONPATH python3 -i -c ....

You can use the -s option while starting the command line. The details are given in the documentation here

I think I know what you want to do. You might want to check IPython, because you cannot start the python interpreter without giving the -i option (at least not directly).
This is what I did in my project:
def ipShell():
'''Starts the interactive IPython shell'''
import IPython
from IPython.config.loader import Config
cfg = Config()
cfg.TerminalInteractiveShell.confirm_exit = False
IPython.embed(config=cfg, display_banner=False)
# Then add the following line to start the shell
ipShell()
You need to be careful, though, because the shell will have the namespace of the module that the function ipShell() is defined. If you put the definition in the file you run, then you will be able to access the globals() you want. There could be other workarounds to inject the namespace you want, b̶u̶t̶ ̶y̶o̶u̶ ̶w̶o̶u̶l̶d̶ ̶h̶a̶v̶e̶ ̶t̶o̶ ̶g̶i̶v̶e̶ ̶a̶r̶g̶u̶m̶e̶n̶t̶s̶ ̶t̶o̶ ̶t̶h̶e̶ ̶f̶u̶n̶c̶t̶i̶o̶n̶ ̶i̶n̶ ̶t̶h̶a̶t̶ ̶c̶a̶s̶e̶.
EDIT
The following function defaults to caller's namespace (__main__.__dict__).
def ipShell():
'''Starts the interactive IPython shell
with the namespace __main__.__dict__'''
import IPython
from __main__ import __dict__ as ns
from IPython.config.loader import Config
cfg = Config()
cfg.TerminalInteractiveShell.confirm_exit = False
IPython.embed(config=cfg, user_ns=ns, display_banner=False)
without any extra arguments.