I want a function that programmatically returns completion options from either bash or zsh. There are lots of examples of related questions on stackoverflow but no proper, generic answers anywhere. I do NOT want to know how to write a specific completer function for bash.
I've already tried implementing this by reading debian /etc/completion shell code, by echoing control-codes for tab into "bash -i", and even tried using automated subprocess interaction with python-pexpect. Every time I thought I was successful, I find some small problem that invalidates the whole solution. I'd accept a solution in any language, but ideally it would be python. Obviously the exact input output would vary depending on systems, but take a look at the example I/O below:
function("git lo") returns ["log","lol","lola"]
function("apt-get inst") returns ["apt-get install"]
function("apt-get") returns []
function("apt-get ") returns ["apt-get autoclean","apt-get autoremove", ...]
function ("./setup") returns ["./setup.py"]
If you are thinking of a solution written in shell, it would ideally be something I can execute without "source"ing. For instance bash "compgen" command looks interesting (try "compgen -F _git"), but note that "bash -c 'compgen -F _git'" does not work because the completion helper "_git" is not in scope.
This gist is my best solution so far. It meets all the requirements, works well for multiple versions of bash on multiple OS's but it requires a subprocess call and it's so complicated it's absurd. The comments includes full documentation of all the outrageous slings and arrows. I'm still hoping for something more reasonable to come along, but unless it does.. this is it!
Related
I am working with multiprocessing and I want to ask if there is some option to create new process with new terminal window in Ubuntu
I have 3 processes starting simultaneously, but I want results from them in separated terminals for each
Thanks
No. Sadly there is no* way to do what you want within python, as python has no control over the terminal it is running in.
What I think you want though is to separate the messages from your different processes, so you can see what's going on. What I sometimes do for this (in testing only!) is to have each process log to a different file, and then watch those three files in three terminal windows. You can do this with watch or even a simple while loop in bash:
watch -n 3 "cat /my/output/file" # or:
while true; do cat /my/output/file; sleep 3; done
of course you can replace cat with something more useful, perhaps tail. Or you can just open the output files in a text editor which has an auto-revert facility (e.g. Emacs with M-x auto-revert-mode). This does exactly the same thing internally---poll the file for changes and update if need be.
I also really suggest you use logging inside your code, and give each paralleled function it's own logger (with the name derived from the parameters of the function). (This can be easier with a small class rather than a function). That way later on you can send all your output to a file, and if something goes wrong you can easily find out which run failed and extract information from only that run (with grep!). I use this approach in paralleled fuzzy matching code (actually for matching music libraries) and it's invaluable when you need to dig into how some strange result occurred.
*Okay, I'm sure there's some horrible way to control some particular terminal and output to it, but that's not what you meant.
For example, let's say I wanted to set a variable in powershell (or command line), and then get that variable later using something like subprocess.check_output("echo $var", shell=True) (this isn't something I need to do, just an example). Or, lets say I wanted to have a user 'cd' to a directory, and be able to run commands from that directory. I would be able to have some python variable that saves the current directory, and always run "cd {dir};{command}", but that would be inefficient, and still wouldn't work for every situation, I would need to add some special bit of code for every possible situation where a user could want to run a command, then run another command which depends on the first command.
Sorry if I phrased this question badly, let me know if I should clarify. TIA!
Ok, after some searching, I found a way to do this. Using a module called pexpect, MarkBaggett on GitHub has made a simple way to do this: pxpowershell. here's an example:
ps = pxpowershell()
ps.start_process()
ps.run("$x = 10")
print(ps.run("echo $x"))
ps.stop_process()
The only small problems are that 1. colors don't work, and 2. you need to decode() and strip() the output, though you can just add that into the pxpowershell.py.
I am trying to call an .exe file that's not in my local Python directory using subprocess.call(). The command (as I type it into cmd.exe) is exactly as follows: "C:\Program Files\R\R-2.15.2\bin\Rscript.exe" --vanilla C:\python\buyback_parse_guide.r
The script runs, does what I need to do, and I have confirmed the output is correct.
Here's my python code, which I thought would do the exact same thing:
## Set Rcmd
Rcmd = r'"C:\Program Files\R\R-2.15.2\bin\Rscript.exe"'
## Set Rargs
Rargs = r'--vanilla C:\python\buyback_parse_guide.r'
retval = subprocess.call([Rcmd,Rargs],shell=True)
When I call retval in my Python console, it returns 1 and the .R script doesn't run, but I get no errors. I'm pretty sure this is a really simple syntax error... help? Much thanks!
To quote the docs:
If shell is True, it is recommended to pass args as a string rather than as a sequence.
Splitting it up (either manually, or via shlex) just so subprocess can recombine them so the shell can split them again is silly.
I'm not sure why you think you need shell=True here. (If you don't have a good reason, you generally don't want it…) But even without shell=True:
On Windows, if args is a sequence, it will be converted to a string in a manner described in Converting an argument sequence to a string on Windows. This is because the underlying CreateProcess() operates on strings.
So, just give the shell the command line:
Rcmd = r'"C:\Program Files\R\R-2.15.2\bin\Rscript.exe" --vanilla C:\python\buyback_parse_guide.r'
retval = subprocess.call(Rcmd, shell=True)
According to the docs, Rscript:
… is an alternative front end for use in #! scripts and other scripting applications.
… is convenient for writing #! scripts… (The standard Windows command line has no concept of #! scripts, but Cygwin shells do.)
… is only supported on systems with the execv system call.
So, it is not the way to run R scripts from another program under Windows.
This answer says:
Rscript.exe is your friend for batch scripts… For everything else, there's R.exe
So, unless you have some good reason to be using Rscript outside of a batch script, you should switch to R.exe.
You may wonder why it works under cmd.exe, but not from Python. I don't know the answer to that, and I don't think it's worth digging through code or experimenting to find out, but I can make some guesses.
One possibility is that when you're running from the command line, that's a cmd.exe that controls a terminal, while when you're running from subprocess.call(shell=True) or os.system, that's a headless cmd.exe. Running a .bat/.cmd batch file gets you a non-headless cmd, but running cmd directly from another app does not. R has historically had all kinds of complexities dealing with the Windows terminal, which is why they used to have separate Rterm.exe and Rcmd.exe tools. Nowadays, those are both merged into R.exe, and it should work just fine either way. But if you try doing things the docs say not to do, that may not be tested, it's perfectly reasonable that it may not work.
At any rate, it doesn't really matter why it works in some situations even though it's not documented to. That certainly doesn't mean it should work in other situations it's not documented to work in, or that you should try to force it to do so. Just do the right thing and run R.exe instead of Rscript.exe.
Unless you have some information that contradicts everything I've found in the documentation and everywhere else I can find, I'm placing my money on Rscript.exe itself being the problem.
You'll have to read the documentation on the invocation differences between Rscript.exe and R.exe, but they're not identical. According to the intro docs,:
If you just want to run a file foo.R of R commands, the recommended way is to use R CMD BATCH foo.R
According to your comment above:
When I type "C:\R\R-2.15.2\bin\i386\R.exe" CMD BATCH C:\python\buyback_parse_guide.r into cmd.exe, the .R script runs successfully. What's the proper syntax for passing this into python?
That depends on the platform. On Windows, a list of arguments gets turned into a string, so you're better off just using a string so you don't have to debug the joining; on Unix, a string gets split into a list of arguments, so you're better off using a list so you don't have to debug the joining.
Since there are no spaces in the path, I'd take the quotes out.
So:
rcmd = r'C:\R\R-2.15.2\bin\i386\R.exe CMD BATCH C:\python\buyback_parse_guide.r'
retval = subprocess.call(rcmd)
Ok i have these commands used in batch and i wanted to know the commands in python that would have a similar affect, just to be clear i dont want to just use os.system("command here") for each of them. For example in batch if you wanted a list of commands you would type help but in python you would type help() and then modules... I am not trying to use batch in a python script, i just wanna know the similarities in both languages. Like in english you say " Hello" but in french you say "Bonjour" not mix the two languages. (heres the list of commands/functions id like to know:
change the current directory
clear the screen in the console
change the prompt to something other than >>>
how to make a loop function
redirections/pipes
start an exteral program (like notepad or paint) from within a script
how to call or import another python script
how to get help with a specific module without having to type help()
#8: (in batch it would be command /?)
EDITED COMPLETELY
Thanks in Adnvance!
You can't just mechanically translate batch script to Python and hope that it works. It's a different language, with different idioms and ways of doing things, not to mention the different purpose.
I've listed some functions related to what you want below, but there's no substitute for just going and learning Python!
os.chdir
os.system("cls") is probably the simplest solution
Change sys.ps1 and sys.ps2.
Nope, there are no gotos in Python. Use for and while loops instead.
Doesn't make sense, use Python's IO instead.
subprocess.Popen
Doesn't make sense, use import or subprocess.Popen instead.
help
Most of the things you've mentioned (start, cls etc.) are not "batch commands", they're executable programs which perform certain tasks. The DOS "shell" simply executes these when it encounters them in a file. In this sense, "python" is the equivalent of a single executable (like cls).
Now that that's clear, cd (and most other OS specific tasks) are accomplished using the os module. There's no single Python statement to clear the screen - that would be wasteful. Changing the prompt of the python interpreter can be done by assigning to sys.ps1. Loops are done using while or for. Redirection doesn't happen. YOu can however use the subprocess module to run subcommands and send their outputs to files or other streams. Starting commands is done using the subprocess.Popen function. For getting help, you can either do help("command") or if you're using ipython, just say command? and hit enter.
You should really go through the tutorial rather than trying to map batch commands to Python.
The Python docs are excellent, and are the place to start. For doing shell-script like things, you'll want to check out:
http://docs.python.org/library/os.html?module-os
http://docs.python.org/library/os.path.html#module-os.path
http://docs.python.org/library/shutil.html#module-shutil
http://docs.python.org/library/subprocess.html#module-subprocess
Python is not a system shell, Python is a multi-paradigm programming language.
If you want to compare .bat with anything, compare it with sh or bash. (You can have those on various platforms too - for example, sh for windows is in the MinGW package).
I am pretty much facing the same problem as you, daniel11. As a solution, I am learning BATCH commands and their meaning. After I understand those, I am going to write a program in Python that does the same or accomplishes the same task.
Thanks to Adam V. and katrielatex for their insight and suggestions.
Spinning off from another thread, when is it appropriate to use os.system() to issue commands like rm -rf, cd, make, xterm, ls ?
Considering there are analog versions of the above commands (except make and xterm), I'm assuming it's safer to use these built-in python commands instead of using os.system()
Any thoughts? I'd love to hear them.
Rule of thumb: if there's a built-in Python function to achieve this functionality use this function. Why? It makes your code portable across different systems, more secure and probably faster as there will be no need to spawn an additional process.
One of the problems with system() is that it implies knowledge of the shell's syntax and language for parsing and executing your command line. This creates potential for a bug where you didn't validate input properly, and the shell might interpet something like variable substitution or determining where an argument begins or ends in a way you don't expect. Also, another OS's shell might have divergent syntax from your own, including very subtle divergence that you won't notice right away. For reasons like these I prefer to use execve() instead of system() -- you can pass argv tokens directly and not have to worry about something in the middle (mis-)parsing your input.
Another problem with system() (this also applies to using execve()) is that when you code that, you are saying, "look for this program, and pass it these args". This makes a couple of assumptions which may lead to bugs. First is that the program exists and can be found in $PATH. Maybe on some system it won't. Second, maybe on some system, or even a future version of your own OS, it will support a different set of options. In this sense, I would avoid doing this unless you are absolutely certain the system you will run on will have the program. (Like maybe you put the callee program on the system to begin with, or the way you invoke it is mandated by something like POSIX.)
Lastly... There's also a performance hit associated with looking for the right program, creating a new process, loading the program, etc. If you are doing something simple like a mv, it's much more efficient to use the system call directly.
These are just a few of the reasons to avoid system(). Surely there are more.
Darin's answer is a good start.
Beyond that, it's a matter of how portable you plan to be. If your program is only ever going to run on a reasonably "standard" and "modern" Linux then there's no reason for you to re-invent the wheel; if you tried to re-write make or xterm they'd be sending the men in the white coats for you. If it works and you don't have platform concerns, knock yourself out and simply use Python as glue!
If compatibility across unknown systems was a big deal you could try looking for libraries to do what you need done in a platform independent way. Or you need to look into a way to call on-board utilities with different names, paths and mechanisms depending on which kind of system you're on.
The only time that os.system might be appropriate is for a quick-and-dirty solution for a non-production script or some kind of testing. Otherwise, it is best to use built-in functions.
Your question seems to have two parts. You mention calling commands like "xterm", "rm -rf", and "cd".
Side Note: you cannot call 'cd' in a sub-shell. I bet that was a trick question ...
As far as other command-level things you might want to do, like "rm -rf SOMETHING", there is already a python equivalent. This answers the first part of your question. But I suspect you are really asking about the second part.
The second part of your question can be rephrased as "should I use system() or something like the subprocess module?".
I have a simple answer for you: just say NO to using "system()", except for prototyping.
It's fine for verifying that something works, or for that "quick and dirty" script, but there are just too many problems with os.system():
It forks a shell for you -- fine if you need one
It expands wild cards for you -- fine unless you don't have any
It handles redirect -- fine if you want that
It dumps output to stderr/stdout and reads from stdin by default
It tries to understand quoting, but it doesn't do very well (try 'Cmd" > "Ofile')
Related to #5, it doesn't always grok argument boundaries (i.e. arguments with spaces in them might get screwed up)
Just say no to "system()"!
I would suggest that you only use use os.system for things that there are not already equivalents for within the os module. Why make your life harder?
The os.system call is starting to be 'frowned upon' in python. The 'new' replacement would be subprocess.call or subprocess.Popen in the subprocess module. Check the docs for subprocess
The other nice thing about subprocess is you can read the stdout and stderr into variables, and process that without having to redirect to other file(s).
Like others have said above, there are modules for most things. Unless you're trying to glue together many other commands, I'd stick with the things included in the library. If you're copying files, use shutil, working with archives you've got modules like tarfile/zipfile and so on.
Good luck.