I have a Python 3 file. I want to use an open-source tool on the internet (nltk), but unfortunately it only supports Python 2. There is no way for me to convert it to Python 3, nor can I convert my Python 3 file to Python 2.
If the user does not give a certain argument (on argparse) then I do something in my file. If the user does give a certain argument, however, I need to use nltk.
Writing a Python 2 script that uses nltk and then executing script that in my Python 3 script
My current idea is to write a script in Python 2 that does what I want with nltk and then run that from my current Python 3 script. However, I don't actually know how to do this.
I found this code: os.system(command) and so I will modify it to be os.system("python py2.py") (where py2.py is my newly written Python 2 file).
I'm not sure if that will work.
I also don't know if that is the most efficient way to solve my problem. I cannot find any information about it on the internet.
The data transferred will probably be quite large. Currently, my test data is about 6600 lines, utf-8. Functionality is more important than how long it takes (to a certain extent) in my case.
Also, how would I pass values from my Python 2 script to my Python 3 script?
Thanks
Is there any other way to do this?
Well, if you're sure you can't convert your script to Python 2, then having one script call the other by running the Python interpreter probably is the best way. (And, this being Python, the best way is, or at least should be, the only way.)
But are you sure? Between the six module, the 3to2 tool, and __future__ statements, it may not be as hard as you think.
Anyway, if you do need to have one script call the other, you should almost never use os.system. As the docs for that function say:
The subprocess module provides more powerful facilities for spawning new processes and retrieving their results; using that module is preferable to using this function. See the Replacing Older Functions with the subprocess Module section in the subprocess documentation for some helpful recipes.
The simplest version is this:
subprocess.check_call(["python", "py2.py"])
This runs your script, waits for it to finish, and raises an exception if the script returns failure—basically, what you wanted to do with os.system, but better. (For example, it doesn't spawn an unnecessary extra shell, it takes care of error handling, etc.)
That assumes whatever other data you need to share is being shared in some implicit, external way (e.g., by accessing files with the same name). You might be better off passing data to py2.py as command-line arguments and/or stdin, passing data back as via stdout, or even opening an explicit pipe or socket to pass things over. Without knowing more about exactly what you need to do, it's hard to suggest anything, but the docs, especially the section Replacing Older Functions with the subprocess Module have lots of discussion on the options.
To give you an idea, here's a simple example: to pass one of your filename arguments to py2.py, and then get data back from py2.py to py3.py, just have py3.py do this:
py2output = subprocess.check_output(["python", "py2.py", my_args[0]])
And then in py2.py, just print whatever you want to send back.
The Anyone hear when NLTK 3.0 will be out? here in SO points out that...
There's a Python 3 branch:
https://github.com/nltk/nltk/tree/nltk-py3k
The answer is from July 2011. It could be improved since then.
I have just looked at https://github.com/nltk/nltk. There is at least the document that talks about Python 3 port related things https://github.com/nltk/nltk/blob/2and3/web/dev/python3porting.rst.
Here is a longer discussion on NLTK and Python 3 that you may be interested in.
And the Grants to Assist Kivy, NLTK in Porting to Python 3 (published 3 days ago) is directly related to the problem.
Related
How can I call a python function from an advanced scripting voice command in Dragon NaturallySpeaking?
I don't want to use a third-party application such as dragonfly or NatLink (paper).
So, one way is to compile it. You can put a bunch of functions that do different things all into the same program and pass along appropriate arguments to select the function you want, and pass the parameters along. Returning the result can be tricky, though, but I usually use the Clipboard (so copy the py output to clip and read from clip in Dragon). Multi-word params need to have spaces escaped (%20) and process it inside your py.
Something like this:
ShellExecute "path\program.exe myFunc myPar1, my%20Par%202", 6 ' 6 runs minimized
Wait 1
myVar = Clipboard
Hth,
Warning: This is not an answer. I am not a programmer. I don't know any Python and have no way of testing it.
This is just a suggestion on how to solve this problem. I don't know where else to put this. I'd put it in a comment, but it allows no screenshots. Please edit and suggest as you wish.
There is answer on SO that deals with calling Python from Excel, which is a similar concept: https://stackoverflow.com/a/3569988/2101890. I am trying to use that here but don't know how.
When using commands in another programming language, you can sometimes add them by adding a reference in the MyCommands Editor. You can reference DLLs and other "stuff". Some references to libraries appear automatically. I've installed Python and hoped to find Python in the References, but no such luck:
There is no Python entry here that I can find. You may have better luck. If you do find something, check the box and see if you can add python commands without causing an error when saving the command.
Maybe you can browse to %localappdata%\Programs\Python\Python36\ and add some of the DLLs from there and call Python commands from there. Or try getting it to work in the way described under 1.
I have a question regarding python bindings.
I have a command-line which exposes some functionality and code is re-factored to provide the functionality through a shared library. I wanted to know what the real advantage that I get from "writing a python binding for the shared library" vs "calling the command line directly".
One obvious advantage I think will be performance, the shared library will link to the same process and the functionality can called within the same process. It will avoid spawning a new process through the command line.
Any other advantages I can get from writing a python binding for such a case ?
Thanks.
I can hardly imagine a case where one would prefer wrapping a library's command line interface over wrapping the library itself. (Unless there is a library that comes with a neat command line interface while being a total mess internally; but the OP indicates that the same functionality available via the command line is easily accessible in terms of library function calls).
The biggest advantage of writing a Python binding is a clearly defined data interface between the library and Python. Ideally, the library can operate directly on memory managed by Python, without any data copying involved.
To illustrate this, let's assume a library function does something more complicated than printing the current time, i.e., it obtains a significant amount of data as an input, performs some operation, and returns a significant amount of data as an output. If the input data is expected as an input file, Python would need to generate this file first. It must make sure that the OS has finished writing the file before calling the library via the command line (I have seen several C libraries where sleep(1) calls were used as a band-aid for this issue...). And Python must get the output back in some way.
If the command line interface does not rely on files but obtains all arguments on the command line and prints the output on stdout, Python probably needs to convert between binary data and string format, not always with the expected results. It also needs to pipe stdout back and parse it. Not a problem, but getting all this right is a lot of work.
What about error handling? Well, the command line interface will probably handle errors by printing error messages on stderr. So Python needs to capture, parse and process these as well. OTOH, the corresponding library function will almost certainly make a success flag accessible to the calling program. This is much more directly usable for Python.
All of this is obviously affecting performance, which you already mentioned.
As another point, if you are developing the library yourself, you will probably find after some time that the Python workflow has made the whole command line interface obsolete, so you can drop supporting it altogether and save yourself a lot of time.
So I think there is a clear case to be made for the Python bindings. To me, one of the biggest strengths of Python is the ease with which such wrappers can be created and maintained. Unfortunately, there are about 7 or 8 equally easy ways to do this. To get started, I recommend ctypes, since it does not require a compiler and will work with PyPy. For best performance use the native C-Python API, which I also found very easy to learn.
Ok i have these commands used in batch and i wanted to know the commands in python that would have a similar affect, just to be clear i dont want to just use os.system("command here") for each of them. For example in batch if you wanted a list of commands you would type help but in python you would type help() and then modules... I am not trying to use batch in a python script, i just wanna know the similarities in both languages. Like in english you say " Hello" but in french you say "Bonjour" not mix the two languages. (heres the list of commands/functions id like to know:
change the current directory
clear the screen in the console
change the prompt to something other than >>>
how to make a loop function
redirections/pipes
start an exteral program (like notepad or paint) from within a script
how to call or import another python script
how to get help with a specific module without having to type help()
#8: (in batch it would be command /?)
EDITED COMPLETELY
Thanks in Adnvance!
You can't just mechanically translate batch script to Python and hope that it works. It's a different language, with different idioms and ways of doing things, not to mention the different purpose.
I've listed some functions related to what you want below, but there's no substitute for just going and learning Python!
os.chdir
os.system("cls") is probably the simplest solution
Change sys.ps1 and sys.ps2.
Nope, there are no gotos in Python. Use for and while loops instead.
Doesn't make sense, use Python's IO instead.
subprocess.Popen
Doesn't make sense, use import or subprocess.Popen instead.
help
Most of the things you've mentioned (start, cls etc.) are not "batch commands", they're executable programs which perform certain tasks. The DOS "shell" simply executes these when it encounters them in a file. In this sense, "python" is the equivalent of a single executable (like cls).
Now that that's clear, cd (and most other OS specific tasks) are accomplished using the os module. There's no single Python statement to clear the screen - that would be wasteful. Changing the prompt of the python interpreter can be done by assigning to sys.ps1. Loops are done using while or for. Redirection doesn't happen. YOu can however use the subprocess module to run subcommands and send their outputs to files or other streams. Starting commands is done using the subprocess.Popen function. For getting help, you can either do help("command") or if you're using ipython, just say command? and hit enter.
You should really go through the tutorial rather than trying to map batch commands to Python.
The Python docs are excellent, and are the place to start. For doing shell-script like things, you'll want to check out:
http://docs.python.org/library/os.html?module-os
http://docs.python.org/library/os.path.html#module-os.path
http://docs.python.org/library/shutil.html#module-shutil
http://docs.python.org/library/subprocess.html#module-subprocess
Python is not a system shell, Python is a multi-paradigm programming language.
If you want to compare .bat with anything, compare it with sh or bash. (You can have those on various platforms too - for example, sh for windows is in the MinGW package).
I am pretty much facing the same problem as you, daniel11. As a solution, I am learning BATCH commands and their meaning. After I understand those, I am going to write a program in Python that does the same or accomplishes the same task.
Thanks to Adam V. and katrielatex for their insight and suggestions.
Spinning off from another thread, when is it appropriate to use os.system() to issue commands like rm -rf, cd, make, xterm, ls ?
Considering there are analog versions of the above commands (except make and xterm), I'm assuming it's safer to use these built-in python commands instead of using os.system()
Any thoughts? I'd love to hear them.
Rule of thumb: if there's a built-in Python function to achieve this functionality use this function. Why? It makes your code portable across different systems, more secure and probably faster as there will be no need to spawn an additional process.
One of the problems with system() is that it implies knowledge of the shell's syntax and language for parsing and executing your command line. This creates potential for a bug where you didn't validate input properly, and the shell might interpet something like variable substitution or determining where an argument begins or ends in a way you don't expect. Also, another OS's shell might have divergent syntax from your own, including very subtle divergence that you won't notice right away. For reasons like these I prefer to use execve() instead of system() -- you can pass argv tokens directly and not have to worry about something in the middle (mis-)parsing your input.
Another problem with system() (this also applies to using execve()) is that when you code that, you are saying, "look for this program, and pass it these args". This makes a couple of assumptions which may lead to bugs. First is that the program exists and can be found in $PATH. Maybe on some system it won't. Second, maybe on some system, or even a future version of your own OS, it will support a different set of options. In this sense, I would avoid doing this unless you are absolutely certain the system you will run on will have the program. (Like maybe you put the callee program on the system to begin with, or the way you invoke it is mandated by something like POSIX.)
Lastly... There's also a performance hit associated with looking for the right program, creating a new process, loading the program, etc. If you are doing something simple like a mv, it's much more efficient to use the system call directly.
These are just a few of the reasons to avoid system(). Surely there are more.
Darin's answer is a good start.
Beyond that, it's a matter of how portable you plan to be. If your program is only ever going to run on a reasonably "standard" and "modern" Linux then there's no reason for you to re-invent the wheel; if you tried to re-write make or xterm they'd be sending the men in the white coats for you. If it works and you don't have platform concerns, knock yourself out and simply use Python as glue!
If compatibility across unknown systems was a big deal you could try looking for libraries to do what you need done in a platform independent way. Or you need to look into a way to call on-board utilities with different names, paths and mechanisms depending on which kind of system you're on.
The only time that os.system might be appropriate is for a quick-and-dirty solution for a non-production script or some kind of testing. Otherwise, it is best to use built-in functions.
Your question seems to have two parts. You mention calling commands like "xterm", "rm -rf", and "cd".
Side Note: you cannot call 'cd' in a sub-shell. I bet that was a trick question ...
As far as other command-level things you might want to do, like "rm -rf SOMETHING", there is already a python equivalent. This answers the first part of your question. But I suspect you are really asking about the second part.
The second part of your question can be rephrased as "should I use system() or something like the subprocess module?".
I have a simple answer for you: just say NO to using "system()", except for prototyping.
It's fine for verifying that something works, or for that "quick and dirty" script, but there are just too many problems with os.system():
It forks a shell for you -- fine if you need one
It expands wild cards for you -- fine unless you don't have any
It handles redirect -- fine if you want that
It dumps output to stderr/stdout and reads from stdin by default
It tries to understand quoting, but it doesn't do very well (try 'Cmd" > "Ofile')
Related to #5, it doesn't always grok argument boundaries (i.e. arguments with spaces in them might get screwed up)
Just say no to "system()"!
I would suggest that you only use use os.system for things that there are not already equivalents for within the os module. Why make your life harder?
The os.system call is starting to be 'frowned upon' in python. The 'new' replacement would be subprocess.call or subprocess.Popen in the subprocess module. Check the docs for subprocess
The other nice thing about subprocess is you can read the stdout and stderr into variables, and process that without having to redirect to other file(s).
Like others have said above, there are modules for most things. Unless you're trying to glue together many other commands, I'd stick with the things included in the library. If you're copying files, use shutil, working with archives you've got modules like tarfile/zipfile and so on.
Good luck.
I need to be able to list the command line arguments (if any) passed to other running processes. I have the PIDs already of the running processes on the system, so basically I need to determine the arguments passed to process with given PID XXX.
I'm working on a core piece of a Python module for managing processes. The code is written as a Python extension in C and will be wrapped by a higher level Python library. The goal of this project is to avoid dependency on third party libs such as the pywin32 extensions, or on ugly hacks like calling 'ps' or taskkill on the command line, so I'm looking for a way to do this in C code.
I've Googled this around and found some brief suggestions of using CreateRemoteThread() to inject myself into the other process, then run GetCommandLine() but I was hoping someone might have some working code samples and/or better suggestions.
UPDATE: I've found full working demo code and a solution using NtQueryProcessInformation on CodeProject: http://www.codeproject.com/KB/threads/GetNtProcessInfo.aspx - It's not ideal since it's "unsupported" to cull the information directly from the NTDLL structures but I'll live with it. Thanks to all for the suggestions.
UPDATE 2: I managed through more Googling to dig up a C version that does not use C++ code, and is a little more direct/concisely pointed toward this problem. See http://wj32.wordpress.com/2009/01/24/howto-get-the-command-line-of-processes/ for details.
Thanks!
To answer my own question, I finally found a CodeProject solution that does exactly what I'm looking for:
http://www.codeproject.com/KB/threads/GetNtProcessInfo.aspx
As #Reuben already pointed out, you can use NtQueryProcessInformation to retrieve this information. Unfortuantely it's not a recommended approach, but given the only other solution seems to be to incur the overhead of a WMI query, I think we'll take this approach for now.
Note that this seems to not work if using code compiled from 32bit Windows on a 64bit Windows OS, but since our modules are compiled from source on the target that should be OK for our purposes. I'd rather use this existing code and should it break in Windows 7 or a later date, we can look again at using WMI. Thanks for the responses!
UPDATE: A more concise and C only (as opposed to C++) version of the same technique is illustrated here:
http://wj32.wordpress.com/2009/01/24/howto-get-the-command-line-of-processes/
The cached solution:
http://74.125.45.132/search?q=cache:-wPkE2PbsGwJ:windowsxp.mvps.org/listproc.htm+running+process+command+line&hl=es&ct=clnk&cd=1&gl=ar&client=firefox-a
in CMD
WMIC /OUTPUT:C:\ProcessList.txt PROCESS get Caption,Commandline,Processid
or
WMIC /OUTPUT:C:\ProcessList.txt path win32_process get Caption,Processid,Commandline
Also:
http://mail.python.org/pipermail/python-win32/2007-December/006498.html
http://tgolden.sc.sabren.com/python/wmi_cookbook.html#running_processes
seems to do the trick:
import wmi
c = wmi.WMI ()
for process in c.Win32_Process ():
print process.CommandLine
By using psutil ( https://github.com/giampaolo/psutil ):
>>> import psutil, os
>>> psutil.Process(os.getpid()).cmdline()
['C:\\Python26\\python.exe', '-O']
>>>
The WMI approach mentioned in another response is probably the most reliable way of doing this. Looking through MSDN, I spotted what looks like another possible approach; it's documented, but its not clear whether it's fully supported. In MSDN's language, it--
may be altered or unavailable in
future versions of Windows...
In any case, provided that your process has the right permissions, you should be able to call NtQueryProcessInformation with a ProcessInformationClass of ProcessBasicInformation. In the returned PROCESS_BASIC_INFORMATION structure, you should get back a pointer to the target process's process execution block (as field PebBaseAddress). The ProcessParameters field of the PEB will give you a pointer to an RTL_USER_PROCESS_PARAMETERS structure. The CommandLine field of that structure will be a UNICODE_STRING structure. (Be careful not too make too many assumptions about the string; there are no guarantees that it will be NULL-terminated, and it's not clear whether or not you'll need to strip off the name of the executed application from the beginning of the command line.)
I haven't tried this approach--and as I mentioned above, it seems a bit... iffy (read: non-portable)--but it might be worth a try. Best of luck...
If you aren't the parent of these processes, then this is not possible using documented functions :( Now, if you're the parent, you can do your CreateRemoteThread trick, but otherwise you will almost certainly get Access Denied unless your app has admin rights.