I've got a very large Lisp project whose output I'd like to programmatically pipe to a Python program, i.e. use Python to call the Lisp program on some input and get the output back into Python.
The project only compiles in Clozure Common Lisp (ccl64) and I did try to find a way to turn it into an executable (I'm using Mac OS X), but that ran into a lot of dead ends (I am not a Lisp programmer).
This documentation for Clozure Common Lisp should provide the solution to the above, but I was not able to understand it. The examples I made created a file, but Terminal would not run them as executables.
How to create executable for ccl64
I tried to follow this question's answer Compiling Common Lisp to an executable except using ccl64's save application function.
$ ccl64
Welcome to Clozure Common Lisp Version 1.9-dev-r15612M-trunk (DarwinX8664)!
? (in-package :ccl)
#<Package "CCL">
? (defun main () (print "hello"))
MAIN
? (save-application "hello" :toplevel-function #'main)
I am trying to use Python's subprocess to invoke ccl64, run the Lisp program, and get the output. However, subprocess for some reason refuses to run the ccl64 command. Here is what I wrote so far:
import subprocess
process = subprocess.Popen(['ccl64', '-h'], stdout=subprocess.PIPE)
out, err = process.communicate()
The variable out should contain the output of getting the usage/help from ccl64. Instead I get an error:
Traceback (most recent call last):
File "sub.py", line 3, in <module>
process = subprocess.Popen(['ccl64', '-h'], stdout=subprocess.PIPE)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1249, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
How can I get Python to invoke ccl64 and get output from the Lisp project?
The error in your Python code is clear: No such file or directory.
You need to tell in your Python code which application you want to run in a way that it actually finds it.
It's also not clear why you save a Lisp executable somewhere named hello, but you are not trying to call it. With the necessary path. Your code tries to call Clozure CL - without the necessary path - but why? You just saved an executable. Why would you call Clozure CL to run it? I would also save the executable with prepending the kernel - that makes it self-contained.
Example:
Calling Clozure CL:
rjmba:~ joswig$ ccl
Welcome to Clozure Common Lisp Version 1.9-dev-r15612M-trunk (DarwinX8664)!
Defining the main function:
? (defun main () (print "hello"))
MAIN
Saving an executable:
? (save-application "hello" :toplevel-function #'main :prepend-kernel t)
Running the new executable from the same directory:
rjmba:~ joswig$ ./hello
"hello"
Calling a Clozure CL application with an argument:
bash-3.2$ ccl
Welcome to Clozure Common Lisp Version 1.9-dev-r15612M-trunk (DarwinX8664)!
The function ccl::command-line-arguments returns the arguments as a list. The first item is the called application itself.
? (defun main ()
(print (second (ccl::command-line-arguments))))
MAIN
? (save-application "hello"
:toplevel-function #'main
:prepend-kernel t)
Calling it:
bash-3.2$ ./hello hello!
"hello!"
Related
I'm using check_output to do all my SSH and GitHub setup, and I'm attempting to execute eval $(ssh-agent), both to start the agent and to parse the output if I need the process id.
from subprocess import check_output
out = check_output(["eval", "$(ssh-agent)"])
print(out)
But regardless of how I escape things, I get the same error.
Traceback (most recent call last):
File "gitSetup.py", line 3, in <module>
out = check_output(["eval", "$(ssh-agent)"])
File "/usr/lib/python2.7/subprocess.py", line 216, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "/usr/lib/python2.7/subprocess.py", line 394, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1047, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
I'm wondering if I'm not escaping or trimming my arguments properly; if anyone sees my error, I'd appreciate some insight!
Even if you managed to fix the superficial syntax problems, a subprocess which runs successfully simply then terminates without a trace; it doesn't attempt to (and couldn't, even if it wanted to) modify the environment of the parent Python process. This is a common FAQ.
You could run the eval in the parent process which starts your Python script, or perhaps figure out how to communicate with ssh-agent directly from Python. Its output is usually a simple sequence of variable assignments, which you can parse yourself.
from subprocess import check_output
from os import environ
eval_string = check_output(['ssh-agent'])
for line in eval_string.rstrip('\n').split('\n'):
for expr in line.rstrip(';').split(';'):
if expr.startswith((' export ', 'echo ')):
continue
var, value = expr.strip().split('=', 1)
environ[var] = value
If the output from ssh-agent contains shell quoting, you will need to perform additional parsing on it (basically, trim the quotes around the value string). But this is already rather clunky and brittle, so perhaps revert to setting up the environment before launching Python instead of trying to splash some sophistication onto this kludge.
In more detail, ssh-agent and a precious few other shell utilities have a very specific design in order for them to be able to communicate with their parent process. Precisely because a subprocess cannot make any changes in the environment of its parent process, it instead prints a piece of code for its parent process to execute. By default, it prints sh code like this:
SSH_AUTH_SOCK=/tmp/ssh-MUyniqn10506/agent.10506; export SSH_AUTH_SOCK;
SSH_AGENT_PID=10507; export SSH_AGENT_PID;
echo Agent pid 10507;
There is also an option for it to print similar code in csh syntax (this is a historical shell which thankfully isn't used much any longer) but, alas, no option for producing Python code. (It would not be hard to make ssh-agent do that, per se.)
(The above output sample copy/pasted from http://blog.joncairns.com/2013/12/understanding-ssh-agent-and-ssh-add/ which contains a fuller explanation.)
This question already has answers here:
python subprocess.call() "no such file or directory"
(2 answers)
Closed 4 years ago.
I'm attempting to run an old script which takes an .mdb file and converts it into a MySQL database. However, I'm running into an issue where I receive the following error.
File "/usr/lib64/python2.7/subprocess.py", line 568, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
errread, errwrite)
File "/usr/lib64/python2.7/subprocess.py", line 1327, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
I have attempted to research this error, but all I found was fixes where the item was not correctly formatted as a list. So, I've modified the affected code where subprocess.py is called, and changed it to the following:
def get_external_command_output(command):
args = shlex.split(command)
print "################"
print type(args), args
ret = subprocess.check_output(args) # this needs Python 2.7 or higher
return ret
Which in turn returns the same error, but also the below message is printed:
<type 'list'> ['mdb-tables', '-1', '/mnt/data/General/Genit/WIP/IL_Orders.mdb']
So I can safely say that the arguments are correctly formatted as a list. I'm really unsure how else to tackle this issue, because all the help forums I've found suggest the same issue. The argument isn't a list. However, I can see that mine is.
Could anybody provide some guidance here?
Many thanks.
There can be number of reasons for this. The error comes from your underlying OS (ENOENT).
The easiest to would be to try running the same thing:
mdb-tables -1 /mnt/data/General/Genit/WIP/IL_Orders.mdb
(or just mdb-tables really) and see what happens, what complain you get from your shell.
Among possible reasons would be:
As mentioned by Peter Wood in the comments. If you just pass an executable name to run, it must be located in one of the directories listed in your PATH environmental variable. Or, you have to use either path to the executable which is either absolute or relative to current working directory (based on where you ran the parent from or cwd argument passed to subprocess.check_output call). E.g. ./mdb-tables if in the same directory where you were when you ran the parent script or to which cwd was set.
It is also possible. Esp. if it's an older script. That you have specified an interpreter that no longer exists on your host. In that case the same error (ENOENT) would be raised. You might see a slightly different error message (hinting at bad interpreter) if executed directly from shell, but it would look identical when called as subprocess from python.
You've hinted mdb-tables is also a script. But otherwise previous paragraph would also hold true for dynamically linked ELF binaries. If they were built with and expected a version (path) of dynamic linker (their interpreter) no longer available on the system. You can print path of the expected dynamic linker binary for instance by running objdump -sj .interp BINARY_IN_QUESTION.
When I run emconfigure ./configure I get the following error:
ERROR:root:Exception thrown when invoking Popen in configure with
args: "./configure"!
Traceback (most recent call last):
File "/usr/local/bin/emconfigure", line 13, in <module>
emconfigure.run()
File "/usr/local/Cellar/emscripten/1.37.21/libexec/emconfigure.py", line 46, in run
shared.Building.configure(sys.argv[1:])
File "/usr/local/Cellar/emscripten/1.37.21/libexec/tools/shared.py", line 1533, in configure
process = Popen(args, stdout=None if EM_BUILD_VERBOSE_LEVEL >= 2 else stdout, stderr=None if EM_BUILD_VERBOSE_LEVEL >= 1 else stderr, env=env)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1335, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
Break-down:
This one can be tricky to track down because of the depth of the call stack. Let's break it apart:
At the "next-to-top level": a file you're trying to execute cannot be found.
One level lower than that things become clear: subprocess.Popen is trying to execute args.
What's in args? - basically ./configure (as passed by emscripten). export the EM_BUILD_VERBOSE=3 environmental variable and then stderr should display this.
So, the next-to-top level ./configure is directing things in an invalid fashion. Now it's time for a guess at the top level (not knowing what's in the ./configure script): is some line of the shell command in the ./configure script file itself trying to execute something non-existent? E.g.: some point after #! /bin/sh ? - try debugging just ./compile.
It is worth noting that a subprocess can have issues with relative paths (not getting PATH), and permissions (./configure is locked-down), but your error very well may persist even after /you/make/an/absolute/call/to/configure and 777 the permissions of configure. - In that case the previous paragraph is the likeliest culprit.
Editorializing:
The reason this error is interesting as a general one for emscripten is that part (much) of emscripten's raison d'etre is to port [legacy] C/++ code to new JS apps. Well, old C/++ builds get old in the sense of this question. If that's true, this is will be a generally-occurring problem, and, if it is general the community could use a general pattern for resolution.
If you're using Docker, your host computer is Windows, and you mounted it to a Windows directory, make sure that your files aren't \r\n line endings but instead \n line endings. Changing this fixed it for me.
In this case, if you're using git, you might try setting git config core.autocrlf input to stop checking out to \r\n. If you need to convert a file, try this answer out.
I am trying to generate a set of files through a python script on a 48-core cluster which actually has a master-and 3 slave machines. I need to generate a set of files, run some scripts on them, collect results and then delete them. I again repeat the process- regenerate files, execute, delete etc.,
When I delete and regenerate files with the same name, I see that the slave machine complains that it cannot find the files.
I am running python script through os.system()
I learnt from this post that it is better to use subprocess.Popen() rather than os.system so that it actually waits for my script to generate my files, before proceeding with the execution. I could use os.system("pause") or time.sleep(whatever) for waiting, but I want to convert my os.systems to subprocess.popens or subprocess.calls and I am stuck here.
I ran through the python documentation and tried out subprocess.Popen('ls'), but I am not able to get a simple thing like subprocess.Popen('cd /whatever_directory') working.
It might sound silly but how do I execute such a simple command like cd through subprocess rather than os.system('cd')?
Then, I actually want to convert the following into subprocess. How do I do it?
import os,optparse
from optparse import OptionParser
parser.add_option("-m", "--mod",dest='module', help='Enter the entity name')
parser.add_option("-f", "--folder", dest="path",help="Enter the path")
module=options.module
path=options.path
os.system('python %s/python_repeat_deckgen_remote.py -m %s' %(path,module))
I just replaced os.system with subprocess.Popen.
But it gave me a lot of complaints:
File "/usr/lib64/python2.6/subprocess.py", line 633, in __init__
errread, errwrite)
File "/usr/lib64/python2.6/subprocess.py", line 1139, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
As NPE already noted, new processes don't affect the existing one (which means os.system('cd /some/where') also has no effect on the current process). In this case, though, I think you're tripping over the fact that os.system invokes a shell to interpret the command you pass in, while subprocess.Popen does not do so by default. But you can tell it to do so:
proc = subprocess.Popen(
'python %s/python_repeat_deckgen_remote.py -m %s' % (path, module),
shell = True)
status = proc.wait()
If you're invoking a shell built-in command or using shell expansions, it's necessary to invoke the shell (assuming you're not willing to simulate it):
>>> import subprocess
>>> x = subprocess.Popen('echo $$')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/usr/local/lib/python2.7/subprocess.py", line 1249, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
>>> x = subprocess.Popen('echo $$', shell = True); x.wait()
81628
0
>>>
but you can bypass the shell—which can help with security issues—if your problem permits, by breaking up the command arguments into a list:
>>> x = subprocess.Popen(['echo', '$$']); x.wait()
$$
0
>>>
Note that this time the output here is the string $$, rather than a process ID, since this time the shell did not interpret the string.
For your original example, for instance, you might use:
proc = subprocess.Popen(['python',
os.path.join(path, 'python_repeat_deckgen_remote.py'),
'-m',
module])
which avoids issues if path and/or module contain characters special to the shell.
(You might even want to call subprocess.Popen with cwd = path and eliminate the call to os.path.join, although that depends on other things.)
I am not able to get a simple thing like subprocess.Popen('cd /whatever_directory') working.
Each process's current working directory is independent of other processes in the system.
When you start a subprocess (using either of these mechanisms), and get it to cd into another directory, that has no effect on the parent process, i.e. on your Python script.
To change the current working directory of your script, you should use os.chdir().
I might be not answering you question directly, but for such tasks which require running subprocess, I always use plumbum.
IMO, It makes the task much simpler and more intuitive, including running on remote machines.
Using plumbum, in order to set subprocess's working directory, you can run the command in a with local.cwd(path): my_cmd() context.
I'm using emacs23 with tramp to modify python scripts on a remote host.
I found that when I start the python shell within emacs it starts up
python on the remote host.
My problem is that when I then try to call python-send-buffer via C-c C-c it comes up with the error
Traceback (most recent call last):
File "", line 1, in ?
ImportError: No module named emacs
Traceback (most recent call last):
File "", line 1, in ?
NameError: name 'emacs' is not defined
Now, I must admit that I don't really know what's going on here. Is there a way for me to configure emacs so that I can evaluate the buffer on the remote host?
Many thanks.
Edit: I've followed eichin's advice and re-implemented python-send-region. See my answer below.
I'm currently trying to to get my unregistered question merged with this account, after which I'll be able to accept eichin's answer and edit my post to include my solution.
I followed eichin's suggestion and copied the emacs2.py emacs3.py and emacs.py files to the remote host and added their directory to PYTHONPATH in the tramp-remote-process-environment variable.
I then reimplemented the python-send-buffer function in my .emacs
(require 'python)
(defun python-send-region (start end)
"Send the region to the inferior Python process."
(interactive "r")
(let* ((loc_name)
(f (if (file-remote-p default-directory)
(let* ((con (tramp-dissect-file-name default-directory)))
(setq loc_name (tramp-make-tramp-temp-file con))
(concat "/"
(tramp-file-name-method con) ":"
(tramp-file-name-user con) "#"
(tramp-file-name-host con) ":"
loc_name
))
(setq loc_name (make-temp-file "py"))))
(command (format "emacs.eexecfile(%S)" loc_name))
(orig-start (copy-marker start)))
(save-excursion
(let ((curbuf (current-buffer))
(tempbuf (get-buffer-create "*python_temp*")))
(set-buffer tempbuf)
(delete-region (point-min) (point-max))
(insert-buffer-substring curbuf start end)
(python-mode)
(when (save-excursion
(goto-char (point-min))
(/= 0 (current-indentation)))
(python-shift-left (point-min) (point-max)))
(write-region nil nil f nil 'nomsg))
(python-send-command command)
(with-current-buffer (process-buffer (python-proc))
;; Tell compile.el to redirect error locations in file `f' to
;; positions past marker `orig-start'. It has to be done *after*
;; `python-send-command''s call to `compilation-forget-errors'.
(compilation-fake-loc orig-start f)))
))
I essentially copy the region into a new buffer, adjust the indentation and then write it into a temporary file, created with tramp-make-tramp-temp-file or make-temp-file, depending on whether the visited file is remote or local.
I had some problems with tramp-handle-write-region, which didn't seem to accept a string as a first argument, which is why I did all the formatting in a separate buffer first.
Let me know if there are still any problems with the code, but this is my first attempt at elisp coding, so please be gentle.
Short answer: not without writing some missing elisp code.
Long version: In python.el, run-python adds data-directory (which on my Ubuntu 10.10 box is /usr/share/emacs/23.1/etc/ ) to $PYTHONPATH, specifically so that it can find emacs.py (as supplied by the local emacs distribution.) Then it does a (python-send-string "import emacs") and expects it to work...
It looks like the defadvice wrappers that tramp uses don't actually pass PYTHONPATH, so this doesn't work even if you have the matching emacs version on the remote system.
If you M-x customize-variable RET tramp-remote-process-environment RET
then hit one of the INS buttons and add PYTHONPATH=/usr/share/emacs/23.1/etc then hit STATE and set it to "current session" (just to test it, or "save for future sessions" if it works for you) it almost works - the complaint goes away, in any case, because the remote python can now find the remote emacs.py. If you now go back to the original question, doing python-send-buffer, you just run into a different error: No such file or directory: '/tmp/py24574XdA' because python-mode just stuffs the content into a temporary file and tells the python subprocess to load that.
You'd have to change python-send-region (the other functions call it) and particularly the way it uses make-temp-file to be tramp-aware - there's even a tramp-make-tramp-temp-file you could probably build upon. (Be sure to post it if you do...)