(This question was asked here, but the answer was Linux-specific; I'm running on FreeBSD and NetBSD systems which (EDIT: ordinarily) do not have /proc.)
Python seems to dumb down argv[0], so you don't get what was passed in to the process, as a C program would. To be fair, sh and bash and Perl are no better. Is there any way I can work around this, so my Python programs can get that original value? I have administrative privileges on this FreeBSD system, and can do things like changing everyone's default PATH environment variable to point to some other directory before the one that contains python2 and python3, but I don't have control over creating /proc. I have a script which illustrates the problem. First, the script's output:
the C child program gets it right: arbitrary-arg0 arbitrary-arg1
the python2 program dumbs it down: ['./something2.py', 'arbitrary-arg1']
the python3 program dumbs it down: ['./something3.py', 'arbitrary-arg1']
the sh script dumbs it down: ./shscript.sh arbitrary-arg1
the bash script dumbs it down: ./bashscript.sh arbitrary-arg1
the perl script drops arg0: ./something.pl arbitrary-arg1
... and now the script:
#!/bin/sh
set -e
rm -rf work
mkdir work
cd work
cat > childc.c << EOD; cc childc.c -o childc
#include <stdio.h>
int main(int argc,
char **argv
)
{
printf("the C child program gets it right: ");
printf("%s %s\n",argv[0],argv[1]);
}
EOD
cat > something2.py <<EOD; chmod 700 something2.py
#!/usr/bin/env python2
import sys
print "the python2 program dumbs it down:", sys.argv
EOD
cat > something3.py <<EOD; chmod 700 something3.py
#!/usr/bin/env python3
import sys
print("the python3 program dumbs it down:", sys.argv)
EOD
cat > shscript.sh <<EOD; chmod 700 shscript.sh
#!/bin/sh
echo "the sh script dumbs it down:" \$0 \$1
EOD
cat > bashscript.sh <<EOD; chmod 700 bashscript.sh
#!/bin/sh
echo "the bash script dumbs it down:" \$0 \$1
EOD
cat > something.pl <<EOD; chmod 700 something.pl
#!/usr/bin/env perl
print("the perl script drops arg0: \$0 \$ARGV[0]\n")
EOD
cat > launch.c << EOD; cc launch.c -o launch; launch
#include <sys/types.h>
#include <sys/wait.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
int main(int argc,
char **argv,
char **arge)
{
int child_status;
size_t program_index;
pid_t child_pid;
char *program_list[]={"./childc",
"./something2.py",
"./something3.py",
"./shscript.sh",
"./bashscript.sh",
"./something.pl",
NULL
};
char *some_args[]={"arbitrary-arg0","arbitrary-arg1",NULL};
for(program_index=0;
program_list[program_index];
program_index++
)
{
child_pid=fork();
if(child_pid<0)
{
perror("fork()");
exit(1);
}
if(child_pid==0)
{
execve(program_list[program_index],some_args,arge);
perror("execve");
exit(1);
}
wait(&child_status);
}
return 0;
}
EOD
What follows is a generally useful answer to what I meant to ask.
The answer that kabanus gave is excellent, given the way I phrased the problem, so of course he gets the up-arrow and the checkmark. The transparency is a beautiful plus, in my opinion.
But it turns out that I didn't specify the situation completely. Each python script starts with a shebang, and the shebang feature makes it more complicated to launch a python script with an artificial argv[0].
Also, transparency isn't my goal; backward compatibility is. I would like the normal situation to be that sys.argv works as shipped, right out of the box, without my modifications. Also, I would like any program which launches a python script with an artificial argv[0] not to have to worry about any additional argument manipulation.
Part of the problem is to overcome the "shebang changing argv" problem.
The answer is to write a wrapper in C for each script, and the launching program launches that program instead of the actual script. The actual script looks at the arguments to the parent process (the wrapper).
The cool thing is that this can work for script types other than python. You can download a proof of concept here which demonstrates the solution for python2, python3, sh, bash, and perl. You'll have to change each CRLF to LF, using dos2unix or fromdos. This is how the python3 script handles it:
def get_arg0():
return subprocess.run("ps -p %s -o 'args='" % os.getppid(),
shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE
).stdout.decode(encoding='latin1').split(sep=" ")[0]
The solution does not rely on /proc, so it works on FreeBSD as well as Linux.
What I think is the path of least resistance here is a bit hacky, but would probably work on any OS. Basically you double wrap your Python calls. First (using Python 3 as an example), the Python3 in your path is replaced by a small C program, which you know you can trust:
#include<stdlib.h>
#include<string.h>
int main(int argc, char **argv) {
// The python 3 below should be replaced by the path to the original one
// In my tests I named this program wrap_python so there was no problem
// but if you are changing this system wide (and calling the wrapper python3
// you can't leave this.
const char *const program = "python3 wrap_python.py";
size_t size = strlen(program) + 1; // Already added null character at end
for(int count = 0; count < argc; ++count)
size += strlen(argv[count]) + 1; // + 1 for space
char *cmd = malloc(size);
if(!cmd) exit(-1);
cmd[0] = '\0';
strcat(cmd, program);
for(int count = 1; count < argc; ++count) {
strcat(cmd, " ");
strcat(cmd, argv[count]);
}
strcat(cmd, " ");
strcat(cmd, argv[0]);
return system(cmd);
}
You can make this faster, but hey, premature optimization?
Note we are calling a script called wrap_python.py (probably you would need a full path here). We want to pass the "true" argv, but we need to work some in the Python context to make it transparent. The true argv[0] is passed as a last argument, and wrap_python.py is:
from sys import argv
argv[0] = argv.pop(-1)
print("Passing:", argv) # Delete me
exit(exec(open(argv[1]).read())) # Different in Python 2. Close the file handle if you're pedantic.
Our small wrapper replaces argv[0] with the one provided by our C wrapper removing it from the end, and then manually executes in the same context. Specifically __name__ == __main__ is true.
This would be run as
python3 my_python_script arg1 arg2 etc...
where your path now will point to that original C program. Testing this on
import sys
print(__name__)
print("Got", sys.argv)
yields
__main__
Got ['./wrap_python', 'test.py', 'hello', 'world', 'this', '1', '2', 'sad']
Note I called my program wrap_python - you want to name it python3.
Use Python's ctypes module to get the "program name" which by default is set to argv[0]. See Python source code here. For example:
import ctypes
GetProgramName = ctypes.pythonapi.Py_GetProgramName
GetProgramName.restype = ctypes.c_wchar_p
def main():
print(GetProgramName())
if __name__ == '__main__':
main()
Running the command prints:
$ exec -a hello python3 name.py
hello
Related
I am trying to write a code that automate some of Nasa's Heasoft program - this doesn't matter -,
heainit is definied in the bashrc as:
export HEADAS=~/heasoft-6.27.2/x86_64-pc-linux-gnu-libc2.31
alias heainit='. $HEADAS/headas-init.sh
so whenever I write heainit on terminal it works pretty fine. However, whenever I tried to pass "heainit" to the terminal through c++ or Python, the code it self works but it doesn't initiate the program!
Ik that alise doesn't create an actual path there but i tried every possibility ik about.
in c++ I tried:
#include <iostream>
#include <stdlib.h>
#include <string.h>
#include <cstdlib>
#include <string>
#include <sstream>
using namespace std;
int main(int argc, char** argv){ string ind, outd;
const char* heainit = "/home/karim/heasoft-6.27.2/x86_64-pc-linux-gnu-libc2.31/headas-init.sh" ;
system(heainit);
return 0;
}
and
int main(string, char*){
string ind, outd;
std::string HEADAS = "/home/karim/heasoft-6.27.2/x86_64-pc-linux-gnu-libc2.31";
std::string str = std::string(". //") + HEADAS + std::string("/headas-init.sh");
const char *c = str.c_str(". /home/karim/heasoft-6.27.2/x86_64-pc-linux-gnu-libc2.31/heafas-init.sh");
system(const char* c);
return 0;
}
in python I tried:
import os
import subprocess
os.system("heainit")
subprocess.run("heainit",capture_output=True)
and
HEADAS=~/heasoft-6.27.2/x86_64-pc-linux-gnu-libc2.31
heainit="." + "\t" + HEADAS + "/headas-init.sh"
subprocess.run(heainit, shell=True)
My latest attempt was to write a bash script:
#!/bin/bash
export HEADAS=~/heasoft-6.27.2/x86_64-pc-linux-gnu-libc2.31
export HEADAS
alias heainit='. $HEADAS/headas-init.sh'
heainit
and the prompt gives the following: ./hea.sh: line 6: heainit: command not found
whenever I execute any of the aforementioned codes with anyother command like "ls -l" for instance it works perfectly :")) I don't know how should I pass this argument(it should be followed by others when the program is initiated) to the terminal yet and I don't know what i lack to go. Maybe the problem is trivial but i'm stuck there for a long time now.
whenever I tried to pass "heainit" to the terminal through c++ or Python, the code it self works but it doesn't initiate the program!
heainit is an alias. So first: the system call runs the shell in non-interactive mode in which alias is ignored and secondly: most probably you have an if [[ $- ~= *i* ]] on top of your bashrc to return when the shell is not interactive. In short: an alias is not a command and is meant to be run only in an interactive shell. bash shell manual aliases
If you wish to run heainit from the terminal, you could create an executable in PATH. Or you could create a function and add it to /etc/bash.bashrc before the interactive check or like to /etc/profile.d/*.
Still, that would most probably solve nothing, as the file is sourced. system() call and subprocess() runs a subprocess, the subprocess environment doesn't affect the parent environment. So if this assessment is true, sourcing the file is doing effectively nothing as it sources it in subprocess which doesn't affect the current environment. The solution would be to write a parser of the file (either a full blown shell parser and interpreter or something simpler depending on the content of the file) and parse the text from the file inside the current process.
I want to call a Python script from C, passing some arguments that are needed in the script.
The script I want to use is mrsync, or multicast remote sync. I got this working from command line, by calling:
python mrsync.py -m /tmp/targets.list -s /tmp/sourcedata -t /tmp/targetdata
-m is the list containing the target ip-addresses.
-s is the directory that contains the files to be synced.
-t is the directory on the target machines where the files will be put.
So far I managed to run a Python script without parameters, by using the following C program:
Py_Initialize();
FILE* file = fopen("/tmp/myfile.py", "r");
PyRun_SimpleFile(file, "/tmp/myfile.py");
Py_Finalize();
This works fine. However, I can't find how I can pass these argument to the PyRun_SimpleFile(..) method.
Seems like you're looking for an answer using the python development APIs from Python.h. Here's an example for you that should work:
#My python script called mypy.py
import sys
if len(sys.argv) != 2:
sys.exit("Not enough args")
ca_one = str(sys.argv[1])
ca_two = str(sys.argv[2])
print "My command line args are " + ca_one + " and " + ca_two
And then the C code to pass these args:
//My code file
#include <stdio.h>
#include <python2.7/Python.h>
void main()
{
FILE* file;
int argc;
char * argv[3];
argc = 3;
argv[0] = "mypy.py";
argv[1] = "-m";
argv[2] = "/tmp/targets.list";
Py_SetProgramName(argv[0]);
Py_Initialize();
PySys_SetArgv(argc, argv);
file = fopen("mypy.py","r");
PyRun_SimpleFile(file, "mypy.py");
Py_Finalize();
return;
}
If you can pass the arguments into your C function this task becomes even easier:
void main(int argc, char *argv[])
{
FILE* file;
Py_SetProgramName(argv[0]);
Py_Initialize();
PySys_SetArgv(argc, argv);
file = fopen("mypy.py","r");
PyRun_SimpleFile(file, "mypy.py");
Py_Finalize();
return;
}
You can just pass those straight through. Now my solutions only used 2 command line args for the sake of time, but you can use the same concept for all 6 that you need to pass... and of course there's cleaner ways to capture the args on the python side too, but that's just the basic idea.
Hope it helps!
You have two options.
Call
system("python mrsync.py -m /tmp/targets.list -s /tmp/sourcedata -t /tmp/targetdata")
in your C code.
Actually use the API that mrsync (hopefully) defines. This is more flexible, but much more complicated. The first step would be to work out how you would perform the above operation as a Python function call. If mrsync has been written nicely, there will be a function mrsync.sync (say) that you call as
mrsync.sync("/tmp/targets.list", "/tmp/sourcedata", "/tmp/targetdata")
Once you've worked out how to do that, you can call the function directly from the C code using the Python API.
How exactly does Python receive
echo input | python script
and
python script input
differently? I know that one comes through stdin and the other is passed as an argument, but what happens differently in the back-end?
I'm not exactly sure what is confusing you here. stdin and command line arguments are treated as two different things.
Since you're most likely using CPython (the C implementation of Python) the command line args are passed automatically in the argv parameter as with any other c program. The main function for CPython (located in python.c) receives them:
int main(int argc, char **argv) // **argv <-- Your command line args
{
wchar_t **argv_copy;
/* We need a second copy, as Python might modify the first one. */
wchar_t **argv_copy2;
/* ..rest of main omitted.. */
While the contents of the pipe are stored in stdin which you can tap into via sys.stdin.
Using a sample test.py script:
import sys
print("Argv params:\n ", sys.argv)
if not sys.stdin.isatty():
print("Stdin: \n", sys.stdin.readlines())
Running this with no piping performed yields:
(Python3)jim#jim: python test.py "hello world"
Argv params:
['test.py', 'hello world']
While, using echo "Stdin up in here" | python test.py "hello world", we'll get:
(Python3)jim#jim: echo "Stdin up in here" | python test.py "hello world"
Argv params:
['test.py', 'hello world']
Stdin:
['Stdin up in here\n']
Not strictly related, but an interesting note:
Additionally, I remembered that you can execute content that is stored in stdin by using the - argument for Python:
(Python3)jimm#jim: echo "print('<stdin> input')" | python -
<stdin> input
Kewl!
i have 2 codes that are identical in the sense that they complete the same task. one code was written in python, the other in c++. all the codes do is call an executable (the executable generates an ascii file). in c++, i use the system() command to call the executable. in python, i have used many things including os.system subprocess.call subprocess.popen.
i realize that c++ is a compiled language while python is interpreted. and i also realize that the python calls have more overhead. but the c++ code does the job nearly 100 times faster than the python code. c++ time was about 0.004 seconds. python time was around 0.35 seconds.
even a simple pwd command takes more than 10 times longer with python than it does with c++. if the overhead is what is slowing the python code down, is there a faster option in python than what i have already tried?
here is a simple python code:
from os import system
from time import time
t0 = time();
system("pwd");
print "duration: ",time()-t0;
and here is the same thing in c++:
#include <iostream>
#include <sys/time.h>
double diff(timespec start, timespec end) { return (end.tv_nsec-start.tv_nsec)/1e9; }
int main()
{
timespec t0, t1;
clock_gettime(CLOCK_PROCESS_CPUTIME_ID, & t0);
system("pwd");
clock_gettime(CLOCK_PROCESS_CPUTIME_ID, & t1);
std::cout << "duration: " << diff(t0,t1) << "\n";
return 0;
}
i used gcc to compile the c++ code. you have to use the -lrt option to get the code to compile correctly.
you can run the code yourself. my timing methods could be wrong. but if they are okay, then the python script takes more than 10 times as long to execute the pwd command compared to the c++ executable
C 'exec' call executes the program directly.
While the Python 'system' call first executes bash, that executes the program in question.
I cooked up a little script and execution time was much faster than what you are seeing.
td#timsworld2:~/tmp/so$ cat nothing.py
#!/usr/bin/env python
import subprocess
import sys
cmd = ['python', '-V'] if 'py' in sys.argv else ['pwd']
if 'shell' in sys.argv:
subprocess.call(' '.join(cmd), shell=True)
else:
subprocess.call(cmd)
td#timsworld2:~/tmp/so$ time ./nothing.py
/home/td/tmp/so
real 0m0.024s
user 0m0.012s
sys 0m0.008s
td#timsworld2:~/tmp/so$ time python nothing.py
/home/td/tmp/so
real 0m0.020s
user 0m0.012s
sys 0m0.004s
td#timsworld2:~/tmp/so$ time ./nothing.py py
Python 2.7.3
real 0m0.022s
user 0m0.016s
sys 0m0.000s
td#timsworld2:~/tmp/so$ time ./nothing.py sh
/home/td/tmp/so
real 0m0.020s
user 0m0.012s
sys 0m0.004s
td#timsworld2:~/tmp/so$
You can use execvp directly in python
import os
binary = "ls"
options = [binary, "-l"]
newpid = os.fork()
if newpid == 0:
# we are in the child process
os.execvp(binary, options)
os._exit(1)
os.wait()
print "executed", " ".join(options)
I updated my python interpreter, but I think the old one is still called. When I check for the version I get:
$ python -V
Python 3.0.1
But I believe the old interpreter is still being called. When I run the command:
python myProg.py
The script runs properly. But when I invoke it with the command
./myProg.py
I get the error message:
AttributeError: 'str' object has no attribute 'format'
Which apparently is due to the old interpreter being called. How can I fix this? I run Mac OS X 10.5. Has it something to do with the first line:
#!/usr/bin/python
I just started out with python and am not very familiar with interpreted languages, so I am not too sure what is going on.
According to the first line of the script, #!/usr/bin/python, you are calling the Python interpreter at /usr/bin/python (which is most likely the one that ships with Mac OS X). You have to change that path to the path where you installed your Python 3 interpreter (likely /usr/local/bin/python or /opt/local/bin/python); or you can just change that line to read #!/usr/bin/env python, which will call the python listed first in your PATH variable (which seems to be the newer version you installed).
Firstly, the recommended shebang line is:
#!/usr/bin/env python
This will make sure the python interpreter that is invoked when you ./foo.py is the same interpreter that is invoked when you invoke python from the command line.
From your description, I suspect that if you did:
which python
It would not give you /usr/bin/python. It would give you something else, which is where the python 3 interpreter lives. You can either modify your shebang line to the above, or replace the path to the python interpreter with the path returned by which.
Try which python. I will tell you which python interpreter is used in your environment.
If it is not /usr/bin/python like in the script, then your suspicion is confirmed.
It's very possibly what you suspect, that the shebang line is calling the older version. Two things you might want to check:
1) what version is the interpreter at /usr/bin/python:
/usr/bin/python -V
2) where is the python 3 interpreter you installed:
which python
If you get the correct one from the command line, then replace your shebang line with this:
#!/usr/bin/env python
Addendum: You could also replace the older version of python with a symlink to python 3, but beware that any major OS X updates (ie: 10.5.6 to 10.5.7) will likely break this:
sudo mv /usr/bin/python /usr/bin/python25
sudo ln -s /path/to/python/3/python /usr/bin/python
run 'which python' - if this gives a different answer than /usr/bin/python, change #!/usr/bin/python to have that path instead.
It may be a bit odd providing a Perl script to answer a Python question, but it works for Python just as well as it does for Perl. This is a script called 'fixin', meaning 'fix interpreter'. It changes the shebang line to the correct string for your current PATH.
#!/Users/jleffler/perl/v5.10.0/bin/perl
#
# #(#)$Id: fixin.pl,v 1.3 2003/03/11 21:20:08 jleffler Exp $
#
# FIXIN: from Programming Perl
# Usage: fixin [-s] [file ...]
# Configuration
$does_hashbang = 1; # Kernel recognises #!
$verbose = 1; # Verbose by default
# Construct list of directories to search.
#absdirs = reverse grep(m!^/!, split(/:/, $ENV{'PATH'}, 999));
# Process command line arguments
if ($ARGV[0] eq '-s')
{
shift;
$verbose = 0;
}
die "Usage: $0 [-s] [file ...]\n" unless #ARGV || !-t;
#ARGV = '-' unless #ARGV;
# Process each file.
FILE: foreach $filename (#ARGV)
{
open(IN, $filename) || ((warn "Can't process $filename: $!\n"), next);
$_ = <IN>;
next FILE unless /^#!/; # Not a hash/bang file
chop($cmd = $_);
$cmd =~ s/^#! *//;
($cmd, $arg) = split(' ', $cmd, 2);
$cmd =~ s!^.*/!!;
# Now look (in reverse) for interpreter in absolute path
$found = '';
foreach $dir (#absdirs)
{
if (-x "$dir/$cmd")
{
warn "Ignoring $found\n" if $verbose && $found;
$found = "$dir/$cmd";
}
}
# Figure out how to invoke interpreter on this machine
if ($found)
{
warn "Changing $filename to $found\n" if $verbose;
if ($does_hashbang)
{
$_ = "#!$found";
$_ .= ' ' . $arg if $arg ne '';
$_ .= "\n";
}
else
{
$_ = <<EOF;
:
eval 'exec $found $arg -S \$0 \${1+"\$#"}'
if \$running_under_some_shell;
EOF
}
}
else
{
warn "Can't find $cmd in PATH, $filename unchanged\n" if $verbose;
next FILE;
}
# Make new file if necessary
if ($filename eq '-') { select(STDOUT); }
else
{
rename($filename, "$filename.bak") ||
((warn "Can't modify $filename"), next FILE);
open(OUT, ">$filename") ||
die "Can't create new $filename: $!\n";
($def, $ino, $mode) = stat IN;
$mode = 0755 unless $dev;
chmod $mode, $filename;
select(OUT);
}
# Print the new #! line (or the equivalent) and copy the rest of the file.
print;
while (<IN>)
{
print;
}
close IN;
close OUT;
}
The code is derived from a script of the same name in the original Camel Book ('Programming Perl', first edition). This copy has been hacked a bit since then - and should be hacked some more. But I use it routinely -- indeed, I just copied it from one Mac to another, and since I've not installed Perl 5.10.0 on the second, I ran:
$ perl fixin fixin
Changing fixin to /usr/bin/perl
$
Thereby changing from the private install Perl to the standard one.
Exercise for the reader - rewrite the script in Python.