i have 2 codes that are identical in the sense that they complete the same task. one code was written in python, the other in c++. all the codes do is call an executable (the executable generates an ascii file). in c++, i use the system() command to call the executable. in python, i have used many things including os.system subprocess.call subprocess.popen.
i realize that c++ is a compiled language while python is interpreted. and i also realize that the python calls have more overhead. but the c++ code does the job nearly 100 times faster than the python code. c++ time was about 0.004 seconds. python time was around 0.35 seconds.
even a simple pwd command takes more than 10 times longer with python than it does with c++. if the overhead is what is slowing the python code down, is there a faster option in python than what i have already tried?
here is a simple python code:
from os import system
from time import time
t0 = time();
system("pwd");
print "duration: ",time()-t0;
and here is the same thing in c++:
#include <iostream>
#include <sys/time.h>
double diff(timespec start, timespec end) { return (end.tv_nsec-start.tv_nsec)/1e9; }
int main()
{
timespec t0, t1;
clock_gettime(CLOCK_PROCESS_CPUTIME_ID, & t0);
system("pwd");
clock_gettime(CLOCK_PROCESS_CPUTIME_ID, & t1);
std::cout << "duration: " << diff(t0,t1) << "\n";
return 0;
}
i used gcc to compile the c++ code. you have to use the -lrt option to get the code to compile correctly.
you can run the code yourself. my timing methods could be wrong. but if they are okay, then the python script takes more than 10 times as long to execute the pwd command compared to the c++ executable
C 'exec' call executes the program directly.
While the Python 'system' call first executes bash, that executes the program in question.
I cooked up a little script and execution time was much faster than what you are seeing.
td#timsworld2:~/tmp/so$ cat nothing.py
#!/usr/bin/env python
import subprocess
import sys
cmd = ['python', '-V'] if 'py' in sys.argv else ['pwd']
if 'shell' in sys.argv:
subprocess.call(' '.join(cmd), shell=True)
else:
subprocess.call(cmd)
td#timsworld2:~/tmp/so$ time ./nothing.py
/home/td/tmp/so
real 0m0.024s
user 0m0.012s
sys 0m0.008s
td#timsworld2:~/tmp/so$ time python nothing.py
/home/td/tmp/so
real 0m0.020s
user 0m0.012s
sys 0m0.004s
td#timsworld2:~/tmp/so$ time ./nothing.py py
Python 2.7.3
real 0m0.022s
user 0m0.016s
sys 0m0.000s
td#timsworld2:~/tmp/so$ time ./nothing.py sh
/home/td/tmp/so
real 0m0.020s
user 0m0.012s
sys 0m0.004s
td#timsworld2:~/tmp/so$
You can use execvp directly in python
import os
binary = "ls"
options = [binary, "-l"]
newpid = os.fork()
if newpid == 0:
# we are in the child process
os.execvp(binary, options)
os._exit(1)
os.wait()
print "executed", " ".join(options)
Related
(This question was asked here, but the answer was Linux-specific; I'm running on FreeBSD and NetBSD systems which (EDIT: ordinarily) do not have /proc.)
Python seems to dumb down argv[0], so you don't get what was passed in to the process, as a C program would. To be fair, sh and bash and Perl are no better. Is there any way I can work around this, so my Python programs can get that original value? I have administrative privileges on this FreeBSD system, and can do things like changing everyone's default PATH environment variable to point to some other directory before the one that contains python2 and python3, but I don't have control over creating /proc. I have a script which illustrates the problem. First, the script's output:
the C child program gets it right: arbitrary-arg0 arbitrary-arg1
the python2 program dumbs it down: ['./something2.py', 'arbitrary-arg1']
the python3 program dumbs it down: ['./something3.py', 'arbitrary-arg1']
the sh script dumbs it down: ./shscript.sh arbitrary-arg1
the bash script dumbs it down: ./bashscript.sh arbitrary-arg1
the perl script drops arg0: ./something.pl arbitrary-arg1
... and now the script:
#!/bin/sh
set -e
rm -rf work
mkdir work
cd work
cat > childc.c << EOD; cc childc.c -o childc
#include <stdio.h>
int main(int argc,
char **argv
)
{
printf("the C child program gets it right: ");
printf("%s %s\n",argv[0],argv[1]);
}
EOD
cat > something2.py <<EOD; chmod 700 something2.py
#!/usr/bin/env python2
import sys
print "the python2 program dumbs it down:", sys.argv
EOD
cat > something3.py <<EOD; chmod 700 something3.py
#!/usr/bin/env python3
import sys
print("the python3 program dumbs it down:", sys.argv)
EOD
cat > shscript.sh <<EOD; chmod 700 shscript.sh
#!/bin/sh
echo "the sh script dumbs it down:" \$0 \$1
EOD
cat > bashscript.sh <<EOD; chmod 700 bashscript.sh
#!/bin/sh
echo "the bash script dumbs it down:" \$0 \$1
EOD
cat > something.pl <<EOD; chmod 700 something.pl
#!/usr/bin/env perl
print("the perl script drops arg0: \$0 \$ARGV[0]\n")
EOD
cat > launch.c << EOD; cc launch.c -o launch; launch
#include <sys/types.h>
#include <sys/wait.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
int main(int argc,
char **argv,
char **arge)
{
int child_status;
size_t program_index;
pid_t child_pid;
char *program_list[]={"./childc",
"./something2.py",
"./something3.py",
"./shscript.sh",
"./bashscript.sh",
"./something.pl",
NULL
};
char *some_args[]={"arbitrary-arg0","arbitrary-arg1",NULL};
for(program_index=0;
program_list[program_index];
program_index++
)
{
child_pid=fork();
if(child_pid<0)
{
perror("fork()");
exit(1);
}
if(child_pid==0)
{
execve(program_list[program_index],some_args,arge);
perror("execve");
exit(1);
}
wait(&child_status);
}
return 0;
}
EOD
What follows is a generally useful answer to what I meant to ask.
The answer that kabanus gave is excellent, given the way I phrased the problem, so of course he gets the up-arrow and the checkmark. The transparency is a beautiful plus, in my opinion.
But it turns out that I didn't specify the situation completely. Each python script starts with a shebang, and the shebang feature makes it more complicated to launch a python script with an artificial argv[0].
Also, transparency isn't my goal; backward compatibility is. I would like the normal situation to be that sys.argv works as shipped, right out of the box, without my modifications. Also, I would like any program which launches a python script with an artificial argv[0] not to have to worry about any additional argument manipulation.
Part of the problem is to overcome the "shebang changing argv" problem.
The answer is to write a wrapper in C for each script, and the launching program launches that program instead of the actual script. The actual script looks at the arguments to the parent process (the wrapper).
The cool thing is that this can work for script types other than python. You can download a proof of concept here which demonstrates the solution for python2, python3, sh, bash, and perl. You'll have to change each CRLF to LF, using dos2unix or fromdos. This is how the python3 script handles it:
def get_arg0():
return subprocess.run("ps -p %s -o 'args='" % os.getppid(),
shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE
).stdout.decode(encoding='latin1').split(sep=" ")[0]
The solution does not rely on /proc, so it works on FreeBSD as well as Linux.
What I think is the path of least resistance here is a bit hacky, but would probably work on any OS. Basically you double wrap your Python calls. First (using Python 3 as an example), the Python3 in your path is replaced by a small C program, which you know you can trust:
#include<stdlib.h>
#include<string.h>
int main(int argc, char **argv) {
// The python 3 below should be replaced by the path to the original one
// In my tests I named this program wrap_python so there was no problem
// but if you are changing this system wide (and calling the wrapper python3
// you can't leave this.
const char *const program = "python3 wrap_python.py";
size_t size = strlen(program) + 1; // Already added null character at end
for(int count = 0; count < argc; ++count)
size += strlen(argv[count]) + 1; // + 1 for space
char *cmd = malloc(size);
if(!cmd) exit(-1);
cmd[0] = '\0';
strcat(cmd, program);
for(int count = 1; count < argc; ++count) {
strcat(cmd, " ");
strcat(cmd, argv[count]);
}
strcat(cmd, " ");
strcat(cmd, argv[0]);
return system(cmd);
}
You can make this faster, but hey, premature optimization?
Note we are calling a script called wrap_python.py (probably you would need a full path here). We want to pass the "true" argv, but we need to work some in the Python context to make it transparent. The true argv[0] is passed as a last argument, and wrap_python.py is:
from sys import argv
argv[0] = argv.pop(-1)
print("Passing:", argv) # Delete me
exit(exec(open(argv[1]).read())) # Different in Python 2. Close the file handle if you're pedantic.
Our small wrapper replaces argv[0] with the one provided by our C wrapper removing it from the end, and then manually executes in the same context. Specifically __name__ == __main__ is true.
This would be run as
python3 my_python_script arg1 arg2 etc...
where your path now will point to that original C program. Testing this on
import sys
print(__name__)
print("Got", sys.argv)
yields
__main__
Got ['./wrap_python', 'test.py', 'hello', 'world', 'this', '1', '2', 'sad']
Note I called my program wrap_python - you want to name it python3.
Use Python's ctypes module to get the "program name" which by default is set to argv[0]. See Python source code here. For example:
import ctypes
GetProgramName = ctypes.pythonapi.Py_GetProgramName
GetProgramName.restype = ctypes.c_wchar_p
def main():
print(GetProgramName())
if __name__ == '__main__':
main()
Running the command prints:
$ exec -a hello python3 name.py
hello
I can debug Python code using ddd -pydb prog.py. All the python command line arguments can be passed too after prog.py. In my case, many classes have been implemented in C++ that are exposed to python using boost-python. I wish I could debug python code and C++ together. For example I want to set break points like this :
break my_python.py:123
break my_cpp.cpp:456
cont
Of course I am trying it after compiling c++ codes with debug option but the debugger does not cross boost boundary. Is there any way?
EDIT:
I saw http://www.boost.org/doc/libs/1_61_0/libs/python/doc/html/faq/how_do_i_debug_my_python_extensi.html.
I followed it and I can do debugging both for python and C++. But I preferably want to do visual debugging with DDD but I don't know how to give 'target exec python' command inside DDD. If not (just using gdb as in the link) I should be able to debug for a Python script not interactively giving python commands as in the link.
I found out how to debug the C++ part while running python. (realized it while reading about process ID detection in Python book..).
First you run the python program which includes C++ programs. At the start of the python program, use raw_input() to make the program wait for you input. But just before that do print os.getpid() (of course you should have imported os package). When you run the python program, it will have printed the pid of the python program you are running and will be waiting for your keyboard input.
python stop code :
import os
def w1(str):
print (str)
wait = raw_input()
return
print os.getpid()
w1('starting main..press a key')
result :
27352
starting main..press a key
Or, you can use import pdb, pdb.set_trace() as comment below.(thanks #AndyG) and see EDIT* to get pid using ps -aux.
Now, suppose the C++ shared library is _caffe.so (which is my case. This _caffe.so library has all the C++ codes and boost python wrapper functions). 27352 is the pid. Then in another shell start gdb like
gdb caffe-fast-rcnn/python/caffe/_caffe.so 27352
or if you want to use graphical debugging using like DDD, do
ddd caffe-fast-rcnn/python/caffe/_caffe.so 27352
Then you'll see gdb starts and wait with prompt. The python program is interrupted by gdb and waits in stopped mode (it was waiting for your key input but now it's really in stopeed mode, and it needs gdb continue command from the second debugger to proceed with the key waiting).
Now you can give break point command in gdb like
br solver.cpp:225
and you can see message like
Breakpoint 1 at 0x7f2cccf70397: file src/caffe/solver.cpp, line 226. (2 locations)
When you give continue command in the second gdb window(that was holding the program), the python code runs again. Of course you should give a key input in the first gdb window to make it proceed.
Now at least you can debug the C++ code while running python program(that's what I wanted to do)!
I later checked if I can do python and C++ debugging at the same time and it works. You start the debugger(DDD) like ddd -pydb prog1.py options.. and attach another DDD using method explained above. Now you can set breakpoints for python and C++ and using other debug functions in each window(I wish I had known this a couple of months earlier.. It should have helped tons.).
EDIT : to get the pid, you can do ps -aux | grep python instead. This pid is the next of ddd's pid.
I had a similar problem, but failed to get the solutions in Chan's answer to work (on MAC OS X 10.12.4). Instead the following worked for me.
Write a python script test.py that imports and uses the boost.Python module.
start python in the debugger
lldb python3 test.py
giving
> lldb python3 test.py
(lldb) target create "python3"
Current executable set to 'python3' (x86_64).
(lldb) settings set -- target.run-args "test.py"
(lldb) run
Process 46189 launched: '/Users/me/anaconda/bin/python3' (x86_64)
test.cpython-36m-darwin.so was compiled with optimization - stepping may behave oddly; variables may not be available.
Process 46189 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x10d4b3000)
frame #0: 0x00000001019f49c2 test.cpython-36m-darwin.so`std::__1::enable_if<true, void>::type (anonymous namespace)::Render2D<double>::add_particle<true, 5ul>(float*, float, float, float, float) const [inlined] mylib::SSE::packed<8ul, float>::loadu(
944 { return {_mm256_load_ps(p)}; }
945 /// load from unaligned memory location
946 static __always__inline packed loadu(const element_type*p) noexcept
-> 947 { return {_mm256_loadu_ps(p)}; }
948 /// load from aligned memory location, using template arg for alignment
949 template<bool aligned>
950 static __always_inline enable_if_t< aligned, packed>
No need to obtain the pid and start the debugger from a separate window or set any breakpoints.
I can debug Python code using ddd -pydb prog.py. All the python command line arguments can be passed too after prog.py. In my case, many classes have been implemented in C++ that are exposed to python using boost-python. I wish I could debug python code and C++ together. For example I want to set break points like this :
break my_python.py:123
break my_cpp.cpp:456
cont
Of course I am trying it after compiling c++ codes with debug option but the debugger does not cross boost boundary. Is there any way?
EDIT:
I saw http://www.boost.org/doc/libs/1_61_0/libs/python/doc/html/faq/how_do_i_debug_my_python_extensi.html.
I followed it and I can do debugging both for python and C++. But I preferably want to do visual debugging with DDD but I don't know how to give 'target exec python' command inside DDD. If not (just using gdb as in the link) I should be able to debug for a Python script not interactively giving python commands as in the link.
I found out how to debug the C++ part while running python. (realized it while reading about process ID detection in Python book..).
First you run the python program which includes C++ programs. At the start of the python program, use raw_input() to make the program wait for you input. But just before that do print os.getpid() (of course you should have imported os package). When you run the python program, it will have printed the pid of the python program you are running and will be waiting for your keyboard input.
python stop code :
import os
def w1(str):
print (str)
wait = raw_input()
return
print os.getpid()
w1('starting main..press a key')
result :
27352
starting main..press a key
Or, you can use import pdb, pdb.set_trace() as comment below.(thanks #AndyG) and see EDIT* to get pid using ps -aux.
Now, suppose the C++ shared library is _caffe.so (which is my case. This _caffe.so library has all the C++ codes and boost python wrapper functions). 27352 is the pid. Then in another shell start gdb like
gdb caffe-fast-rcnn/python/caffe/_caffe.so 27352
or if you want to use graphical debugging using like DDD, do
ddd caffe-fast-rcnn/python/caffe/_caffe.so 27352
Then you'll see gdb starts and wait with prompt. The python program is interrupted by gdb and waits in stopped mode (it was waiting for your key input but now it's really in stopeed mode, and it needs gdb continue command from the second debugger to proceed with the key waiting).
Now you can give break point command in gdb like
br solver.cpp:225
and you can see message like
Breakpoint 1 at 0x7f2cccf70397: file src/caffe/solver.cpp, line 226. (2 locations)
When you give continue command in the second gdb window(that was holding the program), the python code runs again. Of course you should give a key input in the first gdb window to make it proceed.
Now at least you can debug the C++ code while running python program(that's what I wanted to do)!
I later checked if I can do python and C++ debugging at the same time and it works. You start the debugger(DDD) like ddd -pydb prog1.py options.. and attach another DDD using method explained above. Now you can set breakpoints for python and C++ and using other debug functions in each window(I wish I had known this a couple of months earlier.. It should have helped tons.).
EDIT : to get the pid, you can do ps -aux | grep python instead. This pid is the next of ddd's pid.
I had a similar problem, but failed to get the solutions in Chan's answer to work (on MAC OS X 10.12.4). Instead the following worked for me.
Write a python script test.py that imports and uses the boost.Python module.
start python in the debugger
lldb python3 test.py
giving
> lldb python3 test.py
(lldb) target create "python3"
Current executable set to 'python3' (x86_64).
(lldb) settings set -- target.run-args "test.py"
(lldb) run
Process 46189 launched: '/Users/me/anaconda/bin/python3' (x86_64)
test.cpython-36m-darwin.so was compiled with optimization - stepping may behave oddly; variables may not be available.
Process 46189 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x10d4b3000)
frame #0: 0x00000001019f49c2 test.cpython-36m-darwin.so`std::__1::enable_if<true, void>::type (anonymous namespace)::Render2D<double>::add_particle<true, 5ul>(float*, float, float, float, float) const [inlined] mylib::SSE::packed<8ul, float>::loadu(
944 { return {_mm256_load_ps(p)}; }
945 /// load from unaligned memory location
946 static __always__inline packed loadu(const element_type*p) noexcept
-> 947 { return {_mm256_loadu_ps(p)}; }
948 /// load from aligned memory location, using template arg for alignment
949 template<bool aligned>
950 static __always_inline enable_if_t< aligned, packed>
No need to obtain the pid and start the debugger from a separate window or set any breakpoints.
I have a small python program that parses a text file and writes the output to another file. Currently, I am writing a bash script to call this program several times, it looks something like:
for i in $(seq 1 100); do
python /home/Documents/myProgram.py myTextFile$i
done
This works fine but I want to know if it is possible to have the python program inside the bash script file so when another user runs the script they don't need to have the python program in their memory; i.e., is it possible to copy and paste the python program into the bash script and run it from within the script itself?
#!/bin/bash
python - 1 2 3 << 'EOF'
import sys
print 'Argument List:', str(sys.argv)
EOF
Output:
Argument List: ['-', '1', '2', '3']
I think you should be able to put:
python << END
[Python code here]
END
But I have not tested this.
for simple scripts you can also run python with the -c option. For example
python -c "x=1; print x; print 'great program eh'"
I wouldn't recommend writing anything too complicated, but it could work for something simple.
Pardon the thread necromancy, but here's a technique that is missing that may be useful to someone.
#!/bin/bash
""":" # Hide bash from python
for i in $(seq 1 100); do
python3 $(readlink -f "$0") myTextFile$i
done
exit
"""
# Python script starts here
import sys
print('Argument List: ', str(sys.argv))
However, I do agree with the general recommendation to just do the loop in Python.
I use this python scripts:
python2.6 and 2.7
for i in xrange(1000000):print i
python3.3
for i in range(1000000):print(i)
Results:
python 2.7
real 1m15.973s
user 0m3.392s
sys 0m6.384s
python 2.6
real 1m15.860s
user 0m3.064s
sys 0m6.296s
With python 3.3 I test the script many times and I receive this diff of running .
python 3.3
real 1m34.235s
user 0m10.668s
sys 0m8.988s
real 1m29.431s
user 0m10.304s
sys 0m9.512s
real 1m12.528s
user 0m10.568s
sys 0m9.004s
real 1m4.193s
user 0m9.688s
sys 0m8.812s
real 1m18.332s
user 0m9.728s
sys 0m9.132s
After that I try again python 2.6 and I got this:
real 0m45.003s
user 0m3.056s
sys 0m5.956s
What is the best way to benchmark 2 python scripts 3.3 and 2.7 (or 2.6).
Use the timeit module to compare small Python snippets. It avoids the common pitfalls, making timing values comparable.
However, you are mostly timing write speed to sys.stdout when timing print i (and in Python 3, encoding to the terminal codec).
There are several ways to benchmark Python programs. At least, I can come with two serious ways. You can find an expanded version of what I say in these slides here. You also may benefit from this video from a talk about Python profiling from PyCon 2013 (from Amjith Ramanujam).
cProfile module
The cProfile module gives you an insight of the time spent in every procedure of your program. It can be manipulated in a very efficient and precise manner. But, the drawback of it is that you cannot trust the execution time it gives for each procedure but its relative time spent compared to the others.
Using cProfile is simply done like this:
python -m cProfile ./mypythonprogram.py
If you know gprof, it will give you a similar output but for a Python program.
timeit module
The timeit module is intended to really evaluate the time the program spend in total. On the contrary to cProfile, there is no extra instrumentation of each procedure and, thus, no slowdown during the execution.
def foo ():
for i in range (10):
i = i * i
from timeit import Timer
max_iter = 10000
t = Timer ("foo()", "from __main__ import foo")
print ("foo(%i): %f seconds" %(max_iter, t.timeit(max_iter)))
And, you call it like this within the command line:
$> python2.7 timeit_example.py
foo(10000): 0.012774 seconds
$> python3.2 timeit_example.py
foo(10000): 0.014030 seconds
Don't test with "print", use something else.