I'm running the below command using python subprocess to extract files from rpm.
But the command failes when the rpm size is more than 25 - 30 MB. Tried the command using Popen, call, with stdout as PIPE and os.system as well. This command is working fine when i run it in shell directly. The problem is only when i invoke this by some means from Python
Command:
rpm2cpio <rpm_name>.rpm| cpio -idmv
I did an strace on the process id and found that its always hung on some write system call
ps -ef | grep cpio
root 4699 4698 4 11:05 pts/0 00:00:00 rpm2cpio kernel-2.6.32-573.26.1.el6.x86_64.rpm
root 4700 4698 0 11:05 pts/0 00:00:00 cpio -idmv
strace -p 4699
Process 4699 attached
write(10, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0rc_pixelview_new"..., 8192
strace -p 4700
Process 4700 attached
write(2, "./lib/modules/2.6.32-573.26.1.el"..., 94
I have 2 questions:
Can someone figure out what is the problem here? Why is it failing when the rpm size is more than 25 MB.
Is there any other way i can extract the rpm contents from python?
Your output pipe is full. The python docs note in many places not to do what you are doing:
Do not use stdout=PIPE or stderr=PIPE with this function as that can deadlock based on the child process output volume. Use Popen with the communicate() method when you need pipes.
If all you want is the payload of a *.rpm package, then do the computations to find the beginning of the compressed cpio payload and do the operations directly in python.
See How do I extract the contents of an rpm? for a rpm2cpio.sh shell script that documents the necessary computations. The only subtlety is ensuring that the padding (needed for alignment) between the signature and metadata headers is correct.
Related
I have a web project,use nginx ,uwsgi ,web.py ,nginx is used for load balancing,uwsgi is used as a web server,web.py is a web frameworks
I start it with this commond "/usr/local/bin/uwsgi -d /home/sheng/www/lr-server/../log/lr-server/uwsgi.log -s 127.0.0.1:8666 -w rc_main -t 20 -M -p 20 --pidfile /home/sheng/www/lr-server/master.pid --enable-threads -R 800"
this commond means it will produce twenty process to receive request, and each process receive 800 requests as most
as show below is normal process
sheng 12414 15051 21 10:04 ? 00:01:45 /usr/local/bin/uwsgi -d /home/sheng/www/lr-server/../log/lr-server/uwsgi.log -s 127.0.0.1:8666 -w rc_main -t 20 -M -p 20 --pidfile /home/sheng/www/lr-server/master.pid --enable-threads -R 800
15051 is parent pid
usually it works normal ,but it will produce strange process when server is very busy and many requests take a long time, as shown below is strange process:
sheng 23370 1 0 09:08 ? 00:00:00 /usr/local/bin/uwsgi -d /home/sheng/www/lr-server/../log/lr-server/uwsgi.log -s 127.0.0.1:8666 -w rc_main -t 20 -M -p 20 --pidfile /home/sheng/www/lr-server/master.pid --enable-threads -R 800
you will notice,this process's pid is 23370,but it's parent pid is 1,it like a defunct process.However, in fact，this process take up memory and will not receive any request
I had hoped to produce 20 normal process to receive request,but now ,it process more then 80 starnge process,who can tell me why and what can I do to solve this problem
I have found out the reason
In my python project , sometimes python will use R to make a scientific calculation
R is slow ,so a colleague of mine use parallel technology,R will fork some child process to calculate
Unfortunately，if a request take more than 20 seconds ,uwsgi will kill the python process,but due to unknown reasons, those child process which R forked will not be killed ,they are just strange process what I see
I use the following snippet in a larger Python program to spawn a process in background:
import subprocess
command = "/media/sf_SharedDir/FOOBAR"
subprocess.Popen(command, shell=True)
After that I wanted to check whether the process was running when my Python program returned.
Output of ps -ef | grep -v grep | grep FOOBAR:
ap 3396 937 0 16:08 pts/16 00:00:00 /bin/sh -c /media/sf_SharedDir/FOOBAR
ap 3397 3396 0 16:08 pts/16 00:00:00 /bin/sh /media/sf_SharedDir/FOOBAR
I was surprised to see two lines of and they have differend PIDs so are those two processes running? Is there something wrong with my Popen call?
FOOBAR Script:
#!/bin/bash
while :
do
echo "still alive"
sleep 1
done
EDIT: Starting the script in a terminal ps displayes only one process.
Started via ./FOOBAR
ap#VBU:/media/sf_SharedDir$ ps -ef | grep -v grep | grep FOOBAR
ap 4115 3463 0 16:34 pts/5 00:00:00 /bin/bash ./FOOBAR
EDIT: shell=True is causing this issue (if it is one). But how would I fix that if I required shell to be True to run bash commands?
There is nothing wrong, what you see is perfectly normal. There is no "fix".
Each of your processes has a distinct function. The top-level process is running the python interpreter.
The second process, /bin/sh -c /media/sf_SharedDir/FOOBAR' is the shell that interprets the cmd line (because you want | or * or $HOME to be interpreted, you specified shell=True).
The third process, /bin/sh /media/sf_SharedDir/FOOBAR is the FOOBAR cmd. The /bin/sh comes from the #! line inside your FOOBAR program. If it were a C program, you'd just see /media/sf_SharedDir/FOOBAR here. If it were a python program, you'd see /usr/bin/python/media/sf_SharedDir/FOOBAR.
If you are really bothered by the second process, you could modify your python program like so:
command = "exec /media/sf_SharedDir/FOOBAR"
subprocess.Popen(command, shell=True)
I'm running python on linux, and when I run my script with:
os.system('v4l2-ctl -c exposure_auto=1')
I get this:
VIDIOC_S_EXT_CTRLS: failed: Input/output error
exposure_auto: Input/output error
When I run this command from terminal with my default user, no output/error appears.
Why is this failing when running the script, but not in terminal?
Edit: Corrected the code and error output.
When a program like this dies with a mysterious error, it means that something about its environment when run beneath Python is subtly different, in a way that matters to the special IO calls that it is making. The question is: what could possibly be different? I just, as a test, ran the basic cat command from a shell — letting it sit there so that I could inspect its state before pressing Control-D to exit it again — and then ran it from the os.system() function in Python. In both cases, lsof shows that it has exactly the same files open and terminal connections made:
$ lsof -p 7573
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
cat 7573 brandon cwd DIR 0,24 45056 131082 /home/brandon
cat 7573 brandon rtd DIR 8,2 4096 2 /
cat 7573 brandon txt REG 8,2 46884 661873 /bin/cat
cat 7573 brandon mem REG 8,2 2919792 393288 /usr/lib/locale/locale-archive
cat 7573 brandon mem REG 8,2 1779492 270509 /lib/i386-linux-gnu/libc-2.17.so
cat 7573 brandon mem REG 8,2 134376 270502 /lib/i386-linux-gnu/ld-2.17.so
cat 7573 brandon 0u CHR 136,19 0t0 22 /dev/pts/19
cat 7573 brandon 1u CHR 136,19 0t0 22 /dev/pts/19
cat 7573 brandon 2u CHR 136,19 0t0 22 /dev/pts/19
In your case the command might run and exit so quickly that it is hard for you to catch it in mid-run with lsof to see what it looks like. Really, what you need to do is run it both ways under strace and figure out which system call is failing, and why.
strace -o trace-good.log v4l2-ctl -c exposure_auto=1
trace-bad.log python my-script.py # that has strace -o in its system() call!
The logs will be long, but using grep on them, or opening them in your less pager and using / and ? to search back and forth (and n and N to keep searching once you have entered a search phrase with / or ?), can help you jump around really quickly.
Look near the bottom of trace-bad.log for the system call that is actually giving the error. Then look in trace-good.log for the same call when it succeeds and post the difference here for us.
Context
I'm adding a few pieces to an existing, working system.
There is a control machine (a local Linux PC) running some test scripts which involve sending lots of commands to several different machines remotely via SSH. The test framework is written in Python and uses Fabric to access the different machines.
All commands are handled with a generic calling function, simplified below:
def cmd(host, cmd, args):
...
with fabric.api.settings(host_string=..., user='root', use_ssh_config=True, disable_known_hosts=True):
return fabric.api.run('%s %s' % (cmd, args))
The actual commands sent to each machine usually involve running an existing python script on the remote side. Those python scripts, do some jobs which include invoking external commands (using system and subprocess). The run() command called on the test PC will return when the remote python script is done.
At one point I needed one of those remote python scripts to launch a background task: starting an openvon server and client using openvpn --config /path/to/config.openvpn. In a normal python script I would just use &:
system('openvpn --config /path/to/config.openvpn > /var/log/openvpn.log 2>&1 &')
When this script is called remotely via Fabric, one must explicitly use nohup, dtach, screen and the likes to run the job in background. I got it working with:
system("nohup openvpn --config /path/to/config.openvpn > /var/log/openvpn.log 2>&1 < /dev/null &"
The Fabric FAQ goes into some details about this.
It works fine for certain background commands.
Problem: doesn't work for all types of background commands
This technique doesn't work for all the commands I need. In some scripts, I need to launch a background atop command (it's a top on steroids) and redirect its stdout to a file.
My code (note: using atop -P for parseable output):
system('nohup atop -P%s 1 < /dev/null | grep %s > %s 2>&1 &' % (dataset, grep_options, filename))
When the script containing that command is called remotely via Fabric, the atop process is immediately killed. The output file is generated but it's empty. Calling the same script while logged in the remote machine by SSH works fine, the atop command dumps data periodically in my output file.
Some googling and digging around brought me to interesting information about background jobs using Fabric, but my problem seems to be only specific to certains types of background jobs. I've tried:
appending sleep
running with pty=False
replacing nohup with dtach -n: same symptoms
I read about commands like top failing in Fabric with stdin redirected to /dev/null, not quite sure what to make of it. I played around with different combinations or (non-) redirects of STDIN, STDOUT and STDERR
Looks like I'm running out of ideas.
Fabric seems overkill for what we are doing. We don't even use the "fabfile" method because it's integrated in a nose framework and I run them invoking nosetests. Maybe I should resort to dropping Fabric in favor of manual SSH commands, although I don't like the idea of changing a working system because of it not supporting one of my newer modules.
In my environment, looks like it is working
from fabric.api import sudo
def atop():
sudo('nohup atop -Pcpu 1 </dev/null '
'| grep cpu > /tmp/log --line-buffered 2>&1 &',
pty=False)
result:
fabric:~$ fab atop -H web01
>>>[web01] Executing task 'atop'
>>>[web01] sudo: nohup atop -Pcpu 1 </dev/null | grep cpu > /tmp/log --line-buffered 2>&1 &
>>>
>>>Done.
web01:~$ cat /tmp/log
>>>cpu web01 1374246222 2013/07/20 00:03:42 361905 100 0 5486 6968 0 9344927 3146 0 302 555 0 2494 100
>>>cpu web01 1374246223 2013/07/20 00:03:43 1 100 0 1 0 0 99 0 0 0 0 0 2494 100
>>>cpu web01 1374246224 2013/07/20 00:03:44 1 100 0 1 0 0 99 0 0 0 0 0 2494 100
...
The atop command may need the super user. This doesn't work
from fabric.api import run
def atop():
run('nohup atop -Pcpu 1 </dev/null '
'| grep cpu > /tmp/log --line-buffered 2>&1 &',
pty=False)
On the other hand this work.
from fabric.api import run
def atop():
run('sudo nohup atop -Pcpu 1 </dev/null '
'| grep cpu > /tmp/log --line-buffered 2>&1 &',
pty=False)
I am working on a script in python where first I set ettercap to ARP poisoning and then start urlsnarf to log the URLs. I want to have ettercap to start first and then, while poisoning, start urlsnarf. The problem is that these jobs must run at the same time and then urlsnarf show the output. So I thought it would be nice If I could run ettercap in background without waiting to exit and then run urlsnarf. I tried command nohup but at the time that urlsnarf had to show the url the script just ended. I run:
subprocess.call(["ettercap",
"-M ARP /192.168.1.254/ /192.168.1.66/ -p -T -q -i wlan0"])
But I get:
ettercap NG-0.7.4.2 copyright 2001-2005 ALoR & NaGA
MITM method ' ARP /192.168.1.254/ /192.168.1.66/ -p -T -q -i wlan0' not supported...
Which means that somehow the arguments were not passed correctly
You could use the subprocess module in the Python standard library to spawn ettercap as a separate process that will run simultaneously with the parent. Using the Popen class from subprocess you'll be able to spawn your ettercap process run your other processing and then kill the ettercap process when you are done. More info here: Python Subprocess Package
import shlex, subprocess
args = shlex.split("ettercap -M ARP /192.168.1.254/ /192.168.1.66/ -p -T -q -i wlan0")
ettercap = subprocess.Popen(args)
# program continues without waiting for ettercap process to finish.