We Keep Coding

How to call curl command from python

I have a command as below
Result = os.open(“curl -u username:password https://hostname/stafftools/reports/all_users.csv -k”).read()
It’s returning nothing.
can someone help me out?

You should use the subprocess module:
import subprocess
result = subprocess.call(["curl", "-u", "username:password", "https://hostname/stafftools/reports/all_users.csv", "-k"])
Find more information here.

run shell command on a cluster using python

I want/am trying to run shell command(Usually file manipulation) on a cluster(a spark cluster 1 master and 3 worker node).
There is passwordless ssh between all the machines in the cluster.
File directories are all same on all cluster Nodes.
Currently I am handling file manipulation shell command by
#let's say copy or move a file from one dir to other dir
import os, sys
os.system('ssh user#Ip_of_worker-1 "cp directory_1/file1.csv directory_2"')
os.system('ssh user#Ip_of_worker-2 "cp directory_1/file1.csv directory_2"')
os.system('ssh user#Ip_of_worker-3 "cp directory_1/file1.csv directory_2"')
And I am looking for a python package to do that, generally I am trying to avoid system call every time I want to run a shell command(I should get stdout & stderr for each command run on different cluster_Nodes in running python script log.).
And shell command should run in parallel/simultaneously on all target Nodes.
Please guide, if any such package you guys are aware of or have used before.

You could use a library implementing ssh protocol, for example paramiko, if you are not happy with system or subprocess. http://docs.paramiko.org/en/2.1/
Hannu

Try to look for pdsh and call this using python.
https://linux.die.net/man/1/pdsh
Example
http://www.linux-magazine.com/Issues/2014/166/Parallel-Shells

It sounds like you want Fabric - http://www.fabfile.org/
From their basic example:
from fabric.api import run
def host_type():
run('uname -s')
Gets you:
$ fab -H localhost,linuxbox host_type
[localhost] run: uname -s
[localhost] out: Darwin
[linuxbox] run: uname -s
[linuxbox] out: Linux

You can do something like this:
#!/usr/bin/python
import thread
import subprocess
# Define a function for the thread
def run_remote(host, delay):
remote_cmd='cp directory_1/file1.csv directory_2'
ssh = subprocess.Popen(['ssh', '-oStrictHostKeyChecking=no', host, remote_cmd],
shell=False,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
result = ssh.stdout.readlines()
if result == []:
error = ssh.stderr.readlines()
print "ERROR: %s" % error
else:
print result
# Create two threads as follows
try:
thread.start_new_thread( run_remote, ("Ip_of_worker-1", 1, ) )
thread.start_new_thread( run_remote, ("Ip_of_worker-2", 1, ) )
thread.start_new_thread( run_remote, ("Ip_of_worker-3", 1, ) )
except:
print "Error: unable to start thread"

parallel-ssh is a non-blocking parallel ssh client that can do this:
from pssh.pssh2_client import ParallelSSHClient
client = ParallelSSHClient(['host1', 'host2']
output = client.run_command('cp directory_1/file1.csv directory_2')
client.join(output)

How to redirect command output using os.execvp() in python

I am invoking shell script using os.execvp() in python. my shell script has some echo statements whcih I want to redirect in file.
Here is what I am trying:
cmd = "/opt/rpm/rpm_upgrade.sh >& /opt/rpm/upgrader.log"
cmdline = ["/bin/sh", cmd]
os.execvp(cmdline[0], cmdline)
Below is the error I am getting:
Error: /bin/sh: /opt/rpm/rpm_upgrade.sh >& /opt/rpm/upgrader.log: No such file or directory
Can any one help?

This is happening because you are passing this entire string as if it were the program name to execute:
"/opt/rpm/rpm_upgrade.sh >& /opt/rpm/upgrader.log"
The easy way to fix this is:
cmdline = ["/bin/sh", "/opt/rpm/rpm_upgrade.sh",
">&", "/opt/rpm/upgrader.log"]
os.execvp(cmdline[0], cmdline)
Now sh will receive three arguments rather than one.
Or you can switch to the more full-featured subprocess module, which lets you redirect output in Python:
import subprocess
with open("/opt/rpm/upgrader.log", "wb") as outfile:
subprocess.check_call(["/opt/rpm/rpm_upgrade.sh"], shell=True,
stdout=outfile, stderr=subprocess.STDOUT)

subprocess module in python

I have to connect to a sybase database and run a simple select query using python script
On my server isql command can run only from sybase bin directory, so i have to cd to that directory before firing the query.
---------------------------Edited-----------------------------
Uptill now i'm able to do this:-
#!/usr/bin/python
import subprocess
path = "path/to/sybase/bin"
os.chdir(path)
arguments = ['./isql',"-S server_name", "-U user", "-P password", "-D database","""<<EOF
SELECT * FROM sometable
go
EOF"""]
ps = subprocess.Popen(arguments)
out = ps.communicate()
print out
The errors are just out of my understanding capability :(
Traceback (most recent call last):
File "./test_db.py", line 8, in ?
ps = subprocess.Popen(arguments)
File "/usr/lib64/python2.4/subprocess.py", line 542, in __init__
errread, errwrite)
File "/usr/lib64/python2.4/subprocess.py", line 975, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
I'm able to do this outside my python script using isql command on my unix terminal
how can i use isql in python subprocess module?

There is a particular Popen argument for that: cwd, as mentioned here. Provide your command using an array and then the cwd parameter with where the command must be executed:
subprocess.Popen(['ls', '-l'], cwd="/path/to/folder")

Popen only takes one args argument, for the command to run. You could try calling a shell with both the cd and isql commands as arguments, but changing the working directory from python is probably simpler
For the former approach:
subprocess.Popen('/bin/sh -c "cd /path/to/... && isql -arg1..'...)
for the latter:
os.chdir('/path/to...')
subprocess.Popen('isql -arg1..'...)

Try:
import os
import subprocess
os.chdir('/path/to/sybase/bin')
if os.path.exists('isql') or os.path.exists(os.path.join('/path/to/sybase/bin', 'isql')):
ps = subprocess.Popen('isql -S %s -U %s -P %s -D %s <<EOF SELECT * FROM sometable EOF' % (server,user,passwd,database), stdout=subprocess.PIPE, shell=True)
out, err = ps.communicate()
else:
print "isql does not exists in this folder"
I am not super experienced with subprocess but this is how I generally use it on the odd occasion. Hopefully someone else can give a better answer/explanation.
Edit: removed the square brackets to remove confusion.

i know it's been long but just wanted to close this question
from subprocess import Popen, PIPE
from textwrap import dedent
isql = Popen(['./isql', '-I', '/app/sybase/...',
'-S', mdbserver,
'-U', muserid,
'-P', password, ...,
'-w', '99999'], stdin=PIPE, stdout=PIPE, cwd=sybase_path)
output = isql.communicate(dedent("""\
SET NOCOUNT ON
{}
go
""".format(User_Query)))[0]

commands.getstatusoutput fails saying sh: Syntax error: ";" unexpected

I am running mongoimport command using python commands module as
status = utilities.execute(mongoimport)
in utilities.py
def execute(command):
if not command:
return (-1, 'command can not be empty or null')
return commands.getstatusoutput(command)
When I run this, I see error as
sh: Syntax error: ";" unexpected
I see that documentation says :
commands.getstatusoutput(cmd)
Execute the string cmd in a shell with os.popen() and return a 2-tuple (status, output). cmd is actually run as { cmd ; } 2>&1, so that the returned output will contain output or error messages
How can I fix this to be able to run this command?

Use the subprocess module
from subprocess import check_output
output = check_output(["ls", "-l"])
This will raise an error if the command fails - no need to check for empty string. If you are really sure that you want to pass stuff through the shell then call like this
output = check_output("ls -l", shell=True)
Just note that passing stuff through the shell is an excellent vector for security problems.