We Keep Coding

Passing a list as variable to python script from bash script

I have a python script like below.
# Import necessary libraries and functions
import sys
import traceback
y = '2020'
querysting = "select {} from table where Year={}".format(col_list,y)
df = pd.read_sql(querystring,db)
if __name__ == '__main__':
if len(sys.argv) != 8:
print('Invalid number of args......')
print('Usage: file.py Input_path Output_path')
exit()
_, col_list, FinalDB, final_table, host, dsl_tbl_name, username, password = tuple(sys.argv)
data_load(col_list, FinalDB, final_table, host, tbl_name, username, password)
Now I am calling this python script inside a shell script like below
#!/bin/bash
col_list='id, name, address, phone_number'
FinalDB='abc'
final_table='xyz'
host='xxxxxx.xxxx'
tbl_name='test'
username='USER'
password='test_pass'
python python_script.py ${col_list} ${FinalDB} ${final_table} ${host} ${tbl_name} ${username} ${password}
Now when I run this bash script I am getting Invalid number of args...... error
I am pretty much sure that it is some thing to do with col_list variable being passed to the python script.
Because instead of having columns in a list if I just pass select * in the query and removing col_list variable then I don't get any errors
What am I doing wrong here and how can I resolve this issue.

The problem comes from how you pass your variables that contains spaces from Bash to Python.
You may note that:
In shell scripts command line arguments are separated by
whitespace, unless the arguments are quoted.
Let's have this simple Python script:
python_script.py:
import sys
if __name__ == '__main__':
print(sys.arv)
Then in your Terminal:
$> export var1="hello"
$> python python_script.py $var1
['python_script.py', 'hello']
However:
$> export var1="hello, hi"
$> python python_script.py $var1
['python_script.py', 'hello,', 'hi'] # Note Here: Two args were passed
$> export var1="hello,hi"
$> python python_script.py $var1
['python_script.py', 'hello,hi'] # One arg passed
So, a solution to your problem, is to pass your Bash args as a string even with spaces, like this example:
$> export var1="hello, hi, hola"
$> python python_script.py "$var1"
['python_script.py', 'hello, hi, hola']
For more informations see this article

Download files via curl using python's subprocess

How is it that when I write this in my terminal the file is downloaded,
curl "http://nomads.ncep.noaa.gov/cgi-bin/filter_gfs_0p50.pl?file=gfs.t00z.pgrb2full.0p50.f000&lev_10_m_above_ground=on&var_UGRD=on&var_VGRD=on&leftlon=0&rightlon=360&toplat=90&bottomlat=-90&dir=%2Fgfs.2016121900" -o "tmp_folder/gfs.t00z.pgrb2full.0p50.f000"
but using python's subprocess module, the download just hangs?
import subprocess
URL = "http://nomads.ncep.noaa.gov/cgi-bin/filter_gfs_0p50.pl?file=gfs.t00z.pgrb2full.0p50.f000&lev_10_m_above_ground=on&var_UGRD=on&var_VGRD=on&leftlon=0&rightlon=360&toplat=90&bottomlat=-90&dir=%2Fgfs.2016121900"
pipe = subprocess.Popen("curl " + URL + " -o" + " my_file", shell = True)
pipe.communicate()
What am I missing?
Thank you

The URL is probably not properly quoted, so it is interpreted by the shell (with all the & chars...)
Better run subprocess with explicit arguments as a list:
pipe = subprocess.Popen(["curl",URL,"-o","my_file"])
shell=True can probably be omitted. Since you're not using any shell capabilities, it will save you trouble.

rather than creating a Popen instance, you can just use the call method. Don't pass shell=True... just split the command with shlex so the args can be passed as a list.
import shlex
import subprocess
cmd = 'curl "http://foo.com" -o "foo.txt"'
subprocess.call(shlex.split(cmd))

Want to run a curl command either through python's subprocess or pycurl

I have the following command which adds a user to the administrator group of a gerrit instance,
curl -X POST -H "Content-Type:application/json;charset=UTF-8" -u nidhi:pswd http://host_ip:port/a/groups/Administrators/members.add -d '{"members":["user#example.com"]}'
When I run this command on my terminal, it runs perfectly and gives the expected output.
But, I want to execute this command in python either using the subprocess or pycurl library.
Using subprocess I wrote the following code,
def add_user_to_administrator(u_name,url):
bashCommand = 'curl -X POST -H "Content-Type:application/json;charset=UTF-8" -u nidhi:pswd http://'+url+'/a/groups/Administrators/members.add -d '+"'"+'{"members":["$u_n#example.com"]}'+"'"
bashCommand = string.Template(bashCommand).substitute({'u_n':u_name})
print bashCommand.split()
process = subprocess.Popen(bashCommand.split())
It shows no error but no changes are seen in the administrator group.
I tried the same using pycurl,
def add_user_to_administrator2(u_name,url):
pf = json.dumps({"members":[str(str(u_name)+"#example.com")]})
headers = ['Content-Type:application/json;charset=UTF-8']
pageContents = StringIO.StringIO()
p = pycurl.Curl()
p.setopt(pycurl.FOLLOWLOCATION, 1)
p.setopt(pycurl.POST, 1)
p.setopt(pycurl.HTTPHEADER, headers)
p.setopt(pycurl.POSTFIELDS, pf)
p.setopt(pycurl.WRITEFUNCTION, pageContents.write)
p.setopt(pycurl.VERBOSE, True)
p.setopt(pycurl.DEBUGFUNCTION, test)
p.setopt(pycurl.USERPWD, "nidhi:pswd")
pass_url=str("http://"+url+"/a/groups/Administrators/Administrators/members.add").rstrip('\n')
print pass_url
p.setopt(pycurl.URL, pass_url)
p.perform()
p.close()
pageContents.seek(0)
print pageContents.readlines()
This throws an error, it cannot find the account members.
The variable mentioned url is of the form host_ip:port.
I have tried a lot to fix these errors. I dont know where I am going wrong. Any help would be appreciated.

a) string escaping
For the subprocess/curl usage, you should be escaping your string tokens rather than manually adding extra ':
...stuff'+"'"+'more.stuff...
Escape using \ before the character i.e. using
"curl -X POST -H \"Content-Type:application/json;charset=UTF-8\""
will keep the " around the Content-Type section.
More on escape characters here: Lexical Analysis - String Literals
...The backslash () character is used to escape characters that otherwise have a special meaning...
b) popen
Looking at the popen docs their example uses shlex.split() to split their command line into args. shlex splits the string a bit differently:
print(bashCommand.split())
['curl', '-X', 'POST', '-H', '"Content-Type:application/json;charset=UTF-8"', '-u', 'nidhi:pswd', 'http://TEST_URL/a/groups/Administrators/members.add', '-d', '\'{"members":["TEST_USER#example.com"]}\'']
print(shlex.split(bashCommand))
['curl', '-X', 'POST', '-H', 'Content-Type:application/json;charset=UTF-8', '-u', 'nidhi:pswd', 'http://TEST_URL/a/groups/Administrators/members.add', '-d', '{"members":["TEST_USER#example.com"]}']
you can see shlex removes excess quoting.
c) http response code
Try using -I option in curl to get a HTTP response code back (and the rest of the HTTP headers):
$curl -h
...
-I, --head Show document info only
Even though you're using subprocess to start/make the request, it should still print the return value to the console(stdout).
d) putting it all together
I changed the how the url and u_name are interpolated into the string.
import shlex
import subprocess
def add_user_to_administrator(u_name, url):
bashCommand = "curl -I -X POST -H \"Content-Type:application/json;charset=UTF-8\" -u nidhi:pswd http://%(url)s/a/groups/Administrators/members.add -d '{\"members\":[\"%(u_n)s#example.com\"]}'"
bashCommand = bashCommand % {'u_n': u_name, 'url': url}
args = shlex.split(bashCommand)
process = subprocess.Popen(args)
add_user_to_administrator('TEST_USER', 'TEST_URL')
If none of this helps, and you're getting no response from gerrit, I'd check gerrit logs to see what happens when it receives your request.

you should try [urllib2] (python2) or urllib(python3) to post json data ;
for subprocess.Popen : subprocess.Popen.communicate(https://docs.python.org/3.5/library/subprocess.html?highlight=subprocess#subprocess.Popen.communicate) may give you help,or just exec bashCommand in shell to see the difference;
for pycurl , I have not use it ,but please you add error info.

How to check whether a shell command returned nothing or something

I am writing a script to extract something from a specified path. I am returning those values into a variable. How can i check whether the shell command has returned something or nothing.
My Code:
def any_HE():
global config, logger, status, file_size
config = ConfigParser.RawConfigParser()
config.read('config2.cfg')
for section in sorted(config.sections(), key=str.lower):
components = dict() #start with empty dictionary for each section
#Retrieving the username and password from config for each section
if not config.has_option(section, 'server.user_name'):
continue
env.user = config.get(section, 'server.user_name')
env.password = config.get(section, 'server.password')
host = config.get(section, 'server.ip')
print "Trying to connect to {} server.....".format(section)
with settings(hide('warnings', 'running', 'stdout', 'stderr'),warn_only=True, host_string=host):
try:
files = run('ls -ltr /opt/nds')
if files!=0:
print '{}--Something'.format(section)
else:
print '{} --Nothing'.format(section)
except Exception as e:
print e
I tried checking 1 or 0 and True or false but nothing seems to be working. In some servers, the path '/opt/nds/' does not exist. So in that case, nothing will be there on files. I wanted to differentiate between something returned to files and nothing returned to files.

First, you're hiding stdout.
If you get rid of that you'll get a string with the outcome of the command on the remote host. You can then split it by os.linesep (assuming same platform), but you should also take care of other things like SSH banners and colours from the retrieved outcome.

As perror commented already, the python subprocess module offers the right tools.
https://docs.python.org/2/library/subprocess.html
For your specific problem you can use the check_output function.
The documentation gives the following example:
import subprocess
subprocess.check_output(["echo", "Hello World!"])
gives "Hello World"

plumbum is a great library for running shell commands from a python script. E.g.:
from plumbum.local import ls
from plumbum import ProcessExecutionError
cmd = ls['-ltr']['/opt/nds'] # construct the command
try:
files = cmd().splitlines() # run the command
if ...:
print ...:
except ProcessExecutionError:
# command exited with a non-zero status code
...
On top of this basic usage (and unlike the subprocess module), it also supports things like output redirection and command pipelining, and more, with easy, intuitive syntax (by overloading python operators, such as '|' for piping).

In order to get more control of the process you run, you need to use the subprocess module.
Here is an example of code:
import subprocess
task = subprocess.Popen(['ls', '-ltr', '/opt/nds'], stdout=subprocess.PIPE)
print task.communicate()

What does this Perl XML filter look like in Python?

curl -u $1:$2 --silent "https://mail.google.com/mail/feed/atom" | perl -ne 'print "\t" if /<name>/; print "$2\n" if /<(title|name)>(.*)<\/\1>/;'
I have this shell script which gets the Atom feed with command-line arguments for the username and password. I was wondering if this type of thing was possible in Python, and if so, how I would go about doing it. The atom feed is just regular XML.

Python does not lend itself to compact one liners quite as well as Perl. This is primarily for three reasons:
With Perl, whitespace is insignificant in almost all cases. In Python, whitespace is very significant.
Perl has some helpful shortcuts for one liners, such as perl -ne or perl -pe that put an implicit loop around the line of code.
There is a large body a cargo-cult Perl one liners to do useful things.
That all said, this python is close to what you posted in Perl:
curl -u $1:$2 --silent "https://mail.google.com/mail/feed/atom" | python -c '
import sys
for s in sys.stdin:
s=s.strip()
if not s: print '\t',
else: print s
'
It is a little difficult to do better because, as stated in my comment, the Perl you posted is incomplete. You have:
perl -ne 'print "\t" if //; print "$2\n" if /(.*)/;'
Which is equivalent to:
LINE:
while (<>) {
print "\t" if //; # print a tab for a blank line
print "$2\n" if /(.*)/; # nonsensical. Print second group but only
# a single match group defined...
}
Edit
While it is trivial to rewrite that Perl in Python, here is something a bit better:
#!/usr/bin/python
from xml.dom.minidom import parseString
import sys
def get_XML_doc_stdin(f):
return xml.dom.minidom.parse(f)
def get_tagged_data2(tag, index=0):
xmlData = dom.getElementsByTagName(tag)[index].firstChild.data
return xmlData
data=sys.stdin.read()
dom = parseString(data)
ele2=get_tagged_data2('title')
print ele2
count=int(get_tagged_data2('fullcount'))
print count,"New Messages:"
for i in range(0,count):
nam=get_tagged_data2('name',i)
email=get_tagged_data2('email',i)
print " {0}: {1} <{2}>".format(i+1,nam,email)
Now save that in a text file, run chmod +x on it, then:
curl -u $1:$2 --silent "https://mail.google.com/mail/feed/atom" |
/path/pythonfile.py
It produces this:
Gmail - Inbox for xxxxxxx#gmail.com
2 New Messages:
1: bob smith <bob#smith.com>
2: Google Alerts <googlealerts-noreply#google.com>
edit 2
And if you don't like that, here is the Python 1 line filter:
curl -u $1:$2 --silent "https://mail.google.com/mail/feed/atom" |python -c '
import sys, re
for t,m in re.findall(r"<(title|name)>(.*)<\/\1>",sys.stdin.read()):
print "\t",m
'

You may use an "URL opener" from the urllib2 standard Python module with a handler for authentication. For example:
#!/usr/bin/env python
import getpass
import sys
import urllib2
def main(program, username=None, password=None, url=None):
# Get input if any argument is missing
username = username or raw_input('Username: ')
password = password or getpass.getpass('Password: ')
url = url or 'https://mail.google.com/mail/feed/atom'
# Create password manager
password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
password_mgr.add_password(None, url, username, password)
# Create HTTP Authentication handler and URL opener
authhandler = urllib2.HTTPBasicAuthHandler(password_mgr)
opener = urllib2.build_opener(authhandler)
# Fetch URL and print content
response = opener.open(url)
print response.read()
if __name__ == '__main__':
main(*sys.argv)
If you'd like to extract information from the feed too, you should check how to parse Password-Protected Feeds with feedparser.