We Keep Coding

How to print the first N lines of a file in python with N as argument

How would I go about getting the first N lines of a text file in python? With N have to give as argument
usage:
python file.py datafile -N 10
My code
import sys
from itertools import islice
args = sys.argv
print (args)
if args[1] == '-h':
print ("-N for printing the number of lines: python file.py datafile -N 10")
if args[-2] == '-N':
datafile = args[1]
number = int(args[-1])
with open(datafile) as myfile:
head = list(islice(myfile, number))
head = [item.strip() for item in head]
print (head)
print ('\n'.join(head))
I wrote the program, can let me know better than this code

Assuming that the print_head logic you've implemented need not be altered, here's the script I think you're looking for:
import sys
from itertools import islice
def print_head(file, n):
if not file or not n:
return
with open(file) as myfile:
head = [item.strip() for item in islice(myfile, n)]
print(head)
def parse_args():
result = {'script': sys.argv[0]}
args = iter(sys.argv)
for arg in args:
if arg == '-F':
result['filename'] = next(args)
if arg == '-N':
result['num_lines'] = int(next(args))
return result
if __name__ == '__main__':
script_args = parse_args()
print_head(script_args.get('filename', ''), script_args.get('num_lines', 0))
Running the script
python file.py -F datafile -N 10
Note: The best way to implement it would be to use argparse library

You can access argument passed to the script through sys
sys.argv
The list of command line arguments passed to a Python script. argv[0] is the script name (it is operating system dependent whether this is a full pathname or not). If the command was executed using the -c command line option to the interpreter, argv[0] is set to the string '-c'. If no script name was passed to the Python interpreter, argv[0] is the empty string.
So in code it would look like this:
import sys
print("All of argv")
print(sys.argv)
print("Last element every time")
print(sys.argv[-1])
Reading the documentation you'll see that the first values stored in the sys.argv vary according to how the user calls the script. If you print the code I pasted with different types of calls you can see for yourself the kind of values stored.
For a basic first approach: access n through sys.argv[-1] which returns the last element every time, assuming. You still have to do a try and beg for forgiveness to make sure the argument passed is a number. For that you would have:
import sys
try:
n = int(sys.argv[-1])
except ValueError as v_e:
print(f"Please pass a valid number as argument, not ${sys.argv[-1]}")
That's pretty much it. Obviously, it's quite basic, you can improve this even more by having the users pass values with flags, like --skip-lines 10 and that would be your n, and it could be in any place when executing the script. I'd create a function in charge of translating sys.argv into a key,value dictionary for easy access within the script.

Arguments are available via the sys package.
Example 1: ./file.py datafile 10
#!/usr/bin/env python3
import sys
myfile = sys.argv[1]
N = int(sys.argv[2])
with open("datafile") as myfile:
head = myfile.readlines()[0:args.N]
print(head)
Example 2: ./file.py datafile --N 10
If you want to pass multiple optional arguments you should have a look at the argparse package.
#!/usr/bin/env python3
import argparse
parser = argparse.ArgumentParser(description='Read head of file.')
parser.add_argument('file', help='Textfile to read')
parser.add_argument('--N', type=int, default=10, help='Number of lines to read')
args = parser.parse_args()
with open(args.file) as myfile:
head = myfile.readlines()[0:args.N]
print(head)

unit tests for command line args python

I have a shell script which currently takes 3 args. I run this via a shell script with the shell script file name , a directory to run the python script upon , along with the name of the test data directory. I want to be able to write a unit tests which executes the command below but only if i was to change the date , depending on the data that is available it would either pass or fail.
main_config.sh
yamldir=$1
for yaml in $(ls ${yamldir}/*.yaml | grep -v "export_config.yaml"); do
if [ "$yaml" != "export_config.yaml" ]; then
echo "Running export for $yaml file...";
python valid.py -p ${yamldir}/export_config.yaml -e $yaml -d ${endDate}
wait
fi
done
This is what is executed on the command line
./main_config.sh /Users/name/Desktop/yaml/ 2018-12-23
This will fail and output on the terminal since there is no directory called 2012-12-23 :
./main_config.sh /yaml/ 2018-12-23
Running export for apa.yaml file...
apa.json does not exist
If the directory existed this would pass and would output on the terminal :
Running export for apa.yaml file...
File Name: apa.json Exists
File Size: 234 Bytes
Writing to file
My python script script is as follows :
def main(get_config):
cfg = get_config()[0] # export_config.yaml
data = get_config()[1] # export_apa.yaml
date = get_config()[2] # data folder - YYYY-MM-DD
# Conditional Logic
def get_config():
parser = argparse.ArgumentParser()
parser.add_argument("-p", "--parameter-file", action="store", required=True)
parser.add_argument("-e", "--export-data-file", action="store", required=True)
parser.add_argument("-d", "--export-date", action="store", required=False)
args = parser.parse_args()
return [funcs.read_config(args.parameter_file), funcs.read_config(args.export_data_file), args.export_date]
if __name__ == "__main__":
logging.getLogger().setLevel(logging.INFO)
main(get_config)

To me it looks like this is not a typical unit test (that tests a function or method) but an integration test (that tests a subsystem from the outside). But of course you could still solve this with your typical Python testing tools like unittest.
A simple solution would be to run your script using subprocess, capture the output, and then parse that output as part of your test:
import unittest
import os
import sys
if os.name == 'posix' and sys.version_info[0] < 3:
import subprocess32 as subprocess
else:
import subprocess
class TestScriptInvocation(unittest.TestCase):
def setUp(self):
"""call the script and record its output"""
result = subprocess.run(["./main_config.sh", "/Users/yasserkhan/Desktop/yaml/", "2018-12-23"], stdout=subprocess.PIPE)
self.returncode = result.returncode
self.output_lines = result.stdout.decode('utf-8').split('\n')
def test_returncode(self):
self.assertEqual(self.returncode, 0)
def test_last_line_indicates_success(self):
self.assertEqual(self.output_lines[-1], 'Writing to file')
if __name__ == '__main__':
unittest.main()
Note that this code uses the backport of the Python 3 subprocess module. Also, it tries to decode the contents of result.stdout because on Python 3 that would be a bytes object and not a str as on Python 2. I didn't test it, but these two things should make the code portable between 2 and 3.
Also note that using absolute paths like "/Users/yasserkhan/Desktop/yaml" could easily break, so you will either need to find a relative path or pass a base path to your tests using environment variables for example.
You could add additional tests that parse the other lines and check for reasonable outputs like a file size in the expected range.

How to make linux command for the python program using shell script by giving arguments?

For example I have python files like hello.py, add.py, square.py.
Definitions of these files are
hello.py:-
def hello(a):
print a #a is some name
add.py:-
def add(a,b):
print a+b #where a,b are numbers which I have to pass as arguments in command
square.py:-
def square(a):
print a**2 #where 'a' is a number
I want to execute these files from shell script(For example pyshell.sh) and want to make commands like
pyshell --hello name - then it has to execute hello.py
pyshell --add 4 5 - then it has to execute add.py
pyshell --square 2 - then it has to execute square.py
I am trying with this code
#! /usr/bin/python
import argparse
# Create Parser and Subparser
parser = argparse.ArgumentParser(description="Example ArgumentParser")
subparser = parser.add_subparsers(help="commands")
# Make Subparsers
hai_parser = subparser.add_parser('--hai', help='hai func')
hai_parser.add_argument("arg",help="string to print")
hai_parser.set_defaults(func='hai')
args = parser.parse_args()
def hai(arg):
print arg
if args.func == '--hai':
hai(args.arg)
But I am getting an error like
usage: 1_e.py [-h] {--hai} ...
1_e.py: error: invalid choice: 'name' (choose from '--hai')

Here's an example using argparse all in python.
You can run it by with the following:
python pyshell.py hello "well hi"
python pyshell.py add 20 3.4
python pyshell.py square 24
pyshell.py:-
import argparse
# Create Parser and Subparser
parser = argparse.ArgumentParser(description="Example ArgumentParser")
subparser = parser.add_subparsers(help="commands")
# Make Subparsers
hello_parser = subparser.add_parser('hello', help='hello func')
hello_parser.add_argument("arg",help="string to print")
hello_parser.set_defaults(func='hello')
add_parser = subparser.add_parser('add', help="add func")
add_parser.add_argument("x",type=float,help='first number')
add_parser.add_argument("y",type=float,help='second number')
add_parser.set_defaults(func='add')
square_parser = subparser.add_parser('square', help="square func")
square_parser.add_argument("a",type=float,help='number to square')
square_parser.set_defaults(func='square')
args = parser.parse_args()
def hello(arg):
print arg
def add(x,y):
print x + y
def square(a):
print a**2
if args.func == 'hello':
hello(args.arg)
elif args.func == 'add':
add(args.x,args.y)
elif args.func == 'square':
square(args.a)

You can use shebang inside python scripts like #! /usr/bin/python such that those files can be executed like shell script for example python file.py can be executed as file.py if you use shebang. So according to your question you can call those scripts in switch case in shell scripts like this
#! /bin/bash
case ${1:-''} in
"hello")
/path/to/hello $2
;;
"add")
/path/to/add $2 $3
;;
"square")
/path/to/square $2
;;
*)
echo "Invalid option supplied"
exit 1
;;
exit 0
If you don't use shebang in python script add python in front of /path/to/script.py better use shebang in script and use absolute path. Also make sure that the respective scripts has execute permissions.

Python - passing parameters in a command line app

I need to write a command line application, like a shell. So it will include commands etc. The thing is I don't know how to pass parameters to the funcions in a module. For example:
User writes: function1 folder1
Program should now pass the 'folder1' parameter to the function1 function, and run it. But also it has to support other functions with different parameters ex:
User input: function2 folder2 --exampleparam
How to make this to work? I mean, I could just write a module, import it in python and just use the python console, but this is not the case. I need a script that takes command input and runs it.
I tried to use eval(), but that doesn't solve the problem with params. Or maybe it does but I don't see it?

The first part of your problem -- parsing the command line -- can be solved with argparse.
The second part -- converting the string name of a function into a function call -- can be done with exec or a dispatching dict which maps from strings to function objects.
I would recommend NOT using exec for this, since
allowing a user to call arbitrary Python functions from the command line might be dangerous. Instead, make a whitelist of allowable functions:
import argparse
def foo(path):
print('Running foo(%r)' % (path, ))
def bar(path):
print('Running bar(%r)' % (path, ))
dispatch = {
'foo': foo,
'bar': bar,
}
parser = argparse.ArgumentParser()
parser.add_argument('function')
parser.add_argument('arguments', nargs='*')
args = parser.parse_args()
dispatch[args.function](*args.arguments)
% test.py foo 1
Running foo('1')
% test.py bar 2
Running bar('2')
% test.py baz 3
KeyError: 'baz'
The above works when the command is typed into the command-line itself. If the command is being typed into stdin, then we'll need to do something a bit different.
A simple way would be to call raw_input to grab the string from stdin. We could then parse the string with argparse, as we did above:
shmod.py:
import argparse
def foo(path):
print('Running foo(%r)' % (path, ))
def bar(path):
print('Running bar(%r)' % (path, ))
dispatch = {
'foo': foo,
'bar': bar,
}
def parse_args(cmd):
parser = argparse.ArgumentParser()
parser.add_argument('function')
parser.add_argument('arguments', nargs='*')
args = parser.parse_args(cmd.split())
return args
main.py:
import shmod
while True:
cmd = raw_input('> ')
args = shmod.parse_args(cmd)
try:
shmod.dispatch[args.function](*args.arguments)
except KeyError:
print('Invalid input: {!r}'.format(cmd))
Another, more sophisticated way to handle this is to use the cmd module, as #chepner mentioned in the comments.
from cmd import Cmd
class MyInterpreter(Cmd):
prompt = '> '
def do_prompt(self, line):
"Change the interactive prompt"
self.prompt = line + ': '
def do_EOF(self, line):
return True
def do_foo(self, line):
print('Running foo {l}'.format(l=line))
def do_bar(self, line):
print('Running bar {l}'.format(l=line))
if __name__ == '__main__':
MyInterpreter().cmdloop()
For more information on how to use the cmd module, see Doug Hellman's excellent tutorial.
Running the code above yields a result like this:
% test.py
> foo 1
Running foo 1
> foo 1 2 3
Running foo 1 2 3
> bar 2
Running bar 2
> baz 3
*** Unknown syntax: baz 3

optparse is deprecated since python 2.7 and anyway argparse is much more flexible.
The approach of unutbu is safe, but in case you provide whitelist, I would suggest you to let the user know which functions are accepted
dispatch = {
'foo': foo,
'bar': bar,
}
parser = argparse.ArgumentParser()
parser.add_argument('function', choices=dispatch.keys() )
FYI: if the parsing is not too complicated, docopt looks like a very nice package

How about sys.argv? For more advanced stuff check out argsparse. optparse seems depreciated now, but there's a lot of answers here about this question.

Take a look at the optparse module in python. It's exactly what you would need:
http://docs.python.org/2/library/optparse.html
Or you can write your own custom opt-parser (minimalistic though)
def getopts(argv):
opts = {}
while argv:
if argv[0][0] == '-': # find "-name value" pairs
opts[argv[0]] = argv[1] # dict key is "-name" arg
argv = argv[2:]
else:
argv = argv[1:]
return opts
if __name__ == '__main__':
from sys import argv # example client code
myargs = getopts(argv)
# DO something based on your logic here
But in case your script needs to run on python 3 and beyond, you need to consider argparse module.\
Hope that helps.

Take a look at optparse . This can help passing and receiving shell style parameters to python scripts.
Update:
Apparently optparse is deprecated now and argparse is now preferred option for parsing command line arguments.

import sys
def main(arg):
return arg
print main(sys.argv[1])
where sys.argv[0] is the .py file you're running, and all the ones after it would be each argument. you could check the length of the list, then iterate through it, and parse them as necessary and pass the correct things to each function

Need help with python script with bash commands

I copied this script from internet but idon't know how to use it. i am newbiw to python so please help. When i execute it using
./test.py then i can only see
usage: py4sa [option]
A unix toolbox
options:
--version show program's version number and exit
-h, --help show this help message and exit
-i, --ip gets current IP Address
-u, --usage gets disk usage of homedir
-v, --verbose prints verbosely
when i type py4sa then it says bash command not found
The full script is
#!/usr/bin/env python
import subprocess
import optparse
import re
#Create variables out of shell commands
#Note triple quotes can embed Bash
#You could add another bash command here
#HOLDING_SPOT="""fake_command"""
#Determines Home Directory Usage in Gigs
HOMEDIR_USAGE = """
du -sh $HOME | cut -f1
"""
#Determines IP Address
IPADDR = """
/sbin/ifconfig -a | awk '/(cast)/ { print $2 }' | cut -d':' -f2 | head -1
"""
#This function takes Bash commands and returns them
def runBash(cmd):
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
out = p.stdout.read().strip()
return out #This is the stdout from the shell command
VERBOSE=False
def report(output,cmdtype="UNIX COMMAND:"):
#Notice the global statement allows input from outside of function
if VERBOSE:
print "%s: %s" % (cmdtype, output)
else:
print output
#Function to control option parsing in Python
def controller():
global VERBOSE
#Create instance of OptionParser Module, included in Standard Library
p = optparse.OptionParser(description='A unix toolbox',
prog='py4sa',
version='py4sa 0.1',
usage= '%prog [option]')
p.add_option('--ip','-i', action="store_true", help='gets current IP Address')
p.add_option('--usage', '-u', action="store_true", help='gets disk usage of homedir')
p.add_option('--verbose', '-v',
action = 'store_true',
help='prints verbosely',
default=False)
#Option Handling passes correct parameter to runBash
options, arguments = p.parse_args()
if options.verbose:
VERBOSE=True
if options.ip:
value = runBash(IPADDR)
report(value,"IPADDR")
elif options.usage:
value = runBash(HOMEDIR_USAGE)
report(value, "HOMEDIR_USAGE")
else:
p.print_help()
#Runs all the functions
def main():
controller()
#This idiom means the below code only runs when executed from command line
if __name__ == '__main__':
main()

It seems to me you have stored the script under another name: test.py rather than py4sa. So typing ./test.py, like you did, is correct for you. The program requires arguments, however, so you have to enter one of the options listed under 'usage'.
Normally 'py4sa [OPTIONS]' would mean that OPTIONS is optional, but looking at the code we can see that it isn't:
if options.verbose:
# ...
if options.ip:
# ...
elif options.usage:
# ...
else:
# Here's a "catch all" in case no options are supplied.
# It will show the help text you get:
p.print_help()
Note that the program probably would not be recognized by bash even if you renamed it to py4sa, as the current directory is often not in bash's PATH. It says 'usage: py4sa (..)' because that's hard-coded into the program.

The script is called "test.py". Either invoke it as such, or rename it to "py4sa".

you run a Python script using the interpreter, so
$ python py4sa