For those who are curious as to why I'm doing this: I need specific files in a tar ball - no more, no less. I have to write unit tests for make check, but since I'm constrained to having "no more" files, I have to write the check within the make check. In this way, I have to write bash(but I don't want to).
I dislike using bash for unit testing(sorry to all those who like bash. I just dislike it so much that I would rather go with an extremely hacky approach than to write many lines of bash code), so I wrote a python file. I later learned that I have to use bash because of some unknown strict rule. I figured that there was a way to cache the entire content of the python file into a single string in the bash file, so I could take the string literal in bash and write to a python file and then execute it.
I tried the following attempt (in the following script and result, I used another python file that's not unit_test.py, so don't worry if it doesn't actually look like a unit test):
toStr.py:
import re
with open("unit_test.py", 'r+') as f:
s = f.read()
s = s.replace("\n", "\\n")
print(s)
And then I piped the results out using:
python toStr.py > temp.txt
It looked something like:
#!/usr/bin/env python\n\nimport os\nimport sys\n\n#create number of bytes as specified in the args:\nif len(sys.argv) != 3:\n print("We need a correct number of args : 2 [NUM_BYTES][FILE_NAME].")\n exit(1)\nn = -1\ntry:\n n = int(sys.argv[1])\nexcept:\n print("Error casting number : " + sys.argv[1])\n exit(1)\n\nrand_string = os.urandom(n)\n\nwith open(sys.argv[2], 'wb+') as f:\n f.write(rand_string)\n f.flush()\n f.close()\n\n
I tried taking this as a string literal and echoing it into a new file and see whether I could run it as a python file but it failed.
echo '{insert that giant string above here}' > new_unit_test.py
I wanted to take this statement above and copy it into my "bash unit test" file so I can just execute the python file within the bash script.
The resulting file looked exactly like {insert giant string here}. What am I doing wrong in my attempt? Are there other, much easier ways where I can hold a python file as a string literal in a bash script?
the easiest way is to only use double-quotes in your python code, then, in your bash script, wrap all of your python code in one pair of single-quotes, e.g.,
#!/bin/bash
python -c 'import os
import sys
#create number of bytes as specified in the args:
if len(sys.argv) != 3:
print("We need a correct number of args : 2 [NUM_BYTES][FILE_NAME].")
exit(1)
n = -1
try:
n = int(sys.argv[1])
except:
print("Error casting number : " + sys.argv[1])
exit(1)
rand_string = os.urandom(n)
# i changed ""s to ''s below -webb
with open(sys.argv[2], "wb+") as f:
f.write(rand_string)
f.flush()
f.close()'
Related
I have tried to execute a simple python command from cmd like C:\Users> stat.py < swagger.yaml > output.html, which executes stat.py by taking swagger.yaml as input argument and generates output.html file and it worked fine in cmd. But now i want to execute my stat.py file through another python file demo.py by passing the values swagger.yaml and output.html as sys.argv[0] and sys.argv[1] inside demo.py.
my command from cmd C:\Users> demo.py swagger.yaml output.html and my demo.py file is as follows..
# my demo.py file ....
import os
import sys
os.system('stat.py < sys.argv[1] > sys.argv[2]')
error - the system can not find the file specified.
Why i am getting this error and please any help to resolve it ..
Inside a normal string, no variable interpretation is applied. So you literally asked to read from a file named sys.argv[1] (possibly sys.argv1 if the file exists, thanks to shell globbing), and write to a file named sys.argv[2].
If you want to use the values sys.argv in your script, you need to format them into the string, e.g. with f-strings (modern Python 3.6 or so only):
os.system(f'stat.py < {sys.argv[1]} > {sys.argv[2]}') # Note f at beginning of literal
or on older Python 2.7, with str.format:
os.system('stat.py < {} > {}'.format(sys.argv[1], sys.argv[2]))
Note that however you slice it, this is dangerous; os.system is launching this in a shell, and arguments that contain shell metacharacters will be interpreted as such. It can't do anything the user didn't already have permission to do, but small mistakes by the user could dramatically change the behavior of the program. If you want to do this properly/safely, use subprocess, open the files yourself, and pass them in explicitly as stdin/stdout:
with open(sys.argv[1], 'rb') as infile, open(sys.argv[2], 'wb') as outfile:
subprocess.run(['stat.py'], stdin=infile, stdout=outfile)
This ensures the files can be opened in the first place before launching the process, doesn't allow the shell to interpret anything, and avoids the (minor) expense of launching a shell at all. It's also going to give you more useful errors if opening the files fails.
I have a Python script which ends the following way:
if __name__ == "__main__":
if len(sys.argv) == 2:
file = open(sys.argv[1])
text = file.readline()
... #more statements
This works when I type in the following: $ python3 script.py my_file.txt
However, I want to change it so my script can accept text from standard input (or even a text file). This is what I want to be able to do:
$ ./script.py < my_file.txt
I think I need to use sys.stdin.read() (or maybe sys.stdin.readlines()). Could you tell me what I would need to change from my original script?
I'm sorry if this looks very basic, but I'm new to Python and I find it hard to see the difference.
It's exactly what you said, you don't need to open a file.
Instead of calling file.readline(), call sys.stdin.readline().
You can make it "nice", with something like:
file = sys.stdin if use_stdin else open(sys.argv[1])
Theres a cool module you can use for this! Assuming you want to do processing per line:
import fileinput
for line in fileinput.input():
process_line(line)
I'm trying to make some functions in python so that I can connect to a linux terminal and do stuff (like in this case, create a file). The code I have, works partially. The only thing that doesn't work is if you want to do something after you have entered the code. Like for instance you create the file and then want to navigate somewhere else (cd /tmp) for instance. Instead of doing the next command, it will just add to the file created.
def create_file(self, name, contents, location):
try:
log.info("Creating a file...")
self.device.execute("mkdir -p {}".format(location))
self.cd_path(location)
self.device.sendline("cat > {}".format(name))
self.device.sendline("{}".format(contents))
self.device.sendline("EOF") # send the CTRL + D command to save and exit I tried here with ^D as well
except:
log.info("Failed to create the file!")
The contents of the file is:
cat test.txt
#!/bin/bash
echo "Fail Method Requested"
exit 1
EOF
ls -d /tmp/asdasd
The order of commands executed is:
execute.create_file(test.txt, the_message, the_location)
execute.check_path("/tmp/adsasd") #this function just checks with ls -d if the directory exists.
I have tried with sendline the following combinations:
^D, EOF, <<EOF
I don't really understand how I could make this happen. I just want to create a file with a specific message. (When researching on how to do this with VI I got the same problem, but there the command I needed was the one for ESC)
If anyone could help with some input that would be great!!
Edit: As Rob mentioned below, sending the character "\x04" actually works. For anyone else having this issue, you can also consult this chart for other combinations if needed:
http://donsnotes.com/tech/charsets/ascii.html
You probably need to send the EOF character, which is typically CONTROL-D, not the three characters E, O, and F.
self.device.sendline("\x04")
http://wiki.bash-hackers.org/syntax/redirection#here_documents
Here docs allow you to use any file input termination string you like to represent end of file ( such as the literal EOF you're attempting to use now). Quoting that string tells the shell not to interpret expansions inside the heredoc content, ensuring that said content is treated as literal.
Using pipes.quote() here ensures that filenames with literal quotes, $s, spaces, or other surprising characters won't break your script. (Of course, you'll need to import pipes; on Python 3, by contrast, this has moved to shlex.quote()).
self.device.sendline("cat > {} <<'EOF'".format(pipes.quote(name)))
Then you can write the EOF as is, having told bash to interpret it as the end of file input.
I have a simple bash command here for a script that I am re-writing in Python, and I've done a lot of searching and haven't found a simple answer. I am trying to echo the output of Print to a file, making sure there are no line breaks and that I can pass a variable into it. Here is just a little snippet (there are a lot of lines like this):
echo " ServerName www.${hostName}" >> $prjFile
Now I know it would end up looking something like:
print ("ServerName www.", hostName) >> prjFile
Right? But that doesn't work. Mind you, this is in Python 2.6 (as the machine this script will run on is using that version and there are other dependencies reliant on sticking with that version).
The syntax is;
print >>myfile, "ServerName www.", hostName,
where myfile is a file object opened in mode "a" (for "append").
The trailing comma prevents line breaks.
You might also want to use sys.stdout.softspace = False to prevent the spaces that Python adds between comma-separate arguments to print, and/or to print things as a single string:
print >>myfile, "ServerName www.%s" % hostName,
You can try a simple:
myFile = open('/tmp/result.file', 'w') # or 'a' to add text instead of truncate
myFile.write('whatever')
myFile.close()
In your case:
myFile = open(prjFile, 'a') # 'a' because you want to add to the existing file
myFile.write('ServerName www.{hostname}'.format(hostname=hostname))
myFile.close()
I have a script which connects to database and gets all records which statisfy the query. These record results are files present on a server, so now I have a text file which has all file names in it.
I want a script which would know:
What is the size of each file in the output.txt file?
What is the total size of all the files present in that text file?
Update:
I would like to know how can I achieve my task using Perl programming language, any inputs would be highly appreciated.
Note: I do not have any specific language constraint, it could be either Perl or Python scripting language which I can run from the Unix prompt. Currently I am using the bash shell and have sh and py script. How can this be done?
My scripts:
#!/usr/bin/ksh
export ORACLE_HOME=database specific details
export PATH=$ORACLE_HOME/bin:path information
sqlplus database server information<<EOF
SET HEADING OFF
SET ECHO OFF
SET PAGESIZE 0
SET LINESIZE 1000
SPOOL output.txt
select * from my table_name;
SPOOL OFF
EOF
I know du -h would be the command which I should be using but I am not sure how should my script be, I have tried something in python. I am totally new to Python and it's my first time effort.
Here it is:
import os
folderpath='folder_path'
file=open('output file which has all listing of query result','r')
for line in file:
filename=line.strip()
filename=filename.replace(' ', '\ ')
fullpath=folderpath+filename
# print (fullpath)
os.system('du -h '+fullpath)
File names in the output text file for example are like: 007_009_Bond Is Here_009_Yippie.doc
Any guidance would be highly appreciated.
Update:
How can I move all the files which are present in output.txt file to some other folder location using Perl ?
After doing step1, how can I delete all the files which are present in output.txt file ?
Any suggestions would be highly appreciated.
In perl, the -s filetest operator is probaby what you want.
use strict;
use warnings;
use File::Copy;
my $folderpath = 'the_path';
my $destination = 'path/to/destination/directory';
open my $IN, '<', 'path/to/infile';
my $total;
while (<$IN>) {
chomp;
my $size = -s "$folderpath/$_";
print "$_ => $size\n";
$total += $size;
move("$folderpath/$_", "$destination/$_") or die "Error when moving: $!";
}
print "Total => $total\n";
Note that -s gives size in bytes not blocks like du.
On further investigation, perl's -s is equivalent to du -b. You should probably read the man pages on your specific du to make sure that you are actually measuring what you intend to measure.
If you really want the du values, change the assignment to $size above to:
my ($size) = split(' ', `du "$folderpath/$_"`);
Eyeballing, you can make YOUR script work this way:
1) Delete the line filename=filename.replace(' ', '\ ') Escaping is more complicated than that, and you should just quote the full path or use a Python library to escape it based on the specific OS;
2) You are probably missing a delimiter between the path and the file name;
3) You need single quotes around the full path in the call to os.system.
This works for me:
#!/usr/bin/python
import os
folderpath='/Users/andrew/bin'
file=open('ft.txt','r')
for line in file:
filename=line.strip()
fullpath=folderpath+"/"+filename
os.system('du -h '+"'"+fullpath+"'")
The file "ft.txt" has file names with no path and the path part is '/Users/andrew/bin'. Some of the files have names that would need to be escaped, but that is taken care of with the single quotes around the file name.
That will run du -h on each file in the .txt file, but does not give you the total. This is fairly easy in Perl or Python.
Here is a Python script (based on yours) to do that:
#!/usr/bin/python
import os
folderpath='/Users/andrew/bin/testdir'
file=open('/Users/andrew/bin/testdir/ft.txt','r')
blocks=0
i=0
template='%d total files in %d blocks using %d KB\n'
for line in file:
i+=1
filename=line.strip()
fullpath=folderpath+"/"+filename
if(os.path.exists(fullpath)):
info=os.stat(fullpath)
blocks+=info.st_blocks
print `info.st_blocks`+"\t"+fullpath
else:
print '"'+fullpath+"'"+" not found"
print `blocks`+"\tTotal"
print " "+template % (i,blocks,blocks*512/1024)
Notice that you do not have to quote or escape the file name this time; Python does it for you. This calculates file sizes using allocation blocks; the same way that du does it. If I run du -ahc against the same files that I have listed in ft.txt I get the same number (well kinda; du reports it as 25M and I get the report as 24324 KB) but it reports the same number of blocks. (Side note: "blocks" are always assumed to be 512 bytes under Unix even though the actual block size on larger disc is always larger.)
Finally, you may want to consider making your script so that it can read a command line group of files rather than hard coding the file and the path in the script. Consider:
#!/usr/bin/python
import os, sys
total_blocks=0
total_files=0
template='%d total files in %d blocks using %d KB\n'
print
for arg in sys.argv[1:]:
print "processing: "+arg
blocks=0
i=0
file=open(arg,'r')
for line in file:
abspath=os.path.abspath(arg)
folderpath=os.path.dirname(abspath)
i+=1
filename=line.strip()
fullpath=folderpath+"/"+filename
if(os.path.exists(fullpath)):
info=os.stat(fullpath)
blocks+=info.st_blocks
print `info.st_blocks`+"\t"+fullpath
else:
print '"'+fullpath+"'"+" not found"
print "\t"+template % (i,blocks,blocks*512/1024)
total_blocks+=blocks
total_files+=i
print template % (total_files,total_blocks,total_blocks*512/1024)
You can then execute the script (after chmod +x [script_name].py) by ./script.py ft.txt and it will then use the path to the command line file as the assumed path to the files "ft.txt". You can process multiple files as well.
You can do it in your shell script itself.
You have all the files names in your spooled file output.txt, all you have to add at the end of existing script is:
< output.txt du -h
It will give size of each file and also a total at the end.
You can use the Python skeleton that you've sketched out and add os.path.getsize(fullpath) to get the size of individual file.
For example, if you wanted a dictionary with the file name and size you could:
dict((f, os.path.getsize(f)) for f in file)
Keep in mind that the result from os.path.getsize(...) is in bytes so you'll have to convert it to get other units if you want.
In general os.path is a key module for manipulating files and paths.