We Keep Coding

Convert multiples line bash script to single line

I am trying to write a bash script to use in python code.
Multi-line bash command (this works perfectly when run directly from terminal)
mydatefile="/var/tmp/date"
while IFS= read line
do
echo $line
sh /opt/setup/Script/EPS.sh $(echo $line) | grep "WORD" | awk -F ',' '{print $6}'
sleep 1
done <"$mydatefile"
My single line conversion
mydatefile="/var/tmp/date;" while IFS= read line do echo $line; sh /opt/setup/Script/EPS.sh $(echo $line) | grep "WORD" | awk -F ',' '{print $6}'; sleep 1; done <"$mydatefile";
ERROR
-bash: syntax error near unexpected token `done'

Missing a ; (fatal syntax error):
while IFS= read line; do echo ...
# ^
# here
More in depth :
combined grep+awk in a single command
mydatefile="/var/tmp/date"
while IFS= read line; do
echo "$line"
sh /opt/setup/Script/EPS.sh "$line" |
awk -F ',' '/WORD/{print $6}'
sleep 1
done < "$mydatefile"
use more quotes !
Learn how to quote properly in shell, it's very important :
"Double quote" every literal that contains spaces/metacharacters and every expansion: "$var", "$(command "$var")", "${array[#]}", "a & b". Use 'single quotes' for code or literal $'s: 'Costs $5 US', ssh host 'echo "$HOSTNAME"'. See
http://mywiki.wooledge.org/Quotes
http://mywiki.wooledge.org/Arguments
http://wiki.bash-hackers.org/syntax/words
finally:
mydatefile="/var/tmp/date;" while IFS= read line; do echo $line; sh /opt/setup/Script/EPS.sh "$line" | awk -F ',' '/WORD/{print $6}'; sleep 1; done < "$mydatefile";

One way to do this conversion might be to paste the script onto the command-line, then look up in the history - though this might depend on the version of command-line editor you have. Note that you do need a semicolon before do, but NOT after. You are punished for too many semicolons as well as too few.
Another way would be to line-by-line fold each line in your script and keep testing it.
The binary chop approach is do the first half, test and undo or continue.
Once you have it down to 1 line that works you can pasted it into python.

vim and wc give different line counts

I have two csv files that give different results when I use wc -l (gives 65 lines for the first, 66 for the second) and when I use vim file.csv and then :$ to go to the bottom of the file (66 lines for both). I have tried viewing newline characters in vim using :set list and they look identical.
I have created the second (which shows one extra line with wc) was created from the first using pandas in Python and to_csv.
Is there anything within pandas that might generate new lines or other bash/vim tools I can use to verify the differences?

If the last character of the file is not a newline, wc won't count the last line:
$ printf 'a\nb\nc' | wc -l
2
In fact, that's how wc -l is documented to work: from man wc
-l, --lines
print the newline counts
^^^^^^^^^^^^^

Jump on specific line in txt file

I have a big text file and I want to open it and jump on a line that contains a specific string.
What I managed to do is jump on a specific line number.
import os
os.system('sudo gedit +50 test.txt')
How can i edit the code so that it searches for a specific string instead of a line number?

You can invoke the line as first ocurrence of text that are you looking for with next line:
gedit +$(grep -n "id" test.txt | head -n 1 | cut -f 1 -d ':') test.txt
This grep -n "text_to_find" test.txt | head -n 1 | cut -f 1 -d ':' means:
grep ... Tell me all lines where there are "test_to_find" and prefix with line number
head ...: get first occurrence
cut ... get the line number
You have to fix it in case that there are not ocurrences

You're not really going to be able to open the file and jump straight to a line, without first doing some logic to figure where that line is.
This should do the trick:
with open('test.txt', 'r') as myFile:
for line in myFile:
if 'My Search String Here' in line:
print line
You may also want to look at this: https://docs.python.org/2/library/linecache.html

Split Command - Choose Output Name

I have a text file named myfile.txt. The file contains 50,000 lines and I would like to split it into 50 text files. I know that this is easy with the split command:
split myfile.txt
This will output 50 1000-line files: xaa, xab, and xac.
My question, how do I run split my text file so that it names the output files:
1.txt
2.txt
3.txt
...
50.txt
Seeking answers in python or bash please. Thank you!

Here is a potential solution using itertools.islice to get the chunks and string formatting for the different file names:
from itertools import islice
with open('myfile.txt') as in_file:
for i in range(1, 51):
with open('{0}.txt'.format(i), 'w') as out_file:
lines = islice(in_file, 1000)
out_file.writelines(lines)

its not exactly what you are looking for, but running
split -d myfile.txt
will output
x00
x01
x02
...

To generate test data in empty directory, you can use
seq 50000 | split -d
To rename in the way that you want, you can use
ls x* | awk '{print $0, (substr($0,2)+1) ".txt"}' | xargs -n2 mv

Here's a funny one: if your split command supports the --filter option, you can use it!
If you call
split --filter=./banana myfile.txt
then the command ./banana will be executed with the environmental variable FILE set to the name split would choose to write the chunk it's processing. This command will receive on its standard input the chunk being processed. If this command returns a non-zero status code, then split will interrupt its operations.
Together with the -d option, that's exactly what you want. With the -d option, the name split will choose for the filenames will be x01, x02, etc.
Make a script:
#!/bin/bash
# remove the leading x from FILE
n=${FILE#x}
# check that n is a number
[[ $n = +([[:digit:]]) ]] || exit 1
# remove the leading zeroes from n
n=$((10#$n))
# send stdin to file
cat > "$n.txt"
Call this script banana, chmod +x it and let's go:
split -d --filter=./banana myfile.txt
This --filter option is really funny.

Here's an example of how you could split this file in bash:
split -l 1000 -d --additional-suffix=.txt myfile.txt
The -l argument determines the number of lines included in each split file (1000 in this case, for 50 total files), the -d argument uses numbers instead of letters for the suffixes, and the value we pass to the --additional-suffix argument here gives each file a .txt file extension.
This will create
x00.txt
x01.txt
x01.txt
etc.
If you wanted to change the 'x' portion of the output files, you'd want to add a prefix after the input file (e.g. myfile.txt f would create f01.txt, f02.txt, etc.)
Note that without --additional-suffix, your files will all lack filename extensions.
I've looked to see if there's a way to split a file and name them with only the suffix, but I haven't found anything.

A simple approach:
f=open('your_file')
count_line,file = 0,1
for x in f:
count_line +=1
if count%1000 == 1:
f1 = open(str(file) + '.txt','w')
f1.write(x)
file +=1
elif count_line%1000 == 0:
f1.write(x)
f1.close()
else:f1.write(x)

Need for Performance in bash script

I have 50000 files and each one has 10000 lines. Each line is in the form:
value_1 (TAB) value_2 (TAB) ... value_n
I wanted to remove specific values from every line in every file (i used cut to remove values 14-17) and write the results to a new file.
For doing that in one file, i wrote this code:
file=nameOfFile
newfile=$file".new"
i=0
while read line
do
let i=i+1
echo line: $i
a=$i"p"
lineFirstPart=$(sed -n -e $a $file | cut -f 1-13)
#echo lineFirstPart: $lineFirstPart
lineSecondPart=$(sed -n -e $a $file | cut -f 18-)
#echo lineSecondPart: $lineSecondPart
newline=$lineFirstPart$lineSecondPart
echo $newline >> $newfile
done < $file
This takes ~45 secs for one file, which means for all it will take about: 45x50000 = 625h ~= 26 days!
Well, i need something faster, e.g. a solution that cats the whole file, applies the two cut commands simultaneusly or something like that i guess.
Also solutions in python are accepted + appreciated but bash scripting is preferable!

The entire while loop can be replaced with one line:
cut -f1-13,18- $file > $newfile