We Keep Coding

Change specific column values from a text file using Python

I have a text file as following:
1 1 2 1 1e8
2 1 2 3 1e5
3 2 3 2 2000
4 2 5 6 1000
5 2 4 3 1e4
6 3 6 4 5000
7 3 5 2 2000
8 3 2 3 5000
9 3 4 5 1e9
10 3 2 3 1e6
My question is that how can I change one column(for example divide all the data from the last column to 900) and save the whole data in a new file?
Thanks in advance

You can use numpy library. The code below should do it.
import numpy as np
# assume the data is in.txt file
dat = np.loadtxt('in.txt')
# -1 means the last column
dat[:, -1] = dat[:, -1]/900
# write the result to the out.txt file
# fmt is the format while writing to the file
# %.2f means that it will save it to the 2 digits precision
np.savetxt('out.txt', dat, fmt='%.2f')

for each line of your file,
vals = line.split(' ')
my_val = float(vals[4])/900
output = open('filename', 'wb')
output.write(my_val +'\n')

Using a simple iteration.
res = []
with open(filename, "r") as infile:
for line in infile: # Iterate each line
val = line.split() # Split line by space
res.append( str(float(val[-1])/900) ) # Use negative index to get last element and divide.
with open(filename, "w") as outfile: #Open file to write
for line in res:
outfile.write(line+"\n") #Write Data

You can try the following:
import numpy as np
#Suppose your text data name is "CH3CH2OH_g.txt"
file = np.loadtxt('CH3CH2OH_g.txt')
z= (file[:, -1]/900)
np.savetxt('LogCH3CH2OH.txt', file, fmt='%.2f')

Read text file into list based on specific criterion

I have a text file with the following content:
str1 str2 str3 str4
0 1 12 34
0 2 4 6
0 3 5 22
0 56 2 18
0 3 99 12
0 8 5 7
1 66 78 9
I want to read the above text file into a list such that program starts reading from the row where first column has a value greater than zero.
How do I do it in Python 3.5?
I tried genfromtxt() but i can only skip fixed number of lines from top. Since I will be reading different files i need something else.

This is one way with csv module.
import csv
from io import StringIO
mystr = StringIO("""\
str1 str2 str3 str4
0 1 12 34
0 2 4 6
0 3 5 22
0 56 2 18
0 3 99 12
0 8 5 7
1 66 78 9
2 50 45 4
""")
res = []
# replace mystr with open('file.csv', 'r')
with mystr as f:
reader = csv.reader(mystr, delimiter=' ', skipinitialspace=True)
next(reader) # skip header
for line in reader:
row = list(map(int, filter(None, line))) # convert to integers
if row[0] > 0: # apply condition
res.append(row)
print(res)
[[1, 66, 78, 9], [2, 50, 45, 4]]

lst = []
flag = 0
with open('a.txt') as f:
for line in f:
try:
if float(line.split()[0].strip('.')) >0:
flag = 1
if flag == 1:
lst += [float(i.strip('.')) for i in line.split()]
except:
pass

convert file of numbers (in matrix format) to a list of list in python [duplicate]

I'd like to read numbers from file into two dimensional array.
File contents:
line containing w, h
h lines containing w integers separated with space
For example:
4 3
1 2 3 4
2 3 4 5
6 7 8 9

Assuming you don't have extraneous whitespace:
with open('file') as f:
w, h = [int(x) for x in next(f).split()] # read first line
array = []
for line in f: # read rest of lines
array.append([int(x) for x in line.split()])
You could condense the last for loop into a nested list comprehension:
with open('file') as f:
w, h = [int(x) for x in next(f).split()]
array = [[int(x) for x in line.split()] for line in f]

To me this kind of seemingly simple problem is what Python is all about. Especially if you're coming from a language like C++, where simple text parsing can be a pain in the butt, you'll really appreciate the functionally unit-wise solution that python can give you. I'd keep it really simple with a couple of built-in functions and some generator expressions.
You'll need open(name, mode), myfile.readlines(), mystring.split(), int(myval), and then you'll probably want to use a couple of generators to put them all together in a pythonic way.
# This opens a handle to your file, in 'r' read mode
file_handle = open('mynumbers.txt', 'r')
# Read in all the lines of your file into a list of lines
lines_list = file_handle.readlines()
# Extract dimensions from first line. Cast values to integers from strings.
cols, rows = (int(val) for val in lines_list[0].split())
# Do a double-nested list comprehension to get the rest of the data into your matrix
my_data = [[int(val) for val in line.split()] for line in lines_list[1:]]
Look up generator expressions here. They can really simplify your code into discrete functional units! Imagine doing the same thing in 4 lines in C++... It would be a monster. Especially the list generators, when I was I C++ guy I always wished I had something like that, and I'd often end up building custom functions to construct each kind of array I wanted.

Not sure why do you need w,h. If these values are actually required and mean that only specified number of rows and cols should be read than you can try the following:
output = []
with open(r'c:\file.txt', 'r') as f:
w, h = map(int, f.readline().split())
tmp = []
for i, line in enumerate(f):
if i == h:
break
tmp.append(map(int, line.split()[:w]))
output.append(tmp)

is working with both python2(e.g. Python 2.7.10) and python3(e.g. Python 3.6.4)
with open('in.txt') as f:
rows,cols=np.fromfile(f, dtype=int, count=2, sep=" ")
data = np.fromfile(f, dtype=int, count=cols*rows, sep=" ").reshape((rows,cols))
another way:
is working with both python2(e.g. Python 2.7.10) and python3(e.g. Python 3.6.4),
as well for complex matrices see the example below (only change int to complex)
with open('in.txt') as f:
data = []
cols,rows=list(map(int, f.readline().split()))
for i in range(0, rows):
data.append(list(map(int, f.readline().split()[:cols])))
print (data)
I updated the code, this method is working for any number of matrices and any kind of matrices(int,complex,float) in the initial in.txt file.
This program yields matrix multiplication as an application. Is working with python2, in order to work with python3 make the following changes
print to print()
and
print "%7g" %a[i,j], to print ("%7g" %a[i,j],end="")
the script:
import numpy as np
def printMatrix(a):
print ("Matrix["+("%d" %a.shape[0])+"]["+("%d" %a.shape[1])+"]")
rows = a.shape[0]
cols = a.shape[1]
for i in range(0,rows):
for j in range(0,cols):
print "%7g" %a[i,j],
print
print
def readMatrixFile(FileName):
rows,cols=np.fromfile(FileName, dtype=int, count=2, sep=" ")
a = np.fromfile(FileName, dtype=float, count=rows*cols, sep=" ").reshape((rows,cols))
return a
def readMatrixFileComplex(FileName):
data = []
rows,cols=list(map(int, FileName.readline().split()))
for i in range(0, rows):
data.append(list(map(complex, FileName.readline().split()[:cols])))
a = np.array(data)
return a
f = open('in.txt')
a=readMatrixFile(f)
printMatrix(a)
b=readMatrixFile(f)
printMatrix(b)
a1=readMatrixFile(f)
printMatrix(a1)
b1=readMatrixFile(f)
printMatrix(b1)
f.close()
print ("matrix multiplication")
c = np.dot(a,b)
printMatrix(c)
c1 = np.dot(a1,b1)
printMatrix(c1)
with open('complex_in.txt') as fid:
a2=readMatrixFileComplex(fid)
print(a2)
b2=readMatrixFileComplex(fid)
print(b2)
print ("complex matrix multiplication")
c2 = np.dot(a2,b2)
print(c2)
print ("real part of complex matrix")
printMatrix(c2.real)
print ("imaginary part of complex matrix")
printMatrix(c2.imag)
as input file I take in.txt:
4 4
1 1 1 1
2 4 8 16
3 9 27 81
4 16 64 256
4 3
4.02 -3.0 4.0
-13.0 19.0 -7.0
3.0 -2.0 7.0
-1.0 1.0 -1.0
3 4
1 2 -2 0
-3 4 7 2
6 0 3 1
4 2
-1 3
0 9
1 -11
4 -5
and complex_in.txt
3 4
1+1j 2+2j -2-2j 0+0j
-3-3j 4+4j 7+7j 2+2j
6+6j 0+0j 3+3j 1+1j
4 2
-1-1j 3+3j
0+0j 9+9j
1+1j -11-11j
4+4j -5-5j
and the output look like:
Matrix[4][4]
1 1 1 1
2 4 8 16
3 9 27 81
4 16 64 256
Matrix[4][3]
4.02 -3 4
-13 19 -7
3 -2 7
-1 1 -1
Matrix[3][4]
1 2 -2 0
-3 4 7 2
6 0 3 1
Matrix[4][2]
-1 3
0 9
1 -11
4 -5
matrix multiplication
Matrix[4][3]
-6.98 15 3
-35.96 70 20
-104.94 189 57
-255.92 420 96
Matrix[3][2]
-3 43
18 -60
1 -20
[[ 1.+1.j 2.+2.j -2.-2.j 0.+0.j]
[-3.-3.j 4.+4.j 7.+7.j 2.+2.j]
[ 6.+6.j 0.+0.j 3.+3.j 1.+1.j]]
[[ -1. -1.j 3. +3.j]
[ 0. +0.j 9. +9.j]
[ 1. +1.j -11.-11.j]
[ 4. +4.j -5. -5.j]]
complex matrix multiplication
[[ 0. -6.j 0. +86.j]
[ 0. +36.j 0.-120.j]
[ 0. +2.j 0. -40.j]]
real part of complex matrix
Matrix[3][2]
0 0
0 0
0 0
imaginary part of complex matrix
Matrix[3][2]
-6 86
36 -120
2 -40

To make the answer simple here is a program that reads integers from the file and sorting them
f = open("input.txt", 'r')
nums = f.readlines()
nums = [int(i) for i in nums]
After reading each line of the file converting each string to a digit
nums.sort()
Sorting the numbers
f.close()
f = open("input.txt", 'w')
for num in nums:
f.write("%d\n" %num)
f.close()
Writing them back
As easy as that, Hope this helps

The shortest I can think of is:
with open("file") as f:
(w, h), data = [int(x) for x in f.readline().split()], [int(x) for x in f.read().split()]
You can seperate (w, h) and data if it looks neater.

How to extract columnes from Python list?

My Python code
import operator
with open('index.txt') as f:
lines = f.read().splitlines()
print type(lines)
print len(lines)
l2=lines[1::3]
print len(l2)
print l2[0]
list1 = [0,2]
my_items = operator.itemgetter(*list1)
new_list = [ my_items(x) for x in l2 ]
with open('newindex1.txt','w') as thefile:
for item in l2:
thefile.write("%s\n" % item)
Couple of lines from index.txt
0 0 0
0 1 0
0 2 0
1 0 0
1 1 0
1 2 0
2 0 0
2 1 0
2 2 0
3 0 0
Couple of lines from newindex1.txt
0 1 0
1 1 0
2 1 0
3 1 0
4 1 0
5 1 0
6 1 0
7 1 0
8 1 0
9 1 0
I wanted to read the file as a list,then choose every third row and then finally select first and the third column from that list.It seems that I do not understand how operator works.
If I try with Back2Basics solution
import numpy as np
myarray = np.fromfile('index.txt', dtype=int, sep=' ')
anotherarray = myarray[::3][0,2]
I got
File "a12.py", line 4, in <module>
anotherarray = myarray[::3][0,2]
IndexError: too many indices

You don't need to read all the data into memory at all, you can use itertools.islice to parse the rows you want and the csv lib to read and write the data:
from operator import itemgetter
from itertools import islice
import csv
with open("in.txt") as f, open('newindex1.txt','w') as out:
r = csv.reader(f, delimiter=" ")
wr = csv.writer(out, delimiter=" ")
for row in iter(lambda: list(islice(r, 0, 3, 3)), []):
wr.writerow(map(itemgetter(0, 2), row)[0])

I'd highly suggest using numpy for this. The reason being this is all numerical data that fits so nicely into memory. The code looks like this.
import numpy as np
myarray = np.fromfile('index.txt', dtype=int, sep=' ')
anotherarray = myarray[::3,::2]
and then you want to write the file
anotherarray.tofile('newfile.txt', sep=" ")
The way the array slicing line [::3,::2] reads is "take every 3rd row starting from 0, and take every other column starting from 0"

I think you need something this?
lines = []
with open('index.txt', 'r') as fi:
lines = fi.read().splitlines()
lines = [line.split() for line in lines]
with open('answer.txt', 'w') as fo:
for column in range(len(lines)):
if (column + 1) % 3:
fo.write('%s %s\n' % (lines[column][0], lines[column][2]))

How to add values to existing columns in a text file (using Python)

I am currently generating 8 random values each time I run a program on Python. These 8 values are different each time I run the program, and I would like to be able to now save these 8 values each time I run the program to a text file in 8 separate columns. When saving these values for future runs, though, I would like to still be able to keep previous values. For example: after run 1, the text file will be 8x1, after run 2, the text file will be 8x2, and after run n, the text file will be 8xn.
I have been looking at solutions like this: save output values in txt file in columns python
And it seems using 'a' instead 'w' will append my new values instead of overwriting previous values. I've been trying to follow the documentation on the method .write but just don't quite see how I can write to a particular column using this method. I have been able to simply write each column in its own text file, but I'd rather be able to write the columns together in the same text file for future runs I do with this program.
Edit: my outputs will be 8 floating point numbers and to reiterate, they will be random each time.
So after 1 run, I will create 8 floating point values: Run11, Run12, Run13, Run14, Run15, Run16, Run17, Run18. After my second run, I will create another set of values (8 entries long): Run21, Run22, Run23, Run24, Run25, Run26, Run27, Run28.
In the text file, I would like these values to be placed in specific columns like this: http://imgur.com/zxoxaKM (this is what it would look like after 2 runs).
The "Value n:" titles are the headers for each column.

import csv
from tempfile import NamedTemporaryFile
from shutil import move
from itertools import chain
with open("in.csv") as f, NamedTemporaryFile(dir=".", delete=False) as temp:
r = csv.reader(f)
new = [9, 10, 11, 12, 13, 14, 15, 16]
wr = csv.writer(temp)
wr.writerows(zip(chain.from_iterable(r), new))
move(temp.name, "in.csv")
Input:
1
2
3
4
5
6
7
8
Output:
1,9
2,10
3,11
4,12
5,13
6,14
7,15
8,16
To take the header into account:
with open("in.csv") as f, NamedTemporaryFile(dir=".", delete=False) as temp:
r = csv.reader(f)
header = next(r)
new = [9, 10, 11, 12, 13, 14, 15, 16]
wr = csv.writer(temp)
wr.writerow(header+["Value {}:".format(len(header)+1)])
wr.writerows(zip(chain.from_iterable(r), new))
move(temp.name, "in.csv")
Input:
Value 1:
1
2
3
4
5
6
7
8
Output:
Value 1:,Value 2:
1,9
2,10
3,11
4,12
5,13
6,14
7,15
8,16
If you are adding an actual row each tie and not a column then just append:
with open("in.csv","a") as f:
new = [9, 10, 11, 12, 13, 14, 15, 16]
wr = csv.writer(f)
wr.writerow(new)
Input:
value 1:,value 2:,value 3:,value 4:,value 5:,value 6:,value 7:,value 8:
1,2,3,4,5,6,7,8
Output:
value 1:,value 2:,value 3:,value 4:,value 5:,value 6:,value 7:,value 8:
1,2,3,4,5,6,7,8
9,10,11,12,13,14,15,16

what about
a = [1, 2, 3, 4, 5, 6, 7, 8]
f = open('myFile.txt', 'a')
for n in a:
f.write('%d\t'%n)
f.write('\n')
f.close()
and you get as file content after running it 4 times
1 2 3 4 5 6 7 8
1 2 3 4 5 6 7 8
1 2 3 4 5 6 7 8
1 2 3 4 5 6 7 8
======= EDIT =========
Try this, its ugly but works ;-)
import os.path
h = ['Header 1', 'Hea 2', 'Header 3', 'Header 4', 'H 5', 'Header', 'Header 7', 'Header 8']
a = [1, 2, 3, 4, 5, 6, 7, 8]
fileName = 'myFile.txt'
#write header
withHeader = not os.path.isfile(fileName)
f = open(fileName, 'a')
if withHeader:
print 'Writing header'
for s in h:
f.write('%s\t'%s)
f.write('\n')
#write numbers
for i in range(0, len(a)):
space = len(h[i])/2;
n = a[i]
for c in range(0, space):
f.write(' ')
print 'Writing %d'%n
f.write('%d'%n)
for c in range(0, space):
f.write(' ')
f.write('\t')
f.write('\n')
f.close()
result:
Header 1 Hea 2 Header 3 Header 4 H 5 Header Header 7 Header 8
1 2 3 4 5 6 7 8
1 2 3 4 5 6 7 8
1 2 3 4 5 6 7 8
1 2 3 4 5 6 7 8