We Keep Coding

Data sorting, combining 2 lines

I have a file looking this way:
;1;108/1;4, 109
;1;51;4, 5
;2;109/2;4, 5
;2;108/2;4, 109
;3;108/2;4, 109
;3;51;4, 5
;4;109/2;4, 5
;4;51;4, 5
;5;109/2;4, 5
;5;40/6;5, 6, 7
where
;id1;id2;position_on_shelf_id2
;id1;id3;position_on_shelf_id3
as a result, i want to get:
id1;id2-id3;x
where x are common shelf positions for both id2 and id3, it should look like this
1;108/1-51;4
2;109/2-108/2;4
3;108/2-51;4
4;109/2-51;4, 5
5;109/2-40/6;5
my script works fine up to the moment where I need to type common shelf positions. I tried using .intersection, but it is not working properly, when I have positions consisting of double characters (pos:144-result: 14; pos:551, result: 51; pos:2222-result: 2 i.e)
result = id2_chars.intersection(id3_chars)
any fix for intersection? or maybe some better method on your mind?
code so far:
part1 - merge every 2nd line together
exp = open('output.txt', 'w')
with open("dane.txt") as f:
content = f.readlines()
strng = ""
for i in range(1,len(content)+1):
strng += content[i-1].strip()
if i % 2 == 0:
exp.writelines(strng + '\n')
strng = ""
exp.close()
part2 - intersection:
exp = open('output2.txt', 'w')
imp = open('output.txt')
for line in imp:
none, lp1, dz1, poz1, lp2, dz2, poz2 = line.split(';')
s1 = poz1.lower()
s2 = poz2.lower()
s1_chars = set(s1)
s2_chars = set(s2)
result = s1_chars.intersection(s2_chars)
result = str(result)
exp.writelines(lp1 + ';' + dz1 + '-' + dz2 + ';' + result + '\n')
exp.close()
** i did not filtered the result for my needs yet (it is in "list" form), but it won't be a problem once I get the right intersection result

Your main problem is that you try to intersect 2 sets of characters while you should intersect positions. So you should at least use:
...
s1 = poz1.lower()
s2 = poz2.lower()
s1_poz= set(x.strip() for x in s1.split(','))
s2_poz = set(x.strip() for x in s1.split(','))
result = s1_poz.intersection(s2_poz)
result = ', '.join(result)
...
But in fact, you could easily do the whole processing in one single pass:
exp = open('output.txt', 'w')
with open("dane.txt") as f:
old = None
for line in f: # one line at a time is enough
line = line.strip()
if old is None: # first line of a block, just store it
old = line
else: # second line of a bock, process both
none, lp1, dz1, poz1 = old.split(';')
none, lp2, dz2, poz2 = line.split(';')
poz1x = set(x.strip() for x in poz1.tolower().split(','))
poz2x = set(x.strip() for x in poz2.tolower().split(','))
result = ', '.join(poz1x.intersection(poz2x))
exp.write(lp1 + ';' + dz1 + '-' + dz2 + ';' + result + '\n')
old = None

Extracting data from a text file to an output file

I have alot of files which names are just number. (Starting from 1 to whatever is the maximum number) and each of these files are similar to each other by their "tags" (ObjectID =, X =, Y =, etc.), but the values after those tags are not the same at all.
I wanted to make my job easier from manually copy/pasting the data from one file to another and made a small script using Python (since I am slightly experienced in it).
This is the full script:
import os
BASE_DIRECTORY = 'C:\Users\Tom\Desktop\TheServer\scriptfiles\Objects'
output_file = open('output.txt', 'w')
output = {}
file_list = []
for (dirpath, dirnames, filenames) in os.walk(BASE_DIRECTORY):
for f in filenames:
if 'txt' in str(f):
e = os.path.join(str(dirpath), str(f))
file_list.append(e)
for f in file_list:
print f
txtfile = open(f, 'r')
output[f] = []
for line in txtfile:
if 'ObjectID =' in line:
output[f].append(line)
elif 'X =' in line:
output[f].append(line)
elif 'Y =' in line:
output[f].append(line)
tabs = []
for tab in output:
tabs.append(tab)
tabs.sort()
for tab in tabs:
for row in output[tab]:
output_file.write(row + '')
Now, everything is working fine, the output file looks like this:
ObjectID = 1216
X = -1480.500610
Y = 2610.885742
ObjectID = 970
X = -1517.210693
Y = 2522.842285
ObjectID = 3802
X = -1512.156616
Y = 2521.116210
etc.
But I don't want it to be like that (each value has a new line). I need it to do this for every file:
Read the file.
Remove the tags infront of the values.
Format a single line which will have those values in the output folder. (Let's say I want to make it look like this: "(1216,-1480.500610,2522.842285)" )
Write that line in the output folder.
Repeat for every file.
Any help please?

Hope this helps.
data = open('sam.txt', 'r').read()
>>> print data
ObjectID = 1216
X = -1480.500610
Y = 2610.885742
ObjectID = 970
X = -1517.210693
Y = 2522.842285
ObjectID = 3802
X = -1512.156616
Y = 2521.116210
>>>
Now lets do some string replacements :)
>>> data = data.replace('ObjectID =', '').replace('\nX = ', ',').replace('\nY = ', ',')
>>> print data
1216,-1480.500610,2610.885742
970,-1517.210693,2522.842285
3802,-1512.156616,2521.116210

In your loop, keep track of whether you are 'in' a record:
records = []
in_record = False
id, x, y = 0, 0, 0
for line in txtfile:
if not in_record:
if 'ObjectID =' in line:
in_record = True
id = line[10:]
elif 'X =' in line:
x = line[3:]
elif 'Y =' in line:
y = line[3:]
records.append((id, x, y))
in_record = False
Then you'll have a list of tuples which you can easily write with the csv module.

Find here a version of the loop you have generating the contents.
I rewrote it so the line contents ObjectId, X and Y are in the same line.
It looks that is what you want to do:
for f in file_list:
print f
txtfile = open(f, 'r')
output[f] = []
for line in txtfile:
myline = ''
if 'ObjectID =' in line:
pos = line.rfind("ObjectID =") + len("ObjectID =")
rest = line[pos:]
# Here you set the delimiter after the ObjectID value. Can be ","
numbers = rest.split(" ")
if len(numbers) > 0:
myline.append(numbers[0])
elif 'X =' in line:
pos = line.rfind("X =") + len("X =")
rest = line[pos:]
# Here you set the delimiter after the ObjectID value. Can be ","
numbers = rest.split(" ")
if len(numbers) > 0:
myline.append(numbers[0])
elif 'Y =' in line:
pos = line.rfind("Y =") + len("Y =")
rest = line[pos:]
# Here you set the delimiter after the ObjectID value. Can be ","
numbers = rest.split(" ")
if len(numbers) > 0:
myline.append(numbers[0])
output[f].append(myline)
Note that you need to know which character (in the code the delimiter) separates the names you try to find: ObjectID = from the actual values you want to grab from the line.

Here is what you need. I did not have enough time to write the code for appending the result to a new file. Instead it just prints it, but you get the point.
import os.path
path = "path"
#getting the number of files in your folder
num_files = len([f for f in os.listdir(path)
if os.path.isfile(os.path.join(path, f))])
#function that returns your desired output for a given file
def file_head_ext(file_path, file_num):
with open(file_path + "/" + file_num) as myfile:
head = [next(myfile).split("=") for x in range(3)]
formatted_head = [elm[1].replace("\n",'').replace(" ","") for elm in head]
return(",".join(formatted_head))
for filnum in range(1,num_files):
print(file_head_ext(path, str(filnum)))

Making A Statistics Text File (Python)

I'm making a program that takes a name and certain input in the form of numbers and gives them a score i want this score to be saved in a text file and i want to be able to do it multiple times but when I write to the file it overwrites the last stat is there anyway to change this
Here is the function I'm using:
def calculate():
try:
a = float(enter1.get())
b = float(enter2.get())
c = float(enter3.get())
d = float(enter4.get())
e = float(enter5.get())
f = float(enter6.get())
result =(a+b+(c*2)+(d*2)+e-f)*2.5
n = result
w = "Score:"
label7.config(text=str(result))
myfile = open('Stats.txt','w')
x = str(enter0.get())
y =("(%s) %s" % (w, n))
myfile.write(x)
myfile.write(y)
myfile.close()
except ValueError:
label7.config(text='Enter Numbers!',fg="white")

Maybe change
myfile = open('Stats.txt','w')
into
myfile = open('Stats.txt','a') # append

Making columns of data lists in python

So I have a program, that reads through a bunch of files and appends the necessary data that I need. I need to now take those particular data and show them as a list. To be more specific, these are the parameters I have:
a = Source, b = luminosity, c = luminosity error, d = HST, e = XRS, f = gmag, g = z, and h = rh
I want to display this in a list, each defining a particular column. I just don't know where exactly I should insert the print statement among the various for loops I've done to do this.
I would appreciate any help! Here's the program (the main focus is in the for loops done and how they iterate through the data, and don't worry about indentations, the program so far works I just need to display the data appended in columns):
import sys
import os
import re
import urllib
import urllib2
from os.path import basename
import urlparse
import shutil
base_dirname = '/projects/XRB_Web/apmanuel/499/'
base_sourcefile = base_dirname + 'Sources.txt'
try:
file = open(base_sourcefile, 'r')
except IOError:
print 'Cannot open: '+base_sourcefile
Source = []
Finallist = []
ACS = []
SRC = []
for line in file:
data_line_check = (line.strip())
if data_line_check:
line = re.sub(r'\s+', ' ', line)
point = line.split('|')
temp_source = (point[0]).strip()
if temp_source and len(point) == 3:
Source = (point[0]).strip()
Source = re.sub(r'\s', '_', Source)
print Source+"\n"
temp_finallist = (point[1]).strip()
if temp_finallist:
Finallistaddress = (point[1]).strip()
Finallistaddress = re.sub(r'\s', '_', Finallistaddress)
Luminositybase_dirname1 = '/projects/XRB_Web/apmanuel/499/Lists/' + Finallistaddress
try:
file2 = open(Luminositybase_dirname1, 'r')
except IOError:
print 'Cannot open: '+Luminositybase_dirname1
source = []
luminosity = []
luminosityerr = []
for line in file2:
pointy = line.split()
a = int(pointy[0])
b = float(pointy[5])
c = float(pointy[6])
source.append(a)
luminosity.append(b)
luminosityerr.append(c)
temp_HST = (point[2]).strip()
if temp_HST:
HSTaddress = (point[2]).strip()
HSTaddress = re.sub(r'\s', '_', HSTaddress)
HSTbase_dirname2 = '/projects/XRB_Web/apmanuel/499/Lists/' + HSTaddress
try:
file3 = open(HSTbase_dirname2, 'r')
except IOError:
print 'Cannot open: '+HSTbase_dirname2
HST = []
for line in file3:
pointy2 = line.split()
d = int(pointy2[0])
HST.append(d)
temp_XRS = (point[3]).strip()
if temp_XRS:
XRSaddress = (point[3]).strip()
XRSaddress =re.sub(r'\s', '_', XRSaddress)
XRSbase_dirname3 = '/projects/XRB_Web/apmanuel/499/Lists/' + XRSaddress
try:
file4 = open(XRSbase_dirname3, 'r')
except IOError:
print 'Cannot open: '+XRSbase_dirname3
XRS = []
for line in file4:
pointy3 = line.split()
e = int(pointy3[0])
XRS.append(e)
temp_others = (point[4]).strip()
if temp_others:
othersaddress = (point[4]).strip()
othersaddress =re.sub(r'\s', '_', othersaddress)
othersbase_dirname4 = '/projects/XRB_Web/apmanuel/499/Lists/' + othersaddress
try:
file5 = open(othersbase_dirname4, 'r')
except IOError:
print 'Cannot open: '+othersbase_dirname4
gmag = []
z = []
rh = []
for line in file5:
pointy4 = line.split()
f = float(pointy4[3])
g = float(pointy4[5])
h = float(pointy4[7])
rh.append(f)
gmag.append(g)
z.append(h)

this function will return columns for a list of rows. note that this requires the lists to all have an element in the column you are trying to access, though it would be relatively simple to change this if you need it.
def getcolumn(matrix,index): #index specifies which column of the matrix you want. note that like all other list indexes, this starts from 0, not one.
column = []
for row in matrix:
column.append(row[index])
return column

Python: parsing binary stl file

I'm having some difficulties while parsing a binary STL file with Python (2.7.1 32-bit and Windows 7 64). The file is a about 450k in size, but my parser suddenly stops working after parsing 244 triangles out of ~8600 with en exception of struct.unpack:
Exception unpack requires a string argument of length 12
The cursor position in the file is line 33, row 929. But the line contains about 3400 characters. So it doesn't seem to be newline problem.
This is the code:
import struct
normals = []
points = []
triangles = []
bytecount = []
fb = [] # debug list
def unpack (f, sig, l):
s = f.read (l)
fb.append(s)
return struct.unpack(sig, s)
def read_triangle(f):
n = unpack(f,"<3f", 12)
p1 = unpack(f,"<3f", 12)
p2 = unpack(f,"<3f", 12)
p3 = unpack(f,"<3f", 12)
b = unpack(f,"<h", 2)
normals.append(n)
l = len(points)
points.append(p1)
points.append(p2)
points.append(p3)
triangles.append((l, l+1, l+2))
bytecount.append(b[0])
def read_length(f):
length = struct.unpack("#i", f.read(4))
return length[0]
def read_header(f):
f.seek(f.tell()+80)
def write_as_ascii(outfilename):
f = open(outfilename, "w")
f.write ("solid "+outfilename+"\n")
for n in range(len(triangles)):
f.write ("facet normal {} {} {}\n".format(normals[n][0],normals[n][1],normals[n][2]))
f.write ("outer loop\n")
f.write ("vertex {} {} {}\n".format(points[triangles[n][0]][0],points[triangles[n][0]][1],points[triangles[n][0]][2]))
f.write ("vertex {} {} {}\n".format(points[triangles[n][1]][0],points[triangles[n][1]][1],points[triangles[n][1]][2]))
f.write ("vertex {} {} {}\n".format(points[triangles[n][2]][0],points[triangles[n][2]][1],points[triangles[n][2]][2]))
f.write ("endloop\n")
f.write ("endfacet\n")
f.write ("endsolid "+outfilename+"\n")
f.close()
def main():
infilename = r"cupHemis46_28.stl"
outfilename = r"cupHemis46_28_ascii_test.stl"
try:
f = open ( infilename, "r")
read_header(f)
l = read_length(f)
try:
while True:
read_triangle(f)
except Exception, e:
print "Exception",e[0]
print len(normals), len(points), len(triangles), l
write_as_ascii(outfilename)
except Exception, e:
print e
if __name__ == '__main__':
main()
The unpack function (not from struct) collects all the strings which will be written to a file. When I compare both file they seem equal, up to the file position where unpack stops working. I opened the binary file with Notepad++, the next character is a "SUB".
Are there any restrictions of unpack that I'm not aware of regarding file size or limitation in characters or something? Is there something wrong with my code? Thanks in advance.

Your unpack function calls f.read twice. I suspect you've walked off the end of the file.
You'll also have trouble with reading the file in text mode on Windows. Any incidental occurrences of \r\n will be read in as \n. Make the following change to avoid this problem.
f = open(infilename, "rb")