We Keep Coding

Getting Wrong Output from Operations Performed Between Two Pipe Delimited Files Converted to 2D Arrays

I'm new to coding and my instructions for this problem are:
"Given pipe-delimited files F1 and F2 where F1 contains accounts with fields ACCOUNT NUM|PIN|BALANCE and F2 contains transaction instructions COMMAND|AMOUNT|ACCOUNT NUM|PIN, execute the transactions, storing the results back in F1.
The COMMAND field will be add or sub indicating addition or subtraction from the account.
Transactions which do not provide the correct PIN code or attempt to put the account below zero should be ignored."
My output is:
1000|1234|10000
1020|2222|0
3000|3344|1000
2020|1234|90000
The correct output is:
1000|1234|11000
1020|2222|0
3000|3344|0
2020|1234|90000
The code I've written is:
import sys
F1= sys.argv[1]
F2= sys.argv[2]
records=[]
with open(F1,'r') as fp:
content=fp.readlines()
for row in content:
recList=row.strip("\n").split('|')
records.append(recList)
records2=[]
with open(F2,'r') as fp:
content=fp.readlines()
for row in content:
recList2=row.strip("\n").split('|')
records2.append(recList2)
for i in range(len(records)):
row=records[i]
for i in range(len(records2)):
row=records2[i]
for row in records and records2:
if records[i][1]==records2[i][3]:
if records2[i][0]=="add":
records[i][2]=int(records[i][2])+int(records2[i][1])
elif records2[i][0]=="sub":
if int(records[i][2])>=int(records2[i][1]):
records[i][2]=int(records[i][2])-int(records2[i][1])
else:
records[i][2]=records[i][2]
else:
break
print(records)

You haven't included the COMMANDS. The last numbers displayed in your output rows are without the addition or subtraction, which is why they are wrong.

# Create a function that turns pipe-delimited strings into 2d arrays
def pipe2a(text):
records= text.split("\n")
for i in range(0, len(records)):
records[i]= records[i].split("|")
return records
# Create a function that turns 2d arrays into pipe-delimited strings.
def a2pipe(a):
text= ""
for i in range(0, len(a)):
account= a[i]
for j in range(0, len(account)):
account[j]= str(account[j])
text = text + "|".join(account) + "\n"
return text;
# Read in the accounts and transactions
accounts= pipe2a(open(F1, 'r').read())
transactions= pipe2a(open(F2, 'r').read())
# Main Section
# for each transaction
for transactionIndex in range(0, len(transactions)):
transaction= transactions[transactionIndex]
if(len(transaction) >= 4):
# look through the accounts for the matching account
for accountIndex in range(0,len(accounts)):
account= accounts[accountIndex]
if(len(account) >= 3): # make sure we have
balance= int(account[2]) # enough fields
transactionAmount= int(transaction[1])
if(account[0] == transaction[2]): # account matches?
if(account[1] == transaction[3]): # pin code matches?
if(transaction[0] == 'add'):
accounts[accountIndex][2]= balance + transactionAmount
elif (transaction[0] == 'sub' and transactionAmount <= balance):
accounts[accountIndex][2]= balance - transactionAmount
# Write the answer back out to the original file
open(F1, 'w').write(a2pipe(accounts))
It works

python Pythagoras function

So I have a set of positional data that I comes from a factory sensor. It produces x, y and z info in meters from to a known lat/long position. I have a function that will convert the distance in meters from the lat/long but I need to use the x and y data in a Pythagoras function to determine that. Let me try to clarify with an example of the JSON data the sensor gives.
[
{
"id": "84eb18677194",
"name": "forklift_0001",
"areaId": "Tracking001",
"areaName": "Hall1",
"color": "#FF0000",
"coordinateSystemId": "CoordSys001",
"coordinateSystemName": null,
"covarianceMatrix": [
0.82,
-0.07,
-0.07,
0.55
],
"position": [ #this is the x,y and z data, in meters from the ref point
18.11,
33.48,
2.15
],
In this branch the forklift is 18.11m along and 33.38m up from the reference lat/long. The sensor is 2.15m high and that is a constant piece of info i don't need. To work out the distance from the reference point I need to use Pythagoras and then convert that data back into lat/long so my analysis tool can present it.
My problem (as far as python goes) is that I can't figure out how to make it see 18.11 & 33.38 as the x & y and tell it to disregard 2.15 entirely. Here is what i have so far.
import math
import json
import pprint
import os
from glob import iglob
rootdir_glob = 'C:/Users/username/Desktop/test_folder**/*"' # Note the
added asterisks, use forward slash
# This will return absolute paths
file_list = [f for f in
iglob('C:/Users/username/Desktop/test_folder/13/00**/*', recursive=True)
if os.path.isfile(f)]
for f in file_list:
print('Input file: ' + f) # Replace with desired operations
with open(f, 'r') as f:
distros = json.load(f)
output_file = 'position_data_blob_14' + str(output_nr) + '.csv' #output file name may be changed
def pythagoras(a,b):
value = math.sqrt(a*a + b*b)
return value
result = pythagoras(str(distro['position'])) #I am totally stuck here :/
print(result)
This piece of script is part of a wider project to parse the file by machine and people and also by work and non work times of day.
If someone could give me some tips on how to make the pythagorus part work i'd be really grateful. I am not sure if I should define it as a function but as I've typed this I am wondering if it should be a 'for' loop which uses the x & y and ignores the x.
All help really appreciated.

Try this:
position = distro['position'] # Get the full list
result = pythagoras(position[0], position[1]) # Get the first and second element from the list
print(result)
Why do you use str() for the argument of the function ? What were you trying to do ?

You're passing one input, a list of numbers, into a function that takes two numbers as input. There are two solutions to this - either change what you pass in, or change the function.
distro['position'] = [18.11, 33.48, 2.15], so for the first solution all you need to do is pass in distro['position'][0] and distro['position'][1]:
result = pythagoras(distro['position'][0], distro['position'][1])
Alternatively (which in my opinion is more elegant), pass in the list to the function and have the function extract the values it cares about:
result = pythagoras(distro['position'])
def pythagoras(input_triple):
a,b,c = input_triple
value = math.sqrt(a*a + b*b)
return value

The solution i used was
for f in file_list:
print('Input file: ' + f) # Replace with desired operations
with open(f, 'r') as f:
distros = json.load(f)
output_file = '13_01' + str(output_nr) + '.csv' #output file name may be changed
with open(output_file, 'w') as text_file:
for distro in distros:
position = distro['position']
result = math.sqrt(position[0]*position[0] + position[1]*position[1]),
print((result), file=text_file)
print('Output written to file: ' + output_file)
output_nr = output_nr + 1

Did you check with the data type of the parameters you're passing?
def pythagoras(a,b):
value = math.sqrt(int(a)**2 + int(b)**2)
return value
This is in the case of integers.

Merging lists obtained by a loop

I've only started python recently but am stuck on a problem.
# function that tells how to read the urls and how to process the data the
# way I need it.
def htmlreader(i):
# makes variable websites because it is used in a loop.
pricedata = urllib2.urlopen(
"http://website.com/" + (",".join(priceids.split(",")[i:i + 200]))).read()
# here my information processing begins but that is fine.
pricewebstring = pricedata.split("},{")
# results in [[1234,2345,3456],[3456,4567,5678]] for example.
array1 = [re.findall(r"\d+", a) for a in pricewebstring]
# writes obtained array to my text file
itemtxt2.write(str(array1) + '\n')
i = 0
while i <= totalitemnumber:
htmlreader(i)
i = i + 200
See the comments in the script as well.
This is in a loop and will each time give me an array (defined by array1).
Because I print this to a txt file it results in a txt file with separate arrays.
I need one big array so it needs to merge the results of htmlreader(i).
So my output is something like:
[[1234,2345,3456],[3456,4567,5678]]
[[6789,4567,2345],[3565,1234,2345]]
But I want:
[[1234,2345,3456],[3456,4567,5678],[6789,4567,2345],[3565,1234,2345]]
Any ideas how I can approach this?

Since you want to gather all the elements in a single list, you can simply gather them in another list, by flattening it like this
def htmlreader(i, result):
...
result.extend([re.findall(r"\d+", a) for a in pricewebstring])
i, result = 0, []
while i <= totalitemnumber:
htmlreader(i, result)
i = i + 200
itemtxt2.write(str(result) + '\n')
In this case, the result created by re.findall (a list) is added to the result list. Finally, you are writing the entire list as a whole to the file.
If the above shown method is confusing, then change it like this
def htmlreader(i):
...
return [re.findall(r"\d+", a) for a in pricewebstring]
i, result = 0, []
while i <= totalitemnumber:
result.extend(htmlreader(i))
i = i + 200

How does one update a FITS data array with PyFITS?

I'm trying to create a new FITS file out of two older ones using PyFITS.
import pyfits
from sys import stdout
from sys import argv
import time
file1 = argv[1]
file2 = argv[2]
hdu1 = pyfits.open(file1)
hdu2 = pyfits.open(file2)
new0 = hdu1[0]
new1 = hdu1[0]
sci1 = hdu1[0].data
sci2 = hdu2[0].data
for r in range(0, len(sci1)):
for c in range(0, len(sci1[r])):
add = sci1[r][c] + sci2[r][c]
new0.data[r][c] = add
for r in range(0, len(sci1)):
for c in range(0, len(sci1[r])):
print "(" + str(r) + ", " + str(c) + ") FirstVal = " + str(sci1[r][c]) + " || SecondVal = " + str(sci2[r][c])
print "\t New File/Add = " + str(new0.data[r][c])
All it prints out is the first value, i.e. sci1[r][c]. This means that the variable isn't being modified at all. How can I make it modify? I'm very new to using FITS.

What you have done here is make sci1 a reference to new0.data which means the assignment to new0 also changes sci1, so it is modifying the intended variable but your print loop is printing the same object twice.
If you want to have a copy instead of reference you have to use the objects copy method, in this case sci0 = new0.data.copy()
This is also not the way you are supposed to use numpy which pyfits uses to represent its images. Instead of loops you apply operations to full arrays which is in most cases easier to read and significantly faster. If you want to add two fits images represented as numpy arrays inplace:
new0.data += new1.data
print new0.data
or if you want to create a new image out of the sum of both inputs:
sum_image = new0.data + new1.data
# put it into an pyfits HDU (primary fits extension)
hdu = pyfits.PrimaryHDU(data=sum_image)

Splitting up a data file in Python, round 2

So I have recently found a solution to this question here
in which I wanted to take two columns of a data file and put them into two arrays
I now have this code which does the job nicely.
Xvals=[]; Yvals=[]
i = open('BGBiasRE_IM3_wCSrescaled.txt','r')
lines = [line.split() for line in i if line[:4] not in ('time', 'Step')]
Xvals, Yvals = zip(*lines)
V = [0, 0.004, 0, 0.0004]
pylab.plot(Xvals, Yvals, marker='o')
pylab.axis(V)
pylab.xlabel('Time (ms)')
pylab.ylabel('Current (A)')
pylab.title('Title')
pylab.show()
But I now realise I screwed up the question. I have a data file laid out as below,
time I(R_stkb)
Step Information: Temp=0 (Run: 1/11)
0.000000000000000e+000 0.000000e+000
9.999999960041972e-012 8.924141e-012
1.999999992008394e-011 9.623148e-012
Step Information: Temp=10 (Run: 2/11)
0.000000000000000e+000 0.000000e+000
9.999999960041972e-012 4.924141e-012
1.999999992008394e-011 8.623148e-012
(Note: No empty lines between each data line, and a Tab between the two data values)
The above code appends all the step information into one array, so I get two big long arrays when I want two different arrays for the different steps so I can plot their respective arrays separately later on. I also have to get the Step name, in this case Temp=10 and attach it/name the array to reflect each chunk of step info. Eg. I would like to end up with arrays like such
Temp0_Xvals = [ 0.000000000000000e+000, 9.999999960041972e-012, 1.999999992008394e-011]
Temp0_Yvals = [ 0.000000e+000, 8.924141e-012, 9.623148e-012]
Temp10_Xvals = [...]
Temp10_Yvals = [...] etc etc
Obviously this makes the problem much more complicated and I have no idea where to start.

I would do something along those lines:
i = open('BGBiasRE_IM3_wCSrescaled.txt', 'r')
Xnames, Ynames = [], []
count = 0
for line in i:
if count > 0:
line_tmp = line.split()
if line_tmp[0] == 'Step':
step = (line_tmp[2].split('='))[1]
varnameX = 'Temp' + str(step) +'_Xvals'
varnameY = 'Temp' + str(step) +'_Yvals'
globals()[varnameX] = []
globals()[varnameY] = []
Xnames.append(varnameX)
Ynames.append(varnameY)
else:
globals()[varnameX].append(float(line_tmp[0]))
globals()[varnameY].append(float(line_tmp[1]))
count += 1
i.close()
for name in Xnames:
print name + ' = ' + str(eval(name))
for name in Ynames:
print name + ' = ' + str(eval(name))
This is for sure not the most efficient solution but it works for your specific problem

Use csv with the excel_tab dialect.