We Keep Coding

convert crosstab to columns without using pandas in python

How do I convert the crosstab data from the input file mentioned below into columns based on the input list without using pandas?
Input list
[A,B,C]
Input data file
Labels A,B,C are only for representation, original file only has the numeric values.
We can ignore the colums XX & YY based on the length of the input list
A B C XX YY
A 0 2 3 4 8
B 4 0 6 4 8
C 7 8 0 5 8
Output (Output needs to have labels)
A A 0
A B 2
A C 3
B A 4
B B 0
B C 6
C A 7
C B 8
C C 0
The labels need to be present in the output file even though its present in the input file, hence I have mentioned its representation in the output file.
NB: In reality the labels are sorted city names without duplicates in ascending order & not single alphabets like A or B.
Unfortunately this would have been easier if I could install pandas on the server & use unstack(), but installations aren't allowed on this old server right now.
This is on python 3.5

Considering you tagged the post csv, I'm assuming the actual input data is a .csv file, without header as you indicated.
So example data would look like:
0,2,3,4,8
4,0,6,4,8
7,8,0,5,8
If the labels are provided as a list, matching the order of the columns and rows (i.e. ['A', 'B', 'C'] this would turn the example output into:
'A','A',0
'A','B',2
'A','C',3
'B','A',4
etc.
Note that this implies the number of rows and columns in the file cannot exceed the number of labels provided.
You indicate that the columns you label 'XX' and 'YY' are to be ignored, but you don't indicate how that's supposed to be communicated, but you do mention the length of the input is determining it, so I assume this means 'everything after column n can be ignored'.
This is a simple implementation:
from csv import reader
def unstack_csv(fn, columns, labels):
with open(fn) as f:
cr = reader(f)
row = 0
for line in cr:
col = 0
for x in line[:columns]:
yield labels[row], labels[col], x
col += 1
row += 1
print(list(unstack_csv('unstack.csv', 3, ['A', 'B', 'C'])))
or if you like it short and sweet:
from csv import reader
with open('unstack.csv') as f:
content = reader(f)
labels = ['A', 'B', 'C']
print([(labels[row], labels[col], x)
for row, data in enumerate(content)
for col, x in enumerate(data) if col < 3])
(I'm also assuming using numpy is out, for the same reason as pandas, but that stuff like csv is in, since it's a standard library)
If you don't want to provide the labels explicitly, but just want them generated, you could do something like:
def label(n):
r = n // 26
c = chr(65 + (n % 26))
if r > 0:
return label(r-1)+c
else:
return c
And then of course just remove the labels from the examples and replace with calls to label(col) and label(row).

df.query while using pandas in python produces empty results

I am learning how to manipulate data using pandas in python. I got the following script:
import pandas as pd
df = pd.read_table( "t.txt" ) #read in the file
df.columns = [x.strip() for x in df.columns] #strip spaces in headers
df = df.query('TLD == ".biz"') #select the rows where TLD == ".biz"
df.to_csv('t.txt', sep='\t') #write the output to a tab-separated file
but the output file has no records, headers only. When I check using
print.df
prior to the selection, the output is:
TLD Length Words \
0 .biz 5 ...
1 .biz 4 ...
2 .biz 5 ...
3 .biz 5 ...
4 .biz 3 ...
5 .biz 3 ...
6 .biz 6 ...
So I know the column TLD has rows with the .biz values. I also tried :
>>> print(df.loc[df['TLD'] == '.biz'])
but the results is
Empty DataFrame
With list of my columns
What am I doing wrong please?

It seems some whitespaces are there, so need remove them by strip:
print(df.loc[df['TLD'].str.strip() == '.biz'])
df['TLD'] = df['TLD'].str.strip()
df = df.query('TLD == ".biz"')

how to extract a range of data in my special data structure in python

I have a data file with similar special structure as below:
#F A 1 1 1 3 3 2
2 1 0.002796 0.000005 0.000008 -4.938531 1.039083
3 1 0.002796 0.000005 0.000007 -4.938531 1.039083
4 0 0.004961 -0.000008 -0.000002 -4.088534 0.961486
5 0 0.004961 0.000006 -0.000002 -4.079798 0.975763
First column is only a description (no need to be considered)and I want to (1)separate all data that their second column is 1 from the ones that their second column is 0 and then (2)extract the data lines that their 5th number(for example in first data line, it will be 0.000008) is in a specific range and then took the 6th number of that line (for our example it would be -4.938531), then take average of all of them( captured 6th values) and finally write them in a new file. For that I wrote this code that although does not include the first task, also it is not working. could anyone please help me with debugging or suggest me a new method?
A=0.0 #to be used for separating data as mentioned in the first task
B=0.0 #to be used for separating data as mentioned in the first task
with open('inputdatafile') as fin, open('outputfile','w') as fout:
for line in fin:
if line.startswith("#"):
continue
else:
col = line.split()
6th_val=float(col[-2])
2nd_val=int(col[1])
if (str(float(col[6])) > 0.000006 and str(float(col[6])) < 0.000009):
fout.write(" ".join(col) + "\n")
else:
del line

Varaible names in python can't start with a number, so change 6th_val to val_6 and 2nd_val to val_2.
str(float(col[6])) produces string, which can't be compared with float '0.000006', so change any str(float(...)) > xxx to float(...) > xxx .
You don't have to delete line, garabage collector does it for you, so remove 'del line'
A=0.000006
B=0.000009
S=0.0
C=0
with open('inputdatafile') as fin, open('outputfile','w') as fout:
for line in fin:
if line.startswith("#"):
continue
else:
col = line.split()
if col[1] == '1':
val_6=float(col[-2])
val_5=int(col[-3])
if val_5 > A and val_5 < B:
fout.write(" ".join(col) + "\n")
s += val_6
c += 1
fout.write("Average 6th: %f\n" % (S/C))

How to skip more then one lines of header in RDD in Spark

Data in my first RDD is like
1253
545553
12344896
1 2 1
1 43 2
1 46 1
1 53 2
Now the first 3 integers are some counters that I need to broadcast.
After that all the lines have the same format like
1 2 1
1 43 2
I will map all those values after 3 counters to a new RDD after doing some computation with them in function.
But I'm not able to understand how to separate those first 3 values and map the rest normally.
My Python code is like this
documents = sc.textFile("file.txt").map(lambda line: line.split(" "))
final_doc = documents.map(lambda x: (int(x[0]), function1(int(x[1]), int(x[2])))).reduceByKey(lambda x, y: x + " " + y)
It works only when first 3 values are not in the text file but with them it gives error.
I don't want to skip those first 3 values, but store them in 3 broadcast variables and then pass the remaining dataset in map function.
And yes the text file has to be in that format only. I cannot remove those 3 values/counters
Function1 is just doing some computation and returning the values.

Imports for Python 2
from __future__ import print_function
Prepare dummy data:
s = "1253\n545553\n12344896\n1 2 1\n1 43 2\n1 46 1\n1 53 2"
with open("file.txt", "w") as fw: fw.write(s)
Read raw input:
raw = sc.textFile("file.txt")
Extract header:
header = raw.take(3)
print(header)
### [u'1253', u'545553', u'12344896']
Filter lines:
using zipWithIndex
content = raw.zipWithIndex().filter(lambda kv: kv[1] > 2).keys()
print(content.first())
## 1 2 1
using mapPartitionsWithIndex
from itertools import islice
content = raw.mapPartitionsWithIndex(
lambda i, iter: islice(iter, 3, None) if i == 0 else iter)
print(content.first())
## 1 2 1
NOTE: All credit goes to pzecevic and Sean Owen (see linked sources).

In my case I have a csv file like below
----- HEADER START -----
We love to generate headers
#who needs comment char?
----- HEADER END -----
colName1,colName2,...,colNameN
val__1.1,val__1.2,...,val__1.N
Took me a day to figure out
val rdd = spark.read.textFile(pathToFile) .rdd
.zipWithIndex() // get tuples (line, Index)
.filter({case (line, index) => index > numberOfLinesToSkip})
.map({case (line, index) => l}) //get rid of index
val ds = spark.createDataset(rdd) //convert rdd to dataset
val df=spark.read.option("inferSchema", "true").option("header", "true").csv(ds) //parse csv
Sorry code in scala, however can be easily converted to python

First take the values using take() method as zero323 suggested
raw = sc.textfile("file.txt")
headers = raw.take(3)
Then
final_raw = raw.filter(lambda x: x != headers)
and done.

Formatting list data by table

I am trying to analyse some data, but my data contains letters which require standardising. What I would like to be able to do is, for every datatable in the data (this csv data contains 3 datatables) replace the letter T or any other letter for that matter with the next highest integer for that table. The first table contains no errors, the second table contains 1 T and the third contains 2 x t's.
DatatableA,1
DatatableA,2
DatatableA,3
DatatableA,4
DatatableA,5
DatatableB,1
DatatableB,6
DatatableB,T
DatatableB,3
DatatableB,4
DatatableB,5
DatatableB,2
DatatableC,3
DatatableC,4
DatatableC,2
DatatableC,1
DatatableC,Q
DatatableC,5
DatatableC,T
I am expecting this to be a relatively easy thing to code, however whilst I know how to replace all T's with a number, within a particular column or a particular row, I do not know how to replace each T with a different number depending on the Datatable it is in. Essentially I am looking to produce the following from the above:
DatatableA,1
DatatableA,2
DatatableA,3
DatatableA,4
DatatableA,5
DatatableB,1
DatatableB,6
DatatableB,7
DatatableB,3
DatatableB,4
DatatableB,5
DatatableB,2
DatatableC,3
DatatableC,4
DatatableC,2
DatatableC,1
DatatableC,6
DatatableC,5
DatatableC,6
Here nothing happened in DatatableA, DatatableB the only T was replaced with the next highest integer in this case it was replaced with a 7, in DatatableC there was two anomalous data points which were both replaced with the next highest integer, which was a 6.
If anyone can point me in the right direction or provide a snippet of something, It would be greatly appreciated. As always constructive comments are also appreciated.
Edit in reply to elyase
I attempted to run the code:
import pandas as pd
df = pd.read_csv('test.csv', sep=',', header=None, names=['datatable', 'col'])
def replace_letter(group):
letters = group.isin(['T', 'Q']) # select letters
group[letters] = int(group[~letters].max()) + 1 # replace by next max
return group
df['col'] = df.groupby('datatable').transform(replace_letter)
print df
and i received the traceback:
Traceback (most recent call last):
File "C:/test.py", line 11, in <module>
df['col'] = df.groupby('datatable').transform(replace_letter)
File "C:\Python27\lib\site-packages\pandas\core\groupby.py", line 1981, in transform
res = path(group)
File "C:\Python27\lib\site-packages\pandas\core\groupby.py", line 2006, in <lambda>
slow_path = lambda group: group.apply(lambda x: func(x, *args, **kwargs), axis=self.axis)
File "C:\Python27\lib\site-packages\pandas\core\frame.py", line 4416, in apply
return self._apply_standard(f, axis)
File "C:\Python27\lib\site-packages\pandas\core\frame.py", line 4491, in _apply_standard
raise e
ValueError: ("invalid literal for int() with base 10: 'col'", u'occurred at index col')
Is there something I have used in correctly, I could use AEAs answer, but I have been meaning to use pandas more, as the library seems so useful for data manipulations.

Pandas is ideal for this kind of tasks:
Read your csv:
>>> import pandas as pd
>>> df = pd.read_csv('data.csv', sep=',', header=None, names=['datatable', 'col'])
>>> df.head()
datatable col
0 DatatableA 1
1 DatatableA 2
2 DatatableA 3
3 DatatableA 4
4 DatatableA 5
Group, select and replace max:
def replace_letter(group):
letters = group.isin(['T', 'Q']) # select letters
group[letters] = int(group[~letters].max()) + 1 # replace by next max
return group
>>> df['col'] = df.groupby('datatable').transform(replace_letter)
>>> df
datatable col
0 DatatableA 1
1 DatatableA 2
2 DatatableA 3
3 DatatableA 4
4 DatatableA 5
5 DatatableB 1
6 DatatableB 6
7 DatatableB 7
8 DatatableB 3
9 DatatableB 4
10 DatatableB 5
11 DatatableB 2
12 DatatableC 3
13 DatatableC 4
14 DatatableC 2
15 DatatableC 1
16 DatatableC 6
17 DatatableC 5
18 DatatableC 6
Write to csv:
df.to_csv('result.csv', index=None, header=None)

I suppose I have to answer the question asked my by own alter-ego. Seriously, does StackExchange not sanitize usernames?
Here's a solution, not guaranteeing that it's efficient or simple, but the logic is pretty simple. First you iterate your dataset and check for anything that's not an integer string and record the largest value. Then you iterate again and replace non-integer strings.
I am using StringIO as a replacement for a file just for convenience sake.
import csv
import string
from StringIO import StringIO
raw = """DatatableA,1
DatatableA,2
DatatableA,3
DatatableA,4
DatatableA,5
DatatableB,1
DatatableB,6
DatatableB,T
DatatableB,3
DatatableB,4
DatatableB,5
DatatableB,2
DatatableC,3
DatatableC,4
DatatableC,2
DatatableC,1
DatatableC,Q
DatatableC,5
DatatableC,T"""
fp = StringIO()
fp.write(raw)
fp.seek(0)
reader = csv.reader(fp)
data = []
mapping = {}
for row in reader:
if row[0] not in mapping:
mapping[row[0]] = float("-inf")
if row[1] in string.digits:
x = int(row[1])
if x > mapping[row[0]]:
mapping[row[0]] = x
data.append(row)
for i, row in enumerate(data):
if row[1] not in string.digits:
mapping[row[0]] += 1
row[1] = str(mapping[row[0]])
fp.close()
fp = StringIO()
writer = csv.writer(fp)
writer.writerows(data)
print fp.getvalue()