I have a DataFrame where I need to split the header into multiple rows as headers for the same Dataframe.
The dataframe looks like this,
My data Frame looks like follows,
gene ALL_ID_1 AML_ID_1 AML_ID_2 AML_ID_3 AML_ID_4 AML_ID_5 Stroma_ID_1 Stroma_ID_2 Stroma_ID_3 Stroma_ID_4 Stroma_ID_5 Stroma_CR_Pat_4 Stroma_CR_Pat_5 Stroma_CR_Pat_6 Stroma_CR_Pat_7 Stroma_CR_Pat_8
ENSG 8 1 11 5 10 0 628 542 767 578 462 680 513 968 415 623
ENSG 0 0 1 0 0 0 0 28 1 3 0 1 4 0 0 0
ENSG 661 1418 2580 6817 14727 5968 9 3 5 9 2 9 3 3 5 1
ENSG 20 315 212 8 790 471 1283 2042 1175 2839 1110 857 1880 1526 2262 2624
ENSG 11 26 24 9 11 2 649 532 953 463 468 878 587 245 722 484
And I want the the above header to be spitted as follows,
network ID ID REL
node B_ALL AML Stroma
hemi 1 1 2 3 4 5 1 2 3 4 5 6 7 8 9 10
ENSG 8 1 11 5 10 0 628 542 767 578 462 680 513 968 415 623
ENSG 0 0 1 0 0 0 0 28 1 3 0 1 4 0 0 0
ENSG 661 1418 2580 6817 14727 5968 9 3 5 9 2 9 3 3 5 1
ENSG 20 315 212 8 790 471 1283 2042 1175 2839 1110 857 1880 1526 2262 2624
ENSG 11 26 24 9 11 2 649 532 953 463 468 878 587 245 722 484
Any help is greatly appreciated ..
Probably not the best minimal example you put here, very few people has the subject knowledge to understand what is network, node and hemi in your context.
You just need to create your MultiIndex and replace your column index with the one you created:
There are 3 rules in your example:
1, whenever 'Stroma' is found, the column belongs to REL, otherwise belongs to ID.
2, node is the first field of the initial column names
3, hemi is the last field of the initial column names
Then, just code away:
In [110]:
df.columns = pd.MultiIndex.from_tuples(zip(np.where(df.columns.str.find('Stroma')!=-1, 'REL', 'ID'),
df.columns.map(lambda x: x.split('_')[0]),
df.columns.map(lambda x: x.split('_')[-1])),
names=['network', 'node', 'hemi'])
print df
network ID REL \
node ALL AML Stroma
hemi 1 1 2 3 4 5 1 2 3 4 5
gene
ENSG 8 1 11 5 10 0 628 542 767 578 462
ENSG 0 0 1 0 0 0 0 28 1 3 0
ENSG 661 1418 2580 6817 14727 5968 9 3 5 9 2
ENSG 20 315 212 8 790 471 1283 2042 1175 2839 1110
ENSG 11 26 24 9 11 2 649 532 953 463 468
network
node
hemi 4 5 6 7 8
gene
ENSG 680 513 968 415 623
ENSG 1 4 0 0 0
ENSG 9 3 3 5 1
ENSG 857 1880 1526 2262 2624
ENSG 878 587 245 722 484
Related
From the u.item file, which is divided into [100000 rows x 4columns],
I have to find out which are the best movies.
I try, for each unique item_id (which is 1682) to find the overall rating for each one separately
import pandas as pd
import csv
ratings = pd.read_csv("erg3/files/u.data", encoding="utf-8", delim_whitespace=True,
names = ["user_id", "item_id", "rating", "timestamp"]
)
The data has this form:
196 242 3 881250949
186 302 3 891717742
22 377 1 878887116
....
244 51 2 880606923
166 346 1 886397596
298 474 4 884182806
My expected output :
item_id
1 1753
2 420
3 273
4 742
...
1570 1
1486 1
1626 1
1580 1
i used this best_m = ratings.groupby("item_id")["rating"].sum()
followed by best_m = best_m.sort_values(ascending=False)
And the output looks like :
50 2541
100 2111
181 2032
258 1936
174 1786
...
1581 1
1570 1
1486 1
1626 1
1580 1
I am trying to stack several cyclical data elements of a DataFrame on top of each other to change the DataFrame dimensions. E.g. Go from 100x20 to 500x4.
Sample 11x7 input:
0 1 2 3 4 5 6 7
0 1 713 1622 658 1658 620 1734
1 2 714 1623 657 1700 618 1735
2 3 714 1624 656 1701 617 1736
3 4 714 1625 655 1702 615 1738
4 5 714 1626 654 1703 614 1739
5 6 713 1627 653 1705 612 1740
6 7 713 1628 651 1706 610 1741
7 8 713 1629 650 1707 609 1742
8 9 713 1630 649 1709 607 1744
9 10 713 1631 648 1710 605 1745
10 11 712 1632 646 1711 604 1746
Desired 32x3 output:
0 1 713 1622
1 2 714 1623
2 3 714 1624
3 4 714 1625
4 5 714 1626
5 6 713 1627
6 7 713 1628
7 8 713 1629
8 9 713 1630
9 10 713 1631
10 11 712 1632
11 1 658 1658
12 2 657 1700
13 3 656 1701
14 4 655 1702
15 5 654 1703
16 6 653 1705
17 7 651 1706
18 8 650 1707
19 9 649 1709
20 10 648 1710
21 11 646 1711
22 1 620 1734
23 2 618 1735
24 3 617 1736
25 4 615 1738
26 5 614 1739
27 6 612 1740
28 7 610 1741
29 8 609 1742
30 9 607 1744
31 10 605 1745
32 11 604 1746
I have spent an inordinate amount of time checking this, and I cannot find anything better than
pd.concat([df1, df2], ignore_index=True)
or
df1.append(df2, ignore_index=True)
, which should produce identical solutions inthis case. However, whichever option is used, it is going to be placed at the end of a loop that produces temporary DataFrames to be concatenated with or appended to the permanent DataFrame. The temp df's come out fine, but the allegedly straightforward concatenation step fails consistently. I get an empty DataFrame with a proper header...
for l in range(1,13):
s1 = l * 4 - 4
s2 = l * 4
dft = df0.iloc[:, s1:s2]
dft.columns = new_col
#pd.concat([df1, dft], ignore_index=True, axis = 0)
#df1.append(dft, ignore_index=True)
df1.head()
Either of the commented out lines should produce a stack of 4-wide temp DataFrames... I get an empty DataFrame with a proper header and no error messages...
Solved by #Aryerez in a comment above:
Both of pd.concat() and df.append() are by default not in place. See if df1 = pd.concat(etc...) solves it.
I have following unique values in dataframe column.
['1473' '1093' '1346' '1324' 'NA' '1129' '58' '847' '54' '831' '816']
I want to drop rows which have 'NA' in this column.
testData = testData[testData.BsmtUnfSF != "NA"]
and got error
TypeError: invalid type comparison
Then I tried
testData = testData[testData.BsmtUnfSF != np.NAN]
It doesn't give any error but it doesn't drop rows.
How to solve this issue?
Here is how you can do it. Just change column with the column name you want.
import pandas as pd
import numpy as np
df = pd.DataFrame({"column": [1,2,3,np.nan,6]})
df = df[np.isfinite(df['column'])]
You could use dropna
testData = testData.dropna(subsets = 'BsmtUnfSF']
assuming your dataFrame:
>>> df
col1
0 1473
1 1093
2 1346
3 1324
4 NaN
5 1129
6 58
7 847
8 54
9 831
10 816
You have multiple solutions:
>>> df[pd.notnull(df['col1'])]
col1
0 1473
1 1093
2 1346
3 1324
5 1129
6 58
7 847
8 54
9 831
10 816
>>> df[df.col1.notnull()]
# df[df['col1'].notnull()]
col1
0 1473
1 1093
2 1346
3 1324
5 1129
6 58
7 847
8 54
9 831
10 816
>>> df.dropna(subset=['col1'])
col1
0 1473
1 1093
2 1346
3 1324
5 1129
6 58
7 847
8 54
9 831
10 816
>>> df.dropna()
col1
0 1473
1 1093
2 1346
3 1324
5 1129
6 58
7 847
8 54
9 831
10 816
>>> df[~df.col1.isnull()]
col1
0 1473
1 1093
2 1346
3 1324
5 1129
6 58
7 847
8 54
9 831
10 816
I wrote a solution to this challenge . It successfully handles the example case given, but not the actual case.
Challenge:
A DNA string is a reverse palindrome if it is equal to its reverse complement. For instance, GCATGC is a reverse palindrome because its reverse complement is GCATGC. For example:
5'...GCATGC...3'
3'...CGTACG...5'
Given:
A DNA string of length at most 1 kbp in FASTA format.
Return:
The position and length of every reverse palindrome in the string
having length between 4 and 12. You may return these pairs in any
order.
Sample Dataset
>Rosalind_24 TCAATGCATGCGGGTCTATATGCAT
Sample Output
4 6
5 4
6 6
7 4
17 4
18 4
20 6
21 4
For the sample, it works. However it failed on the actual sample.
Actual Dataset:
>Rosalind_7901 ATATAGTCGGCTGTCCAGGCAATCGCGAGATGGGGAACGACATCTTGGTACTTTACGGAT GCCAAGACTTAATATCTGGCCCGGATATGACCGCGAGCACCCCCTACTCGTCTGTCGGTT TCGGCCGGCATGACCTGTCCTCTTGATAATAGATATAAGTTGCCAACCGCACTATTTCAA GATCAGATGCCCCAAGGCACAAGGCACAGAAGAATCAGGTACTGAGCAAACAGCGCCCAT TTGTCAGCGCAACTCCGAGCGACAGGCACAAGTGGTAGTAACATCTGTAGTCTACGAGCG CGGGACCGATGTAAAAAGCAACGAGAGACGGGGCCGTCGATAGAAAAGCAATGGAGTCCA TATGGGCACGCTGAGCGTGCCTGTACTAATTTCTATGGGCTACTGGCACTAGGGGCTTAA GCCCTCGGTTACCGCGCTTTATGAATATAGTTTTCGTGCCAGGAGTGTCTTGTTTCGAGG AAGCGTGAGCTACACTTAGCACGTCCGGGCTTATTGGAAATTTGTTCAGTCTGTATGCTC CGCAATATCATGTCGGCGCTCATTCAATGTTGCGTGTAATTTAGACCTCTACTACAGCTG GGGTTGGAGCGGTCGGTAGTAAGACGTATGATTACGGTTTACATCCCGCCGGCGGACACG GAACGTGATTTTCAGCATTGTCCCATCGTAGGGATTGGGGCCCTAGTAGGTGTGGGTAGC ACGTTACATGAAGCTATCCAATGGCGTATATACTCCATCCCATCGGACTAGAAGATTTGA GGGACCCAGTCATAACTGGTGCAAAATTACGTTACAAAAGCCGAGGATACAGTATA
Actual Output:
1 4
2 4
23 6
24 4
48 4
70 4
73 4
79 4
82 4
86 4
93 4
124 6
125 4
126 6
127 4
131 4
155 4
156 4
184 4
222 4
236 4
251 4
337 4
342 4
389 4
394 4
415 4
423 4
440 4
441 4
452 4
453 4
482 4
496 4
509 4
513 4
526 6
527 4
554 4
558 4
565 4
587 4
604 6
605 4
634 4
656 10
657 8
658 6
659 4
674 4
709 6
710 4
714 4
733 4
739 4
744 4
758 8
759 4
759 6
760 4
761 4
780 4
813 4
818 4
822 4
846 4
Code:
from string import maketrans
table=maketrans('ATCG','TAGC')
protein=open('rosalind_revp.txt','r').read()[14::].strip()
for i in range(len(protein)):
for ii in range(2,7):
if protein[i:i+ii]==protein[i+2*ii-1:i+ii-1:-1].translate(table):
print str(i+1),str(2*ii)
(When testing sample, the 4th line is
protein=open('rosalind_revp.txt','r').read()[12::].strip()
I even manually matched a bunch of the position-length pairs, and sad to find that they all worked perfectly. I still don't know why the result wasn't accepted.
Could anyone let me know where I was wrong?
This is my github link and it has the solution hope this works
def reverse(l):
t=""
for i in range(len(l)):
if(l[i]=='A'):
t=t+'T'
elif(l[i]=='T'):
t=t+'A'
elif(l[i]=='C'):
t=t+'G'
elif(l[i]=='G'):
t=t+'C'
return t
def rev(d):
return d[len(d)::-1]
k=input()
p=input()
for i in range(len(p)):
for j in range(4,14):
if (p[i:i+j]==rev(reverse(p[i:i+j]))and i+j<=len(p)):
print(i+1, end=" ")
print(j)
https://github.com/jssssv007/stackexcahnge
I have a dataframe which looks like this
Geneid PRKCZ.exon1 PRKCZ.exon2 PRKCZ.exon3 PRKCZ.exon4 PRKCZ.exon5 PRKCZ.exon6 PRKCZ.exon7 PRKCZ.exon8 PRKCZ.exon9 PRKCZ.exon10 ... FLNA.exon31 FLNA.exon32 FLNA.exon33 FLNA.exon34 FLNA.exon35 FLNA.exon36 FLNA.exon37 FLNA.exon38 MTCP1.exon1 MTCP1.exon2
S28 22 127 135 77 120 159 49 38 409 67 ... 112 104 37 83 47 18 110 70 167 19
22 3 630 178 259 142 640 77 121 521 452 ... 636 288 281 538 276 109 242 314 790 484
S04 16 658 320 337 315 881 188 162 769 577 ... 1291 420 369 859 507 208 554 408 1172 706
56 26 663 343 390 314 1090 263 200 844 592 ... 675 243 250 472 280 133 300 275 750 473
S27 13 1525 571 1081 560 1867 427 370 1348 1530 ... 1817 926 551 1554 808 224 971 1313 1293 701
5 rows × 8297 columns
In that above dataframe I need to add an extra column with information about the index. And so I made a list -healthy with all the index to be labelled as h and rest everything should be d.
And so tried the following lines:
healthy=['39','41','49','50','51','52','53','54','56']
H_type =pd.Series( ['h' for x in df.loc[healthy]
else 'd' for x in df]).to_frame()
But it is throwing me following error:
SyntaxError: invalid syntax
Any help would be really appreciated
In the end I am aiming something like this:
Geneid sampletype SSX4.exon4 SSX2.exon11 DUX4.exon5 SSX2.exon3 SSX4.exon5 SSX2.exon10 SSX4.exon7 SSX2.exon9 SSX4.exon8 ... SETD2.exon21 FAT2.exon15 CASC5.exon8 FAT1.exon21 FAT3.exon9 MLL.exon31 NACA.exon7 RANBP2.exon20 APC.exon16 APOB.exon4
S28 h 0 0 0 0 0 0 0 0 0 ... 2480 2003 2749 1760 2425 3330 4758 2508 4367 4094
22 h 0 0 0 0 0 0 0 0 0 ... 8986 7200 10123 12422 14528 18393 9612 15325 8788 11584
S04 h 0 0 0 0 0 0 0 0 0 ... 14518 16657 17500 15996 17367 17948 18037 19446 24179 28924
56 h 0 0 0 0 0 0 0 0 0 ... 17784 17846 20811 17337 18135 19264 19336 22512 28318 32405
S27 h 0 0 0 0 0 0 0 0 0 ... 10375 20403 11559 18895 18410 12754 21527 11603 16619 37679
Thank you
I think you can use numpy.where with isin, if Geneid is column.
EDIT by comment:
There can be integers in column Geneid, so you can cast to string by astype.
healthy=['39','41','49','50','51','52','53','54','56']
df['type'] = np.where(df['Geneid'].astype(str).isin(healthy), 'h', 'd')
#get last column to list
print df.columns[-1].split()
['type']
#create new list from last column and all columns without last
cols = df.columns[-1].split() + df.columns[:-1].tolist()
print cols
['type', 'Geneid', 'PRKCZ.exon1', 'PRKCZ.exon2', 'PRKCZ.exon3', 'PRKCZ.exon4',
'PRKCZ.exon5', 'PRKCZ.exon6', 'PRKCZ.exon7', 'PRKCZ.exon8', 'PRKCZ.exon9',
'PRKCZ.exon10', 'FLNA.exon31', 'FLNA.exon32', 'FLNA.exon33', 'FLNA.exon34',
'FLNA.exon35', 'FLNA.exon36', 'FLNA.exon37', 'FLNA.exon38', 'MTCP1.exon1', 'MTCP1.exon2']
#reorder columns
print df[cols]
type Geneid PRKCZ.exon1 PRKCZ.exon2 PRKCZ.exon3 PRKCZ.exon4 \
0 d S28 22 127 135 77
1 d 22 3 630 178 259
2 d S04 16 658 320 337
3 h 56 26 663 343 390
4 d S27 13 1525 571 1081
PRKCZ.exon5 PRKCZ.exon6 PRKCZ.exon7 PRKCZ.exon8 ... \
0 120 159 49 38 ...
1 142 640 77 121 ...
2 315 881 188 162 ...
3 314 1090 263 200 ...
4 560 1867 427 370 ...
FLNA.exon31 FLNA.exon32 FLNA.exon33 FLNA.exon34 FLNA.exon35 \
0 112 104 37 83 47
1 636 288 281 538 276
2 1291 420 369 859 507
3 675 243 250 472 280
4 1817 926 551 1554 808
FLNA.exon36 FLNA.exon37 FLNA.exon38 MTCP1.exon1 MTCP1.exon2
0 18 110 70 167 19
1 109 242 314 790 484
2 208 554 408 1172 706
3 133 300 275 750 473
4 224 971 1313 1293 701
[5 rows x 22 columns]
You could use pandas isin()
First add an extra column called 'sampletype' and fill it with 'd'. Then, find all samples that have a geneid in health and fill them with 'h'. Suppose your main dataframe is called df, then you would use something like:
healthy = ['39','41','49','50','51','52','53','54','56']
df['sampletype'] = 'd'
df['sampletype'][df['Geneid'].isin(healthy)]='h'