Is there a method in turning user input into csv format?

Is there a method in turning user input into csv format? - python

This is the example data that would be pasted into an input() prompt and ideally I would like it to be processed and made into a csv file through python:
,,,,,,Performance,Performance,Performance,Performance,Performance,Performance,Performance,Performance,Performance,Performance,Performance,Performance,Performance,Expected,Expected,Expected,SCA,SCA,Passes,Passes,Passes,Passes,Carries,Carries,Dribbles,Dribbles,-additional
Player,#,Nation,Pos,Age,Min,Gls,Ast,PK,PKatt,Sh,SoT,CrdY,CrdR,Touches,Press,Tkl,Int,Blocks,xG,npxG,xA,SCA,GCA,Cmp,Att,Cmp%,Prog,Carries,Prog,Succ,Att,-9999
Gabriel Jesus,9,br BRA,FW,25-124,82,0,0,0,0,1,0,0,0,40,13,1,1,0,0.1,0.1,0.0,4,0,20,27,74.1,2,33,1,4,5,b66315ae
Eddie Nketiah,14,eng ENG,FW,23-067,8,0,0,0,0,0,0,0,0,6,2,0,0,0,0.0,0.0,0.1,2,0,4,4,100.0,1,4,1,0,0,a53649b7
Martinelli,11,br BRA,LW,21-048,90,1,0,0,0,2,1,0,0,38,21,0,2,1,0.6,0.6,0.1,1,0,24,28,85.7,1,34,5,3,4,48a5a5d6
Bukayo Saka,7,eng ENG,RW,20-334,90,0,0,0,0,3,0,0,0,52,23,3,0,3,0.2,0.2,0.0,2,1,24,36,66.7,2,37,8,2,2,bc7dc64d
Martin Ødegaard,8,no NOR,AM,23-231,89,0,0,0,0,2,0,0,0,50,22,2,1,2,0.1,0.1,0.0,2,0,30,39,76.9,5,28,3,1,2,79300479
Albert Sambi Lokonga,23,be BEL,CM,22-287,1,0,0,0,0,0,0,0,0,2,0,0,0,0,0.0,0.0,0.0,0,0,1,1,100.0,0,1,1,0,0,1b4f1169
Granit Xhaka,34,ch SUI,DM,29-312,90,0,0,0,0,0,0,1,0,60,5,0,2,3,0.0,0.0,0.0,4,0,42,49,85.7,6,32,2,0,0,e61b8aee
Thomas Partey,5,gh GHA,DM,29-053,90,0,0,0,0,1,0,0,0,62,25,7,1,2,0.1,0.1,0.0,0,0,40,47,85.1,5,26,4,0,1,529f49ab
Oleksandr Zinchenko,35,ua UKR,LB,25-233,82,0,1,0,0,1,1,0,0,64,16,3,3,1,0.0,0.0,0.3,2,1,44,54,81.5,6,36,5,0,0,51cf8561
Kieran Tierney,3,sct SCO,LBWB,25-061,8,0,0,0,0,0,0,0,0,6,1,0,0,0,0.0,0.0,0.0,0,0,2,4,50.0,0,1,0,0,0,fce2302c
Gabriel Dos Santos,6,br BRA,CB,24-229,90,0,0,0,0,0,0,0,0,67,5,1,1,2,0.0,0.0,0.0,0,0,52,58,89.7,1,48,3,0,0,67ac5bb8
William Saliba,12,fr FRA,CB,21-134,90,0,0,0,0,0,0,0,0,58,3,1,2,2,0.0,0.0,0.0,0,0,42,46,91.3,1,35,1,0,0,972aeb2a
Ben White,4,eng ENG,RB,24-301,90,0,0,0,0,0,0,1,0,61,22,7,4,5,0.0,0.0,0.1,1,0,29,40,72.5,5,25,2,1,1,35e413f1
Aaron Ramsdale,1,eng ENG,GK,24-083,90,0,0,0,0,0,0,0,0,33,0,0,0,0,0.0,0.0,0.0,0,0,24,32,75.0,0,21,0,0,0,466fb2c5
14 Players,,,,,990,1,1,0,0,10,2,2,0,599,158,25,17,21,1.1,1.1,0.5,18,2,378,465,81.3,35,361,36,11,15,-9999
The link to the table is: https://fbref.com/en/matches/e62f6e78/Crystal-Palace-Arsenal-August-5-2022-Premier-League#stats_18bb7c10_summary
I have attempted to use pandas dataframe but I am only able to export the first row of headers and nothing else (only the items before player).

Would have been nice for you to include your attempt.
Pandas works just fine:
import pandas as pd
url = 'https://fbref.com/en/matches/e62f6e78/Crystal-Palace-Arsenal-August-5-2022-Premier-League#stats_18bb7c10_summary'
df = pd.read_html(url)[10]
cols = [f'{each[0]}_{each[1]}' if 'Unnamed' not in each[0] else f'{each[1]}' for each in df.columns]
df.columns = cols
df.to_csv('output.csv', index=False)
Output:
print(df.to_markdown())
| | Player | # | Nation | Pos | Age | Min | Gls | Ast | PK | PKatt | Sh | SoT | CrdY | CrdR | Touches | Press | Tkl | Int | Blocks | xG | npxG | xA | SCA | GCA | Cmp | Att | Cmp% | Prog | Carries | Prog.1 | Succ | Att.1 |
|---:|:---------------------|----:|:---------|:------|:-------|------:|------:|------:|-----:|--------:|-----:|------:|-------:|-------:|----------:|--------:|------:|------:|---------:|-----:|-------:|-----:|------:|------:|------:|------:|-------:|-------:|----------:|---------:|-------:|--------:|
| 0 | Gabriel Jesus | 9 | br BRA | FW | 25-124 | 82 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 40 | 13 | 1 | 1 | 0 | 0.1 | 0.1 | 0 | 4 | 0 | 20 | 27 | 74.1 | 2 | 33 | 1 | 4 | 5 |
| 1 | Eddie Nketiah | 14 | eng ENG | FW | 23-067 | 8 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 6 | 2 | 0 | 0 | 0 | 0 | 0 | 0.1 | 2 | 0 | 4 | 4 | 100 | 1 | 4 | 1 | 0 | 0 |
| 2 | Martinelli | 11 | br BRA | LW | 21-048 | 90 | 1 | 0 | 0 | 0 | 2 | 1 | 0 | 0 | 38 | 21 | 0 | 2 | 1 | 0.6 | 0.6 | 0.1 | 1 | 0 | 24 | 28 | 85.7 | 1 | 34 | 5 | 3 | 4 |
| 3 | Bukayo Saka | 7 | eng ENG | RW | 20-334 | 90 | 0 | 0 | 0 | 0 | 3 | 0 | 0 | 0 | 52 | 23 | 3 | 0 | 3 | 0.2 | 0.2 | 0 | 2 | 1 | 24 | 36 | 66.7 | 2 | 37 | 8 | 2 | 2 |
| 4 | Martin Ødegaard | 8 | no NOR | AM | 23-231 | 89 | 0 | 0 | 0 | 0 | 2 | 0 | 0 | 0 | 50 | 22 | 2 | 1 | 2 | 0.1 | 0.1 | 0 | 2 | 0 | 30 | 39 | 76.9 | 5 | 28 | 3 | 1 | 2 |
| 5 | Albert Sambi Lokonga | 23 | be BEL | CM | 22-287 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 2 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 100 | 0 | 1 | 1 | 0 | 0 |
| 6 | Granit Xhaka | 34 | ch SUI | DM | 29-312 | 90 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 60 | 5 | 0 | 2 | 3 | 0 | 0 | 0 | 4 | 0 | 42 | 49 | 85.7 | 6 | 32 | 2 | 0 | 0 |
| 7 | Thomas Partey | 5 | gh GHA | DM | 29-053 | 90 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 62 | 25 | 7 | 1 | 2 | 0.1 | 0.1 | 0 | 0 | 0 | 40 | 47 | 85.1 | 5 | 26 | 4 | 0 | 1 |
| 8 | Oleksandr Zinchenko | 35 | ua UKR | LB | 25-233 | 82 | 0 | 1 | 0 | 0 | 1 | 1 | 0 | 0 | 64 | 16 | 3 | 3 | 1 | 0 | 0 | 0.3 | 2 | 1 | 44 | 54 | 81.5 | 6 | 36 | 5 | 0 | 0 |
| 9 | Kieran Tierney | 3 | sct SCO | LB,WB | 25-061 | 8 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 6 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 2 | 4 | 50 | 0 | 1 | 0 | 0 | 0 |
| 10 | Gabriel Dos Santos | 6 | br BRA | CB | 24-229 | 90 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 67 | 5 | 1 | 1 | 2 | 0 | 0 | 0 | 0 | 0 | 52 | 58 | 89.7 | 1 | 48 | 3 | 0 | 0 |
| 11 | William Saliba | 12 | fr FRA | CB | 21-134 | 90 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 58 | 3 | 1 | 2 | 2 | 0 | 0 | 0 | 0 | 0 | 42 | 46 | 91.3 | 1 | 35 | 1 | 0 | 0 |
| 12 | Ben White | 4 | eng ENG | RB | 24-301 | 90 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 61 | 22 | 7 | 4 | 5 | 0 | 0 | 0.1 | 1 | 0 | 29 | 40 | 72.5 | 5 | 25 | 2 | 1 | 1 |
| 13 | Aaron Ramsdale | 1 | eng ENG | GK | 24-083 | 90 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 33 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 24 | 32 | 75 | 0 | 21 | 0 | 0 | 0 |
| 14 | 14 Players | nan | nan | nan | nan | 990 | 1 | 1 | 0 | 0 | 10 | 2 | 2 | 0 | 599 | 158 | 25 | 17 | 21 | 1.1 | 1.1 | 0.5 | 18 | 2 | 378 | 465 | 81.3 | 35 | 361 | 36 | 11 | 15 |

could you elaborate more?
maybe you could split the raw text by comma and then convert it to a dataframe
like:
list_of_string = input.split(',')
df = pd.DataFrame(list_of_string)
df.to_csv('yourfile.csv')

The correct approach is as proposed by chitown88, however if you want to copy paste the data by hand into the terminal and get a csv you can do something like this:
import pandas as pd
from datetime import datetime
while True:
print("Enter/Paste your content. Ctrl-D or Ctrl-Z ( windows ) to save it.")
contents = []
while True:
try:
line = input()
except EOFError:
break
contents.append(line)
df = pd.DataFrame(contents)
df.to_csv(f"df_{int(datetime.now().timestamp())}.csv", index=None)
Start the Python script, paste the data into the terminal, press CTRL+D and press enter to export the data you pasted into the terminal into a csv file.

You can use user input controlled while loop to get user input. Finally, you may exit depending on the user’s choice. Look at the code below:
user_input = 'Y'
while user_input.lower() == 'y':
# Run your code here.
user_input = input('Do you want to add one more entry: Y or N?')

This is most intuitive and understandable solution I could come up with uses of basic linear algebra to solve the problem which I find pretty neat. I recommend you to find an another way to parse the data. Check out beautifulsoup and requests.
import pandas as pd#for dataframe
data = '''
,,,,,,Performance,Performance,Performance,Performance,Performance,Performance,Performance,Performance,Performance,Performance,Performance,Performance,Performance,Expected,Expected,Expected,SCA,SCA,Passes,Passes,Passes,Passes,Carries,Carries,Dribbles,Dribbles,-additional
Player,#,Nation,Pos,Age,Min,Gls,Ast,PK,PKatt,Sh,SoT,CrdY,CrdR,Touches,Press,Tkl,Int,Blocks,xG,npxG,xA,SCA,GCA,Cmp,Att,Cmp%,Prog,Carries,Prog,Succ,Att,-9999
Gabriel Jesus,9,br BRA,FW,25-124,82,0,0,0,0,1,0,0,0,40,13,1,1,0,0.1,0.1,0.0,4,0,20,27,74.1,2,33,1,4,5,b66315ae
Eddie Nketiah,14,eng ENG,FW,23-067,8,0,0,0,0,0,0,0,0,6,2,0,0,0,0.0,0.0,0.1,2,0,4,4,100.0,1,4,1,0,0,a53649b7
Martinelli,11,br BRA,LW,21-048,90,1,0,0,0,2,1,0,0,38,21,0,2,1,0.6,0.6,0.1,1,0,24,28,85.7,1,34,5,3,4,48a5a5d6
Bukayo Saka,7,eng ENG,RW,20-334,90,0,0,0,0,3,0,0,0,52,23,3,0,3,0.2,0.2,0.0,2,1,24,36,66.7,2,37,8,2,2,bc7dc64d
Martin Ødegaard,8,no NOR,AM,23-231,89,0,0,0,0,2,0,0,0,50,22,2,1,2,0.1,0.1,0.0,2,0,30,39,76.9,5,28,3,1,2,79300479
Albert Sambi Lokonga,23,be BEL,CM,22-287,1,0,0,0,0,0,0,0,0,2,0,0,0,0,0.0,0.0,0.0,0,0,1,1,100.0,0,1,1,0,0,1b4f1169
Granit Xhaka,34,ch SUI,DM,29-312,90,0,0,0,0,0,0,1,0,60,5,0,2,3,0.0,0.0,0.0,4,0,42,49,85.7,6,32,2,0,0,e61b8aee
Thomas Partey,5,gh GHA,DM,29-053,90,0,0,0,0,1,0,0,0,62,25,7,1,2,0.1,0.1,0.0,0,0,40,47,85.1,5,26,4,0,1,529f49ab
Oleksandr Zinchenko,35,ua UKR,LB,25-233,82,0,1,0,0,1,1,0,0,64,16,3,3,1,0.0,0.0,0.3,2,1,44,54,81.5,6,36,5,0,0,51cf8561
Kieran Tierney,3,sct SCO,LBWB,25-061,8,0,0,0,0,0,0,0,0,6,1,0,0,0,0.0,0.0,0.0,0,0,2,4,50.0,0,1,0,0,0,fce2302c
Gabriel Dos Santos,6,br BRA,CB,24-229,90,0,0,0,0,0,0,0,0,67,5,1,1,2,0.0,0.0,0.0,0,0,52,58,89.7,1,48,3,0,0,67ac5bb8
William Saliba,12,fr FRA,CB,21-134,90,0,0,0,0,0,0,0,0,58,3,1,2,2,0.0,0.0,0.0,0,0,42,46,91.3,1,35,1,0,0,972aeb2a
Ben White,4,eng ENG,RB,24-301,90,0,0,0,0,0,0,1,0,61,22,7,4,5,0.0,0.0,0.1,1,0,29,40,72.5,5,25,2,1,1,35e413f1
Aaron Ramsdale,1,eng ENG,GK,24-083,90,0,0,0,0,0,0,0,0,33,0,0,0,0,0.0,0.0,0.0,0,0,24,32,75.0,0,21,0,0,0,466fb2c5
14 Players,,,,,990,1,1,0,0,10,2,2,0,599,158,25,17,21,1.1,1.1,0.5,18,2,378,465,81.3,35,361,36,11,15,-9999
'''
#you can just replace data with user input
def tryNum(x):#input a value and if its a number then it returns a number, if not it returns itself back
try:
x = float(x)
return x
except:
return x
rows = [i.split(',')[:-1] for i in data.split('\n')[2:-2]]#removing useless lines
col_names = [i for i in rows[0]]#fetching all column names
cols = [[tryNum(rows[j][i]) for j in range(1,len(rows))] for i in range(len(rows[0]))]#get all column info by transposing the "matrix" if you will
full = {}#setting up the dictionary
for i,y in zip(col_names,cols):#putting the data in the dict
full[i]=y
df = pd.DataFrame(data = full)#uploading it all to the df
print(df.head())

Related

Pandas cant print the list of objects collected from web using xpath in Jupyter notebook

This is the code I used. I am using Jupternotebook web version. I upgraded the XML, and python version is 3.8.
import numpy as np
import requests
from lxml import html
import csv
import pandas as pd
# getting the web content
r = requests.get('http://www.pro-football-reference.com/years/2017/draft.htm')
data = html.fromstring(r.text)
collecting specific data
pick = data.xpath('//td[#data_stat="draft_pick"]//text()')
player = data.xpath('//td[#data_stat="player"]//text()')
position = data.xpath('//td[#data_stat="pos"]//text()')
age= data.xpath('//td[#data_stat="age"]//text()')
games_played = data.xpath('//td[#data_stat="g"]//text()')
cmp = data.xpath('//td[#data_stat="pass_cmp"]//text()')
att = data.xpath('//td[#data_stat="pass_att"]//text()')
college = data.xpath('//td[#data_stat="college_id"]//text()')
data = list(zip(pick,player,position,age,games_played,cmp,att,college))
df = pd.DataFrame(data)
df
There are two errors showing on two separate files I tried:
<module 'pandas' from 'C:\Users\anaconda3\lib\site-packages\pandas\init.py'>
AttributeError: 'list' object has no attribute 'xpath'
The code is not giving me the list of data I wanted from the webpage. Can anyone help me out with this? Thank you in advance.

You can load html tables directly into a dataframe using read_html:
import pandas as pd
df = pd.read_html('http://www.pro-football-reference.com/years/2017/draft.htm')[0]
df.columns = df.columns.droplevel(0) # drop top header row
df = df[df['Rnd'].ne('Rnd')] # remove mid-table header rows
Output:
| | Rnd | Pick | Tm | Player | Pos | Age | To | AP1 | PB | St | CarAV | DrAV | G | Cmp | Att | Yds | TD | Int | Att | Yds | TD | Rec | Yds | TD | Solo | Int | Sk | College/Univ | Unnamed: 28_level_1 |
|---:|------:|-------:|:-----|:------------------|:------|------:|-----:|------:|-----:|-----:|--------:|-------:|----:|------:|------:|------:|-----:|------:|------:|------:|-----:|------:|------:|-----:|-------:|------:|------:|:---------------|:----------------------|
| 0 | 1 | 1 | CLE | Myles Garrett | DE | 21 | 2020 | 1 | 2 | 4 | 35 | 35 | 51 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 107 | nan | 42.5 | Texas A&M | College Stats |
| 1 | 1 | 2 | CHI | Mitchell Trubisky | QB | 23 | 2020 | 0 | 1 | 3 | 33 | 33 | 51 | 1010 | 1577 | 10609 | 64 | 37 | 190 | 1057 | 8 | 0 | 0 | 0 | nan | nan | nan | North Carolina | College Stats |
| 2 | 1 | 3 | SFO | Solomon Thomas | DE | 22 | 2020 | 0 | 0 | 2 | 15 | 15 | 48 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 73 | nan | 6 | Stanford | College Stats |
| 3 | 1 | 4 | JAX | Leonard Fournette | RB | 22 | 2020 | 0 | 0 | 3 | 25 | 20 | 49 | 0 | 0 | 0 | 0 | 0 | 763 | 2998 | 23 | 170 | 1242 | 2 | nan | nan | nan | LSU | College Stats |
| 4 | 1 | 5 | TEN | Corey Davis | WR | 22 | 2020 | 0 | 0 | 4 | 25 | 25 | 56 | 0 | 0 | 0 | 0 | 0 | 6 | 55 | 0 | 207 | 2851 | 11 | nan | nan | nan | West. Michigan | College Stats |

Filter all rows from groupby object

I have a dataframe like below
+-----------+------------+---------------+------+-----+-------+
| InvoiceNo | CategoryNo | Invoice Value | Item | Qty | Price |
+-----------+------------+---------------+------+-----+-------+
| 1 | 1 | 77 | 128 | 1 | 10 |
| 1 | 1 | 77 | 101 | 1 | 11 |
| 1 | 2 | 77 | 105 | 3 | 12 |
| 1 | 3 | 77 | 129 | 2 | 10 |
| 2 | 1 | 21 | 145 | 1 | 9 |
| 2 | 2 | 21 | 130 | 1 | 12 |
+-----------+------------+---------------+------+-----+-------+
I want to filter the entire group, if any of the items in the list item_list = [128,129,130] is present in that group, after grouping by 'InvoiceNo' &'CategoryNo'.
My desired out put is as below
+-----------+------------+---------------+------+-----+-------+
| InvoiceNo | CategoryNo | Invoice Value | Item | Qty | Price |
+-----------+------------+---------------+------+-----+-------+
| 1 | 1 | 77 | 128 | 1 | 10 |
| 1 | 1 | 77 | 101 | 1 | 11 |
| 1 | 3 | 77 | 129 | 2 | 10 |
| 2 | 2 | 21 | 130 | 1 | 12 |
+-----------+------------+---------------+------+-----+-------+
I know how to filter a dataframe using isin(). But, not sure how to do it with groupby()
so far i have tried below
import pandas as pd
df = pd.read_csv('data.csv')
item_list = [128,129,130]
df.groupby(['InvoiceNo','CategoryNo'])['Item'].isin(item_list)
but nothing happens. please guide me how to solve this issue.

You can do something like this:
s = (df['Item'].isin(item_list)
.groupby([df['InvoiceNo'], df['CategoryNo']])
.transform('any')
)
df[s]

Logical indexing in pandas dataframes [duplicate]

This question already has answers here:
How do I Pandas group-by to get sum?
(11 answers)
Closed 3 years ago.
I have some data like this:
+-----------+---------+-------+
| Duration | Outcome | Event |
+-----------+---------+-------+
| 421 | 0 | 1 |
| 421 | 0 | 1 |
| 261 | 0 | 1 |
| 24 | 0 | 1 |
| 27 | 0 | 1 |
| 613 | 0 | 1 |
| 2454 | 0 | 1 |
| 227 | 0 | 1 |
| 2560 | 0 | 1 |
| 229 | 0 | 1 |
| 2242 | 0 | 1 |
| 6680 | 0 | 1 |
| 1172 | 0 | 1 |
| 5656 | 0 | 1 |
| 5082 | 0 | 1 |
| 7239 | 0 | 1 |
| 127 | 0 | 1 |
| 128 | 0 | 1 |
| 128 | 0 | 1 |
| 7569 | 1 | 1 |
| 324 | 0 | 2 |
| 6395 | 0 | 2 |
| 6196 | 0 | 2 |
| 31 | 0 | 2 |
| 228 | 0 | 2 |
| 274 | 0 | 2 |
| 270 | 0 | 2 |
| 275 | 0 | 2 |
| 232 | 0 | 2 |
| 7310 | 0 | 2 |
| 7644 | 1 | 2 |
| 6949 | 0 | 3 |
| 6903 | 1 | 3 |
| 6942 | 0 | 4 |
| 7031 | 1 | 4 |
+-----------+---------+-------+
Now, for each Event, with the Outcome 0/1 considered as Fail/Pass, I want to sum the total Duration of Fail/Pass events separately in 2 new columns (or 1, whatever ensures readability).
I'm new to dataframes and I feel significant logical indexing is involved here. What is the best way to approach this problem?

df.groupby(['Event', 'Outcome'])['Duration'].sum()
So you group by both the event then the outcome, look at the duration column then take the sum of each group.

You can also try:
pd.pivot_table(index='Event',
columns='Outcome',
values='Duration',
data=df,
aggfunc='sum')
which gives you a table with two columns:
+---------+-------+------+
| Outcome | 0 | 1 |
+---------+-------+------+
| Event | | |
+---------+-------+------+
| 1 | 35691 | 7569 |
| 2 | 21535 | 7644 |
| 3 | 6949 | 6903 |
| 4 | 6942 | 7031 |
+---------+-------+------+

Assigning states of Hidden Markov Models by idealized values intensity values.

I'm running the pomegranate HMM (http://pomegranate.readthedocs.io/en/latest/HiddenMarkovModel.html) on my data, and I load the results into a Pandas DF, and define the idealized intensity as the median of all the points in that state: df["hmm_idealized"] = df.groupby(["hmm_state"],as_index = False)["Raw"].transform("median"). Sample data:
+-----+-----------------+-------------+------------+
| | hmm_idealized | hmm_state | hmm_diff |
|-----+-----------------+-------------+------------|
| 0 | 99862 | 3 | nan |
| 1 | 99862 | 3 | 0 |
| 2 | 99862 | 3 | 0 |
| 3 | 99862 | 3 | 0 |
| 4 | 99862 | 3 | 0 |
| 5 | 99862 | 3 | 0 |
| 6 | 117759 | 4 | 1 |
| 7 | 117759 | 4 | 0 |
| 8 | 117759 | 4 | 0 |
| 9 | 117759 | 4 | 0 |
| 10 | 117759 | 4 | 0 |
| 11 | 117759 | 4 | 0 |
| 12 | 117759 | 4 | 0 |
| 13 | 117759 | 4 | 0 |
| 14 | 124934 | 2 | -2 |
| 15 | 124934 | 2 | 0 |
| 16 | 124934 | 2 | 0 |
| 17 | 124934 | 2 | 0 |
| 18 | 124934 | 2 | 0 |
| 19 | 117759 | 4 | 2 |
| 20 | 117759 | 4 | 0 |
| 21 | 117759 | 4 | 0 |
| 22 | 117759 | 4 | 0 |
| 23 | 117759 | 4 | 0 |
| 24 | 117759 | 4 | 0 |
| 25 | 117759 | 4 | 0 |
| 26 | 117759 | 4 | 0 |
| 27 | 117759 | 4 | 0 |
| 28 | 117759 | 4 | 0 |
| 29 | 117759 | 4 | 0 |
| 30 | 117759 | 4 | 0 |
| 31 | 117759 | 4 | 0 |
| 32 | 117759 | 4 | 0 |
| 33 | 117759 | 4 | 0 |
| 34 | 117759 | 4 | 0 |
| 35 | 117759 | 4 | 0 |
| 36 | 117759 | 4 | 0 |
| 37 | 117759 | 4 | 0 |
| 38 | 117759 | 4 | 0 |
| 39 | 117759 | 4 | 0 |
| 40 | 106169 | 1 | -3 |
| 41 | 106169 | 1 | 0 |
| 42 | 106169 | 1 | 0 |
| 43 | 106169 | 1 | 0 |
| 44 | 106169 | 1 | 0 |
| 45 | 106169 | 1 | 0 |
| 46 | 106169 | 1 | 0 |
| 47 | 106169 | 1 | 0 |
| 48 | 106169 | 1 | 0 |
| 49 | 106169 | 1 | 0 |
| 50 | 106169 | 1 | 0 |
| 51 | 106169 | 1 | 0 |
| 52 | 106169 | 1 | 0 |
| 53 | 106169 | 1 | 0 |
| 54 | 106169 | 1 | 0 |
| 55 | 106169 | 1 | 0 |
| 56 | 106169 | 1 | 0 |
| 57 | 106169 | 1 | 0 |
| 58 | 106169 | 1 | 0 |
| 59 | 106169 | 1 | 0 |
| 60 | 106169 | 1 | 0 |
| 61 | 106169 | 1 | 0 |
| 62 | 106169 | 1 | 0 |
| 63 | 106169 | 1 | 0 |
| 64 | 106169 | 1 | 0 |
| 65 | 106169 | 1 | 0 |
| 66 | 106169 | 1 | 0 |
| 67 | 106169 | 1 | 0 |
| 68 | 106169 | 1 | 0 |
| 69 | 106169 | 1 | 0 |
| 70 | 106169 | 1 | 0 |
| 71 | 106169 | 1 | 0 |
| 72 | 106169 | 1 | 0 |
| 73 | 106169 | 1 | 0 |
| 74 | 106169 | 1 | 0 |
| 75 | 99862 | 3 | 2 |
| 76 | 99862 | 3 | 0 |
| 77 | 99862 | 3 | 0 |
| 78 | 99862 | 3 | 0 |
| 79 | 99862 | 3 | 0 |
| 80 | 99862 | 3 | 0 |
| 81 | 99862 | 3 | 0 |
| 82 | 99862 | 3 | 0 |
| 83 | 99862 | 3 | 0 |
| 84 | 99862 | 3 | 0 |
| 85 | 99862 | 3 | 0 |
| 86 | 99862 | 3 | 0 |
| 87 | 99862 | 3 | 0 |
| 88 | 99862 | 3 | 0 |
| 89 | 99862 | 3 | 0 |
| 90 | 99862 | 3 | 0 |
| 91 | 99862 | 3 | 0 |
| 92 | 99862 | 3 | 0 |
| 93 | 99862 | 3 | 0 |
| 94 | 99862 | 3 | 0 |
| 95 | 99862 | 3 | 0 |
| 96 | 99862 | 3 | 0 |
| 97 | 99862 | 3 | 0 |
| 98 | 99862 | 3 | 0 |
| 99 | 99862 | 3 | 0 |
| 100 | 99862 | 3 | 0 |
| 101 | 99862 | 3 | 0 |
| 102 | 99862 | 3 | 0 |
| 103 | 99862 | 3 | 0 |
| 104 | 99862 | 3 | 0 |
| 105 | 99862 | 3 | 0 |
| 106 | 99862 | 3 | 0 |
| 107 | 99862 | 3 | 0 |
| 108 | 94127 | 0 | -3 |
| 109 | 94127 | 0 | 0 |
| 110 | 94127 | 0 | 0 |
| 111 | 94127 | 0 | 0 |
| 112 | 94127 | 0 | 0 |
| 113 | 94127 | 0 | 0 |
| 114 | 94127 | 0 | 0 |
| 115 | 94127 | 0 | 0 |
| 116 | 94127 | 0 | 0 |
| 117 | 94127 | 0 | 0 |
| 118 | 94127 | 0 | 0 |
| 119 | 94127 | 0 | 0 |
| 120 | 94127 | 0 | 0 |
| 121 | 94127 | 0 | 0 |
| 122 | 94127 | 0 | 0 |
| 123 | 94127 | 0 | 0 |
| 124 | 94127 | 0 | 0 |
| 125 | 94127 | 0 | 0 |
| 126 | 94127 | 0 | 0 |
| 127 | 94127 | 0 | 0 |
| 128 | 94127 | 0 | 0 |
| 129 | 94127 | 0 | 0 |
| 130 | 94127 | 0 | 0 |
| 131 | 94127 | 0 | 0 |
| 132 | 94127 | 0 | 0 |
| 133 | 94127 | 0 | 0 |
| 134 | 94127 | 0 | 0 |
| 135 | 94127 | 0 | 0 |
| 136 | 94127 | 0 | 0 |
| 137 | 94127 | 0 | 0 |
| 138 | 94127 | 0 | 0 |
| 139 | 94127 | 0 | 0 |
| 140 | 94127 | 0 | 0 |
| 141 | 94127 | 0 | 0 |
| 142 | 94127 | 0 | 0 |
| 143 | 94127 | 0 | 0 |
| 144 | 94127 | 0 | 0 |
| 145 | 94127 | 0 | 0 |
| 146 | 94127 | 0 | 0 |
| 147 | 94127 | 0 | 0 |
| 148 | 94127 | 0 | 0 |
| 149 | 94127 | 0 | 0 |
| 150 | 94127 | 0 | 0 |
| 151 | 94127 | 0 | 0 |
| 152 | 94127 | 0 | 0 |
| 153 | 94127 | 0 | 0 |
| 154 | 94127 | 0 | 0 |
| 155 | 94127 | 0 | 0 |
| 156 | 94127 | 0 | 0 |
| 157 | 94127 | 0 | 0 |
| 158 | 94127 | 0 | 0 |
| 159 | 94127 | 0 | 0 |
| 160 | 94127 | 0 | 0 |
| 161 | 94127 | 0 | 0 |
| 162 | 94127 | 0 | 0 |
| 163 | 94127 | 0 | 0 |
| 164 | 94127 | 0 | 0 |
| 165 | 94127 | 0 | 0 |
| 166 | 94127 | 0 | 0 |
| 167 | 94127 | 0 | 0 |
| 168 | 94127 | 0 | 0 |
| 169 | 94127 | 0 | 0 |
| 170 | 94127 | 0 | 0 |
| 171 | 94127 | 0 | 0 |
| 172 | 94127 | 0 | 0 |
| 173 | 94127 | 0 | 0 |
| 174 | 94127 | 0 | 0 |
| 175 | 94127 | 0 | 0 |
| 176 | 94127 | 0 | 0 |
| 177 | 94127 | 0 | 0 |
| 178 | 94127 | 0 | 0 |
| 179 | 94127 | 0 | 0 |
| 180 | 94127 | 0 | 0 |
| 181 | 94127 | 0 | 0 |
| 182 | 94127 | 0 | 0 |
| 183 | 94127 | 0 | 0 |
| 184 | 94127 | 0 | 0 |
| 185 | 94127 | 0 | 0 |
| 186 | 94127 | 0 | 0 |
| 187 | 94127 | 0 | 0 |
| 188 | 94127 | 0 | 0 |
| 189 | 94127 | 0 | 0 |
| 190 | 94127 | 0 | 0 |
| 191 | 94127 | 0 | 0 |
| 192 | 94127 | 0 | 0 |
| 193 | 94127 | 0 | 0 |
| 194 | 94127 | 0 | 0 |
| 195 | 94127 | 0 | 0 |
| 196 | 94127 | 0 | 0 |
| 197 | 94127 | 0 | 0 |
| 198 | 94127 | 0 | 0 |
| 199 | 94127 | 0 | 0 |
| 200 | 94127 | 0 | 0 |
+-----+-----------------+-------------+------------+
When analyzing the results, I want to count the number of increases in the model. I would like to use the df.where(data.hmm_diff > 0).count() function to read off how many times I increment by one state. However, the increase sometimes spans two states (i.e. it skips the middle state), so I need to reassign the HMM-state label by sorting the idealized values, so that the lowest state would be 0, the highest 4, etc. Is there a way to reassign the hmm_state label from arbitrary to dependent on idealized intensity?
For example,the hmm_state labeled as "1" lies in between of hmm_state 3 and 4

It looks like you just need to define a sorted HMM state like this:
state_orders = {v: i for i, v in enumerate(sorted(df.hmm_idealized.unique()))}
df['sorted_state'] = df.hmm_idealized.map(state_orders)
Then you can continue as you did in the question, but taking a diff on this column, and counting the jumps on it.

where condition is met, get last row pandas

+------------+----+----+----+-----+----+----+----+-----+
| WS | 1 | 2 | 3 | 4 | 1 | 2 | 3 | 4 |
+------------+----+----+----+-----+----+----+----+-----+
| w1 | 0 | 0 | 0 | 50 | 0 | 0 | 0 | 50 |
| w2 | 0 | 30 | 0 | 0 | 0 | 30 | 0 | 0 |
| d1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| d2 | 62 | 0 | 0 | 0 | 62 | 0 | 0 | 0 |
| Total | 62 | 30 | 0 | 50 | 62 | 30 | 0 | 50 |
| Cumulative | 62 | 92 | 92 | 142 | 62 | 92 | 92 | 142 |
+------------+----+----+----+-----+----+----+----+-----+
Based on the condition of the column having value more than 0, I would like to get the corresponding value of row "Cumulative".
As shown in the image, when 50 > 0, I would like to get the corresponding "Cumulative" value of 142.
+------------+----+----+---+-----+----+----+---+-----+
| WS | 1 | 2 | 3 | 4 | 1 | 2 | 3 | 4 |
+------------+----+----+---+-----+----+----+---+-----+
| Cumulative | 62 | 92 | 0 | 142 | 62 | 92 | 0 | 142 |
+------------+----+----+---+-----+----+----+---+-----+
I have tried pandas loc and iloc but they cannot perform what I wanted.
Thank you in advanced!

You definitely should have designed your question better, but anyway, here is a possible solution:
Assuming df is your DataFrame:
df[df>5].loc['Cumulative'].dropna()

We Keep Coding

Python is a programming language that lets you work quickly and integrate systems more effectively.

Is there a method in turning user input into csv format? - python

could you elaborate more? maybe you could split the raw text by comma and then convert it to a dataframe like: list_of_string = input.split(',') df = pd.DataFrame(list_of_string) df.to_csv('yourfile.csv')

You can use user input controlled while loop to get user input. Finally, you may exit depending on the user’s choice. Look at the code below: user_input = 'Y' while user_input.lower() == 'y': # Run your code here. user_input = input('Do you want to add one more entry: Y or N?')

Related

Pandas cant print the list of objects collected from web using xpath in Jupyter notebook

Filter all rows from groupby object

Logical indexing in pandas dataframes [duplicate]

Assigning states of Hidden Markov Models by idealized values intensity values.

where condition is met, get last row pandas

Categories

Resources