So I'm doing some time series analysis in Pandas and have a peculiar pattern of outliers which I'd like to remove. The bellow plot is based on a dataframe with the first column as a date and the second column the data
AS you can see those points of similar values interspersed and look like lines are likely instrument quirks and should be removed. Ive tried using both rolling_mean, median and removal based on standard deviation to no avail. For an idea of density, its daily measurements from 1984 to the present. Any ideas?
auge = pd.read_csv('GaugeData.csv', parse_dates=[0], header=None)
gauge.columns = ['Date', 'Gauge']
gauge = gauge.set_index(['Date'])
gauge['1990':'1995'].plot(style='*')
And the result of applying rolling median
gauge = pd.rolling_mean(gauge, 5, center=True)#gauge.diff()
gauge['1990':'1995'].plot(style='*')
After rolling median
You can demand that each data point has at least "N" "nearby" data points within a certain distance "D".
N can be 2 or more.
nearby for element gauge[i] can be a pair like: gauge[i-1] and gauge[i+1], but since some only have neighbors on one side you can ask for at least two elements with distance in indexes (dates) less than 2. So, let's say at least 2 of {gauge[i-2], gauge[i-1] gauge[i+1], gauge[i+2]} should satisfy: Distance(gauge[i], gauge[ix]) < D
D - you can decide this based on how close you expect those real data points to be.
It won't be perfect, but it should get most of the noise out of the dataset.
Related
I have a dataframe with temperature data for a certain period. With this data, I want to calculate the relative frequency of the month of August being warmer than 20° as well as January being colder than 2°. I have already managed to extract these two columns in a separate dataframe, to get the count of each temperature event and used the normalize function to get the frequency for each value in percent (see code).
df_temp1[df_temp1.aug >=20]
df_temp1[df_temp1.jan <= 2]
df_temp1['aug'].value_counts()
df_temp1['jan'].value_counts()
df_temp1['aug'].value_counts(normalize=True)*100
df_temp1['jan'].value_counts(normalize=True)*100
What I haven't managed is to calculate the relative frequency for aug>=20, jan<=2, as well as aug>=20 AND jan<=2 and aug>=20 OR jan<=2.
Maybe someone could help me with this problem. Thanks.
I would try something like this:
proprortion_of_augusts_above_20 = (df_temp1['aug'] >= 20).mean()
proprortion_of_januaries_below_20 = (df_temp1['jan'] <= 2).mean()
This calculates it in two steps. First, df_temp1['aug'] >= 20 creates a boolean array, with True representing months above 20, and False representing months which are not.
Then, mean() reinterprets True and False as 1 and 0. The average of this is the percentage of months which fulfill the criteria, divided by 100.
As an aside, I would recommend posting your data in a question, which allows people answering to check whether their solution works.
I am preparing a dataframe for machine learning. The data set contains weather data from several weather stations in australia over a period of 10 years. One of the measured attributes is Evaporation. It has about 50% missing values.
Now I want to find out, whether the missing values are evenly distributed over all weather stations or if roughly half of the weather stations just never measured Evaporation.
How can I find out about the distribution of a value in combination with another attribute?
I basically want to loop over the weather stations and get a count of NaNs and normal values.
rain_df.query('Location == "Albury"').Location.count()
This gives me the number of measurement points from the weaher station in Albury. Now how can I find out how many NaNs were measured in Albury compared to normal (non-NaN) measurements?
You can use .isnull() to mask a series with True for NaNs and False for everything else. Then you can use .value_counts(normalize=True) to get the proportions of NaN and non NaN in that series.
rain_df.query('Location == "Albury"').Location.isnull().value_counts(normalize=True)
I have two dataframes that are spectral measurements (both have two columns: Intensity and Wavelength) and I need to divide the intensity of one by the intensity of the other in a given Wavelength, as if I were dividing two functions (I1 (λ) / I2 (λ)). The difficulty is that both dataframes have different sizes and the Wavelength values for one are not exactly the same as the other (although obviously they "go close").
One has approximately 200 lines (black line) and the other has 3648 (red line). In short, the red graph is much more "filled" than the black graph, but as I said before, the Wavelength values of the respective dataframes are not exactly the same.
They have different Wavelength ranges as well:
Black starts from 300.2 to 795.5 nm
Red starts at 199.975 and goes up to 1027.43 nm
What I like to do is something like this:
Note that, I divided the Intensity of the black one by the red one, where the result with his corresponding Wavelength is added in a new df. Is it possible to generate a new dataframe with an equivalent Wavelength and make this division between intensities?
Here is working solution of your problem. My current assumption is that the sampling rate of instrument is the same. Since, you didn't provide any sample, I have generated some data. The answer is based on concatenating both dataframes on the Wavelength column.
import pandas as pd
import numpy as np
##generating the test data
black_lambda = np.arange(300.2,795.5,0.1)
red_lambda = np.arange(199.975,1027.43,0.1)
I_black = np.random.random((1,len(black_lambda))).ravel()
I_red = np.random.random((1,len(red_lambda))).ravel()
df = pd.DataFrame([black_lambda,I_black]).T
df1 = pd.DataFrame([red_lambda,I_red]).T
df.columns=['lambda','I_black']
df1.columns=['lambda','I_red']
Follow from here:
#setting lambda as index for both dataframes
df.set_index(['lambda'],inplace=True)
df1.set_index(['lambda'],inplace=True)
#concatenating/merging both dataframes into one
df3 = pd.concat([df,df1],axis=1)
#since both dataframes are not of same length, there will be some missing values. Taking care of them by filling previous values (optional).
df3.fillna(method='bfill',inplace=True)
df3.fillna(method='ffill',inplace=True)
#creating a new column 'division' to finish up the task
df3['division'] = df3['I_black'] / df3['I_red']
print(df3)
Output:
I_black I_red division
lambda
199.975 0.855777 0.683906 1.251308
200.075 0.855777 0.305783 2.798643
200.175 0.855777 0.497258 1.720993
200.275 0.855777 0.945699 0.904915
200.375 0.855777 0.910735 0.939655
... ... ... ...
1026.975 0.570973 0.637064 0.896258
1027.075 0.570973 0.457862 1.247042
1027.175 0.570973 0.429709 1.328743
1027.275 0.570973 0.564804 1.010924
1027.375 0.570973 0.246437 2.316917
Currently, I want to observe the impact of missing values on my dataset. I replace data point (10, 20, 90 %) to missing values and observe the impact. This function below is to replace a certain per cent data point to missing.
def dropout(df, percent):
# create df copy
mat = df.copy()
# number of values to replace
prop = int(mat.size * percent)
# indices to mask
mask = random.sample(range(mat.size), prop)
# replace with NaN
np.put(mat, mask, [np.NaN]*len(mask))
return mat
My question is, I want to replace missing values based on zipf distirbution/power low/long tail. For instance, I have a dataset that contains of 10 columns (5 columns categorical data and 5 columns numerical data). I want to replace some data points on 5 columns categorical based on zipf law, columns in the left sides have more missing rather than in the right side.
I used Python to do this task.
I saw Scipy manual about zipf distirbution in this link: https://docs.scipy.org/doc/numpy-1.15.0/reference/generated/numpy.random.zipf.html but still it's not help me much.
Zipf distributions are a family of distributions on 0 to infinity, whereas you want to delete values from only 5 discrete columns, so you will have to make some arbitrary decisions to do this. Here is one way:
Pick a parameter for your Zipf distribution, say a = 2 as in the example given on the SciPy documentation page.
Looking at the plot given on that same page, you could decide to truncate at 10, i.e. if any sampled value of more than 10 comes up, you're just going to discard it.
Then you could just map the remaining domain of 0 to 10 linearly to your five categorical columns: Any value between 0 and 2 corresponds to the first column, and so on.
So you iteratively sample single values from your Zipf distribution using the SciPy function. For every sampled value, you delete one data point in the column the value corresponds to (see 3.), until you have reached the overall desired percentage of missing values.
Changed question and picture (as I said before... its complicated :)
I have a pandas dataframe 'df' that has a column 'score' (floating point values) with a distribution (lets say a normal distribution). I additionally have an integer 'splits' (lets say 3) and a floating point number 'gap' (lets say 0.5).
I would like to have two dataframes 'gaps_df' and 'rest_df'. 'gaps_df' should consist of all entries from df that are marked orange in the picture (every two red lines have distance 'gap'). 'rest_df' consists of all entries which are marked green.
Here is the tricky part: The green areas have to be of equal size!
To be clear:
the GREEN areas have to be of equal amount of entries!
the ORANGE areas have to consist of entries within the gap-range (amount doesn't matter) between the green areas
So far I have the following:
df.sort('score')
df = df.reset_index(drop=True)
split_markers = []
for marker_index in range(1, classes):
split_markers.append(marker_index * df.size/classes)
But the last two lines are wrong since they split the WHOLE AREA into equal amount of entries. With a normal distribution, I could just move the markers 0.5*gap to the left and to the right. But in fact: I do NOT have a normal distribution (this was just to quickly create a picture with equal green areas).
It gets freaking me out. I really do appreciate every help you can give! Maybe there is a way easier solution...