I have a series of data which consists of values from several experiments (1-40, in the MWE it is 1-5). The overall amount of entries in my original data is ~4.000.000, which I try to smooth in order to display it:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from scipy.interpolate import spline
from statsmodels.nonparametric.smoothers_lowess import lowess
df = pd.DataFrame()
df["values"] = np.random.randint(100000, 200000, 1000)
df["id"] = [1,2,3,4,5] * 200
plt.figure(1, figsize=(11.69,8.27))
# Both fail for my amount of data:
plt.plot(spline(df["values"], df["id"], range(100)), "r-")
plt.plot(lowess(df["values"], df["id"]), "r-")
Both, scipy.interplate and statsmodels.nonparametric.smoothers_lowess.lowess, throw out of memory exceptions for my data. Is there any efficient way to solve this like in, e.g., GNU R using ggplot2 and geom_smooth()?
I can't quite tell what you're getting at with all the dimensions to your data, but one very simple thing you can try is to just use the 'markevery' kwarg like so:
import numpy as np
import matplotlib.pyplot as plt
x=np.linspace(1,100,1E7)
y=x**2
plt.figure(1, figsize=(11.69,8.27))
plt.plot(x,y,markevery=100)
plt.show()
This will only plot every nth point (n=100 here).
If that doesn't help then you may want to try just a simple numpy interpolation with fewer samples like so:
x_large=np.linspace(1,100,1E7)
y_large=x**2
x_small=np.linspace(1,100,1E3)
y_small=np.interp(x_small,x_large,y_large)
plt.plot(x_small,y_small)
Related
I have the following .csv data:
Simulation Run,[urea] (μM),[NO3-] (μM),[NH4+] (μM),[NO2-] (μM),[O2] (μM),[HCO3-] (μM),[OH-] (μM),[H+] (μM),[H2O] (μM)
/Run_01,1124.3139186264032,49.79709670397852,128.31458304321205,0.0,4.0,140000.0,0.1,0.1,55000000.0
/Run_02,1.0017668367460492e-159,2426.7395169966485,3.1544859186304598e-09,1.975005700484566e-10,4.0,140000.0,0.1,0.1,55000000.0
/Run_03,9.905001536507822e-160,2426.739516996945,2.861369463189477e-09,1.7910618538551373e-10,4.0,140000.0,0.1,0.1,55000000.0
/Run_04,1123.3362048916795,49.7956932352008,130.27141398143655,0.0,4.0,140000.0,0.1,0.1,55000000.0
/Run_05,1101.9594005273052,49.792379912298884,173.02833603309404,0.0,4.0,140000.0,0.1,0.1,55000000.0
I would like to plot it in a series of scatterplot matrices to look at the relationships between the different variables. Much like how it is done here. NOTE: In the linked example the person is asking how to accomplish this in altair. I want to do this in Matplotlib.
Using the above code as reference, here is the code I'm working with:
import pandas as pd
from pandas.plotting import scatter_matrix
import matplotlib.pyplot as plt
from matplotlib.backends.backend_pdf import PdfPages
from math import ceil
def graph_data(f: str):
"""
Represents the data
as a series of scatter-plot matrices.
"""
df = pd.read_csv(f)
NROWS = ceil((len(df.columns) - 1) / 3)
# Although the number of variables could vary,
# I would like no more than 3 charts per row.
NCOLS = 3
fname = f[:-4] + '.pdf'
with PdfPages(fname) as pdf:
scatter_matrix(df, alpha=0.2, figsize=(NROWS, NCOLS), diagonal='kde')
pdf.savefig(bbox_inches='tight')
plt.close()
When I try to run this, here is the error I get:
[LOTS OF TRACEBACK]...numpy.linalg.LinAlgError: singular matrix
Is this happening because the number of variables isn't a perfect square number (thereby not yielding a square matrix)? Is there a way to avoid this?
EDIT:
I forgot to specify my import statements so I have those in now.
I want to plot a tendency line on top of a data plot. This must be simple but I have not been able to figure out how to get to it.
Let us say I have the following:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
df = pd.DataFrame(np.random.randint(0,100,size=(100, 1)), columns=list('A'))
sns.lineplot(data=df)
ax.set(xlabel="Index",
ylabel="Variable",
title="Sample")
plt.show()
The resulting plot is:
What I would like to add is a tendency line. Something like the red line in the following:
I thank you for any feedback.
A moving average is one method (my first thought, and already suggested).
Another method is to use a polynomial fit. Since you had 100 points in your original data, I picked a 10th order fit (square root of data length) in the example below. With some modification of your original code:
idx = [i for i in range(100)]
rnd = np.random.randint(0,100,size=100)
ser = pd.Series(rnd, idx)
fit = np.polyfit(idx, rnd, 10)
pf = np.poly1d(fit)
plt.plot(idx, rnd, 'b', idx, pf(idx), 'r')
This code provides a plot like this:
You can do something like this using Rolling Average:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
data = np.random.randint(0,100,size=(100, 1))
df["rolling_avg"] = df.A.rolling(7).mean().shift(-3)
sns.lineplot(data=df)
plt.show()
You could also do a Regression plot to analyse how data can be interpolated using:
ax = sns.regplot(x=df.index, y="A",
data=df,
scatter_kws={"s": 10},
order=10,
ci=None)
As you can see from the picture below, some of the data is cut off on the end. Does anyone know how to fix that? Also I want the intervals for weeks on the x axis to be (1,2,3...13) for weeks 1-13. Thanks.
Since you did not provide the data, I used an example below:
import seaborn as sns
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
np.random.seed(111)
df = pd.DataFrame({'Week':np.tile(np.arange(1,15),2),
'Score':np.random.uniform(np.repeat([0,1],14),
np.repeat([1,2],14),28),
'Win':np.repeat(['0','1'],14)
})
sns.lmplot returns a FacetGrid so you can set the axis ticks like this:
g = sns.lmplot(data=df,x='Week',y='Score',hue='Win')
g.set(xlim = (0.5,14.5))
g.set(xticks=range(14))
This is a very straightforward question. I have and x axis of years and a y axis of numbers increasing linearly by 100. When plotting this with pandas and matplotlib I am given a graph that does not represent the data whatsoever. I need some help to figure this out because it is such a small amount of code:
The CSV is as follows:
A,B
2012,100
2013,200
2014,300
2015,400
2016,500
2017,600
2018,700
2012,800
2013,900
2014,1000
2015,1100
2016,1200
2017,1300
2018,1400
The Code:
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
data = pd.read_csv("CSV/DSNY.csv")
data.set_index("A", inplace=True)
data.plot()
plt.show()
The graph this yields is:
It is clearly very inconsistent with the data - any suggestions?
The default behaviour of matplotlib/pandas is to draw a line between successive data points, and not to mark each data point with a symbol.
Fix: change data.plot() to data.plot(style='o'), or df.plot(marker='o', linewidth=0).
Result:
All you need is sort A before plotting.
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
data = pd.read_csv("CSV/DSNY.csv").reset_index()
data = data.sort_values('A')
data.set_index("A", inplace=True)
data.plot()
plt.show()
I have pandas series of complex numbers, which I would like to plot. Currently, I am looping through each point and assigning it a color. I would prefer to generate the plot without the need to loop over each point... Using Series.plot() would be preferable. Converting series to numpy is ok though.
Here is an example of what I currently have:
import pandas as pd
import numpy as np
from matplotlib import pyplot
s = pd.Series((1+np.random.randn(500)*0.05)*np.exp(1j*np.linspace(-np.pi, np.pi, 500)))
cmap = pyplot.cm.viridis
for i, val in enumerate(s):
pyplot.plot(np.real(val), np.imag(val), 'o', ms=10, color=cmap(i/(len(s)-1)))
pyplot.show()
You can use pyplot.scatter, which allows coloring of points based on a value.
pyplot.scatter(np.real(s), np.imag(s), s=50, c=np.arange(len(s)), cmap='viridis')
Here, we set c to an increasing sequence to get the same result as in the question.
You can simply plot the real and imaginary part of the series without a loop.
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
s = pd.Series((1+np.random.randn(500)*0.05)*np.exp(1j*np.linspace(-np.pi, np.pi, 500)))
plt.plot(s.values.real,s.values.imag, marker="o", ls="")
plt.show()
However, you need to use a scatter plot if you want to have different colors:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
s = pd.Series((1+np.random.randn(500)*0.05)*np.exp(1j*np.linspace(-np.pi, np.pi, 500)))
plt.scatter(s.values.real,s.values.imag, c = range(len(s)), cmap=plt.cm.viridis)
plt.show()