I have the following dataset that was randomly generated through a simulation I am building:
https://drive.google.com/drive/folders/1JF5QrliE9s8VPMaGc8Z-mwpFhNWkeYtk?usp=sharing
For debugging purposes, I would like to be able to view this data in a series of small multiples. Like this:
I am attempting to do this using matplotlib and pandas. Here is my code for that:
import pandas as pd
import matplotlib.pyplot as plt
from matplotlib.backends.backend_pdf import PdfPages
def graph_file(f: str):
"""
Graphs a single file of data
and exports it as a pdf of separate charts.
"""
data = pd.read_csv(f)
header = data.columns
fname = f[:-4] + '.pdf'
with PdfPages(fname) as pdf:
n = len(header)
time: str = header[0]
# Multiple charts on one page
fig = plt.figure()
for i in range(1, n):
y: str = header[i]
ax = fig.add_subplot()
data.plot(x=time, y=y)
pdf.savefig(bbox_inches='tight')
When I open up the .csv file and try to run the function using a Jupyter notebook, I get the same deprecation warning over and over again:
<ipython-input-5-0563709f3c08>:24: MatplotlibDeprecationWarning: Adding an axes using the same arguments as a previous axes currently reuses the earlier instance. In a future version, a new instance will always be created and returned. Meanwhile, this warning can be suppressed, and the future behavior ensured, by passing a unique label to each axes instance.
ax = fig.add_subplot()
The resulting pdf file does not contain a single page with multiple graphs (which is what I want like in the first image) but just a single page with a single graph:
What exactly am I doing wrong? I greatly appreciate any feedback you can give.
Here is a solution that should meet your needs. It reads the csv file into a dataframe and iterates through the columns of the dataframe to plot corresponding subplots.
import pandas as pd
import matplotlib.pyplot as plt
from matplotlib.backends.backend_pdf import PdfPages
def graph_file(f: str):
df = pd.read_csv(f)
fig, axs = plt.subplots(nrows=3, ncols=3)
fig.set_size_inches(20, 10)
fig.subplots_adjust(wspace=0.5)
fig.subplots_adjust(hspace=0.5)
fname = f[:-4] + '.pdf'
with PdfPages(fname) as pdf:
for col, ax in zip(df.columns[1:], axs.flatten()):
ax.plot(df['time (days)'], df[col])
ax.set(xlabel='time (days)', ylabel=col)
ax.tick_params(axis='x', labelrotation=30)
pdf.savefig(bbox_inches='tight')
plt.show()
The following code is a sample showing how the problem arises.
import pandas as pd
import matplotlib.pyplot as plt
#Reading data
data = pd.read_csv("mydata.csv",parse_dates=['date'])
data = data.iloc[0:17, :]
#Plotting data
fig = plt.figure(figsize=(10, 7))
ax = fig.add_subplot(111)
ax.plot(data['date'],data['y'],'-o')
ax.set(xlabel='Date', ylabel='y')
ax.grid()
plt.show()
The result is the following: the grid is displaced with respect to data point grid_displaced.
If I remove ,parse_dates=['date'], everything works fine grid_not_displaced.
Here is the link to the data file https://drive.google.com/file/d/1AWcyIKgtDY_xkT_gaUxsiwjq9vLGfMog/view?usp=sharing
I've been trying several style sheets but none of them seems to be applied to the canvas. This is the first time I'm using twinx() so maybe that's the issue. The code I have tried is below -
import pickle
import numpy as np
import matplotlib.pyplot as plt
with open("acc.pkl", "rb") as a:
acc = pickle.load(a)
with open("loss.pkl", "rb") as b:
loss = pickle.load(b)
x = np.array([point for point in range(100)])
fig, graph_1 = plt.subplots()
points_1 = np.array(acc)
graph_1.plot(x, points_1, 'b')
graph_2 = graph_1.twinx()
points_2 = np.array(loss)
graph_2.plot(x, points_2, 'r')
plt.style.use('fivethirtyeight')
plt.xlabel('epochs')
fig.tight_layout()
plt.show()
The stylesheet parameters are applied at the time the object that uses them is created.
E.g. if you want to have a figure and axes in a given style, you need to set the style sheet before creating them via plt.subplots.
plt.style.use('fivethirtyeight')
fig, graph_1 = plt.subplots()
I have a python animation script (using matplotlib's funcAnimation), which runs in Spyder but not in Jupyter. I have tried following various suggestions such as adding "%matplotlib inline" and changing the matplotlib backend to "Qt4agg", all without success. I have also tried running several example animations (from Jupyter tutorials), none of which have worked. Sometimes I get an error message and sometimes the plot appears, but does not animate. Incidentally, I have gotten pyplot.plot() to work using "%matplotlib inline".
Does anyone know of a working Jupyter notebook with a simple inline animation example that uses funcAnimation.
[Note: I am on Windows 7]
notebook backend
'Inline' means that the plots are shown as png graphics. Those png images cannot be animated. While in principle one could build an animation by successively replacing the png images, this is probably undesired.
A solution is to use the notebook backend, which is fully compatible with FuncAnimation as it renders the matplotlib figure itself:
%matplotlib notebook
jsanimation
From matplotlib 2.1 on, we can create an animation using JavaScript. This is similar to the ani.to_html5() solution, except that it does not require any video codecs.
from IPython.display import HTML
HTML(ani.to_jshtml())
Some complete example:
import matplotlib.pyplot as plt
import matplotlib.animation
import numpy as np
t = np.linspace(0,2*np.pi)
x = np.sin(t)
fig, ax = plt.subplots()
ax.axis([0,2*np.pi,-1,1])
l, = ax.plot([],[])
def animate(i):
l.set_data(t[:i], x[:i])
ani = matplotlib.animation.FuncAnimation(fig, animate, frames=len(t))
from IPython.display import HTML
HTML(ani.to_jshtml())
Alternatively, make the jsanimation the default for showing animations,
plt.rcParams["animation.html"] = "jshtml"
Then at the end simply state ani to obtain the animation.
Also see this answer for a complete overview.
There is a simple example within this tutorial here: http://louistiao.me/posts/notebooks/embedding-matplotlib-animations-in-jupyter-notebooks/
To summarise the tutorial above, you basically need something like this:
from matplotlib import animation
from IPython.display import HTML
# <insert animation setup code here>
anim = animation.FuncAnimation() # With arguments of course!
HTML(anim.to_html5_video())
However...
I had a lot of trouble getting that to work. Essentially, the problem was that the above uses (by default) ffmpeg and the x264 codec in the background but these were not configured correctly on my machine. The solution was to uninstall them and rebuild them from source with the correct configuration. For more details, see the question I asked about it with a working answer from Andrew Heusser: Animations in ipython (jupyter) notebook - ValueError: I/O operation on closed file
So, try the to_html5_video solution above first, and if it doesn't work then also try the uninstall / rebuild of ffmpeg and x264.
Another option:
import matplotlib.animation
import matplotlib.pyplot as plt
import numpy as np
plt.rcParams["animation.html"] = "jshtml"
plt.rcParams['figure.dpi'] = 150
plt.ioff()
fig, ax = plt.subplots()
x= np.linspace(0,10,100)
def animate(t):
plt.cla()
plt.plot(x-t,x)
plt.xlim(0,10)
matplotlib.animation.FuncAnimation(fig, animate, frames=10)
Here is the answer that I put together from multiple sources including the official examples. I tested with the latest versions of Jupyter and Python.
Download FFmpeg ( http://ffmpeg.zeranoe.com/builds/ )
Install FFmpeg making sure that you update the environmental variable ( http://www.wikihow.com/Install-FFmpeg-on-Windows ).
Run this script in Jupyter below. The variable imageList is the only thing that you need to modify. It is an list of images (your input).
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.animation as animation
from IPython.display import HTML
#=========================================
# Create Fake Images using Numpy
# You don't need this in your code as you have your own imageList.
# This is used as an example.
imageList = []
x = np.linspace(0, 2 * np.pi, 120)
y = np.linspace(0, 2 * np.pi, 100).reshape(-1, 1)
for i in range(60):
x += np.pi / 15.
y += np.pi / 20.
imageList.append(np.sin(x) + np.cos(y))
#=========================================
# Animate Fake Images (in Jupyter)
def getImageFromList(x):
return imageList[x]
fig = plt.figure(figsize=(10, 10))
ims = []
for i in range(len(imageList)):
im = plt.imshow(getImageFromList(i), animated=True)
ims.append([im])
ani = animation.ArtistAnimation(fig, ims, interval=50, blit=True, repeat_delay=1000)
plt.close()
# Show the animation
HTML(ani.to_html5_video())
#=========================================
# Save animation as video (if required)
# ani.save('dynamic_images.mp4')
If you have a list of images and want to animate through them, you can use something like this:
from keras.preprocessing.image import load_img, img_to_array
from matplotlib import animation
from IPython.display import HTML
import glob
%matplotlib inline
def plot_images(img_list):
def init():
img.set_data(img_list[0])
return (img,)
def animate(i):
img.set_data(img_list[i])
return (img,)
fig = figure()
ax = fig.gca()
img = ax.imshow(img_list[0])
anim = animation.FuncAnimation(fig, animate, init_func=init,
frames=len(img_list), interval=20, blit=True)
return anim
imgs = [img_to_array(load_img(i)) for i in glob.glob('*.jpg')]
HTML(plot_images(imgs).to_html5_video())
Thank to Kolibril. I finally can run animation on Jupyter and Google Colab.
I modify some code which will generate animation of drawing random line instead.
import matplotlib.animation
import matplotlib.pyplot as plt
from itertools import count
import random
plt.rcParams["animation.html"] = "jshtml"
plt.rcParams['figure.dpi'] = 150
fig, ax = plt.subplots()
x_value = []
y_value = []
index = count();
def animate(t):
x_value.append(next(index))
y_value.append(random.randint(0,10))
ax.cla()
ax.plot(x_value,y_value)
ax.set_xlim(0,10)
matplotlib.animation.FuncAnimation(fig, animate, frames=10, interval = 500)
enter image description here
Here is my code so far:
with open(logfile,'rb') as f:
while True:
lines = sum(1 for line in f)
print lines
X = np.arange(lines)
data = []
for line in f:
a = line.split(',')
data.append(a[1][:-2])
print data
Y = np.array(data)
plt.ion()
graph = plt.plot(X,Y)[0]
graph.set_y_data(Y)
plt.plot(data)
plt.draw()
plt.pause(0.01)
Right now when I print data or Y, it prints an empty array. Then it complains about X not being the same dimension as Y of course. I wonder if perhaps this is because data is not filled quickly enough before the print command is called? But python is supposed to execute sequentially, right?
In any case, I think the logic itself here is probably at fault. This is my best guess - open the file, and while True, try and read everything in and send it into the plot for plot.draw to use. Then as the file is growing as log files do, the chart data and the chart itself will update. How can I ensure that this works?
Use matplotlibs animation features
You need to make an animation like this example.
Version updating the data
Create an empty plot first and update along the way:
import time
import matplotlib.pyplot as plt
from matplotlib.animation import FuncAnimation
def read(logfile):
with open(logfile) as f:
data = []
while True:
line = f.readline()
time.sleep(0.1)
if line:
data.append(float(line.split(',')[1][:-2]))
yield data
def animate(values):
x = list(range(len(values)))
line.set_data(x, values)
ax.set_xlim(x[0], x[-1])
ax.set_ylim(min(values), max(values))
return line,
fig, ax = plt.subplots()
line, = ax.plot([])
ani = FuncAnimation(fig, animate, frames=read('log.txt'), interval=10)
plt.show()
Version creating a new plot each time
Less code, but works only for a few steps:
import time
import matplotlib.pyplot as plt
from matplotlib.animation import FuncAnimation
def read(logfile):
with open(logfile) as f:
data = []
while True:
line = f.readline()
time.sleep(0.1)
if line:
data.append(float(line.split(',')[1][:-2]))
yield data
def animate(values):
line, = plt.plot(values, color='blue')
return line,
fig = plt.figure(figsize = (5,5))
ani = FuncAnimation(fig, animate, frames=read('log.txt'), interval=10)
plt.show()