I'm trying to create a dashboard with two holoviews objects: a panel pn.widgets.Select object that contains a list of xarray variables, and a hvplot object that takes the selected variable on input, like this:
def hvmesh(var=None):
mesh = ds[var].hvplot.quadmesh(x='x', y='y', rasterize=True, crs=crs,
width=600, height=400, groupby=list(ds[var].dims[:-2]), cmap='jet')
return mesh
Here's what an example mesh looks like for a particular variable (one that has both time and height dimensions):
I would like to have the map update when I select a variable from the panel widget:
I tried to do this as a dynamic map, like this:
from holoviews.streams import Params
import holoviews as hv
var_stream = Params(var_select, ['value'], rename={'value': 'var'})
mesh = hv.DynamicMap(hvmesh, streams=[var_stream])
but when I try to display the map, I get:
Exception: Nesting a DynamicMap inside a DynamicMap is not supported.
It would seem a common need to select the variable for hvplot from a panel widget. What is the best way to accomplish this with pyviz?
In case it's useful, here is my full attempt Jupyter Notebook.
Because the groupby changes with each variable selected, a list of variables can not be passed to hvplot. So one solution is to just recreate the plot each time a new variable is selected. This works:
import holoviews as hv
from holoviews.streams import Params
def plot(var=None, tiles=None):
var = var or var_select.value
tiles = tiles or map_select.value
mesh = ds[var].hvplot.quadmesh(x='x', y='y', rasterize=True, crs=crs, title=var,
width=600, height=400, groupby=list(ds[var].dims[:-2]),
cmap='jet')
return mesh.opts(alpha=0.7) * tiles
def on_var_select(event):
var = event.obj.value
col[-1] = plot(var=var)
def on_map_select(event):
tiles = event.obj.value
col[-1] = plot(tiles=tiles)
var_select.param.watch(on_var_select, parameter_names=['value']);
map_select.param.watch(on_map_select, parameter_names=['value']);
col = pn.Column(var_select, map_select, plot(var_select.value) * tiles)
producing:
Here is the full notebook.
So there is a long answer and a short answer here. Let's start with the short answer, which is that there's no need to create a custom select widget for the data variable since hvPlot allows selecting between multiple data variables automatically, so if you change it to this:
rasterized_mesh = ds[time_vars].hvplot.quadmesh(
x='x', y='y', z=time_vars[::-1], crs=crs, width=600, height=400,
groupby=list(ds[var].dims[:-2]), rasterize=True, cmap='jet')
You will get a DynamicMap that lets you select the non-spatial dimensions and the data variable and you can now embed that in your panel, no extra work needed. If that's all you care about stop here as we're about to get into some of the internals to hopefully deliver a better understanding.
Let us assume for a minute hvPlot did not allow selecting between data variables, what would we do then? So the main thing you have to know is that HoloViews allows chaining DynamicMaps but does not allow nesting them. This can be a bit hard to wrap your head around but we'll break the problem down into multiple steps and then see how we can achieve what we want. So what is the chain of events that would give us our plot?
Select a data variable
Apply a groupby over the non-spatial dimensions
Apply rasterization to each QuadMesh
As you know, hvPlot takes care of steps 2. and 3. for us, so how can we inject step 1. before 2. and 3. In future we plan to add support for passing panel widgets directly into hvPlot, which means you'll be able to do it all in a single step. Since panel is still a very new project I'll be pointing out along the way how our APIs will eventually make this process trivial, but for now we'll have to stick with the relatively verbose workaround. In this case we have to rearrange the order of operations:
Apply a groupby over the non-spatial dimensions
Select a data variable
Apply rasterization to each QuadMesh
To start with we therefore select all data variables and skip the rasterization:
meshes = ds[time_vars].hvplot.quadmesh(
x='x', y='y', z=time_vars, crs=crs, width=600, height=400,
groupby=list(ds[var].dims[:-2]))
Now that we have a DynamicMap which contains all the data we might want to display we can apply the next operations. Here we will make use of the hv.util.Dynamic utility which can be used to chain operations on a DynamicMap while injecting stream values. In particular in this step we create a stream from the var_select widget which will be used to reindex the QuadMesh inside our meshes DynamicMap:
def select_var(obj, var):
return obj.clone(vdims=[var])
var_stream = Params(var_select, ['value'], rename={'value': 'var'})
var_mesh = hv.util.Dynamic(meshes, operation=select_var, streams=[var_select])
# Note starting in hv 1.12 you'll be able to replace this with
# var_mesh = meshes.map(select_var, streams=[var_select])
# And once param 2.0 is out we intend to support
# var_mesh = meshes.map(select_var, var=var_select.param.value)
Now we have a DynamicMap which responds to changes in the widget but are not yet rasterizing it so we can apply the rasterize operation manually:
rasterized_mesh = rasterize(var_mesh).opts(cmap='jet', width=600, height=400)
Now we have a DynamicMap which is linked to the Selection widget, applies the groupby and is rasterized, which we can now embed in the panel. Another approach hinted at by #jbednar above would be to do all of it in one step by making the hvPlot call not dynamic and doing the time and height level selection manually. I won't go through that here but it also a valid (if less efficient) approach.
As I hinted at above, eventually we also intend to have all hvPlot parameters become dynamic, which means you'll be able to do something like this to link a widget value to a hvPlot keyword argument:
ds[time_vars].hvplot.quadmesh(
x='x', y='y', z=var_select.param.value, rasterize=True, crs=crs,
width=600, height=400, groupby=list(ds[var].dims[:-2]), cmap='jet')
Related
I have a question regarding the ColumnDataSource in a Bokeh 2.3.0 Server application.
Below is an example that tries to illustrate my question. Eventough it is a littlebit longer, I've spend a lot of effort making it as minimal but complete as possible.
So, there are at least two major ways of editing the data in ColumnDataSource that I know will work.
First one is by using the 'index_way' (I don't know how to call this method correctly) by using source.data['my_column_name'][<numpy_like_array_indexing>] = 'my_new_value' where <numpy_like_array_indexing> can result in something like [0:10] or [[True,False,True]], ect. to subset the data like a numpy array. This way, one can use the source.selected.indices to index the data for example.
The second method is using the .patch() function of ColumnDataSource. Which the reference calls describes as Efficiently update data source columns at specific locations.
The third method I came accross in my code is when editing/changing a complete column in ColumnDataSource like source.data['my_data_column_1'] = source.data['my_data_column_2']. This way, I can set a data column to an already existing one.
My question is: Are they designed to behave differently? I found that changes using the 'index' method are not propagated or updated to the HoverTool, wheares for the other two methods, this seem to work.
This behavior can be seen in the following code example. When changing the first few samples in the plot, by selecting them with the selection tool and editing source.data['Label'] via label_selected_via_index() the HoverTool does not show the correct and updated value of 'Label'. However, the change in the data was acutally performed, which can be seen by check_label() which accesses and prints the first few samples of source.data['Label'].
Changing the Label Value with one of the other methods does indeed show the correct and updated value when hovering over the data.
import pandas as pd
from bokeh.plotting import figure, curdoc
from bokeh.models import ColumnDataSource, LinearColorMapper, Dropdown, Button, HoverTool
from bokeh.layouts import layout
import random
import time
plot_data = 'Value1'
LEN = 1000
df = pd.DataFrame({"ID":[i for i in range(LEN)],
"Value1":[random.random() for i in range(LEN)],
"Value2":[random.random() for i in range(LEN)],
"Color": [int(random.random()*10) for i in range(LEN)] })
df['plot_data'] = df[plot_data]
df['Label'] = "No Label Set"
df['Label_new_col'] = "Label was added"
source = ColumnDataSource(df)
cmap = LinearColorMapper(palette="Turbo256", low = 0, high = 3)
def make_tooltips():
return [('ID', '#ID'),
('Label', '#Label'),
(plot_data, f'#{plot_data}')]
tooltips = make_tooltips()
hover_tool = HoverTool(tooltips=tooltips)
plot1 = figure(plot_width=800, plot_height=250, tooltips=tooltips, tools='box_select')
plot1.add_tools(hover_tool)
circle = plot1.circle(x='ID', y='plot_data', source=source,
fill_color={"field":'Color', "transform":cmap},
line_color={"field":'Color', "transform":cmap})
def update_plot_data(event):
global plot_data
plot_data = event.item
source.data['plot_data'] = source.data[plot_data]
hover_tool.tooltips = make_tooltips()
dropdown = Dropdown(label='Change Value', menu=["Value1","Value2"])
dropdown.on_click(update_plot_data)
def label_selected_via_index(event):
t0 = time.time()
selected = source.selected.indices
source.data['Label'][0:10] = 'Label was added'
hover_tool.tooltips = make_tooltips()
source.selected.indices = []
print(f"Time needed for label_selected_via_index: {time.time()-t0:.5f}")
button_set_label1 = Button(label='Set Label via Index')
button_set_label1.on_click(label_selected_via_index)
def label_selected_via_patch(event):
t0 = time.time()
selected = source.selected.indices
patches = [(ind, 'Label was added') for ind in selected]
source.patch({'Label': patches})
hover_tool.tooltips = make_tooltips()
source.selected.indices = []
print(f"Time needed for label_selected_via_patch: {time.time()-t0:.5f}")
button_set_label2 = Button(label='Set Label via Patch')
button_set_label2.on_click(label_selected_via_patch)
def label_selected_via_new_col(event):
t0 = time.time()
selected = source.selected.indices
source.data['Label'] = source.data['Label_new_col']
hover_tool.tooltips = make_tooltips()
source.selected.indices = []
print(f"Time needed for label_selected_via_new_col: {time.time()-t0:.5f}")
button_set_label3 = Button(label='Set Label via New Column ')
button_set_label3.on_click(label_selected_via_new_col)
def check_label(event):
print(f"first 10 labels: {[l for l in source.data['Label'][0:10]]}")
button_label_check = Button(label='Check Label')
button_label_check.on_click(check_label)
layout_ = layout([[plot1],
[dropdown],
[button_set_label1 ,button_set_label2, button_set_label3],
[button_label_check]])
curdoc().add_root(layout_)
In my application, I have a lot of data and observed, that using .patch() does take significantly longer than the indexing version or the replacement of a complete column. In my application, the indexing method needs less than a millisecond, while the patch method needs more than one seconds, which makes everything a little bit more laggy when interactively changing values. Basically, my application is somehow similar to the above example regarding the process of selecting samples in one plot and assigning a label via multiple buttons. Those labels are also shown in muliple plots via the tooltip, so this update is necessary for me.
Is there a way to A) Make the indexing version also updating the Hovertool? I prefer this method, because it is visually much faster or B) Make the .patch() version somehow faster?
I hope I could make my problem somehow understandable and be thankful for any suggestions.
In the context of a Bokeh server app, it's worth keeping in mind, "what all actually needs to happen for a change to show up in the browser?" And the answer to that is roughly:
a change is detected (or signaled) in Python
a change event is serialized and sent over the network to a browser
the change event is deserialized by BokehJS
the change is applied and view in the browser is updates
Pretty much Bokeh always handles the last three steps (modulo any actual bugs or TBD features). So the question really boils down to "what are the ways to signal a change" to Bokeh?
Let's start from a position of describing what is available and intended (rather than starting from differences or what is not intended).
Direct Assignment to Properties
The number one, primary way to update a Bokeh object in order see a change in the browser is to assign an entire new value to a Bokeh property. If you do that, e.g. .prop_name = new_value, literally including a "dot" and "equal sign", then Bokeh can auto-magically detect the change and send it to the browser. Here are a few examples:
plot.title.text = "New title" # updates the title
glyph.line_color = "red" # change a glyph's line color
slider.value = 10 # sets a slider's value
The examples above all show basic scalar (string, number) values, but this works just as well for more complicated values. Another extremely common example of this general mechanism is updating the entire .data dict of a ColumnDataSource
source.data = {'x': [...], 'y': [...]} # new data for a glyph or table
That updates all the data in a CDS, so that e.g. a line glyph might re-draw itself.
Depending what you are doing, the size of your data, etc., updating the entire .data dict may be expensive (due to serialization, de-serialization, network transit, etc). So there are some other ways that may be more efficient in specific cases.
"In-place" Special Cases
The distinguishing characteristic above is that everything is a "whole" assignment, i.e. there are not mutating or in-place modifications. In a few cases, Bokeh can auto-magically handle in-place updates to mutable values. Without getting too into the weeds, by far the most important example of this is setting a single new column in a ColumnDataSource by using standard Python dict indexing assignment on .data:
source.data['x'] = [...] # Bokeh will automatically handle this
This is your Third method above. It works fine, but only for updating columns in a CDS .data dict. This method only sends the one new column of data over the wire. As long as you only need to update one or a few columns in a large CDS, it is probably faster than assigning a new whole .data value.
What does NOT work is basically any other kind of mutating, in-place assignment:
source.data['x'][100:200] = [...] # Bokeh does not automatically handle this
This is your First ("index") method above, and it is a non-starter. This kind of usage will not trigger any changes in the browsers.
The TLDR is that wrapping every CDS sequence in some custom class that overrides the standard getitem/setitem machinery just makes common usage too inefficient, and the trade-off cannot be justified. Bokeh will not auto-magically notice or do anything with with mutating assignments like this. (If you are purely in BokehJS JavaScript-side, then you can make in-place assignments like this and then manually call a change.emit() to manually trigger updates, but that is only for the pure JS side of things).
Dedicated APIs for Optimized Cases
Recognizing that sometimes even restricting CDS updates to a single column is still not efficient enough, the patch and stream methods were added to ColumnDataSource.
These methods are for cases like:
I want to append few new values to the end of all my columns, instead of sending all the data again. (e.g for streaming new stock ticker or other sensor data efficiently)
I want to update a few specific values in the middle of my large time series or image, but only send the updated values and not re-send everything else.
This is your Second method and is typically much faster for small updates relative to total data size. As for patch specifically, you can see e.g. the patch_app.py example in the examples folder:
This example updates a three separate scatter, multi-line, and image plots all simultaneously at a 20Hz update rate. The checked-in version has fairly modest data sizes, but I tested it again locally by bumping all the data to 10–100x larger, and it still kept up. If you are seeing something different from patch (i.e. multi-second update times), then a complete Minimal Reproducing Example is needed to actually investigate.
When using animation.FuncAnimation, is it possible to optimally reverse an animation after it plays in forward? i.e. equivalent of
FuncAnimation(.., frames=np.hstack([range(10), range(9)[::-1]]))
This redraws frames in reverse order, eating up drawing computation and memory when 50% of data is exactly the same. FuncAnimation does have a caching mechanism, but does it or any other animation. have a reverse=True? Note all my data is pre-computed and retrieved via simple index for each frame.
Are you looking for a solution within the (interactive?) matplotlib backend or are you writing this animation to file?
The latter is quite a lot simpler, as you could simply use a tool like ffmpeg to revert&append the final gif/movie file. See this answer, for example.
If you need a solution within matplotlib then I guess you'd need to store all your required calculations in separate objects (e.g. create a dictionary like frames = { 0: {"x":0, "y":1, … } } and then tell the FuncAnimation to update with a simple "lookup" function like:
def update(frame):
xdata = frames[frame]["x"]
ydata = frames[frame]["y"]
ln.set_data(xdata, ydata)
return ln,
A third option would be to store your axes as pickle'd object, i.e.:
pickle.dump(ax, file('frame_0.pickle', 'w'))
and then try to re-load those pickle'd frames within FuncAnimation's update call like:
ax = pickle.load(file('frame_0.pickle'))
I am generating STL files for 3D printing and then using mlab/mayavi to display them. I would like the Z axis to remain vertical as I rotate the image. According to the mayavi documentation, this can be achieved using the following incantation:
fig = mlab.gcf()
from tvtk.api import tvtk
fig.scene.interactor.interactor_style = tvtk.InteractorStyleTerrain()
Unfortunately, as you can see from this screenshot of my app, it is not the Z axis but the Y axis that gets maintained vertical.
This is an issue because 3D printers always think of Z as the vertical axis, so I really need Z and not Y to be oriented vertically. Is there a way in which this can be achieved?
I have experienced the same problem, vertically aligned along the y-axis.
To get the scene to align vertically along the z-axis (which you want), you can simply add a scene.mlab.view() call before setting the interactor.
The scene.mlab.view() call aligns the camera correctly (with z up), before the interactor is set. I have found this solution by simply testing a bunch of stuff, I could not find this "hack" in the documentation.
New code:
fig = mlab.gcf()
from tvtk.api import tvtk
fig.scene.mlab.view(0, 90)
fig.scene.interactor.interactor_style = tvtk.InteractorStyleTerrain()
You can rename them using the Mayavi button to access the pipeline, then selecting the appropiate axes object and changing the labels.
With code, you can add the xlabel, ylabel and zlabel keyword arguments to specify them. This is specified in the axes API
I am trying to create a grid layout (say 2x2) of plots of the DynamicMap type using holoviews.
This is to be served as a Holoviews/Bokeh app.
After creating my dmaps I Lay them out using
layout = hv.Layout([dmap1, dmap2]).cols(2)
this layout generates a two plots side-by-side
The problem I am facing is that by default, the widgets for each Dynamic Map get clustered to the rightof the row , with no visual association to the plots (the maps toolbars also get combined into just one).
Moreover, If I pass more than two dmaps to the layout, I get an error:
TypeError: DynamicMap does not accept AdjointLayout type, data elements have to be a ('ViewableElement', 'NdMapping', 'Layout').
I am using the renderer in server mode:
renderer = hv.renderer('bokeh')
renderer = renderer.instance(mode='server')
In summary, I want to have a grid of more than 2 independently controlable dynamical plots.
There's a lot to unpack here, so let me start out answering this question:
the widgets for each Dynamic Map get clustered to the right of the row
If you want independent sets of widgets you will have to construct each set of widgets manually and compose the resulting bokeh models yourself. This example demonstrates this approach:
import numpy as np
import holoviews as hv
from bokeh.io import curdoc
from bokeh.layouts import row
renderer = hv.renderer('bokeh').instance(mode='server')
dmap1 = hv.DynamicMap(lambda x: hv.Curve(np.random.rand(10)), kdims='x').redim.range(x=(0,5))
dmap2 = hv.DynamicMap(lambda y: hv.Scatter(np.random.rand(10)), kdims='y').redim.range(x=(0,5))
widget1 = renderer.get_widget(dmap1, None, position='above').state
widget2 = renderer.get_widget(dmap2, None, position='above').state
r = row(widget1, widget2)
doc = curdoc()
doc.add_root(r)
We create two independent DynamicMaps, then use the renderer to generate separate plots and widgets and then compose them using the bokeh row layout. As you can see we can also define a position for the widgets so instead of laying them out on the right they are on top.
the maps toolbars also get combined into just one
In this recent PR a new merge_tools option was added to allow having separate toolbars in a single Layout.
Moreover, If I pass more than two dmaps to the layout, I get an error:
This is probably due to returning an adjoined object, which is not allowed inside a DynamicMap at the moment. Are you by any chance using the .hist method? If so try calling it on the DynamicMap instead of having the DynamicMap return an object with an adjoined Histogram.
Say I have a class that holds some data and implements a function that returns a bokeh plot
import bokeh.plotting as bk
class Data():
def plot(self,**kwargs):
# do something to retrieve data
return bk.line(**kwargs)
Now I can instantiate multiple of these Data objects like exps and sets and create individual plots. If bk.hold() is set they'll, end up in one figure (which is basically what I want).
bk.output_notebook()
bk.figure()
bk.hold()
exps.scatter(arg1)
sets.plot(arg2)
bk.show()
Now I want aggregate these plots into a GridPlot() I can do it for the non overlayed single plots
bk.figure()
bk.hold(False)
g=bk.GridPlot(children=[[sets.plot(arg3),sets.plot(arg4)]])
bk.show(g)
but I don't know how I can overlay the scatter plots I had earlier as exps.scatter.
Is there any way to get a reference to the currently active figure like:
rows=[]
exps.scatter(arg1)
sets.plot(arg2)
af = bk.get_reference_to_figure()
rows.append(af) # append the active figure to rows list
bg.figure() # reset figure
gp = bk.GridPlot(children=[rows])
bk.show(gp)
As of Bokeh 0.7 the plotting.py interface has been changed to be more explicit and hopefully this will make things like this simpler and more clear. The basic change is that figure now returns an object, so you can just directly act on those objects without having to wonder what the "currently active" plot is:
p1 = figure(...)
p1.line(...)
p1.circle(...)
p2 = figure(...)
p2.rect(...)
gp = gridplot([p1, p2])
show(gp)
Almost all the previous code should work for now, but hold, curplot etc. are deprecated (and issue deprecation warnings if you run python with deprecation warnings enabled) and will be removed in a future release.
Ok apparently bk.curplot() does the trick
exps.scatter(arg1)
sets.plot(arg2)
p1 = bk.curplot()
bg.figure() # reset figure
exps.scatter(arg3)
sets.plot(arg4)
p2 = bk.curplot()
gp = bk.GridPlot(children=[[p1,p2])
bk.show(gp)