We Keep Coding

How to cache bokeh plots using redis

I'm using bokeh server to render a timeseries graph over a map. As the timeseries progresses, the focus of the map moves.
The code below works, but each progression creates a call that goes off to the google api (GMAP) to get the backdrop. This then takes time to render. At points where the timeseries has shifted the focus a few times in quick succession, the backdrop hasn't had time to render before it is updated.
I've been trying to work out if/how these requests can be made in advance, cached (using redis), enabling the user is able to view the cache with all data already loaded for each tick on the timeseries.
main.py
import settings
from bokeh.plotting import figure, gmap
from bokeh.embed import components
from bokeh.models import CustomJS, ColumnDataSource, Slider, GMapOptions, GMapPlot, Range1d, Button
from bokeh.models.widgets import DataTable, TableColumn, HTMLTemplateFormatter
from bokeh.layouts import column, row, gridplot, layout
from bokeh.io import show, export_png, curdoc
from filehandler import get_graph_data
"""
Get arguments from request
"""
try:
args = curdoc().session_context.request.arguments
pk = int(args.get('pk')[0])
except:
pass
"""
get data for graph from file and initialise variables
"""
#load data into dictionary from file referenced by pk
data_dict = get_graph_data(pk)
no_of_markers = data_dict.get('markers')
length_of_series = data_dict.get('length')
series_data = data_dict.get('data') #lat/lon position of each series at each point in time
series_names = series_data.get('series_names') #names of series
range_x_axis = data_dict.get('xaxis') #min/max lat co-ords
range_y_axis = data_dict.get('yaxis') #min/max lon co-ords
"""
Build data
"""
graph_source = ColumnDataSource(series_data)
"""
Build markers to show current location
"""
markers = ColumnDataSource(data=dict(lon=[], lat=[]))
"""
Build mapping layer
"""
def create_map_backdrop(centroid, zoom, tools):
"""
Create the map backdrop, centered on the starting point
Using GoogleMaps api
"""
map_options = GMapOptions(lng=centroid[1],
lat=centroid[0],
map_type='roadmap',
zoom=zoom,
)
return gmap(google_api_key=settings.MAP_KEY,
map_options=map_options,
tools=tools,
)
#set map focus
centroid = (graph_source.data['lats'][0][0],
graph_source.data['lons'][0][0],
)
"""
Build Plot
"""
tools="pan, wheel_zoom, reset"
p = create_map_backdrop(centroid, 18, tools)
p.multi_line(xs='lons',
ys='lats',
source=graph_source,
line_color='color',
)
p.toolbar.logo = None
p.circle(x='lon', y='lat', source=markers)
"""
User Interactions
"""
def animate_update():
tick = slider.value + 1
slider.value = tick
def slider_update(attr, old, new):
"""
Updates all of the datasources, depending on current value of slider
"""
start = timer()
if slider.value>series_length:
animate()
else:
tick = slider.value
i=0
lons, lats = [], []
marker_lons, marker_lats = [], []
while i < no_of_markers:
#update lines
lons.append(series_data['lons'][i][0:tick])
lats.append(series_data['lats'][i][0:tick])
#update markers
marker_lons.append(series_data['lons'][i][tick])
marker_lats.append(series_data['lats'][i][tick])
#update iterators
i += 1
#update marker display
markers.data['lon'] = marker_lons
markers.data['lat'] = marker_lats
#update line display
graph_source.data['lons'] = lons
graph_source.data['lats'] = lats
#set map_focus
map_focus_lon = series_data['lons'][tick]
map_focus_lat = series_data['lats'][tick]
#update map focus
p.map_options.lng = map_focus_lon
p.map_options.lat = map_focus_lat
slider = Slider(start=0, end=series_length, value=0, step=5)
slider.on_change('value', slider_update)
callback_id = None
def animate():
global callback_id
if button.label == "► Play":
button.label = "❚❚ Pause"
callback_id = curdoc().add_periodic_callback(animate_update, 1)
else:
button.label = "► Play"
curdoc().remove_periodic_callback(callback_id)
button = Button(label="► Play", width=60)
button.on_click(animate)
"""
Display plot
"""
grid = layout([[p, data_table],
[slider, button],
])
curdoc().add_root(grid)
I've tried caching the plot data (p), but it looks like this is persisted before the call to the google api is made.
I've explored caching the map tiles direct from the api and then stitching them into the plot as a background image (using bokeh ImageURL), but I can't get ImageUrl to recognise the in-memory image.
The server documentation suggests that redis can be used as a backend so I wondered whether this might speed thing up, but when I try to start it bokeh serve myapp --allow-websocket-origin=127.0.0.1:5006 --backend=redis I get --backend is not a recognised command.
Is there a way to either cache the fully rendered graph (possibly the graph document itself), whilst retaining the ability for users to interact with the plot; or to cache the gmap plot once it has been rendered and then add it to the rest of the plot?

If this was standalone Bokeh content (i.e. not a Bokeh server app) then you serialize the JSON representation of the plot with json_items and re-hydrate it explicitly in the browser with Bokeh.embed_items. That JSON could potentially be stored in Redis, and maybe that would be relevant. But a Bokeh server is not like that. After the initial session creation, there is never any "whole document" to store or cache, just a sequence of incremental, partial updates that happen over a websocket protocol. E.g. the server says "this specific data source changed" and the browser says "OK I should recompute bounds and re-render".
That said, there are some changes I would suggest.
The first is that you should not update CDS columns one by one. You should not do this:
# BAD
markers.data['lon'] = marker_lons
markers.data['lat'] = marker_lats
This will generate two separate update events and two separate re-render requests. Apart from the extra work this causes, it's also the case that the first update is guaranteed to have mismatched old/new coordinates. Instead, you should always update CDS .data dict "atomically", in one go:
source.data = new_data_dict
Addtionally, you might try curdoc().hold to collect updates into fewer events.

How to append coordinates from Bokeh Tap event to a python object?

In the example below, I am trying to get the x and y coordinates that appear in the Div next to the plot when the bokeh plot is Tapped to be appended to the data dictionary coordList in their respective list.
import numpy as np
from bokeh.io import show, output_notebook
from bokeh.plotting import figure
from bokeh.models import CustomJS, Div
from bokeh.layouts import column, row
from bokeh.events import Tap
coordList = dict(x=[], y=[])
output_notebook()
def display_event(div, attributes=[], style = 'float:left;clear:left;font_size=10pt'):
"Build a suitable CustomJS to display the current event in the div model."
return CustomJS(args=dict(div=div), code="""
var attrs = %s; var args = [];
for (var i = 0; i<attrs.length; i++) {
args.push(Number(cb_obj[attrs[i]]).toFixed(2));
}
var line = "<span style=%r>(" + args.join(", ") + ")</span>\\n";
var text = div.text.concat(line);
var lines = text.split("\\n")
if (lines.length > 35)
lines.shift();
div.text = lines.join("\\n");
""" % (attributes, style))
x = np.random.random(size=4000) * 100
y = np.random.random(size=4000) * 100
radii = np.random.random(size=4000) * 1.5
colors = ["#%02x%02x%02x" % (int(r), int(g), 150) for r, g in zip(50+2*x, 30+2*y)]
p = figure(tools="pan,wheel_zoom,zoom_in,zoom_out,reset")
p.scatter(x, y, radius=np.random.random(size=4000) * 1.5,
fill_color=colors, fill_alpha=0.6, line_color=None)
div = Div(width=400, height=p.plot_height)
layout = row(p, div)
point_attributes = ['x', 'y']
p.js_on_event(Tap, display_event(div, attributes=point_attributes))
show(layout)
I'm not sure how the coordinates are saved and how to access them and append them to the lists.

There is no way to append to coordinates to a python object with code like above, because that code is generating standalone output (i.e. it is using "show"). Standalone output is pure static HTML and Bokeh JSON that is sent to browser, without any sort of connection to any Python process. If you want to connect Bokeh visualizations to a real running Python process, that is what the Bokeh server is for.
If you run a Bokeh server application, then you can use on_event with a real python callback to run whatever python code you want with the Tap even values:
def callback(event):
# use event['x'], event['y'], event['sx'], event['sy']
p.on_event(Tap, callback)

Bokeh custom tool plotting a vertical line profile of an image

I need to plot a profile of an image, which is, to plot values of a matrix column.
And to implement it as a drag tool, which would automatically update the lower plot based on cursor position over the upper plot:
Based on "A New Custom Tool" example from the docs I've written a code which works fine but has several problems:
import numpy as np
import bokeh.plotting as bp
from bokeh.models import CustomJS
from bokeh.layouts import layout, column, row
from bokeh.io import reset_output
from PIL import Image
im = Image.open(r'C:\Documents\image1.jpg')
z = np.array(im)[:,:,0]
from bokeh.core.properties import Instance, Float
from bokeh.io import output_file, show, output_notebook
from bokeh.models import ColumnDataSource, Tool
from bokeh.plotting import figure
from bokeh.util.compiler import TypeScript
from bokeh.layouts import layout, column, row
#output_file("a.html")
#reset_output()
# image vertical profile tool
TS_CODE = """
import {GestureTool, GestureToolView} from "models/tools/gestures/gesture_tool"
import {ColumnDataSource} from "models/sources/column_data_source"
import {GestureEvent} from "core/ui_events"
import * as p from "core/properties"
export class DrawToolView extends GestureToolView {
model: DrawTool
//this is executed when the pan/drag event starts
_pan_start(_ev: GestureEvent): void {
this.model.source.data = {x: [], y: []}
}
//this is executed on subsequent mouse/touch moves
_pan(ev: GestureEvent): void {
const {frame} = this.plot_view
const {sx, sy} = ev
if (!frame.bbox.contains(sx, sy))
return
const x = frame.xscales.default.invert(sx)
const y = frame.yscales.default.invert(sy)
var res = Array(128);
var rx = Math.round(x);
for(var i=0; i<128; i++) res[i] = this.model.zz.data["z"][i*225+rx];
this.model.source.data = {
x: Array(128).fill(0).map(Number.call, Number),
y: res
};
this.model.source.change.emit()
}
// this is executed then the pan/drag ends
_pan_end(_ev: GestureEvent): void {}
}
export namespace DrawTool {
export type Attrs = p.AttrsOf<Props>
export type Props = GestureTool.Props & {
source: p.Property<ColumnDataSource>,
zz: p.Property<ColumnDataSource>,
width: p.Float
}
}
export interface DrawTool extends DrawTool.Attrs {}
export class DrawTool extends GestureTool {
properties: DrawTool.Props
constructor(attrs?: Partial<DrawTool.Attrs>) {
super(attrs)
}
tool_name = "Drag Span"
icon = "bk-tool-icon-lasso-select"
event_type = "pan" as "pan"
default_order = 12
static initClass(): void {
this.prototype.type = "DrawTool"
this.prototype.default_view = DrawToolView
this.define<DrawTool.Props>({
source: [ p.Instance ],
zz: [ p.Instance ],
width: [ p.Float ]
})
}
}
DrawTool.initClass()
"""
class DrawTool(Tool):
__implementation__ = TypeScript(TS_CODE)
source = Instance(ColumnDataSource)
zz = Instance(ColumnDataSource)
width = Float()
output_notebook()
source = ColumnDataSource(data=dict(x=[], y=[]))
zz = ColumnDataSource(data=dict(z=z.flatten()))
p1 = figure(plot_width=600, plot_height=200, x_range=(0, 225), y_range=(0, 128),
tools=[DrawTool(source=source, zz=zz, width=225)])
im = p1.image(image=[np.flipud(z)], x=0, y=0, dw=225,
dh=128, palette='Greys256')
p2 = figure(plot_width=600, plot_height=200)
p2.line('x', 'y', source=source)
bp.show(column(p1, p2))
1) Image dimensions are hard-coded now: how do I feed image dimensions from python to js?
2) The image is transferred to the client twice: first as an argument to image(), then as source for the button plot. How to access the image "source" from the DrawTool?
3) If (all this code being in one jupyter cell) I run it the second time it refuses to plot anything with a javascript error in console Model 'DrawTool' does not exist. Running it the third time, fourth time and further on works fine. What exactly is bokeh trying to tell me in this error message?

1) Image dimensions are hard-coded now: how do I feed image dimensions
from python to js?
2) The image is transferred to the client twice: first as an argument
to image(), then as source for the button plot. How to access the
image "source" from the DrawTool?
The answer to these is the same, add more properties (on both the Python and the JS sides) for the data you want to store on the DrawTool. E.g. another Instance for another ColumnDataSource that holds the image data, and integer properties for the width and height.
3) If (all this code being in one jupyter cell) I run it the second time it refuses to plot anything with a javascript error in console Model 'DrawTool' does not exist. Running it the third time, fourth time and further on works fine. What exactly is bokeh trying to tell me in this error message?
This message is stating that BokehJS does not know anything about any DrawTool, and the reason for this is that, due to the way things work in the notebook, custom extensions only get registered when you call output_notebook. So you will have to call output_notebook again after you define the custom extension. I don't like this state of affairs but there is nothing we can do about it.

hide annotations in bokeh

I have a Bokeh plot where I add some data in the form of a LabelSet and BoxAnnotation as overlay, but I want to be able to dynamically enable/disable this overlay.
I can enable/hide some of the lines in the plot already, but the system for the Annotations seems to be different. I've got so far already
Initialising
from ipywidgets import interact
from bokeh.plotting import figure as bf
from bokeh.layouts import layout as bl
from bokeh.models import Toggle, BoxAnnotation, CustomJS
from bokeh.io import push_notebook, show, output_notebook
output_notebook()
Widget generation
p = bf(title='test', x_range=(0,1), y_range=(0,1))
x = [1/3, 2/3]
y=[1/3, 2/3]
p.circle(x=x, y=y, size=15)
box = BoxAnnotation(left=None, right=0.5, fill_color='red', fill_alpha=0.1)
p.add_layout(box)
Interactivity
code = '''\
if toggle.active
box.visible = true
console.log 'enabling box'
else
box.visible = false
console.log 'disabling box'
'''
callback = CustomJS.from_coffeescript(code=code, args={})
toggle = Toggle(label="Red Box", button_type="success", callback=callback)
callback.args = {'toggle': toggle, 'box': box}
layout = bl([p], [toggle])
show(layout)
When I check the JS console, the if/else clauses get triggered as expected, so the Toggle works but the red box stays in place, both in Firefox as in IE

I think there might be some plumbing that is not hooked up on the BokehJS side to respond to visible. If so, that's a bug. Please make an issue with all this information in the Project Issue Tracker.
In the mean time, you can accomplish the same visual effect by manipulating the alpha values instead:
code = '''\
if toggle.active
box.fill_alpha = 0.1
box.line_alpha = 1
console.log 'enabling box'
else
box.fill_alpha = 0
box.line_alpha = 0
console.log 'disabling box'
'''
callback = CustomJS.from_coffeescript(code=code, args={})
toggle = Toggle(label="Red Box", button_type="success", callback=callback)
callback.args = {'toggle': toggle, 'box': box}
layout = bl([p], [toggle])
show(layout)

is there a way to save bokeh data table content

I am experimenting with bokeh data table to display data embedded in web page. It works quite nicely.
Is there a way to save the table content from the displayed data table? Other bokeh plots have tool bar for various functions including saving, but the DataTable does not seem to come with it. I know very little about javascript or slickgrid, which bokeh data table uses. And wondering if it can be done.
Thanks!
EDIT - It appears the my original question was not clear enough. Hope following pictures can help to illustrate:
Bokeh plot has toolbars associated:
But data table does not have it by default, and it won't take 'tools' parameter either:
Is it possible to add 'save' button to data table so the person view the table can download as tab delimited or csv files? Not necessarily need to be look the same, but with the same function for saving.

2021 Update: adjusted code that works in python 3.8 and bokeh 2.2.3
For those who have trouble adjusting or finding the example on the bokeh website or are just very lazy, the below code does the minimal job:
from bokeh.models import ColumnDataSource, CustomJS
from bokeh.models.widgets import Button
from bokeh.io import show
import os
source = ColumnDataSource({'list1':[0,1,2,3],'list2':[4,5,6,7]})
button = Button(label="Download", button_type="success")
button.js_on_click(CustomJS(args=dict(source=source),code=open(os.path.join(os.path.dirname(__file__),"download.js")).read()))
show(button)
And the file download.js:
function table_to_csv(source) {
const columns = Object.keys(source.data)
const nrows = source.get_length()
const lines = [columns.join(',')]
for (let i = 0; i < nrows; i++) {
let row = [];
for (let j = 0; j < columns.length; j++) {
const column = columns[j]
row.push(source.data[column][i].toString())
}
lines.push(row.join(','))
}
return lines.join('\n').concat('\n')
}
const filename = 'data_result.csv'
const filetext = table_to_csv(source)
const blob = new Blob([filetext], { type: 'text/csv;charset=utf-8;' })
//addresses IE
if (navigator.msSaveBlob) {
navigator.msSaveBlob(blob, filename)
} else {
const link = document.createElement('a')
link.href = URL.createObjectURL(blob)
link.download = filename
link.target = '_blank'
link.style.visibility = 'hidden'
link.dispatchEvent(new MouseEvent('click'))
}

It would be nice if bokeh provides a tool button for saving/exporting the data table to csv / txt / excel files. If it already does, I have not found it in the document yet.
In the mean time, a possible answer is to export the js array (that is underneath the bokeh data table) to CSV using native javascript. It has been described here and here.
ADD: bokeh has callbacks for using js. A simple description is here. still reading about it ...
EDIT: It is probably there for a while now, but I have just noticed an example on Bokeh website for saving csv from data table.

Related to my response to this stackoverflow question. Response copied below:
Here is a working example with Python 3.7.5 and Bokeh 1.4.0
public github link to this jupyter notebook:
https://github.com/surfaceowl-ai/python_visualizations/blob/master/notebooks/bokeh_save_linked_plot_data.ipynb
environment report:
virtual env python version: Python 3.7.5
virtual env ipython version: 7.9.0
watermark package reports:
bokeh 1.4.0
jupyter 1.0.0
numpy 1.17.4
pandas 0.25.3
rise 5.6.0
watermark 2.0.2
# Generate linked plots + TABLE displaying data + save button to export cvs of selected data
from random import random
from bokeh.io import output_notebook # prevent opening separate tab with graph
from bokeh.io import show
from bokeh.layouts import row
from bokeh.layouts import grid
from bokeh.models import CustomJS, ColumnDataSource
from bokeh.models import Button # for saving data
from bokeh.models.widgets import DataTable, DateFormatter, TableColumn
from bokeh.models import HoverTool
from bokeh.plotting import figure
# create data
x = [random() for x in range(500)]
y = [random() for y in range(500)]
# create first subplot
plot_width = 400
plot_height = 400
s1 = ColumnDataSource(data=dict(x=x, y=y))
fig01 = figure(
plot_width=plot_width,
plot_height=plot_height,
tools=["lasso_select", "reset", "save"],
title="Select Here",
)
fig01.circle("x", "y", source=s1, alpha=0.6)
# create second subplot
s2 = ColumnDataSource(data=dict(x=[], y=[]))
# demo smart error msg: `box_zoom`, vs `BoxZoomTool`
fig02 = figure(
plot_width=400,
plot_height=400,
x_range=(0, 1),
y_range=(0, 1),
tools=["box_zoom", "wheel_zoom", "reset", "save"],
title="Watch Here",
)
fig02.circle("x", "y", source=s2, alpha=0.6, color="firebrick")
# create dynamic table of selected points
columns = [
TableColumn(field="x", title="X axis"),
TableColumn(field="y", title="Y axis"),
]
table = DataTable(
source=s2,
columns=columns,
width=400,
height=600,
sortable=True,
selectable=True,
editable=True,
)
# fancy javascript to link subplots
# js pushes selected points into ColumnDataSource of 2nd plot
# inspiration for this from a few sources:
# credit: https://stackoverflow.com/users/1097752/iolsmit via: https://stackoverflow.com/questions/48982260/bokeh-lasso-select-to-table-update
# credit: https://stackoverflow.com/users/8412027/joris via: https://stackoverflow.com/questions/34164587/get-selected-data-contained-within-box-select-tool-in-bokeh
s1.selected.js_on_change(
"indices",
CustomJS(
args=dict(s1=s1, s2=s2, table=table),
code="""
var inds = cb_obj.indices;
var d1 = s1.data;
var d2 = s2.data;
d2['x'] = []
d2['y'] = []
for (var i = 0; i < inds.length; i++) {
d2['x'].push(d1['x'][inds[i]])
d2['y'].push(d1['y'][inds[i]])
}
s2.change.emit();
table.change.emit();
var inds = source_data.selected.indices;
var data = source_data.data;
var out = "x, y\\n";
for (i = 0; i < inds.length; i++) {
out += data['x'][inds[i]] + "," + data['y'][inds[i]] + "\\n";
}
var file = new Blob([out], {type: 'text/plain'});
""",
),
)
# create save button - saves selected datapoints to text file onbutton
# inspriation for this code:
# credit: https://stackoverflow.com/questions/31824124/is-there-a-way-to-save-bokeh-data-table-content
# note: savebutton line `var out = "x, y\\n";` defines the header of the exported file, helpful to have a header for downstream processing
savebutton = Button(label="Save", button_type="success")
savebutton.callback = CustomJS(
args=dict(source_data=s1),
code="""
var inds = source_data.selected.indices;
var data = source_data.data;
var out = "x, y\\n";
for (i = 0; i < inds.length; i++) {
out += data['x'][inds[i]] + "," + data['y'][inds[i]] + "\\n";
}
var file = new Blob([out], {type: 'text/plain'});
var elem = window.document.createElement('a');
elem.href = window.URL.createObjectURL(file);
elem.download = 'selected-data.txt';
document.body.appendChild(elem);
elem.click();
document.body.removeChild(elem);
""",
)
# add Hover tool
# define what is displayed in the tooltip
tooltips = [
("X:", "#x"),
("Y:", "#y"),
("static text", "static text"),
]
fig02.add_tools(HoverTool(tooltips=tooltips))
# display results
# demo linked plots
# demo zooms and reset
# demo hover tool
# demo table
# demo save selected results to file
layout = grid([fig01, fig02, table, savebutton], ncols=3)
output_notebook()
show(layout)
# things to try:
# select random shape of blue dots with lasso tool in 'Select Here' graph
# only selected points appear as red dots in 'Watch Here' graph -- try zooming, saving that graph separately
# selected points also appear in the table, which is sortable
# click the 'Save' button to export a csv
# TODO: export from Bokeh to pandas dataframe