I'm using bokeh server to render a timeseries graph over a map. As the timeseries progresses, the focus of the map moves.
The code below works, but each progression creates a call that goes off to the google api (GMAP) to get the backdrop. This then takes time to render. At points where the timeseries has shifted the focus a few times in quick succession, the backdrop hasn't had time to render before it is updated.
I've been trying to work out if/how these requests can be made in advance, cached (using redis), enabling the user is able to view the cache with all data already loaded for each tick on the timeseries.
main.py
import settings
from bokeh.plotting import figure, gmap
from bokeh.embed import components
from bokeh.models import CustomJS, ColumnDataSource, Slider, GMapOptions, GMapPlot, Range1d, Button
from bokeh.models.widgets import DataTable, TableColumn, HTMLTemplateFormatter
from bokeh.layouts import column, row, gridplot, layout
from bokeh.io import show, export_png, curdoc
from filehandler import get_graph_data
"""
Get arguments from request
"""
try:
args = curdoc().session_context.request.arguments
pk = int(args.get('pk')[0])
except:
pass
"""
get data for graph from file and initialise variables
"""
#load data into dictionary from file referenced by pk
data_dict = get_graph_data(pk)
no_of_markers = data_dict.get('markers')
length_of_series = data_dict.get('length')
series_data = data_dict.get('data') #lat/lon position of each series at each point in time
series_names = series_data.get('series_names') #names of series
range_x_axis = data_dict.get('xaxis') #min/max lat co-ords
range_y_axis = data_dict.get('yaxis') #min/max lon co-ords
"""
Build data
"""
graph_source = ColumnDataSource(series_data)
"""
Build markers to show current location
"""
markers = ColumnDataSource(data=dict(lon=[], lat=[]))
"""
Build mapping layer
"""
def create_map_backdrop(centroid, zoom, tools):
"""
Create the map backdrop, centered on the starting point
Using GoogleMaps api
"""
map_options = GMapOptions(lng=centroid[1],
lat=centroid[0],
map_type='roadmap',
zoom=zoom,
)
return gmap(google_api_key=settings.MAP_KEY,
map_options=map_options,
tools=tools,
)
#set map focus
centroid = (graph_source.data['lats'][0][0],
graph_source.data['lons'][0][0],
)
"""
Build Plot
"""
tools="pan, wheel_zoom, reset"
p = create_map_backdrop(centroid, 18, tools)
p.multi_line(xs='lons',
ys='lats',
source=graph_source,
line_color='color',
)
p.toolbar.logo = None
p.circle(x='lon', y='lat', source=markers)
"""
User Interactions
"""
def animate_update():
tick = slider.value + 1
slider.value = tick
def slider_update(attr, old, new):
"""
Updates all of the datasources, depending on current value of slider
"""
start = timer()
if slider.value>series_length:
animate()
else:
tick = slider.value
i=0
lons, lats = [], []
marker_lons, marker_lats = [], []
while i < no_of_markers:
#update lines
lons.append(series_data['lons'][i][0:tick])
lats.append(series_data['lats'][i][0:tick])
#update markers
marker_lons.append(series_data['lons'][i][tick])
marker_lats.append(series_data['lats'][i][tick])
#update iterators
i += 1
#update marker display
markers.data['lon'] = marker_lons
markers.data['lat'] = marker_lats
#update line display
graph_source.data['lons'] = lons
graph_source.data['lats'] = lats
#set map_focus
map_focus_lon = series_data['lons'][tick]
map_focus_lat = series_data['lats'][tick]
#update map focus
p.map_options.lng = map_focus_lon
p.map_options.lat = map_focus_lat
slider = Slider(start=0, end=series_length, value=0, step=5)
slider.on_change('value', slider_update)
callback_id = None
def animate():
global callback_id
if button.label == "► Play":
button.label = "❚❚ Pause"
callback_id = curdoc().add_periodic_callback(animate_update, 1)
else:
button.label = "► Play"
curdoc().remove_periodic_callback(callback_id)
button = Button(label="► Play", width=60)
button.on_click(animate)
"""
Display plot
"""
grid = layout([[p, data_table],
[slider, button],
])
curdoc().add_root(grid)
I've tried caching the plot data (p), but it looks like this is persisted before the call to the google api is made.
I've explored caching the map tiles direct from the api and then stitching them into the plot as a background image (using bokeh ImageURL), but I can't get ImageUrl to recognise the in-memory image.
The server documentation suggests that redis can be used as a backend so I wondered whether this might speed thing up, but when I try to start it bokeh serve myapp --allow-websocket-origin=127.0.0.1:5006 --backend=redis I get --backend is not a recognised command.
Is there a way to either cache the fully rendered graph (possibly the graph document itself), whilst retaining the ability for users to interact with the plot; or to cache the gmap plot once it has been rendered and then add it to the rest of the plot?
If this was standalone Bokeh content (i.e. not a Bokeh server app) then you serialize the JSON representation of the plot with json_items and re-hydrate it explicitly in the browser with Bokeh.embed_items. That JSON could potentially be stored in Redis, and maybe that would be relevant. But a Bokeh server is not like that. After the initial session creation, there is never any "whole document" to store or cache, just a sequence of incremental, partial updates that happen over a websocket protocol. E.g. the server says "this specific data source changed" and the browser says "OK I should recompute bounds and re-render".
That said, there are some changes I would suggest.
The first is that you should not update CDS columns one by one. You should not do this:
# BAD
markers.data['lon'] = marker_lons
markers.data['lat'] = marker_lats
This will generate two separate update events and two separate re-render requests. Apart from the extra work this causes, it's also the case that the first update is guaranteed to have mismatched old/new coordinates. Instead, you should always update CDS .data dict "atomically", in one go:
source.data = new_data_dict
Addtionally, you might try curdoc().hold to collect updates into fewer events.
Related
In Bokeh, I have a CustomJS triggering a callback updating a ColumnDataSource. I would like to have any update to this trigger some Python code. However, this does not work by specifying an on_change callback to the data source nor can I access the updated data. What am I doing wrong? My code...
from bokeh.models import ColumnDataSource, CustomJS, Button
from bokeh.layouts import layout
from bokeh.tile_providers import get_provider
from bokeh.plotting import figure
from bokeh.io import curdoc
from bokeh.events import Tap
def callback():
print('button checks source:')
print(source.data)
def callback_source(attr, old, new):
print('source changed!')
print(source.data)
# define the source, initially empty
source = ColumnDataSource(data=dict(lon=[], lat=[]))
# set a map
tile_provider = get_provider('CARTODBPOSITRON')
initial_bounds = ((-1000000, 4000000), (3000000, 8000000))
plot = figure(
x_range=initial_bounds[0], y_range=initial_bounds[1],
x_axis_type="mercator", y_axis_type="mercator"
)
plot.add_tile(tile_provider)
# show the current lon/lat pair in source
plot.circle(x="lon", y="lat", size=15, fill_alpha=0.8, source=source)
plot.js_on_event(Tap, CustomJS(args=dict(source=source), code="""
// get the source.data contents
const x = source.data['lon'];
const y = source.data['lat'];
// if source.data is empty, append the cursor coords
if (x.length == 0) {
x.push(cb_obj.x);
y.push(cb_obj.y);
// if source.data is populated, replace the cursor coords
} else {
x[0] = cb_obj.x;
y[0] = cb_obj.y;
}
source.change.emit();
"""))
# how to detect changes in source.data which trigger Python code?
source.on_change('data', callback_source)
button = Button(label="show source.data status", button_type="success")
# print the source status
button.on_click(callback)
curdoc().add_root(layout([button, plot]))
leads to a plot, where my Tap Events are correctly updating the circle, but apparently not updating the source variable.
2022-01-07 08:12:46,924 Starting Bokeh server version 2.4.2 (running on Tornado 6.1)
2022-01-07 08:12:46,927 User authentication hooks NOT provided (default user enabled)
2022-01-07 08:12:46,931 Bokeh app running at: http://localhost:5006/temp_test
2022-01-07 08:12:46,931 Starting Bokeh server with process id: 4797
2022-01-07 08:12:48,212 WebSocket connection opened
2022-01-07 08:12:48,214 ServerConnection created
button checks source:
{'lon': [], 'lat': []}
Based on the answer provided by #bigreddot, the following new JS code (source.change.emit() is not necessary) works:
const x=[cb_obj.x];
const y=[cb_obj.y];
const new_data = {lon:x, lat:y};
source.data = new_data;
To get a property update event sent back to Python you will have to actually assign an entire new ("dict") value to source.data, and not just update the arrays "in place". The call to source.change.emit() is purely a JS-side event and has no effect whatsoever on the Python side.
I am running a bokeh server with a simple circle glyph and a TapTool to select individual circles.
Now I want to have a button to select all glyphs and update the selection in the plot.
Here is my atempt:
from bokeh import plotting as bplt
from bokeh import layouts as blayouts
from bokeh import models as bmodels
from bokeh import io as bio
from bokeh.server.server import Server
fig = bplt.figure(tools="tap")
source = bmodels.ColumnDataSource(dict(x=[0,1], y=[0,1]))
r = fig.circle('x', 'y', source=source, size=10)
def handler(attr, old, new):
print('attr: {} old: {} new: {}'.format(attr, old, new))
# r.data_source.on_change('selected', handler)
r.data_source.selected.on_change('indices', handler)
button = bmodels.Button(label="select all", button_type="success", width=200)
def callback(event):
'''Here I would like to select all points in the plot with python code'''
# this is my atempt:
print('event: {}'.format(event))
print('data source selected:', r.data_source.selected.indices)
r.data_source.selected.indices = [0]
print('data source selected:', r.data_source.selected.indices)
button.on_click(callback)
def modify(doc):
layout = blayouts.row(fig, button)
doc.add_root(layout)
doc.title = "title"
print('modify', type(doc))
if __name__ == '__main__':
print('Opening Bokeh application on http://localhost:5006/')
server = Server({'/': modify}, num_procs=1)
server.start()
server.io_loop.add_callback(server.show, "/")
server.io_loop.start()
You can run the example just by using:
python3 example_code.py
My question now is the following: How can I select all Bokeh circle glyphs with a python callback identically to when I would select the same glyphs with the TapTool manually?
In Bokeh the TapTool lets you select just one glyph and mute the others. You can still select both circles in a Python callback in your example by doing r.data_source.selected.indices = [0,1] but with what purpose?
For multi-glyph selection you could use BoxSelectTool, LassoSelectTool or PolySelectTool
In the example below, I am trying to get the x and y coordinates that appear in the Div next to the plot when the bokeh plot is Tapped to be appended to the data dictionary coordList in their respective list.
import numpy as np
from bokeh.io import show, output_notebook
from bokeh.plotting import figure
from bokeh.models import CustomJS, Div
from bokeh.layouts import column, row
from bokeh.events import Tap
coordList = dict(x=[], y=[])
output_notebook()
def display_event(div, attributes=[], style = 'float:left;clear:left;font_size=10pt'):
"Build a suitable CustomJS to display the current event in the div model."
return CustomJS(args=dict(div=div), code="""
var attrs = %s; var args = [];
for (var i = 0; i<attrs.length; i++) {
args.push(Number(cb_obj[attrs[i]]).toFixed(2));
}
var line = "<span style=%r>(" + args.join(", ") + ")</span>\\n";
var text = div.text.concat(line);
var lines = text.split("\\n")
if (lines.length > 35)
lines.shift();
div.text = lines.join("\\n");
""" % (attributes, style))
x = np.random.random(size=4000) * 100
y = np.random.random(size=4000) * 100
radii = np.random.random(size=4000) * 1.5
colors = ["#%02x%02x%02x" % (int(r), int(g), 150) for r, g in zip(50+2*x, 30+2*y)]
p = figure(tools="pan,wheel_zoom,zoom_in,zoom_out,reset")
p.scatter(x, y, radius=np.random.random(size=4000) * 1.5,
fill_color=colors, fill_alpha=0.6, line_color=None)
div = Div(width=400, height=p.plot_height)
layout = row(p, div)
point_attributes = ['x', 'y']
p.js_on_event(Tap, display_event(div, attributes=point_attributes))
show(layout)
I'm not sure how the coordinates are saved and how to access them and append them to the lists.
There is no way to append to coordinates to a python object with code like above, because that code is generating standalone output (i.e. it is using "show"). Standalone output is pure static HTML and Bokeh JSON that is sent to browser, without any sort of connection to any Python process. If you want to connect Bokeh visualizations to a real running Python process, that is what the Bokeh server is for.
If you run a Bokeh server application, then you can use on_event with a real python callback to run whatever python code you want with the Tap even values:
def callback(event):
# use event['x'], event['y'], event['sx'], event['sy']
p.on_event(Tap, callback)
I'm studying gmaps and I'm trying refresh gmap marker using widgets.button, but I cannot refresh map when I click in button.
Maybe is a simple question, but I'm trying it for hours and can't solve.
Follow my code.
from IPython.display import display
import ipywidgets as widgets
import gmaps
gmaps.configure(api_key='')
class AcledExplorer(object):
"""
Jupyter widget for exploring the ACLED dataset.
The user uses the slider to choose a year. This renders
a heatmap of civilian victims in that year.
"""
def __init__(self):
self.marker_locations = [(None, None)]
self._slider = None
self._slider2 = None
title_widget = widgets.HTML(
'<h3>MY TEST, my test</h3>'
'<h4>test1 ACLED project</h4>'
)
map_figure = self._render_map(-15.7934036, -47.8823172)
control = self._render_control()
self._container = widgets.VBox([title_widget, control, map_figure])
def render(self):
display(self._container)
def on_button_clicked(self, b):
latitude = self.FloatSlider1.value
longitude = self.FloatSlider2.value
print("Button clicked.")
self.markers = gmaps.marker_layer([(latitude, longitude)])
return self._container
def _render_control(self):
""" Render the widgets """
self.FloatSlider1 = widgets.FloatSlider(
value=-15.8,
min=-34,
max=4.5,
step=0.2,
description='Latitude:',
disabled=False,
continuous_update=False,
orientation='horizontal',
readout=True,
readout_format='.1f',
)
self.FloatSlider2 = widgets.FloatSlider(
value=-47.9,
min=-74,
max=-33,
step=0.2,
description='Longitude:',
disabled=False,
continuous_update=False,
orientation='horizontal',
readout=True,
readout_format='.1f',
)
self.button = widgets.Button(
description="Plot!"
)
self.button.on_click(self.on_button_clicked)
controls = widgets.VBox(
[self.FloatSlider1, self.FloatSlider2, self.button])
return controls
def _render_map(self, latitude, longitude):
""" Render the initial map """
self.marker_locations = [(latitude, longitude)]
brasilia_coordinates = (-15.7934036, -47.8823172)
fig = gmaps.figure(center=brasilia_coordinates, zoom_level=3)
self.markers = gmaps.marker_layer(self.marker_locations)
fig.add_layer(self.markers)
return fig
AcledExplorer().render()
I start creating widgets, after I link values from Sliders to button. I need refresh marker position when click in button.
In function on_button_click I can view that news values of latitude and longitude are being getting from slider bar, so I'm update self.marker, maybe my mistake stay here.
Problem with your code
In on_button_click, you are not actually updating the marker layer. You currently write:
self.markers = gmaps.marker_layer([(latitude, longitude)])
but that just sets the markers attribute of your class. What you actually want to do is mutate the set of markers in your marker layer. The simplest change you can make is to change that line to:
self.markers.markers = [gmaps.Marker(location=(latitude, longitude))]
This mutates the markers attribute of your marker layer — basically the list of markers. Every time you press plot, it destroys the marker on the map and replaces it with a new one at an updated location.
Improving your solution
Using the high-level factory methods like marker_layer can obscure how jupyter-gmaps uses widgets internally. To make it somewhat more understandable, let's introduce a _create_marker() method that creates a gmaps.Marker object:
def _create_marker(self, latitude, longitude):
return gmaps.Marker(location=(latitude, longitude))
We can now use this in the initial render:
def _render_map(self, latitude, longitude):
""" Render the initial map """
brasilia_coordinates = (-15.7934036, -47.8823172)
fig = gmaps.figure(center=brasilia_coordinates, zoom_level=3)
self.marker_layer = gmaps.Markers()
initial_marker = self._create_marker(latitude, longitude)
self.marker_layer.markers = [initial_marker] # set the first marker
fig.add_layer(self.marker_layer)
return fig
Note that I have renamed self.markers to self.marker_layer to make it clear it's a layer.
Finally, the update code is now:
def on_button_clicked(self, _):
latitude = self.FloatSlider1.value
longitude = self.FloatSlider2.value
# look how closely the following two lines match the construction code
new_marker = self._create_marker(latitude, longitude)
self.marker_layer.markers = [new_marker]
The below code is a minimal example of an issues where a bokeh model doesn't update when it's attribute is set via a callback. I've found that removing and adding back a model object (not even the suspect one) from the layout of the curdoc forces it to refresh. I've shown this via the first button press.
Is there a more elegant way to force bokeh to redraw the figure?
The example is for DataTable.columns.formatter, but I've noticed that this applies to other model attributes as well (including axis ranges, where I've seen a workaround involving setting the range explicitly at figure creation to allow updates).
from bokeh.models.widgets import Dropdown, RadioButtonGroup, CheckboxGroup, \
Toggle, DataTable, TableColumn, NumberFormatter
from bokeh.plotting import figure, curdoc, ColumnDataSource
from bokeh.layouts import column, layout
def update_format(attr, old, new):
if toggle_commas.active == 1:
(t.columns[1].formatter)
# remove the commas
t.columns[1].formatter = NumberFormatter(format='0,0.[00]')
# show that it updates the actual attribute
print(t.columns[1].formatter)
del doc_layout.children[-1]
doc_layout.children.insert(1, toggle_commas)
else:
# change the formatter back and note that it doesn't update the table unless you remove and add something
(t.columns[1].formatter)
# remove the commas
t.columns[1].formatter = NumberFormatter(format='0.[00]')
# show that it updates the actual attribute
print(t.columns[1].formatter)
table_data = dict(
percentiles=['min', '1st', '5th', '10th', '25th', '50th',
'75th', '90th', '95th', '99th', 'max', '', 'mean', 'std'],
values=[i for i in range(1000, 1014)]
)
table_source = ColumnDataSource(table_data)
table_columns = [
TableColumn(field="percentiles", title="Percentile"),
TableColumn(field="values", title="Value", formatter=NumberFormatter(format='0.[00]'))
]
t = DataTable(source=table_source, columns=table_columns, width=400, height=600,
name='pct_table')
toggle_commas = Toggle(label='Commas', active=False)
toggle_commas.on_change('active', update_format)
doc_layout = layout(t, toggle_commas)
curdoc().add_root(doc_layout)