We Keep Coding

Python/Jupyter: scroll scatter-plot with many data points horizontally

I want to visualize time-series-like data with several measurements over time.
There are a lot of such measurements in a dataset, in the order of tens to hundreds of thousands.
In order to view these in a notebook or HTML page, I would like some efficient method to show a subrange of the whole time range with just a view hundred to thousand database and have controls to scroll lef/right i.e. forward/backward in time through the data.
I have tried doing this with Plotly and a range slider, but unfortunately this does not scale to a lot of data at all. Apparently, this approach creates all the graph data in the output javascript, which slows down everything and at some point makes the browser hang or crash.
What I would need is an approach that actually only renders the data in the subrange and interacts with the python code via the scrolling widgets to update the view.
Ideally, this would work with Plotly as I am using it for all other visualizations, but any other efficient solution would also be welcome.

Plotly runs into rendering issues when there are too many data points within the window (see Plotly Benchmarks). I would suggest using Plotly-Resampler which resamples data that is within the user's view.

Choroplethmapbox slow to render?

I’ve been playing around with Plotly and Dash for the first time over the past few days, with the hope of developing a browser-based data explorer for geographic NetCDF4 data. I’ve been impressed at how straightforward this has been so far, however I’m finding that some interactions with choroplethmapbox are taking longer to update and render than expected. I believe this may be the same issue discussed here
The following refers to the code and sample data available here, where the Dash application can be run using:
python choropleth.py (Python 3.7).
The source of my data comes from a 4D NetCDF4 file (in this case a model of ocean temperature - temp.nc) with dimensions of time, depth, lat and lon. In my case I’m only plotting a 2D chloropleth map, but I’d like the user to interactively select the desired time interval (and eventually depth) as well (the render will always be in 2D space).
Using the examples from here, I’m using a GeoJSON file of the 2D grid cells coupled with a Pandas DataFrame to render ocean temperature. Everything is working as expected, however any changes to the slider value (time) take a long time to update (approx six seconds on my machine). It appears as though there’s a second or so between selecting the slider value and running the update_figure() callback, then another 4-5 seconds before the new render starts to take place in the browser.
The update_figure() callback reads the requested data directly from the NetCDF4 file, then directly updates the Z values in the existing figure dictionary and returns this as a new figure (see code fragment below). At first I was concerned that the slow response time was due to reading from the NetCDF4, however a basic timing function shows that the update_figure() callback runs in less than 0.01 seconds in most cases. So it appears the delay is either coming from the #app.callback or the render function (post update_figure()) in Dash?
# Create the callback and callback function (update_figure)
#app.callback(Output('plot', 'figure'),
[Input('slide', 'value')],
[State('plot','relayoutData'),State('plot', 'figure')])
def update_figure(x,r,f):
t0 = tme.time()
f['layout']['mapbox']['center']['lat'] = f['layout']['mapbox']['center']['lat']
f['layout']['mapbox']['center']['lon'] = f['layout']['mapbox']['center']['lon']
f['layout']['mapbox']['zoom'] = f['layout']['mapbox']['zoom']
# If the map window has been panned or zoomed, grab those values for the new figure
if r is not None:
if 'mapbox.center' in r:
f['layout']['mapbox']['center']['lat'] = r['mapbox.center']['lat']
f['layout']['mapbox']['center']['lon'] = r['mapbox.center']['lon']
f['layout']['mapbox']['zoom'] = r['mapbox.zoom']
# Extract the new time values from the NetCDF file
tmp = nc['temp'][x, -1, :, :].values.flatten()
# Repace the Z values in the original figure with the updated values, leave everything else (e.g. cell geojson and max/min ranges) as-is
f['data'][0]['z'] = np.where(np.isnan(tmp), None, tmp).tolist()
print("update_figure() time: ",tme.time()-t0)
return f
I suspect that the slow render times are somehow related to the GeoJSON of each cell polygon (47k grid cell polygons are being rendered in total, with each polygon being defined by 6 points (i.e. 284k points total)), and unfortunately this can’t be simplified any further.
I'm seeking suggestions on how I can speed up the update/render when a user is interacting with the application. Two ideas I've had include:
Utilising WebGL if possible? It's unclear to me from the documentation whether choroplethmapbox already uses WebGL? If not, is there a pathway for making use of this for faster rendering?
Implementing some form of client side callback, although I don't know if this is possible given that I need to read the values directly out of the NetCDF file when requested by the user? Perhaps it's possible to just read/return the new Z values, then merge that with the existing GeoJSON on the client side?
Suggestions appreciated.

Trigger update of Bokeh plot upon new data point in stream

I am trying to create a plot in Bokeh to visualize data from a live data-feed. I'm fairly new to Bokeh at this point. The data stream is a stream of files where the data for the plots first requires to be extracted and then pre-processed before being visualized. This part is currently handled by the python watchdog package, where the processing is triggered upon the appearance of a new file in the streams that are monitored.
The output of this is a dictionary holding all the information for this particular datapoint needed in the plots handled by the Bokeh app.
My question is, how would I trigger an update of the Bokeh plot when a new datapoint arrives?
I had looked into add_periodic_callback, but since I do not know up front when a new datapoint will arrive, nor how much time there will be between them, I risk missing data in the plot. What would be the best way to solve this?
1) Use functionality "x" in Bokeh that I am unaware of, that will trigger an update of the ColumnDataSource and the actual plots, exactly when a new datapoint arrives (this would be my preferred solution).
2) Create a form of buffer data source that stores the data for the past NN files and then use add_periodic_callback to a function that queries this source to update the ColumnDataSource
3) Another solution than the two above, that I don't know of with my limited software development skills.

matplotlib, draw multiple graphs / points in figure

I am trying to develop a telemetry system which sends sensor data from an Arduino, plotted in realtime. For this I'm using Python and the matplotlib-library. My problem is that every time a new data point arrives, I want to add that data point by plotting it into the same figure as the other data points. So far I could not find a solution to this.

You can stream data from an Arduino into a Plotly graph with the Arduino API in Plotly. You have two options: continuously transmit data (which it sounds like you'll want to do), or transmit a single chunk.
It will update the graph every few seconds if you refresh the page.
The Arduino API is available here. And, if you're already using Python, you can use the extend option to update data into another plot. The Python API is here.
Here's an example of how it looks to transmit from an Arduino, and you can see the interactive version here
Full disclosure: I work at Plotly.

as far as I can see, you have a few different ways of doing this (i'll list them in what I consider increasing difficulty
Making a bitmap file, eg .png, which has to be regenerated each time a new datapoint arrives. To do this you need to have your old data stored somewhere in a file or in a database.
Using svg in a browser. Then you can add on points or lines using javascript (e.g. http://sickel.net/blogg/?p=1506 )
Make a bitmap, store it and edit it to add in new points - this really gets tricky if you either wants to "roll old points off" at one end, or rescale the image when more data arrives.
Make a series of bitmaps, and have the total graph as a combination of a lot of slices. - here you can easily "roll off" old points, but you are out of luck if you want to rescale.

How to visualize IP addresses as they change in python?

I've written a little script that collects my external IP address every time I open a new terminal window and appends it, at well as the current time, to a text file. I'm looking for ideas on a way to visualize when/how often my IP address changes. I bounce between home and campus and could separate them using the script, but it would be nice to visualize them separately.
I frequently use matplotlib. Any ideas?

Plot your IP as a point on the xkcd internet map (or some zoomed in subset of the map, to better show different but closely neighboring IPs).
Plot each point "stacked" proportional to how often you've had that IP, and color the IPs to make more recent points brighter, less recent points proportionally darker.

"When" is one dimensional temporal data, which is well shown by a timeline. At larger timescales, you'd probably lose the details, but most any plot of "when" would have this defect.
For "How often", a standard 2d (bar) plot of time vs frequency, divided into buckets for each day/week/month, would be a standard way to go. A moving average might also be informational.
You could combine the timeline & bar plot, with the timeline visible when you're zoomed in & the frequency display when zoomed out.
How about a bar plot with time on the horizontal axis where the width of each bar is the length of time your computer held a particular IP address and the height of each bar is inversely proportional to the width? That would also give a plot of when vs how often plot.
You could also interpret the data as a pulse density modulated signal, like what you get on a SuperAudio CD. You could graph this or even listen to the data. As there's no obvious time length for an IP change event, the length of a pulse would be a tunable parameter. Along similar lines, you could view the data as a square wave (triangular wave, sawtooth &c), where each IP change event is a level transition. Sounds like a fun Pure Data project.

There's a section in the matplotlib user guide about drawing bars on a chart to represent ranges. I've never done that myself but it seems appropriate for what you're looking for.

Assuming you specified terminal, i'll assume you are on a UNIX variant system. Using the -f switch along with the command line utility tail can allow you to constantly monitor the end of a file. You could also use something like IBM's inotify, which can monitor file changes or dnotify (and place the file in it's own directory) which usually comes standard on most distributions (you can then call tail -n 1 to get the last line). Once the line changes, you can grab the current system time since epoch using Python's time.time() and subtract it from the time of the last change, then plot this difference using matplotlib. I assume you could categorize the times
into ranges to make the graphing easier on yourself. 1 Bar for less than 1 hour change intervals, another for changes between 1 - 5 hours, and so on.
There is a Python implementation of tail -f located here if you don't want to use it directly. Upon a detection of a change in the file, you could perform the above.