I have created a lot of visualization for a transaction data set of my company using jupyter notebook. Now I want to make that code reusable so that after my internship other people from my company will be able to generate those visualizations on next quarter data set. Also, they don't know python. Hence I need to create something that grabs data and throws the visualization without much hassle. How can I do this.
On top of my head i could think of many ideas here are some of them.
Using python interface (tkinter) that accepts the data from the user and show visualization.
Another solution could be you can create a web app using flask and accept the data from the user and visualize it.
If all of this is not possible you could easily create a script that accepts user data and then separate in 2 categories or dataframe for eg. object datatypes and numerical datatype and then plot using for loops.
Related
I am working on creating an app that access PLC to gather data and then display it on the screen with the option of viewing the data as an interactive graph. There has been some work done in python that collects data, store them in excel and at the end process the data to get some meaning full data points and visualization using matplotlib.
They have used tkinter to be the interface to get data and display the current values on 2 screens, I have plans to update the program to store the data in a database and query them as per the needs using python.
What would be the ideal GUI package and the data visualization tools I can use to make the app interactive and easy to use on a PC?
As for as why I chose python, I feel comfortable using python.
Thank you guys in advance!
It's not a free solution, but you might want to look at Ignition [link]
I have been using it for several years now and it has some great depth and flexibility for capturing data from most brands of PLCs and then displaying the data on a HMI, webpage, uploading to SQL, and so on.
The scripting language is jython, so you should feel very comfortable there. You can also try if for free for two-hours at a time. Then reset the trial as many times as you like.
I am using Python and gspread to upload local .csv data to a google SpreadsheetA.
I have a separate google SpreadsheetB that uses =IMPORTRANGE to import the data from SpreadsheetA and create a pivot table and corresponding chart (both located on SpreadsheetB).
If I were to manually adjust any data in SpreadsheetA (e.g., alter value of any cell, add a value to an empty cell, etc), then the data in SpreadsheetB—with its corresponding pivot table and chart—update dynamically with the new data from SpreadsheetA.
However, when SpreadsheetA is updated with new data programmatically via Python, IMPORTRANGE in SpreadsheetB does not capture the new data.
Any ideas as to why this happens and how I might be able to fix?
Both Sheet A and B show the same number of rows. I am a bit confused with your IMPORTRANGE() formula though, why the ampersand?
=IMPORTRANGE("https://docs.google.com/spreadsheets/d/16DyWC8rsQB1ThpLiQh0p5xH9CYK2cPqbPH547ybw2Fo/edit#gid=1875728384",""&"TestgAPI!A:J")
I changed to this:
=IMPORTRANGE("https://docs.google.com/spreadsheets/d/16DyWC8rsQB1ThpLiQh0p5xH9CYK2cPqbPH547ybw2Fo/edit#gid=1875728384","TestgAPI!A:J")
Although probably not the ideal, my solution to this was to use gspread to add a new worksheet to spreadsheetA, which somehow manages to kickstart importrange() in SpreadsheetB.
I would still love to see a cleaner solution, if anyone knows of one—but this has continued to work since implementing a week ago.
Let's say I have a page I'd like to render which will present some (expensive to compute) data in a few ways. For example, I want to hit my database and get some large-size pile of data. Then I want to group that data and otherwise manipulate it in Python (for example, using Pandas). Say the result of this manipulation is some Pandas DataFrame that I'll call prepped_data. And say everything up to this point takes 3 seconds. (I.e. it takes a while...)
Then I want to summarize that data at a single URL (/summary): I'd like to show a bar graph, a pie chart and also an HTML table. Each of these elements depends on a subset of prepped_data.
One way I could handle this is to make 3 separate views hooked up to 3 separate URL's. I could make pie_chart_view which would make a dynamically generated pie chart available at /piechart.svg. I could make bar_graph_view which would make a dynamically generated bar graph available at /bargraph.svg. And I could make summary_view which would finish by rendering a template. That template would make use of context variables generated by summary_view itself to make my HTML table. And it would also include the graphs by linking to their URL's from within the template. In this structure, all 3 view functions would need to independently calculate prepped_data. That seems less-than-ideal.
As an alternative. I could turn on some kind of caching. Maybe I could make a view called raw_data_view which would make the data itself available at /raw_data.json. I could set this to cache itself (using whatever Django caching backend) for a short amount of time (30 seconds?). Then each of the other views could hit this URL to get their data and that way I could avoid doing the expensive calculations 3 times. This seems a bit dicey as well, though, because there's some real judgement involved in setting the cache time.
One other route could involve creating both graphs within summary_view and embedding the graphics directly within the rendered HTML (which is possible with .svg). But I'm not a huge fan of that since you wind up with bulky HTML files and graphics that are hard for users to take with them. More generally, I don't want to commit to doing all my graphics in that format.
Is there a generally accepted architecture for handling this sort of thing?
Edit: How I'm making the graphs:
One comment asked how I'm making the graphs. Broadly speaking, I'm doing it in matplotlib. So once I have a Figure I like generated by the code, I can save it to an svg easily.
I think your idea to store the prepped data in a file is a good one. I might name the file something like this:
/tmp/prepped-data-{{session_id}}.json
You could then just have a function in each view called get_prepped_data(session_id) that either computes it or reads it from the file. You could also delete old files when that function is called.
Another option would be to store the data directly in the user's session so it is cleaned up when their session goes away. The feasibility of this approach depends a bit on how much data needs to be stored.
I am relatively new to (Python) programming and would like some direction for my current project.
I have been learning to web scrape and parse data. I am now sitting on a program that can create lots of information in the form of lists and dictionaries.
I now need a way to create formatted tables and output to some sort of web-based system. I am looking at tables of about 10-40 rows and 20 columns of alphanumeric (names and numbers) data. If I can produce basic bar/line charts that would be useful. It also needs to be entirely automated - the program will run once a day and download information. I need it to output seamlessly in report form to something like dropbox that I can access on-the-go. The table template will always be the same and will be heavily formatted (colouring mostly, like Excel conditional formatting).
I am also keen to learn to create web apps and I'm wondering if this is something I can do with this project? I'm not sure what I'll need to do and I'd like some direction. I'm pretty new to programming and the jargon is tough to wade through. Is it possible to create a website that takes automated input and creates good-looking data tables? Is this efficient? What are the available tools for this? If not efficient what are the other available options?
I am attempting to add an Excel like filtering option to a program that is processing 3 xml files and optionally an xls file. One of the problems I am running into is finding good examples of applying multiple filters at once. I understand that Filter.Chain allows for multiple "pre-built" filters to be applied at once, the main trouble is being able to dynamically create the filters then applying them. For example, one of the things being looked at by the program is vehicle makes such as Honda, Ford, etc and I would like to be able to select which makes I want to see in the ObjectListView.
Due to the amount of code in use it is hosted at pastebin:
Main Gui
Worker Script
Input Panel
Primary Output
Secondary Output
What I am hoping to find are examples of how I would be able to add an Excel like filtering. Currently I have a menu option set that opens a MultiChoiceDialog window which provides a list of options, but I am unable to find a good pythonic way of taking the selections and applying them as a filter or series of filters.
Thanks in advance.
It appears that you are reading an Excel file into your ObjectListView widget. I think it would be easier to load the data into a sqlite database and then use SQL commands to do your filtering. I prefer using SQLAlchemy. That way I can create a class that represents the data that I can use both for SQLAlchemy and for my ObjectListView widget.
You can read about some of this sort of thing in the following articles:
http://www.blog.pythonlibrary.org/2012/06/04/wxpython-and-sqlalchemy-loading-random-sqlite-databases-for-viewing/
http://www.blog.pythonlibrary.org/2011/11/10/wxpython-and-sqlalchemy-an-intro-to-mvc-and-crud/