I created a sqlite3 database which has a table to store some data(table includes time, day, weather...etc. options). And ideally, I'd like to use scroll down selection way to show these data on web and let user to choose the specific time to see the result. Is there any way I can to that?
Please assume I have no experience in developing the structure of websites.
I am currently working on a website that displays data fetched from an API. (I use Django and python for the web app)
I want to display the current Covid-19 data and analyze the trend in graphs and charts.
I am able to grab the data from the API, but question is for a site like that needs an update on the numbers on a daily basis, what is the best practice for retrieving data. Should I be saving the api data to my database for easier access or should I just call the API everytime when the web page is visited (which is already a necessity due to the day to day update) and process data? (Performance wise or easily of processing data.)
I would appreciate if anyone would share their experience.
I have an input form with couple of input fields mostly numerical. I have a python script which takes the input values then performs some medium calculations (arrays, loops, etc) and gives results.
I would like to display results without a page refresh below the input form. Simplest example would be to display an average from the three input fields below when user clicks calculate button or instantly without even clicking.
Could anybody point me to the right direction how this would be implemented? I don't know what's the best framework to use for this. Just Ajax or Angular? Perform calls on client side or server?
I have a basic personal project website that I am looking to learn some web dev fundamentals with and database (SQL) fundamentals as well (If SQL is even the right technology to use??).
I have the basic skeleton up and running but as I am new to this, I want to make sure I am doing it in the most efficient and "correct" way possible.
Currently the site has a main index (landing) page and from there the user can select one of a few subpages. For the sake of understanding, each of these sub pages represents a different surf break and they each display relevant info about that particular break i.e. wave height, wind, tide.
As I have already been able to successfully scrape this data, my main questions revolve around how would I go about inserting this data into a database for future use (historical graphs, trends)? How would I ensure data is added to this database in a continuous manner (once/day)? How would I use data that was scraped from an earlier time, say at noon, to be displayed/used at 12:05 PM rather than scraping it again?
Any other tips, guidance, or resources you can point me to are much appreciated.
This kind of data is called time series. There are specialized database engines for time series, but with a not-extreme volume of observations - (timestamp, wave heigh, wind, tide, which break it is) tuples - a SQL database will be perfectly fine.
Try to model your data as a table in Postgres or MySQL. Start by making a table and manually inserting some fake data in a GUI client for your database. When it looks right, you have your schema. The corresponding CREATE TABLE statement is your DDL. You should be able to write SELECT queries against your table that yield the data you want to show on your webapp. If these queries are awkward, it's a sign that your schema needs revision. Save your DDL. It's (sort of) part of your source code. I imagine two tables: a listing of surf breaks, and a listing of observations. Each row in the listing of observations would reference the listing of surf breaks. If you're on a Mac, Sequel Pro is a decent tool for playing around with a MySQL database, and playing around is probably the best way to learn to use one.
Next, try to insert data to the table from a Python script. Starting with fake data is fine, but mold your Python script to read from your upstream source (the result of scraping) and insert into the table. What does your scraping code output? Is it a function you can call? A CSV you can read? That'll dictate how this script works.
It'll help if this import script is idempotent: you can run it multiple times and it won't make a mess by inserting duplicate rows. It'll also help if this is incremental: once your dataset grows large, it will be very expensive to recompute the whole thing. Try to deal with importing a specific interval at a time. A command-line tool is fine. You can specify the interval as a command-line argument, or figure out out from the current time.
The general problem here, loading data from one system into another on a regular schedule, is called ETL. You have a very simple case of it, and can use very simple tools, but if you want to read about it, that's what it's called. If instead you could get a continuous stream of observations - say, straight from the sensors - you would have a streaming ingestion problem.
You can use the Linux subsystem cron to make this script run on a schedule. You'll want to know whether it ran successfully - this opens a whole other can of worms about monitoring and alerting. There are various open-source systems that will let you emit metrics from your programs, basically a "hey, this happened" tick, see these metrics plotted on graphs, and ask to be emailed/texted/paged if something is happening too frequently or too infrequently. (These systems are, incidentally, one of the main applications of time-series databases). Don't get bogged down with this upfront, but keep it in mind. Statsd, Grafana, and Prometheus are some names to get you started Googling in this direction. You could also simply have your script send an email on success or failure, but people tend to start ignoring such emails.
You'll have written some functions to interact with your database engine. Extract these in a Python module. This forms the basis of your Data Access Layer. Reuse it in your Flask application. This will be easiest if you keep all this stuff in the same Git repository. You can use your chosen database engine's Python client directly, or you can use an abstraction layer like SQLAlchemy. This decision is controversial and people will have opinions, but just pick one. Whatever database API you pick, please learn what a SQL injection attack is and how to use user-supplied data in queries without opening yourself up to SQL injection. Your database API's documentation should cover the latter.
The / page of your Flask application will be based on a SQL query like SELECT * FROM surf_breaks. Render a link to the break-specific page for each one.
You'll have another page like /breaks/n where n identifies a surf break (an integer that increments as you insert surf break rows is customary). This page will be based on a query like SELECT * FROM observations WHERE surf_break_id = n. In each case, you'll call functions in your Data Access Layer for a list of rows, and then in a template, iterate through those rows and render some HTML. There are various Javascript and Python graphing libraries you can feed this list of rows into and get graphs out of (client side or server side). If you're interested in something like a week-over-week change, you should be able to express that in one SQL query and get that dataset directly from the database engine.
For performance, try not to get in a situation where more than one SQL query happens during a page load. By default, you'll be doing some unnecessary work by going back to the database and recomputing the page every time someone requests it. If this becomes a problem, you can add a reverse proxy cache in front of your Flask app. In your case this is easy, since nothing users do to the app cause its content to change. Simply invalidate the cache when you import new data.
I am building an app that uses the Oanda API to retrieve historical forex data. There are 28 forex pairs and 10 years worth of historical data. Since none of this data changes I was planning on saving it into my database rather than blowing up the API.
My goal is to initially populate the database for all pairs and then update the data once per minute from then on.
What I can't figure out is how to do this effectively.
Where should the logic for this live inside the Django app?
How can I execute the initial population of the data so that it will save?
It's the saving that is giving me the most problems. As far as I can tell Django only likes to save model instances from the shell.
Any help would be greatly appreciated.
You might want to take a look at this answer and to the django-admin commands.
I hope this helps! =)
Generally you should do operations like this inside proper view.
If you want to save some data once per minute just create a method that will implement it and refresh it with for example Ajax from time to time (for example once per minute). You don't have to render page from beginning - everything can work in the background.
Remember that you will need psycopg2 module to interact with Postgres