I am working on a little personal project with the aim of using Python to obtain and process data from a running game and use HTML/CSS to output this data using Flask.
I am very new to Python, but have some basic exposure to HTML/CSS and JavaScript.
Using example code from the web and various tutorials I have been able to get two distinct sections of this working to a proof of concept stage:
My python code runs in an interuptible loop and will output a variable every X seconds to the console.
I have setup a Flask application that I can pass a variable to in a Python script and display in a browser.
My problem is I am stumped with how I go about joining these two together. In the Python script that runs the Flask app nothing else happens while the Flask app is running, I assume because it is executing synchronously. This means that I can't start the Flask app and then have my loop run after that, how can I get both to work simultaneously?
EDIT:
After more thinking about this and some further research I think I have framed the problem incorrectly due to the way I have setup my local environment.
My original thinking was that the Python script which runs the Flask server could also be used to recieve/process and output my game logic*. I now believe this to be incorrect because in typical usage Flask would be running on a web server somewhere serving the website while the data from the game would have to be collected and processed by an application local to the running of the game.
So, given the above the question could be better posed as:
How to dynamically display (say with a maximum refresh interval of 1s) a Python variable that is constantly changing using a Flask server? How do I get these two separate parts of my project to communicate with one-another?
*If this were possible (and I don't know its not) it would actually accomplish what I want from this project but is not likely to be a particularly useful skill so I would like to figure out how to do this the correct way in case this becomes something I would like to make available to others.
Related
I have a processing engine built in python and a driver program with several components that uses this engine to process some files.
See the pictorial representation here.
The Engine is used for math calculations.
The Driver program has several components.
Scanner keeps scanning a folder to check for new files, if found makes entry into DB by calling a API.
Scheduler picks new entries made by scanner and schedules them for processing (makes entry into 'jobs' table in DB)
Executer picks entries from job table and executes them using the engine and outputs new files.
All the components run as separate python process continuously. This is very in efficient, how can I improve this? The use of Django is to provide a DB (so the multiple processes could communicate) and keep a record of how many files are processed.
Next came a new requirement to manually check the processed files for errors so a UI was developed for this. Also the assess to the engine was to be made API based. See the new block diagram here
Now the entire thing is a huge mess in my opinion. For start, the Django now has to serve 2 different sets of API - one for the UI and other for the driver program. If the server stops, the UI stops working and also the Driver program stops working.
Since the engine is API based there is a huge amount of data passed to it in the request. The Engine takes several minutes (3 to 4) to process the files and most of the time the request to engine get timeout. The Driver program is started separately from terminal and it fails if Django server is not running as the DB APIs are required to schedule jobs and execute the jobs.
I want to ask what is the beast way to structure such projects.
should I keep the Engine and driver program logic inside Django? In this case how do I start the driver program?
Should I keep both of them outside Django, in which case how do I communicate with Django such that even if the Django server is down I can still keep processing the files.
I would really appreciate any sort of improvement ideas in any of the areas.
Im new to Javascript so I would like to keep it at the bare minimum. Is there a way that I can use the Electron to communicate with python script without having node.js? My app is just a basic app that takes some input from users from a html page and I need this text input to be processed in python and write an excel file. So there is not much happening in html so is there a simple way to transfer the input to python file? I want to use Electron because I need this html to be my UI and also I need to distribute this app.
I guess the answer is "no": the main process running node will always be there.
An Electron app consists of a JavaScript main process, and one or more JavaScript renderer processes. There is no built-in Python support. And the user will need Python already installed. So, it sounds like a poor fit for what you need.
The answers here may be useful, and will show how to call the python script. I took a quick look at the flexx toolkit mentioned there. It seems to work with the user's browser, rather than producing a single executable.
Recently i have done it with some sort of trick hope it will help you and there are the following step which i followed-
Created a stand alone python exe using pyinstaller and the exe has flask server internally then i put the flask server inside my node application.
Now we have to initiate our flask server and send a request to it for processing, i have done this with the help of "execFile" function as a child process, for which i have created a function and the code was something like that-
async function callFlask(){
var child = require('child_process').execFile;
child('path_to_python_exe ', function(err, data) {
if(err){
console.error(err);
return;
}
});
}
Now we have initiated our flask server then will send the request with the help of fetch request like
await callFlask().then(
await fetch('host_ip_defined_in_flask'+encodeURIComponent('data'))
Now further we can extend our then chain to get response from python if any and proceed further forexample -
await callFlask().then(
await fetch('host_ip_defined_in_flask'+encodeURIComponent('data'))
.then(res => res.text())
.then(body => console.log(body)))
Here, your output data which python return will be printed in console then you can make your node application behave differently depending on output returned by it.
Also you can package your app with available packagers for electron like electron-packager it will work like a charm.
Also there is are some disadvantage for using python as like it will increase your package size and the process will be difficult to kill from electron after processing so it will increase burden on host machine.
I am assuming that Explaining to create a flask server is not the scope of this question instead if you face any issues let me know, i hope it will help...
Really newbie questions.
I made a Python bot which receives some data and has to analyze it,then prints everything. To use it, i need it to run for the whole day, the problem is that i can't leave my computer on 24/7, so i need a server or something similar for it and i need to be able to check what it prints whenever i want.
I made some research and found Heroku, but i'm having some problems understanding it: i tried to deploy it there and it's working but it prints all the stuff on a shell in the app's page and not on the webpage that heroku assigned to my app, so my problem is partially solved, since i can run it for the whole day but checking what it prints is way harder.
I was thinking of making it as a Telegram bot in order to have everything there but since it prints a lot of stuff, Telegram would not be the best platform for this kind of thing.
Is there another resource to deploy it and have it, for example, on a webpage?
You can look into renting a cheapish cloud server (from digitalocean for example).
There are multiple ways of transferring data from your python script to your bot, either directly, through a websocket, or a webpage that displays it in a JSON format or otherwise.
Since you're already using python you could look into running a flask app on your node alongside your script or even combine them together.
If ran separately you could modify your script to output it's content into a file and then read the file with your flask application to display it on a webpage. For example:
with open('/tmp/data.txt', 'w') as f:
f.write(yourdata)
then in your flask application:
from flask import Flask
app = Flask(__name__)
#app.route('/')
def show_data():
with open('/tmp/data.txt', 'r') as f:
data = f.read()
return data
There are way more efficient ways of transferring data. Example above is a quick and dirty solution I wouldn't recommend running it due to security reasons especially if you are transmitting sensitive data.
Newbie on appengine and I really don't know how to phrase the question which sadly results in me not knowing what keywords to google and I hope that i really do get help other than the bashing that a lot of people do.
I'm confused between the behavior of appengine online and the appengine on the local server.
Background info:
Btw this is in Python
Initially i assumed that , when needed or as authored
an instance of the app or module will be created.
And that instance will be the one serving multiple requests from different clients.
In this behavior any initialization code will only be run once.
But in the local development server.
Every time i add something new, specially in the main.py,
the server is able to catch the new changes,
then on browser-refresh be able to run it.
This made me think, wait...
Does it run the entire script over and over again
on every request?
Question:
Does an instance/module run the entire code on every request or is this just an added behavior to the dev server to make development easier?
Both your assumptions - about behaviour in production and development - are wrong.
In production, GAE spins up instances as required. This may be in response to increased load, or the host may simply decide after a certain amount of time to recycle an instance by killing it and starting a new one. Initialization code will always be run whenever a new instance is started.
In development, you only get a single instance. However, the server watches your file system for changes. If it detects a change to the code itself, it will restart itself, and therefore re-run the initialization code. But if you don't make any code changes between requests, the existing process continues indefinitely, and init code will not be re-run.
I am an experienced Python developer starting to work on web service
backend system. The system feeds data (constantly) from the web to a
MySQL database. This data is later displayed by a frontend side (there
is no connection between the frontend and the backend). The backend
system constantly downloads flight information from the web (some of
the data is fetched via APIs, and some by downloading and parsing
text / xls files). I already have a script that downloads the data,
parses it, and inserts it to the MySQL db - all in a big loop. The
frontend side is just a bunch of php pages that properly display the
data by querying the MySQL server.
It is crucial that this web service be robust, strong and reliable.
Therefore, I have been looking into the proper ways to design it, and came across the following parts to comprise my system:
1) django as a framework (for HTTP connections and for using Piston)
2) Piston as an API provider (this is great because then my front-end can use the API instead of actually running queries)
3) SQLAlchemy as the DB layer (I don't like the little control you get when using django ORM, I want to be able to run a more complex DB framework)
4) Apache with mod_wsgi to run everything
5) And finally, Celery (or django-cron) to actually run my infinite loop that pulls the data off the web - hopefully in some sort of organized tasks format). This is the part I am least sure of, and any pointers are appreciated.
This all sounds great. I used django before to write websites (aka
request handlers that return data). However, other than using Celery or django-cron I can't really see how it fits a role of a constant data feeding backend.
I just wanted to run this by you guys to hear your ideas / comments. Any input you have / pointers to documentation and/or other libraries would be greatly greatly appreciated!
If You are about to use SQLAlchemy, I would refrain from using Django: Django is fine if You are using the whole stack, but as You are about to rip Models off, I do not see much value in using it and I would take a look at another option (perhaps Pylons or pure old CherryPy would do).
Even more so if FEs will not run queries, but only ask API providers.
As for robustness, I am more satisfied with starting separate fcgi processess with supervise and using more lightweight web server (ligty / nginx), but that's a matter of taste.
For the "infinite loop" part, it depends on what behavior you want: if there is a problem with the source, would you just like to skip the step or repeat it multiple times when source is back up?
Periodic Tasks might be good for former, while cron that would just spawn scraping tasks is better for latter.