I'm learning Django on the fly so bear with me. My goal here is for a user to be able to upload a file, the server script process it and send the results on each iteration to the client live.
My current structure is: User uploads a file (csv), this is read via views.py then renders the blank html view, JS in the html creates a new websocket, and well this is where I'm not able to pass data outside of consumers.py or process the data.
I'm also not sure the structure of view > JS script > consumers > ? is the best route for processing files but I've not found much documentation on file uploads and channels.
views.py
from django.shortcuts import render
import pandas as pd
def someFunc(request):
if request.method == "POST":
global csvUpload
csvUpload = pd.read_csv(request.FILES['filename'])
return render(request, 'app/appPost.html')
I render the view here first so that the blank page displays prior to the script running.
appPost.html JS script
var socket = new WebSocket('ws://localhost:8000/ws/app_ws/')
socket.onmessage = function(event){
var data = JSON.parse(event.data);
console.log(data);
var para = document.createElement("P");
para.innerText = data.message;
document.body.appendChild(para);
}
consumers.py
from channels.generic.websocket import WebsocketConsumer
from asgiref.sync import async_to_sync
class WSConsumer(WebsocketConsumer):
def connect(self):
self.accept()
self.render()
async_to_sync(self.add_group)('appDel_updates_group')
Ultimately I'd then like to pass what's in this alarm file to the websocket connection.
app_alarm.py
from requests.api import request
import pandas as pd
import requests as r
def app_process(csvUpload):
csvUpload=pd.read_csv(request.FILES['filename'])
csvFile = pd.DataFrame(csvUpload)
colUsernames = csvFile.usernames
for user in colUsernames:
req = r.get('https://reqres.in/api/users/2')
t = req.json()
data = t['data']['email']
message = user + " " + data
channel_layer = get_channel_layer()
async_to_sync(channel_layer.group_send)(
'appDel_updates_group',
{'type': 'render', 'message': message}
)
How do you do something after you render the view? (Django)
This seems somewhat relevant, I'm going to see about creating a thread to run the file processing only after the render is done in views.
Edit: Threading allowed me to render the page then pass the data over to the alarm.py file for processing. This has solved the issue I was having by adding the following code into the views.py:
x = threading.Thread(target=app_alarm.sub_process, args=[csvUpload])
x.setDaemon(False)
x.start()
return render(request, 'efax/appPost.html')
This allows views to grab the file using request.FILES and pass it to the script before the page was rendered without waiting on the completion of the processing. That would have rendered the entire point of channels in this situation pointless. Hopefully this helps someone.
Copying from the original post edit:
Threading allowed me to render the page then pass the data over to the alarm.py file for processing. This has solved the issue I was having by adding the following code into the views.py:
x = threading.Thread(target=app_alarm.sub_process, args=[csvUpload])
x.setDaemon(False)
x.start()
return render(request, 'efax/appPost.html')
This allows views to grab the file using request.FILES and pass it to the script before the page was rendered without waiting on the completion of the processing. That would have rendered the entire point of channels in this situation pointless. Hopefully this helps someone.
Related
I need some help understanding the concepts needed to get my application to work. I am running two instances of localhost, one on 5000: and one on 8000:. One is a Flask instance (backend) and the other is a VUE instance (frontend). On my frontend, I have an onSubmit() function that does two things:
Sends the form data to the Flask instance via axios.post
Redirects the frontend to a result webpage.
On my backend, my Flask instance is supposed to return the information I need.
if request.method == "GET":
#Accessing page for first time
response_object['msg'] = "Waiting for user input"
elif request.method == "POST":
#Form has been submitted
data = request.get_json()
user = parseFormData(data)
itinerary = getResults(user)
return itinerary
I am unsure / unable how to check if this is working. On the redirected page, I have the following method that is called when the page is created.
getResponse(){
const path = "http://127.0.0.1:5000";
axios.post(path)
.then((res) => {
this.tmp = res.data;
console.log(this.tmp);
})
.catch((err) => {
console.error(err);
});
},
I am unable to get anything from this page actually. Any advice on where to go from here will be appreciated!
Edit:
The problem appears to lie in the sequence of events. On my Frontend, the form is submitted then the page is immediately redirected to a result page. While this is happening, my Backend receives the data, processes it and returns the information. That information appears to be lost. I believe this is sort of an example of async but I am unfamiliar with that concept as of now.
I'm building a website with Django and among other things I want to display the latest news about a certain topic. I have a python script in the back-end that I would like to program to retrieve the latest news once every 1 hour for example. In the meantime I want to be able to display the most recently retrieved news. I'm doing this in order to avoid that the script is being activated every time someone opens my website.
My script is in news.py:
import pprint
import requests
import datetime
import pandas as pd
import dateutil.parser
secret = "********"
url = 'https://newsapi.org/v2/everything?'
quote = 'Amazon'
parameters1 = {
'q': quote,
'pageSize': 100,
'sortby': 'publishedAt',
'apiKey': secret,
}
response1 = requests.get(url, params=parameters1)
response_json1 = response1.json()
text_combined1 = []
for i in response_json1['articles']:
if i['content'] != None:
case = {'Source': i['source']['name'], 'Title': i['title'], 'url': i['url'],
'Published on:': dateutil.parser.parse(i['publishedAt']).strftime('%Y-%m-%d'), 'Image': i['urlToImage']}
text_combined1.append(case)
data_amazon = pd.DataFrame.from_dict(text_combined1)
news1 = data_amazon.iloc[0]
news2 = data_amazon.iloc[1]
news3 = data_amazon.iloc[2]
My views.py looks like this:
from django.shortcuts import render
from .news import *
def dashboard(request):
content = {'data': data, 'news1': news1, 'news2': news2, 'news3': news3}
return render(request, 'dashboard.html',
content)
I'm new to web development but my understanding as of now is that every time someone sends a request to my webpage that script would be run, which would result in delay in the display of the news and most likely access denial to the news.api due to too many requests.
Thank you in advance!
A good way to do this is with Celery. It will let you schedule tasks in Django.
You can read more about it here, and see some other options as well.
Set up a scheduled job?
I have two different pages, one (A) that displays data taken from a model object, and one (B) that changes its fields.
I would like that when the post data is sent from B to the server, the server changes the values in A.
What is the best way to do it?
This example could work for me but it's in PHP... there is a way to replicate it whit Python?
https://www.w3schools.com/html/html5_serversentevents.asp
This is working example from w3schools in Django:
template
<!DOCTYPE html>
<html>
<body>
<h1>Getting server updates</h1>
<div id="result"></div>
<script>
if(typeof(EventSource) !== "undefined") {
var source = new EventSource("stream/");
source.onmessage = function(event) {
document.getElementById("result").innerHTML += event.data + "<br>";
};
} else {
document.getElementById("result").innerHTML = "Sorry, your browser does not support server-sent events...";
}
</script>
</body>
</html>
views
import datetime
import time
from django.http import StreamingHttpResponse
def stream(request):
def event_stream():
while True:
time.sleep(3)
yield 'data: The server time is: %s\n\n' % datetime.datetime.now()
return StreamingHttpResponse(event_stream(), content_type='text/event-stream')
urls
urlpatterns = [
path('stream/', views.stream, name='stream')
]
Update:
If you want to manage your notifications you can create the model like:
from django.db import models
class Notification(models.Model):
text = models.CharField(max_length=200)
user = models.ForeignKey(User, on_delete=models.CASCADE)
sent = models.BooleanField(default=False)
Then create the view that is looking for the first unsent notification and sends it:
#login_required
def stream(request):
def event_stream():
while True:
time.sleep(3)
notification = Notification.objects.filter(
sent=False, user=request.user
).first()
text = ''
if notification:
text = notification.text
notification.sent = True
notification.save()
yield 'data: %s\n\n' % text
return StreamingHttpResponse(event_stream(), content_type='text/event-stream')
And the send_notification function that creates an entry in the Notification model (just call this function from anywhere in your code):
def send_notification(user, text):
Notification.objects.create(
user=user, text=text
)
After reading this, I think I understood the whole thing (please comment if I’m wrong).
Django does NOT natively support keep-alive connections.
This means, when the client gets the message from the server, the connection is immediately closed after (like any classic HTTP request/response cycle).
What’s different with text/event-stream request, is that the client automatically tries to reconnect to the server every second (the length can be changed with a retry parameter).
Unfortunately, it seems using SSE in that case has no interest since it has the same con’s that polling (ie. a request/response cycle occurs each X seconds).
As expected and mentioned in other answers, I would need django-channels to create a persistent connection that prevent HTTP request/response overheads and ensure the message is sent immediately.
As mentioned in other answers, you will need to use Django Channels to properly handle asynchronous communication without tying up threads.
For an example, see the django-eventstream library which uses Channels to implement SSE.
I wanted to know, which of the emails, that I have sent to have opened the email.
Here is how I approach to solve the problem -
Create an image file in the html template that should be rendered.
< img src="{{ tracking_url }}" height="1" width="1">
Once an email is opened a request will be made to a url, which will have base64 encoded url pattern:
base64_pattern = r'([A-Za-z0-9+/]{4})*([A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)'
url(r'^tracking/(?P{})/$'.format(base64_pattern), 'tracking_image_url', name='tracking_image_url'),
That URL will serve an image, and update the counter, as follows -
TRANSPARENT_1_PIXEL_GIF = "\x47\x49\x46\x38\x39\x61\x01\x00\x01\x00\x80\x00\x00\xff\xff\xff\x00\x00\x00\x21\xf9\x04\x01\x00\x00\x00\x00\x2c\x00\x00\x00\x00\x01\x00\x01\x00\x00\x02\x02\x44\x01\x00\x3b"
view -
def tracking_image_url(request, mail_64=None):
eml = base64.b64decode(mail_64)
// Update the counters in the DB table for the email address.
return HttpResponse(TRANSPARENT_1_PIXEL_GIF, content_type='image/gif')
I do all this, but can't track the opens, what am I missing?
Try this
import base64
from django.http import HttpResponse
PIXEL_GIF_DATA = base64.b64decode(
b"R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7")
def index(request):
return HttpResponse(PIXEL_GIF_DATA, content_type='image/gif')
I am trying to execute a redirect but it does not seem to be happening. The XHR outputs that the page has finished loading but my page is not redirected at all. The database has the correct data that I queried for and all.
def post(self):
modcode = self.request.get("code")
email = users.get_current_user().email()
query = db.GqlQuery("SELECT * from ModuleReviews where code =:1 and email =:2", modcode, email).get()
if query != None:
self.redirect('/errorR')
else:
module = ModuleReviews(code=self.request.get("code"),text=self.request.get("review"))
module.text = self.request.get("review")
module.code = self.request.get("code")
module.ratings = self.request.get("ratings")
module.workload = self.request.get("workload")
module.diff = self.request.get("diff")
module.email = users.get_current_user().email()
module.put()
self.redirect('/display')
If you're using XHR, you will have to get your Javascript handler to do the redirect via window.location. However, since you always want a redirect, you should consider whether using Ajax is the right thing at all: just submitting via a normal POST would provide exactly the functionality you want without any Javascript needed.