I have a webpage showing some data. I have a python script that continuously updates the data(fetches the data from database, and writes it to the html page).It takes about 5 minutes for the script to fetch the data. I have the html page set to refresh every 60 seconds using the meta tag. However, I want to change this and have the page refresh as soon as the python script updates it, so basically I need to add some code to my python script that refreshes the html page as soon as it's done writing to it.
Is this possible ?
Without diving into complex modern things like WebSockets, there's no way for the server to 'push' a notice to a web browser. What you can do, however, it make the client check for updates in a way that is not visible to the user.
It will involve writing Javascript & writing an extra file. When writing your main webpage, add, inside Javascript, a timestamp (Unix timestamp will be easiest here). You also write that same timestamp to a file on the web server (let's call it updatetime.txt). Using an AJAX request on the page, you pull in updatetime.txt & see if the number in the file is bigger than the number stored when you generate the document, refresh the page if you see an updated time. You can alter how 'instantly' the changes get noticed but controlling how quickly you poll.
I won't go into too much detail on writing the code but I'd probably just use $.ajax() from JQuery (even though it's sort of overkill for one function) to make the calls. The trick to putting something on a time in JS is setinterval. You should be able to find plenty of documentation on using both of them already written.
Related
I want to have some python code use selenium or bs4 to check a website hourly for updates and the get that information and put it on a website in whatever way I want to format it. I would like to know what's the best way to connect these two together.
I would probably set up a cron job (scheduled task on windows) to run hourly to run your script to scrape the website you're getting data from. The logic in your script will depend on the website you're entering data into, but broadly speaking I would scrape the data, process it, then either send a POST request to the website or update the data source (a db, file, whatever) directly.
So I had a number of amino acid sequence strings that I wanted to use as input into a tool that studies its interactions with certain components of the human immune system (http://www.cbs.dtu.dk/services/NetMHCcons/).
I wanted to ask what, if any, would be a way of accessing, inputting data and getting the output, via a script (R or python preferably). My main issue was I had a lot of sequences that need to be queried separately so wanted to automate the whole thing. The website has one field that reads "Submission" which takes in the string input. There is another field "select species/loci" which gives a drop down menu from which an option needs to be selected. Lastly there's a "submit" button. The output simply loads on the page after hitting submit.
I've tentatively poked around with RSelenium and Rcurl but wanted to ask if there was a more efficient method.
I took a look at what it'd take to send a POST request to this service from Python, and it looks possible:
this form takes in "multipart/form-data" (see: How to send a "multipart/form-data" with requests in python?), you'll need to send your data in this format. You could inspect a request from the browser (using the dev tools) and copy the fields from there as a starting point.
once the form is submitted, it doesn't give you the result right away. You'd need to get your job ID from the response, and then poll the URL: http://www.cbs.dtu.dk/cgi-bin/webface2.fcgi?jobid={your_job_id}&wait=20 until it gives you the result
the result will then need to be downloaded and parsed
This tool is however available as a portable version for linux/mac: https://services.healthtech.dtu.dk/software.php
Perhaps downloading this version would make it easier?
Try this :
Submitting to a web form using python
This link is an answer to how to send web forms in python, using urllib. Check your source code and extract the necessary data using re module from the source code of the link you have put up, and send the request.
save the HTML source code of http://www.cbs.dtu.dk/services/NetMHCcons/ in the python file as
source_code = '''...'''
The HTML can be found by using CTRL+U in firefox.
Let me provide a little background.
An organization i am volunteering for delivers meals to people who are unable to come pick them up during the holidays.
They currently have a SQL Server DB that stores the information of all their clients along with the meal information for each year.
Currently a Java desktop application connects to the SQL Server DB and allows several functions to happen.
i.e. Add a client, add meals, remove clients, print delivery sheets.
I am hoping to use python Flask to rewrite the application as a web based application. The one function i am interested in at the moment is the print delivery sheets function.
The way this works is there is a setting for the current year. When you click the print deliveries for year button it will batch print a document for each customer onto an 8.5" x 11.5" paper. The sheet will be split in two with the same exact information on each side. This information includes the customer name, address, number of meals and so forth.
What i am wondering is how/what would be the best way to setup this template so that i could batch print it using python. I was thinking of creating an html template for the page but i am not sure how that would work.
Again i need to pass in every customer within that year into the template and batch print to 8.5" by 11.5" sheet.
What i am asking is.....
How could i create a template for the print that i can pass every customer two.
How would i print that template for every customer in a batch manner for every customer.
I was hoping to do this all in python if possible.
Thank you for your time and help.
If you are already deploying this as a web app, it will probably be easier to design and generate a pdf. You can use an html to pdf converter, which there are several of on PyPI, or there are plenty of resources online, such as:
How to convert webpage into PDF by using Python
https://www.smallsurething.com/how-to-generate-pdf-reports-with-jinja2-and-pyqt/
Once you have found a way to generate PDFs, you can then just use them like any other PDF, and either have the user download them or print them from the browser (this may require a little bit of Javascript, but this shouldn't be that hard since it should pretty much just be a window.open call.
For instance, you can add a button
<button onclick="getPDF()">Download PDF</button>
Which will then call a function called getPDF() which you define, which finds a way to generate the pdf.
function getPDF() {
// Find the uri for the pdf by some method
var urlToPdf = getUrlToPdf();
// Open PDF in new window
window.open(urlToPdf, "_blank");
}
Note Since you are using Flask, you can include the URL for the pdf in the source for the page, even the Javascript using the {{ }} syntax. Then the pdfs are only generated when someone requests that route.
This way you will not have to worry about connecting to a printer yourself at all, just use the browser to handle those kinds of tasks.
First I guess I should say I am still a bit of a Django/Python noob. I am in the midst of a project that allows users to enter a URL, the site scrapes the content from that page and returns images over a certain size and the page title tag so the user can then pick which image they want to use on their profile. A pretty standard scenario I assume. I have this working by using Selenium (headless Chrome browser) to grab the destination page content, some python to determine the file size and then my Django view spits it all out into a template. I then have it coded in such a way that the image the user selects will be downloaded and stored locally.
However I seriously doubt the scalability of this, its currently just running locally and I am very concerned about how this would cope if there were lots of users all running at the same time. I am firing up that headless chrome browser every time a request is made which doesn't sound efficient, I am having to download the image to determine it's size so I can decide whether it's large enough. One example took 12 seconds to get from me submitting the URL to displaying the results to the user, whereas the same destination URL put through www.kit.com (they have very similar web scraping functionality) took 3 seconds.
I have not provided any code as the code I have does what it should, I think the approach however is incorrect. To summarise what I want is:
To allow a user to enter a URL and for it to return all images (or just the URLs to those images) from that page over a certain size (width/height), and the page title.
For this to be the most efficient solution, taking into account it would be run concurrently between many users at once.
For it to work in a Django (2.0) / Python (3+) environment.
I am not completely against using the API from a 3rd party service if one exists, but it would be my least preferred option.
Any help/pointers would be much appreciated.
You can use 2 python solutions in your case:
1) BeautifulSoup, and here is a good answer how to download the images using it. You just have to make it a separate function and pass site as the argument into it. But also it is very easy to parse only images links as u said - depending on speed what u need (obviously scraping files, specially when there is a big amount of them, will be much slower, than links). This tool is just for parsing and scrapping the content of the page.
2) Scrapy - this is much more powerful tool, framework, via it you can connect your spider to a Django models, operate with images much more efficiently, using its built-in image-pipelines. It is much more flexible with a lot of features how to operate with scrapped data. I am not sure if u need to use it in your project and if it is not overpowered in your case.
Also my advice is to run the spider in some background task like Queue or Celery, and call the result via AJAX, cuz it may take some time to parse the content - so don't make a user wait for the response.
P.S. You can even combine those 2 tools in some cases :)
There is a website that I frequently go to and play a browser game. I want to be able to have some kind of firefox plugin that can scrape data off of the page and send it to a python script. I want the controls for the program (toggle on/off) to be a HTML display which is added on to the webpage after every time I load it.
I am aware of plugins like Greasemonkey, but I don't want to use this because if I want to send any data to python, I have to setup a python http server and manually launch it every time I want to use my program.
Essentially this is what I want to be able to do:
Open Firefox as I would normally to do any kind of internet browsing
Go to the website which has my game.
The game is loaded, javascript code is executed which adds some basic HTML controls which can be used to toggle settings in my backend python program
If I choose to enable the program, javascript will parse the page when necessary and send that data to a python script on my machine.
The python program executes, recieves the data, and does what I want.
I feel like this should be a simple task, but I can't find anything straightforward. From what I have been reading, I can make a Firefox extension which can do this, but the tutorials I have seen are all for things like adding extra features to the browser. I just want a minimal tutorial since all I need to do is just run my own javascript when visiting website "X" and then call a python script.