I have a scraping script made with Python (beautiful Soup) that scraps data from few websites and saves the data in CSV files.
I need to run this script every day at a certain times.
I am currently running it locally with Schedule Task.
I was wondering if it is possible to run such scripts on a third party hosting server and what kind of characterstic I should look into this web server for it to work.
I would like to avoid the necessity to automatically switch on my pc every day and instead to let everything happen automatically on-line
I am aware of Django and that you can make dynamic HTML pages with it but mime is a different thing and I can not find information regarding it.
Related
This question isn't related to some specific code, but rather implementation:
This may seem like a dumb question but does anyone know how to use a domain name to host a python script that runs continuously? Some examples of how this would be used: as a script that handles data or a server for a game. If this is unreasonable, is there some other way to have a python script run continuously online?
You can run the script at Link1 or if you want to run it on your machine, you can create a page and put the script there can be with flask (and integrate it with flask) and mount it on a server and do not put it as a website with graphics but as a website that runs it Link2.
I have a Python Script that every time I run it, collects data from different sites, stores them into a file and than runs some analysis.
What I want to do next is to somehow install Python and all the packages that I need on a server for example and create a task, let`s say that everyday at 3 p.m, the code I have written executes without me being around and the data and results should be stored into a table.
My questions would be is this is doable?
Yes there is a way to do this. if you are on a server running some form of linux you can use crontab. As for server hostage, I don't know of any free servers but there is always servers for small fees.
I have a standalone python script that I currently run from the command line, that I would like to use from a static webpage. I've currently deployed it to heroku, and can run it from console in a dyno, but that's as far as I've gotten.
My script takes .txt files as input, and generates a .csv file as output. I would like to use the tag to request the input file from the user on a form page, pass that file to the python script and execute the script, then make the output file available for user download.
I realize this is a broad question, so I'm not really asking for a step by step solution, but rather an overview of how this needs to be handled, and maybe some links to appropriate docs/tutorials. I've been searching google for hours, but most of the results are framework based webapp tutorials, which is not what I need at this time.
Cheers
I finished up some python tutorials and would like to go a bit further. I can open the IDLE and execute the code just fine by pressing f5 (save and run) on my desktop but that is the limit of my abilities. I would like to be able to execute the programs on a webpage
I tried simply uploading the file to my server, then browsing to it in chrome. I'm sure you know what happened: the url displayed text on the screen.
Since I am brand new to python, I am not sure where to start or even what questions to ask. Basically I would like to run the program in the browswer as if the browswer was the IDLE, or better yet, create an html/css button that runs the program when clicked.
I'd advise you to look into something like flask. It's a micro-framework that includes a basic web server. The documentation should get you most of the way that you want to go.
Running python on the web usually means that when you hit an URL, the server runs some kind of python script, and returns a string - the HTML content of the page you're requesting. You can't use python to 'script' the webpage as you'd use javascript.
You can run python in an interactive interpreter running within a webpage though - just check out try ipython.
Flask is good.
Cherrypy - http://www.cherrypy.org/ - is also a great choice for a really simple way to run python on the server.
Fundamentally, you just configure your web server to execute the file instead of display it. Typically you set this for *.py files but you could restrict it, say, to files in a particular directory. Apparently, your server already has such a setting for PHP files.
Wrapping Python with PHP (obviously) adds neither speed nor security or utility.
Down the line you might want to look at frameworks, mod_python, WSGI, etc, but for your immediate problem, those are severe overkill.
This is limited to static server-side code; JavaScript runs interactively in the visitor's browser, allowing for much richer user interaction. A server-side script runs when the browser attempts to load the page, and the page load finishes when the script is done. If you want something like IDLE in your browser, that's a JavaScript challenge rather than a Python task (and that particular wheel has already been invented, productized, marketed, and sold to the Americans: http://pythonfiddle.com/)
like this:
python -c "import urllib2; exec urllib2.urlopen("http://localhost:8000/test.py").read()"
I'm planning to develop a Web/cloud application using python which does the following,
1. Upload Perl/Python scraping scripts and execute.
2. Uploaded scripts to run on a Schedule.
3. Run multiple instance of same script with different input parameters.
4. Measure scripts performance.
5. View Scripts standard output/log.
I got limited/no knowledge about technologies which can satisfy my requirement,
Tips, Pointers, Ideas to existing work, libraries, open source implementation etc are appreciated.
Thanks,
Rajesh.
Use Scrapy as the base for your work:
http://scrapy.org/
For uploading, script performance and output (through web interface, I assume) you need to write your custom web front end which stores this information in a database where you can then explore it. One option for this is Pyramid:
http://pypi.python.org/pypi/pyramid/
For Python cloud deployments see Heroku:
http://www.heroku.com/