I wrote a fairly complex script using URLLIB and BeautifulSoup, and last night I wondered if there was any way to produce the same results as a web application.
I'm not asking for a tutorial, but can someone point me in the general direction of how/what proficiency's would be needed to write an application that would let someone input scraping criteria, and a URL, and output the correct source, all in a webpage?
For a basic one page web application, I'd recommend integrating your existing code into one of the available python web micro-frameworks. Try Flask to start; this framework is lightweight and seems ideal for your use-case (another options is bottle, and pyramid and django for larger apps). The tutorials for these frameworks should be enough to get you on the right track.
Related
I'm just curious for future reference, how exactly would I link a python program and how would the code be worded to a html document.
Thanks in advance
Here is a list of frameworks you can check out:
https://wiki.python.org/moin/WebFrameworks
I prefer Django but that's just a personal opinion:
https://www.djangoproject.com/
Frameworks make it easier than ever to build applications and websites. Learn python well, learn how the web works, follow a few tutorials (for example the django tutorial) and you are ready to start developing.
Here is how Django works with templates(html):
https://docs.djangoproject.com/en/1.11/intro/tutorial03/
You could use frameworks like django and flask.
i always prefer flask. its simple and light
I have 10 line python code and a third party python library that I want to use in a html website. I do not want to use a full fledged framework like django to complete the task. I want to use the following library https://github.com/nficano/pytube in my project. Thank You looking forward for some assistance.
The most simple thing come to my mind is following.
1) You'll need to use some sort of WSGI (https://uwsgi-docs.readthedocs.io/en/latest/, for example), to serve your script as a web service.
2) You'll also need a little bit of JavaScript code to make AJAX (for example) requests to that script.
I want to create a special wiki page on my local Redmine server. It should contain an inventory of some executables from my server. My goal is a script which scans certain folders on my server for these files and put them (with some additional information) in a nice Redmine wiki page.
My first thought was to traverse my server's file system with a simple batch file and to create a SQL expression for putting the results directly into the underlying mySQL database (which contains Redmine's wiki pages). But I consider this too risky and too error-prone.
Then I had the idea to use a script language like python (which I always wanted to learn) to retrieve the information and send it back to the Redmine server, like a web browser would do. This should be a much safer way. But this doesn't seems to be an easy beginner's task when just starting with python - I fail to authenticate myself on the Redmine server.
My last idea was to create a HTML page with python, which could be displayed within a Redmine wiki page with the plugin 'Redmine Wiki Extensions'. But I consider this only as a solution light, because it's not very elegant.
So what I seek is either a new idea to solve this problem or some clues on how to do a proper authentification with python on my Redmine server - maybe I could use a cookie for easier authentification...
I'm not familiar with redmine, but if you are looking for something like having a script that performs some actions the same way you would do in a browser, then mechanize is a library that might be helpful for you unless there's some javascript involved. In that case, then I'd look into something like windmill or selenium to let you drive the web browser.
However, please note using web scraping is also error-prone since any change in the design of the web pages involved might break your scripts.
Regarding the option of using an API as pointed out by the comment from AdamKG, that would be a good option, since there's a REST API that you can use from python if you like. Unfortunately, I don't see anything to let you do what you're looking for and it seems it hasn't yet reached the stable status yet. Anyway, as I said, it's still a good option to consider in the future.
This week, I want to start a web mapping and data visualization site for my work.
Unfortunately, I just found out my work place will be using Drupal in a few months down the road. (Most of my web development experience is with App Engine.)
My problem is that I need to make sure my web application embeds nicely into the larger Drupal site that outside consultants plan to make.
I am most comfy with Python, and I was expecting to use the Python-Django combo instead. There are important python libraries and modules I must have that cant be found or re-written in PHP.
I was thinking I will avoid all django on the web pages so things dont get confusing when the Drupal switch is made.
I will have the javascript on the web page make calls to python on the server which then spits out JSON data, and I think this will stay the same even after the Drupal switch.
Does this make sense?
Any general or specific suggestions that may guide me are greatly appreciated!
If you write API calls and utilize Drupal Services module, you can hook into just about anything and send/receive JSON/XML data.
I want to have a browser page that updates some information on a timer or events. I'd like to use Python on the server side. It's quite simple, I don't need anything massively complex.
I can spend some time figuring out how to do all this the "AJAX way", but I'm sure someone has written a nice Python library to do all the heavy lifting. If you have used such a library please let me know the details.
Note: I saw how-to-implement-a-minimal-server-for-ajax-in-python but I want a library to hide the implementation details.
AJAX stands for Asynchronous JavaScript and XML. You don't need any special library, other than the Javascript installed on the browser to do AJAX calls. The AJAX requests comes from the client side Javascript code, and goes to the server side which in your case would be handled in python.
You probably want to use the Django web framework.
Check out this tutorial on Django tips: A simple AJAX example.
Here is a simple client side tutorial on XmlHTTPRequest / AJAX
You can also write both the client and server side of the ajax code using python with pyjamas:
Here's an RPC style server and simple example:
http://www.machine-envy.com/blog/2006/12/10/howto-pyjamas-pylons-json/
Lots of people use it with Django, but as the above example shows it will work fine with Pylons, and can be used with TurboGears2 just as easily.
I'm generally in favor of learning enough javascript to do this kind of thing yourself, but if your problem fits what pygjamas can do, you'll get results from that very quickly and easily.
I suggest you to implement the server part in Django, which is in my opinion a fantastic toolkit. Through Django, you produce your XML responses (although I suggest you to use JSON, which is easier to handle on the web browser side).
Once you have something that generates your reply on server side, you have to code the javascript code that invokes it (through the asynchronous call), gets the result (in JSON) and uses it to do something clever on the DOM tree of the page. For this, you need a JavaScript library.
I did some experience with various javascript libraries for "Web 2.0". Scriptaculous is cool, and Dojo as well, but my absolute favourite is MochiKit, because they focus on a syntax which is very pythonic, so it will hide you quite well the differences between javascript and python.