I want to cache a webpage as an image upon a user request, but I don't know where to start with this.
I'm developing on App Engine with python.
Here's a good library for capturing a webpage as a png image:
http://github.com/AdamN/python-webkit2png
One way is to use a web service such as thumbalizr since a lot of the programs for this type of thing aren't always install-able on appengine (because they use C++, etc). Other options include girafa and browsershots.
There are websites that to this for you. Google is your friend. If you build a script around them, you have what you need. As a demonstration, see http://webshots.velocitysc.com/sandbox/.
There are also downloadable programs that do it, such as the one at http://download.cnet.com/Advanced-Website-to-Image-JPG-BMP-Converter-Free/3000-2094_4-10900902.html. These are just examples; google a while and you'll find better implementations.
If you want to do it yourself, you basically need to duplicate a web browser (the HTML rendering part, anyway), which is unrealistic, or use a preexisting rendering engine like webkit as Zach suggests. If I were you, I would forget about doing it myself and use a preexisting web service, unless this is going to be the core of your application.
Related
I have written an application in python to collect data from a javascript form and returned the processed text. It is based entirely off of the code here (but a lot more complex, so I have to use python for this).
https://kooneiform.wordpress.com/2010/02/28/python-and-ajax-for-beginners-with-webpy-and-jquery/
(note to people who like to edit...please leave this link in place since it shows all the relevant code sections in python and javascript).
I need to use this in wordpress (since that's what runs my site) and I honestly have no idea how to pull this off. Webpy can run with Apache CGI, but the documentation (http://webpy.org/cookbook/cgi-apache) is only clear if one wants to navigate directly to the python app as its own page.
I'm hoping someone here has expertise in how to embed this all within a Wordpress page/post?
Thanks!!
As far as I know, there is no native way to run Python code inside a WordPress site just like php. In fact, if you are not doing anything unique to Python, I would suggest you to use php, which supports regular expression and can be used in WordPress by installing the plugin "Insert PHP".
If you really want to use Python, then you need an API endpoint where you connect the function to your website. You would have to look into Azure Function App/AWS lambda on which you write a function app to work as a backend. Then whenever someone request your website, your website would do an HTTP request to that API.
Can you explain what exactly you want to do on your website?
I have 10 line python code and a third party python library that I want to use in a html website. I do not want to use a full fledged framework like django to complete the task. I want to use the following library https://github.com/nficano/pytube in my project. Thank You looking forward for some assistance.
The most simple thing come to my mind is following.
1) You'll need to use some sort of WSGI (https://uwsgi-docs.readthedocs.io/en/latest/, for example), to serve your script as a web service.
2) You'll also need a little bit of JavaScript code to make AJAX (for example) requests to that script.
I've been working on a small project which requires me to create a proxy with access to only a few sites.
Am using the code from here: https://github.com/labnol/google-proxy
Now, basically am a PHP guy, but haven't found anything as better as the above for setting up a web proxy server.
What I need here is:
A way to filter out URLs. Like I want people to access only the sites I allow.
In the allowed sites, I'd like to block certain scripts. e.g. On wikipedia.org, the person shouldn't be allowed to login.
Am a complete noob in Python. Can anyone suggest me something here, or provide a code snippet which I can use?
Thanks! :)
P.S.: For a PHP version, I've tried using Glype and miniProxy. But not as good as the one I mentioned. They break the CSS/JS of the websites.
You may install Squid and write your own filtering rules.
I want to create a special wiki page on my local Redmine server. It should contain an inventory of some executables from my server. My goal is a script which scans certain folders on my server for these files and put them (with some additional information) in a nice Redmine wiki page.
My first thought was to traverse my server's file system with a simple batch file and to create a SQL expression for putting the results directly into the underlying mySQL database (which contains Redmine's wiki pages). But I consider this too risky and too error-prone.
Then I had the idea to use a script language like python (which I always wanted to learn) to retrieve the information and send it back to the Redmine server, like a web browser would do. This should be a much safer way. But this doesn't seems to be an easy beginner's task when just starting with python - I fail to authenticate myself on the Redmine server.
My last idea was to create a HTML page with python, which could be displayed within a Redmine wiki page with the plugin 'Redmine Wiki Extensions'. But I consider this only as a solution light, because it's not very elegant.
So what I seek is either a new idea to solve this problem or some clues on how to do a proper authentification with python on my Redmine server - maybe I could use a cookie for easier authentification...
I'm not familiar with redmine, but if you are looking for something like having a script that performs some actions the same way you would do in a browser, then mechanize is a library that might be helpful for you unless there's some javascript involved. In that case, then I'd look into something like windmill or selenium to let you drive the web browser.
However, please note using web scraping is also error-prone since any change in the design of the web pages involved might break your scripts.
Regarding the option of using an API as pointed out by the comment from AdamKG, that would be a good option, since there's a REST API that you can use from python if you like. Unfortunately, I don't see anything to let you do what you're looking for and it seems it hasn't yet reached the stable status yet. Anyway, as I said, it's still a good option to consider in the future.
I need to create a simple web service, and am leaning towards the Python package soaplib.
I would also like to create a simple web page that can consume the web service. This will serve two purposes:
Allow for easier testing by multiple people including non-programmers.
Allow knowledgeable / power users to call the service directly in Production on a few occasions.
Any suggestions for creating this web page? Ideally, I want to generate it automatically and not craft it by hand. I have used this type of feature with Visual Studio .NET which autocreates a basic web page as part of the process of creating a web service.
Any ideas are appreciated. I'd prefer an automated solution based on soaplib but am open to any non-soaplib or non-Python solution as well.
Have you considered using REST (here or here)? I was creating small app using GAE and I wanted to have programmable interface for that too. So using exactly the same URLs I've created a web service (REST) which based on client accepted mime type answered either using json (for software) or html (for users). You have wide range of possibilities and it's lot cleaner that soap-based ws.