I am curious if there might be a way to dynamically alter source from a web page automatically.
For instance, I know the firebug plugin for Firefox allows the capability to modify the source and see the reaction in real-time. So, say I want to login to a particular form. Could I alter this dynamic source with a login name and password and enter the website in question via some automated script? If not, are there any potential alternatives to this approach that may fair better?
Thanks.
If you want something which can automate IE browser, what I can recommend to you is: Watir and WatiN. Watir is developed in ruby while WatiN is developed in c#. They are both quite powerful, more than enough to meet your requirements.
If you have to use Python script. then I would recommend C# + WatiN + IronPython. You can write python scripts to call WatiN's dll. Please note that, IronPython is not the same as Python, it is based on microsoft's .net framework. Currently I don't know any pure python product which can do the same as WatiN and Watir.
If you want to login to a website automatically you don't need to edit the source, you need to interact with the webserver. Try curl and use it to submit login details and fetch the resulting web page.
for firefox automation, I recommend chickenfoot to you. It can meet your needs:
alter this dynamic source with a login
name and password and enter the
website in question via some automated
script
But chechenfoot only supports up to firefox 3. If you want to support the newest version of firefox, you might have to get the source code and compile it yourself.
If you can use Javascript, try:
document.write("HTML CODE HERE");
But if you need in Python, I think you can use REPLACE in the HTML Source Code.
Related
I am working on application where I am giving a functionality to user where they can signin from dropbox by using my application but when I am trying to get the file from Dropbox I am unable to do it. When user choose the desired file from dropbox and click on choose and then nothing happen. Can anyone please help out either is it possible to do it or not? If yes, how can we do it? I am at beginner level. Please help me out and explain the whole process to do it in detail.
Are you using public Dropbox files?
Then you just need to fetch the item by URL and download it. If this happens in a browser tool, you'll need JavaScript and not Python to download it.
Or you leave out JS and just use Python to render an HTML page where a button is for a Dropbox file and clicking the button triggers a download of the file. That is a generic HTML task you can search for.
If you need access to sign in Dropbox and view private files, consider using a Python library built around Dropbox.
See the Python Dropbox guide. Are you using a library like that? Please share as your question was vague.
https://www.dropbox.com/developers/documentation/python#
Also please share an explanation of what your logic is or a small code snippet. I can't see what you are doing yet so I don't know where you are missing something or making a mistake.
From the screenshot, I see you're using the Dropbox Chooser. That's a pre-built way to let your end-users select files from their Dropbox accounts and give them to your app.
Make sure you implement the success and cancel callback methods as documented there for the Chooser.
In the success callback, you'll get the information for the selected file(s). That occurs in JavaScript in the browser though, so if you need that on your server, you'll need to write some JavaScript to send that up to your server, e.g., via an AJAX call or a form or whatever means you use in your app.
I have a requirement where i have to use my python project into WordPress web page, where i ask user for input in one tab than execute it and gives back the result on display on the web-page.
I found one plugin for the same namely: "Create With Code".
It worked pretty well, only challenge was to expose my code into third party website which i can not do.
Now i have started to create my own custom plugin to do the same.
Could anyone suggest whether i am on a right track or is there any other secure straightforward way to achieve that.
Can I have any highlight kind of things using Python 2.7? Say when my script clicking on the submit button,feeding data into the text field or selecting values from the drop-down field, just to highlight on that element to make sure to the script runner that his/her script doing what he/she wants.
EDIT
I am using selenium-webdriver with python to automate some web based work on a third party application.
Thanks
This is something you need to do with javascript, not python.
[NOTE: I'm leaving this answer for historical purposes but readers should note that the original question has changed from concerning itself with Python to concerning itself with Selenium]
Assuming you're talking about a browser based application being served from a Python back-end server (and it's just a guess since there's no information in your post):
If you are constructing a response in your Python back-end, wrap the stuff that you want to highlight in a <span> tag and set a class on the span tag. Then, in your CSS define that class with whatever highlighting properties you want to use.
However, if you want to accomplish this highlighting in an already-loaded browser page without generating new HTML on the back end and returning that to the browser, then Python (on the server) has no knowledge of or ability to affect the web page in browser. You must accomplish this using Javascript or a Javascript library or framework in the browser.
I want to create a special wiki page on my local Redmine server. It should contain an inventory of some executables from my server. My goal is a script which scans certain folders on my server for these files and put them (with some additional information) in a nice Redmine wiki page.
My first thought was to traverse my server's file system with a simple batch file and to create a SQL expression for putting the results directly into the underlying mySQL database (which contains Redmine's wiki pages). But I consider this too risky and too error-prone.
Then I had the idea to use a script language like python (which I always wanted to learn) to retrieve the information and send it back to the Redmine server, like a web browser would do. This should be a much safer way. But this doesn't seems to be an easy beginner's task when just starting with python - I fail to authenticate myself on the Redmine server.
My last idea was to create a HTML page with python, which could be displayed within a Redmine wiki page with the plugin 'Redmine Wiki Extensions'. But I consider this only as a solution light, because it's not very elegant.
So what I seek is either a new idea to solve this problem or some clues on how to do a proper authentification with python on my Redmine server - maybe I could use a cookie for easier authentification...
I'm not familiar with redmine, but if you are looking for something like having a script that performs some actions the same way you would do in a browser, then mechanize is a library that might be helpful for you unless there's some javascript involved. In that case, then I'd look into something like windmill or selenium to let you drive the web browser.
However, please note using web scraping is also error-prone since any change in the design of the web pages involved might break your scripts.
Regarding the option of using an API as pointed out by the comment from AdamKG, that would be a good option, since there's a REST API that you can use from python if you like. Unfortunately, I don't see anything to let you do what you're looking for and it seems it hasn't yet reached the stable status yet. Anyway, as I said, it's still a good option to consider in the future.
I am trying to write a Python-based Web Bot that can read and interpret an HTML page, then execute an onClick function and receive the resulting new HTML page. I can already read the HTML page and I can determine the functions to be called by the onClick command, but I have no idea how to execute those functions or how to receive the resulting HTML code.
Any ideas?
The only tool in Python for Javascript, that I am aware of is python-spidermonkey. I have never used it though.
With Jython you could (ab-)use HttpUnit.
Edit: forgot that you can use Scrapy. It supports Javascript through Spidermonkey, and you can even use Firefox for crawling the web.
Edit 2: Recently, I find myself using browser automation more and more for such tasks thanks to some excellent libraries. QtWebKit offers full access to a WebKit browser, which can be used in Python thanks to language bindings (PySide or PyQt). There seem to be similar libraries and bindings for Gtk+ which I haven't tried. Selenium WebDriver API also works great and has an active community.
Well obviously python won't interpret the JS for you (though there may be modules out there that can). I suppose you need to convert the JS instructions to equivalent transformations in Python.
I suppose ElementTree or BeautifulSoup would be good starting points to interpret the HTML structure.
To execute JavaScript, you need to do much of what a full web browser does, except for the rendering. In particular, you need a JavaScript interpreter, in addition to the Python interpreter.
One starting point might be python-spidermonkey. Depending on the specific JavaScript, you might have to provide a good DOM API to the spidermonkey, in addition to providing an XmlHttpRequest implementation.
You can try to leverage V8,
V8 is Google's open source, high performance JavaScript engine. It is written in C++ and is used in Google Chrome, Google's open source browser.
Calling it from Python may not be straightforward, without a framework to provide the DOM.
Pyjamas has an experimental project, Pyjamas Desktop, providing V8 integration for Javascript execution.
Pyv8 is an experimental python v8 bindings and a python-javascript compiler.
For the browser part of this you might want to look into Mechanize, which basically is a webbrowser implemented as a Python library. http://pypi.python.org/pypi/mechanize/0.1.11
But as mentioned, the text n onClick is Javascript, and you'll need spidermonkey for that.
If you can make a generic support for spidermonkey in mechanize, I'm sure many people would be extremely happy. ;)
Mechanize may be overkill, maybe you just want to find specific parts of the HTML, and then lxml and BeautifulSoup both work well.
Why don't you just sniff what gets sent after the onclick event and replicate that with your bot?
For web automation , you can look into "webbot" library. It makes autmation damn simple and pain free.
webbot works even for webpages with dynamically changing id and classnames and has more methods and features than selenium and mechanize.
Here's a snippet of webbot
from webbot import Browser
web = Browser()
web.go_to('google.com')
web.click('Sign in')
web.type('mymail#gmail.com' , into='Email')
web.click('NEXT' , tag='span')
web.type('mypassword' , into='Password' , id='passwordFieldId') # specific selection
web.click('NEXT' , tag='span') # you are logged in ^_^
Docs are at : https://webbot.readthedocs.io