I have created simple test cases using selenium web driver in python. I want to log the execution of the test cases at different levels. How do I do it? Thanks in advance.
I created library in python for logging info messages and screenshots in HTML file called selenium-logging
There is also video explanation of package on youtube (25s) called "Python HTML logging"
Related
I have a problem getting javascript content into HTML to use it for scripting. I used multiple methods as phantomjs or python QT library and they all get most of the content in nicely but the problem is that there are javascript buttons inside the page like this:
Pls see screenshot here
Now when I load this page from a script these buttons won't default to any value so I am getting back 0 for all SELL/NEUTRAL/BUY values below. Is there a way to set these values when you load the page from a script?
Example page with all the values is: https://www.tradingview.com/symbols/NEBLBTC/technicals/
Any help would be greatly appreciated.
If you are trying to achieve this with scrapy or with derivation of cURL or urrlib I am afraid that you can't do this. Python has another external packages such selenium that allow you to interact with the javascript of the page, but the problem with selenium is too slow, if you want something similar to scrapy you could check how the site works (as i can see it works through ajax or websockets) and fetch the info that you want through urllib, like you would do with an API.
Please let me know if you understand me or i misunderstood your question
I used seleneum which was perfect for this job, it is indeed slow but fits my purpose. I also used the seleneum firefox plugin to generate the python script as it was very challenging to find where exactly in the code as the button I had to press.
I am a python beginner and I need some help to create a web service that calls a python web scraping script (a task for a course).
I can use Bottle to create the web service. I wanted to use static_file to call the script but I am not sure about that because the documentation says that static_file is for CSS.
The idea is first to create the web service and later used the web scraping script from a server.
Thanks for your help and greetings from Colombia!
P.S. I don't have an excellent English but I hope someone can understand me and help me.
unless it's already in a function, edit your scraping script so your code is contained within a function that returns whatever information you want. it should be as easy as indenting everything unindented and adding a def main():
let's say your script is called scraper.py and it's located along with your bottle controllers, at the top of your controller file add a import scraper.
in your callback you can call scraper.main()
(not sure why pasting the code here is not formatting it's below)
Having that said, it's usually bad practice to have something long running like a scraping script in a request. You'd usually want to use a queue of scraping jobs where your controller posts work to do, your scraper subscibes to it and notifies it when it's done caching the results somewhere.
from bottle import route, run
import scraper
#route('/scrape')
def scrape():
return scraper.main()
you could try this guide I found:
http://docs.python-guide.org/en/latest/scenarios/scrape/
For the xpath stuff, I would suggest using Mozilla Firefox with the "Firebug" pluggin. It can generate xpaths for you which will help you write your script faster
I have written my scripts in Selenium Python. In the Jenkins, I have added Publish JUnit test results report(http://nose.readthedocs.org/en/latest/plugins/xunit.html). However, I want to add the screenshot in that report as well. I feel like my tests are not useful without screenshots. How can I achieve this? I also would like receive error screenshots as attachments in my email.
Can I achieve this just by using a Jenkins plugin? or I need to add something in my selenium script? I am not able to find anything useful in selenium python.
I went over the following question, but it does explain the full process
Adding a png image and table html while sending email in Jenkins
Please help.....
I have a task where in I need to submit a form to website but they dont provide any API. I am currently using webdriver and faced many problems because of asynchronous nature between my code and browser. I am looking for a light weight reliable library/tool using with I can do all the tasks a user do with a browser.
Casperjs is one of the option which can do my job but I am more familiar with python and scrapy has larger developer community compare to casperjs.
Navigation utility without browser, light weight and fail-proof is one of the related question.
in short answer is No.
scrapy can't render java script but browser can.
you can use Selenium.
if you are sure to use scrapy and there is javascript that you need to run you can use
scrapy with selenium
scrapy with gtk/webkit/jswebkit
scrapy with webdrivers
If you like CasperJS but want to stick with Python, you should have a look at Ghost.py
I'm not sure how to find this information, I have found a few tutorials so far about using Python with selenium but none have so much as touched on this.. I am able to run some basic test scripts through python that automate selenium but it just shows the browser window for a few seconds and then closes it.. I need to get the browser output into a string / variable (ideally) or at least save it to a file so that python can do other things on it (parse it, etc).. I would appreciate if anyone can point me towards resources on how to do this. Thanks
using Selenium Webdriver and Python, you would simply access the .page_source property to get the source of the current page.
for example, using Firefox() driver:
from selenium import webdriver
driver = webdriver.Firefox()
driver.get('http://www.example.com/')
print(driver.page_source)
driver.quit()
There's a Selenium.getHtmlSource() method in Java, most likely it is also available in Python. It returns the source of the current page as string, so you can do whatever you want with it
Ok, so here is how I ended up doing this, for anyone who needs this in the future..
You have to use firefox for this to work.
1) create a new firefox profile (not necessary but ideal so as to separate this from normal firefox usage), there is plenty of info on how to do this on google, it depends on your OS how you do this
2) get the firefox plugin: https://addons.mozilla.org/en-US/firefox/addon/2704/ (this automatically saves all pages for a given domain name), you need to configure this to save whichever domains you intend on auto-saving.
3) then just start the selenium server to use the profile you created (below is an example for linux)
cd /root/Downloads/selenium-remote-control-1.0.3/selenium-server-1.0.3
java -jar selenium-server.jar -firefoxProfileTemplate /path_to_your_firefox_profile/
Thats it, it will now save all the pages for a given domain name whenever selenium visits them, selenium does create a bunch of garbage pages too so you could just delete these via a simple regex parsing and its up to you, from there how to manipulate the saved pages