Selenium --headless option behaviour? - python

I`m using Python Selenium to navigate TripAdvisor.com
After running my script for a while, I get banned by IP (or at least it seems that way)
However, when I retry or even run a somewhat similar script from another machine in the same network, the script is still working properly and does not fail due to HTTP 403 or something like that.
I've managed to deduce that --headless option for Chrome webdriver has something to do with it. Without it, the script fails as it is supposed to, showing Access Denied as on the screenshot.
So, my question is, what does the webdriver in --headless mod do differently as opposed to not using it?
UPD: I have rephrased the question, as commentors pointed out, nobody has insight into server logic

Related

Python application using eel "unable to connect" to localhost on startup

I am trying to create a python application while using eel to create a user interface in html. My operating system is Ubuntu Linux and I'm using Firefox to display the web interface.
The problem I'm having is every time I run the python code, Firefox opens a blank page saying "Unable to connect" followed by "Firefox can't establish a connection to the server at localhost:8000". However, if I click the "Try Again" button once, twice, or three times, my interface is displayed.
Once open, I can navigate to different pages but I also noticed that once I navigate to a different page, some of my javascript stops working (specifically a window.close() function). I don't know if this is related but I thought I would mention it just in case.
Any advice on the matter would be greatly appreciated.
Thank you.
I changed my browser from firefox to chromium and now my interface loads on startup the first time. I know some documentation says it can be used with firefox, and it can, but it seems to be kind of buggy and works better with other browsers.
However, I'm still having trouble with my javascript not running but that will be another question.

How to close webdriver by different python script

I want to know is there a way to create a webdriver by script1.py , but close the webdriver by script2.py . I don't use the time.sleep() because I set these scripts to execute after few month. And I'm afraid that the scripts will delay due to the network crash. Any ideas are welcome.
Thanks!
The Python Selenium framework, SeleniumBase, comes with a special pytest command-line option called, --reuse-session, which tells all your tests to reuse the same browser session, even if all the tests live in different Python files. (More info on SeleniumBase command-line options here.) When using --reuse-session mode, the first test run will spin up the web browser, and then the last test run will close it. This should give you what you're looking for, where a different Python script closes the browser than the one that originally opened it.

Running chromedriver Through Django Selenium Testing

It seems that a lof of people are having trouble getting Selenium to find chromedriver, so this may apply to them aswell if they actually have the chromedriver.exe in the correct path.
It seems I have everything I need to have these selenium tests working, and when I manually try running the following 2 lines, everything works fine (it finds chromedriver and opens Chrome).
from selenium import webdriver
webdriver.Chrome()
However, when I put the exact same code into a Django test and try running the test through Django, I get a "ChromeDriver executable needs to be available in the path" error. I've tried re-installing Django and Selenium to no success.
Any help would be appreciated!
The problem was actually caused by enabling Celery tasks.

selenium headlessly run on server over SSH

I am now developing a webpage crawler, unfortunately the website generates the results by ajax. Following some coders suggestion, I tried to use selenium, a test automation tool for python.
As the example given in the documentation:
driver = webdriver.Firefox()
This code executes to open the Firefox browser. And then do something just like filling the form, submitting and so on.
Frankly speaking, this example works well on my PC(ubuntu 12.10), but my project will finally transfer to a CentOS server.
What I am considering is whether the code(need to open a browser gui) can be successfully ran on the CentOS server over ssh because no KDE such as gnome provided on that machine.....
And if without browser gui, the code cannot work well, then is there any other solutions?
Any reply would be admired~
You can probably use the HtmlUnit driver if you enable javascript. The only way to be sure though is to test it out. Another option would be to try and run with an X framebuffer.

Why doesn’t input.send_keys() work in my Selenium WebDriver Python script when run as www-data?

I have a Python script that uses Selenium WebDriver (with PyVirtualDisplay as the display) to log into Flickr.
http://pastebin.com/dqmf4Ecw (you’ll need to add your own Flickr credentials)
When I run it as myself on my Debian server, it works fine. (I’m a sudoer, but I don’t use sudo when running the script.)
When I run it as the user www-data (which is what it’ll be running as eventually, because I want to trigger it from a Django website), I get two problems, one small, one big:
(Small): the webdriver.Firefox() call takes 30–45 seconds to return, compared to 2 seconds when run as myself
(Big): the script fails to log into Flickr. In order to log in, I find the username and password fields on the Flickr signin page (http://www.flickr.com/signin/), and use element.send_keys() to enter the username and password. Although Selenium seems to find the elements (i.e. no NoSuchElementException is thrown), the values do not get entered in the fields when the script is run as www-data (according to the screenshots I take using browser.save_screenshot), unlike when the script is run as myself.
Why does send_keys() not work when the script is run as www-data? (And is it related to the browser taking much longer to start?)
Maybe you have something different in your environment.
Try copy by example your ~/.bashrc in /home/www-data
If it's not sufficient, run this command both as your current user & as www-data:
strace -tt -f -s 1000 -o /tmp/trace ./script.py
And paste it (filter out your logins/passwords) somewhere.
We will see what's happens.
Sometimes, Firefox performs some nasty plugin compatibility check during startup. As each user can have a different set of browser plugins, this could be responsible for the difference in startup times. You could try to sync your Firefox profiles between users.
Then, are you sure that Firefox as user www-data has proper network/internet access? Can you confirm that the Flickr site loads properly via SeleniumHQ? "The script fails to log into Flickr" is too unprecise. Some more details about why it fails might reveal the problem instantaneously.
Edit: Sorry, I just understood that there shouldn't be a difference in profiles, because Selenium creates one. Nevertheless, my second point might be useful, so I won't delete this answer.
Some more things to ponder about:
Could you spawn firefox manually from www-data account once and make sure that Firefox is not updating itself before every execution of the script? I once faced this problem with Selenium RC on Windows and had to let the update finish before starting the script with the updated binary.
As a workaround, I guess you could you try running the script as www-data user but connecting remotely to a webdriver server running in your login (aka "grid" mode). Would that work for you?
I would suggest getting the latest chrome from google and trying input.send_keys() in that browser instead.
Sometimes some features of webdriver get broken with new releases.. If you are bent on testing with firefox, you might have better luck with an older/newer version of selenium webdriver.
I remember having a similar issue regarding send_keys() on a mac.. My issue was that send_keys() did not work in certain modal windows after I updated selenium webdriver.. I fixed it by reverting to an older webdriver that I knew to work. However, I was using Ruby and not Python to drive webdriver.
sometimes, there might also be a problem with getting the correct ENV variables in your shell if you use it as a different user. I would suggest trying to troubleshoot and see if all the shell ENV variables are set properly under www-data.

Categories