when using selenium chromedriver in headless i need to initally log in into the website im trying to scrape so that the browserdata gets stored via
option.add_argument(r'--user-data-dir=.\UserData').
unfortunately the cookies dont work for headless if i put in the information via non headless.
the easiest way arround that for me was to let it display via option.add_argument('--remote-debugging-port=9222') while in headless.
this worked on my previous pc like a charm however on my new, current one it just displays a white screen of death on the localhost.
python: 3.7.9
i pasted the code from my previous pc where it worked perfectly fine so its not a code side error
Related
I am automating a process using Selenium Firefox and Python. I am running Selenium on a Google Virtual Instance and accessing using RDP. All went fine when I ran it through RDP, but the problem is when I disconnect from RDP, Selenium doesn't work.
The part of selenium that doesn't work is :
WebDriverWait(driver, 15).until(ec.element_to_be_clickable((By. CSS_SELECTOR,'div.van-key__wrapper:nth-child(9) div:nth-child(1)'))).click()
I have tested everything all over the internet like gpedit regedit everything and also not tell to put bat file or command to stay connected to local user like chromedriver (selenium).
I am running selenium webdriver (firefox) using python on a headless server. I am using pyvirtualdisplay to start and stop the Xvnc display to grab the image of the sites I am visiting. This is working great except flash content is not loading on the pages (I can tell because I am taking screenshots of the pages and I just see empty space where flash content should be on the screenshots).
When I run the same program on my local unix machine, the flash content loads just fine. I have installed flash on my server, and have libflashplayer.so in /usr/lib/mozilla/plugins. The only difference seems to be that I am using the Xvnc display on the server (unless plash wasn't installed properly? but I believe it was since I used to get a message asking me to install flash when I viewed a site that had flash content but since installing flash I dont get that message anymore).
Does anyone have any ideas or experience with this- is there a trick to getting flash to load using a firefox webdriver on a headless server? Thanks
It turns out, I needed to use selenium to scroll down the page to load all the content.
I am using selenium webdriver (Python version) to take images of a web page after it loads. When I go to a page and save an image using a python script on my local computer, it looks fine. However, I am running the script on a server and there the screenshots are mangled- the edge might be cut off with text missing, banners on the right side might be pushed to the bottom in a jumbled fashion, etc. I even tried maximizing the window
driver.get(url)
driver.maximize_window()
time.sleep(4)
driver.save_screenshot('screen.png')
On the server, I cannot load firefox in a headed version and must manually start/stop the display in my script before/after running selenium
from pyvirtualdisplay.xvnc import XvncDisplay
display = XvncDisplay(rfbport='####')
display.start()
So I'm thinking this might have to do with the settings of my display.
Anyone have any ideas on how to fix this? Thanks
Try chomium + chromedriver instead. It works for me, even though I didn't have the problem you describe. Just an idea.
Download chromedriver to /usr/local/bin https://sites.google.com/a/chromium.org/chromedriver/downloads and don't forget chmod a+x /usr/local/bin/chromedriver
I used this blog post: http://coreygoldberg.blogspot.cz/2011/07/python-taking-browser-screenshots-with.html
I am now developing a webpage crawler, unfortunately the website generates the results by ajax. Following some coders suggestion, I tried to use selenium, a test automation tool for python.
As the example given in the documentation:
driver = webdriver.Firefox()
This code executes to open the Firefox browser. And then do something just like filling the form, submitting and so on.
Frankly speaking, this example works well on my PC(ubuntu 12.10), but my project will finally transfer to a CentOS server.
What I am considering is whether the code(need to open a browser gui) can be successfully ran on the CentOS server over ssh because no KDE such as gnome provided on that machine.....
And if without browser gui, the code cannot work well, then is there any other solutions?
Any reply would be admired~
You can probably use the HtmlUnit driver if you enable javascript. The only way to be sure though is to test it out. Another option would be to try and run with an X framebuffer.
I'm trying to use selenium from python but I'm having a problem running it on a RHEL5.5 server. I don't seem to be able to really start firefox.
from selenium import webdriver
b = webdriver.Firefox()
On my laptop with ubuntu this works fine and it starts a brings up a firefox window. When I log in to the server with ssh I can run firefox from the command line and get it displayed on my laptop. It is clearly firefox from the server since it has the RHEL5.5 home page.
When I run the python script above on the server it (or run it in ipython) the script hangs at webdriver.Firefox()
I have also tried
from selenium import webdriver
fb = webdriver.FirefoxProfile()
fb.native_events_enabled=True
b=webdriver.Firefox(fb)
Which also hangs on the final line there.
I'm using python2.7 installed in /opt/python2.7. In installed selenium with /opt/python2.7/pip-2.7.
I can see the firefox process on the server with top and it is using a lot of CPU. I can also see from /proc/#/environ that the DISPLAY is set to localhost:10.0 which seems right.
How can I get a browser started with selenium on RHEL5.5? How can I figure out why Firefox is not starting?
It looks like the problem I'm encountering is this selenium bug:
http://code.google.com/p/selenium/issues/detail?id=2852
I used the fix described in comment #9 http://code.google.com/p/selenium/issues/detail?id=2852#c9
That worked for me.