i use the command
browser.get_screenshot_as_file('google2.png')
to take pictures of my headless chrome in ubuntu server.
But the pictures are from the hole page without including the console with the errors. Problem is i am trying to connect linkedin using webscraping knowledge, but it is giving me an error. So i want to see if this error appears in the console, in order to solve it.
If you import ActionChains As well as Keys, you should be able to press F12 using the following:
actions = ActionChains(browser)
actions.send_keys(Keys.F12).perform()
Let me know how that works for you. Action change can be flaky sometimes but there are a couple of other options we could try if this doesn’t work.
Related
I want to be able to use pure selenium webdriver to open a zoom link in Chrome and then redirect me to the zoom.us application.
When I execute this:
from selenium import webdriver
def main():
driver = webdriver.Chrome()
driver.get("https://zoom.us/j/000-000-000")
main()
I receive a pop-up saying
https://zoom.us wants to open this application.
and I must press a button titled open zoom.us to open the app.
Is there a way to press this pop-up button through selenium. Or, is there some other way to open zoom from chromedriver?
NOTE: I only want to use selenium. I have been able to implement pyautogui to click on the button but that is not what I am looking for.
Solution for Java:
driver.switchTo().alert().accept();
Solution for Python:
driver.switch_to.alert.accept()
There are a lot of duplicated questions regarding this issue. Here is one of them, and it is quite sure that selenium is not capable of achieving such job since it only interacts with the chrome page. I previously encountered this issue as well and here is my solution to it. It might look really unprofessional, but fortunately it works.
The logic of my solution is to change the setting of chrome in order to skip the popup and directly open the application you want. However, the Chrome team has removed this feature in the latter version for some reasons, and we need to get it back manually. Then, we know that everytime when selenium starts to do the thing it opens a new Chrome page with NO customized settings just like the incognito page. Therefore we need to do something to let selenium opened a Chrome page with your customized setting, so that we can make sure that the popup, which we changed manually to skip, can be skipped successfully.
Type the following code in your terminal.
defaults write com.google.Chrome ExternalProtocolDialogShowAlwaysOpenCheckbox -bool true
This enables you to change the setting of skipping popups, which is the feature Chrome team removed.
Restart Chrome,and open the zoom (or whatever application) page to let the popup display. If you do the 1st step correctly you will be able to see there is a checkbox shown next to the "Open Zoom.us" saying if you check it chrome will open this application without asking, that is, to skip the popup for this application.
Now we need to let selenium open the Chrome with our customized setting. To do this, type "chrome://version" in the search tab of your ordinary Chrome (Not automated page opened by selenium). Go to "Profile Path", and copy this path without the last word "default". For example:
/Users/MYNAME/Library/Application Support/Google/Chrome/Default
This is my profile path, but I only copy everything except the last word Default, so this is what I need to copy.
/Users/MYNAME/Library/Application Support/Google/Chrome/
This is for Mac users, but for Windows only the path is different(starts with C:// or something), steps are same.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
option = Options()
option.add_argument('THE PATH YOU JUST COPIED')
driver = webdriver.Chrome(executable_path='YOUR PATH TO CHROMEDRIVER', options=option)
driver.get("google.com") #Or anything else
We use "options" to let selenium open a page with our customized profile. Now you will see selenium opens a Chrome page with all your account profile, settings, and it just appears like your ordinary chrome page.
Run your code. But before that, remember to quit ALL CHROME sessions manually. For Mac, make sure that there is no dot under Chrome icon indicating that Chrome is not running for any circumstances. THIS STEP IS CRITICAL otherwise selenium will open a chrome page and it just stops there.
Here are all the steps. Again, this solution is vert informal and I personally don't think it is a "solution" to this problem. I will try to figure out a better way of achieving this in the future. But I still posted this as an alternative simply because I guess it might be helpful to some extent for somebody just like me. Hope it works for you, and good luck.
I'm trying to run a Python script using Selenium, and while everything runs fine, my personal print() to console lines get hidden between a ton of Selenium/Chromedriver outputs like this:
1128/150256.806:INFO:CONSOLE(0)] "The SSL certificate used to load resources from [some-link.com] will be distrusted in the future. Once distrusted, users will be prevented from loading these resources. See https://g.co/chrome/symantecpkicerts for more information.", source: [current-page.com] (0)
I checked what these links are, and they're just the ads on the pages I'm looking at, so it's completely useless. Also, since the ads are randomly generated every time a page loads/is reloaded, the links are different so the outputs are never-ending. This is incredibly annoying and makes it very hard to see what is actually happening within my program. Is there any way to turn this off with some Selenium options or something?
The strange thing is, running the program in Eclipse Oxygen with PyDev doesn't show any of Selenium's output at all, only if I run it using command line.
EDIT: following the instructions from the possible duplicate mentioned didn't help. I tried setting the logging level to the highest, CRITICAL, and the output mentioned above still went through and flooded the console.
The best Way to solve this problem is to add the --log-level option to your Driver. That would look something like this:
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument("--log-level=3")
driver = webdriver.Chrome(chrome_options=chrome_options)
Whoever uses the ChromeDriverManager via the webdriver_manager package can use the following to disable the [WDM] log messages:
import os
os.environ["WDM_LOG_LEVEL"] = str(logging.WARNING)
Please note, this question is Python 3.5.2, only Python answers will be accepted. Unless this can definitely be handled in Java? Automating a process as part of an internal project. Everything works just fine using the IE webdriver, but not phantomJS web driver (which is expected due to limited functionality). However, a work-around / solution is required.
When opening the internal site, a Windows Security login dialog box comes up prompting for a username, password and press 'Ok'. With the IE web driver, it is handled just fine with:
loginAlert = driver.switch_to_alert()
loginAlert.authenticate(username, password)
The javascript:
driver.execute_script("window.confirm = function(){return true;}")
Being run before loading the page that gives the prompt, doesn't seem to confirm the login alert, for either phantom or IE. Even if it did, this doesn't type in the login details. As mentioned, it's a Windows Security prompt from the browser, not an element.
Once logged in, the page is reloaded with an ASP.NET_SessionId Cookie which expires once the session is ended. I've tried logging in through IE, then adding the cookie into Phantom, but it doesn't seem to match up the domains.
I've tried using:
driver.save_screenshot(filename) to see what's happening in phantom
Which works with IE driver, but with PhantomJS, only a transparent image is saved. The whole http://username:pass#site.com thing doesn't work for either IE or phantom driver. It can't load / use the URL when this is done.
How can the Windows Security login dialog be handled, or worked around? I tried looking into alternatives, such as pyvirtualdisplay, but found no information on how to get this working with Python 3 on windows.
I have also tried setting phantomjs desired capabilities custom header authentication, but that doesn't seem to do anything for this either.
I have also tried using ActionChains, however they don't work when the Alert is there (in either IE or phantom driver). An UnexpectedAlertPresentException is thrown, even if this is caught and you try to perform the actions, once caught, the alert seems to close.
My bad!
Whilst the username:pass#domain.com didn't work in the IE webdriver - it did work in the PhantomJS web driver.
However, the website has limited browser compatibility - it doesn't load properly in either Chrome or Firefox, it is IE particular.
PhantomJS seems to handle the site the same way as Chrome / Firefox based on page source comparisons.
As such, I am trying to find a way to make the current IE driver invisible / hidden.
I have found:
headless-selenium-for-win using Python
However, despite the user here saying they got it to work, when I try to initialize the driver, it just hangs, the code doesn't proceed and no error messages are provided.
Asking another question regarding this.
I have written a Python code to open my gmail account. Here is the code that I am using:
from selenium import webdriver
browser = webdriver.Firefox()
browser.get('https://www.gmail.com')
emailElem = browser.find_element_by_id('email')
emailElem.send_keys(myemail)
emailElem = browser.find_element_by_id('password')
emailElem.send_keys(mypassword)
emailElem = browser.find_element_by_id('signInSubmit')
emailElem.submit()
Everything is working fine. I have also found out that there are sites that lets one Log In only after entering a Captcha, to prevent scripts from logging in.
Is there a way in which I can use my above code get around this problem??
Experimentation. If the site is not showing a captcha to normal users you'll have to mimic being a human with your code. So that could mean that you use time.sleep(x) to make it seem like it takes a while before certain actions happen.
Otherwise there are services out there that solve captchas for you.
If you perform the same actions repetitively, gmail(or any other site which tries to block automation) will identify your actions as automated ones. To get around this you need to pass random sleep time in your script. Also, switching between multiple credential helps.
For that you must used some Captcha resolver API. Here I will provide you website which provide text code of captcha https://2captcha.com/
Alright, I'm confused. So I want to scrape a page using Selenium Webdriver and Python. I've recorded a test case in the Selenium IDE. It has stuff like
Command Taget
click link=14
But I don't see how to run that in Python. The desirable end result is that I have the source of the final page.
Is there a run_test_case command? Or do I have to write individual command lines? I'm rather missing the link between the test case and the actual automation. Every site tells me how to load the initial page and how to get stuff from that page, but how do I enter values and click on stuff and get the source?
I've seen:
submitButton=driver.find_element_by_xpath("....")
submitButton.click()
Ok. And enter values? And get the source once I've submitted a page? I'm sorry that this is so general, but I really have looked around and haven't found a good tutorial that actually shows me how to do what I thought was the whole point of Selenium Webdriver.
I've never used the IDE. I just write my tests or site automation by hand.
from selenium import webdriver
browser = webdriver.Firefox()
browser.get("http://www.google.com")
print browser.page_source
You could put that in a script and just do python wd_script.py or you could open up a Python shell and type it in by hand, watch the browser open up, watch it get driven by each line. For this to work you will obviously need Firefox installed as well. Not all versions of Firefox work with all versions of Selenium. The current latest versions of each (Firefox 19, Selenium 2.31) do though.
An example showing logging into a form might look like this:
username_field = browser.find_element_by_css_selector("input[type=text]")
username_field.send_keys("my_username")
password_field = browser.find_element_by_css_selector("input[type=password]")
password_field.send_keys("sekretz")
browser.find_element_by_css_selector("input[type=submit]").click()
print browser.page_source
This kind of stuff is much easier to write if you know css well. Weird errors can be caused by trying to find elements that are being generated in JavaScript. You might be looking for them before they exist for instance. It's easy enough to tell if this is the case by putting in a time.sleep for a little while and seeing if that fixes the problem. More elegantly you can abstract some kind of general wait for element function.
If you want to run Webdriver sessions as part of a suite of integration tests then I would suggest using Python's unittest to create them. You drive the browser to the site under test, and make assertions that the actions you are taking leave the page in a state you expect. I can share some examples of how that might work as well if you are interested.