As I learn selenium with python, I find that I may have made a mistake that leads to an exception while I adapt it to the exact flow of the web application. As such, the scripts exits with an error, and python loses control over the driver.
Fixing the mistake takes some time while a new browser is created; that also creates a lot of temporary files that are required for the browser profile (rust_mozprofile). In short, fixing the error loses me significant time.
I'm looking for an answer specifying a workflow with selenium that allows me to reuse the browser instance, make mistakes in syntax, intent or error handling, and correct them without neither reloading the script, respawning the browser, nor redoing all steps in the website.
I'd prefer a Firefox focused solution, but Chrome answers are also welcome.
Use the VS Code Debugger
Most of the issues mentioned could be avoided by first doing an exhaustive review of the website so that you start coding your script with "perfect" knowledge of the every site transition, delay, button identifier, and so on. Realistically, this would be too much work.
I will provide a simple example of the workflow I use to develop:
Example. Retrieve the first picture from #cats in Instagram
Step 1.
Start with boilerplate code that imports all relevant modules so that if I need them, I can call them immediately. A good start is:
bot_template.py:
import logging
import pickle
import time
from inspect import getsourcefile
from os.path import abspath
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.common.exceptions import NoSuchElementException, TimeoutException
from selenium.webdriver.remote.webelement import WebElement
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.firefox.options import Options
driver = webdriver.Firefox()
# your breakpoint should go in this line
driver.quit()
At this point try doing the process manually of going to Instagram and getting the first picture in the tags feed. If you did you would find that you can go to https://www.instagram.com/explore/tags/cats/ for the cat tag.
Step 2
Add a breakpoint in the line right after driver = webdriver.Firefox() which spawns Firefox. Breakpoints are added by clicking in the empty region immediately left of the line numbers or with the default shortcut (F9).
Step 3
In VSCode, Press F1 and type Start Debugging, and select Python File, or use the default shortcut (F5).
Step 4
In VSCode, Press F1 and type Focus on Debug Console View. Switch over to the debug console.
Step 5
Make mistakes in the debug console as you try to build your script. You need the bot to go to https://www.instagram.com/explore/tags/cats/, so you write: driver.Get(r'https://www.instagram.com/explore/tags/cats/'). Next thing you see is:
Error: AttributeError: 'WebDriver' object has no attribute 'Get'
Step 6
Correct your mistake. You realize the method is actually called get (lowercase). Type that again in the VSCode Debug Console. The browser will now navigate to the right URL, no need to respawn a new one due to a typo.
>>> driver.get('https://www.instagram.com/explore/tags/cats/')
Step 6
Switch over to the browser you are controlling from Python. You are free to do whatever you want to it, in whatever tempo/order you want.
You open the inspector and realize that you can use XPATH in the browser terminal $x('//div[#class="_aabd _aa8k _aanf"]') to get the array of images. Therefore, you type in the VSCode debug console:
>>> image = driver.find_element(By.XPATH, '//div[#class="_aabd _aa8k _aanf"]')
Step 7
You start trying to get the image src. Therefore, you type image.get_attribute('src') in the debug console. It does not work because you selected a div that does have a 'src' html attribute, rookie mistake. The code would have terminated in regular execution and you would have lost control over the browser, but you are on the debugger, so you retain control and your precious time. You keep trying in your debugging playground, until you figure out that you can fix this by changing the xpath. You try typing in the debug console and that gets you the image url.
>>> image = driver.find_element(By.XPATH, '//*[#class="_aabd _aa8k _aanf"]//img[#src]')
>>> src = image.get_attribute('src')
Step 8
Download the image from the url. You keep the browser and vscode session open and go look for ways to download the image. You notice you need other libraries. No problem, try typing the following in same debug console session and you will have your image downloaded.
# Changes the current dir to the project folder and downloads as 'image.png'
>>> import os
>>> import urllib
>>> os.chdir('.')
>>> urllib.urlretrieve(src, 'image.png')
Step 9
Keep building interactively until your bot does all it has to do. Commit the lines of code that gave you the desired result to your file bot_template.py. Once you are done, test the whole thing by running the script without debugging. You never had to respawn another browser session and lose time.
Related
I want to be able to use pure selenium webdriver to open a zoom link in Chrome and then redirect me to the zoom.us application.
When I execute this:
from selenium import webdriver
def main():
driver = webdriver.Chrome()
driver.get("https://zoom.us/j/000-000-000")
main()
I receive a pop-up saying
https://zoom.us wants to open this application.
and I must press a button titled open zoom.us to open the app.
Is there a way to press this pop-up button through selenium. Or, is there some other way to open zoom from chromedriver?
NOTE: I only want to use selenium. I have been able to implement pyautogui to click on the button but that is not what I am looking for.
Solution for Java:
driver.switchTo().alert().accept();
Solution for Python:
driver.switch_to.alert.accept()
There are a lot of duplicated questions regarding this issue. Here is one of them, and it is quite sure that selenium is not capable of achieving such job since it only interacts with the chrome page. I previously encountered this issue as well and here is my solution to it. It might look really unprofessional, but fortunately it works.
The logic of my solution is to change the setting of chrome in order to skip the popup and directly open the application you want. However, the Chrome team has removed this feature in the latter version for some reasons, and we need to get it back manually. Then, we know that everytime when selenium starts to do the thing it opens a new Chrome page with NO customized settings just like the incognito page. Therefore we need to do something to let selenium opened a Chrome page with your customized setting, so that we can make sure that the popup, which we changed manually to skip, can be skipped successfully.
Type the following code in your terminal.
defaults write com.google.Chrome ExternalProtocolDialogShowAlwaysOpenCheckbox -bool true
This enables you to change the setting of skipping popups, which is the feature Chrome team removed.
Restart Chrome,and open the zoom (or whatever application) page to let the popup display. If you do the 1st step correctly you will be able to see there is a checkbox shown next to the "Open Zoom.us" saying if you check it chrome will open this application without asking, that is, to skip the popup for this application.
Now we need to let selenium open the Chrome with our customized setting. To do this, type "chrome://version" in the search tab of your ordinary Chrome (Not automated page opened by selenium). Go to "Profile Path", and copy this path without the last word "default". For example:
/Users/MYNAME/Library/Application Support/Google/Chrome/Default
This is my profile path, but I only copy everything except the last word Default, so this is what I need to copy.
/Users/MYNAME/Library/Application Support/Google/Chrome/
This is for Mac users, but for Windows only the path is different(starts with C:// or something), steps are same.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
option = Options()
option.add_argument('THE PATH YOU JUST COPIED')
driver = webdriver.Chrome(executable_path='YOUR PATH TO CHROMEDRIVER', options=option)
driver.get("google.com") #Or anything else
We use "options" to let selenium open a page with our customized profile. Now you will see selenium opens a Chrome page with all your account profile, settings, and it just appears like your ordinary chrome page.
Run your code. But before that, remember to quit ALL CHROME sessions manually. For Mac, make sure that there is no dot under Chrome icon indicating that Chrome is not running for any circumstances. THIS STEP IS CRITICAL otherwise selenium will open a chrome page and it just stops there.
Here are all the steps. Again, this solution is vert informal and I personally don't think it is a "solution" to this problem. I will try to figure out a better way of achieving this in the future. But I still posted this as an alternative simply because I guess it might be helpful to some extent for somebody just like me. Hope it works for you, and good luck.
I'm trying to make an auto bump program for forums but I can't seem to interact with anything.
import webbrowser
import pyautogui
import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Chrome()
driver.get("https://ogusers.com/member.php?action=login")
#Login
inputElement = driver.find_element_by_class_name("textbox")
inputElement.send_keys("xKyle")
Can't really tell where problem lies, you need to say exactly what is happening, let's start with not giving selenium path to ChromeDriver. You need to download it from here, make sure to get driver compatible with your browser verison.
Next give path to it, if you are on windows path will look like:
r"C:\dir\dir\chromedriver.exe'
you need to give 'r' for raw data, otherwise interpreter will think that you are trying to give some weird in-string command like \n for new line
Also try not to search by class name, for 99% of time you want to use Xpath, especially if it's relative. So just click on element, and copy xpath.
For more answers you need to tell exactly what are you trying to do and when it fails :)
I'm trying to run a Python script using Selenium, and while everything runs fine, my personal print() to console lines get hidden between a ton of Selenium/Chromedriver outputs like this:
1128/150256.806:INFO:CONSOLE(0)] "The SSL certificate used to load resources from [some-link.com] will be distrusted in the future. Once distrusted, users will be prevented from loading these resources. See https://g.co/chrome/symantecpkicerts for more information.", source: [current-page.com] (0)
I checked what these links are, and they're just the ads on the pages I'm looking at, so it's completely useless. Also, since the ads are randomly generated every time a page loads/is reloaded, the links are different so the outputs are never-ending. This is incredibly annoying and makes it very hard to see what is actually happening within my program. Is there any way to turn this off with some Selenium options or something?
The strange thing is, running the program in Eclipse Oxygen with PyDev doesn't show any of Selenium's output at all, only if I run it using command line.
EDIT: following the instructions from the possible duplicate mentioned didn't help. I tried setting the logging level to the highest, CRITICAL, and the output mentioned above still went through and flooded the console.
The best Way to solve this problem is to add the --log-level option to your Driver. That would look something like this:
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument("--log-level=3")
driver = webdriver.Chrome(chrome_options=chrome_options)
Whoever uses the ChromeDriverManager via the webdriver_manager package can use the following to disable the [WDM] log messages:
import os
os.environ["WDM_LOG_LEVEL"] = str(logging.WARNING)
I keep getting this error:
https://sites.google.com/a/chromium.org/chromedriver/help/chromedriver-crashes
I get it when running the command:
python Web.py
However when I go into the file and run the lines 1 by 1, I don't get the error. However I always get the error when the Web.py file has finished. When I run the lines 1 by 1, it's very basic things but i feel like I"m not ending my script correctly.
import selenium
from selenium.webdriver.common.keys import Keys
import time
from selenium import webdriver
driver = webdriver.Chrome()
driver.get('espn.com')
I want the window with espn.com to stay on the screen, not quit when the script has finished.
I'm running this on Python. I can share my setup, maybe that's something I did incorreclty but any help would be appeciated.
You're passing an invalid url.
You need to pass the url like this:
driver.get("http://www.espn.com")
This might work in your browser, but it won't with selenium. Type in "espn.com" in your browser and then copy / paste the url and you'll see that it's actually the above url.
You should also specify the "chromedriver.exe" path.
You are getting this error because you had not installed the chrome driver for selenium on your Machine. Selenium by default provides the driver for Firefox so when you use the webdriver for Firefox, it won't rise any error. To resolve this issue with Chrome you can download the Chrome webdriver from here.
and you can specify the driver as
from selenium import webdriver
d = webdriver.Chrome(executable_path='<your Chrome driver path>')
Adding to what #Pythonista said , it's better if you keep the URL as a raw string than a normal string
driver.get(r'http://www.espn.com')
so that it won't take the slash as an escape sequence in few cases.
Hope it helps.
Try to update chrome and get updated/latest chrome driver, recently chrome made several updates in its driver you can download the last one from the link below:
https://chromedriver.storage.googleapis.com/2.27/chromedriver_win32.zip
I'm able to upload a file to a remote server using Selenium, but the file selector dialog sticks around even once the file is uploaded. The Selenium FAQ notes that, "You can't interact with the native OS file browser dialog directly, but we do some magic so that...." Given the use of "magic" here it's not surprising that the behavior I get is a little rough around the edges. But there appear to be workarounds. Taking my cues from this answer I have the following code:
import contextlib, time
from selenium import webdriver
import selenium.webdriver.common.action_chains as action_chains
with contextlib.closing(webdriver.Chrome()) as driver:
driver.get("http://www.bing.com/images")
driver.find_element_by_id("sbi_t").click()
driver.find_element_by_id("sbi_file").click()
driver.find_element_by_id("sbi_file_upload").send_keys("//Loch Ness Monster.jpg")
print driver.current_url # Still `http://www.bing.com/images` :(
file_upload = driver.find_element_by_id("sbi_file_upload")
action_chains.ActionChains(driver).click(file_upload).perform() # https://stackoverflow.com/a/16864547/2829764
But at the end of this the file upload window is still there. I suspect I need a slightly different workaround since I'm on a Mac. Can anyone help?
Don't click upload button at all.
Set the filename via send_keys() and click "Go" (tested and works for me):
element = driver.find_element_by_id("sbi_file_upload")
element.send_keys('/Path/to/file.jpeg')
driver.find_element_by_css_selector('div#sbi_sb_ipt span[name=go]').click()