how do I enable javascript in selenium using python? I tried several methods but mine isnt working. Does anyone know how could i fix this issue? Thanks
My code
from selenium import webdriver
import urllib
import urllib.request
import string
from bs4 import BeautifulSoup
import mysql.connector
import time
chrome_path = r"C:\chromedriver.exe"
driver = webdriver.Chrome(chrome_path)
driver.add_argument("--enable-javascript")
Error i got
Exception has occurred: AttributeError
'WebDriver' object has no attribute 'add_argument'
File "C:\Users\bustillo-ronald\Desktop\python-basics\Scrape\propertyguru.py", line 11, in <module>
driver.add_argument("--enable-javascript")
You need to create an instance of a ChromeOptions object first and add_argument to that. Then pass the options object as an argument to the Chrome webdriver.
something like this
options = webdriver.ChromeOptions()
options.add_argument("--enable-javascript")
# Now back to your code
driver = webdriver.Chrome(chrome_path, options=options)
...
Related
I've been struggling with this problem for sometime, but now I'm coming back around to it. I'm attempting to use selenium to scrape data from a URL behind a company proxy using a pac file.
I'm using Chromedriver, which my browser uses the pac file in it's configuration.
I've been trying to use desired_capabilities, but the documentation is horrible or I'm not grasping something. Originally, I was attempting to webscrape with beautifulsoup, which I had working except the data I need now is in javascript, which can't be read with bs4.
Below is my code:
import pandas as pd
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.proxy import Proxy, ProxyType
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
desired_capabilities = webdriver.DesiredCapabilities.CHROME.copy()
PAC_PROXY = {
'proxyAutoconfigUrl': 'http://proxy-pac/proxy.pac',
}
proxy = Proxy()
proxy.proxy_autoconfig_url = PAC_PROXY['proxyAutoconfigUrl']
desired_capabilities = {}
proxy.add_to_capabilities(desired_capabilities)
URL = "https://mor.nlm.nih.gov/RxClass/search?query=ALIMENTARY%20TRACT%20AND%20METABOLISM%7CATC1-4&searchBy=class&sourceIds=a&drugSources=atc1-4%7Catc%2Cepc%7Cdailymed%2Cmeshpa%7Cmesh%2Cdisease%7Cmedrt%2Cchem%7Cdailymed%2Cmoa%7Cdailymed%2Cpe%7Cdailymed%2Cpk%7Cmedrt%2Ctc%7Cfmtsme%2Cva%7Cva%2Cdispos%7Csnomedct%2Cstruct%7Csnomedct%2Cschedule%7Crxnorm"
service = Service('C:\Program Files\Chrome Driver\chromedriver.exe')
driver = webdriver.Chrome(service=service)
driver.get(URL)
print(driver.requests[0].headers, driver.requests[0].response)
WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR, 'tr.dbsearch')))
print(pd.read_html(driver.page_source)[1].iloc[:,:-1])
pd.read_html(driver.page_source)[1].iloc[:,:-1].to_csv('table.csv',index=False)
I'm not sure why I'm receiving an:
TypeError: __init__() got an unexpected keyword argument 'service'
even when I have the path added correctly to my system environment variables as shown below:
Essentially what I'm attempting to do is scrape the data in the table from https://mor.nlm.nih.gov/RxClass/search?query=ALIMENTARY%20TRACT%20AND%20METABOLISM%7CATC1-4&searchBy=class&sourceIds=a&drugSources=atc1-4%7Catc%2Cepc%7Cdailymed%2Cmeshpa%7Cmesh%2Cdisease%7Cmedrt%2Cchem%7Cdailymed%2Cmoa%7Cdailymed%2Cpe%7Cdailymed%2Cpk%7Cmedrt%2Ctc%7Cfmtsme%2Cva%7Cva%2Cdispos%7Csnomedct%2Cstruct%7Csnomedct%2Cschedule%7Crxnorm then store it to a pandas dataframe and pass it to a csv file.
If you are still using Selenium v3.x then you shouldn't use the Service() and in that case the key executable_path is relevant. In that case the lines of code will be:
driver = webdriver.Chrome(executable_path='C:\Program Files\Chrome Driver\chromedriver.exe')
Else, if you are using selenium4 then you have to use Service() and in that case the key executable_path is no more relevant. So you need to change the line of code:
service = Service(executable_path='C:\Program Files\Chrome Driver\chromedriver.exe')
driver = webdriver.Chrome(service=service)
as:
service = Service('C:\Program Files\Chrome Driver\chromedriver.exe')
I've been having this problem also since I switched from pip Jupyter to Anaconda Jupyter. This worked for me:
driver = webdriver.Chrome(ChromeDriverManager().install())
Apparently you don't need the service in the Jupyter package.
I keep receiving the same error with my code "AttributeError: module 'selenium.webdriver' has no attribute 'get'"
any help?
code:
from selenium import webdriver as wd
import chromedriver_binary
wd = wd.Chrome()
wd.implicitly_wait(10)
wd.get("http://google.com")
You have a problem with naming.
Try this, I hope it will fix your problem
from selenium import webdriver as wd
import chromedriver_binary
driver = wd.Chrome()
driver.implicitly_wait(10)
driver.get("http://google.com")
With your original names you are attempting to assign wd.Chrome() to the selenium.webdriver module.
After that you attempting to set implicitly_wait(10) on selenium.webdriver
And finally you attempting to apply .get("http://google.com") on selenium.webdriver
That is how Python interprets your code
Why the first code doesn't work while the second does?
First code:
import selenium
driver = selenium.webdriver.Firefox()
AttributeError: 'module' object has no attribute 'webdriver'
Second code:
from selenium import webdriver
driver = webdriver.Firefox()
Nested packages are not automatically loaded; not until you import selenium.webdriver is it available as an attribute. Importing just selenium is not enough.
Do this:
import selenium.webdriver
driver = selenium.webdriver.Firefox()
Sometimes the package itself will import a nested package in the __init__.py package initializer; os imports os.path, so os.path is immediately available even if you import just os.
I am trying to use selenium webdriver in centos to test my webpage.
But,I got an error message when I execute the process.
Can someone help me?
from pyvirtualdisplay import Display
from selenium import webdriver
display=Display(visible=0, size=(320, 240)).start()
path = "/usr/bin/firefox"
driver= webdriver.Firefox(path)
driver.get("www.google.com")
html_source = driver.page_source
print html_source
driver.close()
And here is the error message:
File "/var/www/test/test.py", line 19, in <module>
driver= webdriver.Firefox(path)
File "/usr/local/lib/python2.7/site-packages/selenium/webdriver/firefox/webdriver.py", line 54, in __init__
self.NATIVE_EVENTS_ALLOWED and self.profile.native_events_enabled)
AttributeError: 'str' object has no attribute 'native_events_enabled'
Pretty sure your problem has to do with the fact that your trying to pass the path to your firefox binary as a string, instead as a "FirefoxBinary" object, furthermore the first argument to Firefox() is a FirefoxProfile(). Doing the following should resolve the issue.
from pyvirtualdisplay import Display
from selenium import webdriver
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
display=Display(visible=0, size=(320, 240)).start()
binary = FirefoxBinary("/usr/bin/firefox")
driver= webdriver.Firefox(firefox_binary=binary)
driver.get("www.google.com")
html_source = driver.page_source
print html_source
driver.close()
see this post for an answer to a very similar problem.
I am using Selenium and for this task I need to maximize the browser after the page is loaded, the problem is that I am getting the following error and can't seem to understand how to solve it.
AttributeError: 'WebDriver' object has no attribute 'window_maximize'
Here is the code I am testing
from pyvirtualdisplay import Display
from ftplib import FTP
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait # available since 2.4.0
from selenium.webdriver.common.keys import Keys
#initialize HIDDEN display
display = Display(visible=0, size=(1366, 768))
display.start()
browser = webdriver.Firefox()
browser.get('http://youtube.com/')
browser.window_maximize();
...
Isn't window_maximize an attribute of the browser?
I am using python and Selenium Server 2.28
Any tip much appreciated!
You can use browser.maximize_window(). I think the problem was in the wrong function name.
OK, after much looking for how to use window_maximize I found out I can use browser.set_window_size(800, 600) instead.
I tested and it works fine. It is important to set the browser to browser.set_window_position(0, 0)
The answer was found here How to maximize a browser window using the Python bindings for Selenium 2-WebDriver?