I want to use selenium to scrape off some website. I can't access the website via my own internet connection, so I need to use browsec mozilla addon for that.
I am unable to launch firefox with selenium with the add-on enabled.
Here is what I have tried:
import selenium
from selenium import webdriver
url = "http://url"
profile = webdriver.FirefoxProfile()
profile.add_extension('browsec#browsec.com.xpi')
#profile.add_extension("C:\Users\urs\AppData\Roaming\Mozilla\Firefox\Profiles\abc.default\extensions\browsec#browsec.com.xpi")
driver = webdriver.Firefox(firefox_profile=profile)
if __name__ == "__main__":
driver.get(url)
driver.wait(5)
driver.quit()
I have tried putting the extension in the same directory where my script is and using the following
profile.add_extension('browsec#browsec.com.xpi')
which gives me this error when I run:
Traceback (most recent call last): File
"C:\Python36\lib\site-packages\selenium\webdriver\firefox\firefox_profile
.py", line 346, in _addon_details
with open(os.path.join(addon_path, 'install.rdf'), 'r') as f: FileNotFoundError: [Errno 2] No such file or directory:
'C:\Users\Usr\AppD
ata\Local\Temp\tmp0hny31u3.browsec#browsec.com.xpi\install.rdf'
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "test.py", line 7, in
profile.add_extension("browsec#browsec.com.xpi") File "C:\Python36\lib\site-packages\selenium\webdriver\firefox\firefox_profile
.py", line 95, in add_extension
self._install_extension(extension) File "C:\Python36\lib\site-packages\selenium\webdriver\firefox\firefox_profile
.py", line 274, in _install_extension
addon_details = self._addon_details(addon) File "C:\Python36\lib\site-packages\selenium\webdriver\firefox\firefox_profile
.py", line 351, in _addon_details
raise AddonFormatError(str(e), sys.exc_info()[2]) selenium.webdriver.firefox.firefox_profile.AddonFormatError: ("[Errno
2] No such file or directory:
'C:\\Users\\Usr\\AppData\\Local\\Temp\\tmp0hn
y31u3.browsec#browsec.com.xpi\\install.rdf'", )
I also tried giving the path to the extension:
profile.add_extension("C:\Users\urs\AppData\Roaming\Mozilla\Firefox\Profiles\abc.default\extensions\browsec#browsec.com.xpi")
And I ran into this error:
profile.add_extension("C:\Users\Hassan\AppData\Roaming\Mozilla\Firefox\Profi
les\n5jwlj9l.default\extensions\browsec#browsec.com.xpi")
^ SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in positio n 2-3: truncated
\UXXXXXXXX escape
Formatting the path string like below doesn't help either.
profile.add_extension(r"C:\Users\urs\AppData\Roaming\Mozilla\Firefox\Profiles\abc.default\extensions\browsec#browsec.com.xpi")
I get the following:
Traceback (most recent call last): File "test.py", line 7, in
profile.add_extension(r"C:\Users\Hassan\AppData\Roaming\Mozilla\Firefox\Prof
iles\n5jwlj9l.default\extensions\browsec#browsec.com.xpi") File
"C:\Python36\lib\site-packages\selenium\webdriver\firefox\firefox_profile
.py", line 95, in add_extension
self._install_extension(extension) File "C:\Python36\lib\site-packages\selenium\webdriver\firefox\firefox_profile
.py", line 274, in _install_extension
addon_details = self._addon_details(addon) File "C:\Python36\lib\site-packages\selenium\webdriver\firefox\firefox_profile
.py", line 351, in _addon_details
raise AddonFormatError(str(e), sys.exc_info()[2]) selenium.webdriver.firefox.firefox_profile.AddonFormatError: ("[Errno
2] No such file or directory:
'C:\\Users\\usr\\AppData\\Local\\Temp\\tmp1he
0fym_.browsec#browsec.com.xpi\\install.rdf'", )
How do I configure selenium to run firefox with browsec enabled by default?
I found this article rather helpful.
Instead of adding the extension to the profile, you install it after the browser has been created:
from selenium import webdriver
driver = webdriver.Firefox()
# This installs adblock plus
driver.install_addon("/home/your_username/coding/Project/seleniumTest/adblock.xpi", temporary=True)
driver.get('https://www.stackoverflow.com')
Be sure to add the .xpi to your project folder!
You can try to create profile on firefox browser like - On windows Run --> type
"firefox.exe -P"
It will open profile manager. Create new profile. Start firefox from that profile, add plugins. And use that same profile with code..Sometime it worked for me..
Sorry for my English))
Most likely you are using the new version of Firefox (Quantum - from the 57th version inclusive). In newer versions of Firefox, the extension metadata is not stored in the install.rdf file, but in the manifest.json file. Selenium does not know this yet (in version 3.11, and learns only in 3.14). Therefore, when trying to connect an extension, it looks for habit install.rdf.
Here the author wrote a class that slightly changes the connection function of the extension, and instead of install.rdf, selenium looks for metadata in manifest.json.
What you need to do:
# Add Import
import json
import os
import sys
from selenium.webdriver.firefox.firefox_profile import AddonFormatError
# Add class
class FirefoxProfileWithWebExtensionSupport(webdriver.FirefoxProfile):
def _addon_details(self, addon_path):
try:
return super()._addon_details(addon_path)
except AddonFormatError:
try:
with open(os.path.join(addon_path, 'manifest.json'), 'r') as f:
manifest = json.load(f)
return {
'id': manifest['applications']['gecko']['id'],
'version': manifest['version'],
'name': manifest['name'],
'unpack': False,
}
except (IOError, KeyError) as e:
raise AddonFormatError(str(e), sys.exc_info()[2])
# Declare Firefox_profile written class
profile = FirefoxProfileWithWebExtensionSupport()
Further as usual)))
Good luck)))
Related
I am using selenium with python and have downloaded the chromedriver for my windows computer from this site. After downloading the zip file, I unpacked the zip file to my downloads folder. Then I add the path to the the Environment Variable "Path".
I want to get information from the site, but when I run code, I get this error and comletely don't understand what is it about.
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
browser = webdriver.Chrome(ChromeDriverManager().install())
url = 'any_url'
browser.get(url)
if browser.find_element_by_id("yearlist_1").get_attribute("type") == "checkbox":
print("Element is a checkbox")
else:
print("Element is not a checkbox")
The error:
*'reg' is not recognized as an internal or external command,
operable program or batch file.
Traceback (most recent call last):
File "C:/Users/miair/Python/script_gks/code_for_ticks.py", line 4, in <module>
browser = webdriver.Chrome(ChromeDriverManager().install())
File "C:\Program Files\Python37\lib\site-packages\webdriver_manager\chrome.py", line 28, in install
driver_path = self.download_driver(self.driver)
File "C:\Program Files\Python37\lib\site-packages\webdriver_manager\manager.py", line 36, in download_driver
driver_version, is_latest = self.__get_version_to_download(driver)
File "C:\Program Files\Python37\lib\site-packages\webdriver_manager\manager.py", line 27, in __get_version_to_download
return self.__get_latest_driver_version(driver), True
File "C:\Program Files\Python37\lib\site-packages\webdriver_manager\manager.py", line 21, in __get_latest_driver_version
return driver.get_latest_release_version()
File "C:\Program Files\Python37\lib\site-packages\webdriver_manager\driver.py", line 58, in get_latest_release_version
self._latest_release_url + '_' + chrome_version(self.chrome_type))
File "C:\Program Files\Python37\lib\site-packages\webdriver_manager\utils.py", line 114, in chrome_version
.format(cmd)
ValueError: Could not get version for Chrome with this command: reg query "HKEY_CURRENT_USER\Software\Google\Chrome\BLBeacon" /v version*
Sorry, I am very new to Python, but if somebody could help, I'll be very pleased and thankful.
I solved problem. I don't understand why I've hadn't think of this solution before - all you need to do is to write the path of your chromedriver (only if you have a correct version)
browser = webdriver.Chrome('path')
important to know for people searching for the same error and ending here. chrome needs to have started at least once to generate the reg key in windows.
I suspect that you are using a selenium web-driver that is not matching your chrome driver version.
you can update your chrome using the following link:
chrome://settings/help
I'm trying to use selenium for a python web scraper but when I try to run the program I get the following error:
/usr/local/bin/python3 /Users/xxx/Documents/Python/hello.py
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/selenium/webdriver/common/service.py", line 72, in start
self.process = subprocess.Popen(cmd, env=self.env,
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 854, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 1702, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/xxx/Documents/Python/chromedriver.exe'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/xxx/Documents/Python/hello.py", line 9, in <module>
wd = webdriver.Chrome(executable_path=DRIVER_PATH)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/selenium/webdriver/chrome/webdriver.py", line 73, in __init__
self.service.start()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/selenium/webdriver/common/service.py", line 81, in start
raise WebDriverException(
selenium.common.exceptions.WebDriverException: Message: 'chromedriver.exe' executable needs to be in PATH. Please see https://sites.google.com/a/chromium.org/chromedriver/home
Here is the python code:
from urllib.request import urlopen
from bs4 import BeautifulSoup
import re
from selenium import webdriver
DRIVER_PATH = '/Users/xxx/Documents/Python/chromedriver.exe'
wd = webdriver.Chrome(executable_path=DRIVER_PATH)
I think the problem is that I'm not specifying the file path in the variable DRIVER_PATH properly but I'm not sure
I am using a Mac
You need to update DRIVER_PATH to include your root directory, which is usually C:\:
DRIVER_PATH = 'C:/Users/xxx/Documents/Python/chromedriver.exe'
Alternatively, you can follow this tutorial to add the path to containing folder of chromedriver.exe (usually chromedriver_win32 folder) to your Path environment variable:
https://docs.telerik.com/teststudio/features/test-runners/add-path-environment-variables
I would try this out (Just adding the 'r'):
wd = webdriver.Chrome(executable_path=r'/Users/xxx/Documents/Python/chromedriver.exe')
if you think it's the filepath then have a go with checking:
import os.path
os.path.exists(DRIVER_PATH)
Also, Beautifulsoup is used will with urllib2
https://www.pythonforbeginners.com/beautifulsoup/beautifulsoup-4-python
import urllib2
url = "https://www.URL.com"
content = urllib2.urlopen(url).read()
soup = BeautifulSoup(content)
You have a mistake in the name of the file.
"chomedriver.exe" is for windows.
If you use macOS and chromedriver for Mac, then the file name should be "chomedriver" without ".exe".
I had the same problem, but this solved it.
I need to launch Firefox with Browsec add-on. I've written some code, but get an error:
from selenium import webdriver
url = input("Enter url to scrape from: ")
profile = webdriver.FirefoxProfile()
profile.add_extension('/home/myusername/.mozilla/firefox/b5qyukpg.default/extensions/browsec#browsec.com.xpi')
browser = webdriver.Firefox()
browser.get(url)
What i get is:
Traceback (most recent call last):
File "scraper.py", line 7, in <module>
profile.add_extension('/home/myusername/.mozilla/firefox/b5qyukpg.default/extensions/browsec#browsec.com.xpi')
File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/firefox/firefox_profile.py", line 95, in add_extension
self._install_extension(extension)
File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/firefox/firefox_profile.py", line 274, in _install_extension
addon_details = self._addon_details(addon)
File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/firefox/firefox_profile.py", line 351, in _addon_details
raise AddonFormatError(str(e), sys.exc_info()[2])
selenium.webdriver.firefox.firefox_profile.AddonFormatError: ("[Errno 2] No such file or directory: '/tmp/tmpd7nyxubj.browsec#browsec.com.xpi/install.rdf'", <traceback object at 0x7f23f786eb08>)
How can i fix that error?
I am attempting to run a simple program on an Ubuntu 16.04 instance using Python 3.5. The program is below;
from bs4 import BeautifulSoup
from selenium import webdriver
driver = webdriver.PhantomJS("p/phantomjs")
driver.get("http://www.bbc.co.uk")
s = BeautifulSoup(driver.page_source, "lxml")
print(s.findAll("a"))
try:
driver.close()
except AttributeError:
pass
All the modules are installed correctly. However, when I run the program, I receive the following errors:
Traceback (most recent call last):
File "t.py", line 4, in <module>
driver = webdriver.PhantomJS("p/phantomjs")
File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/phantomjs/webdriver.py", line 52, in __init__
self.service.start()
File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/common/service.py", line 64, in start
stdout=self.log_file, stderr=self.log_file)
File "/usr/lib/python3.5/subprocess.py", line 947, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.5/subprocess.py", line 1551, in _execute_child
raise child_exception_type(errno_num, err_msg)
OSError: [Errno 8] Exec format error
Exception ignored in: <bound method Service.__del__ of <selenium.webdriver.phantomjs.service.Service object at 0x7fb05cd964a8>>
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/common/service.py", line 163, in __del__
self.stop()
File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/common/service.py", line 135, in stop
if self.process is None:
AttributeError: 'Service' object has no attribute 'process'
It seems as though it is an issue with Selenium rather than with PhantomJS. However, I have no idea how to make the program work properly.
In other questions similar to this, the issue seems to be with closing the headless instance. However, this error is received as soon as I try to instantiate PhantomJS.
How can this be fixed?
If p folder (as you've mentioned) located in the same directory as your script, then you might need to start your code with something like
from bs4 import BeautifulSoup
from selenium import webdriver
import os
path_to_phantom_js = os.path.dirname(__file__) + '/p/phantomjs'
driver = webdriver.PhantomJS(path_to_phantom_js)
P.S. If it not works, tell me output of print(path_to_phantom_js)
I am trying to automate downloading mp3 files from Youtube using this site.
Objective:
Visit download website
Paste Youtube link
Click "Convert Video"
Wait for download link to appear, then click it
When I use browser.find_by_id('dl_link').click(), my code executes without any errors but the file is not downloaded. Why?
When I use browser.click_link_by_partial_text('Download'), I get the error shown below.
Here is my code:
from splinter.browser import Browser
import time
with Browser() as browser:
browser.visit("http://www.youtube-mp3.org")
browser.find_by_id('youtube-url').fill("https://www.youtube.com/watch?v=lgT1AidzRWM")
browser.find_by_id('submit').click()
if browser.is_element_present_by_id('dl_link'):
time.sleep(2)
browser.click_link_by_partial_text('Download')
# browser.find_by_id('dl_link').click()
print "Clicked Download"
time.sleep(2)
Here is the error I am getting:
Traceback (most recent call last):
File "/Users/anon/Dropbox/Programs/Proj/splinter2.py", line 11, in <module>
browser.click_link_by_partial_text('Download')
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/splinter/driver/__init__.py", line 332, in click_link_by_partial_text
return self.find_link_by_partial_text(partial_text).first.click()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/splinter/driver/webdriver/__init__.py", line 539, in click
self._element.click()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/selenium/webdriver/remote/webelement.py", line 74, in click
self._execute(Command.CLICK_ELEMENT)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/selenium/webdriver/remote/webelement.py", line 457, in _execute
return self._parent.execute(command, params)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 233, in execute
self.error_handler.check_response(response)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 194, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.ElementNotVisibleException: Message: Element is not currently visible and so may not be interacted with
Stacktrace:
at fxdriver.preconditions.visible (file:///var/folders/wj/t56fgsms4rdcgptcy94k2w8m0000gn/T/tmpAVUrd2/extensions/fxdriver#googlecode.com/components/command-processor.js:10092)
at DelayedCommand.prototype.checkPreconditions_ (file:///var/folders/wj/t56fgsms4rdcgptcy94k2w8m0000gn/T/tmpAVUrd2/extensions/fxdriver#googlecode.com/components/command-processor.js:12644)
at DelayedCommand.prototype.executeInternal_/h (file:///var/folders/wj/t56fgsms4rdcgptcy94k2w8m0000gn/T/tmpAVUrd2/extensions/fxdriver#googlecode.com/components/command-processor.js:12661)
at DelayedCommand.prototype.executeInternal_ (file:///var/folders/wj/t56fgsms4rdcgptcy94k2w8m0000gn/T/tmpAVUrd2/extensions/fxdriver#googlecode.com/components/command-processor.js:12666)
at DelayedCommand.prototype.execute/< (file:///var/folders/wj/t56fgsms4rdcgptcy94k2w8m0000gn/T/tmpAVUrd2/extensions/fxdriver#googlecode.com/components/command-processor.js:12608)
[Finished in 7.2s with exit code 1]
[shell_cmd: python -u "/Users/anon/Dropbox/Programs/Proj/splinter2.py"]
[dir: /Users/anon/Dropbox/Programs/Proj]
[path: /usr/bin:/bin:/usr/sbin:/sbin]
I have already checked other questions with the same (Error: “Element is not currently visible and so may not be interacted with” with selenuim) title, but I am still not able to resolve this since the Download button is visible in my case.
Any help would be much appreciated
EDIT:
Whenever I try to set a profile using browser = Browser('firefox', profile=profile), I get the following error:
Traceback (most recent call last):
File "/Users/adb/Dropbox/Programs/Proj/Youtube Playlist MP3/splinter2.py", line 11, in <module>
browser = Browser('firefox', profile=profile)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/splinter/browser.py", line 63, in Browser
return driver(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/splinter/driver/webdriver/firefox.py", line 23, in __init__
firefox_profile = FirefoxProfile(profile)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/selenium/webdriver/firefox/firefox_profile.py", line 77, in __init__
ignore=shutil.ignore_patterns("parent.lock", "lock", ".parentlock"))
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 171, in copytree
names = os.listdir(src)
TypeError: coercing to Unicode: need string or buffer, FirefoxProfile found
When I use browser.find_by_id('dl_link').click(), my code executes without any errors but browser.click_link_by_partial_text('Download') throws?
Because you interact with a "static" element.
When you hit the "convert video" button, you refresh the DOM. But your driver still works on the old DOM. That's why you don't find by using browser.click_link_by_partial_text('Download')
When it wokrs, the file is not downloaded. Why?
Probably because there's a pop up windows asking you if you want to save and execute it. Check this out