The newest Firefox version 67 has private browsing add-ons disabled. These can be added by following this guide. https://support.mozilla.org/en-US/kb/extensions-private-browsing
The issue is that even using that guide the add-ons are not consistently enabled under the geckodriver.
This was functioning in Firefox v66 without issue.
I have run an instance with the default firefox profile (Profile0 from profile.ini) with two add-ons (coil and LastPass). I see the addons popup for about 2 seconds and then disappear.
I have manually enabled them inside the private browser by going to about: addons. It appears that if they are already enabled in the browser for private browser use and are not visible in private browsing changing permissions to disallow still enables them in private browsing, then allowing them keeps them enabled. Very strange.
I uninstalled an addon closed all browsers, started the script again and it worked the first time. Then on the second run, it was back to the 2 seconds enabled then disappear mode as mentioned above.
I've checked the temp profile created under temp when geckodriver copies the profile for consumption. The add-ons are included.
Manually opening up a private browser shows two add-ons so it appears specific to geckodriver + firefox v67. However, the private browser doesn't have me logged in on my addons.
def get_firefox_profile_dir(self):
from pathlib import Path
self.gecko_path = os.path.dirname(__file__)
if sys.platform in ['linux', 'linux2']:
import subprocess
self.ff_gecko = Path(self.gecko_path + '/geckodriver')
bits = 'uname -m'
ver_32_64 = subprocess.getstatusoutput(bits)
cmd = "ls -d /home/$USER/.mozilla/firefox/*.default/"
fp = subprocess.Popen([cmd], shell=True, stdout=subprocess.PIPE)
FF_PRF_DIR = fp.communicate()[0][0:-2]
FF_PRF_DIR_DEFAULT = str(FF_PRF_DIR, 'utf-8')
ff_ext_path = os.path.join(FF_PRF_DIR_DEFAULT, 'extensions')
self.ff_coil_loc = os.path.join(ff_ext_path, self.ff_coil_extId)
ff_coil_enabled = os.path.exists(self.ff_coil_loc)
if ff_coil_enabled:
if 'x86_64' in ver_32_64:
if not self.ff_gecko.is_file():
import wget
self.gecko_targz = 'geckodriver-v0.24.0-linux64.tar.gz'
wget.download(self.gecko_source_linux64, self.gecko_path)
self.file_unzip_tar(self.gecko_path + '/' + self.gecko_targz)
os.remove(self.gecko_path + '/' + self.gecko_targz)
if self.ff_gecko.is_file():
self.data_path = FF_PRF_DIR_DEFAULT
self.gecko = self.ff_gecko
return
if 'i368' in ver_32_64:
if not self.ff_gecko.is_file():
import wget
self.gecko_targz = 'geckodriver-v0.24.0-linux32.tar.gz'
wget.download(self.gecko_source_linux32, self.gecko_path)
self.file_unzip_tar(self.gecko_path + '/' + self.gecko_targz)
os.remove(self.gecko_path + '/' + self.gecko_targz)
if self.ff_gecko.is_file():
self.data_path = FF_PRF_DIR_DEFAULT
self.gecko = self.ff_gecko
return
elif sys.platform == 'win32' or 'nt':
from pathlib import Path
self.gecko = self.gecko_path + "\geckodriver.exe"
mozilla_profile = os.path.join(os.getenv('APPDATA'), r'Mozilla\Firefox')
mozilla_profile_ini = os.path.join(mozilla_profile, r'profiles.ini')
profile = configparser.ConfigParser()
profile.read(mozilla_profile_ini)
FF_PRF_DIR_DEFAULT = os.path.normpath(os.path.join(mozilla_profile, profile.get('Profile0', 'Path')))
ff_ext_path = os.path.join(FF_PRF_DIR_DEFAULT, 'extensions')
self.ff_coil_loc = os.path.join(ff_ext_path, self.ff_coil_extId)
ff_coil_enabled = os.path.exists(self.ff_coil_loc)
if ff_coil_enabled:
ff_gecko = Path(self.gecko)
if ff_gecko.is_file():
self.data_path = FF_PRF_DIR_DEFAULT
return
else:
import wget
wget.download(self.gecko_source_win64)
self.file_zunip('geckodriver-v0.24.0-win64.zip')
if ff_gecko.is_file():
os.remove('geckodriver-v0.24.0-win64.zip')
self.data_path = FF_PRF_DIR_DEFAULT
return
self.get_firefox_profile_dir()
self.driver = webdriver.Firefox(options=self.options, firefox_profile=self.data_path, executable_path=self.gecko)
self.driver.get(self.url) # OPEN URL
I expect the FF addons to be enabled as they have been manually permitted from the browser and that they will be logged in as they are in normal mode.
It appears that the core issue was related to the upgrade to v67 and some unknown items in the profile became broken. A profile reset fixed the issue.
https://support.mozilla.org/en-US/kb/refresh-firefox-reset-add-ons-and-settings
UPDATE: After futher testing I see that it shows short term stability and that extensions are hit and miss (more miss) when loading dynamically. In other words after profile reset addons work the first few times and then they do not load anymore with selenium
Related
This is for a project where a user can download all their GitHub Gists.
This code gets the directory of the user's Download folder on their computer for files to download into. But what if the user's browser's download location is not the computer's Download folder? Maybe it's the Desktop or some random folder.
Am I supposed to check what browser the user is using and somehow get the path of where their download location is? Though a Google search says there's 200 different browsers...
Even if I was to ignore the user's browser's download location and save to the operating system's Download folder there are at least 33 according to a search.
# Find the user's download folder
# Get the operating system
system = platform.system()
# Set the path to save the files to the user's Download folder location
if system == "Windows":
save_path = os.path.join(os.environ['USERPROFILE'], 'Downloads')
elif system == "Darwin":
save_path = os.path.expanduser("~/Downloads")
elif system == "Linux":
save_path = os.path.expanduser("~/Downloads")
on windows to get the path of the Downloads folder, it should be done through win32Api, specifically through SHGetKnownFolderPath, python has access to it through ctypes, the way to access this specific function is taken from Windows Special and Known Folders from python stack overflow answer. with some modifications to read c_wchar_p.
you have to pass in the GUID for the downloads folder from KNOWNFOLDERID which is "{374DE290-123F-4565-9164-39C4925E467B}".
, so you end up with the following code that works only on 64-bit python, for 32-bit you will probably have to change the argument types.
from ctypes import windll, wintypes
from ctypes import *
from uuid import UUID
from itertools import count
from functools import partial
# ctypes GUID copied from MSDN sample code
class GUID(Structure):
_fields_ = [
("Data1", wintypes.DWORD),
("Data2", wintypes.WORD),
("Data3", wintypes.WORD),
("Data4", wintypes.BYTE * 8)
]
def __init__(self, uuidstr):
uuid = UUID(uuidstr)
Structure.__init__(self)
self.Data1, self.Data2, self.Data3, self.Data4[0], self.Data4[1], rest = uuid.fields
for i in range(2, 8):
self.Data4[i] = rest>>(8-i-1)*8 & 0xff
FOLDERID_Downloads = '{374DE290-123F-4565-9164-39C4925E467B}'
SHGetKnownFolderPath = windll.shell32.SHGetKnownFolderPath
SHGetKnownFolderPath.argtypes = [
POINTER(GUID), wintypes.DWORD, wintypes.HANDLE, POINTER(c_char_p)]
def get_known_folder_path(uuidstr):
pathptr = c_char_p()
guid = GUID(uuidstr)
if SHGetKnownFolderPath(byref(guid), 0, 0, byref(pathptr)):
raise Exception('Whatever you want here...')
resp = cast(pathptr,POINTER(c_wchar))
iterator = (resp.__getitem__(i) for i in count(0))
result = ''.join(list(iter(iterator.__next__, '\x00')))
return result
print(get_known_folder_path(FOLDERID_Downloads))
this will return the Downloads folder location even if the user changes it through the properties, or for different languages.
on linux a similar method is to get it from $HOME/.config/user-dirs.dirs under the name of XDG_DOWNLOAD_DIR, which is changed with user settings changes.
$ grep XDG_DOWNLOAD_DIR ~/.config/user-dirs.dirs
XDG_DOWNLOAD_DIR="$HOME/Downloads"
This is obviously only the "default" location, you should allow the user to manually specify his own custom downloads path.
Using a hardcoded path is a recipe for "but it works on my machine", so just ask the OS about its path.
I'm using python+selenium, with profile that has addons.
On startup it shows them all momentarily, but then they are concealed. They exist, they are not disabled, but invisible and don't work. I can disable and enable it, then it appears on the taskbar and is functional.
When invoking firefox manually with the profile, it works.
Here's what printed to the log.
1596129664500 mozrunner::runner INFO Running command: "/usr/bin/firefox" "-marionette" "-foreground" "-no-remote" "-profile" "/tmp/rust_mozprofileO34n0s"
JavaScript error: resource:///modules/sessionstore/SessionStore.jsm, line 1325: uncaught exception: 2147746065
JavaScript error: resource://gre/modules/ExtensionContent.jsm, line 554: TypeError: Argument 1 of PrecompiledScript.executeInGlobal is not an object.
1596129672037 Marionette INFO Listening on port 41285
1596129672136 Marionette WARN TLS certificate errors will be ignored for this session
JavaScript error: undefined, line 14: Error: An unexpected error occurred
JavaScript error: moz-extension://45aaa1ae-14fe-4a8f-841d-6a9416fd5d09/lib/picture_in_picture_overrides.js, line 15: Error: Incorrect argument types for pictureInPictureParent.setOverrides.
1596129683512 Marionette INFO Stopped listening on port 41285
Can it be because of these errors?
The code itself is of no interest at all:
#!/usr/bin/env python
from selenium.webdriver import Firefox
from selenium.webdriver.firefox.options import Options
profile_path='./profile'
opts=Options()
opts.profile=profile_path
driver=Firefox(options=opts)
breakpoint()
Versions are probably more important:
Mozilla Firefox 68.9.0esr
geckodriver 0.26.0 (e9783a644016 2019-10-10
13:38 +0000)
I'm creating an empty directory, then running firefox --new-instance --profile ./profile, then installing addon manually into the profile. But here there is that profile
In your profile there is a file: prefs.js which contains a line user_pref("extensions.lastAppBuildId", "20200707180101"); this line might be at fault for the disabled addons. You could therefore test to decrease this number by 1 or remove the whole line (untested).
profile.set_preference("extensions.lastAppBuildId", "<apppID> -1 ")
Full example code:
from selenium.webdriver import FirefoxProfile
from selenium import webdriver
path = '%APPDATA%\Mozilla\Firefox\Profiles\azk4wioue.default' #path to your profile
profile = FirefoxProfile(path)
profile.set_preference("extensions.lastAppBuildId", "<apppID> -1 ")
driver = webdriver.Firefox(profile)
Example to use an existing firefox profile:
# go to the the following folder %APPDATA%\Mozilla\Firefox\Profiles\
# there the firefox profiles should be stored, the default one ending with .default
# now provide the profile to the driver like this:
profile = FirefoxProfile('%APPDATA%\Mozilla\Firefox\Profiles\azk4wioue.default')
driver = webdriver.Firefox(firefox_profile=profile)
Alternatively install the addon clean on each run through a temporary profile.
# go to https://addons.mozilla.org and search for the plugin you want, e.g.:https://addons.mozilla.org/en-US/firefox/addon/ublock-origin/
# rightclick on the button "add to firefox"
# download the target file to a folder of your choice
# then include the addon like this:
driver.install_addon('/Users/someuser/app/extension.xpi', temporary=True)
Alternatively 2, you can try setting the extension this way:
from selenium.webdriver import FirefoxProfile
from selenium import webdriver
profile = webdriver.FirefoxProfile()
profile.add_extension(extension='/Users/someuser/app/extension.xpi')
driver = webdriver.Firefox(profile)
If the addons are there after loading the profile but disabled, you can also try this:
def newTab(fx, url="about:blank"):
wnd = fx.execute(selenium.webdriver.common.action_chains.Command.NEW_WINDOW)
handle = wnd["value"]["handle"]
fx.switch_to.window(handle)
fx.get(url) # changes handle
return fx.current_window_handle
def retoggleAllTheAddons(fx):
initialHandlesCount = len(fx.window_handles)
addonsTabHandle = newTab(fx, "about:addons")
fx.execute_script("""
let hb = document.getElementById("html-view-browser");
let al = hb.contentWindow.window.document.getElementsByTagName("addon-list")[0];
let cards = al.getElementsByTagName("addon-card");
for(let card of cards){
card.addon.disable();
card.addon.enable();
}
""")
if len(fx.window_handles) != 1:
fx.switch_to.window(addonsTabHandle)
fx.close()
i want following script as an executable... it become build and everything... but it doesnt make a token.txt when I run it (also as admin). I dont know if it dont work at all or just the file creation part.
But think its dont work at all... run the exe in cmd dont show the print ...
Also there is opening a geckodriver.exe window.
When I dont drive as admin the exe ask for firewall permission.
When I start the exe every seound time (realy only every secound time)there is comming up an error that say:
"Failed to execute script Etherscrape" (Ethersscrape is the name of the .exe)
Also is there a geckodriver.log what shows an error:
*** You are running in headless mode.
JavaScript error: resource://gre/modules/XULStore.jsm, line 66: Error: Can't find profile directory.
1591714269803 Marionette INFO Listening on port 54219
1591714270054 Marionette WARN TLS certificate errors will be ignored for this session
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from collections import defaultdict
counted = defaultdict(int)
tokenliste = []
options = Options()
options.headless = True
driver = webdriver.Firefox(firefox_options=options)
for x in range(1,10):
my_url = "https://etherscan.io/tokentxns?ps=100&p="+str(x)
driver.get(my_url)
for i in range(1,100):
xpath = "/html/body/div[1]/main/div[2]/div/div/div[2]/table/tbody/tr["+str(i)+"]/td[9]/a"
p_element = driver.find_element_by_xpath(xpath)
tokenliste.append(p_element.text)
for x in tokenliste:
counted[x] += 1
print(counted)
with open("token.txt","w",encoding="utf-8") as f:
for key, value in sorted(counted.items(), key=lambda item: item[1]):
stri = str(key)+ ": " + str(value)+ "\n"
f.write(stri)
I have to download some files from a website, I'm using Python - Selenium - Chrome - Osx.
I have his code so far:
lnk = "www.foobar.com"
CHROMEDRIVER=webdriver.Chrome()
options = webdriver.ChromeOptions()
profile = {"plugins.plugins_list": [{"enabled":False, "name":"Chrome PDF Viewer"}],
"download.default_directory" : TEMP_DOWNLOAD}
options.add_experimental_option("prefs",profile)
driver = webdriver.Chrome(chrome_options = options)
driver.get(lnk)
while True:
if filter(os.path.isfile, glob.glob(TEMP_DOWNLOAD+'/*.crdownload')):
pass
else:
break
driver.quit()
This code starts the download of the file, waits the end of the download and then closes the webdriver.
Everything is working properly except that it opens 2 Chrome windows, one to open the link and the other to download the file, and the quit() method closes only the latter.
Is there a way to kill all the windows opened by Selenium (I'm trying to avoid firing a terminal command to kill the processes brute force)?
EDIT:
as Mukesh Takhtani said in comment in my code the problem is a pointless webdriver instance.
Use this. This I used for Firefox. You can use this for Chrome. Call kill_waste() in your python code and it would kill idle useless Chrome. Please note that this would work for OSX or FreeBSD. For Linux distros you would have to change the way you are going to use grep and cut
import commands
def kill_waste():
(_,firefox_processes) = commands.getstatusoutput("ps -ax | grep '/usr/local/bin/firefox -foreground' | cut -c1-24")
sleep(0.5)
firefox_processes = firefox_processes.splitlines()
for pid in firefox_processes:
values = pid.split()
time_value = values[3].split(':')
if ((values[2] == 'I' or values[2] == 'I+') and time_value[0] == '1') or time_value[0] == '2':
commands.getstatusoutput("kill -9 " + values[0])
I am trying to create symlinks using Python on Windows 8. I found This Post and this is part of my script.
import os
link_dst = unicode(os.path.join(style_path, album_path))
link_src = unicode(album_path)
kdll = ctypes.windll.LoadLibrary("kernel32.dll")
kdll.CreateSymbolicLinkW(link_dst, link_src, 1)
Firstly, It can create symlinks only when it is executed through administrator cmd. Why is that happening?
Secondly, When I am trying to open those symlinks from windows explorer I get This Error:
...Directory is not accessible. The Name Of The File Cannot Be Resolved By The System.
Is there a better way of creating symlinks using Python? If not, How can I solve this?
EDIT
This is the for loop in album_linker:
def album_Linker(album_path, album_Genre, album_Style):
genre_basedir = "E:\Music\#02.Genre"
artist_basedir = "E:\Music\#03.Artist"
release_data_basedir = "E:\Music\#04.ReleaseDate"
for genre in os.listdir(genre_basedir):
genre_path = os.path.join(genre_basedir, "_" + album_Genre)
if not os.path.isdir(genre_path):
os.mkdir(genre_path)
album_Style_list = album_Style.split(', ')
print album_Style_list
for style in album_Style_list:
style_path = os.path.join(genre_path, "_" + style)
if not os.path.isdir(style_path):
os.mkdir(style_path)
album_path_list = album_path.split("_")
print album_path_list
#link_dst = unicode(os.path.join(style_path, album_path_list[2] + "_" + album_path_list[1] + "_" + album_path_list[0]))
link_dst = unicode(os.path.join(style_path, album_path))
link_src = unicode(album_path)
kdll = ctypes.windll.LoadLibrary("kernel32.dll")
kdll.CreateSymbolicLinkW(link_dst, link_src, 1)
It takes album_Genre and album_Style And then It creates directories under E:\Music\#02.Genre . It also takes album_path from the main body of the script. This album_path is the path of directory which i want to create the symlink under E:\Music\#02.Genre\Genre\Style . So album_path is a variable taken from another for loop in the main body of the script
for label in os.listdir(basedir):
label_path = os.path.join(basedir, label)
for album in os.listdir(label_path):
album_path = os.path.join(label_path, album)
if not os.path.isdir(album_path):
# Not A Directory
continue
else:
# Is A Directory
os.mkdir(os.path.join(album_path + ".copy"))
# Let Us Count
j = 1
z = 0
# Change Directory
os.chdir(album_path)
Firstly, It can create symlinks only when it is executed through administrator cmd.
Users need "Create symbolic links" rights to create a symlink. By default, normal users don't have it but administrator does. One way to change that is with the security policy editor. Open a command prompt as administrator, run secpol.msc and then go to Security Settings\Local Policies\User Rights Assignment\Create symbolic links to make the change.
Secondly, When I am trying to open those symlinks from windows explorer I get This Error:
You aren't escaping the backslashes in the file name. Just by adding an "r" to the front for a raw string, the file name changes. You are setting a non-existant file name and so explorer can't find it.
>>> link_dst1 = "E:\Music\#02.Genre_Electronic_Bass Music\1-800Dinosaur-1-800-001_[JamesBlake-Voyeur(Dub)AndHolyGhost]_2013-05-00"
>>> link_dst2 = r"E:\Music\#02.Genre_Electronic_Bass Music\1-800Dinosaur-1-800-001_[JamesBlake-Voyeur(Dub)AndHolyGhost]_2013-05-00"
>>> link_dst1 == link_dst2
False
>>> print link_dst1
E:\Music\#02.Genre_Electronic_Bass Music☺-800Dinosaur-1-800-001_[JamesBlake-Voyeur(Dub)AndHolyGhost]_2013-05-00
os.symlink works out of the box since python 3.8 on windows, as long as Developer Mode is turned on.
If you're just trying to create a link to a directory, you could also create a "Junction", no admin privileges required:
import os
import _winapi
src_dir = "C:/Users/joe/Desktop/my_existing_folder"
dst_dir = "C:/Users/joe/Desktop/generated_link"
src_dir = os.path.normpath(os.path.realpath(src_dir))
dst_dir = os.path.normpath(os.path.realpath(dst_dir))
if not os.path.exists(dst_dir):
os.makedirs(os.path.dirname(dst_dir), exist_ok=True)
_winapi.CreateJunction(src_dir, dst_dir)