I have to download some files from a website, I'm using Python - Selenium - Chrome - Osx.
I have his code so far:
lnk = "www.foobar.com"
CHROMEDRIVER=webdriver.Chrome()
options = webdriver.ChromeOptions()
profile = {"plugins.plugins_list": [{"enabled":False, "name":"Chrome PDF Viewer"}],
"download.default_directory" : TEMP_DOWNLOAD}
options.add_experimental_option("prefs",profile)
driver = webdriver.Chrome(chrome_options = options)
driver.get(lnk)
while True:
if filter(os.path.isfile, glob.glob(TEMP_DOWNLOAD+'/*.crdownload')):
pass
else:
break
driver.quit()
This code starts the download of the file, waits the end of the download and then closes the webdriver.
Everything is working properly except that it opens 2 Chrome windows, one to open the link and the other to download the file, and the quit() method closes only the latter.
Is there a way to kill all the windows opened by Selenium (I'm trying to avoid firing a terminal command to kill the processes brute force)?
EDIT:
as Mukesh Takhtani said in comment in my code the problem is a pointless webdriver instance.
Use this. This I used for Firefox. You can use this for Chrome. Call kill_waste() in your python code and it would kill idle useless Chrome. Please note that this would work for OSX or FreeBSD. For Linux distros you would have to change the way you are going to use grep and cut
import commands
def kill_waste():
(_,firefox_processes) = commands.getstatusoutput("ps -ax | grep '/usr/local/bin/firefox -foreground' | cut -c1-24")
sleep(0.5)
firefox_processes = firefox_processes.splitlines()
for pid in firefox_processes:
values = pid.split()
time_value = values[3].split(':')
if ((values[2] == 'I' or values[2] == 'I+') and time_value[0] == '1') or time_value[0] == '2':
commands.getstatusoutput("kill -9 " + values[0])
Related
i want following script as an executable... it become build and everything... but it doesnt make a token.txt when I run it (also as admin). I dont know if it dont work at all or just the file creation part.
But think its dont work at all... run the exe in cmd dont show the print ...
Also there is opening a geckodriver.exe window.
When I dont drive as admin the exe ask for firewall permission.
When I start the exe every seound time (realy only every secound time)there is comming up an error that say:
"Failed to execute script Etherscrape" (Ethersscrape is the name of the .exe)
Also is there a geckodriver.log what shows an error:
*** You are running in headless mode.
JavaScript error: resource://gre/modules/XULStore.jsm, line 66: Error: Can't find profile directory.
1591714269803 Marionette INFO Listening on port 54219
1591714270054 Marionette WARN TLS certificate errors will be ignored for this session
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from collections import defaultdict
counted = defaultdict(int)
tokenliste = []
options = Options()
options.headless = True
driver = webdriver.Firefox(firefox_options=options)
for x in range(1,10):
my_url = "https://etherscan.io/tokentxns?ps=100&p="+str(x)
driver.get(my_url)
for i in range(1,100):
xpath = "/html/body/div[1]/main/div[2]/div/div/div[2]/table/tbody/tr["+str(i)+"]/td[9]/a"
p_element = driver.find_element_by_xpath(xpath)
tokenliste.append(p_element.text)
for x in tokenliste:
counted[x] += 1
print(counted)
with open("token.txt","w",encoding="utf-8") as f:
for key, value in sorted(counted.items(), key=lambda item: item[1]):
stri = str(key)+ ": " + str(value)+ "\n"
f.write(stri)
The newest Firefox version 67 has private browsing add-ons disabled. These can be added by following this guide. https://support.mozilla.org/en-US/kb/extensions-private-browsing
The issue is that even using that guide the add-ons are not consistently enabled under the geckodriver.
This was functioning in Firefox v66 without issue.
I have run an instance with the default firefox profile (Profile0 from profile.ini) with two add-ons (coil and LastPass). I see the addons popup for about 2 seconds and then disappear.
I have manually enabled them inside the private browser by going to about: addons. It appears that if they are already enabled in the browser for private browser use and are not visible in private browsing changing permissions to disallow still enables them in private browsing, then allowing them keeps them enabled. Very strange.
I uninstalled an addon closed all browsers, started the script again and it worked the first time. Then on the second run, it was back to the 2 seconds enabled then disappear mode as mentioned above.
I've checked the temp profile created under temp when geckodriver copies the profile for consumption. The add-ons are included.
Manually opening up a private browser shows two add-ons so it appears specific to geckodriver + firefox v67. However, the private browser doesn't have me logged in on my addons.
def get_firefox_profile_dir(self):
from pathlib import Path
self.gecko_path = os.path.dirname(__file__)
if sys.platform in ['linux', 'linux2']:
import subprocess
self.ff_gecko = Path(self.gecko_path + '/geckodriver')
bits = 'uname -m'
ver_32_64 = subprocess.getstatusoutput(bits)
cmd = "ls -d /home/$USER/.mozilla/firefox/*.default/"
fp = subprocess.Popen([cmd], shell=True, stdout=subprocess.PIPE)
FF_PRF_DIR = fp.communicate()[0][0:-2]
FF_PRF_DIR_DEFAULT = str(FF_PRF_DIR, 'utf-8')
ff_ext_path = os.path.join(FF_PRF_DIR_DEFAULT, 'extensions')
self.ff_coil_loc = os.path.join(ff_ext_path, self.ff_coil_extId)
ff_coil_enabled = os.path.exists(self.ff_coil_loc)
if ff_coil_enabled:
if 'x86_64' in ver_32_64:
if not self.ff_gecko.is_file():
import wget
self.gecko_targz = 'geckodriver-v0.24.0-linux64.tar.gz'
wget.download(self.gecko_source_linux64, self.gecko_path)
self.file_unzip_tar(self.gecko_path + '/' + self.gecko_targz)
os.remove(self.gecko_path + '/' + self.gecko_targz)
if self.ff_gecko.is_file():
self.data_path = FF_PRF_DIR_DEFAULT
self.gecko = self.ff_gecko
return
if 'i368' in ver_32_64:
if not self.ff_gecko.is_file():
import wget
self.gecko_targz = 'geckodriver-v0.24.0-linux32.tar.gz'
wget.download(self.gecko_source_linux32, self.gecko_path)
self.file_unzip_tar(self.gecko_path + '/' + self.gecko_targz)
os.remove(self.gecko_path + '/' + self.gecko_targz)
if self.ff_gecko.is_file():
self.data_path = FF_PRF_DIR_DEFAULT
self.gecko = self.ff_gecko
return
elif sys.platform == 'win32' or 'nt':
from pathlib import Path
self.gecko = self.gecko_path + "\geckodriver.exe"
mozilla_profile = os.path.join(os.getenv('APPDATA'), r'Mozilla\Firefox')
mozilla_profile_ini = os.path.join(mozilla_profile, r'profiles.ini')
profile = configparser.ConfigParser()
profile.read(mozilla_profile_ini)
FF_PRF_DIR_DEFAULT = os.path.normpath(os.path.join(mozilla_profile, profile.get('Profile0', 'Path')))
ff_ext_path = os.path.join(FF_PRF_DIR_DEFAULT, 'extensions')
self.ff_coil_loc = os.path.join(ff_ext_path, self.ff_coil_extId)
ff_coil_enabled = os.path.exists(self.ff_coil_loc)
if ff_coil_enabled:
ff_gecko = Path(self.gecko)
if ff_gecko.is_file():
self.data_path = FF_PRF_DIR_DEFAULT
return
else:
import wget
wget.download(self.gecko_source_win64)
self.file_zunip('geckodriver-v0.24.0-win64.zip')
if ff_gecko.is_file():
os.remove('geckodriver-v0.24.0-win64.zip')
self.data_path = FF_PRF_DIR_DEFAULT
return
self.get_firefox_profile_dir()
self.driver = webdriver.Firefox(options=self.options, firefox_profile=self.data_path, executable_path=self.gecko)
self.driver.get(self.url) # OPEN URL
I expect the FF addons to be enabled as they have been manually permitted from the browser and that they will be logged in as they are in normal mode.
It appears that the core issue was related to the upgrade to v67 and some unknown items in the profile became broken. A profile reset fixed the issue.
https://support.mozilla.org/en-US/kb/refresh-firefox-reset-add-ons-and-settings
UPDATE: After futher testing I see that it shows short term stability and that extensions are hit and miss (more miss) when loading dynamically. In other words after profile reset addons work the first few times and then they do not load anymore with selenium
I 've a porgram called searchMachine.py that if I run it straight from IDLE (F5..)it works just fine, but If I run it by just clicking on searchMachine.py it works untill somepoint and then I get an error. See below.
Thanks for any input you could give me.
By the way, any idea how I can make this chrome work in silent? for example minimized?
from selenium import webdriver
from selenium.webdriver.support.ui import Select
print('Please type the machine name.')
machinename = input()
print('')
print('Thank you.')
print('')
print('Results will show below.')
print('')
path_to_chromedriver = 'C:\python34\chromedriver\chromedriver.exe' # change path as needed
browser = webdriver.Chrome(executable_path = path_to_chromedriver)
url = 'http://app.corp'
browser.get(url)
browser.find_element_by_xpath("/html/body/div[4]/form/table[1]/tbody/tr/td[5]/input[1]").send_keys(machinename)
browser.find_element_by_css_selector('input[type=\"submit\"]').click() # working to press submit
xpath = '/html/body/div[4]/form/center[2]/table/thead'
for i in browser.find_elements_by_xpath(xpath):
print (i.text)
xpath = '/html/body/div[4]/form/center[2]/table/tbody/tr'
for i in browser.find_elements_by_xpath(xpath):
print (i.text)
browser.close()
import os
os.system("c:/code/close_chrome_driver.bat")
error is:
The error occurs because the unicode character ▾ or U+25BE BLACK DOWN-POINTING SMALL TRIANGLE cannot be written into Windows console window, as it does not have a match in the CP850 character set as used by your Windows console.
Now, Python croaks loudly when it cannot print the correct value. You could set the PYTHONIOENCODING environment variable to say cp850:ignore which would make the Python process ignore errors; or you can print to stderr instead of stdout, which would use the backslashescape error handling.
I have a cron job that loads a Python script on reboot but it will just not work.
I have checked the Python script and that works fine from CLi.
The .py basically loads a browser to Google and then sends it to full screen.
(It actually loads another website and enters login details also, but removed for obvious reasons)
Been at this for weeks now and driving me crazy, any ideas?
Raspberry Pi running Raspbian.
$crontab -e
#reboot DISPLAY=:0 python /prtgboot.py
prtgboot.py
'#'!/usr/bin/env python
import commands
import time
webbrowser = "iceweasel"
pgrepcmd = "pgrep %s " % (webbrowser)
process = commands.getoutput(pgrepcmd)
if process == "":
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver import ActionChains
browser = webdriver.Firefox()
actions = ActionChains(browser)
browser.get('http://google.co.uk')
elemFullscreen = browser.find_element_by_tag_name('html')
time.sleep(30)
elemFullscreen.send_keys (Keys.F11)
exit()
else:
exit()
Ok, so Petesh was correct. It was #reboot not working correctly.
Changed the cron to * * * * * so my script runs every minute. Normally bad practice, but already setup script to end if browser already running. Working a treat now.
On a positive note, if the browser crashes it will start again for me :)
I want to know what is the current url of active tab in running firefox instance from python module. Does FireFox have any API for this and does python know to work with it?
The most convenient way maybe insatll a firefox extension to open up a tcp service, then you can exchange info with firefox.
mozrepl can set up a telnet service, you can call js-like command to get info.
With telnetscript (http: //code.activestate.com/recipes/152043/), you can write:
import telnetscript
script = """rve
w content.location.href;
ru repl>
w repl.quit()
cl
"""
conn = telnetscript.telnetscript( '127.0.0.1', {}, 4242 )
ret = conn.RunScript( script.split( '\n' )).split( '\n' )
print ret[-2][6:]
If on windows you can use win32com
import win32clipboard
import win32com.client
shell = win32com.client.Dispatch("WScript.Shell")
shell.AppActivate('Some Application Title')
Then use shell.SendKeys to do a ctrl+l and a ctrl+c
Then read the string in the clipboard.
It's horkey though it will work, alternatly you can use something like AutoIt an compile the code to an exe that you can work with.
Hope this helps.