I'm working on a GUI desktop app to give users control over a web scraper. I have two executables, one runs a Selenium scraper (Py27) and one runs the GUI which starts and stops the scraper using subprocess.Popen (Python 38). Unfortunately, using the same Python version in both apps is not possible for me right now. Although we don't seem to be hitting environment problems because of the Python version discrepancy.
I'm on Windows, using Selenium 3.141, Geckodriver 0.26, cx_Freeze 5, and Firefox 77.
I am able to run both exes manually without problem, but when I try to run the scraper exe through the GUI exe, it gets caught in an infinite loop of creating a tmp directory with User.js inside, opening a geckodriver command prompt, and then closing and starting over. It ends up continually generating more and more of these tmp directories until I kill the GUI window. The traceback tells me a few things -- Geckodriver is hitting a permissions error, and the app is having trouble establishing the home directory.
I've tested starting a scrape process using Popen, so that's not the issue.
When I run it manually and dump the Firefox options, and compare it to the Firefox options when I run it through the GUI, there is some difference. This is from the FF profile on a manual (successful) run:
'userPrefs': 'c:\\users\\atadmin\\appdata\\local\\temp\\tmpqctvna\\user.js',
'profile_dir': 'c:\\users\\atadmin\\appdata\\local\\temp\\tmpqctvna',
'extensionsDir': 'c:\\users\\atadmin\\appdata\\local\\temp\\tmpqctvna\\extensions',
and this is the profile from an unsuccessful GUI run:
'userPrefs': 'c:\\users\\atadmin\\documents\\aeleads-master\\aeleads-portal-testing\\build\\exe.win-amd64-3.8\\tmpu9ktpl\\user.js',
'profile_dir': 'c:\\users\\atadmin\\documents\\aeleads-master\\aeleads-portal-testing\\build\\exe.win-amd64-3.8\\tmpu9ktpl',
'extensionsDir': 'c:\\users\\atadmin\\documents\\aeleads-master\\aeleads-portal-testing\\build\\exe.win-amd64-3.8\\tmpu9ktpl\\extensions',
And here is the latest traceback:
Can't determine home directory
utils.tss_logging: Logging initailizing with None/None
Error: Traceback (most recent call last):
File "li_scraper.py", line 63, in run_scraper
File "C:\Users\atadmin\Documents\aeleads-master\aeleads-portal-testing\scrape\scraper.py", line 394, in collection
self.login()
File "C:\Users\atadmin\Documents\aeleads-master\aeleads-portal-testing\scrape\scraper.py", line 85, in login
driver = self._get_driver(use_proxy=self.use_proxy)
File "C:\Users\atadmin\Documents\aeleads-master\aeleads-portal-testing\scrape\scraper.py", line 622, in _get_driver
use_proxy=use_proxy, reserve_proxy=False)
File "C:\Users\atadmin\Documents\aeleads-master\aeleads-portal-testing\lib\webdrivers.py", line 164, in open_webdriver
capabilities=capabilities, log_path="C:\\Users\\atadmin\\Documents\\aeleads-master\\geckodriver.log")
File "C:\Python27\Py27\lib\site-packages\selenium\webdriver\firefox\webdriver.py", line 174, in __init__
keep_alive=True)
File "C:\Python27\Py27\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 157, in __init__
self.start_session(capabilities, browser_profile)
File "C:\Python27\Py27\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 252, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "C:\Python27\Py27\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "C:\Python27\Py27\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
WebDriverException: Message: permission denied
It seems like cx_Freeze creates its own build environment, maybe causing me to lose the user, based off the permissions error. I'm fairly new to Python.
Figured it out. When passing the environment into Popen, I was passing a stripped out environment that didn't include the TEMP path. I passed in my full environment and then changed the variables that needed to be changed, and now it is working.
Related
I want to execute my python script by a batch file. I have no problem if i execute my python script manually, but if i use a batch file to execute it then i get the "'geckodriver' executable needs to be in Path" error. In both ways I use the same interpreter.
My batch file:
C:\Users\murac\PycharmProjects\IP_Klima_Strom\venv\Scripts\python.exe C:\Users\murac\PycharmProjects\IP_Klima_Strom\main.py
pause
my error warning:
Traceback (most recent call last):
File "C:\Users\murac\PycharmProjects\IP_Klima_Strom\main.py", line 166, in <module>
driver = webdriver.Firefox(firefox_profile=browser_profile, options=opts)
File "C:\Users\murac\PycharmProjects\IP_Klima_Strom\venv\lib\site-packages\selenium\webdriver\firefox\webdriver.py", line 164, in __init__
self.service.start()
File "C:\Users\murac\PycharmProjects\IP_Klima_Strom\venv\lib\site-packages\selenium\webdriver\common\service.py", line 83, in start
os.path.basename(self.path), self.start_error_message)
selenium.common.exceptions.WebDriverException: Message: 'geckodriver' executable needs to be in PATH.
My path Confi
I alreay updated my geckodriver and updated the path too, but nothing has changed. DO you have any idea to fix this?
i guess if you put you geckodriver in the same folder with script its gonna work
I'm trying to run a python script when the system boots on a Windows 10 workstation. My python script runs fine when I kick it off in a different directory as a user with administrative privileges but when I run it from a bat file in the C:\Windows\System32\GroupPolicy\Machine\Scripts\Startup folder on system boot I receive the following error in my logs.
2018-11-13 04:43:58,913 - VP - Level 55 - nmap args: ['"C:\\Program Files (x86)\\Nmap\\nmap.exe"', '-oX', 'C:\\\\nmap\\NmapResults\\scan-192.168.1.5-20181113044358.xml', '-sS', '-n', '-p-', '--allports', '--scan-delay', '1ms', '-sU', '-sV', '--version-all', '192.168.1.5']
Traceback (most recent call last):
File "secnmap\nmap_scanner.py", line 596, in <module>
exit_code = main()
File "secnmap\nmap_scanner.py", line 248, in main
run_nmap(args, scan_options, host_data_list, use_service_name_check)
File "secnmap\nmap_scanner.py", line 268, in run_nmap
nmap_proc = subprocess.Popen(nmap_cmd_list, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
File "C:\Python27\lib\subprocess.py", line 390, in __init__
errread, errwrite)
File "C:\Python27\lib\subprocess.py", line 640, in _execute_child
startupinfo)
WindowsError: [Error 5] Access is denied
Here is the content of the .bat file.
cd C:\\nmap_startup
python nmap_startup.py
I have tried using the full nmap.exe path as shown in the code snippet as well as plain old nmap. I get the access denied error either way. I didn't see much in subprocess.py that would lead me to believe this to be an error with the python code either so I am leaning on it being an issue with System user running the script. If someone thinks otherwise or has any ideas, I'd love to hear them. Thanks
EDIT: I chose to run from a bat file because my python version is 32bit and I was having issues running from the C:\windows\system32 directory
It turns out that the path to nmap, C:\Program Files (x86)\Nmap, was not included in the system environment variables. When the script ran as a System user, it did not recognize the location of nmap
I have a web scraping script in Python, using Selenium and (tor) Firefox. This runs completely fine when I start the script manually, in my IDE or from cmd. However, this script needs to be run when I am not in the office, so I run it via a batch script (along with some other scraping scripts) which is started by the windows task scheduler. Again, when I run this batch file manually, the script runs perfectly. When the scheduled task runs, on the other hand, it fails as soon as the webdriver is set with the following unhelpful error message:
Traceback (most recent call last):
...
#private library traceback
...
File "redacted.py", line 322, in redacted_func
driver = webdriver.Firefox()
File "C:\Program Files\Python 3.5\lib\site-packages\selenium\webdriver\firefox\webdriver.py", line 152, in __init__
keep_alive=True)
File "C:\Program Files\Python 3.5\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 98, in __init__
self.start_session(desired_capabilities, browser_profile)
File "C:\Program Files\Python 3.5\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 188, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "C:\Program Files\Python 3.5\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 256, in execute
self.error_handler.check_response(response)
File "C:\Program Files\Python 3.5\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 194, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: Process unexpectedly closed with status: 0
Does anyone have any idea why this error occurs only when run from the task scheduler?
After further investigation, I found that the error happens when the task is processed in the background. This happens when the task is set to 'Run whether user is logged on or not'. Changing this setting to 'Run only when user is logged on' allows the task to run without issue, in the foreground.
For some reason, running in the background causes firefox to crash, but chromedriver (which is used in some other scraping scripts I run) is unaffected by this. This seems slightly bizarre, but this workaround is sufficient to get this running on windows.
Saw a lot of people have had problems like this, but in all my searches I saw a lot of conflicting and confusing information that I didn't understand - this is all a bit out of my newbie depth.
I installed Selenium in PyCharm and was attempting to run this code from the book Automate The Boring Stuff with Python:
from selenium import webdriver
browser = webdriver.Firefox()
browser.get('http://inventwithpython.com')
linkElem = browser.find_element_by_link_text('Read It Online')
type(linkElem)
linkElem.click() # follows the "Read It Online" link
Running it throws the following exceptions:
Traceback (most recent call last):
File "C:\Users\LB\AppData\Local\Programs\Python\Python36-32\lib\site-packages\selenium\webdriver\common\service.py", line 74, in start
stdout=self.log_file, stderr=self.log_file)
File "C:\Users\LB\AppData\Local\Programs\Python\Python36-32\lib\subprocess.py", line 707, in __init__
restore_signals, start_new_session)
File "C:\Users\LB\AppData\Local\Programs\Python\Python36-32\lib\subprocess.py", line 990, in _execute_child
startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:/Users/LB/Desktop/PythonProjects/AutomateTheBoringStuffProjects/generalTestingFile.py", line 2, in <module>
browser = webdriver.Firefox()
File "C:\Users\LB\AppData\Local\Programs\Python\Python36-32\lib\site-packages\selenium\webdriver\firefox\webdriver.py", line 142, in __init__
self.service.start()
File "C:\Users\LB\AppData\Local\Programs\Python\Python36-32\lib\site-packages\selenium\webdriver\common\service.py", line 81, in start
os.path.basename(self.path), self.start_error_message)
selenium.common.exceptions.WebDriverException: Message: 'geckodriver' executable needs to be in PATH.
I've downloaded the latest geckodriver.exe (specifically the 64bit version since I'm on 64bit OS but I'm running 32bit Firefox if that's okay?), but I'm not sure where to put it.
I've looked up how to change a PATH but I don't know what exactly I'm supposed to change or where it's supposed to point to. (Firefox folders? Python folders?)
Followed someone's advice to put geckodriver.exe in C:\Users\LB\ and edit the System Path to ADD that location to the Variable called Path. And now the code works! (You can apparently put geckodriver.exe anywhere, as long as you point the path to that specific folder.)
Source: https://www.howtogeek.com/118594/how-to-edit-your-system-path-for-easy-command-line-access/
This answer was posted as a comment to the question How to set up Python 3 Selenium/Geckodriver for Firefox on Windows 10? by the OP LBoot.
I'm fairly new to programming and decided to setup a simple python script that would open all the applications I use for webapp development. The code I am using is (for GAE):
google_appengine = r'C:\Applications\google_app_engine\launcher\GoogleAppEngineLauncher.exe'
subprocess.Popen(google_appengine)
This works fine for the other programs I am opening, but I am unable to run any applications within App Engine after I have opened it this way. I get the following error in my App Engine log file:
Exception in thread Thread-2:
Traceback (most recent call last):
File "threading.pyc", line 486, in __bootstrap_inner
File "launcher\taskthread.pyc", line 65, in run
File "subprocess.pyc", line 587, in __init__
File "subprocess.pyc", line 700, in _get_handles
File "subprocess.pyc", line 745, in _make_inheritable
WindowsError: [Error 6] The handle is invalid
I'm guessing it is the way subprocess.Popen() works, but I haven't been able to find any alternatives. I'm running Windows 7 if that makes a difference. Thanks for looking.
if you want to manage the local dev_appserver, this is the wrong approach.
the best way to do this is clone the sdk repository (https://code.google.com/p/googleappengine/) directly to your drive and then add that path to your environment PYTHONPATH variable.
here's a link to a script template i created & often use to manage startup & killing of the dev_appserver process: https://gist.github.com/4514647
i'm not too familiar with managing a python environment on Windows, so you'd have to take my notes on a highlevel and research the specific implementation for that platform.