I have a scraping python script and a batch file that when run from CMD works perfectly however when I try to run it from Task Scheduler nothing happens.
I know there are a lot of questions regarding the same issue but I have tried all of the proposed answers and non of them seem to work.
Don't know if this is relevant but script would open Firefox and scrape some websites.
Have tried adding full permissions to the folders and files that I am using.
Also, have tried in Task Scheduler to set up "Run wheter user is logged on or not", "Run with highest privileges" , "Start in (optional): add/bactch/file/path" and so on
Batch file:
py "C:\python_test\myscript.py"
It should run the python script which opens Firefox and scrapes some websites, gets their links and saves them in a csv file
Here is myscript.py:
import datetime
import time
import csv
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
import os
file_path = r"C:\General\{0:%Y%m%d}\results{0:%Y%m%d%H%M%S}.csv".format(datetime.datetime.now())
directory = os.path.dirname(file_path)
try:
os.stat(directory)
except:
os.mkdir(directory)
from bs4 import BeautifulSoup
def init_driver():
caps = DesiredCapabilities.FIREFOX
caps['marionette'] = True
driver = webdriver.Firefox(capabilities=caps)
driver.wait = WebDriverWait(driver, 15)
return driver
xpath = {
'english': '//*[#id="xxx"]',
'soort': '//*[#id="xxx"]/option[text()=\'|- Announcement of change in denominator or in thresholds\']',
'start': '//*[#id="xxx"]',
'end': '//*[#id="xxx"]',
'submit': '//*[#id="xxx"]',
'pub_sort': '//*[#id="xxx"]',
}
if __name__ == "__main__":
driver = init_driver()
try:
driver.get("http://SOMEWEBSITE")
driver.find_element_by_css_selector('[id$=hplEnglish]').click()
except Exception:
DoNothing = ""
time.sleep(2)
driver.find_element_by_css_selector('[id$=hplEnglish]').click()
time.sleep(3)
#alert_obj = driver.switch_to.alert
#alert_obj.dismiss()
#driver.find_element_by_xpath(xpath['english']).click()
today = datetime.datetime.now()
driver.wait.until(EC.element_to_be_clickable((By.XPATH, xpath['soort']))).click()
driver.find_element_by_xpath(xpath['start']).send_keys((today-datetime.timedelta(weeks=1)).strftime('%d/%m/%Y'))
driver.find_element_by_xpath(xpath['end']).send_keys(today.strftime('%d/%m/%Y'))
driver.find_element_by_xpath(xpath['submit']).click()
for i in range(2):
driver.wait.until(EC.element_to_be_clickable((By.XPATH, xpath['pub_sort']))).click()
time.sleep(5)
html = driver.page_source
driver.quit()
results = BeautifulSoup(html, 'html.parser').find('div', { 'id': 'SearchResults'}).table
res = [['issuer', 'type', 'date']]
for x in results.find_all('tr')[1:]:
# print(x.text.strip())
try:
a, b, c = [i.text.strip() for i in x.find_all('td', class_='resultItemTop')[:3]]
res.append([a,b,c])
except ValueError:
continue
with open(file_path, 'w', newline='') as result:
writer = csv.writer(result, delimiter=',')
writer.writerows(res)
print('finished')
Introduction
I suggest to read first What must be taken into account on executing a batch file as scheduled task?
Issue 1: Current directory
The first issue here is most likely the current directory on running the batch file.
Windows sets the directory of the batch file as current directory on double clicking a batch file, except the batch file path is a UNC path starting with \\computername\share\.
The current directory for a scheduled task is by default %SystemRoot%\System32, i.e. the Windows system directory which of course is special protected against modifications. Many batch files expect that the current directory is the directory of the batch file and not any other directory.
Solution 1: Define Start in directory in properties of scheduled task.
Start Windows Task Scheduler.
Navigate to the task and double click on it to open the Properties of the task.
Select tab Action and click on button Edit.
There is Start in (optional). Enter here the path of the executed batch file.
Click two times on button OK to save this important modification in properties.
Solution 2: Make batch file directory the current directory using CD.
In the batch file insert after first line being usually #echo off the line:
cd /D "%~dp0"
This command line changes the current directory from default %SystemRoot%\System32 to the directory of the batch file as long as the batch file is not started using a UNC path.
Open a command prompt window and run cd /? for help on command CD and option /D.
Solution 3: Make batch file directory the current directory using PUSHD.
This solution is best if the batch file is stored on a network resource accessed using UNC path and of course the scheduled task is configured to run with credentials (user account and password) with permissions to read content of the batch file on the network resource.
In the batch file insert after first line being usually #echo off the lines:
setlocal EnableExtensions DisableDelayedExpansion
pushd "%~dp0"
The batch file should additionally contain as last two line executed before exiting batch file processing the two lines:
popd
endlocal
Open a command prompt window and run pushd /?, popd /?, setlocal /? and endlocal /? for help on these four commands and read also this answer for even more details on these four commands.
Solution 4: Code everything to be independent on current directory.
The fourth solution is writing batch file and Python script for being independent on which directory is the current directory on execution of batch file and Python script.
This solution requires that all files and directories are specified with full qualified file/folder name which means full path + file/folder name + file extension.
The full file/folder paths can be derived from known paths on execution like path of the batch file which can be referenced with %~dp0 inside the batch file and which expands to a path string always ending with backslash which means it can be concatenated with file/folder names without usage of an additional backslash.
See also Wikipedia article about Windows Environment Variables.
Issue 2: Environment variables
Scheduled tasks are often executed with built-in local system account. So all environment variables defined just for the user account used on developing and testing the batch file either are not defined at all or are defined different.
Open from Windows Control Panel from item System the Advanced system settings or press key combination Win+Break if keyboard has a key Break (often as alternate function requiring pressing additionally the key Fn) and clicking next on Advanced system settings on left side. The System Properties dialog is opened wit tab Advanced selected containing at bottom the button Environment Variables... which must be clicked to open the Environment Variables window.
There are two lists of environment variables: User variables for ... and System variables. The system variables are defined for all accounts including built-in local system account. The user variables are defined only for the displayed user.
The user variables defined for current user are important and could be a reason why a batch file executed with a double click by current user works, but does not work on being executed as scheduled task with built-in system account. A user PATH is often a main source of not working batch file on execution as scheduled task if the used scripts and executables depend on specific folder paths in local PATH defined in user PATH.
Please take a look on What is the reason for "X is not recognized as an internal or external command, operable program or batch file"? for more information about system, user and local PATH and environment variable PATHEXT used on writing in a batch file just py instead of the script/executable file with full qualified file name.
So it is definitely better to use
"C:\Python27\python.exe" "C:\python_test\myscript.py"
instead of using
py "C:\python_test\myscript.py"
which results in cmd.exe searching around for a file py using local PATHEXT and local PATH environment variables which can file if the folder containing file py is defined in user PATH.
I have not installed Python and so don't know what py really is. It could be a batch file with file name py.cmd or py.bat in which case the command CALL must be used if the batch file contains additional command lines after the command line with py and which perhaps depends on user environment variables. It could be a symbolic link to python.exe in installation folder of Python. I don't know.
Issue 3: Network resources
Many scheduled tasks access files, folders or data via a network. In this case the scheduled task must be configured to run with the credentials (user account and password) which has the required permissions to access the files, folders or data on the network resource. The usage of the built-in local system account is in this case nearly never the right choice to run the scheduled task because of local system account has usually no permissions to read/write data on any network resource.
Conclusion
It is good practice in my opinion to write a batch file being executed as scheduled task to be as independent as possible on other batch files and environment variables not defined in the batch file executed by Windows task scheduler itself. A design with dependencies on other script files and variables defined outside main script cause often sooner or later (sometimes years later) unexpected execution problems.
The programmer of a script written for execution as scheduled task should really know very well on which files, libraries, environment variables and registry keys/values the script and called executables depend on to work properly.
A batch file containing only one line to execute one application with some parameters is completely unnecessary and just a potential source of not working scheduled task because of the executable to run and its parameters can be in this case also directly set in properties of the scheduled task. There is absolutely no need to run %SystemRoot%\System32\cmd.exe with implicit option /C to process a specified batch file containing only one command line to execute an application with zero or more parameters because of Windows task scheduler can directly run the application with its parameters, too.
Related
I am a new Robot Framework user and have a question regarding the log file that it makes at the end of executing testcases. I would like to use the html file it creates and upload it automatically to the correct ticket. I already have python code that works to upload the file and it can be used as a keyword, but I am not sure how I can call upon that keyword as a test teardown step as at that point the logging probably is not created yet..
Is this correct and if so: is there another way to automatically call a python function to upload the html file after executing a testcase?
Yes, you can use things as below:
Test Teardown Run keyword if test Passed/Failed Name_of_kw
Add this to your settings section of .robot file.
Now, define the kw: (Import Process Library first in your robot file)
Name_of_kw
Start Process <tab> python <tab> path_to_file.py<tab> alias=prog
Wait for Process prog #wait until it gets completed
Get Process Result prog stdout=yes #to make sure you have uploaded it
================
more details at -https://robotframework.org/robotframework/latest/libraries/Process.html#library-documentation-top
I've been facing the same problem and I've got an alternative solution for getting log files (log.html, report.html, output.xml) in same execution and upload them to a FTP Server.
I created a python script in Robot Framework's root project folder:
import subprocess
import os.path
import time
arguments = sys.argv
subprocess.run(arguments)
logs = ['log.html', 'report.html', 'output.xml']
while not os.path.exists(logs[0] and logs[1] and logs[2]):
time.sleep(1)
do_something()
Instead of running:
robot .\tests\...\suite
Run:
python script.py robot .\tests\...\suite
In that way, you will be able to work with output files when tests or suites are finished.
If you run robot's command with -d flag to save results on a different folder, consider using automatic variable from Robot Framework called ${OUTPUT DIR} to get full source path and save it to txt file on root folder. That's because you'll be able to read it and find the logs in your script.py.
So I have a server which runs on windows system, and I need to schedule a daily task with windows task scheduler for a batch file which executes a python script which contains selenium operation (I am using a chrome driver), the expected outcome is to automatically download a file from online and then unzipped it to another folder. i.e. Program download the file and stored it in C:\download then unzipped it to E:\Data.
Everything is working fine if I set "Run only when user is logged on" in windows scheduler or I just ran the program manually. However, when I set "Run whether user is logged on or not", I won't see any file gets downloaded, so I suspect in order to let it work, one must log on to get webdriver interacts with website? I have refer to this site and seems like it is the case though the author is using edge (https://developer.microsoft.com/en-us/microsoft-edge/platform/issues/7290550/). Can anyone help to confirm this or can someone offer me a solution?
Moreover, if I manually leave the zipped file in i.e. C:\download, when the task starts with "Run whether user is logged on or not", I can see the file gets unzipped in the destination folder i.e. E:\Data which means part of my script (the extract part) is executed. So I am positive the task started successfully, but there's just no interaction in between selenium and website. Thanks.
P.S. My script does not contain any mapped driver, so this solution doesn't apply (https://social.technet.microsoft.com/Forums/windows/en-US/c03d6691-b058-4f8d-961c-e8eba25bbaed/task-scheduler-problem-run-whether-user-is-logged-on-or-not?forum=w7itprogeneral), and neither does this help (Task scheduler cannot open batch file when set to run whether user is logged on or not).
I've got a process I'm trying to automate through the Windows task scheduler. The file is located right next to the script on a subfolder in my desktop. A mockup that shows the problem I'm running into is:
import os
if __name__ == "__main__":
lines = []
with open(r'C:\Users\user\Desktop\folder\config.txt') as file:
for line in file:
lines.append(line)
with open(r'C:\Users\user\Desktop\folder\output.txt', 'w') as file:
for line in lines:
file.write(line)
When I run this through the command line, the code works fine. The config file is read in properly and the output works properly.
However, in task scheduler, the program fails to execute with an 0x2 error code. In task scheduler, I have the tried with the following task scheduler options selected:
Run only when user is logged on/Run with highest priveledges Run only
when user is logged on/Highest priveledges not checked Run whether
user is logged on or not/Run with highest priveledges Run whether
user is logged on or not/Highest priveledges not checked
I am using absolute paths - as you can see in the above example. I'm also using the "start in" option inside of task scheduler.
This is seriously annoying me.
Edit: I am using absolute paths inside of task scheduler to both the Python executable and the script.
I have a program which is running on daily basis. I would like to have a log created for each run of it. Here is the snip of code responsible for logging:
logging.basicConfig(filename = 'log.txt', level = logging.DEBUG, format = '%(asctime)s - %(levelname)s - %(message)s')
logging.debug('Start of program') # example logging
Everything runs perfectly fine as long as I launch it by .py file or .bat file (mouse clicking in file explorer). Unfortunately, when I put it on the schedule the program runs fine but log file doesn't get created.
I have tried multiple scheduler settings but the problem seems to be located in the code of the script.
Thanks for feedback!
I wanted to add the solution as an answer too, since this seems to be a somewhat common problem which has caused me some grief in the past.
When executing something from the Windows Task Scheduler the working directory of that process is not by default the path of the executable but some other directory. E.g. when your Scheduler action calls C:\sample.py this is not executed in C:\ but C:\Windows\system32 (in my case).
You can verify this by adding a Scheduled Task calling a simple Python/Batch script which saves the active working directory to some file, such as this
import os
with open("C:\\cwd.txt", "w") as fh:
fh.write(os.getcwd())
or this
echo %cd% >> C:\cwd.txt
This messes with any relative paths your program may contain and, I suspect, it has some more subtle issues that I have not been able to pin down yet.
Problems can be avoided by explicitly setting the optional 'Start in' path for your action:
Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 7 years ago.
Improve this question
I have a python which can run normally but cannot be scheduled to run successfully on Windows 7 Task scheduler. I even created a batch file to call the python script.
Under the task action ("Start a program"), I have;
C:\Backup\backup.bat
It is so simple that I cannot find anything that I have done wrong. Anything else that I need to take note?
Create on drive C: a directory Temp and make sure that security permissions are set to full control for everyone.
Put into your batch file at top:
#echo off
echo Current directory: %CD%>C:\Temp\Environment.txt
echo.>>C:\Temp\Environment.txt
echo Environment variables:>>C:\Temp\Environment.txt
echo.>>C:\Temp\Environment.txt
set >>C:\Temp\Environment.txt
When you double click on your batch file it writes the current directory to file C:\Temp\Environment.txt which will be the directory of the batch file. And it writes all environment variables defined for your user account also to file C:\Temp\Environment.txt.
Now rename Environment.txt to DoubleClickedEnvironment.txt.
Then do what is necessary to run the same batch file as scheduled task and later look on C:\Temp\Environment.txt.
You will see most likely by comparing C:\Temp\Environment.txt with C:\Temp\DoubleClickedEnvironment.txt that the current directory is now C:\Windows\System32 (respectively %SystemRoot%\System32) instead of the directory containing the batch file and the list of environment variables as well as their values differ.
Most important on environment variables are PATH and PATHEXT when not referencing executables in batch file with name of file with extension and with full path enclosed in double quotes if name or path contains 1 or more spaces. Also all environment variables defined for Python and evaluated by Python are important on your batch file.
Another common mistake on running something as scheduled task is thinking that the used account for the scheduled task has same permissions on accessing files and directories as the current user. This is not the case if the scheduled task is not executed with using your user account.
And last mapped network drives are not mapped on running a batch file as scheduled task. Mapping network drives is done by Windows only on login of a user. So in batch file designed for running as scheduled task
the UNC paths must be used, or
the commands pushd and popd are used to map a network share temporarily to a drive letter using credentials of the account defined for the scheduled task, or
%SystemRoot%\System32\net.exe X: \\ComputerName\ShareName password /user:domain\username /persistent:no
is used at beginning of batch file and
%SystemRoot%\System32\net.exe X: /delete
is used at end of batch file as an example for drive X:.
The last method is very insecure as this makes it possible to everyone with permissions to read the batch file to get user name and password for the share.