The problem: we wish to run an Azure Devops pipeline which contains a step that runs a python command "mkdocs" to generate documentation. However, the pipeline does not have access to this script.
How are you supposed to access python scripts inside a Windows Service?
Environment
The pipeline runs on a self hosted Windows agent (more info). This pipeline runs as a Windows Service, with a "Network Service" account. Important to note is that this account does not have access to the AppData folder, because it is not a user.
mkdocs is a python package installed using pip, which comes with a mkdocs.exe which we attempt to run in the pipeline. This package is normally installed under the user's AppData folder.
Attempt 1:
Simply adding C:\Users\[REDACTED]\AppData\Roaming\Python\Python310\Scripts to PATH. This gives the following expected output in the logs of the pipeline:
##[error]'mkdocs' is not recognized as an internal or external command, operable program or batch file.
This is not surprising since the pipeline does not have access to the [REDACTED] user's AppData folder.
Attempt 2:
Changing the installation directory of pip packages using either --target=c:/PythonScripts, or with the C:\Users\[REDACTED]\AppData\Roaming\pip\pip.ini file:
[global]
target = c:\PythonScripts
When running pip install mkdocs, it successfully installs it into c:/PythonScripts. We add the folder c:/PythonScripts/bin to the PATH. Now when we try to run mkdocs from the command line, we get the following error:
C:\Users\[REDACTED]>mkdocs
Traceback (most recent call last):
File "C:\Program Files\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Program Files\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\PythonScripts\bin\mkdocs.exe\__main__.py", line 4, in <module>
ModuleNotFoundError: No module named 'mkdocs'
Somehow it does find the mkdocs.exe because it is on the path, but it cannot load the module mkdocs because that's installed in another directory?
Attempt 3:
Simply installing in the AppData folder and copying the contents. This also didn't work since the module was again not found.
Attempt 4:
Running the service as the user. This is really really undesirable, but it works. I would like to know a proper solution to this problem.
Related
I have my project installed as a windows service and everything worked smoothly. I had some errors with using venv and pywin32, but eventually figuret that out. Stackoverflow link
However when I use pyinstaller to make onefile executable from it problems start occuring.
I use following command to create executable:
[Path_to_my_venv_pyinstaller] --onefile --hidden-import win32timezone myservice.py
Then I am executing it from cmd with install arg, service is installed.
However next step, executing it with start arg fails. Error message:
Starting service [Service_Name]
Error starting service: The service did not respond to the start or control request in a timely fashion.
So I tried debugging it with ``debug``` argument. It works normally, when I install service without executable, from source files. When service is installed from executable I am getting following error.
Debugging service [Service_Name] - press Ctrl+C to stop.
Traceback (most recent call last):
File "[Service]\[Service_Folder]\service.py", line 48, in <module>
File "lib\site-packages\win32\lib\win32serviceutil.py", line 640, in HandleCommandLine
File "lib\site-packages\win32\lib\win32serviceutil.py", line 461, in DebugService
AttributeError: module 'servicemanager' has no attribute 'Debugging'
[14252] Failed to execute script service
I found similar topic on stack, unanswered, link here. So the problem indeed lays in installing service from pyinstaller executable.
I have no idea, how to track error on this, Event Viewer does not display anything more than i could see from console.
Is there other way to debug a service?
I am new to Python. I am new to Vagrant. However, my team runs their project using a Vagrant VM and IDEs of their own choosing. I chose PyCharm, because I've used some JetBrains products in the past.
I'd really like to be able to visually debug through it as it is running. Set a break point, view the values of variables, etc.
PyCharm has a help section (and the related articles above it):
https://www.jetbrains.com/help/pycharm/configuring-product-to-work-on-the-vm.html
I've done all of them, but under Project->Project Interpreter the Path Mappings seem to list all the shared folders on the between the host machine and the Vagrant VM. Those shared folders are
one directory up from the actual project
my vagrant directory
My home directory
I don't think it is pointing to the project or its dependency libraries correctly.
I also get a yellow message at the bottom that says Python packaging tools not found.
If I hit debug, I get the following output in a terminal:
bash: line 0: cd: /vagrant/app: No such file or directory
pydev debugger: process 2032 is connecting
Connected to pydev debugger (build 191.6183.50)
Traceback (most recent call last):
File "/home/vagrant/.pycharm_helpers/pydev/pydevd.py", line 1741, in
<module>
main()
File "/home/vagrant/.pycharm_helpers/pydev/pydevd.py", line 1735, in main
globals = debugger.run(setup['file'], None, None, is_module)
File "/home/vagrant/.pycharm_helpers/pydev/pydevd.py", line 1135, in run
pydev_imports.execfile(file, globals, locals) # execute the script
IOError: [Errno 2] No such file or directory: '/vagrant/app/__main__.py'
It also has opened itself a tab for 'pydev.py' that has in it:
Remote file /home/vagrant/.pycharm_helpers/pydev/pydevd.py is mapped to the
local path C:\Users\<my username>\vagrant\.pycharm_helpers\pydev\pydevd.py
and can't be found. You can continue debugging, but without the source. To
fix that you can do one of the following:
How can setup to debug from PyCharm on my host machine through code running on the Vagrant VM?
I'm trying to set up a Windows slave for our Jenkins server, but some of the unit tests are failing due to a sqlite error.
We're using conda and have built a custom version of sqlite that includes rtree. Then our application is built using this custom sqlite. When we install our app on the windows server, the correct version of sqlite is installed:
$ conda list
sqlite 3.16.2 vc9_0 [vc9] file://[package_server]
Running the app from the command line it works as expected. When we run tests though,
$ nosetests test
...
======================================================================
ERROR: test_vectorise_1 (test.system_test_wallet.test_rasters.test_vectorise.Test)
----------------------------------------------------------------------
Traceback (most recent call last):
File "farm\rasters\vectorise.py", line 189, in run_vectorise
proc(progressfn)
File "farm\process.py", line 1278, in __call__
for feature in lyr:
File "C:\bin\Miniconda2\envs\farm_system_test_windows\lib\site-packages\osgeo\
ogr.py", line 2876, in next
feature = self.GetNextFeature()
File "C:\bin\Miniconda2\envs\farm_system_test_windows\lib\site-packages\osgeo\
ogr.py", line 1460, in GetNextFeature
return _ogr.Layer_GetNextFeature(self, *args)
RuntimeError: sqlite3_exec(CREATE VIRTUAL TABLE "rtree_GB_TEST_RP100_VE_5m_27700
_geom" USING rtree(id, minx, maxx, miny, maxy)) failed: no such module: rtree
I added some logging to show python executable path and sqlite3 path and it is the same every time, so it's not a virtualenv issue:
C:\bin\Miniconda2\envs\farm_system_test_windows\python.exe
C:\bin\Miniconda2\envs\farm_system_test_windows\lib\sqlite3\__init__.pyc
Printing the sqlite version though, shows some differences:
print('sql ver', sqlite3.sqlite_version_info)
Running the app from the command line, you get (3, 16, 2), which is the correct version. Running nosetests, you get (3, 8, 11) - an older version and presumably the reason for rtree not being found.
Weirdly, if you call a test file with
python path/to/test.py
and let unittest do its stuff, version (3, 16, 2) is initially imported, but then when the test runs, you get (3, 8, 11) and the test errors.
What is going on here? Why do I get a different version of sqlite depending on how I call the application?
I'm using python 2.7.14, conda 4.4.11 on Windows Server 2012 R2.
UPDATE:
I'm still getting this error on Jenkins and now on my dev machine too. It's not just a nose issue, as when I run the test using unittest, I get the error. The PYTHONPATH and sys.path are the same in both cases, and the call stack is identical, except with unittest, there are an extra two frames at the top.
File "C:\ProgramData\Anaconda2\envs\farm\lib\runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "C:\ProgramData\Anaconda2\envs\farm\lib\runpy.py", line 72, in _run_code
exec code in run_globals
Interestingly, the rest of the call stack shows an absolute path for main.py, the entry point, and relative paths for everything else when using unittest. When calling main.py directly however, main.py has a relative path and all the other files have absolute paths.
Does anyone have any idea why we're seeing this difference? It's really frustrating having to skip a load of tests when we're running on windows.
By looking for a solution in sys.path you are looking at the wrong place. You need to see what os.environ['PATH'] is at the critical points here since it's the Windows Loader that has picked the 'wrong' DLL, where by wrong I mean the first one it found according to its well documented DLL lookup procedure.
I'm compiling a python application using pyinstaller.
The structure is like so -
d:\app\myprog.exe
d:\app\config\settings.conf
If I run myprog.exe --switch value from d:\app it runs fine, if I try to run from anywhere else like c:\windows it's not finding my settings.conf file complaining with the message:
Traceback (most recent call last):
File "<string>", line 284, in <module>
File "<string>", line 218, in main
File "ConfigParser.py", line 330, in get
ConfigParser.NoSectionError: No section: 'database'
myApp returned -1
database being the first line in the config file I'm trying to reference.
I'm referencing the BASE_DIR from the app here -
# Global Path and Config Info
try:
BASE_DIR = os.path.dirname(os.path.abspath(__file__))
except NameError: # We are the main py2exe script, not a module
BASE_DIR = os.path.dirname(os.path.abspath(sys.argv[0]))
So that I can get the config file like this -
config = RawConfigParser()
config.read(os.path.join(BASE_DIR, 'config/settings.conf'))
But I guess BASE_DIR is whatever folder I'm running the EXE from (like c:\windows, not the location of the EXE (which is d:\app)?
As mentioned in PyInstaller documentation (in How the One-File Program Works section) when a single executable file (created by PyInstaller) is executed, what happens is that a directory structure of the required library modules (Python VM, libraries and packages, etc.) are extracted from that single executable to a temporary directory, and then the application is started from there.
This means that __file__ in your Python code is not going to be the path to the single executable file, and that's the reason for this issue.
I'd write my app to accept the path to the configuration file from command line arguments, fall back to using an environment variable if available, and then a hard coded default value based on the platform (as most multi platform applications do to reach to their configurations).
Another approach is to use the one-directory output from PyInstaller and include the configuration file in the same directory.
Then to help make it easier to distribute the software to end users, use another tool to create a single Windows installer file from that directory structure. This way you'll be distributing the Windows installer, so users can run to install the application. But when the app runs, it runs from the directory where the configuration file resides.
I'm new to Google App Engine and I followed multiple video tutorials (One from Udacity) and for some reason I am unable to get it to run on the local host with the simple "Hello World" program.
I do have the Python SDK installed as well as the Google App Engine program installed. I did modify the YAML file so that it matches with my application (it did by default). When I click "run" in the Google App Engine launcher, it shows a yellow triangle caution sign next to the program I'm attempting to run.
When I type the localhost:8080 in the search bar it says:
This webpage is not available
I've also tried reinstalling both Python 2.7.9 and the Google App Engine and to no avail. In short I would like to understand why the program shows no content when I attempt to run it. Here are the log files if it's any help:
2015-03-14 18:36:21 Running command: "['E:\\Python\\pythonw.exe', 'C:\\Program Files (x86)\\Google\\google_appengine\\dev_appserver.py', '--skip_sdk_update_check=yes', '--port=8080', '--admin_port=8000', 'C:\\Program Files (x86)\\Google\\google_appengine\\new_project_template']"
Traceback (most recent call last):
File "C:\Program Files (x86)\Google\google_appengine\dev_appserver.py", line 83, in <module>
_run_file(__file__, globals())
File "C:\Program Files (x86)\Google\google_appengine\dev_appserver.py", line 79, in _run_file
execfile(_PATHS.script_file(script_name), globals_)
NameError: name 'execfile' is not defined
2015-03-14 18:36:21 (Process exited with code 1)
It looks like the default pythonw on your system (E:\Python\pythonw.exe) is some version of Python 3. That's where execfile is indeed not defined (and the GAE launcher's incompatible with Py3 in other ways, anyway, at this time).
To verify, run E:\Python\python.exe at a cmd prompt -- it should greet you with a version banner which I bet will mention Python 3.something.
Where did you (re-)install 2.7.9? How's your PATH environment variable? Likely with E:\Python before wherever 2.7.9 is installed.
Simplest might be to change your PATH so that wherever 2.7.9 is installed comes before E:\Python...!