We Keep Coding

How to access python scripts from Windows service

The problem: we wish to run an Azure Devops pipeline which contains a step that runs a python command "mkdocs" to generate documentation. However, the pipeline does not have access to this script.
How are you supposed to access python scripts inside a Windows Service?
Environment
The pipeline runs on a self hosted Windows agent (more info). This pipeline runs as a Windows Service, with a "Network Service" account. Important to note is that this account does not have access to the AppData folder, because it is not a user.
mkdocs is a python package installed using pip, which comes with a mkdocs.exe which we attempt to run in the pipeline. This package is normally installed under the user's AppData folder.
Attempt 1:
Simply adding C:\Users\[REDACTED]\AppData\Roaming\Python\Python310\Scripts to PATH. This gives the following expected output in the logs of the pipeline:
##[error]'mkdocs' is not recognized as an internal or external command, operable program or batch file.
This is not surprising since the pipeline does not have access to the [REDACTED] user's AppData folder.
Attempt 2:
Changing the installation directory of pip packages using either --target=c:/PythonScripts, or with the C:\Users\[REDACTED]\AppData\Roaming\pip\pip.ini file:
[global]
target = c:\PythonScripts
When running pip install mkdocs, it successfully installs it into c:/PythonScripts. We add the folder c:/PythonScripts/bin to the PATH. Now when we try to run mkdocs from the command line, we get the following error:
C:\Users\[REDACTED]>mkdocs
Traceback (most recent call last):
File "C:\Program Files\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Program Files\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\PythonScripts\bin\mkdocs.exe\__main__.py", line 4, in <module>
ModuleNotFoundError: No module named 'mkdocs'
Somehow it does find the mkdocs.exe because it is on the path, but it cannot load the module mkdocs because that's installed in another directory?
Attempt 3:
Simply installing in the AppData folder and copying the contents. This also didn't work since the module was again not found.
Attempt 4:
Running the service as the user. This is really really undesirable, but it works. I would like to know a proper solution to this problem.

Virtual Python enviroment with WINSW

I am trying to create a Windows Service which uses a Virtual Python Environment. I create the environment as described here with venv. To create the Windows Service I use winsw.
While I can both run a program in the virtual environment and setup a windows service I cannot figure out how to do both at once.
My first attempt was to call "call win_env/Scripts/activate.bat" in the .bat file which is used by the windows service. This did not work, as he still used the main enviroment and therefore ignored this line of code. Also the second answer in this issue suggests an solution. This also did not work for me as the program still used the main enviroment.
Possible this is because I do not use the env variables in the code but I cannot figure out how I would start the virtual env from there.
When I write that it uses the "main environment" I track this because the program fails because of some missing dependencies (wrong versions). These dependencies are installed in the virtual environment but not in the main environment. If I call the bat files without going into the virtual environment the program fails with the same message.
Error Message:
Exception in thread {X}:
Traceback (most recent call last):
File "{Path}\scoop\apps\anaconda3\current\lib\threading.py", line 973, in _bootstrap_inner
self.run()
File "{Path}\scoop\apps\anaconda3\current\lib\threading.py", line 910, in run
self._target(*self._args, **self._kwargs)
File "{Path}\Sources\{Project}\.\populate_cache.py", line 25, in query_failures_from_db
df = pd.read_sql(statement, con, params=[date_from, date_to])
File "{Path}\Sources\{Project}\win_env\lib\site-packages\pandas\io\sql.py", line 563, in read_sql
pandas_sql = pandasSQL_builder(con)
File "{Path}\Sources\{Project}\win_env\lib\site-packages\pandas\io\sql.py", line 744, in pandasSQL_builder
import sqlite3
File "{Path}\scoop\apps\anaconda3\current\lib\sqlite3\__init__.py", line 23, in <module>
from sqlite3.dbapi2 import *
File "{Path}\scoop\apps\anaconda3\current\lib\sqlite3\dbapi2.py", line 27, in <module>
from _sqlite3 import *
ImportError: DLL load failed while importing _sqlite3: The specified module could not be found.
.bat file:
call uvicorn main:app --port 8590 --host 0.0.0.0
winsw file:
<?xml version="1.0"?>
<!--This configuration file should be placed near the WinSW executable, the name should be the same.E.g. for myapp.exe the configuration file name should be myapp.xmlYou can find more information about configuration options here: https://github.com/kohsuke/winsw/blob/master/doc/xmlConfigFile.md -->
<configuration>
<id>Test Programm</id>
<name>Test Programm</name>
<description> XXX. </description>
<executable>{PathToBatfile}</executable>
<onfailure delay="10 sec" action="restart"/>
<onfailure delay="20 sec" action="restart"/>
<onfailure action="none"/>
<resetfailure>1 hour</resetfailure>
<workingdirectory>{PathToWorkingdirectory}</workingdirectory> // .bat file is located here
<priority>Normal</priority>
<stoptimeout>15 sec</stoptimeout>
<stopparentprocessfirst>true</stopparentprocessfirst>
<startmode>Automatic</startmode>
<waithint>15 sec</waithint>
<sleeptime>1 sec</sleeptime>
<logpath>{PathToLog}</logpath>
<!--OPTION: logDefines logging mode for logs produced by the executable.Supported modes:* append - Rust update the existing log* none - Do not save executable logs to the disk* reset - Wipe the log files on startup* roll - Rotate logs based on size* roll-by-time - Rotate logs based on time* rotate - Rotate logs based on size, (8 logs, 10MB each). This mode is deprecated, use "roll"Default mode: appendEach mode has different settings.See https://github.com/kohsuke/winsw/blob/master/doc/loggingAndErrorReporting.md for more details -->
<log mode="roll"> </log>
</configuration>

Based on the input you provided, I think the following batch file should work:
call %~dp0win_env/Scripts/activate.bat
uvicorn main:app --port 8590 --host 0.0.0.0
(The %~dp0 resolves to the directory where the batch file is stored, so it works independent of your current working directory.)
Note that the activate.bat does nothing special. It just updates your PATH environment variable and changes the command prompt, which is convenient during development. However, you could also just call the Python executable from your virtual environment without "activating" it. I'm not familiar with uvicorn, but it seems like it can be called as a Python module as well. In that case you could also use the following batch file:
win_env/Scripts/python.exe -m uvicorn main:app --port 8590 --host 0.0.0.0

How to Create Python virtual environment inside an USB Flash

I am running ubuntu 18.04.1 in virtual box, on windows 10. The Ubuntu is running 2.7 version python. The USB stick is mounted to the path /opt/ext_usb/yyyy. When I tried executing the command virtualenv my-env-name --no-site-packages , I am getting the following error.
Running virtualenv with interpreter /usr/bin/python2
New python executable in /opt/ext_usb/jpr_lamp/bin/python2
Also creating executable in /opt/ext_usb/jpr_lamp/bin/python
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/virtualenv.py", line 2375, in <module>
main()
File "/usr/lib/python3/dist-packages/virtualenv.py", line 724, in main
symlink=options.symlink)
File "/usr/lib/python3/dist-packages/virtualenv.py", line 946, in create_environment
site_packages=site_packages, clear=clear, symlink=symlink))
File "/usr/lib/python3/dist-packages/virtualenv.py", line 1417, in install_python
os.symlink(py_executable_base, full_pth)
OSError: [Errno 1] Operation not permitted
I would like to know why the permission is denied when creating python virtual environment inside an USB stick. Is it because the ubuntu is running in VM and USB is shared between the windows. Please Help !
Note:
I have tried creating symlink on USB stick using ln -s /usr/bin/python /sampe/path/inUsb and failed to create the symbolic link.
Is this the actual cause of the issue? If so, how to get rid of the issue. Thanks in advance !

Thanks for your comment #wholevinski
The USB stick that I have used has Fat32 file system and it won't support symbolic links. Able to create the symlinks after changing the file system from fat32 to NTFS. Now I am able to create python virtual environment in an USB flash.

python - compiled calling config file

I'm compiling a python application using pyinstaller.
The structure is like so -
d:\app\myprog.exe
d:\app\config\settings.conf
If I run myprog.exe --switch value from d:\app it runs fine, if I try to run from anywhere else like c:\windows it's not finding my settings.conf file complaining with the message:
Traceback (most recent call last):
File "<string>", line 284, in <module>
File "<string>", line 218, in main
File "ConfigParser.py", line 330, in get
ConfigParser.NoSectionError: No section: 'database'
myApp returned -1
database being the first line in the config file I'm trying to reference.
I'm referencing the BASE_DIR from the app here -
# Global Path and Config Info
try:
BASE_DIR = os.path.dirname(os.path.abspath(__file__))
except NameError: # We are the main py2exe script, not a module
BASE_DIR = os.path.dirname(os.path.abspath(sys.argv[0]))
So that I can get the config file like this -
config = RawConfigParser()
config.read(os.path.join(BASE_DIR, 'config/settings.conf'))
But I guess BASE_DIR is whatever folder I'm running the EXE from (like c:\windows, not the location of the EXE (which is d:\app)?

As mentioned in PyInstaller documentation (in How the One-File Program Works section) when a single executable file (created by PyInstaller) is executed, what happens is that a directory structure of the required library modules (Python VM, libraries and packages, etc.) are extracted from that single executable to a temporary directory, and then the application is started from there.
This means that __file__ in your Python code is not going to be the path to the single executable file, and that's the reason for this issue.
I'd write my app to accept the path to the configuration file from command line arguments, fall back to using an environment variable if available, and then a hard coded default value based on the platform (as most multi platform applications do to reach to their configurations).
Another approach is to use the one-directory output from PyInstaller and include the configuration file in the same directory.
Then to help make it easier to distribute the software to end users, use another tool to create a single Windows installer file from that directory structure. This way you'll be distributing the Windows installer, so users can run to install the application. But when the app runs, it runs from the directory where the configuration file resides.

Google App Engine Local Host Issue

I'm new to Google App Engine and I followed multiple video tutorials (One from Udacity) and for some reason I am unable to get it to run on the local host with the simple "Hello World" program.
I do have the Python SDK installed as well as the Google App Engine program installed. I did modify the YAML file so that it matches with my application (it did by default). When I click "run" in the Google App Engine launcher, it shows a yellow triangle caution sign next to the program I'm attempting to run.
When I type the localhost:8080 in the search bar it says:
This webpage is not available
I've also tried reinstalling both Python 2.7.9 and the Google App Engine and to no avail. In short I would like to understand why the program shows no content when I attempt to run it. Here are the log files if it's any help:
2015-03-14 18:36:21 Running command: "['E:\\Python\\pythonw.exe', 'C:\\Program Files (x86)\\Google\\google_appengine\\dev_appserver.py', '--skip_sdk_update_check=yes', '--port=8080', '--admin_port=8000', 'C:\\Program Files (x86)\\Google\\google_appengine\\new_project_template']"
Traceback (most recent call last):
File "C:\Program Files (x86)\Google\google_appengine\dev_appserver.py", line 83, in <module>
_run_file(__file__, globals())
File "C:\Program Files (x86)\Google\google_appengine\dev_appserver.py", line 79, in _run_file
execfile(_PATHS.script_file(script_name), globals_)
NameError: name 'execfile' is not defined
2015-03-14 18:36:21 (Process exited with code 1)

It looks like the default pythonw on your system (E:\Python\pythonw.exe) is some version of Python 3. That's where execfile is indeed not defined (and the GAE launcher's incompatible with Py3 in other ways, anyway, at this time).
To verify, run E:\Python\python.exe at a cmd prompt -- it should greet you with a version banner which I bet will mention Python 3.something.
Where did you (re-)install 2.7.9? How's your PATH environment variable? Likely with E:\Python before wherever 2.7.9 is installed.
Simplest might be to change your PATH so that wherever 2.7.9 is installed comes before E:\Python...!