Python external project script execution - python

After many researches on the web, on many different topics, I wasn't able to find the solution answering my problem.
I am developing a tool (with graphic interface) allowing to process python scripts from different projects.
To not have any problem of dependences, I ask the user to target the project directory as well as the script to process.
Thus to develop this tool I have in my IDE a project "benchmark_tool" which is the project integrating the GUI, and the project "test_project" which groups various testing scripts. Obviously, the final goal is the tool to be compiled and thus that the project "benchmark_tool" disappears from IDE.
Here is the IDE architecture:
benchmark_tool (python project)
__init__.py
main.py
test_project (python project)
__init__.py
module1
__init__.py
test_script.py
module2
__init__.py
imports.py
How it works: The main.py shall call test_script.py.
test_script.py calls imports.py at the first beggining of the script.
UPDATE
I tried some modifications in my main code:
import sys
import subprocess
subprocess.check_call([sys.executable, '-m', 'test_project.module1.test_script'], cwd='D:/project/python')
I got this error
Traceback(most recent call last):
File "C:/Python31/lib/runpy.py", line 110, in run module as main
mod_name, loader, code, fname = _get_module_detail(mod_name)
File "C:/Python31/lib/runpy.py", line 91, in get module details
code = loader.get_code(mod_name)
File "C:/Python31/lib/pkgutil.py", line 272, in get code
self.code = compile(source, self.filename, 'exec')
File "D:/project/python/test_project/module1/test_script.py", line 474
SyntaxError: invalid syntax
Traceback (most recent call last):
File "D:/other_project/benchmark_tool/main.py", line 187, in read
subprocess.check_call([sys.executable, '-m', 'module1.test_script.py'], cwd='D:/project/python/test_project')
File "C:/Python31/lib/subprocess.py", line 446, in check call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['C:/Python31/python.exe', '-m', 'module1.test_script.py']' returned non-zero exit status 1
Note
It works with subprocess.Popen(['python','D:/project/python/test_project/module1/test_script.py])
What's the main difference between both methods ? I'll also have to pass arguments to test_scripts.py, which one is the best to use to communicate with a python script (input and output datas are exchanged) ?
Thanks by advance

There are 3 problems I see that you have to fix:
In order to import from module2, you need to turn that directory into a package by placing an empty *__init__.py file in it
When executing test_script.py, you need the full path to the file, not just file name
Fixing up sys.path only work for your script. If you want to propagate that to test_script.py, set the PYTHONPATH environment variable
Therefore, my solution is:
import os
import sys
import subprocess
# Here is one way to determine test_project
# You might want to do it differently
script_dir = os.path.abspath(os.path.dirname(__file__))
test_project = os.path.normpath(os.path.join(script_dir, '..', 'test_project'))
python_path = '{}:{}'.format(os.environ.get('PYTHONPATH', ''), test_project)
python_path = python_path.lstrip(':') # In case PYTHONPATH was not set
# Need to determine the full path to the script
script = os.path.join(test_project, 'module1', 'test_script.py')
proc = subprocess.Popen(['python', script], env=dict(PYTHONPATH=python_path))

To run an "external project script", install it e.g.: pip install project (run it in a virtualenv if you'd like) and then run the script as you would with any other executable:
#!/usr/bin/env python
import subprocess
subprocess.check_call(['executable', 'arg 1', '2'])
If you don't want to install the project and it has no dependencies (or you assume they are already installed) then to run a script from a given path as a subprocess (assuming there is module1/__init__.py file):
#!/usr/bin/env python
import subprocess
import sys
subprocess.check_call([sys.executable, '-m', 'module1.test_script'],
cwd='/abs/path/to/test_project')
Using -m to run the script, avoids many import issues.
The above assumes that test_project is not a Python package (no test_project/__init__.py). If test_project is itself a Python package then include it in the module name and start from the parent directory instead:
subprocess.check_call([sys.executable, '-m', 'test_project.module1.test_script'],
cwd='/abs/path/to')
You could pass the path to test_project directory as a command-line parameter (sys.argv) or read it from a config file (configparser,json). Avoid calculating it relative to the installation path of your parent script (if you have to then use pkgutil, setuptools' pkg_resources, to get the data). To find a place to store user data, you could use appdirs Python package.

Related

How can I access other files in the same Docker container?

I'm new to Docker. When I am running my docker container, the last line in the Dockerfile is the following:
CMD ["python3", "./poses/server/server_multithreaded.py"]
In the server_multithreaded.py file described above, I am importing another file, as seen below:
from poses.poseapp.poseapp_sockets import PoseAppWSockets
When I run container using the command docker run -p 8089:8089 zoheezus/capstone I get the following error:
Traceback (most recent call last):
File "./poses/server/server_multithreaded.py", line 18, in <module>
from poses.poseapp.poseapp_sockets import PoseAppWSockets
ImportError: No module named 'poses'
From what I understand, the 'poses' directory is not accessible or I am not accessing it the right way. What do I need to do for server_multithreaded.py to be able to access the other files when I run it?
The file structure of the project is the following:
Python won't add current working path into module search path, it will just add top level script's path into search path, of course PYTHONPATH, sys path etc. also be searched.
For your case, the current working path /usr/pose_recognizer won't be searched, only /usr/pose_recognizer/poses/server will be searched. So, definitely, it won't find module named 'poses'.
To make it work for you, give you some options:
Option 1: Execute the server_multithreaded.py as module:
python -m poses.server.server_multithreaded
Option 2: Change sys.path in server_multithreaded.py as next before from poses.poseapp.poseapp_sockets import PoseAppWSockets:
import sys
import os
file_path = os.path.abspath(os.path.dirname(__file__)).replace('\\', '/')
lib_path = os.path.abspath(os.path.join(file_path, '../..')).replace('\\', '/')
sys.path.append(lib_path)
Option 3: Change PYTHONPATH in dockerfile:
WORKDIR /usr/pose_recognizer
ENV PYTHONPATH=.
CMD ["python3", "./poses/server/server_multithreaded.py"]
this problem is most likely not related to docker but to your PYTHONPATH environment variable (inside the docker container). You must make sure that capstone-pose-estimation is in it. Furthermore, you should make poses and poseapp a package (containing __init___.py) in order to import from it

Open file in IDLE through python script

How would I open a specific file in IDLE through a python script?
I understand that an app could be opened through subprocess:
import subprocess
subprocess.call('C:\\program.exe')
But I can't figure out how to make it open a file.
If it helps, this:
import os.path
import sys
# Enable running IDLE with idlelib in a non-standard location.
# This was once used to run development versions of IDLE.
# Because PEP 434 declared idle.py a public interface,
# removal should require deprecation.
idlelib_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
if idlelib_dir not in sys.path:
sys.path.insert(0, idlelib_dir)
from idlelib.pyshell import main
main()
also opens IDLE. I checked, and main() does not take any parameters such as files to open.
I am using Windows 10 with Python 3.6.4.
Any help is greatly appreciated.
Here are 2 ways to open any python file through IDLE
import subprocess
p = subprocess.Popen(["idle.exe", path_to_file])
# ... do other things while idle is running
returncode = p.wait() # wait for notepad to exit
OR:
import subprocess
import os
subprocess.call([path_to_idle, path_to_file])
You can also use these methods to open any file with any installed app (How can I open files in external programs in Python?)
One can run IDLE from a command line on any platform with <python> -m idlelib <IDLE args>, where <python> could be 'python', 'python3', or something line 'py -3.8', depending on the platform. <IDLE args> are defined in the "Command line usage" subsection of the IDLE doc, also available within IDLE as Help => IDLE Help.
A possible 'IDLE arg' is the path of a file to be opened in an editor window. Relative paths are relative to the current working directory, which can be changed with the 'cd' command. A working command line can used quoted or turned into a list for a subprocess.run or subprocess.Popen call. In Python code, the working directory is changed with os.chdir('newdir').

Import from my package recognized by PyCharm but not by command line

I've created the following package tree:
/main_package
/child_package
version.py
where version.py contains a single string variable (VERSION)
Inside my script in child package I'm importing version.py by the following line:
from main_package.version import VERSION
While I'm running the code from PyCharm everything works great, however when I'm running the code via the command line I'm getting the following error message:
C:\Users\usr\PycharmProjects\project\main_package\child_package>python script.py
Traceback (most recent call last):
File "script.py", line 2, in <module>
from main_package.version import VERSION
ModuleNotFoundError: No module named 'main_package'
I've found in the internet that I might need to add my package to the python path, however it doesn't seems to work for me
PyCharm sets the Python Path at the root of the project (by default). To mimic this in a quick'n'dirty fashion, you just need to do this once in your shell session before invoking python whatever:
set PYTHONPATH=C:\Users\usr\PycharmProjects\project
The pythonic way is to have a setup.py file to install your project in the system (check python Minimal Structure):
from setuptools import setup
setup(name='main_package',
version='0.1',
description='main package',
license='MIT',
packages=['main_package'],
zip_safe=False)
Then you install it as follow:
python setup.py install for global installation
OR
python setup.py develop for local installation in editable mode

How do I replicate the way PyCharm is running my Python 3.4 project at the command line?

My project looks like this:
running-pycharm-project-at-cmd
- main.py
- c
- run_project.py
- z
- __init__.py
- the_module.py
- y
- __init__.py
- template.md
- the_module_module.py
- the_support_function.py
The contents of the .py files are shown below:
main.py
from c.run_project import run
print('running main.py...')
run()
c/run_project.py
from c.z.the_module import the_module_function, the_module_write_function
def run():
print('Running run_project.py!')
the_module_function()
# write a file:
the_module_write_function(read_file='./z/y/template.md', write_file='../README.md')
if __name__ == '__main__':
run()
c/z/the_module.py
from c.z.y.the_module_module import the_module_module_function
def the_module_function():
print('the_module_function is running!')
the_module_module_function()
pass
def the_module_write_function(read_file, write_file):
with open(read_file, 'r') as fid:
with open(write_file, 'w') as fid_out:
contents = fid.read()
contents.replace('{}', 'THE-FINAL-PRODUCT!')
fid_out.write(contents)
c/z/y/the_module_module.py
from .the_support_function import this_support_data
def the_module_module_function():
print('The module module function is running!')
print("Here is support data: {}".format(this_support_data))
pass
c/z/y/the_support_function.py
this_support_data = "I am the support data!"
GitHub repo to make replication easier: running-pycharm-project-at-cmd
The problem: in Pycharm load up the project with running-pycharm-project-at-cmd as the root. Then I right click and run the run_project.py file. Everything works fine. I cannot figure out how to run run_project.py from the command line in the same way. Creating main.py was a workaround suggested elsewhere, but it fails due to the relative references to the template file (in actual project, the y folder is a git submodule and therefore cannot have absolute references in it).
If you copy your main.py file into the c folder and rename it __init__.py, then you could run:
$ python -m c
The -m argument tells python to run a module or package (in this case c). Python will look in the c's folder for an __init__.py file and run that. You just need to ensure that the folder c is either in your PYTHONPATH or is a subfolder of the current working directory.
I circled back to this and was able to get it working. Note that I am on Windows 7, and some of this might be platform dependent. In order to get this working without changing any code, the key is setting PYTHONPATH environment variable before running. PyCharm does this automatically (by default, but it is configurable in the Run Configuration options).
Steps:
Recreating the issue:
Clone the github repo to G:\TEST.
Navigate to G:\TEST\running-pycharm-project-at-cmd-master\c
python run_project.py
Traceback (most recent call last):
File "run_project.py", line 1, in <module>
from c.z.the_module import the_module_function, the_module_write_function
ImportError: No module named 'c'
Simplest Solution (if able to run Python from the script location):
set the PYTHONPATH environment variable before running. e.g.
SET PYTHONPATH=G:\TEST\running-pycharm-project-at-cmd-master
Now python run_project.py
Running run_project.py!
the_module_function is running!
The module module function is running!
Here is support data: I am the support data!
To run from a remote location (this is definitely Windows specific):
Create a .bat file that navigates to correct working directory before running Python. i.e.:
START cmd /K "cd G:\TEST\running-pycharm-project-at-cmd-master\c & python run_project.py"
G:\TEST\myotherlocation
run_it.bat
Running run_project.py!
the_module_function is running!
The module module function is running!
Here is support data: I am the support data!

No such file or directory with Flask

flask question: I have my run.py file in the following dir
/Users/`<username>`/Python_stuff/flask
but when I run it it says
(api_automation)MacBook-Pro:flask `<username>`$ ./run.py
-bash: ./run.py: flask/bin/python: bad interpreter: No such file or directory
I'm confused as to why this is happening when its worked in the past on other virtualenv's
here is what the run.py looks like:
#!flask/bin/python
from app import app
app.run(debug = True)
Your file starts with a shebang telling the shell what program to load to run the script. The shebang line is the first line starting with #!.
In this case, the shebang tells the shell to run flask/bin/python, and that file does not exist in your current location.
The tutorial you got this from expects you to create a virtualenv directory called flask, and the script is set up to run the Python binary installed in that directory.
If you are using a different Python location, edit the shebang line to point to the correct location, or use python run.py to explicitly name the executable on the command line. In that case the shebang line is ignored.

Categories