I have been searching and every solution I find is for Windows specific. I am looking for a way to run multiple python programs at one time. I have tried
import test1
import test2
and i only get one of them at a time. I am needing both to be ran at the same time.
Any help is appreciated!
To run Python code in parallel, please refer to the module multiprocessing.
from multiprocessing import Process
def first_import():
import test1
def second_import():
import test2
if __name__ == '__main__':
first_process = Process(target=first_import)
second_process = Process(target=second_import)
first_process.start()
second_process.start()
first_process.join()
second_process.join()
I am currently doing a little python project, and I am wondering if there is any way that I can import a script which is hosted on my web page. The script includes some vars like
CurrentV = 'The current version is 1.2'
...so I don't need to text everyone that I made a new version. They can see it then in their scripts, because it's way faster to just update the version on my web host.
I tried it that way:
import requests
oimport = requests.get('https://website.com/version.py')
from oimport.content import CurrentV
print(CurrentV)
print('your current version is 1.1')
But it's not working that way.
One way you can make this work is to write the "module" to disk and then import it. For example:
from pathlib import Path
import sys
oimport = requests.get('https://website.com/version.py')
module_path = "./modules"
# Write the code to modules/version.py
with open(f"{module_path}/version.py", "w") as module_file:
module_file.write(module_path)
# Make sure our path in sys.path
sys.path.append(module_path)
from version import CurrentV
THIS OPENS UP A CAN OF WORMS IN THE FORM OF CODE INJECTION! but it does/should do what you need it to do.
As we all know we need to protect the main() when running code with multiprocessing in Python using if __name__ == '__main__'.
I understand that this is necessary in some cases to give access to functions defined in the main but I do not understand why this is necessary in this case:
file2.py
import numpy as np
from multiprocessing import Pool
class Something(object):
def get_image(self):
return np.random.rand(64,64)
def mp(self):
image = self.get_image()
p = Pool(2)
res1 = p.apply_async(np.sum, (image,))
res2 = p.apply_async(np.mean, (image,))
print(res1.get())
print(res2.get())
p.close()
p.join()
main.py
from file2 import Something
s = Something()
s.mp()
All of the functions or imports necessary for Something to work are part of file2.py. Why does the subprocess need to re-run the main.py?
I think the __name__ solution is not very nice as this prevents me from distribution the code of file2.py as I can't make sure they are protecting their main.
Isn't there a workaround for Windows?
How are packages solving that (as I never encountered any problem not protecting my main with any package - are they just not using multiprocessing?)
edit:
I know that this is because of the fork() not implemented in Windows. I was just asking if there is a hack to let the interpreter start at file2.py instead of main.py as I can be sure that file2.py is self-sufficient
When using the "spawn" start method, new processes are Python interpreters that are started from scratch. It's not possible for the new Python interpreters in the subprocesses to figure out what modules need to be imported, so they import the main module again, which in turn will import everything else. This means it must be possible to import the main module without any side effects.
If you are on a different platform than Windows, you can use the "fork" start method instead, and you won't have this problem.
That said, what's wrong with using if __name__ == "__main__":? It has a lot of additional benefits, e.g. documentation tools will be able to process your main module, and unit testing is easier etc, so you should use it in any case.
As others have mentioned the spawn() method on Windows will re-import the code for each instance of the interpreter. This import will execute your code again in the child process (and this will make it create it own child, and so on).
A workaround is to pull the multiprocessing script into a separate file and then use subprocess to launch it from the main script.
I pass variables into the script by pickling them in a temporary directory, and I pass the temporary directory into the subprocess with argparse.
I then pickle the results into the temporary directory, where the main script retrieves them.
Here is an example file_hasher() function that I wrote:
main_program.py
import os, pickle, shutil, subprocess, sys, tempfile
def file_hasher(filenames):
try:
subprocess_directory = tempfile.mkdtemp()
input_arguments_file = os.path.join(subprocess_directory, 'input_arguments.dat')
with open(input_arguments_file, 'wb') as func_inputs:
pickle.dump(filenames, func_inputs)
current_path = os.path.dirname(os.path.realpath(__file__))
file_hasher = os.path.join(current_path, 'file_hasher.py')
python_interpreter = sys.executable
proc = subprocess.call([python_interpreter, file_hasher, subprocess_directory],
timeout=60,
)
output_file = os.path.join(subprocess_directory, 'function_outputs.dat')
with open(output_file, 'rb') as func_outputs:
hashlist = pickle.load(func_outputs)
finally:
shutil.rmtree(subprocess_directory)
return hashlist
file_hasher.py
#! /usr/bin/env python
import argparse, hashlib, os, pickle
from multiprocessing import Pool
def file_hasher(input_file):
with open(input_file, 'rb') as f:
data = f.read()
md5_hash = hashlib.md5(data)
hashval = md5_hash.hexdigest()
return hashval
if __name__=='__main__':
argument_parser = argparse.ArgumentParser()
argument_parser.add_argument('subprocess_directory', type=str)
subprocess_directory = argument_parser.parse_args().subprocess_directory
arguments_file = os.path.join(subprocess_directory, 'input_arguments.dat')
with open(arguments_file, 'rb') as func_inputs:
filenames = pickle.load(func_inputs)
hashlist = []
p = Pool()
for r in p.imap(file_hasher, filenames):
hashlist.append(r)
output_file = os.path.join(subprocess_directory, 'function_outputs.dat')
with open(output_file, 'wb') as func_outputs:
pickle.dump(hashlist, func_outputs)
There must be a better way...
The main module is imported (but with __name__ != '__main__' because Windows is trying to simulate a forking-like behavior on a system that doesn't have forking). multiprocessing has no way to know that you didn't do anything important in you main module, so the import is done "just in case" to create an environment similar to the one in your main process. If it didn't do this, all sorts of stuff that happens by side-effect in main (e.g. imports, configuration calls with persistent side-effects, etc.) might not be properly performed in the child processes.
As such, if they're not protecting their __main__, the code is not multiprocessing safe (nor is it unittest safe, import safe, etc.). The if __name__ == '__main__': protective wrapper should be part of all correct main modules. Go ahead and distribute it, with a note about requiring multiprocessing-safe main module protection.
the if __name__ == '__main__' is needed on windows since windows doesnt have a "fork" option for processes.
In linux, for example, you can fork the process, so the parent process will be copied and the copy will become the child process (and it will have access to the already imported code you had loaded in the parent process)
Since you cant fork in windows, python simply imports all the code that was imported by the parent process, in the child process. This creates a similar effect, but if you dont do the __name__ trick, this import will execute your code again in the child process (and this will make it create it own child, and so on).
so even in your example main.py will be imported again (since all the files are imported again). python cant guess what specific python script the child process should import.
FYI there are other limitations you should be aware of like using globals, you can read about it here https://docs.python.org/2/library/multiprocessing.html#windows
I am trying to connect Modelica and Python using the Python27 block, provided by the Berkeley Simulations Lab:
http://simulationresearch.lbl.gov/modelica
I use this block to call a Python function:
def Test2(WriteValues):
''' Connection Test - Works if started from Dymola
'''
#Doing nothing and returning the input
ReturnList=WriteValues
return (ReturnList)
works perfectly.
Now I need to import some modules
#Importing Python modules works in general
import sys
import thread
import time
works aswell
Only now I want to import a module that is not part of Python but a site-package:
def Test1(WriteValues):
'''Connection Test - Doesnt work if started from Dymola
'''
#Importing some Bacpypes Module
#Path is by default C:\Python27\Lib\site-packages\BACpypes-0.7-py2.7.egg\bacpypes
#My os is win7
from bacpypes.core import run
#Doing nothing and returning the input
ReturnList=WriteValues
return (ReturnList)
This does not work. It does not matter if I import the BACpypes module inside a function or globally - the error is always
'module' object has no attribute 'argv'
Colleagues pointed me to the idea that it might be related to a multiple import problem. The function is being called by Modelica every 10 seconds (real-time-simualtion).
If I call the function Test1 outside of Modelica, there is no problem. It only fails using the Python27 block!
Does anyone have an idea about how to make the BACpypes import work?
UPDATE 2013-10-16:
I printed out the value of sys.argv for the script excecution in the Python directory and an excecution from Modelica.
sys.argv from Python directory:
['C:\\Python27\\Testcon.py']
sys.argv if function is called from inside Modelica:
['modpython']
Might this in any way be related to the error message I get?
The bug is caused because bacpypes uses sys.argv but the Python interpreter did not call PySys_SetArgv.
This will be fixed in the next version of the Modelica Buildings library, see https://github.com/lbl-srg/modelica-buildings/issues/191
I have successfully created a Python Windows service using pywin32. In testing my application I attempted to have it print (which didn't work as I expected) and I also had it write to a file. It was able to write to a file, but the file ended up in the python library site-packages folder. This appears to be where the working directory is, though I'm not sure why? I would like to know the best way to specify what the working directory should be.
I could open files with full path names, or I could maybe use os.cwd? What is the best practice?
Here are the two files which compose my Windows service.
import os
import sys
import win32service
import win32serviceutil
from twisted.internet import reactor
import xpress
class XPressService(win32serviceutil.ServiceFramework):
_svc_name_ = 'XPress'
_svc_display_name_ = 'XPress Longer Name'
def SvcStop(self):
self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
reactor.callFromThread(reactor.stop)
def SvcDoRun(self):
xpress.main()
reactor.run(installSignalHandlers=False)
if __name__ == "__main__":
win32serviceutil.HandleCommandLine(XPressService)
Below is "xpress.py" which is imported by the above script.
import datetime
def main():
with open('times', 'a') as f:
print str(datetime.datetime.now())
f.write(str(datetime.datetime.now()))
if __name__ == '__main__':
main()
They both work, it's what your needs are. For various reasons, it's probably best to use absolute paths to the file names, this way you don't have to worry about 'where' your app is working, you just know where the output will be (which is most important). In *nix, apps generally work in '/' when they don't have a specified working directory. If you do choose to work in another directory it's os.chdir(newDir), do this before you call win32serviceutil.HandleCommandLine
I don't know the windows default, but you probably nailed it with the library's directory in site-packages.