I downloaded offline zim archives of websites such as wikipedia.com, stackoverflow.com, unix.stackexchange.com etc. and serve them via below script.
#!/usr/bin/env python3
import os
import subprocess
import sys
import threading
from zimply import ZIMServer
ZIMServer("/home/user/zim/wikipedia_en_all_novid_2018-05.zim", index_file="/home/user/zim/wikipedia_en_all_novid_2018-05.idx", port=8790, encoding="utf-8")
it works well but when I add another line
ZIMServer("/home/user/zim/math.stackexchange.com_eng_all_2018-08.zim", index_file="/home/user/zim/math.stackexchange.com_eng_all_2018-08.idx", port=7890, encoding="utf-8")
it doesn't load the second ZIMServer. I think it's a python multithreading issue. How can I run both of them in the single script?
By using multiprocessing — Process-based parallelism, plus port incrementation.
from multiprocessing import Process
import os
import subprocess
import sys
import threading
from zimply import ZIMServer
def f(port, file, idx):
ZIMServer(file, index_file=idx, port=port, encoding="utf-8")
if __name__ == '__main__':
p1 = Process(target=f, args=(8790,"/home/user/zim/wikipedia_en_all_novid_2018-05.zim",""))
p2 = Process(target=f, args=(8791,"/home/user/zim/whatever.zim",""))
p1.start()
p2.start()
Related
I make a python script running in the console, and I want to create another console for printing important messages without running another python script to do that.
I first tried to use win32console.AllocConsole() directly, but it got Access is denied
(Seemingly because one process can attach to at most one console according to the docs).
So I tried creating a new process by using multiprocessing :
import sys, os
import win32api, win32con, win32console
import multiprocessing
def ShowConsole():
win32console.FreeConsole()
win32console.AllocConsole()
sys.stdout = open("CONOUT$", "w")
sys.stderr = open("CONOUT$", "w")
print("Test")
os.system("pause")
if __name__ == '__main__':
p = multiprocessing.Process(target=ShowConsole)
p.start()
But when I ran the code in Powershell, it exited directly with no message while no new console is created.
None of the possible solutions I found in stackoverflow works for me. What should I do?
Update: It turns out that it is because multiprocessing.Process fails to call ShowConsole function. I use multiprocessing.dummy.Process as the alternative and it works as expected.
The reason why multiprocessing.Process fails to call target is still unclear.
There's nothing wrong with your example above, it pops the console as shown below. I added a "hello" in the main section to differentiate.
But since you want to values from the first console to the second,
here's a better example. Utilize put/get to pass the information from the first console to the second console.
import win32console
import multiprocessing
import time
def secondconsole(output):
win32console.FreeConsole()
win32console.AllocConsole()
while True:
print(output.get())
if __name__ == "__main__":
output = multiprocessing.Queue()
multiprocessing.Process(target=secondconsole, args=[output]).start()
while True:
print("Hello World")
output.put("Hello to second console") #here you will provide the data to the second console
time.sleep(3) #sleep for 3 seconds just for testing
It looks like the issue might be with the way you are trying to open the console using sys.stdout and sys.stderr. Try using the following code instead:
import sys, os
import win32api, win32con, win32console
import multiprocessing
def ShowConsole():
win32console.FreeConsole()
win32console.AllocConsole()
os.dup2(win32console.GetStdHandle(win32console.STD_OUTPUT_HANDLE), sys.stdout.fileno())
os.dup2(win32console.GetStdHandle(win32console.STD_ERROR_HANDLE), sys.stderr.fileno())
print("Test")
os.system("pause")
if __name__ == '__main__':
p = multiprocessing.Process(target=ShowConsole)
p.start()
Below is my code for running two python code in parallel using multiprocessing :
defs.py
import os
def pro(process):
#print(process)
os.system('python {}'.format(process))
Multiprocessing.py
import os
from multiprocessing import Pool
import multiprocessing as mp
import defs
import datetime
import pandas as pd
processes = ('python_code1.py','python_code2.py')
if __name__ == '__main__':
pool = Pool(processes=4)
start = datetime.datetime.now()
print('Start:',start)
pool.map(defs.pro, processes)
end = datetime.datetime.now()
print('End :',end)
total = end-start
print('Total :', end-start)
This code is running perfectly fine. But my requirement is I need to run the python code 'python_code1.py' and 'python_code2.py' from two different directory.
so I made the below changes in Multiprocessing.py:
path1 = r'C:\Users\code1\python_code1.py'
path2 = r'C:\Users\code2\python_code2.py'
processes = (path1,path2)
but this is not working for me.
My Multiprocessing.py and defs.py are kept on path `C:\Users\Multiprocessing\'
Well an elegant solution using asyncio. It is used as a foundation for multiple Python asynchronous frameworks that provide high-performance network and web-servers, database connection libraries, distributed task queues, etc. Plus it has both high-level and low-level APIs to accomodate any kind of problem. And you might find syntax easier as I do:
import os
import asyncio
def background(f):
def wrapped(*args, **kwargs):
return asyncio.get_event_loop().run_in_executor(None, f, *args, **kwargs)
return wrapped
#background
def pro(process):
#print(process)
os.system('python {}'.format(process))
processes = (r'C:\Users\code1\python_code1.py',r'C:\Users\code2\python_code2.py')
for process in processes:
pro(process)
Detailed answer on parallelizing for loop. You might find useful.
How to End task this exe file in background processes:
my code:
import os
import subprocess
import time
import win32com.client
wmi=win32com.client.GetObject('winmgmts:')
for p in wmi.InstancesOf('win32_process'):
# print(p.Name)
if p.Name == 'boltzmann.exe':
print("Ok")
In my Code I used process chrome.exe as an example (replace it with boltzmann.exe):
import os
import subprocess
import time
import win32com.client
import signal
wmi=win32com.client.GetObject('winmgmts:')
def getpid(process_name):
import os
return [item.split()[1] for item in os.popen('tasklist').read().splitlines()[4:] if process_name in item.split()]
for p in wmi.InstancesOf('win32_process'):
if p.Name == 'chrome.exe':
process_id = getpid('chrome.exe')
if len(process_id) > 1:
for x in process_id:
pid = int(x)
os.kill(pid, signal.SIGTERM)
else:
pid = int("".join(getpid('chrome.exe')))
os.kill(pid, signal.SIGTERM)
I tried my best to make this all using inbuilt libraries so that you won't have to import lot of stuff
SOME NOTES:
The process name is case sensitive
If the Process is running on higher priority then process won't terminate, and a access denied error will appear
I'm using Pyinstaller to make an .app bundle on Mac OS X, my app generates a config.ini, the script version worked flawlessly, but when it's an .app, it doesn't work.
I'm using ConfigParser to read and write.
I'm using Pyside for GUI.
It doesn't generate my .ini file, hence it doesn't read or write.
Yes, it's an Anime Notifier, got sick of checking it every time.
Code:
import sys
import urllib2
import ConfigParser
import re
import time
import thread
import atexit
from datetime import date
import lxml.html as lx
from PySide.QtCore import *
from PySide.QtGui import *
from pync import Notifier
def layout_widgets(self, layout):
return (layout.itemAt(i) for i in range(layout.count()))
def notify(self):
while True:
config.read('animeConfig.ini')
for newAnime in animeNotify.xpath('//*[#id="frontpage_left_col"]//*[#class="blue"]/text()'):
if config.has_section(newAnime):
for newEp in animeNotify.xpath('//*[#id="frontpage_left_col"]//*[text()="'+newAnime+'"]/parent::h2/following-sibling::h3/a/text()'):
if not config.has_option(newAnime, newEp):
Notifier.notify(newEp+' has been uploaded!', title=newAnime+' '+newEp.lower(), open='http://www.animeseason.com' +
animeNotify.xpath('//*[#id="frontpage_left_col"]//*[text()="'+newEp+'"]/#href')[0])
m = re.findall('\d+', newEp)
config.set(newAnime, newEp, int(m[0]))
with open('animeConfig.ini', 'wb') as configFile:
config.write(configFile)
time.sleep(300)
def checkChecked(self):
while True:
config.read('animeConfig.ini')
for checkbox in self.layout_widgets(self.vLayout):
if checkbox.widget().isChecked() and not config.has_section(checkbox.widget().text()):
config.add_section(checkbox.widget().text())
for anime in animeList.xpath('//*[#class="series_alpha"]/li/span/preceding-sibling::a/text()'):
if config.has_section(anime):
self.EUrl = animeList.xpath('//*[#class="series_alpha"]/li/*[text()="'+anime+'"]/#href')[0]
self.EUrl = lx.parse(urllib2.urlopen("http://www.animeseason.com" + self.EUrl))
for ep in self.EUrl.xpath('//tr/*[#class="text_center"]/a/text()'):
config.set(anime, 'episode '+ep, ep)
with open('animeConfig.ini', 'wb') as configFile:
config.write(configFile)
elif not checkbox.widget().isChecked() and config.has_section(checkbox.widget().text()):
config.remove_section(checkbox.widget().text())
with open('animeConfig.ini', 'wb') as configFile:
config.write(configFile)
time.sleep(300)
I'm also using the thread module so they can operate simultaneously.
Just like so:
thread.start_new_thread(self.notify, ())
thread.start_new_thread(self.checkChecked, ())
Left out the GUI part, because that isn't very interesting, I guess.
I'm using Python 2.7.6 and I have two scripts:
outer.py
import sys
import os
print "Outer file launching..."
os.system('inner.py')
calling inner.py:
import sys
import os
print "[CALLER GOES HERE]"
I want the second script (inner.py) to print the name of the caller script (outer.py).
I can't pass to inner.py a parameter with the name of the first script because I have tons of called/caller scripts and I can't refactor all the code.
Any idea?
One idea is to use psutil.
#!env/bin/python
import psutil
me = psutil.Process()
parent = psutil.Process(me.ppid())
grandparent = psutil.Process(parent.ppid())
print grandparent.cmdline()
This is ofcourse dependant of how you start outer.py.
This solution is os independant.
On linux you can get the process id and then the caller name like so.
p1.py
import os
os.system('python p2.py')
p2.py
import os
pid = os.getppid()
cmd = open('/proc/%d/cmdline' % (pid,)).read()
caller = ' '.join(cmd.split(' ')[1:])
print caller
running python p1.py will yield p1.py
I imagine you can do similar things in other OS as well.
Another, a slightly shorter version for unix only
import os
parent = os.system('readlink -f /proc/%d/exe' % os.getppid())
If applicable to your situation you could also simply pass an argument that lets inner.py differentiate:
import sys
import os
print "Outer file launching..."
os.system('inner.py launcher')
innter.py
import sys
import os
try:
if sys.argv[0] == 'launcher':
print 'outer.py called us'
except:
pass