We Keep Coding

Convert multiple .py file to .exe

I am trying to convert my python program to an executable (.exe) file using PyInstaller, I have 2 python fils: dummy_script.py and GUI.py.
Basically, GUI.py consists of a button that executes dummy_script.py.
dummy_script.py:
import sys
import time
def flush_then_wait():
sys.stdout.flush()
sys.stderr.flush()
time.sleep(0.5)
sys.stdout.write("Script stdout 1\n")
sys.stdout.write("Script stdout 2\n")
sys.stdout.write("Script stdout 3\n")
sys.stderr.write("Total time: 00:05:00\n")
sys.stderr.write("Total complete: 10%\n")
flush_then_wait()
sys.stdout.write("name=Martin\n")
sys.stdout.write("Script stdout 4\n")
sys.stdout.write("Script stdout 5\n")
sys.stderr.write("Total complete: 30%\n")
flush_then_wait()
sys.stderr.write("Elapsed time: 00:00:10\n")
sys.stderr.write("Elapsed time: 00:00:50\n")
sys.stderr.write("Total complete: 50%\n")
sys.stdout.write("country=Nederland\n")
flush_then_wait()
sys.stderr.write("Elapsed time: 00:01:10\n")
sys.stderr.write("Total complete: 100%\n")
sys.stdout.write("Script stdout 6\n")
sys.stdout.write("Script stdout 7\n")
sys.stdout.write("website=www.mfitzp.com\n")
flush_then_wait()
GUI.py:
from PyQt5.QtWidgets import (QApplication, QMainWindow, QPushButton, QPlainTextEdit,
QVBoxLayout, QWidget, QProgressBar)
from PyQt5.QtCore import QProcess
import sys
import re
# A regular expression, to extract the % complete.
progress_re = re.compile("Total complete: (\d+)%")
def simple_percent_parser(output):
"""
Matches lines using the progress_re regex,
returning a single integer for the % progress.
"""
m = progress_re.search(output)
if m:
pc_complete = m.group(1)
return int(pc_complete)
class MainWindow(QMainWindow):
def __init__(self):
super().__init__()
self.p = None
self.btn = QPushButton("Execute")
self.btn.pressed.connect(self.start_process)
self.text = QPlainTextEdit()
self.text.setReadOnly(True)
self.progress = QProgressBar()
self.progress.setRange(0, 100)
l = QVBoxLayout()
l.addWidget(self.btn)
l.addWidget(self.progress)
l.addWidget(self.text)
w = QWidget()
w.setLayout(l)
self.setCentralWidget(w)
def message(self, s):
self.text.appendPlainText(s)
def start_process(self):
if self.p is None: # No process running.
self.message("Executing process")
self.p = QProcess() # Keep a reference to the QProcess (e.g. on self) while it's running.
self.p.readyReadStandardOutput.connect(self.handle_stdout)
self.p.readyReadStandardError.connect(self.handle_stderr)
self.p.stateChanged.connect(self.handle_state)
self.p.finished.connect(self.process_finished) # Clean up once complete.
self.p.start("python",["dummy_script.py"])
def handle_stderr(self):
data = self.p.readAllStandardError()
stderr = bytes(data).decode("utf8")
# Extract progress if it is in the data.
progress = simple_percent_parser(stderr)
if progress:
self.progress.setValue(progress)
self.message(stderr)
def handle_stdout(self):
data = self.p.readAllStandardOutput()
stdout = bytes(data).decode("utf8")
self.message(stdout)
def handle_state(self, state):
states = {
QProcess.NotRunning: 'Not running',
QProcess.Starting: 'Starting',
QProcess.Running: 'Running',
}
state_name = states[state]
self.message(f"State changed: {state_name}")
def process_finished(self):
self.message("Process finished.")
self.p = None
app = QApplication(sys.argv)
w = MainWindow()
w.show()
app.exec_()
The concept used to execute dummy_script.py in GUI.py is:
p = QProcess()
p.start("python3", ['dummy_script.py'])
Now when I pack the whole program in an executable using PyInstaller, dummy_script.py is missing.
How can I make sure dummy_script.py gets included in the PyInstaller bundle?
Reference: https://www.pythonguis.com/tutorials/qprocess-external-programs/

The concept used to execute (dummy_script.py) in (GUI.py) is,
p = QProcess()
p.start("python3", ['dummy_script.py'])
PyInstaller is pretty good at finding and bundling dependencies, but it won't be able to figure out that GUI.py needs dummy_script.py if you run it this way.
A better approach would be to import the code you need and use it directly, e.g something like
from dummy_script import some_function
some_function()
Other options
If you simply modify GUI.py like this, PyInstaller should find dummy_script.py on its own.
If that is not practical for some reason, you should be able to declare it as a hidden import using a spec file. You may have a spec file from an earlier build, but if you need to create a new one you can do that with something like this:
pyi-makespec GUI.py
Then edit the spec file to add dummy_script to the list of hidden imports:
a = Analysis(['GUI.py'],
# ...
hiddenimports=['dummy_script'],
# ...,
)
Then build again from the modified spec file:
pyinstaller foo.spec
That may not work either, since you still aren't importing the other module. In that case you may need to declare it as a data file instead.

It looks like Chris has you covered with a great answer. If you have any further issues with pyinstaller.
Additional references:https://pyinstaller.readthedocs.io/en/stable/usage.html
I use the following:
pathex to find additional paths for imports
hiddenimports = missing or not visible in the python scripts. Sometimes modules have hidden imports themselves.
datas = all data from a module
Sometimes a hook is required, an example hook used for Hydra:
from PyInstaller.utils.hooks import
collect_data_files,collect_submodules
datas = collect_data_files('hydra.conf.hydra')
hiddenimports = collect_submodules('hydra')

Cythonised application not updating

I'm using an updating script and trying to get it to work with cython. It seems the pyupdater script is ignored (no print and not updating). It works fine with standard python, but calling a pyx file it seems it skips over this so no print and update.
If python main.py (Calls a cython script there is no print and to a larger extent it does not update).
def main():
import collections
import os
import sys
import time
import bb.bb_module
import progressbar
import urllib3.poolmanager
from pyupdater.client import Client, AppUpdate
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.chrome.options import Options
from client_config import ClientConfig
if __name__ == '__main__':
main()
Cython pyx file:
import urllib3.poolmanager
orig_urlopen = urllib3.poolmanager.PoolManager.urlopen
def new_urlopen(self, method, url, redirect=True, **kw):
if "s3.amazonaws.com" in url and 'authorization' in self.headers:
self.headers.pop('authorization')
return orig_urlopen(self, method, url, redirect, **kw)
import sys
urllib3.poolmanager.PoolManager.urlopen = new_urlopen
import logging
import http.client as http_client
import logging
from selenium import webdriver
from client_config import ClientConfig
from pyupdater.client import Client, AppUpdate
import progressbar
import sys
bar = None
import sys
import os
import sys
def check_for_update():
stdout_save = sys.stdout
sys.stdout = open(os.devnull, 'w')
def cb(status):
global bar
from tqdm import tqdm
if bar is None:
bar = progressbar.ProgressBar(widgets = [progressbar.Percentage(), progressbar.Bar(), progressbar.FileTransferSpeed(), ' ', progressbar.ETA()
], fd=sys.stdout).start()
zz = float(status['percent_complete'])
bar.update(zz)
stdout_save = sys.stdout
sys.stdout = open(os.devnull, 'w')
client = Client(ClientConfig(), refresh=True,
headers={'basic_auth': 'brofewfefwefewef:Er8qUc9c48LHAtH5mGz5'})
sys.stdout = stdout_save
client.platform = "win"
app_update = client.update_check(ClientConfig.APP_NAME, ClientConfig.APP_VERSION, channel='stable')
if app_update is not None:
app_update.progress_hooks.append(cb)
if app_update.download():
if isinstance(app_update, AppUpdate):
app_update.extract_restart()
return True
else:
app_update.extract()
return True
return False
def main():
import sys
class DevNull:
def write(self, msg):
pass
print('Current version is ', ClientConfig.APP_VERSION)
if check_for_update():
pass
else:
pass
import os
from contextlib import contextmanager
import sys, os
driver = webdriver.Firefox()
if __name__ == "__main__":
main()
driver = webdriver.Chrome()
sys.stdout = sys.__stdout__
print('This will not print anything')
Output:
DevTools listening on ws://127.0.0.1:12284/devtools/browser/b2f98849-8daa-4442-b594-6e7a904c2091
This will not print anything
It looks as if pyupdater is being ignored when calling cython script. It does not print or update.
I've created a repo to reproduce these issues. How can I get it to update when using cython? I have also included a working pythonic version to see the difference if needed
I suspect:
if __name__ == "__main__":
main()
used in the cython script may be the issue as other than pyupdater the cython script runs perfectly.

Pyinstaller doesn't generate my file config-file

I'm using Pyinstaller to make an .app bundle on Mac OS X, my app generates a config.ini, the script version worked flawlessly, but when it's an .app, it doesn't work.
I'm using ConfigParser to read and write.
I'm using Pyside for GUI.
It doesn't generate my .ini file, hence it doesn't read or write.
Yes, it's an Anime Notifier, got sick of checking it every time.
Code:
import sys
import urllib2
import ConfigParser
import re
import time
import thread
import atexit
from datetime import date
import lxml.html as lx
from PySide.QtCore import *
from PySide.QtGui import *
from pync import Notifier
def layout_widgets(self, layout):
return (layout.itemAt(i) for i in range(layout.count()))
def notify(self):
while True:
config.read('animeConfig.ini')
for newAnime in animeNotify.xpath('//*[#id="frontpage_left_col"]//*[#class="blue"]/text()'):
if config.has_section(newAnime):
for newEp in animeNotify.xpath('//*[#id="frontpage_left_col"]//*[text()="'+newAnime+'"]/parent::h2/following-sibling::h3/a/text()'):
if not config.has_option(newAnime, newEp):
Notifier.notify(newEp+' has been uploaded!', title=newAnime+' '+newEp.lower(), open='http://www.animeseason.com' +
animeNotify.xpath('//*[#id="frontpage_left_col"]//*[text()="'+newEp+'"]/#href')[0])
m = re.findall('\d+', newEp)
config.set(newAnime, newEp, int(m[0]))
with open('animeConfig.ini', 'wb') as configFile:
config.write(configFile)
time.sleep(300)
def checkChecked(self):
while True:
config.read('animeConfig.ini')
for checkbox in self.layout_widgets(self.vLayout):
if checkbox.widget().isChecked() and not config.has_section(checkbox.widget().text()):
config.add_section(checkbox.widget().text())
for anime in animeList.xpath('//*[#class="series_alpha"]/li/span/preceding-sibling::a/text()'):
if config.has_section(anime):
self.EUrl = animeList.xpath('//*[#class="series_alpha"]/li/*[text()="'+anime+'"]/#href')[0]
self.EUrl = lx.parse(urllib2.urlopen("http://www.animeseason.com" + self.EUrl))
for ep in self.EUrl.xpath('//tr/*[#class="text_center"]/a/text()'):
config.set(anime, 'episode '+ep, ep)
with open('animeConfig.ini', 'wb') as configFile:
config.write(configFile)
elif not checkbox.widget().isChecked() and config.has_section(checkbox.widget().text()):
config.remove_section(checkbox.widget().text())
with open('animeConfig.ini', 'wb') as configFile:
config.write(configFile)
time.sleep(300)
I'm also using the thread module so they can operate simultaneously.
Just like so:
thread.start_new_thread(self.notify, ())
thread.start_new_thread(self.checkChecked, ())
Left out the GUI part, because that isn't very interesting, I guess.

How to get the caller script name

I'm using Python 2.7.6 and I have two scripts:
outer.py
import sys
import os
print "Outer file launching..."
os.system('inner.py')
calling inner.py:
import sys
import os
print "[CALLER GOES HERE]"
I want the second script (inner.py) to print the name of the caller script (outer.py).
I can't pass to inner.py a parameter with the name of the first script because I have tons of called/caller scripts and I can't refactor all the code.
Any idea?

One idea is to use psutil.
#!env/bin/python
import psutil
me = psutil.Process()
parent = psutil.Process(me.ppid())
grandparent = psutil.Process(parent.ppid())
print grandparent.cmdline()
This is ofcourse dependant of how you start outer.py.
This solution is os independant.

On linux you can get the process id and then the caller name like so.
p1.py
import os
os.system('python p2.py')
p2.py
import os
pid = os.getppid()
cmd = open('/proc/%d/cmdline' % (pid,)).read()
caller = ' '.join(cmd.split(' ')[1:])
print caller
running python p1.py will yield p1.py
I imagine you can do similar things in other OS as well.

Another, a slightly shorter version for unix only
import os
parent = os.system('readlink -f /proc/%d/exe' % os.getppid())

If applicable to your situation you could also simply pass an argument that lets inner.py differentiate:
import sys
import os
print "Outer file launching..."
os.system('inner.py launcher')
innter.py
import sys
import os
try:
if sys.argv[0] == 'launcher':
print 'outer.py called us'
except:
pass

Open a program with python minimized or hidden

What I'm trying to do is to write a script which would open an application only in process list. Meaning it would be "hidden". I don't even know if its possible in python.
If its not possible, I would settle for even a function that would allow for a program to be opened with python in a minimized state maybe something like this:
import subprocess
def startProgram():
subprocess.Hide(subprocess.Popen('C:\test.exe')) # I know this is wrong but you get the idea...
startProgram()
Someone suggested to use win32com.client but the thing is that the program that i want to launch doesn't have a COM server registered under the name.
Any ideas?

It's easy :)
Python Popen Accept STARTUPINFO Structure...
About STARTUPINFO Structure: https://msdn.microsoft.com/en-us/library/windows/desktop/ms686331(v=vs.85).aspx
Run Hidden:
import subprocess
def startProgram():
SW_HIDE = 0
info = subprocess.STARTUPINFO()
info.dwFlags = subprocess.STARTF_USESHOWWINDOW
info.wShowWindow = SW_HIDE
subprocess.Popen(r'C:\test.exe', startupinfo=info)
startProgram()
Run Minimized:
import subprocess
def startProgram():
SW_MINIMIZE = 6
info = subprocess.STARTUPINFO()
info.dwFlags = subprocess.STARTF_USESHOWWINDOW
info.wShowWindow = SW_MINIMIZE
subprocess.Popen(r'C:\test.exe', startupinfo=info)
startProgram()

You should use win32api and hide your window e.g. using win32gui.EnumWindows you can enumerate all top windows and hide your window
Here is a small example, you may do something like this:
import subprocess
import win32gui
import time
proc = subprocess.Popen(["notepad.exe"])
# lets wait a bit to app to start
time.sleep(3)
def enumWindowFunc(hwnd, windowList):
""" win32gui.EnumWindows() callback """
text = win32gui.GetWindowText(hwnd)
className = win32gui.GetClassName(hwnd)
#print hwnd, text, className
if text.find("Notepad") >= 0:
windowList.append((hwnd, text, className))
myWindows = []
# enumerate thru all top windows and get windows which are ours
win32gui.EnumWindows(enumWindowFunc, myWindows)
# now hide my windows, we can actually check process info from GetWindowThreadProcessId
# http://msdn.microsoft.com/en-us/library/ms633522(VS.85).aspx
for hwnd, text, className in myWindows:
win32gui.ShowWindow(hwnd, False)
# as our notepad is now hidden
# you will have to kill notepad in taskmanager to get past next line
proc.wait()
print "finished."

What is the purpose?
if you want a hidden(no window) process working in background, best way would be to write a windows service and start/stop it using usual window service mechanism. Windows service can be easily written in python e.g. here is part of my own service (it will not run without some modifications)
import os
import time
import traceback
import pythoncom
import win32serviceutil
import win32service
import win32event
import servicemanager
import jagteraho
class JagteRahoService (win32serviceutil.ServiceFramework):
_svc_name_ = "JagteRaho"
_svc_display_name_ = "JagteRaho (KeepAlive) Service"
_svc_description_ = "Used for keeping important services e.g. broadband connection up"
def __init__(self,args):
win32serviceutil.ServiceFramework.__init__(self,args)
self.stop = False
def SvcStop(self):
self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
self.log('stopping')
self.stop = True
def log(self, msg):
servicemanager.LogMsg(servicemanager.EVENTLOG_INFORMATION_TYPE,
servicemanager.PYS_SERVICE_STARTED,
(self._svc_name_,msg))
def SvcDoRun(self):
self.log('folder %s'%os.getcwd())
self.ReportServiceStatus(win32service.SERVICE_RUNNING)
self.start()
def shouldStop(self):
return self.stop
def start(self):
try:
configFile = os.path.join(jagteraho.getAppFolder(), "jagteraho.cfg")
jagteraho.start_config(configFile, self.shouldStop)
except Exception,e:
self.log(" stopped due to eror %s [%s]" % (e, traceback.format_exc()))
self.ReportServiceStatus(win32service.SERVICE_STOPPED)
if __name__ == '__main__':
win32serviceutil.HandleCommandLine(AppServerSvc)
and you can install it by
python svc_jagteraho.py--startup auto install
and run it by
python python svc_jagteraho.py start
I will be also be seen in services list e.g. services.msc will show it and you can start/stop it else you can use commandline
sc stop jagteraho

Run Hidden:
from subprocess_maximize import Popen
Popen("notepad.exe",show='hidden', priority=0)
Before the code above, use the following command:
pip install subprocess-maximize

If what is appearing is a terminal, redirect the process's stdout.