So, I create a small program that uses flask to receive some requests and do a few things over selenium. All bits that deal with selenium are in another file that I tried to run first using a thread, and when it did not worked, a process. I believe that the problem is because I use a while true to keep my selenium working. The selenium part knows what to do because it keep checking the variable that I update from them flask part...
This is pretty much my main class that runs the selenium and them start flask, but it never start flask. It get locked on the .start().
if __name__ == "__main__":
# Logging
log_format = '%(asctime)s [%(filename)s:%(lineno)d] %(message)s'
logging.basicConfig(format=log_format,
level=logging.INFO,
stream=sys.stdout)
# Start Selenium
browser = Process(target=selenium_file.run_stuff())
browser.start()
print('TEST')
# Flask
app.run(debug=True)
Not really sure how I could solve this problem (if it's a problem)...
Exchange browser = Process(target=selenium_file.run_stuff()) with browser = Process(target=selenium_file.run_stuff)
You don't pass the function run_stuff but you already execute it and hence it blocks your program until run_stuff returns.
Related
I have a pywebview, that launches a Qt window utilizing the QtWebEngine with Chromium on Windows.
When starting the webview in debug mode, the following output is of particular interest for me:
DevTools listening on ws://127.0.0.1:8228/devtools/browser/<GUID>
This line is output by the Chromium engine itself and I want to make the debug port and GUID available in my application. For that, my idea is to redirect all terminal outputs to a StringIO stream.
Here is a minimum example of what I have done so far, to redirect all outputs:
from contextlib import redirect_stdout, redirect_stderr
from io import StringIO
import logging
from PyQt5 import QtCore
import webview
def qt_message_handler(mode, context, message):
# Redirect message to stdout
print(message)
if __name__ == '__main__':
# Let the handler redirect all qt messages to stdout
QtCore.qInstallMessageHandler(qt_message_handler)
stream = StringIO()
# Redirect stdout and stderr to stream
with redirect_stdout(stream), redirect_stderr(stream):
# Redirect all loging outputs to stream
stream_handler = logging.StreamHandler(stream)
loggers = [logging.getLogger(name) for name in logging.root.manager.loggerDict]
for logger in loggers:
for handler in logger.handlers:
logger.removeHandler(handler)
logger.addHandler(stream_handler)
# Start the webview window
window = webview.create_window('Webview window')
webview.start(gui='qt', debug=True)
Ths approach redirects all outputs to the stream object as intended except the one line output by Chromium mentioned at the beginning.
So i played around with the QTWEBENGINE_CHROMIUM_FLAGS environment variable. Since pywebview overrides this environment variable, I changed it directly in the pywebview module. However no argument passed to Chromium by this environment variable seems to change the output behaviour of this specific line. Looking at the Chromium source code, this makes sense:
std::string message = base::StringPrintf(
"\nDevTools listening on ws://%s%s\n", ip_address->ToString().c_str(),
browser_guid.c_str());
fprintf(stderr, "%s", message.c_str());
fflush(stderr);
Since the message is directly printed to stderr no Chromium logging flags have an influence. But it's clearly evident, that the message gets printed to stderr, so why does a redirect of stderr in Python have no effect on this message? Is there any way to redirect this message in python?
Bonus question:
A further look into the Chromium source code shows, that the debug port and GUID are also written into a file:
if (!output_directory.empty()) {
base::FilePath path =
output_directory.Append(kDevToolsActivePortFileName);
std::string port_target_string = base::StringPrintf(
"%d\n%s", ip_address->port(), browser_guid.c_str());
if (base::WriteFile(path, port_target_string.c_str(),
static_cast<int>(port_target_string.length())) < 0) {
LOG(ERROR) << "Error writing DevTools active port to file";
}
}
I just can't figure out where Chromium stores this data. Where can this file be located on Windows?
I have made some findings that, while not directly answering the question, do lead to a conclusion:
Why is the stderr of Chromium not redirected? This is because Chromium is launched in a child process by the QtWebEngine. While it would be possible, to get the output of a child process in Python, it would require to mess around with the QtWebEngine. At this point I decided, that this is not worth the effort.
Regarding my bonus question, I found out that the file containting the debug port and GUID should be named DevToolsActivePort. However, this file is not created when using the QtWebEngine and I haven't found out yet, why this is the case. See this question for a possible answer in the future: QtWebEngine: DevToolsActivePort file is missing
How did I solve my issue? I found out, that a query exists which returns the websocket url among other things: http://<debug_server_url>/json/version (see this answer). While you can determine the debug port within the QtWebEngine, I solved it for my pywebview application similar to this:
import requests
from PySide6 import QtCore
import webview
debug_server_url = None
def qt_message_handler(mode, context, message):
# Check if the message is the one containing the debug server url
if message.startswith('Remote debugging server started successfully.'):
# Debug server url is at the end of the message
global debug_server_url
debug_server_url = message.split()[-1]
def on_loaded():
if debug_server_url is not None:
websocket_debugger_url = requests.get(debug_server_url + '/json/version').json()['webSocketDebuggerUrl']
print(websocket_debugger_url)
if __name__ == '__main__':
# Define a custom qt message handler
QtCore.qInstallMessageHandler(qt_message_handler)
window = webview.create_window('Webview window')
window.events.loaded += on_loaded
webview.start(gui='qt', debug=True)
When I run my python script which is supposed to open github in edge browser, it works, but closes soon after it is done
#imports
from selenium import webdriver
from selenium.webdriver.edge.service import Service
# Change the directory
#Github
def github():
browser = webdriver.Edge(r'C:\Program Files\Browser Drivers\Edge Drivers\msedgedriver.exe')
browser.get("https://github.com")
if __name__ == "__main__":
github()
The error
DevTools listening on ws://127.0.0.1:52908/devtools/browser/7ebfcee1-74f9-43ba-bf79-
99549f4d7ccf
[2916:7184:0428/150623.910:ERROR:fallback_task_provider.cc(124)] Every renderer should have at
least one task provided by a primary task provider. If a "Renderer" fallback task is shown, it
is a bug. If you have repro steps, please file a new bug and tag it as a dependency of
crbug.com/739782.
I have some simple code testing latency for browsers which opens multiple instances of Selenium:
with Pool(processes=args.number_of_browsers) as pool:
for i in range(args.number_of_browsers):
logging.info("Starting job on browser #" + str(i))
pool.apply_async(run, args=(args.refresh_rate, args.jitter, args.duration, args.url, str(i)))
For the purposes of the question, the run function could be as simple as:
def run():
logging.debug("ANYTHING")
I haven't been able to figure out how to get console output from the pool library.
here a basic working logging example in python
import logging
logging.basicConfig(format='%(asctime)s %(levelname)s:%(message)s',
datefmt='%Y%m%d%H%M%S%p', level=logging.DEBUG)
NODE_NAME = 'Test'
logger = logging.getLogger(NODE_NAME)
logger.info('hello')
correct logging needs more config
I built a python desktop webapp with a fairly simple flask backend which I'm currently serving simply through Chrome. I'm trying to package it using pywebview, but I ran into an intestesting roadblock.
I have code that looks like this in the main file of my package:
import os
import sys
from flasher import app
import httplib
import webview
import threading
def start_prod_server():
runserver(debug=False, reloader=False)
def start_gui():
webview.create_window("MyAppName", "http://localhost:5000", height=1000)
def runserver(debug=True, reloader=False):
port = int(os.environ.get('PORT', 5000))
url = "http://localhost:{}".format(port)
print("ready!")
app.run(host='::', port=port, debug=debug, use_reloader=reloader)
if __name__ == '__main__':
t = threading.Thread(target=start_prod_server)
t.daemon = True
t.start()
# This never works
threading.Timer(1.5, start_gui).start()
sys.exit()
This fails almost systematically, the window shows up blank and the backend doesn't respond to anything (including requests from a separate desktop browser).
I discovered accidentally that if I start two webview threads, it always works:
if __name__ == '__main__':
t = threading.Thread(target=start_prod_server)
t.daemon = True
t.start()
# This works every time
threading.Timer(1.5, start_gui).start()
threading.Timer(1.5, start_gui).start()
sys.exit()
What could be causing this? My understanding of threading in Python is fairly limited, so I'm not sure where to look. Is this likely to be a bug in pywebview, or am I doing something wrong with my threads?
I'm also open to suggestions of alternatives for the webview part, but I want to keep the python/flask part since the app already works fine as it is.
Well, after a bit of digging, I think I found the solution myself.
For whatever reason, webview.create_window() wasn't navigating to the page and getting stuck. Opening up the second window was causing both windows to navigate to the url and allowing the app to continue. (I still haven't understood why)
The problem is solved simply by adding webview.load_url("http://localhost:5000") like so:
if __name__ == '__main__':
t = threading.Thread(target=start_prod_server)
t.daemon = True
t.start()
threading.Timer(1.5, start_gui).start()
webview.load_url("http://localhost:5000")
sys.exit()
Any insight as to why this happens is still welcome though... In the source for pywebview, the two methods are making the exact same call to self.browser.web_browser.Navigate(url)
I am preparing a web service using django in python. When I start web service, I would like to have a main function to run. However, code below did not work in web service. I want to have an initializer function which will fill the structures defined global. I can achieve it by directly defining an init function and call it end of the module but I don't know whether this is the proper way of doing so.
if __name__ == '__main__':
main()
Here similar question was posted. As I understand there is no clear solution for that and I find my way to do this. I defined a dummy function in web service module like:
def warmup(request):
response = HttpResponse()
response.write("ok")
return response
We start application with command sudo python manage.py runserver 0.0.0.0:8081 so I add a piece of code in manage.py such that:
#!/usr/bin/env python
import os
import sys
import requests
import threading
import time
def warmup_request():
print "warmup.."
time.sleep(1)
try:
r = requests.get("http://localhost:8081/warmup/")
if r.content == "ok":
return
except Exception, e:
pass
threading.Timer(0,warmup_request).start()
if __name__ == "__main__":
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "mysite.settings")
from django.core.management import execute_from_command_line
warmup_request()
execute_from_command_line(sys.argv)
If warmup service API returns ok it means loading initial data process has finished. By defining a global flag, we can block all other service API requests before warming up has finished. Or instead defining global flag we may use request_started signal stated here to block all incoming requests with more little effort.