I have two scripts running, one on port :80 and one on port :81. Because some of our users are having issues with stuff happening on the server with port :81, I'm trying to implement a workaround like this;
Old way of doing it, which works fine for most users:
AngularJS app makes request to example.com:81/getpdf/1
Flask server generates PNG and PDF files using PhantomJS and ImageMagick using two separate subprocess.Popen calls and the .wait() method
Using Flask's send_file(), the PDF gets sent back to the user and starts downloading
My workaround for this issue:
AngularJS makes request to example.com/getpdf/1
Flask server (:80) makes a new GET request, r = requests.get(url_with_port_81), faking the old AngularJS request to create the PNG/PDF
Instead of using send_file(), I now return the path of the generated PDF
I return send_file(r.text)
Now, using my workaround, the subprocesses I run to create the PNG/PDFs somehow crash. I have to sudo pkill python, and only when I do so, I'm getting a PNG with no data in the folder on my server.
Basically, PhantomJS has run but hasn't loaded any data (only html/css, but no important stuff that needs to come from the Flask server) and crashes. How is this even possible? I'm just faking the request the browser makes using requests.get, or am I not aware of something here?
I thought subprocess.Popen is non-blocking, so my requests for data could still be answered to fill the PNG/PDFs?
I finally found the reason my subprocess kept crashing.
Apparently, it's a bug in Python < 2.7.3, described here: http://bugs.python.org/issue12786
I had to use 'close_fds=True' in my Popen call and all was fixed. Thanks for your effort either way, #Mark Hildreth!
Related
Im new to Javascript so I would like to keep it at the bare minimum. Is there a way that I can use the Electron to communicate with python script without having node.js? My app is just a basic app that takes some input from users from a html page and I need this text input to be processed in python and write an excel file. So there is not much happening in html so is there a simple way to transfer the input to python file? I want to use Electron because I need this html to be my UI and also I need to distribute this app.
I guess the answer is "no": the main process running node will always be there.
An Electron app consists of a JavaScript main process, and one or more JavaScript renderer processes. There is no built-in Python support. And the user will need Python already installed. So, it sounds like a poor fit for what you need.
The answers here may be useful, and will show how to call the python script. I took a quick look at the flexx toolkit mentioned there. It seems to work with the user's browser, rather than producing a single executable.
Recently i have done it with some sort of trick hope it will help you and there are the following step which i followed-
Created a stand alone python exe using pyinstaller and the exe has flask server internally then i put the flask server inside my node application.
Now we have to initiate our flask server and send a request to it for processing, i have done this with the help of "execFile" function as a child process, for which i have created a function and the code was something like that-
async function callFlask(){
var child = require('child_process').execFile;
child('path_to_python_exe ', function(err, data) {
if(err){
console.error(err);
return;
}
});
}
Now we have initiated our flask server then will send the request with the help of fetch request like
await callFlask().then(
await fetch('host_ip_defined_in_flask'+encodeURIComponent('data'))
Now further we can extend our then chain to get response from python if any and proceed further forexample -
await callFlask().then(
await fetch('host_ip_defined_in_flask'+encodeURIComponent('data'))
.then(res => res.text())
.then(body => console.log(body)))
Here, your output data which python return will be printed in console then you can make your node application behave differently depending on output returned by it.
Also you can package your app with available packagers for electron like electron-packager it will work like a charm.
Also there is are some disadvantage for using python as like it will increase your package size and the process will be difficult to kill from electron after processing so it will increase burden on host machine.
I am assuming that Explaining to create a flask server is not the scope of this question instead if you face any issues let me know, i hope it will help...
My end goal here is to turn on my tv using my Pi. I've already setup and configured everything I can think of, I can access the pi remotely via http, but I constantly get a 404 when trying to call a macro via the REST API. Script runs fine on its own, just can't seem to be called from http.
At this point, I'd take any solution that can be executed via http. Php, cgi, etc, don't care, I just need it to run beside the current setup.
Added to config file as follows:
myscript = /home/pi/harmony.py
harmony.py
import webiopi
import sys
import os
#webiopi.macro
def HarAll():
os.system("/home/pi/Desktop/harmonycontrol/HarmonyHubControl em#i.l passwort start_activity 6463490")
When I attempt to access http://piaddress:8000/macros/HarAll I get a 404. I'm positive I'm missing a step here, for some reason, webIOPi simply isn't adding the macro to the web server.
Got it figured out, this whole time I was trying to test it instead of just adding it to the app I made, I was sending http GET from web browser instead of http POST. Works perfectly.
As far as I know Bottle when used with CherryPy server should behave multi-threaded. I have a simple test program:
from bottle import Bottle, run
import time
app = Bottle()
#app.route('/hello')
def hello():
time.sleep(5)
#app.route('/hello2')
def hello2():
time.sleep(5)
run(app, host='0.0.0.0', server="cherrypy", port=8080)
When I call localhost:8080/hello by opening 2 tabs and refreshing them at the same time, they don't return at the same time but one of them is completed after 5 seconds and the other is completed after 5 more seconds.
But when I call /hello in one tab and /hello2 in another at the same time they finish at the same time.
Why does Bottle not behave multi-threaded when the same end-point is called twice? Is there a way to make it multi-threaded?
Python version: 2.7.6
Bottle version: 0.12.8
CherryPy version: 3.7.0
OS: Tried on both Ubuntu 14.04 64-Bit & Windows 10 64-Bit
I already met this behaviour answering one question and it had gotten me confused. If you would have searched around for related questions the list would go on and on.
The suspect was some incorrect server-side handling of Keep-Alive, HTTP pipelining, cache policy or the like. But in fact it has nothing to do with server-side at all. The concurrent requests coming to the same URL are serialised because of a browser cache implementation (Firefox, Chromium). The best answer I've found before searching bugtrackers directly, says:
Necko's cache can only handle one writer per cache entry. So if you make multiple requests for the same URL, the first one will open the cache entry for writing and the later ones will block on the cache entry open until the first one finishes.
Indeed, if you disable cache in Firebug or DevTools, the effect doesn't persist.
Thus, if your clients are not browsers, API for example, just ignore the issue. Otherwise, if you really need to do concurrent requests from one browser to the same URL (normal requests or XHRs) add random query string parameter to make request URLs unique, e.g. http://example.com/concurrent/page?nocache=1433247395.
It's almost certainly your browser that's serializing the request. Try using two different ones, or better yet a real client. It doesn't reproduce for me using curl.
I use to program on python. I have started few months before, so I am not the "guru" type of developer. I also know the basics of HTML and CSS.
I see few tutorials about node.js and I really like it. I cannot create those forms, bars, buttons etc with my knowledge from html and css.
Can I use node.js to create what user see on browser and write with python what will happen if someone push the "submit" button? For example redirect, sql write and read etc.
Thank you
You can call python scripts in the back end at the node server, in response to button click by user. For that you can use child_process package. It allows you to call programs installed on your machine.
For example here is how to run your script when user POST's something on /reg page:
app.post('/reg', function(request, response){
spawn = require('child_process').spawn;
path = "location of your script";
// create child process of your script and pass two arguments from the request
backend = spawn('python',[path, request.body.name, request.body.email]);
backend.on('exit', function(code) {
console.log(path + ' exited with code ' + code);
if(code==0)
response.render('success'); //show success page if script runs successfully
else
response.redirect('bad');
});
});
Python has to be installed in your system, along with other python libraries you will need. It cannot respond / redirect to requests to node, else why would you use node then. When in Rome, do as the Romans do. Use JavaScript in node, calling external programs is not as fast using JS libraries.
Node.js is a serverside JavaScript environment (like Python). It runs on the server and interacts with the database, generates the HTML that the clients see and isn't actually directly accessed by the browser.
Browsers, on the other hand, run clientside JavaScript directly.
If you want to use Python on the server, there are a bunch of frameworks that you can work with:
Django
Flask
Bottle
Web.py
CherryPy
many, many more...
I think you're thinking about this problem backwards. Node.js lets you run browser Javascript without a browser. You won't find it useful in your Python programming. You're better off, if you want to stick with Python, using a framework such as Pyjamas to write Javascript with Python or another framework such as Flask or Twisted to integrate the Javascript with Python.
I have some test code (as a part of a webapp) that uses urllib2 to perform an operation I would usually perform via a browser:
Log in to a remote website
Move to another page
Perform a POST by filling in a form
I've created 4 separate, clean virtualenvs (with --no-site-packages) on 3 different machines, all with different versions of python but the exact same packages (via pip requirements file), and the code only works on the two virtualenvs on my local development machine(2.6.1 and 2.7.2) - it won't work on either of my production VPSs
In the failing cases, I can log in successfully, move to the correct page but when I submit the form, the remote server replies telling me that there has been an error - it's an application server error page ('we couldn't complete your request') and not a webserver error.
because I can successfully log in and maneuver to a second page, this doesn't seem to be a session or a cookie problem - it's particular to the final POST
because I can perform the operation on a particular machine with the EXACT same headers and data, this doesn't seem to be a problem with what I am requesting/posting
because I am trying the code on two separate VPS rented from different companies, this doesn't seem to be a problem with the VPS physical environment
because the code works on 2 different python versions, I can't imagine it being an incompabilty problem
I'm completely lost at this stage as to why this wouldn't work. I've even 'turned-it-off-and-turn-it-on-again' because I just can't see what the problem could be.
I think it has to be something to do with the final POST coming from a VPS that the remote server doesn't like, but I can't figure out what that could be. I feel like there is something going on under the hood of URLlib that is causing the remote server to dislike the reply.
EDIT
I've installed the exact same Python version (2.6.1) on the VPS as is on my working local copy and it doesn't work remotely, so it must be something to do with originating from a VPS. How could this effect the Http request? Is it something lower level?
You might try setting the debuglevel=1 for urllib2 and see what it comes up with:
import urllib2
h=urllib2.HTTPHandler(debuglevel=1)
opener = urllib2.build_opener(h)
...
This is a total shot in the dark, but are your VPSs 64-bit and your home computer 32-bit, or vice versa? Maybe a difference in default sizes or accuracies of something could be freaking out the server.
Barring that, can you try to find out any information on the software stack the web server is using?
I had similar issues with urllib2 (working with Zimbra's REST api), in the end switched to pycurl with success.
PS
for operations of login/navigate/post, I usually find Mechanize useful and easier to use. Maybe you can give it a show.
Well, it looks like I know why the problem was happening, but I'm not 100% the reason for it.
I simply had to make the server wait (time.sleep()) after it sent the 2nd request (Move to another page) before doing the 3rd request (Perform a POST by filling in a form).
I don't know is it because of a condition with the 3rd party server, or if it's some sort of odd issue with URLlib? The reason it seemed to work on my development machine is presumably because it was slower then the server at running the code?