Problem: .so(shared object) as library in python works well when python calls it and fails in uWSGI-running python(Django) application.
More info: I've build Go module with go build -buildmode=c-shared -o output.so input.go to call it in Python with
from ctypes import cdll
lib = cdll.LoadLibrary('path_to_library/output.so')
When django project is served via uWSGI the request handler that calling Go library freezes, causing future 504 in Nginx. After getting in "so called freeze", uWSGI is locked there and only restarting helps to enliven app. No logs AT ALL! It just freezes.
Everything works correctly when i run in python interpreter on the same machine.
My thoughts: i've tried to debug this and put a lot of log messages in library, but it won't give much info because everything is fine with library(because it works in interpreter). Library loads correctly, because some log messages that i've putted in library. I think it some sort of uWSGI limitation. I don't think putting uwsgi.ini file is somehow helpful.
Additional info:
Go dependencies:
fmt
github.com/360EntSecGroup-Skylar/excelize
log
encoding/json
OS: CentOS 6.9
Python: Python 3.6.2
uWSGI: 2.0.15
What limitations can be in uWSGI in that type of shared object work and if there a way to overcome them?
Firstly, are you absolutely positive you need to call Go as a library from uWSGI process?
uWSGI are usually for interpreted languages such as PHP, Python, Ruby and others. It bootstraps the interpreter and manages the master/worker processes to handle requests. It seems strange to be using it on Go library.
You mentioned having nginx as your webserver, why not just use your Go program as the http server (which it does great) and call it from nginx directly using it's URL:
location /name/ {
proxy_pass http://127.0.0.1/go_url/;
}
See nginx docs.
If you really want to use Go as a python imported library via a .so module, you have to be aware Go has its own runtime, thread management and may not work well with uWSGI which handles threads/processes in a different way. In this case I'm unable to help you, since I never actually tried this.
If you could clarify your question with what are you actually tring to do, we might me able to answer more helpfully.
Attempts and thoughts
I tried to avoid separation of shared library from my python code, since it requires support of at least one more process, and i would have to rewrite some of the library to create new api.
As #Lu.nemec kindly noted that:
Go has its own runtime, thread management and may not work well with uWSGI which handles threads/processes in a different way
Since uWSGI is the problem i started seaching for a solution there. One of the hopes was installing GCCGO uWSGI plugin somehow solve that problem. But even it's hard to install on old OSes, because it lacks of pre-builded plugins and manual build haven't gone very well, it haven't helped, nothing changes, it still freezes.
And then i thought that i wan't to disable coroutines and that type of stuff that differs from uWSGI and one of the changes that i am able to do is to set GOMAXPROCS
GOMAXPROCS sets the maximum number of CPUs that can be executing simultaneously and returns the previous setting. If n < 1, it does not change the current setting. The number of logical CPUs on the local machine can be queried with NumCPU. This call will go away when the scheduler improves.
And it worked like a charm!!!
The solution
import (
...
"runtime"
)
...
//export yourFunc
func yourFunc(yourArgs argType) {
runtime.GOMAXPROCS(1)
...
}
My previous answer works in some cases. HOWEVER, when i tried to run the same project on another server with same OS, same Python, same uWSGI (version, plugins, config files), same requirements, same .so file, it freezed the save way as i described in answer.
I personally didn't want to run this as separate process and bind it to socket/port and create API for communicating with shared library.
The solution:
Required only separate process. Run with celery.
Some caveats:
1.You cannot run task with task.apply() since it would be run in main application, not in celery:
result = task.apply_async()
while result.ready():
time.wait(5)
2.You neeed to run celery with solo execution pool
celery -A app worker -P solo
Related
I've already looked at other threads on this, but most don't go into enough setup detail which is where I need help.
I have an Ubuntu based VPS running with nginx, serving PHP sites through php-cgi on port 9000.
I'd like to start doing more with Python, so I've written a deployment script which I essentially want to use as a post-receive hook on my local GitLab server as my first python script. I can run this script successfully by running python script.py on the command line but in order to use this as a post-receive hook I need it be able to access it via http.
I looked at this guide on the nginx wiki but partway down is says to:
And start the django fastcgi process:
python ./manage.py runfcgi host=127.0.0.1 port=8080
Now, like I said I am pretty new to python, and I have never used the Django framework. Can anyone assit on how I am supposed to start the fastcgi server? Do I replace ./manage.py with the name of my script? Any help would be appriciated as everything I've found online refers to working with Django.
Do I replace ./manage.py with the name of my script?
No. It's highly unlikely your script is a FastCGI server, or that it can accept HTTP requests of any kind since you mention running it over the command line. (From what little I know of FastCGI, an app supporting it has to be able to handle a stream of requests coming in over stdin in a specific format, so there's definitely some plumbing involved.)
I'd say the easiest approach would be to use some web framework just to act as HTTP/FastCGI middleware. For your use a "microframework" like Flask (or even Paste but I found the documentation inscrutable) sounds like it'd work fine. The idea would be to have two interfaces to your main code, one that can handle command line arguments, and one that can handle a HTTP request, ultimately both would just call one function that actually does the work. (If you want to keep the command-line version of the app.)
The Flask documentation also mentions using uWSGI or standalone workers as deployment options. I'm not familiar with the former; the latter I wouldn't recommend for a simple, low-traffic app for the same reasons as the approach in the next paragraph.
Considering you use a VPS, you might even be able to just run the app as a standalone server process using the http.server module, but I'm not sure that's the better choice unless you absolutely want to avoid using any sort of framework. You'd have to make sure the app starts up if the server is rebooted or that it restarts when it crashes and it seems easier to just have nginx do the job of the supervisor.
UPDATE: Scratch that, it seems that nginx won't handle supervising a FastCGI worker process for you, which would've been the main advantage of the approach. In light of that it doesn't matter which of the three approaches you use since you'll have to set up a service supervisor one way or the other. I'd say go with uWSGI since flup (which is needed for Flask+FastCGI) seems abandoned since 2011, and the uWSGI protocol is apparently supported in nginx natively. Otherwise you'd need to use a different webserver than nginx, one that will manage a FastCGI worker for you. If this is an option, I'd consider Cherokee, which can be configured using a web GUI.
tl;dr: you need to write a (very simple) webapp. While it is feasible to do this without a web framework of any kind, in my opinion using one is easier, since you some (nontrivial) plumbing for free and there's a lot of guidance available on how to deploy them.
I had a similar issue when running fast-cgi and I was told there is no way to fix it: Files being served are stale / cached ; Python as fcgi + web.py + nginx without doing custom work. I was told to use the python method, which invokes a local "web server" to host the python page.
Even doing that, the files served are stale / cached. If I make edits to the files, save and refresh, the python web server is still serving the stale / cached file.
The only way to get it to serve the modified file is to kill (ctrl+c) the script, and then restart...this takes about 5 seconds every-time and seriously impedes my development workflow.
Ideally any change to the script would be reflected next time the page is requested from the web server.
EDIT
#Jordan: Thanks for the suggestions. I've tried #2, which yields the following error:
app = web.application(urls, globals(), web.reloader)
AttributeError: 'module' object has no attribute 'reloader'
Per the documentation here: http://webpy.org/tutorial2.en
I then tried suggestion #4,
web.config.debug = True
Both still cause 'stale' files to get served.
Understandably you want a simple, set it up once and never worry about it again, solution. But you might be making this problem more difficult than it needs to be.
I generally write applications for an apache/modwsgi/nginx stack. If I have a caching problem, I just restart apache and voila, my python files are re-interpreted. I don't remember the commands to restart apache on all of my different boxes (mac's, ubuntu, centos, etc), and I shouldn't need to.
That is what command line aliases are for...
A python application is interpreted before it is run, and when run on a webserver, it is run once and should be considered stateless. This is unlike javascript running in a browser, which can be considered to have state since it is a continually running VM. You can edit javascript while it is running and that is probably fine for most applications of the language.
In python you generally write the code, run it, and if that doesn't work you start over. You don't edit the code in real time. That means you are knowingly saving the source and changing contexts to run it.
I am betting that you are editing your source from a Graphical IDE instead of a command-line editor like vi or emacs (I might be wrong, and I'm not saying there is anything 'wrong' with that). I only write iOS applications using an IDE, everything else I stick to ViM. Why? Because then I am always on the command line, and I am not distracted by anything (animations, mouse pointers, notifications). I finish writing my code, i quickly type ':wq' (write and quit), and then quickly type 'restartweb' (actually i usually type 're' then <\tab> to auto-complete) which is my alias to whatever the command to restart apache is. Voila my python is reinterpreted.
My point is that you should probably keep it simple and use something like an alias to solve your problem. It might not be the coolest thing you could do. But it is what Ninja coders have been doing for the last 20 years to get work done fast and simple.
Now obviously I only suggested a solution for apache, and I have never used web.py before. But the same possible solution still applies. Make a bash script that goes in your project directory, call it something like restart.bash. In it put something like:
rm -r *.pyc
Which will recursively remove all compiled pyc files, forcing your app to reload. Then make an alias in your ~/.bashrc that runs that file
Something like:
alias restartproject="bash /full/path/to/restart.bash"
Magical, now you have a solution that works everywhere, regardless of which type of web server you choose to run your application from.
Edit:
Now you have a solution that works everywhere but on a Windows IIS server. And if you are trying to run python from Windows, you should probably Stahp! hugz
We are using virtualenv right? :) We want to keep our python nice and system-agnostic so we can sell it to anyone right? :) And you should really check out ViM and emacs if you don't use them... you will bang your head against the wall for a week getting used to it, then never want to touch a mouse again after that.
Right, so Python is a compiled language when run on a web server. It's outputting a .pyc file that's the compiled version. Your goal is to tell the web server that the .pyc file is out of date and is no longer valid.
You have a few options:
Delete the relevant .pyc file
For web.py, use the reloader middleware
Send it a HUP signal (I'm lazy and usually do killall -SIGHUP python). You can do this automatically with a file watching tool like watchdog (thanks barracel).
web.config.debug = True should be the default in your application
None of those options are working for you?
I am trying to access video data (e.g., frames, video length) from within python.
Spawning something like mplayer is not an option because of a weird bug which apparently exists between mod_wsgi and python.
pyffmpeg and ffvideo no longer compile, and are not in sync with the newest ffmpeg versions.
I want a simple library, if anyone knows of it.
The bug being referred to would have to be the bug in Python 2.7.2. In short they broke the ability to do a fork from within a sub interpreter. See:
http://bugs.python.org/issue13156
The workaround in mod_wsgi is to force your WSGI application to run in the main Python interpreter. This is done using:
WSGIApplicationGroup %{GLOBAL}
If you are hosting multiple WSGI applications with embedded mode and needed to do this to more than one, you would need to start using daemon mode instead and delegate each WSGI application to separate daemon process group, with all being forced to run in the main interpreter of their respective daemon process groups.
So, any reason you aren't simply using this work around for the bug in Python 2.7.2?
So I've been working on my first Django / Python project and I got my production server up and running. I was wondering if it's possible to make Python/FastCGI (not really sure which is responsible for the task) to recompile my code. As of right now, when I upload updated code, I need to restart the server for the changes to take place. I read that you can add some kind of mysite.fcgi file to lighttpd so it see's that you've updated the code, can you do the same for Nginx / FastCGI?
for anyone else that was interested in my question.. this is only a partial solution, but I ended up finding my answer here: How to gracefully restart django running fcgi behind nginx?
You can just run the script (I'm going to modify it a bit), everytime you edit your code and it will gracefully restart everything without dropping connections.
This is a general guide from the mod_wsgi project that outlines how you can monitor code changes from your app_wsgi.py and restart the current process if any of the modules have changed. You need to restart the Python process, because threads contending over modules could mean that a freshly reloaded module has outdated references from other modules that are still waiting to get discovered for reload.
If you want something that works nicely with nginx, Django and wsgi apps in general, take a peek at Spawning as your wsgi server. It's approach to code reloading is about as graceful as it gets.
It has great documentation, well documented request handling model and it just works, which makes it such a no-brainer to configure. You'd need less than five minutes from now to having your Django instance running on Spawning. Here's another topical blog to get your juices running.
It seems that all roads lead to having to use PyISAPIe to get Django running on IIS6. This becomes a problem for us because it appears you need separate application pools per PyISAPIe/Django instance which is something we'd prefer not to do.
Does anyone have any advice/guidance, or can share their experiences (particularly in a shared Windows hosting environment)?
You need separate application pools no matter what extension you use. This is because application pools split the handler DLLs into different w3wp.exe process instances. You might wonder why this is necessary:
Look at Django's module setting: os.environ["DJANGO_SETTINGS_MODULE"]. That's the environment of the process, so if you change it for one ISAPI handler and then later another within the same application pool, they both point to the new DJANGO_SETTINGS_MODULE.
There isn't any meaningful reason for this, so feel free to convince the Django developers they don't need to do it :)
There are a few ways to hack around it but nothing works as cleanly as separate app pools.
Unfortunately, isapi-wsgi won't fix the Django problem, and I'd recommend that you keep using PyISAPIe (disclaimer: I'm the developer! ;)
Django runs well on any WSGI infrastructure (much like any other modern Python web app framework) and there are several ways to run WSGI on IIS, e.g. see http://code.google.com/p/isapi-wsgi/ .