So, I have decided to write my next project with python3, why? Due to the plan for Ubuntu to gradually drop all Python2 support within the next year and only support Python3. (Starting with Ubuntu 13.04)
gevent and the memcached modules aren't officially ported to Python3.
What are some alternatives, already officially ported to Python3, for gevent and pylibmc or python-memcached?
Circuits has now support for Python 3, try it it is great.
for memcached you probably know alternative: redis+python3
I am stuck in the same point.
Its core is greenlet 0.4.0, which is available in python 3, but not the full libraries (gevent, evenlet or concurrence).
There are some attempts to migrate it, but with no luck.
You can check packages availability in this website: http://py3ksupport.appspot.com/pypi/greenlet
If I find any alternative I would let you know.
pymemcache : A comprehensive, fast, pure-Python memcached client.
Comparison with other clients
pylibmc
The pylibmc library is a wrapper around libmemcached, implemented in
C. It is fast, implements consistent hashing, the full memcached
protocol and timeouts. It does not provide access to the "noreply"
flag. It also isn't pure Python, so using it with libraries like
gevent is out of the question, and its dependency on libmemcached
poses challenges (e.g., it must be built against the same version of
libmemcached that it will use at runtime).
Python-memcache
The python-memcache library implements the entire memcached text
protocol, has a single timeout for all socket calls and has a flexible
approach to serialization and deserialization. It is also written
entirely in Python, so it works well with libraries like gevent.
However, it is tied to using thread locals, doesn't implement
"noreply", can't treat errors as cache misses and is slower than both
pylibmc and pymemcache. It is also tied to a specific method for
handling clusters of memcached servers.
Related
I'm looking to set up a distributed system where there are compute/worker machines running resource-heavy Python 3 code, and there is a single web server that serves the results of the Python computation to clients. I would very much like to write the web server in Node.js.
I've looked into using an RPC framework—specifically, this question lead me to ZeroRPC, but it's not compatible with Python 3 (the main issue is that it requires gevent, which isn't that close to a Python 3 version yet). There doesn't seem to be another viable option for Python–Node.js RPC as far as I can tell.
In light of that, I'm open to using something other than RPC, especially since I've read that the RPC strategy hides too much from the programmer.
I'm also open to using a different language for the web server if that really makes more sense; for example, it may be much simpler from a development point of view to just use Python for the server too.
How can I achieve this?
You have a few options here.
First, it sounds like you like ZeroRPC, and your only problem is that it depends on gevent, which is not 3.x-ready yet.
Well, gevent is close to 3.x-ready. There are a few forks of it that people are testing and even using, which you can see on issue #38. As of mid-September 2014 the one that seems to be getting the most traction is Michal Mazurek's. If you're lucky, you can just do this:
pip3 install git+https://github.com/MichalMazurek/gevent
pip3 install ZeroRPC
Or, if ZeroRPC has metadata that says it's Python 2-only, you can install it from its repo the same way as gevent.
The down side is that none of the gevent-3.x forks are quite battle-tested yet, which is why none of them have been accepted upstream and released yet. But if you're not in a huge hurry, and willing to take a risk, there's a pretty good chance you can start with a fork today, and switch to the final version when it's released, hopefully before you've reached 1.0 yourself.
Second, ZeroRPC is certainly not the only RPC library available for either Python or Node. And most of them have a similar kind of interface for exposing methods over RPC. And, while you may ultimately need something like ZeroMQ for scalability or deployment reasons, you can probably use something simpler and more widespread like JSON-RPC over HTTP—which has a dozen or more Python and Node implementations—for early development, then switch to ZeroRPC later.
Third, RPC isn't exactly complicated, and binding methods to RPCs the way most libraries do isn't that hard. Making it asynchronous can be tricky, but again, for early development you can just use an easy but nonscalable solution—creating a thread for each request—and switch to something else later. (Of course that solution is only easy if your service is stateless; otherwise you're just eliminating all of your async problems and replacing them with race condition problems…)
ZeroMQ offers several transport classes, while the first two will be best suited for the case of a heterogenous RPC layer
ipc://
tcp://
pgm://
epgm://
inproc://
ZeroMQ has ports for both systems, so will definitely serve also your projected needs.
Recently, AWS announced ElastiCache's auto-discovery feature, although they only officially released a client for Java. Does anyone know of a Python Memcached library with support for this feature?
As far as I know, ElastiCache cluster is just a bunch of memcached servers, so you need to give your memcached client the list of all of your servers and have the client do the relevant load balancing.
For Python, you have a couple of options:
pylibmc - which is a wrapper around libmemcached - one of the best and fastest memcached clients there is
python-memcached - a native Python client - very basic, but easy to work with, install and use
They haven't provided a client yet in python to deal with the new auto-discovery feature unfortunately.
There is library for django: django-elasticache.
You can find more details in my answer on same question here.
I wrote a package, it can do elasticache auto discovery:
https://github.com/yupeng820921/elasticache_pyclient
Yes elasticache_pyclient supports aws elasticache auto discovery.
It uses python-memcached implements memcache command, and use hash_ring implements consistent hash.
# Usage (from documentation)
>>> from elasticache_pyclient import MemcacheClient
>>> mc = MemcacheClient('test.lwgyhw.cfg.usw2.cache.amazonaws.com:11211')
>>> mc.set('foo', 'bar')
True
>>> mc.get('foo')
'bar'
Both 'pypy' and 'gevent' are supposed to provide high performance. Pypy is supposedly faster than CPython, while gevent is based on co-routines and greenlets, which supposedly makes for a faster web server.
However, they're not compatible with each other.
I'm wondering which setup is more efficient (in terms of speed/performance):
The builtin Flask server running on pypy
or:
The gevent server, running on CPython
The short answer is: It's faster with PyPy, and it has higher concurrency with gevent.
It is possible to use gevent and PyPy simultaneously (we do this at PubNub for multiple projects) although it can be tricky. Starting with PyPy 2.2, a few patches are required to gevent on their socket implementation. We have an experimental branch on github for it: https://github.com/pubnub/gevent/tree/pypy-hacks - To be used in conjunction with pypycore.
Our recommendation? Use Flask with PyPy and gevent. Get the best of both worlds!
Pypy is compatible with Gevent +1.1 (http://www.gevent.org/changelog.html). It is also compatible with Python 3. So, why not using both? Pypy will improve processing perfomance while Gevent will help in IO bound tasks (e.g. database queries, web requests) by using underground asynchronous connections.
Builtin flask server is a BaseHTTPServer or so, never use. The best scenario is very likely tornado + pypy or something like that. Benchmark before using though. It also depends quite drastically on what you're doing. The web server + web framework benchmarks are typically hello world kind of benchmarks. Is your application really like that?
Cheers, fijal
I just built a small app with the very cool and minimalistic web.py.
I am using a cheap shared hosting package (at WebFaction) and have installed web.py via virtualenv. I cannot use the system python since I need additional packages which I'm not allowed to install into the system python.
So now I start my app with
/home/me/mypython/python myapp.py <myport>
It feels like a cumbersome solution, and I'm not sure how much traffic this setup can take. Any general hints?
Thanks in advance
Is there a reason you're not using fastcgi? That's probably considerably better than trying to use some high-numbered port, particularly since your webhost may not be very happy about that at all. There are a few notes on doing that (on dreamhost, but it should be similar for you) in this post:
http://thefire.us/archives/261
I need to use gets and cas (check and set) commands of memcached from Python application. The only Python client library supporting them I found is Twisted. But Twisted requires quite different design of application, so it's not an option. Is there any other full-featured (not listed on official page) Python library for memcached? Pure Python solution is preferred.
Summary: Thanks to piquadrat there is patch for for python-libmemcached to add support of missing methods. The patch is already applied to trunk in August, but there was no new release yet. Using development branch is OK in many cases, but I'm still looking for stable pure Python solution.
I don't see pylibmc listed there, but I have no idea if it supports those commands you need (edit: it doesn't, sorry).
/edit: if everything else fails, you could perhaps use this patch for python-libmemcached, which adds support for cas and gets.
/edit: The latest git version of pylibmc supports cas and gets