Had a quick question here, I am used to devpi and was wondering what is the difference between devpi and pypi server?
Is one better than another? Which of this one scale better?
PyPI (Python Package Index)- is the official repository for third-party Python software packages. Every time you use e.g. pip to install a package that is not in the standard it will get downloaded from the PyPI server.
All of the packages that are on PyPI are publicly visible. So if you upload your own package then anybody can start using it. And obviously you need internet access in order to use it.
devpi (not sure what the acronym stands for) - is a self hosted private Python Package server. Additionally you can use it for testing and releasing of your own packages.
Being self hosted it's ideal for proprietary work that maybe you wouldn't want (or can't) share with the rest of the world.
So other features that devpi offers:
PyPI mirror - cache locally any packages that you download form PyPI. This is excellent for CI systems. Don't have to worry if a package or server goes missing. You can even still use it if you don't have internet access.
multiple indexes - unlike PyPI (which has only one index) in devpi you can create multiple indexes. For example a main index for packages that are rock solid and development where you can release packages that are still under development. Although you have to be careful with this because a large amount of indexes can make things hard to track.
The server has a simple web interface where you can you and search for packages.
You can integrate it with pip so that you can use your local devpi server as if you were using PyPI.
So answering you questions:
Is one better than the other? - well these are two different tools really. No clear answer here, depends on what your needs are.
Which scales better? - definitely devpi.
The official website is very useful with good examples: http://doc.devpi.net/latest/
So, I have decided to write my next project with python3, why? Due to the plan for Ubuntu to gradually drop all Python2 support within the next year and only support Python3. (Starting with Ubuntu 13.04)
gevent and the memcached modules aren't officially ported to Python3.
What are some alternatives, already officially ported to Python3, for gevent and pylibmc or python-memcached?
Circuits has now support for Python 3, try it it is great.
for memcached you probably know alternative: redis+python3
I am stuck in the same point.
Its core is greenlet 0.4.0, which is available in python 3, but not the full libraries (gevent, evenlet or concurrence).
There are some attempts to migrate it, but with no luck.
You can check packages availability in this website: http://py3ksupport.appspot.com/pypi/greenlet
If I find any alternative I would let you know.
pymemcache : A comprehensive, fast, pure-Python memcached client.
Comparison with other clients
pylibmc
The pylibmc library is a wrapper around libmemcached, implemented in
C. It is fast, implements consistent hashing, the full memcached
protocol and timeouts. It does not provide access to the "noreply"
flag. It also isn't pure Python, so using it with libraries like
gevent is out of the question, and its dependency on libmemcached
poses challenges (e.g., it must be built against the same version of
libmemcached that it will use at runtime).
Python-memcache
The python-memcache library implements the entire memcached text
protocol, has a single timeout for all socket calls and has a flexible
approach to serialization and deserialization. It is also written
entirely in Python, so it works well with libraries like gevent.
However, it is tied to using thread locals, doesn't implement
"noreply", can't treat errors as cache misses and is slower than both
pylibmc and pymemcache. It is also tied to a specific method for
handling clusters of memcached servers.
Recently, AWS announced ElastiCache's auto-discovery feature, although they only officially released a client for Java. Does anyone know of a Python Memcached library with support for this feature?
As far as I know, ElastiCache cluster is just a bunch of memcached servers, so you need to give your memcached client the list of all of your servers and have the client do the relevant load balancing.
For Python, you have a couple of options:
pylibmc - which is a wrapper around libmemcached - one of the best and fastest memcached clients there is
python-memcached - a native Python client - very basic, but easy to work with, install and use
They haven't provided a client yet in python to deal with the new auto-discovery feature unfortunately.
There is library for django: django-elasticache.
You can find more details in my answer on same question here.
I wrote a package, it can do elasticache auto discovery:
https://github.com/yupeng820921/elasticache_pyclient
Yes elasticache_pyclient supports aws elasticache auto discovery.
It uses python-memcached implements memcache command, and use hash_ring implements consistent hash.
# Usage (from documentation)
>>> from elasticache_pyclient import MemcacheClient
>>> mc = MemcacheClient('test.lwgyhw.cfg.usw2.cache.amazonaws.com:11211')
>>> mc.set('foo', 'bar')
True
>>> mc.get('foo')
'bar'
I was hoping to implement an SVN communicator in my python program so that any file being worked on is automatically stored into the user's SVN account without any user interaction (username and password already provided so Python takes care of storage). Are there any libraries that can handle this kind of communication?
Thanks!
There are Python bindings for SVN. They follow the C API, so present a fairly low-level interface, not very "Pythonic". I'm not sure how easy they are to install these days. I've tried to use it in the past, and found that it requires some digging into the C API documentation to figure out how to make it work.
pysvn provides a more "Pythonic" API. I've used this, and found it very simple in comparison.
There is a pysvn project, which provides python interface to various svn tasks. You could use that invoke the svn commit operation for the user action which you want to act upon.
I currently work with .NET exclusively and would like to have a go at python. To this end I need to set up a python development environment. I guide to this would be handy. I guess I would be doing web development so will need a web server and probably a database. I also need pointers to popular ORM's, an MVC framework, and a testing library.
One of my main criteria with all this is that I want to understand how it works, and I want it to be as isolated as possible. This is important as i am wary of polluting what is a working .NET environment with 3rd party web and database servers. I am perfectly happy using SQLite to start with if this is possible.
If I get on well with this I am also likely to want to set up automated build and ci server (On a virtual machine, probably ubuntu). Any suggestions for these would be useful.
My ultimate aim if i like python is to have similar sorts of tools that i have available with .NET and to really understand the build and deployment of it all. To start with I will settle for a simple development environment that is as isolated as possible and will be easy to remove if I don't like it. I don't want to use IronPython as I want the full experience of developing a python solution using the tools and frameworks that are generally used.
It's not that hard to set up a Python environment, and I've never had it muck up my .NET work. Basically, install Python --- I'd use 2.6 rather than 3.0, which is not yet broadly accepted --- and add it to your PATH, and you're ready to go with the language. I wouldn't recommend using a Ubuntu VM as your development environment; if you're working on Windows, you might as well develop on Windows, and I've had no significant problems doing so. I go back and forth from Windows to Linux with no trouble.
If you have an editor that you're comfortable with that has basic support for Python, I'd stick with it. If not, I've found Geany to be a nice, light, easy-to-use editor with good Python support, though I use Emacs myself because I know it; other people like SCITE, NotePad++, or any of a slew of others. I'd avoid fancy IDEs for Python, because they don't match the character of the language, and I wouldn't bother with IDLE (included with Python), because it's a royal pain to use.
Suggestions for libraries and frameworks:
Django is the standard web framework, but it's big and you have to work django's way; I prefer CherryPy, which is also actively supported, but is light, gives you great freedom, and contains a nice, solid webserver that can be replaced easily with httpd.
Django includes its own ORM, which is nice enough; there's a standalone one for Python, though, which is even nicer: SQL Alchemy
As far as a testing library goes, pyunit seems to me to be the obvious choice
Good luck, and welcome to a really fun language!
EDIT summary: I originally recommended Karrigell, but can't any more: since the 3.0 release, it's been continuously broken, and the community is not large enough to solve the problems. CherryPy is a good substitute if you like a light, simple framework that doesn't get in your way, so I've changed the above to suggest it instead.
Well, if you're thinking of setting up an Ubuntu VM anyway, you might as well make that your development environment. Then you can install Apache and MySQL or Postgres on that VM just via the standard packaging tools (apt-get install), and there's no danger of polluting your Windows environment.
You can either do the actual development on your Windows machine via your favourite IDE, using the VM as a networked drive and saving the code there, or you can just use the VM as a full desktop environment and do everything there, which is what I would recommend.
Install the pre-configured ActivePython release from activestate.
Among other features, it includes the PythonWin IDE (Windows only) which makes it easy to explore Python interactively.
The recommended reference is Dive Into Python, mentioned many times on similar SO discussions.
You should install python 2.4, python 2.5, python 2.6 and python 3.0, and add to your path the one you use more often (Add c:\Pythonxx\ and c:\Pythonxx\Scripts).
For every python 2.x, install easy_install; Download ez_setup.py and then from the cmd:
c:\Python2x\python.exe x:\path\to\ez_setup.py
c:\Python2x\Scripts\easy_install virtualenv
Then each time you start a new project create a new virtual environment to isolate the specific package you needs for your project:
mkdir <project name>
cd <project name>
c:\Python2x\Scripts\virtualenv --no-site-packages .\v
It creates a copy of python and its libraries in .v\Scripts and .\v\Lib. Every third party packages you install in that environment will be put into .\v\Lib\site-packages. The -no-site-packages don't give access to the global site package, so you can be sure all your dependencies are in .\v\Lib\site-packages.
To activate the virtual environment:
.\v\Scripts\activate
For the frameworks, there are many. Django is great and very well documented but you should probably look at Pylons first for its documentions on unicode, packaging, deployment and testing, and for its better WSGI support.
For the IDE, Python comes with IDLE which is enough for learning, however you might want to look at Eclipse+PyDev, Komodo or Wingware Python IDE. Netbean 6.5 has beta support for python that looks promising (See top 5 python IDE).
For the webserver, you don't need any; Python has its own and all web framework come with their own. You might want to install MySql or ProgreSql; it's often better to develop on the same DB you will use for production.
Also, when you have learnt Python, look at Foundations of Agile Python Development or Expert Python Programming.
Using Python on Windows
SO: Python tutorial for total beginners?
Take a look at Pylons, read about WSGI and Paste.
There's nice introductory Google tech talk about them: ReUsable Web Components with Python and Future Python Web Development.
Here's my answer to similar question:
Django vs other Python web frameworks?
NOTE: I included a lot of links to frameworks, projects and what-not, but as a new user I was limited to 1 link per answer. If someone else with enough reputation to edit wants/can edit them into this answer instead of the footnotes, I'd be grateful.
There are some Python IDE's such as Wing IDE[1], I believe some people use Eclipse[2] with a python plugin[3] as well. A lot of people in the #python channel of FreeNode seem to prefer vim, emacs, nano and similar text editors in favor of IDE's. My personal preffered editor is Vim, but if you've mostly done .NET development on windows, presumably with the usual Visual X IDE's, vim and emacs will probably cause you culture shock and you'd be better of using an IDE.
Nearly all python web frameworks* support the WSGI standard[4], most of the large web servers have some sort of plugin to support WSGI, the others support WSGI via fast cgi or plain cgi.
The Zope[5] and Django[6] frameworks have their own ORM's, of other ORM's the two most well known appear to be SQL Alchemy[7] and SQL Object[8]. I only have experience with the former, but both support all possible sane database choices, including SQLite which is installed together with Python and hence perfectly suited to testing and experimenting without polluting your .NET environment with 3rd part web servers and database servers.
The builtin unittest[9] and pyunit[10] frameworks seem to be the preffered solutions for unit testing, but I don't have much experience with these.
bpython[11] and ipython[12] offer enhanced interactive python shells which can greatly help speed up and testing small bits of code and hence worth looking in to.
As for a list of well known and often used web frameworks, look into the following frameworks**:
Twisted[13] is a generic networking framework, which supports almost every single protocol under the sun.
Pylons[14] is light-weight framework aimed at being as flexible as possible and leaving all the choices about what ORM, templating language and what-not to you.
CherryPy[15] tries to provide an interface to expose Python objects to the web.
Django[6] attempts to be an all-in-one solution, builtin template system, ORM, admin pages and internationalization. While the previous frameworks have more DIY wiring together various frameworks work involved with them.
Zope[5] is aimed to be suitable for large enterprise applications, I've heard nothing but good things about it, but consensus seems to be that for smaller you're probably better off with one of the simpler and smaller frameworks.
TurboGears[16] is the framework I know the least about, but it seems to be mostly competition for Django.
This is everything I can think of right now, I'll edit and add stuff if I can think of it. I hope this helps you some in the wonderful world of python.
* - The main exception would be Apache's mod_python, which you should avoid for exactly that reason, use mod_wsgi instead.
** - Word of warning, I have not personally used these frameworks this is just a very short impression I have gotten from talking to other people about each framework, it may be wildly inaccurate. (If anyone has any corrections, do comment and I'll try to edit and fix this answer).
(The http:// is missing since they're recognized as links otherwise)
[1] www.wingware.com/
[2] www.eclipse.org/
[3] pydev.sourceforge.net/
[4] wsgi.org/wsgi/
[5] www.zope.org/
[6] www.djangoproject.com/
[7] www.sqlalchemy.org/
[8] www.sqlobject.org/
[9] docs.python.org/library/unittest.html
[10] pyunit.sourceforge.net/pyunit.html
[11] www.bpython-interpreter.org/
[12] ipython.scipy.org/
[13] twistedmatrix.com/trac/
[14] pylonshq.com/
[15] www.cherrypy.org/
[16] turbogears.org/
Environment?
Here is the simplest solution:
Install Active Python 2.6. Its the Python itself, but comes with some extra handy useful stuff, like DiveintoPython chm.
Use Komodo Edit 5. It is among the good free editor you can use for Python.
Use IDLE. Its the best simplest short snippet editor, with syntax highlighting and auto complete unmatched by most other IDEs. It comes bundled with python.
Use Ipython. Its a shell that does syntax highlighting and auto complete, bash functions, pretty print, logging, history and many such things.
Install easy_install and/or pip for installing various 3rd party apps easily.
Coming from Visual Studio and .Net it will sound a lot different, but its an entirely different world.
For the framework, django works the best. Walk thro the tutorial and you will be impressed enough. The documentation rocks. The community, you have to see for yourself, to know how wonderful it is!!
Python has build in SQL like database and web server, so you wouldn't need to install any third party apps. Remember Python comes with batteries included.
If you've worked with Eclipse before you could give Pydev a try