Related
This question already has answers here:
How do Rpy2, pyrserve and PypeR compare?
(4 answers)
Closed 7 years ago.
I am quite new to R, and pretty much used to python. I am not so comfortable writing R code. I am looking for python interface to R, which lets me use R packages in pythonic way.
I have done google research and found few packages which can do that:
Rpy2
PypeR
pyRserve
But not sure which one is better ? Which has more contributers and more actively used ?
Please note my main requirement is pythonic way for accessing R packages.
As pointed out by #lgautier, there is already another answer on this subject. I leave my answer here as it adds the experience of approaching R as a novice, knowing Python first.
I use both Python and R and sympathise with your need as a newcomer to R.
Since any answer you get will be subjective, I summarise a few points from my experience:
I use rpy2 as my interface and find it is 'Pythonic', stable, predictable, and effective enough for my needs. I have not used the other packages so this is not a comment on them, rather on the merits of rpy2 itself.
BUT do not expect that there will be an easy way of using R in Python without learning both. I find that adding an interface between the two languages allows ease of coding when you know both, but a nightmare of debugging for someone who is deficient in one of the languages.
My advice:
For most applications, Python has packages that allow you to do most of the things that you want to do in R, from data wrangling to plotting. Check out SciPy, NumPy, pandas, BioPython, matplotlib and other scientific packages, or even the full Anaconda or Enthought python distributions. This allows you to stay within the Python environment and provides you most of the power that you need.
At the same time, you will want R's vast range of specialised packages, so spend some time learning it in an interactive environment. I found it almost impossible to master even basic R on the command line, but RStudio and the tutorials at Quick-R and Learn-R got me going very fast.
Once you know both, then you will do magic with rpy2 without the horrors of cross-language debugging.
New Resources
Update on 29 Jan 2015
This answer has proved popular and so I thought it would be useful to point out two more recent resources:
Ralph Heinkel gave a great talk on this subject at EuroPython 2014. The video on Combining the powerful worlds of Python and R is available on the EuroPython YouTube channel. Quoting him:
The triplet R, Rserve, and pyRserve allows the building up of a network bridge from Python to R: Now R-functions can be called from Python as if they were implemented in Python, and even complete R scripts can be executed through this connection.
It is now possible to combine R and Python using rmagic in IPython/Jupyter greatly easing the work of producing reproducible research and notebooks that combine both languages.
A question about comparing rpy2, pyrserve, and pyper with each other was answered on the site earlier.
Regarding the number of contributors, I'd say that all 3 have a relatively small number. A site like Ohloh can give a more detailled answer.
How actively a package is used is tricky to determine. One indication might be the number of downloads, an other might be the number of posts on mailing lists or the number questions on a site like stackoverflow, the number of other packages using it or citing it, the number of CVs or job openings mentioning the package. As much as I believe that I could give a fair evaluation, I might also be seen as having a conflict of interest. ;-)
All three have their pros and cons. I'd say that you base you choice on that.
My personal experience has been with Rpy, not Rpy2. I used it for a while, but dropped it in favor of using system commands. A typical case for me was running a FORTRAN model using Python scripts, and post-processing with R. In my experience the easiest solution was to create a command line tool using R, which is quite straightforward (at least under Linux). The command line tool could be executed in the root of the model run, and the script would produce a set of R objects and plots in an Routput directory. The advantage of disconnecting R and Python in this way was that I could easily debug the R code separate from the Python code.
I think Rpy really shines when a lot of back and forth communication between R and Python is needed. But if the functionality is nicely separable, and the overhead of disk i/o is not too bad, I would stick to system calls. See ?system for more information regarding system calls, and Rscript for running R scripts as a command line tool.
Regarding your wish to write R code in a Python way, this is not possible as all the solutions require you to write R code in R syntax. For Rpy this means R syntax, but a little different (no . for example). I agree with #gauden that there is no shortcut in using R through Rpy.
We are a medium-sized academic research lab whose main outputs are new statistical methods for analyzing large datasets. We generally develop in R and MATLAB/Octave.
We would like to expand the reach of our work by building simple, wizard-style user interfaces to access our methods, either web-apps like RNAfold or stand-alone applications to analyze private data.
Ideally, we would like the interfaces to do some data checking, to only use FOSS, to run in Mac and Windows environments, and to be able to generate simple charts and graphs that can be output as figures suitable for publication. Also, we like Python because it’s a popular language in the lab and in our research community.
However, we want to be able to develop and release quickly and cheaply. We are lucky to be able to fund one developer in the lab and s/he has to support multiple projects.
There are a lot of groups with the same needs and constraints as us, so it would be useful to be able to develop a consistent long-term strategy for this type of challenge.
Edit
I asked for comments on four possible approaches, included below with summary of comments:
Calling the R and Octave methods from Python.
Answers:
R / Python integration
Rpy (from ChrisC)
pyRserve (from radek)
PypeR (from radek, see answer for paper link)
Using a Python client to call in to an Octave server (from ChrisC)
Re-implementing our methods in Python. In general, this is not easy because our methods rely on other methods also implemented in R and Octave.
Answers:
There was no comment on this approach, it's probably a bad idea.
Using a different framework to build user-interfaces.
Answers:
How about building GUIs and web-apps directly in R? (from hadley, see answer for tool refs)
Other ideas that we haven’t considered, e.g., integrating with workflow management systems like Taverna.
Answers:
Try a visual programming interface like RedR (from chrisamiller)
My preferred answer
I'm first going to look into the approach suggested by Spacedman: using QtDesigner to build the UI for PyQt4 and calling R with RPy. We get a lot of students without a programming background who want to volunteer in the lab, and QtDesigner looks like something that we could train them to use effectively.
I would like develop GUIs directly in R as recommended by hadley but RQt, which I presume that I need to use QtDesigner, does not appear to be in active development (last update was summer of 2009).
Also, it looks like we may have to start implementing our methods exclusively in R, and translating over from MATLAB/Octave to R. In my field, this is probably not a bad idea anyways.
Appendix:
Python libraries to simplify user interface construction:
wxPython (ChrisC)
PyQt4 (from Spacedman)
Other projects that integrate Python, R, and Octave:
SAGE (Dirk Eddelbuettel)
Shogun (Dirk Eddelbuettel)
GUIs to make GUIs:
QtDesigner with PyQt4 (from Spacedman). Note Qt4 is both commercial and available under a LGPL.
Why not continue to develop directly in R? There are a number of packages that allow you to develop GUIs (gWidgets RGtk, tcl/tk, RQt, Rwxwidgets, rjava) or web applications.
I'd go with Python and PyQt4 for the UI, and use Rpy to interface to R.
There's the QtDesigner for interface designing and you can generate python from that. QtAssistant gives you a fully hyperlinked documentation set for Qt which is the best I've ever used.
Well worth it!
That's a tall order -- multi-platform, multi-backend (R and Ocatve), as well and cheap, fast and easy to use!
From the top of my head, look at these
Sage: they have what they call notebooks which are interactive web pages and showed a nice demo at useR! 2010; I think Octave is covered too
Shogun: which also integrates to Python, R and Octave and may have some tools for you to leverage.
I've been looking for something similar - simple rapid GUI in R. It seems there's a new alternative from RStudio guys.
Shiny should be sufficient for small web interfaces without much coding. Seems like a viable option if you need a simple input/output style of UI.
I haven't tried directly connecting Octave with Python beyond setting up Octave apps as a server and having a Python client call in, but I've had some luck integrating R with Python courtesy RPy. In some of our applications we build e.g. a wxPython + matplotlib Python frontend, and call the R scripts directly through RPy.
For R and Python integration you might also want to have a look at
pyRserve
PypeR [more in the paper]
You might try looking at RedR, which lets you wrap R packages in a Python GUI.
I'm currently working on a component of a trading product that will allow a quant or strategy developer to write their own custom strategies. I obviously can't have them write these strategies in natively compiled languages (or even a language that compiles to a bytecode to run on a vm) since their dev/test cycles have to be on the order of minutes.
I've looked at lua, python, ruby so far and really enjoyed all of them so far, but still found them a little "low level" for my target users. Would I need to somehow write my own parser + interpreter to support a language with a minimum of support for looping, simple arithmatic, logical expression evaluation, or is there another recommendation any of you may have? Thanks in advance.
Mark-Jason Dominus, the author of Perl's Text::Template module, has some insights that might be relevant:
When people make a template module
like this one, they almost always
start by inventing a special syntax
for substitutions. For example, they
build it so that a string like %%VAR%%
is replaced with the value of $VAR.
Then they realize the need extra
formatting, so they put in some
special syntax for formatting. Then
they need a loop, so they invent a
loop syntax. Pretty soon they have a
new little template language.
This approach has two problems: First,
their little language is crippled. If
you need to do something the author
hasn't thought of, you lose. Second:
Who wants to learn another language?
If you write your own mini-language, you could end up in the same predicament -- maintaining a grammar and a parser for a tool that's crippled by design.
If a real programming language seems a bit too low-level, the solution may not be to abandon the language but instead to provide your end users with higher-level utility functions, so that they can operate with familiar concepts without getting bogged down in the weeds of the underlying language.
That allows beginning users to operate at a high level; however, you and any end users with a knack for it -- your super-users -- can still leverage the full power of Ruby or Python or whatever.
It sounds like you might need to create some sort of Domain Specific Language (DSL) for your users that could be built loosely on top of the target language. Ruby, Python and Lua all have their various quirks regarding syntax, and to a degree some of these can be massaged with clever function definitions.
An example of a fairly robust DSL is Cucumber which implements a an interesting strategy of converting user-specified verbiage to actual executable code through a series of regular expressions applied to the input data.
Another candidate might be JavaScript, or some kind of DSL to JavaScript bridge, as that would allow the strategy to run either client-side or server-side. That might help scale your application since client machines often have surplus computing power compared to a heavily loaded server.
Custom-made modules are going to be needed, no matter what you choose, that define your firm's high level constructs.
Here are some of the needs I envision -- you may have some of these covered already: a way to get current positions, current and historical quotes, previous performance data, etc... into the application. Define/backtest/send various kinds of orders (limit/market/stop, what exchange, triggers) or parameters of options, etc... You probably are going to need multiple sandboxes for testing as well as the real thing.
Quants want to be able to do matrix operations, stochastic calculus, PDEs.
If you wanted to do it in python, loading NumPy would be a start.
You could also start with a proprietary system designed to do mathematical financial research such as something built on top of Mathematica or Matlab.
I've been working on a Python Algorithmic Trading Library (actually for backtesting, not for real trading). You may want to take a look at it: http://gbeced.github.com/pyalgotrade/
Check out http://www.tadeveloper.com for a backtesting framework using MATLAB as a scripting language. MATLAB has the advantage that it is very powerful but you do not need to be a programmer to use it.
This might be a bit simplistic, but a lot of quant users are used to working with Excel & VBA macros. Would something like VBSCript be usable, as they may have some experience in this area.
Existing languages are "a little "low level" for my target users."
Yet, all you need is "a minimum of support for looping, simple arithmatic, logical expression evaluation"
I don't get the problem. You only want a few features. What's wrong with the list of languages you provided? They actually offer those features?
What's the disconnect? Feel free to update your question to expand on what the problem is.
I would use Common Lisp, which supports rapid development (you have a running image and can compile/recompile individual functions) and tailoring the language to your domain. You would provide functions and macros as building blocks to express strategies, and the whole language would be available to the user for combining these.
Is something along the lines of Processing the complexity level that you're shooting for? Processing is a good example of taking a full-blown language (Java) and reducing/simplifying the available syntax into only a subset applicable to the problem domain (problem domain = visualization in the case of Processing).
Here's a little side-by-side comparison from the Processing docs.
Java:
g.setColor(Color.black)
fillRect(0, 0, size.width, size.height);
Processing:
background(0);
As others have suggested, you may be able to simply write enough high-level functions such that most of the complexity is hidden from the user but you still retain the ability to do more low-level things when necessary. The Wiring language for Arduino follows this strategy of using a thin layer of high-level functions on top of C in order to make it more accessible to non-programmers and hobbyists.
Define the language first -- if possible, use the pseudo-language called EBN, it's very simple (see the Wikipedia entry).
Then once you have that, pick the language. Almost certainly you will want to use a DSL. Ruby and Lua are both really good at that, IMO.
Once you start working on it, you may find that you go back to your definition and tweak it. But that's the right order to do things, I think.
I have been in the same boat building and trading with my own software. Java is not great because you want something higher level like you say. I have had a lot of success using the eclipse project xtext. http://www.eclipse.org/Xtext It does all the plumbing of building parsers etc. for you and using eclipse you can quickly generate code with functional editors. I suggest looking into this as you consider other options as well. This combined with the eclipse modeling framework is very powerful for quickly building DSL's which sounds like you need. - Duncan
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 3 years ago.
Improve this question
I know some Python and I'm really impressed by the language's ease of use. From what I've seen of Objective-C it looks a lot less pretty, but it seems to be the lingua franca for Mac OS X development (which means it has better documentation).
I'm thinking about starting Mac development - will using PyObjC+Python make me a second class citizen?
Yes.
For one thing, as you note, all the documentation is written for Objective-C, which is a very different language.
One difference is method name. In Objective-C, when you send a message to (Python would say “call a method of”) an object, the method name (selector) and arguments are mixed:
NSURL *URL = /*…*/;
NSError *error = nil;
QTMovie *movie = [QTMovie movieWithURL:URL
error:&error];
This isn't possible in Python. Python's keyword arguments don't count as part of the method name, so if you did this:
movie = QTMovie.movieWithURL(URL, error = ???)
you would get an exception, because the QTMovie class has no method named movieWithURL; the message in the Objective-C example uses the selector movieWithURL:error:. movieWithURL: and movieWithURL would be two other selectors.
There's no way they can change this, because Python's keyword arguments aren't ordered. Suppose you have a hypothetical three-argument method:
foo = Foo.foo(fred, bar=bar, baz=baz)
Now, this calls foo:bar:baz:, right?
Not so fast. Foo may also have a method named foo:baz:bar:. Because Python's keyword arguments aren't ordered, you may actually be calling that method. Likewise, if you tried to call foo:baz:bar:, you may actually end up calling foo:bar:baz:. In reality, this case is unlikely, but if it ever happens, you would be unable to reliably call either method.
So, in PyObjC, you would need to call the method like this:
movie = QTMovie.movieWithURL_error_(URL, ???)
You may be wondering about the ???. C doesn't allow multiple return values, so, in Objective-C, the error: argument takes a pointer to a pointer variable, and the method will store an object in that variable (this is called return-by-reference). Python doesn't have pointers, so the way the bridge handles arguments like this is that you pass None, and the method will (appear to) return a tuple. So the correct example is:
movie, error = QTMovie.movieWithURL_error_(URL, None)
You can see how even a simple example deviates from what documentation might show you in Objective-C.
There are other issues, such as the GIL. Cocoa apps are only going to get more concurrent, and you're going to want in on this, especially with tempting classes like NSOperation lying around. And the GIL is a serious liability, especially on multi-core machines. I say this as a Python guy myself (when not writing for Cocoa). As David Beazley demonstrates in that video, it's a cold, hard fact; there's no denying it.
So, if I were going to switch away from Objective-C for my apps, I would take up MacRuby. Unlike with PyObjC and RubyCocoa, messages to Cocoa objects don't cross the language bridge; it's a from-the-ground-up Ruby implementation in Cocoa, with language extensions to better support writing Cocoa code in it.
But that's too far ahead of you. You're just getting started. Start with Objective-C. Better to avoid all impedance mismatches between the language you're using and the one the documentation is written for by keeping them the same language.
Plus, you'll find some bugs (such as messages to deceased objects) harder to diagnose without knowledge of how Objective-C works. You will write these bugs as a new Cocoa programmer, regardless of which language you're writing the code in.
So, learn C, then learn Objective-C. A working knowledge of both shouldn't take more than a few weeks, and at the end of it, you'll be better prepared for everything else.
I won't go into how I learned C; suffice to say that I do not recommend the way I did it. I've heard that this book is good, but I've never owned nor read it. I do have this book, and can confirm that it's good, but it's also not Mac-specific; skip the chapter on how to compile the code, and use Xcode instead.
As for Objective-C: The Hillegass book is the most popular, but I didn't use it. (I have skimmed it, and it looks good.) I read Apple's document on the language, then jumped right in to writing small Cocoa apps. I read some of the guides, with mixed results. There is a Currency Converter tutorial, but it didn't help me at all, and doesn't quite reflect a modern Cocoa app. (Modern apps still use outlets and actions, but also Bindings, and a realistic Currency Converter would be almost entirely a couple of Bindings.)
This really says it all:
As the maintainer of PyObjC for nearly
15 years, I will say it bluntly. Use
Objective-C. You will need to know
Objective-C to really understand Cocoa
anyway and PyObjC is just going to add
a layer of bugs & issues that are
alien to 99% of Cocoa programmers.
a comment in an answer to this question. This question is also interesting.
DO NOT ATTEMPT to avoid learning objective-C if you're going to write apps for the Mac. The purpose of PyObjC and the other language bindings is to let you re-use existing libraries in your apps, not to let you avoid learning the native tools.
Second class citizen seems a bit strong. The Objective-C API's are available from Python as well, should you need them, and that's mostly if you want to make Cocoa apps. But then they are restricted to OS X anyway. Personally, I have no interest in building apps that isn't cross-platform, but that's me. That also means I haven't actually done this, so I don't know how tricky it is, but there was an article in the Python Magazine not long ago, and it didn't look that horrible.
The major drawback of Python is execution time, and that mainly comes from it being a dynamic language. This can be solved with Cython and C-extensions, etc, but then you get a mix of Python + ObjectiveC API's + Cython which can be daunting.
So it depends a lot of what kinds of applications you are going to make. Something uniquely OSX-ish that makes no sense anywhere else? ObjectiveC is probably the ticket. Cross-platform servers, well then Python rocks! Something else? Then it depends.
This is something I've been wondering myself, and although I hope someone comes by with more experience, from what I know you will not be seriously constrained by Python itself. Along with Java and GCC, Python is an excellent way to write native cross-platform applications. Once you get the hang of it you should be able to map example code in Objective C to your Python code.
Since you have access to all libraries and events, everything that you can do in Objective C will be there in Python. Of course, the more OS X-only calls and functions you use, the less easy it will be to port to another platform, but that's beside the point. Usually graphics programming and working with device drivers is somewhat of a limiting factor - but in both cases I'm finding evidence of good support and community libraries (search for Python and Quartz, Lightblue, libhid, PyUSB, for some examples).
The decisive factor for me would be: what is the level of tooling and IDE support that is needed. Apple provides some great software for building new software, but then again with something like Pydev you've got a great place to write Python code too! http://pydev.org/
So give it a try, I'm sure you won't regret it, and there will be a supportive community to draw on for help and insipiration.
You're going to need Objective-C: that's what all the tutorials, documentation, sample code, and everything is written in. In addition to a wilder variety of people being able to help you.
So learn ObjC first. If, on your second or third project, or a year down the road, you start a project that needs a Python module (like, say, Twisted, or SQLAlchemy. But a SERIOUS need like foundation of your app need, where the extra boost your app gets makes everything worth it), then you can write a PyObjC app and get a lot of the speed benefits of that language, with your background in Cocoa.
Just as an extra option, consider that wxPython can produce some pretty good applications on Mac as well as on Linux and Windows. For the most part you can get native appearance but maintain portability with little or no attention to platform-specific issues.
In other words, PyObjC + Python is not the only way to do Mac development with Python.
No you dont need to know Objective C you dont need to use PyObjC , and you wont be a second class citizent.
Unless you want to do something extremely specific to the MAC platform , coding in Objective C or using PyObjC is a really bad idea.
The reason is obvious, once you go the objc route you say a big "goodbye" to other platforms. Its that simple.
Apple does not want you to code for other platforms the same way Microsoft does not want you to code for other platforms. And that is why more and more developers are turning to open source languages like, python, java, ruby etc. Because you dont care what Apple and Microsot , you only care about an App that is the most useful and most easy to develop. And making your App available only for MAC will make it less useful and obviously developing in Objective C is way more difficult.
Python has more than enough libraries to accomodate you , hundrends of them , readily available for the mac platform. I for instance develope a new application in pygame, no its not a game, if I have done the same thing in ObjC or PyObj I would have to rewrite the code for windows and linux. While with pygame my code works exactly the same in windows and linux even though my main platform is macos.
Thats the appeal of most python libraries , they are cross platform. WxPython is another example, someone mentioned that "it does not exactly look natively" , do you want this to stop you from making your application available for windows and linux. Why limit yourself only on the MAC platform ? Do you think the average user will care how natively your app will look. Even macos apps do not look native , many of them introduce their own "eye candy" gui. Not that you cant make WxPython look 100% native, the way you code is always importnat.
Objc makes sense when you intend to develop for Iphone OS , as Apple thought it a great idea to exclude python (and not only python), even though they were forced to include javascript (or else websurfing would have being a nightmare on iphoneos) . Pyjamas, can make python available for iphone os as well (with no hacks or jailbroken phones), but with the obvious limitations since it translates python code to javascript, but still its a valid solution till Apple decide that excluding python from iphone os is a really bad idea.
link text
There is no harm done in studying Objective C though. You can always use the native libraries via pyobjc.
But to be absolutely sincere with you, If my app reaches a dead end with the python libraries ( a very unlikely scenario) I would rather wrap an existing cross platform C/C++ Libraries with Cython than go the objective c pyobjc route and detroy the cross platform ability of my app. The last thing I would be using is anything platoform specifc.
Now if you dont care about other platforms at all, then I guess Objective C can be a valid choice. It certainly looks ugly as hell, but I have heard that it gets much better the more you use it and there are many people that prefer it over C/C++.
I am trying to learn Python and referencing the documentation for the standard Python library from the Python website, and I was wondering if this was really the only library and documentation I will need or is there more? I do not plan to program advanced 3d graphics or anything advanced at the moment.
Edit:
Thanks very much for the responses, they were very useful. My problem is where to start on a script I have been thinking of. I want to write a script that converts images into a web format but I am not completely sure where to begin. Thanks for any more help you can provide.
For the basics, yes, the standard Python library is probably all you'll need. But as you continue programming in Python, eventually you will need some other library for some task -- for instance, I recently needed to generate a tone at a specific, but differing, frequency for an application, and pyAudiere did the job just right.
A lot of the other libraries out there generate their documentation differently from the core Python style -- it's just visually different, the content is the same. Some only have docstrings, and you'll be best off reading them in a console, perhaps.
Regardless of how the other documentation is generated, get used to looking through the Python APIs to find the functions/classes/methods you need. When the time comes for you to use non-core libraries, you'll know what you want to do, but you'll have to find how to do it.
For the future, it wouldn't hurt to be familiar with C, either. There's a number of Python libraries that are actually just wrappers around C libraries, and the documentation for the Python libraries is just the same as the documentation for the C libraries. PyOpenGL comes to mind, but it's been a while since I've personally used it.
As others have said, it depends on what you're into. The package index at http://pypi.python.org/pypi/ has categories and summaries that are helpful in seeing what other libraries are available for different purposes. (Select "Browse packages" on the left to see the categories.)
One very common library, that should also fit your current needs, is the Python Image Library (PIL).
Note: the latest version is still in beta, and available only at Effbot site.
If you're just beginning, all you'll need to know is the stuff you can get from the Python website. Failing that a quick Google is the fastest way to get (most) Python answers these days.
As you develop your skills and become more advanced, you'll start looking for more exciting things to do, at which point you'll naturally start coming across other libraries (for example, pygame) that you can use for your more advanced projects.
It's very hard to answer this without knowing what you're planning on using Python for. I recommend Dive Into Python as a useful resource for learning Python.
In terms of popular third party frameworks, for web applications there's the Django framework and associated documentation, network stuff there's Twisted ... the list goes on. It really depends on what you're hoping to do!
Assuming that the standard library doesn't provide what we need and we don't have the time, or the knowledge, to implement the code we reuse 3rd party libraries.
This is a common attitude regardless of the programming language.
If there's a chance that someone else ever wanted to do what you want to do, there's a chance that someone created a library for it. A few minutes Googling something like "python image library" will find you what you need, or let you know that someone hasn't created a library for your purposes.