I've discovered a function in the Python C API named PyEval_CallFunction which seems to be useful. It allows you to invoke a Python callable by saying something like:
PyEval_CallFunction(obj, "OOO", a, b, c);
However, I can't find any official documentation on this function. A google search brings up various unofficial tutorials which discuss this function, but:
The function isn't
documented in the official
Python docs, so I don't know if it's
even something that is supposed to
be part of the public API.
Searching the web turns up
inconsistent usage policies. Some
tutorials indicate the
format string needs parenthesis
around the type list, like
"(OiiO)", whereas other times I
see it used without the parenthesis.
When I actually try the function in
a real program, it seems to require
the parenthesis, otherwise it
segfaults.
I'd like to use this function because it's convenient. Does anyone know anything about this, or know why it isn't documented? Is it part of the public API?
I couldn't find many references to it either, but the tutorial you linked to mentions this:
The string format and the following
arguments are as for Py_BuildValue
(XXX so i really should have described
that by now!). A call such as
PyEval_CallFunction(obj, "iii", a, b, c);
is equivalent to
PyEval_CallObject(obj, Py_BuildValue("iii", a, b, c));
I suppose PyEval_CallFunction is not public API, as its value seems rather limited. There is not much of a difference between these two. But then again, I'm not really involved in python extensions, so this is just my view on this.
PyEval_CallObject itself is just a macro around PyEval_CallObjectWithKeywords.
#define PyEval_CallObject(func,arg) \
PyEval_CallObjectWithKeywords(func, arg, (PyObject *)NULL)
On the matter of "What is public API?" here is a recent message from Martin v. Löwis:
Just to stress and support Georg's
explanation: the API is not defined
through the documentation, but instead
primarily through the header files.
All functions declared as PyAPI_FUNC
and not starting with _Py are public
API. There used to be a lot of undocumented API (up to 1.4, there was no API documentation at all, only the extension module tutorial); these days, more and more API gets documented.
http://mail.python.org/pipermail/python-dev/2011-February/107973.html
The reason it isn't documented is because you should be using PyObject_CallFunction instead.
The PyEval_* function family are the raw internal calls for the interpreter evaluation loop. The corresponding documented PyObject_* calls include all the additional interpreter state integrity checks, argument validation and stack protection.
Related
http://www.swig.org/papers/PyTutorial98/PyTutorial98.pdf
It comes from above link:
I know that it is an old publication so it is possible that information is outdated.
I would like to ask:
"Seems to work fine with C++ if you aren't being too clever"
What does it mean, to be too clever?
Is there known situation/case that I shuold be very careful where I am programming C++ modules and extending Python using swig tool?
This PDF appears to be a copy of slides from a presentation given by David Beazley at the 7th International Python Conference. My guess is there was a joke or verbal explanation of what he meant by that phrase.
Seems to work fine with C++ if you aren't being too clever
Here is a link to his website if you want to get in touch with him and ask him directly. His twitter account is dabeaz, which may (or may not) be a better way of contacting him.
The slide is strange and misleading. SWIG does not transform pass-by-value into pass-by-reference at all. Let me try to clarify by an example:
Let's say that as in the example you have the C++ function
double dot_product(Vector a, Vector b);
Now in plain C++ (no SWIG, no wrapping) you may use this function as in the following examples:
1.
Vector a = Vector(1,0);
Vector b = Vector(0,1);
double zero = dot_product(a, b);
2.
Vector *a = new Vector(1,0);
Vector *b = new Vector(0,1);
double zero = dot_product(*a, *b);
In both cases, the function is in fact called in exactly the same way using call-by-value.
SWIG wraps all objects into a structure that contains a pointer to the object, so under the hood SWIG passes pointers around for everything, and therefore uses a syntax as in the second example. But there is no conversion / transformation of call semantics going on whatsoever.
To answer your questions:
"Seems to work fine with C++ if you aren't being too clever" What does it mean, to be too clever?
I have no idea. As stated in another answer, likely a joke.
Is there known situation/case that I shuold be very careful where I am programming C++ modules and extending Python using swig tool?
This is a very broad question, and there certainly are pitfalls, especially related to memory management. However, this particular "transformation" is not an issue.
For reference, here is the relevant entry in the SWIG manual. Note that it is worded differently: The function is transformed to accept pointers. Nothing is said about "call semantics" (since this is a non-issue).
I have a Python program, in which there's a function that should be able to accept a Python lambda, and then pass to a running C++ program.
I was thinking of using pickle.dumps to serialize the lambda into a string. Then I just pass the string to the C++ program, after which I do deserialization using boost.python and get the lambda in C++. However, I just realized that functions cannot be pickled.
I believe there should be existing solutions that I'm not aware of. Thanks in advance for any advice!
Have a look at the C API tutorial page. The link goes to the section about calling Python functions from C. It sounds like you're new to the C API, so start at the top.
Python does not distinguish between the various types of callable objects. Functions, lambdas, objects that have a __call__ method are interchangeable as far as the C API is concerned.
I won't copy/paste everything here as it's very well written in the link, but basically you make a Python module in C (or C++, the API works in both). The first example shows you how. Then you can call a C function from Python, and pass it whatever you need to pass it. All Python objects are exposed to C via PyObject* pointers, and the C API provides many functions to manipulate the objects, convert to/from C datatypes, and do things like call callable objects.
Wrappers like boost.python and SWIG use this API internally.
Both epydoc and Sphinx document generators permit the coder to annotate what the types should be of any/all function parameter.
My question is: Is there a way (or module) that enforces these types (at run-time) when documented in the docstring. This wouldn't be strong-typing (compile-time checking), but (more likely) might be called firm-typing (run-time checking). Maybe raising a "ValueError", or even better still... raising a "SemanticError"
Ideally there would already be something (like a module) similar to the "import antigravity" module as per xkcd, and this "firm_type_check" module would already exist somewhere handy for download.
FYI: The docstring for epydoc and sphinz are as follows:
epydoc:
Functions and Methods parameters:
#param p: ... # A description of the parameter p for a function or method.
#type p: ... # The expected type for the parameter p.
#return: ... # The return value for a function or method.
#rtype: ... # The type of the return value for a function or method.
#keyword p: ... # A description of the keyword parameter p.
#raise e: ... # A description of the circumstances under which a function or method
raises exception e.
Sphinx: Inside Python object description directives, reST field lists with these fields are recognized and formatted nicely:
param, parameter, arg, argument, key, keyword: Description of a parameter.
type: Type of a parameter.
raises, raise, except, exception: That (and when) a specific exception is raised.
var, ivar, cvar: Description of a variable.
returns, return: Description of the return value.
rtype: Return type.
The closest I could find was a mention by Guido in mail.python.org and created by Jukka Lehtosalo at Mypy Examples. CMIIW: mypy cannot be imported as a py3 module.
Similar stackoverflow questions that do not use the docstring per se:
Pythonic Way To Check for A Parameter Type
What's the canonical way to check for type in python?
To my knowledge, nothing of the sort exists, for a few important reasons:
First, docstrings are documentation, just like comments. And just like comments, people will expect them to have no effect on the way your program works. Making your program's behavior depend on its documentation is a major antipattern, and a horrible idea all around.
Second, docstrings aren't guaranteed to be preserved. If you run python with -OO, for example, all docstrings are removed. What then?
Finally, Python 3 introduced optional function annotations, which would serve that purpose much better: http://legacy.python.org/dev/peps/pep-3107/ . Python currently does nothing with them (they're documentation), but if I were to write such a module, I'd use those, not docstrings.
My honest opinion is this: if you're gonna go through the (considerable) trouble of writing a (necessarily half-baked) static type system for Python, all the time it will take you would be put to better use by learning another programming language that supports static typing in a less insane way:
Clojure (http://clojure.org/) is incredibly dynamic and powerful (due to its nature as a Lisp) and supports optional static typing through core.typed (https://github.com/clojure/core.typed). It is geared towards concurrency and networking (it has STM and persistent data structures <3 ), has a great community, and is one of the most elegantly-designed languages I've seen. That said, it runs on the JVM, which is both a good and a bad thing.
Golang (http://golang.org/) feels sort-of Pythonic (or at least, it's attracting a lot of refugees from Python), is statically typed and compiles to native code.
Rust (http://www.rust-lang.org/) is lower-level than that, but it has one of the best type systems I've seen (type inference, pattern matching, traits, generics, zero-sized types...) and enforces memory and resource safety at compile time. It is being developed by Mozilla as a language to write their next browser (Servo) in, so performance and safety are its main goals. You can think of it as a modern take on C++. It compiles to native code, but hasn't hit 1.0 yet and as such, the language itself is still subject to change. Which is why I wouldn't recommend writing production code in it yet.
I have heard many times that C and Python/Ruby code can be integrated.
Now, my question is, can I use, for example a Python/Ruby ORM from within C?
Yes, but the API would be unlikely to be very nice, especially because the point of an ORM is to return objects and C doesn't have objects, hence making access to the nice OOP API unwieldy.
Even in C++ is would be problematic as the objects would be Python/Ruby objects and the values Python/Ruby objects/values, and you would need to convert back and forth.
You would be better off using a nice database layer especially made for C.
For Ruby, yes, you can by using the Ruby C API. After including ruby.h you can use rb_funcall:
To invoke methods directly, you can use the function below
VALUE rb_funcall(VALUE recv, ID mid, int argc, ...)
This function invokes a method on the recv, with the method name specified by the symbol mid.
This will allow you to call any Ruby method, and thus use any Ruby code from C. It won’t be pretty, though. There are a lot of good resources in SO’s Ruby C API tag wiki.
To ask my very specific question I find I need quite a long introduction to motivate and explain it -- I promise there's a proper question at the end!
While reading part of a large Python codebase, sometimes one comes across code where the interface required of an argument is not obvious from "nearby" code in the same module or package. As an example:
def make_factory(schema):
entity = schema.get_entity()
...
There might be many "schemas" and "factories" that the code deals with, and "def get_entity()" might be quite common too (or perhaps the function doesn't call any methods on schema, but just passes it to another function). So a quick grep isn't always helpful to find out more about what "schema" is (and the same goes for the return type). Though "duck typing" is a nice feature of Python, sometimes the uncertainty in a reader's mind about the interface of arguments passed in as the "schema" gets in the way of quickly understanding the code (and the same goes for uncertainty about typical concrete classes that implement the interface). Looking at the automated tests can help, but explicit documentation can be better because it's quicker to read. Any such documentation is best when it can itself be tested so that it doesn't get out of date.
Doctests are one possible approach to solving this problem, but that's not what this question is about.
Python 3 has a "parameter annotations" feature (part of the function annotations feature, defined in PEP 3107). The uses to which that feature might be put aren't defined by the language, but it can be used for this purpose. That might look like this:
def make_factory(schema: "xml_schema"):
...
Here, "xml_schema" identifies a Python interface that the argument passed to this function should support. Elsewhere there would be code that defines that interface in terms of attributes, methods & their argument signatures, etc. and code that allows introspection to verify whether particular objects provide an interface (perhaps implemented using something like zope.interface / zope.schema). Note that this doesn't necessarily mean that the interface gets checked every time an argument is passed, nor that static analysis is done. Rather, the motivation of defining the interface is to provide ways to write automated tests that verify that this documentation isn't out of date (they might be fairly generic tests so that you don't have to write a new test for each function that uses the parameters, or you might turn on run-time interface checking but only when you run your unit tests). You can go further and annotate the interface of the return value, which I won't illustrate.
So, the question:
I want to do exactly that, but using Python 2 instead of Python 3. Python 2 doesn't have the function annotations feature. What's the "closest thing" in Python 2? Clearly there is more than one way to do it, but I suspect there is one (relatively) obvious way to do it.
For extra points: name a library that implements the one obvious way.
Take a look at plac that uses annotations to define a command-line interface for a script. On Python 2.x it uses plac.annotations() decorator.
The closest thing is, I believe, an annotation library called PyAnno.
From the project webpage:
"The Pyanno annotations have two functions:
Provide a structured way to document Python code
Perform limited run-time checking "