I'm writing a Python (2.6) extension, and I have a situation where I need to pass an opaque binary blob (with embedded null bytes) to my extension.
Here is a snippet of my code:
from authbind import authenticate
creds = 'foo\x00bar\x00'
authenticate(creds)
which throws the following:
TypeError: argument 1 must be string without null bytes, not str
Here is some of authbind.cc:
static PyObject* authenticate(PyObject *self, PyObject *args) {
const char* creds;
if (!PyArg_ParseTuple(args, "s", &creds))
return NULL;
}
So far, I have tried passing the blob as a raw string, like creds = '%r' % creds, but that not only gives me embedded quotes around the string but also turns the \x00 bytes into their literal string representations, which I do not want to mess around with in C.
How can I accomplish what I need? I know about the y, y# and y* PyArg_ParseTuple() format characters in 3.2, but I am limited to 2.6.
Ok, I figured out a with the help of this link.
I used a PyByteArrayObject (docs here) like this:
from authbind import authenticate
creds = 'foo\x00bar\x00'
authenticate(bytearray(creds))
And then in the extension code:
static PyObject* authenticate(PyObject *self, PyObject *args) {
PyByteArrayObject *creds;
if (!PyArg_ParseTuple(args, "O", &creds))
return NULL;
char* credsCopy;
credsCopy = PyByteArray_AsString((PyObject*) creds);
}
credsCopy now holds the string of bytes, exactly as they are needed.
Related
I am writing a python extension in C and I am trying to pass a bytes object to my function. Obviously the 's' token is for strings; I have tried 'O', 'N', and a few others with no luck. Is there a token I can use to parse a bytes object? If not is there an alternative method to parse bytes objects?
static PyObject *test(PyObject *self, PyObject *args)
{
char *dev;
uint8_t *key;
if(!PyArg_ParseTuple(args, "ss", &dev, &key))
return NULL;
printf("%s\n", dev);
for (int i = 0; i < 32; i++)
{
printf("Val %d: %d\n", i, key[i]);
}
Py_RETURN_NONE;
}
Calling from python: test(b"device", f.read(32)).
If you read the parsing format string docs, it's pretty clear.
s is solely for getting a NUL terminated UTF-8 encoded C-style string from a str object (so it's appropriate for your first argument, but not your second).
y* is specifically called out in the docs with (emphasized in original text):
This is the recommended way to accept binary data.
y# would also work, at the expense of requiring the caller to provide immutable bytes-like objects, excluding stuff like bytearray and mmap.mmaps.
Hi I've a simple example of addressbook.proto I am serializing using the protobuf SerailizeToString() function in python. Here's the code.
import address_pb2
person = address_pb2.Person()
person.id = 1234
person.name = "John Doe"
person.email = "jdoe#example.com"
phone = person.phones.add()
phone.number = "555-4321"
phone.type = address_pb2.Person.HOME
print(person.SerializeToString())
Where address_pb2 is the file I generated from the protobuf compiler. Note that the example is copied from the protoBuf tutorials. This gives me the following string.
b'\n\x08John Doe\x10\xd2\t\x1a\x10jdoe#example.com"\x0c\n\x08555-4321\x10\x01'
Now I want to import this string into c++ protobuf. For this I wrote the following code.
#include <iostream>
#include <fstream>
#include <string>
#include "address.pb.h"
using namespace std;
int main(int argc, char* argv[]) {
GOOGLE_PROTOBUF_VERIFY_VERSION;
tutorial::AddressBook address_book;
string data = "\n\x08""John Doe\x10""\xd2""\t\x1a""\x10""jdoe#example.com\"\x0c""\n\x08""555-4321\x10""\x01""";
if(address_book.ParseFromString(data)){
cout<<"working"<< endl;
}
else{
cout<<"not working" << endl;
}
// Optional: Delete all global objects allocated by libprotobuf.
google::protobuf::ShutdownProtobufLibrary();
return 0;
}
Here I am simply trying to import the script using ParseFromString() fucntion but this doesn't work and I am not sure how it will work as I've been stuck on this since a long time now.
I tried changing the binary a bit to suit the c++ version but still no idea if I am on the right path or not.
How can I achieve this ? Does anybody have a clue ?
In Python, you are serializing a Person object. In C++, you are trying to parse an AddressBook object. You need to use the same type on both ends.
(Note that protobuf does NOT guarantee that it will detect these errors. Sometimes when you parse a message as the wrong type, the parse will appear to succeed, but the content will be garbage.)
There's another issue with your code that happens not to be a problem in this specific case, but wouldn't work in general:
string data = "\n\x08""John Doe\x10""\xd2""\t\x1a""\x10""jdoe#example.com\"\x0c""\n\x08""555-4321\x10""\x01""";
This line won't work if the string has any NUL bytes, i.e. '\x00'. If so, that byte would be interpreted as the end of the string. To avoid this problem you need to specify the length of the data, like:
string data("\n\x08""John Doe\x10""\xd2""\t\x1a""\x10""jdoe#example.com\"\x0c""\n\x08""555-4321\x10""\x01""", 45);
I am using Python C API to connect to my python v2.7.2
As the title suggests, I am looking to use unicode string as key in my dictionary.
I am aware that we can use unicode string as key in python dictionary.
But how is that possible through Python C API.
int PyDict_SetItemString(PyObject *p, const char *key, PyObject *val)
we have the above for using the ascii values. Is there any way for it?
Thanks in advance.
Use
int PyDict_SetItem(PyObject *p, PyObject *key, PyObject *val)
and pass in a:
PyObject *PyUnicode_FromString(const char *u)
as the key
I am extending Python with some C++ code.
One of the functions I'm using has the following signature:
int PyArg_ParseTupleAndKeywords(PyObject *arg, PyObject *kwdict,
char *format, char **kwlist, ...);
(link: http://docs.python.org/release/1.5.2p2/ext/parseTupleAndKeywords.html)
The parameter of interest is kwlist. In the link above, examples on how to use this function are given. In the examples, kwlist looks like:
static char *kwlist[] = {"voltage", "state", "action", "type", NULL};
When I compile this using g++, I get the warning:
warning: deprecated conversion from string constant to ‘char*’
So, I can change the static char* to a static const char*. Unfortunately, I can't change the Python code. So with this change, I get a different compilation error (can't convert char** to const char**). Based on what I've read here, I can turn on compiler flags to ignore the warning or I can cast each of the constant strings in the definition of kwlist to char *. Currently, I'm doing the latter. What are other solutions?
Sorry if this question has been asked before. I'm new.
Does PyArg_ParseTupleAndKeywords() expect to modify the data you are passing in? Normally, in idiomatic C++, a const <something> * points to an object that the callee will only read from, whereas <something> * points to an object that the callee can write to.
If PyArg_ParseTupleAndKeywords() expects to be able to write to the char * you are passing in, you've got an entirely different problem over and above what you mention in your question.
Assuming that PyArg_ParseTupleAndKeywords does not want to modify its parameters, the idiomatically correct way of dealing with this problem would be to declare kwlist as const char *kwlist[] and use const_cast to remove its const-ness when calling PyArg_ParseTupleAndKeywords() which would make it look like this:
PyArg_ParseTupleAndKeywords(..., ..., ..., const_cast<char **>(kwlist), ...);
There is an accepted answer from seven years ago, but I'd like to add an alternative solution, since this topic seems to be still relevant.
If you don't like the const_cast solution, you can also create a write-able version of the string array.
char s_voltage[] = "voltage";
char s_state[] = "state";
char s_action[] = "action";
char s_type[] = "type";
char *kwlist[] = {s_voltage, s_state, s_action, s_type, NULL};
The char name[] = ".." copies the your string to a writable location.
I need to convert between python objects and c strings of various encodings. Going from a c string to a unicode object was fairly simple using PyUnicode_Decode, however Im not sure how to go the other way
//char* can be a wchar_t or any other element size, just make sure it is correctly terminated for its encoding
Unicode(const char *str, size_t bytes, const char *encoding="utf-16", const char *errors="strict")
:Object(PyUnicode_Decode(str, bytes, encoding, errors))
{
//check for any python exceptions
ExceptionCheck();
}
I want to create another function that takes the python Unicode string and puts it in a buffer using a given encodeing, eg:
//fills buffer with a null terminated string in encoding
void AsCString(char *buffer, size_t bufferBytes,
const char *encoding="utf-16", const char *errors="strict")
{
...
}
I suspect it has somthing to do with PyUnicode_AsEncodedString however that returns a PyObject so I'm not sure how to put that into my buffer...
Note: both methods above are members of a c++ Unicode class that wraps the python api
I'm using Python 3.0
I suspect it has somthing to do with PyUnicode_AsEncodedString however that returns a PyObject so I'm not sure how to put that into my buffer...
The PyObject returned is a PyStringObject, so you just need to use PyString_Size and PyString_AsString to get a pointer to the string's buffer and memcpy it to your own buffer.
If you're looking for a way to go directly from a PyUnicode object into your own char buffer, I don't think that you can do that.