Is it possible to use the installed version of SQLite3 from Python, with ctypes? If so, how?
On a Mac, the below works without error:
from ctypes import CDLL
libsqlite3 = CDLL("libsqlite3.dylib")
... but then from https://www.sqlite.org/c3ref/sqlite3.html
Each open SQLite database is represented by a pointer to an instance of the opaque structure named "sqlite3".
(emphasis mine)
which to me suggests you can't really make a ctypes.Structure for the database, say to then pass to sqlite3_open.
(Context: I want to use parts of SQLite from Python that are not exposed by the built-in sqlite3 module)
The sqlite3-API uses an opaque pointer, so in the end there is no need to know its memory layout - one just could use a void-pointer.
For example, opening a sqlite3-database would create such a pointer:
int sqlite3_open(
const char *filename, /* Database filename (UTF-8) */
sqlite3 **ppDb /* OUT: SQLite db handle */
);
i.e. the second parameter is a pointer to pointer. This function will create the structure and give its address to us - no need to know the exact layout of the the structur at all.
Later, we only need the address of this structure to be able to use further functionality, i.e.:
int sqlite3_close(sqlite3*);
The type-safety is something ensured by the compiler, once we have the machine code, the gloves are off and we can pass anything instead of sqlite3* to the function, but we have to ensure that it would work. Any pointer can be replaced by void* as long as it points to a valid memory (i.e. with correct memory layout). That leads to:
import ctypes
libsqlite3 = ctypes.CDLL("libsqlite3.dylib")
sqlite3_handle = ctypes.c_void_p() # nullptr
# pass handle by reference:
res = libsqlite3.sqlite3_open(b"mydb.db", ctypes.byref(sqlite3_handle))
print("open result", res) # check res == 0
print("pointer value:", sqlite3_handle) # address is set
# do what ever needed...
# example usage of handle:
res = libsqlite3.sqlite3_close(sqlite3_handle)
print("close result", res)# check res == 0
sqlite3_handle = None # make sure nobody accesses dangling pointer
This is somewhat quick and dirty: usually one needs to set argument-types and return-value-type. But in the functions above, defaults get correct behavior so I've skipped this (otherwise important) step.
Based on ead's answer, this is a more complete example of how to use libsqlite3 from Python, which is also at https://gist.github.com/michalc/a3147997e21665896836e0f4157975cb
The below defines a (generator) function, query
from contextlib import contextmanager
from collections import namedtuple
from ctypes import cdll, byref, string_at, c_char_p, c_int, c_double, c_int64, c_void_p
from sys import platform
def query(db_file, sql, params=()):
libsqlite3 = cdll.LoadLibrary({'linux': 'libsqlite3.so', 'darwin': 'libsqlite3.dylib'}[platform])
libsqlite3.sqlite3_errstr.restype = c_char_p
libsqlite3.sqlite3_errmsg.restype = c_char_p
libsqlite3.sqlite3_column_name.restype = c_char_p
libsqlite3.sqlite3_column_double.restype = c_double
libsqlite3.sqlite3_column_int64.restype = c_int64
libsqlite3.sqlite3_column_blob.restype = c_void_p
libsqlite3.sqlite3_column_bytes.restype = c_int64
SQLITE_ROW = 100
SQLITE_DONE = 101
SQLITE_TRANSIENT = -1
SQLITE_OPEN_READWRITE = 0x00000002
bind = {
type(0): libsqlite3.sqlite3_bind_int64,
type(0.0): libsqlite3.sqlite3_bind_double,
type(''): lambda pp_stmt, i, value: libsqlite3.sqlite3_bind_text(pp_stmt, i, value.encode('utf-8'), len(value.encode('utf-8')), SQLITE_TRANSIENT),
type(b''): lambda pp_stmt, i, value: libsqlite3.sqlite3_bind_blob(pp_stmt, i, value, len(value), SQLITE_TRANSIENT),
type(None): lambda pp_stmt, i, _: libsqlite3.sqlite3_bind_null(pp_stmt, i),
}
extract = {
1: libsqlite3.sqlite3_column_int64,
2: libsqlite3.sqlite3_column_double,
3: lambda pp_stmt, i: string_at(
libsqlite3.sqlite3_column_blob(pp_stmt, i),
libsqlite3.sqlite3_column_bytes(pp_stmt, i),
).decode(),
4: lambda pp_stmt, i: string_at(
libsqlite3.sqlite3_column_blob(pp_stmt, i),
libsqlite3.sqlite3_column_bytes(pp_stmt, i),
),
5: lambda pp_stmt, i: None,
}
def run(func, *args):
res = func(*args)
if res != 0:
raise Exception(libsqlite3.sqlite3_errstr(res).decode())
def run_with_db(db, func, *args):
if func(*args) != 0:
raise Exception(libsqlite3.sqlite3_errmsg(db).decode())
#contextmanager
def get_db(db_file):
db = c_void_p()
run(libsqlite3.sqlite3_open_v2, db_file.encode(), byref(db), SQLITE_OPEN_READWRITE, None)
try:
yield db
finally:
run_with_db(db, libsqlite3.sqlite3_close, db)
#contextmanager
def get_pp_stmt(db, sql):
pp_stmt = c_void_p()
run_with_db(db, libsqlite3.sqlite3_prepare_v3, db, sql.encode(), -1, 0, byref(pp_stmt), None)
try:
yield pp_stmt
finally:
run_with_db(db, libsqlite3.sqlite3_finalize, pp_stmt)
with \
get_db(db_file) as db, \
get_pp_stmt(db, sql) as pp_stmt:
for i, param in enumerate(params):
run_with_db(db, bind[type(param)], pp_stmt, i + 1, param)
row_constructor = namedtuple('Row', (
libsqlite3.sqlite3_column_name(pp_stmt, i).decode()
for i in range(0, libsqlite3.sqlite3_column_count(pp_stmt))
))
while True:
res = libsqlite3.sqlite3_step(pp_stmt)
if res == SQLITE_DONE:
break
if res != SQLITE_ROW:
raise Exception(libsqlite3.sqlite3_errstr(res).decode())
yield row_constructor(*(
extract[libsqlite3.sqlite3_column_type(pp_stmt, i)](pp_stmt, i)
for i in range(0, len(row_constructor._fields))
))
which can be used as, for example:
for row in query('my.db', 'SELECT * FROM my_table WHERE a = ?;', ('b',)):
print(row)
Related
I'm trying to use following function with ctypes and I do have troubles how declare all the parameters and variables right.
The Documentation of the C code is following,
/* global variables */
int main ()
char sDeviceSerialNumber[32];
FEUSB_GetScanListPara( 0, "Device-ID", sDeviceSerialNumber ) ;
sDeviceSerialNumber is supposed to be a return value of the function which I need in Python for further use.
Python code:
def FEUSB_GetScanListPara(iIndex, cPara):
libfeusb.FEUSB_GetScanListPara.argtypes = [ctypes.c_int,
ctypes.c_wchar_p,
ctypes.POINTER(ctypes.c_char_p)
]
libfeusb.FEUSB_GetScanListPara.restype = ctypes.c_int
iIndex = ctypes.c_int(iIndex)
cValue_buffer = ctypes.create_string_buffer(32)
cValue = ctypes.c_char_p(ctypes.addressof(cValue_buffer))
value = libfeusb.FEUSB_GetScanListPara(iIndex,
cPara,
ctypes.byref(cValue)
)
if __name__ == "__main__":
i = 0
RFID.FEUSB_GetScanListPara(i, "Device-ID")
When I call the function with the code above, I get an error code, FEUSB_ERR_UNKNOWN_PARAMETER, therefore I assume that I do not declare the parameters correctly.
Any input is appreciated!
EDIT 1
def FEUSB_GetScanListPara(iIndex, cPara):
libfeusb.FEUSB_GetScanListPara.argtypes = [ctypes.c_int,
ctypes.c_char_p,
ctypes.c_char_p
]
libfeusb.FEUSB_GetScanListPara.restype = ctypes.c_int
cValue = ctypes.create_string_buffer(32)
value = libfeusb.FEUSB_GetScanListPara(iIndex, cPara,
ctypes.byref(cValue))
print("1.0", cPara, "back value", " = ", value)
print("1.1", cPara, " = ", cValue.value)
print("######")
if __name__ == "__main__":
data = RFID.FEUSB_GetScanListPara(i, b"Device-ID")
Python Console:
FEUSB_ClearScanList = 0
FEUSB_Scan = 0
FEUSB_GetScanListSize = 1
Traceback (most recent call last):
File "C:\xxxxx\3.1_ObidRFID_test\OBID_RFID_06.py", line 265, in <module>
data = RFID.FEUSB_GetScanListPara(i, b"Device-ID")
File "C:\xxxxx\3.1_ObidRFID_test\OBID_RFID_06.py", line 89, in FEUSB_GetScanListPara
value = libfeusb.FEUSB_GetScanListPara(iIndex, cPara, ctypes.byref(cValue))
ArgumentError: argument 3: <class 'TypeError'>: wrong type
EDIT 2
working code
def FEUSB_GetScanListPara(iIndex, cPara):
libfeusb.FEUSB_GetScanListPara.argtypes = [ctypes.c_int,
ctypes.c_char_p,
ctypes.c_char_p
]
libfeusb.FEUSB_GetScanListPara.restype = ctypes.c_int
cValue = ctypes.create_string_buffer(32)
return_value = libfeusb.FEUSB_GetScanListPara(0, b'Device-ID',
cValue)
Your declaration of .argtypes would match the C prototype of:
int FEUSB_GetScanListPara(int, wchar_t*, char**)
You haven't provided the exact C prototype, but from your example of:
FEUSB_GetScanListPara( 0, "Device-ID", sDeviceSerialNumber ) ;
and knowing wchar_t* is not a common interface parameter, you probably actually have simple char* declarations like:
int FEUSB_GetScanListPara(int, const char*, char*);
I'm assuming the 2nd parameter is an input parameter and 3rd parameter is an output parameter. Note that c_char_p corresponds to a byte string so use b'DeviceID' for cPara. Also if you have to allocate the buffer, the 3rd parameter is unlikely to be char**. If the API itself is not returning a pointer, but filling out an already allocated buffer, char* and hence ctypes.c_char_p is appropriate. You correctly use create_string_buffer() for an output parameter.
Note you don't need to wrap iIndex in a c_int. From .argtypes, ctypes knows the 1st parameter is a c_int and converts it for you. That's also the default if no .argtypes is provided, but better to be explicit and provide .argtypes.
This code should work. I don't have the DLL to verify:
import ctypes as ct
libfeusb = CDLL('./FESUB') # assuming in same directory
libfeusb.FEUSB_GetScanListPara.argtypes = ct.c_int, ct.c_char_p, ct.c_char_p
libfeusb.FEUSB_GetScanListPara.restype = ct.c_int
cValue = ct.create_string_buffer(32)
ret = libfeusb.FEUSB_GetScanListPara(0, b'Device-ID', cValue)
if ret == 0:
print(cValue.value)
else:
print('error:',ret)
If you still have issues, edit your question with a minimal, reproducible example. Make sure to provide the real C prototype.
We have an old, legacy database that needs input from another system. SendInput method of data input into database forms is slow and unreliable, setting clipboard and then ^v is not reliable either (I have no idea why, but database interface is very old, early 2000s). After a lot of fiddling I discovered that using SendMessage to set text and then sending VK_RETURN is fast (much faster than SendInput/keybd_event) and reliable with our database. Now this code in plain C works:
HWND fghwnd = GetForegroundWindow();
DWORD threadId = GetWindowThreadProcessId(fghwnd, NULL);
DWORD myId = GetCurrentThreadId();
if (AttachThreadInput(myId, threadId, true)) {
HWND ctrl = GetFocus();
SendMessage(ctrl, WM_SETTEXT, 0, (LPARAM) sendbuf); // TESTING
PostMessage(ctrl, WM_KEYDOWN, VK_RETURN, 0);
PostMessage(ctrl, WM_KEYUP, VK_RETURN, 0);
AttachThreadInput(myId, threadId, false);
} else {
printf("\nError: AttachThreadInput failure!\n");
}
But this one in python does not:
foregroundHwnd = win32gui.GetForegroundWindow()
foregroundThreadID = win32process.GetWindowThreadProcessId(foregroundHwnd)[0]
ourThreadID = win32api.GetCurrentThreadId()
if foregroundThreadID != ourThreadID:
win32process.AttachThreadInput(foregroundThreadID, ourThreadID, True)
focus_whd = win32gui.GetFocus()
win32gui.SendMessage(focus_whd, win32con.WM_SETTEXT, None, "test text")
win32gui.PostMessage(focus_whd, win32con.WM_KEYDOWN, win32con.VK_RETURN, None)
win32gui.PostMessage(focus_whd, win32con.WM_KEYUP, win32con.VK_RETURN, None)
win32process.AttachThreadInput(foregroundThreadID, ourThreadID, False)
The trouble is, most of our new logic in python. I turned that C code into a small python module and it works, but as result now I've got dependency on Microsoft's huge compiler and a lot of fiddling with module building. I'd like to have a python-only solution.
Any ideas why this python code does not work? These system calls look the same...
Yes, AttachThreadInput failed. According to the comment here https://toster.ru/q/79336 win32process.GetWindowThreadProcessId returns wrong value, ctypes must be used. This code works:
"""
Fast "paste" implemented via calls to Windows internals, sends parameter
string and RETURN after that
Usage:
from paste import paste
paste("test")
"""
import time
import random
import string
from ctypes import windll
import ctypes
import win32con
def random_string(string_length=10):
"""Generate a random string of fixed length """
letters = string.ascii_lowercase
return ''.join(random.choice(letters) for i in range(string_length))
ERROR_INVALID_PARAMETER = 87
def paste(text_to_paste):
"""Fast "paste" using WM_SETTEXT method + Enter key"""
current_hwnd = windll.user32.GetForegroundWindow()
current_thread_id = windll.kernel32.GetCurrentThreadId()
thread_process_id = windll.user32.GetWindowThreadProcessId(current_hwnd, None)
if thread_process_id != current_thread_id:
res = windll.user32.AttachThreadInput(thread_process_id, current_thread_id, True)
# ERROR_INVALID_PARAMETER means that the two threads are already attached.
if res == 0 and ctypes.GetLastError() != ERROR_INVALID_PARAMETER:
print("WARN: could not attach thread input to thread {0} ({1})"
.format(thread_process_id, ctypes.GetLastError()))
return
focus_whd = windll.user32.GetFocus()
windll.user32.SendMessageW(focus_whd, win32con.WM_SETTEXT, None, text_to_paste)
windll.user32.PostMessageW(focus_whd, win32con.WM_KEYDOWN, win32con.VK_RETURN, None)
windll.user32.PostMessageW(focus_whd, win32con.WM_KEYUP, win32con.VK_RETURN, None)
res = windll.user32.AttachThreadInput(thread_process_id, current_thread_id, True)
if __name__ == '__main__':
time.sleep(5) # time to switch to the target
# paste random 150 char string
paste(random_string(150))
I'v done researched a lot about this problem.. But there's no where and I couldn't find it. I'm trying to call double c structure by calling c dll.
My question is, did i right way to declare "Class Structure" in python? I couldn't think that i'm right on my way. Because even though the Functions that I want to call from dll, It didn't come output anything.
[Visual C++/C]
I did try to C Syntax code,
typedef sturct {
int nBoardNum;
struct{
char pBoardName[16];
int nBoardID;
}BOARDINDEX[8];
}AAPBOARDINFO, *PAAPBOARDINFO;
HANDLE AcapOpen(char* cpBoardName, int nBoardNo, int nCh)
[Python]
I changed Python Syntax like this.
import ctypes as c
class BOARDINDEX(c.Structure):
_field_ = [("nBoardName", c.c_char_p * 16),("nBoardID", c.c_int)]
class AAPBOARDINFO(c.Structure):
_field_ = [("nBoardNum", c.c_int), ("BOARDINDEX", BOARDINDEX * 8)]
AapLib2 = c.WinDLL("AapLib2.dll")
BoardName = ["ABC","FWD","HGW"]
BoardNo = 0
ch = 1
output = Open(BoardName, BoardNo, ch)
def Open(BoardName, BoardNo, ch)
func = AapLib2.AcapOpen
func.argtypes = [c.POINTER(BOARDINDEX),c.c_int, c.c_int]
func.restype = c.c_int
ref = BOARDINDEX()
res = func(c.byref(ref.nBoardName),BoardNo, ch)
return res
Nothing outcomes when call Open() function...
please consider my request and any answer would be great...
Everything you need to know, can be found in the [Python.Docs]: ctypes - A foreign function library for Python.
There are a couple of problems with the code:
Structure members are specified in the _fields_ (not _field_) attribute
char pBoardName[16] maps to ctypes.c_char * 16 (not c_char_p)
HANDLE should be mapped to wintypes.HANDLE
Function prototype differs between C and Python
Using globals like AapLib2 is best to be be avoided, but I left them unchanged as they are outside the question scope
#1. and #3. will generate Undefined Behavior! Check [SO]: C function called from Python via ctypes returns incorrect value (#CristiFati's answer) for more details.
Here's a modified version of your code. Needless to say that I didn't actually test it, as I don't have the .dll:
#!/usr/bin/env python
import ctypes as cts
import sys
from ctypes import wintypes as wts
class BOARDINDEX(cts.Structure):
_fields_ = [
("nBoardName", cts.c_char * 16),
("nBoardID", cts.c_int),
]
class AAPBOARDINFO(cts.Structure):
_fields_ = [
("nBoardNum", cts.c_int),
("BOARDINDEX", BOARDINDEX * 8),
]
def open_board(board_name, board_no, ch):
AcapOpen = aaplib2.AcapOpen
AcapOpen.argtypes = (cts.c_char_p, cts.c_int, cts.c_int)
AcapOpen.restype = wts.HANDLE
ref = BOARDINDEX(board_name, board_no) # Probably this line should be replaced by the 3 (commented) ones below (AcapGetBoardInfo prototype would have to be specified as well)
#abi = AAPBOARDINFO()
#AcapGetBoardInfo(cts.byref(abi))
#ref = abi.BOARDINDEX[0]
res = AcapOpen(ref.nBoardName, ref.nBoardID, ch)
return res
def main(*argv):
board_names = (
"ABC",
"FWD",
"HGW",
)
board_no = 0
ch = 1
aaplib2 = cts.WinDLL("AapLib2.dll")
output = open_board(board_names[0], board_no, ch)
print(output)
if __name__ == "__main__":
print("Python {:s} {:03d}bit on {:s}\n".format(" ".join(elem.strip() for elem in sys.version.split("\n")),
64 if sys.maxsize > 0x100000000 else 32, sys.platform))
rc = main(*sys.argv[1:])
print("\nDone.\n")
sys.exit(rc)
Let me know how this works out.
I've been trying to use the digi Advanced Device Discovery protocol library with python using ctypes.
the context:
Windows 7 x64
python 2.7.5
dll library
here's my current code:
guid = (0xbf6db409,0xc83d,0x44a3,0xa3,0x6d,0x21,0x79,0x7d,0x2f,0x73,0xf9)
class ADDP():
from ctypes import Structure
class GUID(Structure):
from ctypes.wintypes import DWORD,WORD,BYTE
_fields_ = [("Data1",DWORD),
("Data2",WORD),
("Data3",WORD),
("Data4",BYTE * 8)]
def __init__(self, guid):
from ctypes import windll, c_void_p, c_byte, pointer,c_char,POINTER
from ctypes.wintypes import HANDLE
import ctypes
self.dll = windll.LoadLibrary("D:\\Lib\\addp.dll")
self.guid = self.GUID()
self.guid.Data1 = guid[0]
self.guid.Data2 = guid[1]
self.guid.Data3 = guid[2]
self.guid.Data4 = (c_byte * 8)(guid[3],guid[4],guid[5],guid[6],guid[7],guid[8],guid[9],guid[10])
addpopen = self.dll[1]
addpopen.argtypes = [POINTER(self.GUID),]
addpopen.restype = c_void_p
#print addpopen.restype
self.handler = addpopen(pointer(self.guid))
if self.handler == None:
raise RuntimeError()
self.opened = False
else:
self.opened = True
def isOpen(self):
return self.opened
def Discover(self):
from ctypes import c_int
srch = self.dll[6]
srch.restype = c_int
print srch(self.handler,10,10)
def Close(self):
close = self.dll[3]
close.restype = None
self.opened = False
#print close(self.handler)
conn = ADDP(guid)
#print conn.handler
conn.Discover()
#conn.Close()
print conn.handler
i searched a lot for how to handle a handle returned from a c function, but couldn't find much about it, i read the ctypes docs for a while, and then inspected the header file too..
the handle is defined in the header file with
typedef void* addp_handle_t;
so i assumed i had to set 'restype' to 'c_void_p', the function always returns 'None'
its specified in the header file that it returns 'None' when an error has occurred, else it return the handle to ADDP session.
another thing, this dll does not export functions by name... i had to, more or less, guess what function is what by expected bytes in arguments.
any ideas on this?
i've found a project on google code but apparently it didn't go far...
if you need any other details, just say
I am trying to use the ctypes module to make calls to Windows' Common Item Dialog API. The code shown below is roughly based on the steps outlined in the MSDN documentation. Its only dependency is the comtypes.GUID module.
import ctypes
from ctypes import byref, POINTER, c_int, c_long
from ctypes.wintypes import HWND, HRESULT
from comtypes import GUID
CLSID_FileOpenDialog = '{DC1C5A9C-E88A-4DDE-A5A1-60F82A20AEF7}'
IID_IFileDialog = '{42F85136-DB7E-439C-85F1-E4075D135FC8}'
#IID_IFileOpenDialog = '{D57C7288-D4AD-4768-BE02-9D969532D960}'
CLSCTX_SERVER = 5
COINIT_APARTMENTTHREADED = 2
FOS_PICKFOLDERS = 32
FOS_FORCEFILESYSTEM = 64
ole32 = ctypes.windll.ole32
CoCreateInstance = ole32.CoCreateInstance
CoInitializeEx = ole32.CoInitializeEx
CoInitializeEx(None, COINIT_APARTMENTTHREADED)
ptr = c_int()
error = CoCreateInstance(
byref(GUID(CLSID_FileOpenDialog)), None, CLSCTX_SERVER,
byref(GUID(IID_IFileDialog)), byref(ptr))
assert error == 0
ptr = ptr.value
c_long_p = ctypes.POINTER(ctypes.c_int)
print('Pointer to COM object: %s' % ptr)
vtable = ctypes.cast(ptr, c_long_p).contents.value
print('Pointer to vtable: %s' % vtable)
func_proto = ctypes.WINFUNCTYPE(HRESULT, HWND)
# Calculating function pointer offset: 3rd entry in vtable; 32-bit => 4 bytes
show_p = ctypes.cast(vtable + 3*4, c_long_p).contents.value
print('Pointer to show(): %s' % show_p)
show = func_proto(show_p)
show(0)
Everything works as intended until the first call to show(0):
WindowsError: exception: access violation reading 0xXXXXXXXX
(Output may vary.) For comparison, I have carried out the same steps in AutoHotkey_L, which has direct access to COM.
CLSID := "{DC1C5A9C-E88A-4DDE-A5A1-60F82A20AEF7}"
IID := "{42F85136-DB7E-439C-85F1-E4075D135FC8}"
ptr := ComObjCreate(CLSID, IID)
vtable := NumGet(ptr + 0, 0, "Ptr")
show := NumGet(vtbl + 0, 3 * A_PtrSize, "Ptr")
MsgBox ptr: %ptr% vtable: %vtable% show: %A_PtrSize%
DllCall(show, "Ptr", ptr, "Ptr", 44)
The resulting macro pops up an Open File dialog, as expected. The vtable pointer offsets are the same in both cases, but only the Python version throws up an access violation.
Can anyone shed some light on this?
[I apologize for not adding more hyperlinks where appropriate, but as a new user I am limited to two at a time.]
Background:
I am putting together a lightweight module which provides a native save/open file dialog for use in Python scripts. So far I have been unable to find an implementation in pure Python. Those that exist rely on UI toolkits such as Tkinter or wxPython.
Here is the solution:
COM methods take an additional parameter: The 'this' pointer. It is implicit when you call the method from C++, in C (and in ctypes) you must supply it yourself.
Change the line
func_proto = ctypes.WINFUNCTYPE(HRESULT, HWND)
into
func_proto = ctypes.WINFUNCTYPE(HRESULT, c_long, HWND)
and this line
show(0)
into
show(ptr, 0)
and your code will work.