We Keep Coding

Python, ctypes, How to fix: OSError: exception: access violation writing 0x0000000000000000

I am trying to configure the connection of Python program with the trading terminal(QUIK) using this official dll: http://arqatech.com/upload/iblock/80a/Trans2QuikAPI_1.3_x64.zip .
There is no convenient connection for this dll with python, but I found a python wrapper written 10 years ago for this dll of the previous version: https://pypi.org/project/PyQUIK/ .
The previous versions of the dll were 32 bit, and the new 64 bit, the old versions did not start at all, when I tried the new version of the dll, it worked, but it constantly gives errors:
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 232, in 'calling callback function'
OSError: exception: access violation writing 0x0000000000000000
The error occurs only when the trading terminal that the dll is trying to connect to is ready to accept transactions from the dll. When the terminal does not accept transactions, no errors occur. In this case, errors occur at the stage of connecting to the terminal, and when the connection with it is stopped. The error messages say about ctypes, but I am not familiar with either this library or the C language, so I don’t know how to fix this problem.
Here is the part of the code that causes the error:
from ctypes import WinDLL, c_char, c_long, c_double, POINTER, \
WINFUNCTYPE, byref, create_string_buffer, pointer, string_at
from ctypes.wintypes import LPCSTR, LPSTR, DWORD, CHAR, PDWORD
from ctypes.util import find_library
c_long_p = POINTER(c_long)
c_double_p = POINTER(c_double)
c_void = None
try:
TRANS2QUIK = WinDLL("TRANS2QUIK.dll")
except OSError:
import os
libpath = os.path.join(os.path.dirname(__file__), "TRANS2QUIK.dll")
TRANS2QUIK = WinDLL(libpath)
pnExtendedErrorCode_data = c_long()
pnExtendedErrorCode = byref(pnExtendedErrorCode_data)
pnReplyCode_data = c_long()
pnReplyCode = byref(pnReplyCode_data)
pdOrderNum_data = c_double()
pdOrderNum = byref(pdOrderNum_data)
dwErrorMessageSize = DWORD(200)
dwResultMessageSize = DWORD(200)
lpstrErrorMessage = create_string_buffer(dwErrorMessageSize.value)
lpstrResultMessage = create_string_buffer(dwResultMessageSize.value)
def _declare_extended(func, *args):
func.restype = c_long
func.argtypes = args + (c_long_p, LPSTR, DWORD)
def __caller__(*args):
args += (
pnExtendedErrorCode,
lpstrErrorMessage,
dwErrorMessageSize
)
return func(*args)
return __caller__
def _declare_method(func, restype, *args):
func.restype = restype
func.argtypes = args
return func
# long TRANS2QUIK_CONNECT(
# LPCSTR lpcstrConnectionParamsString,
# _________________________
# long* pnExtendedErrorCode,
# LPSTR lpstrErrorMessage,
# DWORD dwErrorMessageSize
# )
CONNECT = _declare_extended(
TRANS2QUIK.TRANS2QUIK_CONNECT, LPCSTR
)
def connect(path):
lpcstrConnectionParamsString = LPCSTR(path.encode())
return CONNECT(lpcstrConnectionParamsString)
# long TRANS2QUIK_DISCONNECT(
# _________________________
# long* pnExtendedErrorCode,
# LPSTR lpstrErrorMessage,
# DWORD dwErrorMessageSize
# )
disconnect = DISCONNECT = _declare_extended(
TRANS2QUIK.TRANS2QUIK_DISCONNECT
)
connection = connect("C:\SBERBANK\QUIK")
print(connection)
disconnect = disconnect()
print(disconnect)
I use python version 3.7.6, library ctypes version 1.1.0, windows 10 64 bit
I have experience in python programming for about a year, I'm not a professional in English either, so sorry if something is wrong, this is the fault of Google translator.

Python sendto() not executing

I have a program that accepts coordinates over UDP, moves some equipment around, and then replies when the job is done.
I seem to have the same issue as this guy:
Python sendto doesn't seem to send
My code is here:
import socket
import struct
import traceback
def main():
sock = socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
sock.bind(('',15000))
reply_sock = socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
while True:
try:
data,addr = sock.recvfrom(1024)
if data is not None:
try:
coords = struct.unpack('>dd',data)
#Stuff happens here
print(f'moved probe to {coords}')
reply_sock.sendto(bytearray.fromhex('B'),('10.0.0.32',15001))
except:
traceback.print_exc()
try:
reply_sock.sendto(bytearray.fromhex('D'),('10.0.0.32',15001))
except:
traceback.print_exc()
break
except:
pass
The program behaves as though the sendto call is just passed over; it accepts the packet, executes the print statements, and loops back around (It can execute the loop multiple times but never replies). I'm looking at wireshark and no packets are ever sent outbound. No errors are ever thrown.
Any ideas why this is happening?

From the documentation:
The string must contain two hexadecimal digits per byte, with ASCII
whitespace being ignored.
So this happens:
$ python3
Python 3.6.6 (default, Sep 12 2018, 18:26:19)
[GCC 8.0.1 20180414 (experimental) [trunk revision 259383]] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> bytearray.fromhex('B')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: non-hexadecimal number found in fromhex() arg at position 1
>>>
Try this:
reply_sock.sendto(bytearray.fromhex('0B'),('10.0.0.32',15001))
if that's what you mean.
Note that your except is catching all the exceptions, not just the ones you're expecting, so you're not seeing the error you're causing. Consider using something like except OSError here instead.
Also, think about reducing the amount of code in your try sections:
coords = struct.unpack('>dd',data)
#Stuff happens here
print(f'moved probe to {coords}')
bytes_to_send = bytearray.fromhex('0B')
try:
reply_sock.sendto(bytes_to_send,('10.0.0.32',15001))
except IOError as e1:
print(e1)
traceback.print_exc()
bytes_to_send = bytearray.fromhex('0D')
try:
reply_sock.sendto(bytes_to_send,('10.0.0.32',15001))
except IOError as e2:
print(e2)
traceback.print_exc()
break
This way you're protecting only the code which you want to.

Python os.environ throws key error?

I'm accessing an environment variable in a script with os.environ.get and it's throwing a KeyError. It doesn't throw the error from the Python prompt. This is running on OS X 10.11.6, and is Python 2.7.10.
What is going on?
$ python score.py
Traceback (most recent call last):
File "score.py", line 4, in <module>
setup_logging()
File "/score/log.py", line 29, in setup_logging
config = get_config()
File "/score/log.py", line 11, in get_config
environment = os.environ.get('NODE_ENV')
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/UserDict.py", line 23, in __getitem__
raise KeyError(key)
KeyError: 'NODE_ENV'
$ python -c "import os; os.environ.get('NODE_ENV')"
$
As requested, here's the source code for score.py
from __future__ import print_function
from log import get_logger, setup_logging
setup_logging()
log = get_logger('score')
And here's log.py
import json
import os
import sys
from iron_worker import IronWorker
from logbook import Logger, Processor, NestedSetup, StderrHandler, SyslogHandler
IRON_IO_TASK_ID = IronWorker.task_id()
def get_config():
environment = os.environ.get('NODE_ENV')
if environment == 'production':
filename = '../config/config-production.json'
elif environment == 'integration':
filename = '../config/config-integration.json'
else:
filename = '../config/config-dev.json'
with open(filename) as f:
return json.load(f)
def setup_logging():
# This defines a remote Syslog handler
# This will include the TASK ID, if defined
app_name = 'scoreworker'
if IRON_IO_TASK_ID:
app_name += '-' + IRON_IO_TASK_ID
config = get_config()
default_log_handler = NestedSetup([
StderrHandler(),
SyslogHandler(
app_name,
address = (config['host'], config['port']),
level = 'ERROR',
bubble = True
)
])
default_log_handler.push_application()
def get_logger(name):
return Logger(name)

Try running:
find . -name \*.pyc -delete
To delete your .pyc files.
Researching your problem I came across this question, where a user was experiencing the same thing: .get() seemingly raising a KeyError. In that case, it was caused, according to this accepted answer, by a .pyc file which contained code where a dict value was being accessed by key (i.e., mydict['potentially_nonexistent_key']), while the traceback was showing the code from the updated .py file where .get() was used. I have never heard of this happening, where the traceback references current code from a .py file, but shows an error raised by an outdated .pyc file, but it seems to have happened at least once in the history of Python...
It is a long shot, but worth a try I thought.

I encountered a similar error when I set the environment variable without exporting it. So if you do this:
me#host:/# NODE_ENV=foo
You will get this:
me#host:/# python3
Python 3.8.2 (default, Apr 27 2020, 15:53:34)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> node_env = os.environ['NODE_ENV']
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.8/os.py", line 675, in __getitem__
raise KeyError(key) from None
KeyError: 'NODE_ENV'
>>>
But if you do this:
me#host:/# NODE_ENV=foo
me#host:/# export NODE_ENV
It works:
me#host:/# python3
Python 3.8.2 (default, Apr 27 2020, 15:53:34)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> node_env = os.environ['NODE_ENV']
>>> print(node_env)
foo
>>>

Command for windows to delete the .pyc files:
del /S *.pyc

I had the same problem. I solved that by making some corrections on the .env file:
Before:
Key = Value
After my correction:
Key=Value
without blank spaces and worked!

I was getting this error while trying to source from a .env file.
I didn't explicitly export the env vars so I had to change this.
ENVIRONMENT=DEV
to this
export ENVIRONMENT=DEV

Use export a=10 instead of a=10 while setting env variable. Add the same in ~./bashrc to reload the env var wherever you login.
Doing this resolved the issue

I'd recommend you start debugging os.py, for instance, on windows it's being used this implementation:
def get(self, key, failobj=None):
print self.data.__class__
print key
return self.data.get(key.upper(), failobj)
And if I test it with this:
import os
try:
os.environ.get('NODE_ENV')
except Exception as e:
print("-->{0}".format(e.__class__))
os.environ['NODE_ENV'] = "foobar"
try:
os.environ.get('NODE_ENV')
except Exception as e:
print("{0}".format(e.__class__))
The output will be:
<type 'dict'>
PYTHONUSERBASE
<type 'dict'>
APPDATA
<type 'dict'>
NODE_ENV
<type 'dict'>
NODE_ENV
So it makes sense the exception is not spawned reading dict.get docs.
In any case, if you don't want to mess up or debugging the python modules, try cleaning up the *.pyc files, try to set up properly NODE_ENV. And if all that don't work, restart your terminal to clear up.

How to save a configuration file / python file IO

I have this python code for opening a .cfg file, writing to it and saving it:
import ConfigParser
def get_lock_file():
cf = ConfigParser.ConfigParser()
cf.read("svn.lock")
return cf
def save_lock_file(configurationParser):
cf = configurationParser
config_file = open('svn.lock', 'w')
cf.write(config_file)
config_file.close()
Does this seem normal or am I missing something about how to open-write-save files? Is there a more standard way to read and write config files?
I ask because I have two methods that seem to do the same thing, they get the config file handle ('cf') call cf.set('blah', 'foo' bar) then use the save_lock_file(cf) call above. For one method it works and for the other method the write never takes place, unsure why at this point.
def used_like_this():
cf = get_lock_file()
cf.set('some_prop_section', 'some_prop', 'some_value')
save_lock_file(cf)

Just to note that configuration file handling is simpler with ConfigObj.
To read and then write a config file:
from configobj import ConfigObj
config = ConfigObj(filename)
value = config['entry']
config['entry'] = newvalue
config.write()

Looks good to me.
If both places call get_lock_file, then cf.set(...), and then save_lock_file, and no exceptions are raised, this should work.
If you have different threads or processes accessing the same file you could have a race condition:
thread/process A reads the file
thread/process B reads the file
thread/process A updates the file
thread/process B updates the file
Now the file only contains B's updates, not A's.
Also, for safe file writing, don't forget the with statement (Python 2.5 and up), it'll save you a try/finally (which you should be using if you're not using with). From ConfigParser's docs:
with open('example.cfg', 'wb') as configfile:
config.write(configfile)

Works for me.
C:\temp>type svn.lock
[some_prop_section]
Hello=World
C:\temp>python
ActivePython 2.6.2.2 (ActiveState Software Inc.) based on
Python 2.6.2 (r262:71600, Apr 21 2009, 15:05:37) [MSC v.1500 32 bit (Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import ConfigParser
>>> def get_lock_file():
... cf = ConfigParser.ConfigParser()
... cf.read("svn.lock")
... return cf
...
>>> def save_lock_file(configurationParser):
... cf = configurationParser
... config_file = open('svn.lock', 'w')
... cf.write(config_file)
... config_file.close()
...
>>> def used_like_this():
... cf = get_lock_file()
... cf.set('some_prop_section', 'some_prop', 'some_value')
... save_lock_file(cf)
...
>>> used_like_this()
>>> ^Z
C:\temp>type svn.lock
[some_prop_section]
hello = World
some_prop = some_value
C:\temp>

Determine if an executable (or library) is 32 -or 64-bits (on Windows)

I am trying to find out if a given executable (or library) is compiled for 32-bits or 64-bits from Python. I am running Vista 64-bits and would like to determine if a certain application in a directory is compiled for 32-bits or 64-bits.
Is there a simple way to do this using only the standard Python libraries (currently using 2.5.4)?

The Windows API for this is GetBinaryType. You can call this from Python using pywin32:
import win32file
type=GetBinaryType("myfile.exe")
if type==win32file.SCS_32BIT_BINARY:
print "32 bit"
# And so on
If you want to do this without pywin32, you'll have to read the PE header yourself. Here's an example in C#, and here's a quick port to Python:
import struct
IMAGE_FILE_MACHINE_I386=332
IMAGE_FILE_MACHINE_IA64=512
IMAGE_FILE_MACHINE_AMD64=34404
f=open("c:\windows\explorer.exe", "rb")
s=f.read(2)
if s!="MZ":
print "Not an EXE file"
else:
f.seek(60)
s=f.read(4)
header_offset=struct.unpack("<L", s)[0]
f.seek(header_offset+4)
s=f.read(2)
machine=struct.unpack("<H", s)[0]
if machine==IMAGE_FILE_MACHINE_I386:
print "IA-32 (32-bit x86)"
elif machine==IMAGE_FILE_MACHINE_IA64:
print "IA-64 (Itanium)"
elif machine==IMAGE_FILE_MACHINE_AMD64:
print "AMD64 (64-bit x86)"
else:
print "Unknown architecture"
f.close()

If you're running Python 2.5 or later on Windows, you could also use the Windows API without pywin32 by using ctypes.
from ctypes import windll, POINTER
from ctypes.wintypes import LPWSTR, DWORD, BOOL
SCS_32BIT_BINARY = 0 # A 32-bit Windows-based application
SCS_64BIT_BINARY = 6 # A 64-bit Windows-based application
SCS_DOS_BINARY = 1 # An MS-DOS-based application
SCS_OS216_BINARY = 5 # A 16-bit OS/2-based application
SCS_PIF_BINARY = 3 # A PIF file that executes an MS-DOS-based application
SCS_POSIX_BINARY = 4 # A POSIX-based application
SCS_WOW_BINARY = 2 # A 16-bit Windows-based application
_GetBinaryType = windll.kernel32.GetBinaryTypeW
_GetBinaryType.argtypes = (LPWSTR, POINTER(DWORD))
_GetBinaryType.restype = BOOL
def GetBinaryType(filepath):
res = DWORD()
handle_nonzero_success(_GetBinaryType(filepath, res))
return res
Then use GetBinaryType just like you would with win32file.GetBinaryType.
Note, you would have to implement handle_nonzero_success, which basically throws an exception if the return value is 0.

I've edited Martin B's answer to work with Python 3, added with statements and ARM/ARM64 support:
import struct
IMAGE_FILE_MACHINE_I386 = 332
IMAGE_FILE_MACHINE_IA64 = 512
IMAGE_FILE_MACHINE_AMD64 = 34404
IMAGE_FILE_MACHINE_ARM = 452
IMAGE_FILE_MACHINE_AARCH64 = 43620
with open('foo.exe', 'rb') as f:
s = f.read(2)
if s != b'MZ':
print('Not an EXE file')
else:
f.seek(60)
s = f.read(4)
header_offset = struct.unpack('<L', s)[0]
f.seek(header_offset + 4)
s = f.read(2)
machine = struct.unpack('<H', s)[0]
if machine == IMAGE_FILE_MACHINE_I386:
print('IA-32 (32-bit x86)')
elif machine == IMAGE_FILE_MACHINE_IA64:
print('IA-64 (Itanium)')
elif machine == IMAGE_FILE_MACHINE_AMD64:
print('AMD64 (64-bit x86)')
elif machine == IMAGE_FILE_MACHINE_ARM:
print('ARM eabi (32-bit)')
elif machine == IMAGE_FILE_MACHINE_AARCH64:
print('AArch64 (ARM-64, 64-bit)')
else:
print(f'Unknown architecture {machine}')

I was able to use Martin B's answer successfully in a Python 3.5 program after making this adjustment:
s=f.read(2).decode(encoding="utf-8", errors="strict")
Originally it worked just fine with my program in Python 2.7, but after making other necessary changes, I discovered I was getting b'MZ', and decoding it appears to fix this.

Using Python 3.7, 32 bit on 64 bit Win 7, the first code fragment in the top answer doesn't run for me. It fails because GetBinaryType is an unknown symbol. Solution is to use win32file.GetBinaryType.
Also running it on a .pyd file doesn't work, even if it is renamed to a .dll. See next:
import shutil
import win32file
from pathlib import Path
myDir = Path("C:\\Users\\rdboylan\\AppData\\Roaming\\Python\\Python37\\site-packages\\pythonwin")
for fn in ("Pythonwin.exe", "win32ui.pyd"):
print(fn, end=": ")
myf = myDir / fn
if myf.suffix == ".pyd":
mytemp = myf.with_suffix(".dll")
if mytemp.exists():
raise "Can not create temporary dll since {} exists".format(mytemp)
shutil.copyfile(myf, mytemp)
type = win32file.GetBinaryType(str(mytemp))
mytemp.unlink()
else:
type=win32file.GetBinaryType(str(myf))
if type==win32file.SCS_32BIT_BINARY:
print("32 bit")
else:
print("Something else")
# And so on
Results in
Pythonwin.exe: 32 bit
win32ui.pyd: Traceback (most recent call last):
File "C:/Users/rdboylan/Documents/Wk devel/bitness.py", line 14, in <module>
type = win32file.GetBinaryType(str(mytemp))
pywintypes.error: (193, 'GetBinaryType', '%1 is not a valid Win32 application.')