I'm using Python 3.7 SDK of apache beam 2.17.0 for dataflow. Code is running locally, but I gather data from pubsub. I try to combine per key and everything goes fine until the pipeline calls the "merge_accumulators" function. From this point on, all the underlying code is executed twice.
After debugging and going deep in the source code, I found the task is not properly finalized and that is why it is executed twice.
This is the pipeline code:
options = {
"runner": "DirectRunner",
"streaming": True,
"save_main_session": True
}
p = beam.Pipeline(options = PipelineOptions(flags=[], **options))
processRows = (p
|'Read from topic' >> beam.io.ReadFromPubSub(subscription=get_subscription_address())
|'Filter do not track' >> beam.ParDo(TakeOutNoTrack)
|'Map Data' >> beam.ParDo(mapData)
|'Filter metatags' >> beam.ParDo(filterMetatags)
|'Label admin' >> beam.ParDo(labelAdmin)
|'Process row' >> beam.ParDo(processRow)
)
sessionRow = (processRows
|'Add timestamp' >> beam.Map(lambda x: window.TimestampedValue(x, x['timestamp']))
|'Key on uuid' >> beam.Map(lambda x: (x['capture_uuid'], x))
|'User session window' >> beam.WindowInto(window.Sessions(config_triggers['session_gap']),
trigger=trigger.AfterWatermark(
early=trigger.AfterCount(config_triggers['after_count'])),
accumulation_mode=trigger.AccumulationMode.ACCUMULATING)
|'CombineValues' >> beam.CombinePerKey(JoinSessions())
)
printing = (sessionRow
|'Printing' >> beam.Map(lambda x: print(x))
)
print('running pipeline')
p.run().wait_until_finish()
print('done running the pipeline')
return
This is the config_triggers:
config_triggers = {
"session_gap": 1320,
"after_count": 1,
"session_length": 20
}
This is the combine class:
class JoinSessions(beam.CombineFn):
def define_format(self):
try:
data = {
"session_uuid": [],
"capture_uuid": "",
"metatags": [],
"timestamps": [],
"admin": []
}
return data
except Exception:
logging.error("error at define data: \n%s" % traceback.format_exc())
def create_accumulator(self):
try:
return self.define_format()
except Exception:
logging.error("error at create accumulator: \n%s " % traceback.format_exc())
def add_input(self, metatags, input):
try:
metatags["session_uuid"].append(input.get('session_uuid'))
metatags["capture_uuid"] = input.get('capture_uuid')
metatags["metatags"].append(input.get('metatags'))
metatags["timestamps"].append(input.get('timestamp'))
metatags["admin"].append(input.get('admin'))
print('test add_input')
return metatags
except Exception:
logging.error("error at add input: \n%s" % traceback.format_exc())
def merge_accumulators(self, accumulators):
# print(accumulators)
try:
global test_counter
tags_accumulated = self.define_format()
for tags in accumulators:
tags_accumulated["session_uuid"] += tags['session_uuid']
tags_accumulated["capture_uuid"] += tags['capture_uuid']
tags_accumulated["metatags"] += tags['metatags']
tags_accumulated["timestamps"] += tags['timestamps']
tags_accumulated["admin"] += tags['admin']
test_counter += 1
print('counter = ', test_counter)
return tags_accumulated
except Exception:
logging.error("Error at merge Accumulators: \n%s" % traceback.format_exc())
def extract_output(self, metatags):
try:
# print('New input in the pipeline:')
# print('Extract_output: ')
# print(metatags, '\n')
return metatags
except Exception:
logging.error("error at return input: \n%s" % traceback.format_exc())
No errors are thrown nor exceptions or some kind of information. Just the output of the 'printing' label is printed twice. Also the global counter goes up two times, but there is just one data entry in the pipeline.
The print on the add_input function is executed just once.
I'm new to dataflow, so, sorry if I made a dumb mistake.
I think this is due to the trigger you have set.
trigger=trigger.AfterWatermark(early=trigger.AfterCount(config_triggers['after_count'])),accumulation_mode=trigger.AccumulationMode.ACCUMULATING)
config_triggers['after_count] is 1.
So you've set up a trigger that fires after every 1 element and you also accumulate elements produced by trigger firings. So a second trigger within the same window will include elements from the first trigger and so on. See following for details regarding setting the trigger of your pipeline correctly according to your use-case.
https://beam.apache.org/documentation/programming-guide/#triggers
Related
I simulated a batch of 5 FMUs by putting a function containing the fmpy simulate in a loop.
4 of them simulate correctly. The fifth one due to a wrong parameter shows an error.
My question is, is there any way to find out in which FMU the error comes during simulation
Ideally the output should look like:
FMU1: OK
FMU2: OK
FMI3: 'Error' and so on
def wf(i):
result = simulate_fmu(
fmupath[i],
validate=False,
start_time=0,
stop_time=endtime,
solver='CVode',
output_interval=stepsize,
record_events=False,
start_values = parameters[i],
output = resultvariables,
)
dfindres = pd.DataFrame.from_dict(result)
return dfindres
results = [wf(i) for i in range(5)]
You can catch the exception and return the status message together with the result like so
def wf(i):
try:
result = simulate_fmu(
fmupath[i],
validate=False,
start_time=0,
stop_time=endtime,
solver='CVode',
output_interval=stepsize,
record_events=False,
start_values=parameters[i],
output=resultvariables,
)
except Exception as e:
return None, "FMU%d: Error" % i
dfindres = pd.DataFrame.from_dict(result)
return dfindres, "FMU%d: OK" % i
# list of tuples (result, status)
results = [wf(i) for i in range(5)]
I have the Below python code which I'm using to determine the Linux bond/team status. This code works just fine. I am not good at aligning the output formatting thus getting little hiccup.
I wanted the Printing Format into a Certain format, would appreciate any help on the same.
Below is the code exercise:
#!/usr/bin/python
# Using below file to process the data
# cat /proc/net/bonding/bond0
import sys
import re
def usage():
print '''USAGE: %s [options] [bond_interface]
Options:
--help, -h This usage document
Arguments:
bond_interface The bonding interface to query, eg. 'bond0'. Default is 'bond0'.
''' % (sys.argv[0])
sys.exit(1)
# Parse arguments
try:
iface = sys.argv[1]
if iface in ('--help', '-h'):
usage()
except IndexError:
iface = 'bond0'
# Grab the inf0z from /proc
try:
bond = open('/proc/net/bonding/%s' % iface).read()
except IOError:
print "ERROR: Invalid interface %s\n" % iface
usage()
# Parse and output
active = 'NONE'
Link = 'NONE'
slaves = ''
state = 'OK'
links = ''
bond_status = ''
for line in bond.splitlines():
m = re.match('^Currently Active Slave: (.*)', line)
if m:
active = m.groups()[0]
m = re.match('^Slave Interface: (.*)', line)
if m:
s = m.groups()[0]
slaves += ', %s' % s
m = re.match('^Link Failure Count: (.*)', line)
if m:
l = m.groups()[0]
links += ', %s' % l
m = re.match('^MII Status: (.*)', line)
if m:
s = m.groups()[0]
if slaves == '':
bond_status = s
else:
slaves += ' %s' % s
if s != 'up':
state = 'FAULT'
print "%s %s (%s) %s %s %s" % (iface, state, bond_status, active, slaves, links)
Result:
$ ./bondCheck.py
bond0 OK (up) ens3f0 , ens3f0 up, ens3f1 up , 0, 0
Expected:
bond0: OK (up), Active Slave: ens3f0 , PriSlave: ens3f0(up), SecSlave: ens3f1(up) , LinkFailCountOnPriInt: 0, LinkFailCountOnSecInt: 0
I tried to format in a very basic way as shown below :
print "%s: %s (%s), Active Slave: %s, PriSlave: %s (%s), SecSlave: %s (%s), LinkFailCountOnPriInt: %s, LinkFailCountOnSecInt: %s" % (iface, state, bond_status, active, slaves.split(',')[1].split()[0], slaves.split(',')[1].split()[1], slaves.split(',')[2].split()[0], slaves.split(',')[2].split()[1], links.split(',')[1], links.split(',')[2])
RESULT:
bond0: OK (up), Active Slave: ens3f0, PriSlave: ens3f0 (up), SecSlave: ens3f1 (up), LinkFailCountOnPriInt: 1, LinkFailCountOnSecInt: 1
However, I would suggest to get the values into variables prior and then use them in the print statement so as to avoid "out of index" issues during print() , as in rare cases like bond with only one interface will report indexing error while splitting hence good to get the values in variable and suppress the out of index into exception for those cases.
Do not use the way with "/proc/net/bonding/%s' for querying bond status. It could trigger system panic. Try to use "/sys/class/net/bondX/bonding", it is more safe.
I am using python SDK to submit jobs to Azure using adlaJobClient , I have around 30 dynamic USQLs constructed using JINJA2, which I am populating in a list and then pushing them off to Azure using adlaJobClient one by one, The problem which I am facing is after a random number of successful deployments python throws an exception in the program console without any further explanation.there is no instance of U-SQL job failure in Azure either , below mentioned is error stack trace in python , when I run the same U-SQL query,which I am generating dynamically.. for which the execution is stopping it runs without fail in Azure (manually)
***** starting query number **** 24
Job is not yet done, waiting for 3 seconds. Current state: Compiling
Job is not yet done, waiting for 3 seconds. Current state: Compiling
Job is not yet done, waiting for 3 seconds. Current state: Compiling
Job is not yet done, waiting for 3 seconds. Current state: starting
Job is not yet done, waiting for 3 seconds. Current state: starting
An exception has occurred, use %tb to see the full traceback.
SystemExit: 1
for script in sql_query_list:
jobId = str(uuid.uuid4())
jobResult = adlaJobClient.job.create(adla,jobId,JobInformation(name='Submit ADLA Job '+jobId,type='USql',properties=USqlJobProperties(script=script)))
try:
while(jobResult.state != JobState.ended):
print('Job is not yet done, waiting for 3 seconds. Current state: ' + jobResult.state.value)
time.sleep(3)
jobResult = adlaJobClient.job.get(adla, jobId)
print(' ******* JOB ID ********',jobId)
print("****QUERY no FINISHED *****",sql_query_list.index(script))
print ('**** JOB ID RESULT: ****** ' + jobResult.result.value)
except Exception as e:
raise ValueError
print ("xxxxxx JOB SUBMISSION TO ADLA FAILED xxxxxxx")
print(e)
Option A: Manually log into Portal
The easiest way to check on a failed job is to log into your Azure Portal (http://portal.azure.com), navigate to the Data Lake Analytics account, and click "View all jobs". From the list of jobs you can navigate to your job and view the output with specific error messages. (Keep reading for the automated method.)
Option B: Automated Python Job
You can get the job properties with error messages using job_result.properties.errors. The below code sample will perform a "pretty print" of any U-SQL job error, raising an exception with those details.
Parsing the error info:
def get_pretty_error(error_message_obj):
"""
Returns a string describing the USQL error.
error_message_obj can be obtained via `job_result.error_message[0]`
"""
err_info = error_message_obj.__dict__
error_msgs = "=" * 80 + "\n"
error_msgs += "=" * 6 + " ERROR: {}".format(err_info.pop("description", None)) + "\n"
error_msgs += "=" * 80 + "\n"
error_msgs += err_info.pop("details", None).replace("\\r\\n", "\n").replace("\\n", "\n").replace("\\t", "\t").rstrip() + "...\n"
error_msgs += "=" * 80 + "\n"
error_msgs += "Message: {}\n".format(err_info.pop("message", None))
error_msgs += "Severity: {}\n".format(str(err_info.pop("severity", None)).upper())
error_msgs += "Resolution: {}\n".format(err_info.pop("resolution", None))
inner = err_info.pop("inner_error", None)
for key in ["end_offset", "line_number", "start_offset", "source", "additional_properties"]:
# ignore (don't print these)
err_info.pop(key, None)
err_info = {x: y for x, y in err_info.items() if y} # Remove empty keys
error_msgs += "Addl. Info:\n\t{}\n".format(
yaml.dump(err_info,
default_flow_style=True
).replace("\\t", "\t").replace("\\n", "\n").replace("\n", "\n\t"))
if inner:
# If there's an inner error, concatenate that message as well recursively
error_msgs += _get_pretty_error(inner, ordinal_text + " (B)")
return error_msgs
Making use of this in a wait-for-job function:
def wait_for_usql_job(adlaJobClient, adla_account, job_id):
"""Wait for completion, on error raise an exception (with specific details)"""
print("Waiting for job ID '{}'".format(job_id))
job_result = adlaJobClient.job.get(adla_account, job_id)
while(job_result.state != JobState.ended):
print('Job is not yet done, waiting for 3 seconds. Current state: ' + job_result.state.value)
time.sleep(3)
job_result = adlaJobClient.job.get(adla_account, job_id)
job_properties = job_result.properties.__dict__
detail_msg = (
"\tCompilation Time: {}\n".format(job_properties.pop("total_running_time", None)) +
"\tQueued Time: {}\n".format(job_properties.pop("total_compilation_time", None)) +
"\tExecution Time: {}\n".format(job_properties.pop("total_queued_time", None)))
print('Job completed with result: {}\n{}'
.format(job_result.result.value, detail_msg))
if job_result.result.value == "Succeeded":
return job_result.result.value
elif job_result.result.value == "Failed":
error_msgs = ""
for error in job_result.error_message:
# Loop through errors and concatenate error messages
error_msgs += get_pretty_error(error)
raise Exception("Job execution failed for job_id '{}':\n{}"
.format(job_id, error_msgs))
Notes:
Because these classes are not well documented online, I used the __dict__ property to explore all properties of the JobProperties and ErrorInfo objects, disposing of any the properties that are blank or which I don't need and printing the rest.
You can optionally rewrite this code to call those properties explicitly, without using __dict__.
Any error can have an inner error, which is why I wrote get_pretty_error() as it's own function - so that it can then call itself recursively.
I'm looking to do something in this example: Python - How to get the start/base address of a process?. I'm having the same issue as the person in that topic, in that the pointers cheat engine provides is in reference to the base address of the process itself.
I've looked around and it looks like the best solution is to use ctypes and the MODULEENTRY32 to store snapshots of processes and analyze their modBaseAddr.
Here is my current code
import os.path, ctypes, ctypes.wintypes
from ctypes import *
from ctypes.wintypes import *
PROCESS_QUERY_INFORMATION = (0x0400)
PROCESS_VM_OPERATION = (0x0008)
PROCESS_VM_READ = (0x0010)
PROCESS_VM_WRITE = (0x0020)
TH32CS_SNAPMODULE = (0x00000008)
CreateToolhelp32Snapshot= ctypes.windll.kernel32.CreateToolhelp32Snapshot
Process32First = ctypes.windll.kernel32.Process32First
Process32Next = ctypes.windll.kernel32.Process32Next
Module32First = ctypes.windll.kernel32.Module32First
Module32Next = ctypes.windll.kernel32.Module32Next
GetLastError = ctypes.windll.kernel32.GetLastError
OpenProcess = ctypes.windll.kernel32.OpenProcess
GetPriorityClass = ctypes.windll.kernel32.GetPriorityClass
CloseHandle = ctypes.windll.kernel32.CloseHandle
class MODULEENTRY32(Structure):
_fields_ = [ ( 'dwSize' , DWORD ) ,
( 'th32ModuleID' , DWORD ),
( 'th32ProcessID' , DWORD ),
( 'GlblcntUsage' , DWORD ),
( 'ProccntUsage' , DWORD ) ,
( 'modBaseAddr' , POINTER(BYTE)) ,
( 'modBaseSize' , DWORD ) ,
( 'hModule' , HMODULE ) ,
( 'szModule' , c_char * 256 ),
( 'szExePath' , c_char * 260 ) ]
def GetBaseAddr(ProcId, ProcName):
me32 = MODULEENTRY32()
me32.dwSize = sizeof(me32)
hSnapshot = CreateToolhelp32Snapshot( TH32CS_SNAPMODULE, ProcId)
if GetLastError() != 0:
CloseHandle(hSnapshot)
print 'Handle Error %s' % WinError()
return 'Error'
else:
if Module32First(hSnapshot, byref(me32)):
if me32.szModule == ProcName:
CloseHandle(hSnapshot)
return id(me32.modBaseAddr)
else:
Module32Next(hSnapshot, byref(me32))
while int(GetLastError())!= 18:
if me32.szModule == ProcName:
CloseHandle(hSnapshot)
return id(me32.modBaseAddr)
else:
Module32Next(hSnapshot, byref(me32))
CloseHandle(hSnapshot)
print 'Couldn\'t find Process with name %s' % ProcName
else:
print 'Module32First is False %s' % WinError()
CloseHandle(hSnapshot)
def GetProcessIdByName( pName):
if pName.endswith('.exe'):
pass
else:
pName = pName+'.exe'
ProcessIds, BytesReturned = EnumProcesses()
for index in range(BytesReturned / ctypes.sizeof(ctypes.wintypes.DWORD)):
ProcessId = ProcessIds[index]
hProcess = ctypes.windll.kernel32.OpenProcess(PROCESS_QUERY_INFORMATION, False, ProcessId)
if hProcess:
ImageFileName = (ctypes.c_char*MAX_PATH)()
if ctypes.windll.psapi.GetProcessImageFileNameA(hProcess, ImageFileName, MAX_PATH)>0:
filename = os.path.basename(ImageFileName.value)
if filename == pName:
return ProcessId
CloseHandle(hProcess)
def EnumProcesses():
count = 32
while True:
ProcessIds = (ctypes.wintypes.DWORD*count)()
cb = ctypes.sizeof(ProcessIds)
BytesReturned = ctypes.wintypes.DWORD()
if ctypes.windll.Psapi.EnumProcesses(ctypes.byref(ProcessIds), cb, ctypes.byref(BytesReturned)):
if BytesReturned.value<cb:
return ProcessIds, BytesReturned.value
break
else:
count *= 2
else:
return None
if __name__ == '__main__':
ProcId = GetProcessIdByName('RocketLeague.exe')
#print ProcId
print hex(GetBaseAddr(ProcId, 'RocketLeague.exe'))
#print hex(GetBaseAddr(8252,'RocketLeague.exe'))
Now my understanding of memory isn't the greatest, but I'd figure that the base address should be static while a program is running. When I do get this code to run, the ModBaseAddr I get back changes every time I run it. Another weird Issue I'm having is that without that print ProcId statement, running the program returns an ERROR_ACCESS_DENIED (error 5) from line 41 (This has something to do with the CreateToolhelp32Snapshot function I assume as I have admin rights on the computer). With the print statement, however, the program runs through giving me a different ModBaseAddr every time. If I feed the GetBaseAddr function the ProcessId manually it also works without the print statement, again however, it's giving me a random address every time.
If anyone could provide me any help or point me in the right direction I'd really appreciate it!
Clarification: MODULEENTRY32 stores information about modules, not processes. when you call CreateToolhelp32Snapshot using TH32CS_SNAPMODULE you are getting modules loaded by the process, not processes themselves.
Instead of getting the MODULEENTRY32 in combination with EnumProcesses you can instead use CreateToolHelp32Snapshot with TH32CS_SNAPPROCESS to get a list of processes in the form of PROCESSENRTY32 structs, which also contains the process identifier.
Despite being a user with administrator privileges, you must also run the process as an administrator.
You should also ensure you're initializing your MODULEENTRY32 to {0} for proper error handling and not running into an issue of the returned value being subject to undefined behavior of uninitialized memory.
I do not know the specific cause of your issue but I have used a source code for this purpose that is very robust that may be a plug and play alternative to what you're currently using, the important snippet will follow, but the full source is available here.
def ListProcessModules( ProcessID ):
hModuleSnap = c_void_p(0)
me32 = MODULEENTRY32()
me32.dwSize = sizeof( MODULEENTRY32 )
hModuleSnap = CreateToolhelp32Snapshot( TH32CS_SNAPMODULE, ProcessID )
ret = Module32First( hModuleSnap, pointer(me32) )
if ret == 0 :
print 'ListProcessModules() Error on Module32First[%d]' % GetLastError()
CloseHandle( hModuleSnap )
return False
while ret :
print " MODULE NAME: %s"% me32.szModule
print " executable = %s"% me32.szExePath
print " process ID = 0x%08X"% me32.th32ProcessID
print " ref count (g) = 0x%04X"% me32.GlblcntUsage
print " ref count (p) = 0x%04X"% me32.ProccntUsage
print " base address = 0x%08X"% me32.modBaseAddr
print " base size = %d"% me32.modBaseSize
ret = Module32Next( hModuleSnap , pointer(me32) )
CloseHandle( hModuleSnap )
return True
This block of code is loading a cryptoki.so library and retrieving slot info. This is getting a list of objects in slot num. 0. I don't need to access all the keys to perform a few functions, just a specific key pair. Is there a way to get a single desired token by using the label name, object ID, or handle?
pkcs11 = PyKCS11.PyKCS11Lib()
pkcs11.load(lib)
pkcs11.initialize()
info = pkcs11.getInfo()
i = pkcs11.getSlotInfo(0)
pkcs11.openSession(0)
print "Library manufacturerID: " + info.manufacturerID
slots = pkcs11.getSlotList()
print "Available Slots:", len(slots)
for s in slots:
try:
i = pkcs11.getSlotInfo(s)
print "Slot no:", s
print format_normal % ("slotDescription", i.slotDescription.strip())
print format_normal % ("manufacturerID", i.manufacturerID.strip())
t = pkcs11.getTokenInfo(s)
print "TokenInfo"
print format_normal % ("label", t.label.strip())
print format_normal % ("manufacturerID", t.manufacturerID.strip())
print format_normal % ("model", t.model.strip())
session = pkcs11.openSession(s)
print "Opened session 0x%08X" % session.session.value()
if pin_available:
try:
session.login(pin=pin)
except:
print "login failed, exception:", str(sys.exc_info()[1])
objects = session.findObjects()
print
print "Found %d objects: %s" % (len(objects), [x.value() for x in objects])
The specific script i'm running only has a few commands defined, such as -pin --sign --decrypt --lib do i need to define a common pkcs11-tool such as --init-token or --token-label to pass it as an argument when executing my script ? Or can i directly assign a variable to the desired LabelName within the python script?
So from command line i'm running
$./Test.py --pin=pass and getting the following
Library manufacturerID: Safenet, Inc.
Available Slots: 4
Slot no: 0
slotDescription: ProtectServer K5E:00045
manufacturerID: SafeNet Inc.
TokenInfo
label: CKM
manufacturerID: SafeNet Inc.
model: K5E:PL25
Opened session 0x00000002
Found 52 objects: [5021, 5022, 5014, 5016, 4, 5, 6, 7, 8, 9, 16, 18, 23, 24, 26, 27, 29, 30, 32, 33, 35, 36, 38, 39, 5313, 5314, 4982, 5325, 5326, 5328, 5329, 5331, 5332, 5335, 5018, 4962, 5020, 4963, 5357, 5358, 5360, 5361, 5363, 5364, 5366, 5367, 5369, 5370, 5372, 5373, 5375, 5376]
I'm ultimately trying to get only one of these objects to run some tests.For instance objectID = 201603040001 contains a private/cert file. I want to specify this particular handle. The actual label is something like 000103...3A0. How can I define this so I don't get the rest of the objects inside the library.
Here's the a list of just a couple of the HSM objects
HANDLE LABEL TYPE OBJECT-ID
5314 00000103000003A1 X509PublicKeyCertificate 201603040001
5313 00000103000003A1 RSAPrivateKey 201603040001
I'm trying to pull just one of the Labels.
Here's the defined usage
def usage():
print "Usage:", sys.argv[0],
print "[-p pin][--pin=pin]",
print "[-s slot][--slot=slot]",
print "[-c lib][--lib=lib]",
print "[-h][--help]",
print "[-o][--opensession]"
try:
opts, args = getopt.getopt(sys.argv[1:], "p:c:Sd:h:s", ["pin=", "lib=", "sign", "decrypt", "help","slot="])
except getopt.GetoptError:
# print help information and exit:
usage()
sys.exit(2)
I'm not aware of how I can add an argument so I can use --sign with just a specific label. End game I want to use $./Test.py --pin=pass --sign --label "00000103000003A4" or by handle $./Test.py --pin=pass --sign --handle=5313
UPDATED from suggested comments below. still having trouble getting attributes for rsa private key and cert. Using the specific token has worked but those objects inside of it are returning bad attribute types
t = pkcs11.getTokenInfo(s)
print "TokenInfo"
if 'CKM' == t.label.decode('ascii').strip():
tokenInfo = pkcs11.getTokenInfo(slot)
if '00000103000003A1' == tokenInfo.label.decode('ascii').strip():
print format_normal % ("label", t.label.strip())
print format_normal % ("manufacturerID", t.manufacturerID.strip())
print format_normal % ("model", t.model.strip())
session = pkcs11.openSession(s)
print("Opened session 0x%08X" % session.session.value())
if pin_available:
try:
if (pin is None) and \
(PyKCS11.CKF_PROTECTED_AUTHENTICATION_PATH & t.flags):
print("\nEnter your PIN for %s on the pinpad" % t.label.strip())
session.login(pin=pin)
except:
print("login failed, exception:", str(sys.exc_info()[1]))
break
objects = session.findObjects([(CKA_LABEL, "00000103000003A4")])
print()
print("Found %d objects: %s" % (len(objects), [x.value() for x in objects]))
all_attributes = list(PyKCS11.CKA.keys())
# only use the integer values and not the strings like 'CKM_RSA_PKCS'
all_attributes.remove(PyKCS11.CKA_PRIVATE_EXPONENT)
all_attributes.remove(PyKCS11.CKA_PRIME_1)
all_attributes.remove(PyKCS11.CKA_PRIME_2)
all_attributes.remove(PyKCS11.CKA_EXPONENT_1)
all_attributes.remove(PyKCS11.CKA_EXPONENT_2)
all_attributes.remove(PyKCS11.CKA_COEFFICIENT)
all_attributes = [e for e in all_attributes if isinstance(e, int)]
n_obj = 1
for o in objects:
print()
print((red + "==================== Object: %d ====================" + normal) % o.value())
n_obj += 1
try:
attributes = session.getAttributeValue(o, all_attributes)
except PyKCS11.PyKCS11Error as e:
continue
attrDict = dict(zip(all_attributes, attributes))
if attrDict[PyKCS11.CKA_CLASS] == PyKCS11.CKO_PRIVATE_KEY \
and attrDict[PyKCS11.CKA_KEY_TYPE] == PyKCS11.CKK_RSA:
m = attrDict[PyKCS11.CKA_MODULUS]
e = attrDict[PyKCS11.CKA_PUBLIC_EXPONENT]
if m and e:
mx = eval(b'0x' + ''.join("%02X" %c for c in m))
ex = eval(b'0x' + ''.join("%02X" %c for c in e))
if sign:
try:
toSign = "12345678901234567890123456789012" # 32 bytes, SHA256 digest
print("* Signing with object 0x%08X following data: %s" % (o.value(), toSign))
signature = session.sign(o, toSign)
sx = eval(b'0x' + ''.join("%02X" % c for c in signature))
print("Signature:")
print(dump(''.join(map(chr, signature))))
if m and e:
print("Verifying using following public key:")
print("Modulus:")
print(dump(''.join(map(chr, m))))
print("Exponent:")
print(dump(''.join(map(chr, e))))
decrypted = pow(sx, ex, mx) # RSA
print("Decrypted:")
d = binascii.unhexlify(hexx(decrypted))
print(dump(d))
if toSign == d[-20:]:
print("*** signature VERIFIED!\n")
the following is what prints. nothing seems to be working using the specific objects, there are no error messages
Slot no: 0
slotDescription: ProtectServer K5E:00045
manufacturerID: SafeNet Inc.
TokenInfo
Opened session 0x00000002
Found 2 objects: [5328, 5329]
==================== Object: 5328 ====================
==================== Object: 5329 ====================
You can work only with one token by checking its label before use, e.g.:
tokenInfo = pkcs11.getTokenInfo(slot)
if 'DesiredTokenLabel' == tokenInfo.label.decode('ascii').strip():
# Start working with this particular token
session = pkcs11.openSession(s)
You can enumerate only specific object using a template argument for the findObjects call, e.g.:
# get objects labelled "PRIV"
objects = session.findObjects([(CKA_LABEL, "PRIV")])
# get all private key objects
objects = session.findObjects([(CKA_CLASS, CKO_PRIVATE_KEY)])
# get all private key objects labelled "PRIV"
objects = session.findObjects([(CKA_CLASS, CKO_PRIVATE_KEY),(CKA_LABEL, "PRIV")])
# get all RSA private key objects labelled "PRIV"
objects = session.findObjects([(CKA_CLASS, CKO_PRIVATE_KEY),(CKA_KEY_TYPE, CKK_RSA),(CKA_LABEL, "PRIV")])
Below is an example code with hard-coded parameters:
from PyKCS11 import *
pkcs11 = PyKCS11.PyKCS11Lib()
pkcs11.load("your module path...")
slots = pkcs11.getSlotList()
for s in slots:
t = pkcs11.getTokenInfo(s)
if 'CKM' == t.label.decode('ascii').strip():
session = pkcs11.openSession(s)
objects = session.findObjects([(CKA_LABEL, "00000103000003A1")])
print ("Found %d objects: %s" % (len(objects), [x.value() for x in objects]))
Please note that it is Python 3 as I can't use PyKCS11 in Python 2.x right now.
Soma additional (random) notes:
do not rely on handles -- they may (and will) be different for different program runs
your program's command line arguments are up to you -- you must decide yourself if your program needs arguments like --token-label
Disclaimer: I am not that into python so please do validate my thoughts
Good luck!
EDIT(Regarding you recent edit)>
No error is (most probably) shown because the exception is caught and ignored:
try:
attributes = session.getAttributeValue(o, all_attributes)
except PyKCS11.PyKCS11Error as e:
continue