I've just started using pycassa, so if this is a stupid question, I apologize upfront.
I have a column family with the following schema:
create column family MyColumnFamilyTest
with column_type = 'Standard'
and comparator = 'CompositeType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.TimeUUIDType)'
and default_validation_class = 'BytesType'
and key_validation_class = 'UTF8Type'
and read_repair_chance = 0.1
and dclocal_read_repair_chance = 0.0
and populate_io_cache_on_flush = false
and gc_grace = 864000
and min_compaction_threshold = 4
and max_compaction_threshold = 32
and replicate_on_write = true
and compaction_strategy = 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
and caching = 'KEYS_ONLY'
and compression_options = {'sstable_compression' : 'org.apache.cassandra.io.compress.SnappyCompressor'};
When I try to do a get() with a valid key (works fine in cassandra-cli) I get:
Traceback (most recent call last):
File "<pyshell#19>", line 1, in <module>
cf.get('mykey',column_count=3)
File "/usr/local/lib/python2.7/dist-packages/pycassa-1.11.0-py2.7.egg/pycassa/columnfamily.py", line 664, in get
return self._cosc_to_dict(list_col_or_super, include_timestamp, include_ttl)
File "/usr/local/lib/python2.7/dist-packages/pycassa-1.11.0-py2.7.egg/pycassa/columnfamily.py", line 368, in _cosc_to_dict
ret[self._unpack_name(col.name)] = self._col_to_dict(col, include_timestamp, include_ttl)
File "/usr/local/lib/python2.7/dist-packages/pycassa-1.11.0-py2.7.egg/pycassa/columnfamily.py", line 444, in _unpack_name
return self._name_unpacker(b)
File "/usr/local/lib/python2.7/dist-packages/pycassa-1.11.0-py2.7.egg/pycassa/marshal.py", line 140, in unpack_composite
components.append(unpacker(bytestr[2:2 + length]))
File "/usr/local/lib/python2.7/dist-packages/pycassa-1.11.0-py2.7.egg/pycassa/marshal.py", line 374, in <lambda>
return lambda v: uuid.UUID(bytes=v)
File "/usr/lib/python2.7/uuid.py", line 144, in __init__
raise ValueError('bytes is not a 16-char string')
ValueError: bytes is not a 16-char string
Here's some more information I've discovered:
When using cassandra-cli I can see the data as:
% cassandra-cli -h 10.249.238.131
Connected to: "LocalDB" on 10.249.238.131/9160
Welcome to Cassandra CLI version 1.2.10-SNAPSHOT
Type 'help;' or '?' for help.
Type 'quit;' or 'exit;' to quit.
[default#unknown] use Keyspace;
[default#Keyspace] list ColumnFamily;
Using default limit of 100
Using default cell limit of 100
-------------------
RowKey: urn:keyspace:ColumnFamily:a36e8ab1-7032-4e4c-a53d-e3317f63a640:
=> (name=autoZoning:::, value=01, timestamp=1391298393966000)
=> (name=creationTime:::, value=00000143efd8b76e, timestamp=1391298393966000)
=> (name=inactive:::14fe78e0-8b9b-11e3-b171-005056b700bb, value=00, timestamp=1391298393966000)
=> (name=label:::14fe78e0-8b9b-11e3-b171-005056b700bb, value=726a6d2d766e782d76613031, timestamp=1391298393966000)
1 Row Returned.
Elapsed time: 16 msec(s).
Since it was unclear what was causing the exception, I decided to add a print prior to the 'return self._name_unpacker(b)' line in columnfamily.py and I see:
>>> cf.get(dict(cf.get_range(column_count=0,filter_empty=False)).keys()[0])
Attempting to unpack: <00>\rautoZoning<00><00><00><00><00><00><00><00><00><00>
Traceback (most recent call last):
File "<pyshell#172>", line 1, in <module>
cf.get(dict(cf.get_range(column_count=0,filter_empty=False)).keys()[0])
File "/usr/local/lib/python2.7/dist-packages/pycassa-1.11.0-py2.7.egg/pycassa/columnfamily.py", line 665, in get
return self._cosc_to_dict(list_col_or_super, include_timestamp, include_ttl)
File "/usr/local/lib/python2.7/dist-packages/pycassa-1.11.0-py2.7.egg/pycassa/columnfamily.py", line 368, in _cosc_to_dict
ret[self._unpack_name(col.name)] = self._col_to_dict(col, include_timestamp, include_ttl)
File "/usr/local/lib/python2.7/dist-packages/pycassa-1.11.0-py2.7.egg/pycassa/columnfamily.py", line 445, in _unpack_name
return self._name_unpacker(b)
File "/usr/local/lib/python2.7/dist-packages/pycassa-1.11.0-py2.7.egg/pycassa/marshal.py", line 140, in unpack_composite
components.append(unpacker(bytestr[2:2 + length]))
File "/usr/local/lib/python2.7/dist-packages/pycassa-1.11.0-py2.7.egg/pycassa/marshal.py", line 374, in <lambda>
return lambda v: uuid.UUID(bytes=v)
File "/usr/lib/python2.7/uuid.py", line 144, in __init__
raise ValueError('bytes is not a 16-char string')
ValueError: bytes is not a 16-char string
I have no idea where the extra characters are coming from around the column name. But that got me curious so I added another print in _cosc_to_dict in columnfamily.py and I see:
>>> cf.get(dict(cf.get_range(column_count=0,filter_empty=False)).keys()[0])
list_col_or_super is: []
list_col_or_super is: [ColumnOrSuperColumn(column=Column(timestamp=1391298393966000,
name='\x00\rautoZoning\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', value='\x01', ttl=None),
counter_super_column=None, super_column=None, counter_column=None),
ColumnOrSuperColumn(column=Column(timestamp=1391298393966000,
name='\x00\x0ccreationTime\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00',
value='\x00\x00\x01C\xef\xd8\xb7n', ttl=None), counter_super_column=None, super_column=None,
counter_column=None), ColumnOrSuperColumn(column=Column(timestamp=1391298393966000,
name='\x00\x08inactive\x00\x00\x00\x00\x00\x00\x00\x00\x10\x14\xfex\xe0\x8b\x9b\x11\xe3\xb1q\x00PV\xb7\x00\xbb\x00', value='\x00', ttl=None), counter_super_column=None, super_column=None,
counter_column=None), ColumnOrSuperColumn(column=Column(timestamp=1391298393966000,
name='\x00\x05label\x00\x00\x00\x00\x00\x00\x00\x00\x10\x14\xfex\xe0\x8b\x9b\x11\xe3\xb1q\x00PV\xb7\x00\xbb\x00', value='thisIsATest', ttl=None), counter_super_column=None, super_column=None, counter_column=None)]
autoZoning unpack:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib64/python2.6/site-packages/pycassa-1.11.0-py2.6.egg/pycassa/columnfamily.py", line 666, in get
return self._cosc_to_dict(list_col_or_super, include_timestamp, include_ttl)
File "/usr/local/lib64/python2.6/site-packages/pycassa-1.11.0-py2.6.egg/pycassa/columnfamily.py", line 369, in _cosc_to_dict
ret[self._unpack_name(col.name)] = self._col_to_dict(col, include_timestamp, include_ttl)
File "/usr/local/lib64/python2.6/site-packages/pycassa-1.11.0-py2.6.egg/pycassa/columnfamily.py", line 446, in _unpack_name
return self._name_unpacker(b)
File "/usr/local/lib64/python2.6/site-packages/pycassa-1.11.0-py2.6.egg/pycassa/marshal.py", line 140, in unpack_composite
components.append(unpacker(bytestr[2:2 + length]))
File "/usr/local/lib64/python2.6/site-packages/pycassa-1.11.0-py2.6.egg/pycassa/marshal.py", line 374, in <lambda>
return lambda v: uuid.UUID(bytes=v)
File "/usr/lib64/python2.6/uuid.py", line 144, in __init__
raise ValueError('bytes is not a 16-char string')
ValueError: bytes is not a 16-char string
Am I correct in assuming that the extra characters around the column names are what is responsible for the 'ValueError: bytes is not a 16-char string' exception?
Also if I try to use the column name and select it I get:
>>> cf.get(u'urn:keyspace:ColumnFamily:a36e8ab1-7032-4e4c-a53d-e3317f63a640:',columns=['autoZoning:::'])
Traceback (most recent call last):
File "<pyshell#184>", line 1, in <module>
cf.get(u'urn:keyspace:ColumnFamily:a36e8ab1-7032-4e4c-a53d-e3317f63a640:',columns=['autoZoning:::'])
File "/usr/local/lib/python2.7/dist-packages/pycassa-1.11.0-py2.7.egg/pycassa/columnfamily.py", line 651, in get
cp = self._column_path(super_column, column)
File "/usr/local/lib/python2.7/dist-packages/pycassa-1.11.0-py2.7.egg/pycassa/columnfamily.py", line 383, in _column_path
self._pack_name(column, False))
File "/usr/local/lib/python2.7/dist-packages/pycassa-1.11.0-py2.7.egg/pycassa/columnfamily.py", line 426, in _pack_name
return self._name_packer(value, slice_start)
File "/usr/local/lib/python2.7/dist-packages/pycassa-1.11.0-py2.7.egg/pycassa/marshal.py", line 115, in pack_composite
packed = packer(item)
File "/usr/local/lib/python2.7/dist-packages/pycassa-1.11.0-py2.7.egg/pycassa/marshal.py", line 298, in pack_uuid
randomize=True)
File "/usr/local/lib/python2.7/dist-packages/pycassa-1.11.0-py2.7.egg/pycassa/util.py", line 75, in convert_time_to_uuid
'neither a UUID, a datetime, or a number')
ValueError: Argument for a v1 UUID column name or value was neither a UUID, a datetime, or a number
Any further thoughts?
Thanks,
Rob
Turns out that the problem wasn't with the key, it was being caused, in part, by a bug in pycassa that wasn't handling an empty (null) string in the column UUID. A short-term fix is in the answer in google groups:
https://groups.google.com/d/msg/pycassa-discuss/Vf_bSgDIi9M/KTA1kbE9IXAJ
The other part of the answer was to get at the columns by using tuples (with the UUID as a UUID and not a str) instead of a string with ':' separators because that's, as I found out, a cassandra-cli thing.
Related
I am using the code below to simulate a model.
def run_demo(with_plots=True):
traj = np.array([[start_time,2.25]])
input_object = ('input_1[1]', traj)
model = load_fmu('[pyfmimodel.fmu',log_level=7)
opts = model.simulate_options ()
opts['ncp']=266
# Simulate
res = model.simulate(options=opts, input=input_object,final_time=stop_time )
This is the error I am getting. I need help to resolve this error.
Traceback (most recent call last):
File "D:\Projects\Python\DOCKER\model_2.py", line 55, in <module>
run_demo()
File "D:\Projects\Python\DOCKER\model_2.py", line 38, in run_demo
res = model.simulate(options=opts, input=input_object,final_time=stop_time )
File "src\pyfmi\fmi.pyx", line 7519, in pyfmi.fmi.FMUModelCS2.simulate
File "src\pyfmi\fmi.pyx", line 378, in pyfmi.fmi.ModelBase._exec_simulate_algorithm
File "src\pyfmi\fmi.pyx", line 372, in pyfmi.fmi.ModelBase._exec_simulate_algorithm
File "C:\Users\tcto5k\Miniconda3\lib\site-packages\pyfmi\fmi_algorithm_drivers.py", line 984, in __init__
self.result_handler.simulation_start()
File "C:\Users\tcto5k\Miniconda3\lib\site-packages\pyfmi\common\io.py", line 2553, in simulation_start
[parameter_data, sorted_vars_real_vref, sorted_vars_int_vref, sorted_vars_bool_vref] = fmi_util.prepare_data_info(data_info, sorted_vars,
File "src\pyfmi\fmi_util.pyx", line 257, in pyfmi.fmi_util.prepare_data_info
File "src\pyfmi\fmi_util.pyx", line 337, in pyfmi.fmi_util.prepare_data_info
File "src\pyfmi\fmi.pyx", line 4377, in pyfmi.fmi.FMUModelBase2.get_boolean
pyfmi.fmi.FMUException: Failed to get the Boolean values.
This is the FMU model variable definition which accepts 1D array as input:
<ScalarVariable name="input_1[1]" valueReference="0" description="u" causality="input" variability="continuous">
<Real start="2.0"/>
</ScalarVariable>
<!-- 2 -->
<ScalarVariable name="dense_3[1]" valueReference="614" description="y (1st order)" causality="output" variability="continuous" initial="calculated">
<Real/>
</ScalarVariable>
I'm having trouble using the Python email module to parse emails where the FROM header has parentheses in it. This only seems to be the problem when using email.policy.default as opposed to email.policy.compat32.
Is there a solution to this problem, other than switching policies?
A minimum working example is below, for Python 3.6.5:
import email
import email.policy as email_policy
raw_mime_msg=b"from: James Mishra \\(says hi\\) <james#example.com>"
compat32_obj = email.message_from_bytes(
raw_mime_msg, policy=email_policy.compat32)
default_obj = email.message_from_bytes(
raw_mime_msg, policy=email_policy.default)
print(compat32_obj['from'])
print(default_obj['from'])
The first print statement returns:
James Mishra \(says hi\) <james#example.com>
and the second print statement returns:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/email/_header_value_parser.py", line 1908, in get_address
token, value = get_group(value)
File "/usr/local/lib/python3.6/email/_header_value_parser.py", line 1867, in get_group
"display name but found '{}'".format(value))
email.errors.HeaderParseError: expected ':' at end of group display name but found '\(says hi\) <james#example.com>'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/email/_header_value_parser.py", line 1734, in get_mailbox
token, value = get_name_addr(value)
File "/usr/local/lib/python3.6/email/_header_value_parser.py", line 1720, in get_name_addr
token, value = get_angle_addr(value)
File "/usr/local/lib/python3.6/email/_header_value_parser.py", line 1646, in get_angle_addr
"expected angle-addr but found '{}'".format(value))
email.errors.HeaderParseError: expected angle-addr but found '\(says hi\) <james#example.com>'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "test_email.py", line 12, in <module>
print(default_obj['from'])
File "/usr/local/lib/python3.6/email/message.py", line 391, in __getitem__
return self.get(name)
File "/usr/local/lib/python3.6/email/message.py", line 471, in get
return self.policy.header_fetch_parse(k, v)
File "/usr/local/lib/python3.6/email/policy.py", line 162, in header_fetch_parse
return self.header_factory(name, value)
File "/usr/local/lib/python3.6/email/headerregistry.py", line 589, in __call__
return self[name](name, value)
File "/usr/local/lib/python3.6/email/headerregistry.py", line 197, in __new__
cls.parse(value, kwds)
File "/usr/local/lib/python3.6/email/headerregistry.py", line 340, in parse
kwds['parse_tree'] = address_list = cls.value_parser(value)
File "/usr/local/lib/python3.6/email/headerregistry.py", line 331, in value_parser
address_list, value = parser.get_address_list(value)
File "/usr/local/lib/python3.6/email/_header_value_parser.py", line 1931, in get_address_list
token, value = get_address(value)
File "/usr/local/lib/python3.6/email/_header_value_parser.py", line 1911, in get_address
token, value = get_mailbox(value)
File "/usr/local/lib/python3.6/email/_header_value_parser.py", line 1737, in get_mailbox
token, value = get_addr_spec(value)
File "/usr/local/lib/python3.6/email/_header_value_parser.py", line 1583, in get_addr_spec
token, value = get_local_part(value)
File "/usr/local/lib/python3.6/email/_header_value_parser.py", line 1413, in get_local_part
obs_local_part, value = get_obs_local_part(str(local_part) + value)
File "/usr/local/lib/python3.6/email/_header_value_parser.py", line 1454, in get_obs_local_part
token, value = get_word(value)
File "/usr/local/lib/python3.6/email/_header_value_parser.py", line 1340, in get_word
if value[0]=='"':
IndexError: string index out of range
email.policy.default is intended to be compliant with the email RFCs, and your message is not compliant with RFC 5322. If the parenthesized part is supposed to be a comment, then the message should look like
raw_mime_msg=b"from: James Mishra (says hi) <james#example.com>"
to be compliant. If it is not supposed to be a comment, then the parentheses should appear inside a quoted string. That might look something like
raw_mime_msg=b'from: "James Mishra (says hi)" <james#example.com>'
Since your message is not compliant, using the policy that expects compliance is a poor fit. If you want to handle non-compliant messages, email.policy.compat32 is a better choice than email.policy.default.
I have put an item in AWS DynamoDB which is in Binary form . I mean my data is encrypted and when I try to access that data to print or assign then I am not able to dousing python
import boto3
import base64
dynamodb = boto3.resource('dynamodb', region_name='us-west-2')
table = dynamodb.Table('xyz')
response = table.get_item(
Key={
'id': "2",
}
)
item = response['Item']
dec_otp=item['otp']
print(dec_otp) #not able to print
a=dec_otp #not able to assign
print(type(dec_otp))
print(item)
output:-
Traceback (most recent call last):
File "aws_kms3.py", line 23, in <module>
'name': 'otp'
File "/home/vinit/Desktop/kms/venv/lib/python3.6/site-packages/botocore/client.py", line 314, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/home/vinit/Desktop/kms/venv/lib/python3.6/site-packages/botocore/client.py", line 586, in _make_api_call
api_params, operation_model, context=request_context)
File "/home/vinit/Desktop/kms/venv/lib/python3.6/site-packages/botocore/client.py", line 621, in _convert_to_request_dict
api_params, operation_model)
File "/home/vinit/Desktop/kms/venv/lib/python3.6/site-packages/botocore/validate.py", line 291, in serialize_to_request
raise ParamValidationError(report=report.generate_report())
File "/home/vinit/Desktop/kms/venv/lib/python3.6/site-packages/botocore/validate.py", line 95, in generate_report
error_messages.append(self._format_error(error))
File "/home/vinit/Desktop/kms/venv/lib/python3.6/site-packages/botocore/validate.py", line 112, in _format_error
', '.join(additional['valid_types']))
TypeError: __str__ returned non-string (type bytes)
output2:-
Traceback (most recent call last):
File "aws_kms3.py", line 25, in <module>
CiphertextBlob=bytes(base64.b64decode(dec_otp))
File "/home/vinit/Desktop/kms/venv/lib/python3.6/base64.py", line 80, in b64decode
s = _bytes_from_decode_data(s)
File "/home/vinit/Desktop/kms/venv/lib/python3.6/base64.py", line 46, in _bytes_from_decode_data
"string, not %r" % s.__class__.__name__) from None
TypeError: argument should be a bytes-like object or ASCII string, not 'Binary'
output3:-
Binary object is not itterable
I refer to Neo4j create nodes and relationships from pandas dataframe with py2neo and code with belows.
But I got py2neo.database.status.CypherSyntaxError. Please verify below is right approach and let me know what is correct Cypher syntax.
My Code:
for line in reader:
print(line['word'], line['similar_word'], line['probability'] )
w1 = Node("Word", name = line['word'])
w2 = Node("Word", name = line['similar_word'])
graph.merge(w1|w2)
graph.run('''
MATCH (a:Paper),(b:Word)
WHERE (a.name = 'Paper10' AND b.name = {$word1})
CREATE (a)<-[o:ORIGINAL]-(b)
''', parameters = {'word1':line['word']})
py2neo.database.status.CypherSyntaxError:
Traceback (most recent call last): File "test.py", line 24, in <module>
''', parameters = {'word1':line['word']}) File "/root/miniconda3/lib/python3.6/site-packages/py2neo/database/__init__.py", line 731, in run
return self.begin(autocommit=True).run(statement, parameters, **kwparameters) File "/root/miniconda3/lib/python3.6/site-packages/py2neo/database/__init__.py", line 1277, in run
self.finish() File "/root/miniconda3/lib/python3.6/site-packages/py2neo/database/__init__.py", line 1296, in finish
self._sync() File "/root/miniconda3/lib/python3.6/site-packages/py2neo/database/__init__.py", line 1286, in _sync
connection.fetch() File "/root/miniconda3/lib/python3.6/site-packages/py2neo/packages/neo4j/v1/bolt.py", line 344, in fetch
handler(*fields) File "/root/miniconda3/lib/python3.6/site-packages/py2neo/database/__init__.py", line 961, in on_failure
raise GraphError.hydrate(metadata) py2neo.database.status.CypherSyntaxError: Invalid input '$': expected whitespace, an identifier, UnsignedDecimalInteger, a property key name or '}' (line 3, column 53 (offset: 78)) " WHERE (a.name = 'Paper10' AND b.name = {$word1})"
For referencing a parameter, you can either use the $ syntax, or the {}, but not both. Try this with just $word1.
I am trying to write a XML file but as I run ET.dump(root) I am getting:
<Reviews><Review rid="en_India'sGrill_477960693"><sentences><sentenceTraceback (most recent call last):
File "/usr/lib/python3.5/xml/etree/ElementTree.py", line 1078, in _escape_attrib
if "&" in text:
TypeError: argument of type 'Sentence' is not iterable
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/media/Data/workspaces/git/master-thesis/python/thesis/semeval/slot1/nlp_cnn.py", line 421, in <module>
f.write(ET.dump(root))
File "/usr/lib/python3.5/xml/etree/ElementTree.py", line 1165, in dump
elem.write(sys.stdout, encoding="unicode")
File "/usr/lib/python3.5/xml/etree/ElementTree.py", line 775, in write
short_empty_elements=short_empty_elements)
File "/usr/lib/python3.5/xml/etree/ElementTree.py", line 940, in _serialize_xml
short_empty_elements=short_empty_elements)
File "/usr/lib/python3.5/xml/etree/ElementTree.py", line 940, in _serialize_xml
short_empty_elements=short_empty_elements)
File "/usr/lib/python3.5/xml/etree/ElementTree.py", line 940, in _serialize_xml
short_empty_elements=short_empty_elements)
File "/usr/lib/python3.5/xml/etree/ElementTree.py", line 932, in _serialize_xml
v = _escape_attrib(v)
File "/usr/lib/python3.5/xml/etree/ElementTree.py", line 1090, in _escape_attrib
_raise_serialization_error(text)
File "/usr/lib/python3.5/xml/etree/ElementTree.py", line 1056, in _raise_serialization_error
"cannot serialize %r (type %s)" % (text, type(text).__name__)
TypeError: cannot serialize <thesis.semeval_data.Sentence object at 0x7f9087f20400> (type Sentence)
No idea what the problem is. I'm constructing this in a very straight forward way:
root = ET.Element("Reviews")
i = 0
for k in review_dict:
review_element = ET.Element("Review")
root.append(review_element)
review_element.set("rid", k)
sentences_element = ET.Element("sentences")
review_element.append(sentences_element)
sentence_dict = review_dict[k]
for k in sentence_dict:
sentence_element = ET.Element("sentence")
sentence_element.set("id", k)
sentences_element.append(sentence_element)
sentence = sentence_dict[k]
text_element = ET.Element("text")
text_element.text = sentence.text
sentences_element.append(text_element)
opinions_element = ET.Element("Opinions")
sentence_element.append(opinions_element)
for category in aspect_categories[y_pred[i] > 0]:
opinion_element = ET.Element("Opinion")
opinion_element.set("category", category)
opinions_element.append(opinion_element)
i += 1
f = open(os.path.join(os.path.curdir, "..", "..", "pred.xml"), "w")
f.write(ET.dump(root)) # Exception
Any idea why I am getting this error? At first I thought it's because I didn't escape things but it appears that xml.etree.ElementTree does that already.
Just look at the traceback: _escape_attrib() raises a serialization error. Then refer to the source: ElementTree expects that element attribute value is a string and catches exceptions if it is not.
Check the type of values being used as <sentence> attributes: sentence_element.set("id", k). One of them is not a string.