We Keep Coding

Convert boto3 BaseCondition to its string representation

I am trying to convert boto3 dynamoDB conditional expressions (using types from boto3.dynamodb.conditions) to its string representation. Of course this could be hand coded but naturally I would prefer to be able to find something developed by AWS itself.
Key("name").eq("new_name") & Attr("description").begins_with("new")
would become
"name = 'new_name' and begins_with(description, 'new')"
I have been checking in the boto3 and boto core code but so far no success, but I assume it must exist somewhere in the codebase...

In the boto3.dynamodb.conditions module there is a class called ConditionExpressionBuilder. You can convert a condition expression to string by doing the following:
condition = Key("name").eq("new_name") & Attr("description").begins_with("new")
builder = ConditionExpressionBuilder()
expression = builder.build_expression(condition, is_key_condition=True)
expression_string = expression.condition_expression
expression_attribute_names = expression.attribute_name_placeholders
expression_attribute_values = expression.attribute_value_placeholders
I'm not sure why this isn't documented anywhere. I just randomly found it looking through the source code at the bottom of this page https://boto3.amazonaws.com/v1/documentation/api/latest/_modules/boto3/dynamodb/conditions.html.
Unfortunately, this doesn't work for the paginator format string notation, but it should work for the Table.query() format.

From #Brian's answer with
ConditionExpressionBuilder I had to add the Dynamodb's {'S': 'value'} type notation before the query execution.
I changed it with expression_attribute_values[':v0'] = {'S': pk_value}, where :v0 is the first Key/Attr in the condition. Not sure but should work for next values (:v0, :v1, :v2...).
Here is the full code, using pagination to retrieve only part of data
from boto3.dynamodb.conditions import Attr, Key, ConditionExpressionBuilder
from typing import Optional, List
import boto3
client_dynamodb = boto3.client("dynamodb", region_name="us-east-1")
def get_items(self, pk_value: str, pagination_config: dict = None) -> Optional[List]:
if pagination_config is None:
pagination_config = {
# Return only first page of results when no pagination config is not provided
'PageSize': 300,
'StartingToken': None,
'MaxItems': None,
}
condition = Key("pk").eq(pk_value)
builder = ConditionExpressionBuilder()
expression = builder.build_expression(condition, is_key_condition=True)
expression_string = expression.condition_expression
expression_attribute_names = expression.attribute_name_placeholders
expression_attribute_values = expression.attribute_value_placeholders
# Changed here to make it compatible with dynamodb typing
python expression_attribute_values[':v0'] = {'S': pk_value}
paginator = client_dynamodb.get_paginator('query')
page_iterator = paginator.paginate(
TableName="TABLE_NAME",
IndexName="pk_value_INDEX",
KeyConditionExpression=expression_string,
ExpressionAttributeNames=expression_attribute_names,
ExpressionAttributeValues=expression_attribute_values,
PaginationConfig=pagination_config
)
for page in page_iterator:
resp=page
break
if ("Items" not in resp) or (len(resp["Items"]) == 0):
return None
return resp["Items"]
EDIT:
I used this question to get string representation for Dynamodb Resource's query, which is not compatible (yet) with dynamodb conditions, but then I found a better solution from Github (Boto3)[https://github.com/boto/boto3/issues/2300]:
Replace paginator with the one from meta
dynamodb_resource = boto3.resource("dynamodb")
paginator = dynamodb_resource.meta.client.get_paginator('query')
And now I can simply use Attr and Key

python byte array print() gives wrong values

Using python 3, I made the following script consisting of a model class containing hundreds of bytearrays and in the same script outside of model class i am printing some of these out to verify they are correct. When i print the values some of values are not what i expected.(i put coded comments to identify these in the code below)
Here is a shortend version of my script with some of the bytearrays
`
class Model:
def __init__(self):
# weird values:
self.bp_diastole_118 = bytearray(b'\xff\x01\x02\x01\x01\x00\x02\x3b')
self.bp_diastole_120 = bytearray(b'\xff\x01\x02\x01\x01\x00\x02\x3c')
self.bp_diastole_122 = bytearray(b'\xff\x01\x02\x01\x01\x00\x02\x3d')
self.bp_diastole_124 = bytearray(b'\xff\x01\x02\x01\x01\x00\x02\x3e')
self.bp_diastole_126 = bytearray(b'\xff\x01\x02\x01\x01\x00\x02\x3f')
self.bp_diastole_128 = bytearray(b'\xff\x01\x02\x01\x01\x00\x02\x40')
self.bp_diastole_160 = bytearray(b'\xff\x01\x02\x01\x01\x00\x02\x50')
# correct values:
self.pupil_r_normal = bytearray(b'\xff\x01\x02\x01\x01\x00\x03\xc3')
self.pupil_r_dilated = bytearray(b'\xff\x01\x02\x01\x01\x00\x03\xc4')
self.pupil_r_constriced = bytearray(b'\xff\x01\x02\x01\x01\x00\x03\xc5')
self.pupil_r_reaction_on = bytearray(b'\xff\x01\x02\x01\x01\x00\x03\xc6')
self.pupil_r_reaction_off = bytearray(b'\xff\x01\x02\x01\x01\x00\x03\xc7')
m = Model()
print('--------------weird value------------------')
print('bp_diastole_118 = {}'.format(m.bp_diastole_118))
print('bp_diastole_120 = {}'.format(m.bp_diastole_120))
print('bp_diastole_122 = {}'.format(m.bp_diastole_122))
print('bp_diastole_124 = {}'.format(m.bp_diastole_124))
print('bp_diastole_126 = {}'.format(m.bp_diastole_126))
print('bp_diastole_128 = {}'.format(m.bp_diastole_128))
print('bp_diastole_160 = {}'.format(m.bp_diastole_160))
print('-------------correct value--------------------')
print('pupil_r_normal = {}'.format(m.pupil_r_normal))
print('pupil_r_dilated = {}'.format(m.pupil_r_dilated))
print('pupil_r_constriced = {}'.format(m.pupil_r_constriced))
print('pupil_r_reaction_on = {}'.format(m.pupil_r_reaction_on))
print('pupil_r_reaction_off = {}'.format(m.pupil_r_reaction_off))
here is what is printed to the console:
`
--------------weird value------------------
bp_diastole_118 = bytearray(b'\xff\x01\x02\x01\x01\x00\x02;')
bp_diastole_120 = bytearray(b'\xff\x01\x02\x01\x01\x00\x02<')
bp_diastole_122 = bytearray(b'\xff\x01\x02\x01\x01\x00\x02=')
bp_diastole_124 = bytearray(b'\xff\x01\x02\x01\x01\x00\x02>')
bp_diastole_126 = bytearray(b'\xff\x01\x02\x01\x01\x00\x02?')
bp_diastole_128 = bytearray(b'\xff\x01\x02\x01\x01\x00\x02#')
bp_diastole_160 = bytearray(b'\xff\x01\x02\x01\x01\x00\x02P')
-------------correct value--------------------
pupil_r_normal = bytearray(b'\xff\x01\x02\x01\x01\x00\x03\xc3')
pupil_r_dilated = bytearray(b'\xff\x01\x02\x01\x01\x00\x03\xc4')
pupil_r_constriced = bytearray(b'\xff\x01\x02\x01\x01\x00\x03\xc5')
pupil_r_reaction_on = bytearray(b'\xff\x01\x02\x01\x01\x00\x03\xc6')
pupil_r_reaction_off = bytearray(b'\xff\x01\x02\x01\x01\x00\x03\xc7')
As you can see the good values print exactly what i would expect and are identical to the values i initialized. However if you look at what was printed from the weird values you can see the last 3 characters do not match the values i initialized.
i.e.
initialized:
self.bp_diastole_118 = bytearray(b'\xff\x01\x02\x01\x01\x00\x02\x3b')
is not the same as printed:
bp_diastole_118 = bytearray(b'\xff\x01\x02\x01\x01\x00\x02;')
Does anybody know why this is happening and how I could remedy the problem?

you are seeing the utf representation of the hex value you set for example utf('0x3b') == ';'

The problem I found is x3b through x50 they seem to be outputting the last part as special character with x50 being P. If you test using bytearray(b'\x3b') it will display the ending for you

Getting wrong result from JSON - Python 3

Im working on a small project of retrieving information about books from the Google Books API using Python 3. For this i make a call to the API, read out the variables and store those in a list. For a search like "linkedin" this works perfectly. However when i enter "Google", it reads the second title from the JSON input. How can this happen?
Please find my code below (Google_Results is the class I use to initialize the variables):
import requests
def Book_Search(search_term):
parms = {"q": search_term, "maxResults": 3}
r = requests.get(url="https://www.googleapis.com/books/v1/volumes", params=parms)
print(r.url)
results = r.json()
i = 0
for result in results["items"]:
try:
isbn13 = str(result["volumeInfo"]["industryIdentifiers"][0]["identifier"])
isbn10 = str(result["volumeInfo"]["industryIdentifiers"][1]["identifier"])
title = str(result["volumeInfo"]["title"])
author = str(result["volumeInfo"]["authors"])[2:-2]
publisher = str(result["volumeInfo"]["publisher"])
published_date = str(result["volumeInfo"]["publishedDate"])
description = str(result["volumeInfo"]["description"])
pages = str(result["volumeInfo"]["pageCount"])
genre = str(result["volumeInfo"]["categories"])[2:-2]
language = str(result["volumeInfo"]["language"])
image_link = str(result["volumeInfo"]["imageLinks"]["thumbnail"])
dict = Google_Results(isbn13, isbn10, title, author, publisher, published_date, description, pages, genre,
language, image_link)
gr.append(dict)
print(gr[i].title)
i += 1
except:
pass
return
gr = []
Book_Search("Linkedin")
I am a beginner to Python, so any help would be appreciated!

It does so because there is no publisher entry in volumeInfo of the first entry, thus it raises a KeyError and your except captures it. If you're going to work with fuzzy data you have to account for the fact that it will not always have the expected structure. For simple cases you can rely on dict.get() and its default argument to return a 'valid' default entry if an entry is missing.
Also, there are a few conceptual problems with your function - it relies on a global gr which is bad design, it shadows the built-in dict type and it captures all exceptions guaranteeing that you cannot exit your code even with a SIGINT... I'd suggest you to convert it to something a bit more sane:
def book_search(search_term, max_results=3):
results = [] # a list to store the results
parms = {"q": search_term, "maxResults": max_results}
r = requests.get(url="https://www.googleapis.com/books/v1/volumes", params=parms)
try: # just in case the server doesn't return valid JSON
for result in r.json().get("items", []):
if "volumeInfo" not in result: # invalid entry - missing volumeInfo
continue
result_dict = {} # a dictionary to store our discovered fields
result = result["volumeInfo"] # all the data we're interested is in volumeInfo
isbns = result.get("industryIdentifiers", None) # capture ISBNs
if isinstance(isbns, list) and isbns:
for i, t in enumerate(("isbn10", "isbn13")):
if len(isbns) > i and isinstance(isbns[i], dict):
result_dict[t] = isbns[i].get("identifier", None)
result_dict["title"] = result.get("title", None)
authors = result.get("authors", None) # capture authors
if isinstance(authors, list) and len(authors) > 2: # you're slicing from 2
result_dict["author"] = str(authors[2:-2])
result_dict["publisher"] = result.get("publisher", None)
result_dict["published_date"] = result.get("publishedDate", None)
result_dict["description"] = result.get("description", None)
result_dict["pages"] = result.get("pageCount", None)
genres = result.get("authors", None) # capture genres
if isinstance(genres, list) and len(genres) > 2: # since you're slicing from 2
result_dict["genre"] = str(genres[2:-2])
result_dict["language"] = result.get("language", None)
result_dict["image_link"] = result.get("imageLinks", {}).get("thumbnail", None)
# make sure Google_Results accepts keyword arguments like title, author...
# and make them optional as they might not be in the returned result
gr = Google_Results(**result_dict)
results.append(gr) # add it to the results list
except ValueError:
return None # invalid response returned, you may raise an error instead
return results # return the results
Then you can easily retrieve as much info as possible for a term:
gr = book_search("Google")
And it will be far more tolerant of data omissions, provided that your Google_Results type makes most of the entries optional.

Following #Coldspeed's recommendation it became clear that missing information in the JSON file caused the exception to run. Since I only had a "pass" statement there it skipped the entire result. Therefore I will have to adapt the "Try and Except" statements so errors do get handled properly.
Thanks for the help guys!

Python - LibreOffice Calc - Find & Replace with Regular Expression

I try to code a Find & Replace method with Python in LibreOffice's Calc to replace all the ".+" with "&" (in a single column - not so important) - unfortunately, even a standard Find & Replace method seems to be impossible (to me). That's what I have up to now:
import uno
def search()
desktop = XSCRIPTCONTEXT.getDesktop()
document = XSCRIPTCONTEXT.getDocument()
ctx = uno.getComponentContext()
sm = ctx.ServiceManager
dispatcher = sm.createInstanceWithContext("com.sun.star.frame.DispatchHelper", ctx)
model = desktop.getCurrentComponent()
doc = model.getCurrentController()
sheet = model.Sheets.getByIndex(0)
replace = sheet.createReplaceDescriptor()
replace.SearchRegularExpression = True
replace.SearchString = ".+$"
replace.ReplaceString ="&"
return None
And what happens: totally nothing! I will be happy and thankful for every hint, sample code and motivating words!

This code changes all non-empty cells in column A to &:
def calc_search_and_replace():
desktop = XSCRIPTCONTEXT.getDesktop()
model = desktop.getCurrentComponent()
sheet = model.Sheets.getByIndex(0)
COLUMN_A = 0
cellRange = sheet.getCellRangeByPosition(COLUMN_A, 0, COLUMN_A, 65536);
replace = cellRange.createReplaceDescriptor()
replace.SearchRegularExpression = True
replace.SearchString = r".+$"
replace.ReplaceString = r"\&"
cellRange.replaceAll(replace)
Notice that the code calls replaceAll to actually do something. Also, from the User Guide:
& will insert the same string found with the Search RegExp.
So the replace string needs to be literal -- \&.

Using Keys as Variables in Python

There is probably a term for what I'm attempting to do, but it escapes me. I'm using peewee to set some values in a class, and want to iterate through a list of keys and values to generate the command to store the values.
Not all 'collections' contain each of the values within the class, so I want to just include the ones that are contained within my data set. This is how far I've made it:
for value in result['response']['docs']:
for keys in value:
print keys, value[keys] # keys are "identifier, title, language'
#for value in result['response']['docs']:
# collection = Collection(
# identifier = value['identifier'],
# title = value['title'],
# language = value['language'],
# mediatype = value['mediatype'],
# description = value['description'],
# subject = value['subject'],
# collection = value['collection'],
# avg_rating = value['avg_rating'],
# downloads = value['downloads'],
# num_reviews = value['num_reviews'],
# creator = value['creator'],
# format = value['format'],
# licenseurl = value['licenseurl'],
# publisher = value['publisher'],
# uploader = value['uploader'],
# source = value['source'],
# type = value['type'],
# volume = value['volume']
# )
# collection.save()

for value in result['response']['docs']:
Collection(**value).save()
See this question for an explanation on how **kwargs work.

Are you talking about how to find out whether a key is in a dict or not?
>>> somedict = {'firstname': 'Samuel', 'lastname': 'Sample'}
>>> if somedict.get('firstname'):
>>> print somedict['firstname']
Samuel
>>> print somedict.get('address', 'no address given'):
no address given
If there is a different problem you'd like to solve, please clarify your question.