I have a few parameters typed in a string that need to be extracted & substituted using the string format() function. The code snippet is as follows:
# Filter list (output from another script)
filters ='recipient_id=100, reporting_date=2020-10-12'
# Fetch SQL file from location
sql_file = 'configs/sql_files/{sql_name}.sql'.format(sql_name=sql_name)
file_path = os.path.realpath(__file__)
final_file_path = "/".join(file_path.split("/")[:-2]) + "/" + sql_file
with open(final_file_path) as sql_file:
# Pass in filters to the SQL snippet. The SQL snippet has two parameters
# recipient_id & reporting_date
sql = sql_file.read().format(filters)
try:
sf = get_sql_client()
except Exception as e:
print("Error connecting to the DB!")
sys.exit()
df = sf.fetch_df(sql)
The code snippet fails as "filters" is being passed as a string.
sql_file.read().format('recipient_id=100, reporting_date=2020-10-12')
Instead, it should be passed as below:
sql_file.read().format(recipient_id=100, reporting_date='2020-10-12')
Is there a way to extract the "filters" string & format it as above?
SQL file sample:
SELECT columns..
FROM A
join B on <condition>
WHERE true
AND REPORTING_LEVEL_1_ID = '{recipient_id}'
AND date_trunc('day', delivered_date_pt) >= DATEADD(day, -7, last_day('{reporting_date}'::date, 'week') + 1)
AND date_trunc('day', delivered_date_pt) <= last_day('{reporting_date}'::date, 'week')
try using double quotes like so...
filters ="recipient_id=100, reporting_date='2020-10-12'"
Assuming the sql file with contents like that and for filtering information formatting as shown (as a series of name and value pairs, separated by comma and space characters), you could create dictionary from the latter and then pass it to the format() method as shown (using a ** prefix to unpack the items in it).
filter_data = 'recipient_id=100, reporting_date=2020-10-12'
final_file_path = 'sql_file'
with open(final_file_path) as sql_file:
pairs = (pair.split('=') for pair in filter_data.replace(',', '').split())
mapping = {k: v for (k, v) in pairs}
sql = sql_file.read().format(**mapping)
print(sql)
Output:
SELECT columns..
FROM A
join B on <condition>
WHERE true
AND REPORTING_LEVEL_1_ID = '100'
AND date_trunc('day', delivered_date_pt) >= DATEADD(day, -7, last_day('2020-10-12'::date, 'week') + 1)
AND date_trunc('day', delivered_date_pt) <= last_day('2020-10-12'::date, 'week')
Related
cursor.execute("insert into demoapp_site ('name','firstname','lastname') values (%s,%s,%s)",site_data)
in above query I want to pass column name dynamically and %s which is inside values also not predefine that how many times it need to write
so, how can I write my sql query according to my data which comes from html table in 2d array from??
We need to build the query before using it. As follows
field_string = ""
value_string = ""
# This will string in this format "'field1','field2'"
# Assuming array is in format [["field1", "value1"],..]
# Update the for loop statement based on your input, other lines remain same
for field, value in my_field_array:
temp_field_string = "'%s'" % field
field_string = field_string + temp_field_string + ","
temp_value_string = "'%s'" % value
value_string = value_string + temp_value_string + ","
# Removing trailing comma's before passing the values
cmd = "insert into demoapp_site (%s) values (%s)" % (field_string[:-1], value_string[:-1])
cursor.execute(cmd)
I need to format an SQL query, and it looks like this:
s += " t{}.{} = '{}' and".format(t_c, filter_c, filter_value)
but when the filter_value is something like m's it will result in
psycopg2.errors.SyntaxError: syntax error
if I use the double quote, it will say there's no such column
Any way I can resolve this problem, please?
Caused by injection vulnerability. Use parameters for filter_value and let the database API handle it.
If the table/schema names are coming from user input, whitelist those too. Parameters aren't possible for table names).
try this:
def format_value( string ):
if '\'' in string:
k = string.split('\'')
string_list = list(map(lambda x: '\'' + x+ '\'', k ))
return ''.join(string_list )
else:
return string
then
filter_value = format_value(filter_value)
before you pass filter_value to your query
I wrote code to append a json response into a list for some API work I am doing, but it stores the single quotes around the alphanumerical value I desire. I would like to get rid of the single quotes. Here is what I have so far:
i = 0
deviceID = []
while i < deviceCount:
deviceID.append(devicesRanOn['resources'][i])
deviceID[i] = re.sub('[\W_]', '', deviceID[i])
i += 1
if i >= deviceCount:
break
if (deviceCount == 1):
print ('Device ID: ', deviceID)
elif (deviceCount > 1):
print ('Device IDs: ', deviceID)
the desired input should look like this:
input Device IDs:
['14*************************00b29', '58*************************c3df4']
Output:
['14*************************00b29', '58*************************c3df4']
Desired Output:
[14*************************00b29, 58*************************c3df4]
As you can see, I am trying to use RegEx to filter non Alphanumeric and replace those with nothing. It is not giving me an error nor is it preforming the actions I am looking for. Does anyone have a recommendation on how to fix this?
Thank you,
xOm3ga
You won't be able to use the default print. You'll need to use your own means of making a representation for the list. But this is easy with string formatting.
'[' + ', '.join(f'{id!s}' for id in ids) + ']'
The f'{id:!s} is an f-string which formats the variable id using it's __str__ method. If you're on a version pre-3.6 which doesn't use f-strings, you can also use
'%s' % id
'{!s}'.format(id)
PS:
You can simplify you're code significantly by using a list comprehension and custom formatting instead of regexes.
ids = [device for device in devicesRanOn['resources'][:deviceCount]]
if deviceCount == 1:
label = 'Device ID:'
elif deviceCount > 1:
label = 'Device IDs:'
print(label, '[' + ', '.join(f'{id!s}' for id in ids) + ']')
I am trying to make query in mongodb where I don't know how many variables I have from the beginning.
You get an array with variables(strings) and then mongo should return via the $or operator so there are no duplicates.
def findRecipes(IngredientsHome):
#Query and return all the alternatives
arrayCompleteSearch = [];
if len(IngredientsHome) > 1:
#theHardStuuff
#Create the query based on ingredientsarray
argsCount = len(IngredientsHome)
argsFunction = "";
for i in range(0, len(IngredientsHome)):
if i < (len(IngredientsHome)-1):
argsFunction += ("{"+'"_Ingredients":' + '"' + IngredientsHome[i] + '"' "}, ")
else:
argsFunction += "{" + '"_Ingredients":' + '"' + IngredientsHome[i] + '"' + "}"
if I then do like this
print("coll.find({" + '$or' + ':' + '[' + argsFunction + ']' "})")
the commandline will print(not execute) the command i want to do in mongodb but I don't know how to make it work in mongo and return the results to an array. I am using pymongo and there i don't if it's possible to use a string as a full query. In sql I could put the whole "query" as a string "SELECT * FROM ..." while pymongo need : outside the string, more like coll.find({"String":"String"}). Is there a similar method or another way to make this query when I don't know how many expressions I have from the beginning in my mongodb query?
Instead of constructing your query as a string, use PyMongo methods to query your database, for example:
coll.find({'_Ingredients': IngredientsHome[i]})
or you can pass the Python list and use $in in your query:
coll.find({'_Ingredients': {'$in': IngredientsHome}})
Read more about how to use PyMongo here.
I'm wondering if you can help me. I'm trying to change the value in each column if the text matches a corresponding keyword. This is the loop:
for i in range(0, 20, 1):
cur.execute("UPDATE table SET %s = 1 WHERE text rlike %s") %(column_names[i], search_terms[i])
The MySQL command works fine on its own, but not when I put it in the loop. It's giving an error at the first %s
Does anyone have any insights?
This is the error:
_mysql_exceptions.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '%s = 1 WHERE text rlike %s' at line 1")
Column names looks like
column_names = ["col1","col2","col3"...]
Search terms look like
search_terms = ["'(^| |.|-)word1[;:,. ?-]'","'(^| |.|-)word2[;:,. ?-]'",...]
The right way to do this is to give values to Python, which will quote things correctly.
adapted from voyager's post:
for i in range(0, 20, 1):
cur.execute("UPDATE table SET {} = 1 WHERE text rlike %s".format(column_names[i]),
(search_terms[i],),
)
In this case it's confusing because the column_name isn't a value, it's part of the table structure, so it's inserted using good old string formatting. The search_term is a value, so is passed to cursor.execute() for correct, safe quoting.
(Don't use string manipulation to add the quotes -- you're exposing yourself to SQL injection.)
Missing quotes and wrong parenthesis placement...
for i in range(0, 20, 1):
cur.execute("UPDATE table SET %s = 1 WHERE text rlike '%s'" %(column_names[i], search_terms[i]))
# ^ ^
# (-----------------------------------------------------------------------------------)
Please note, this is not the right way of doing this, if your string may contain quotes by itself...
What about that instead:
for i in range(0, 20, 1):
cur.execute("UPDATE table SET %s = 1 WHERE text rlike ?" % (column_names[i],),
(search_terms[i],))
This uses the % operator to set the column name, but uses an executes parameter to bind the data, letting the DB driver escape all characters that need so.