Concatenating strings containing many quotations results in slashes in output - python

I am trying to build a string that needs to contain specific double and single quotation characters for executing a SQL expression.
I need my output to be formatted like this:
" "Full_Stree" = 'ALLENDALE RD' "
where the value of ALLENDALE RD will be a variable defined through a For Loop. In the following code sample, the variable tOS is what I am trying to pass into the query variable.
tOS = "ALLENDALE RD"
query = '" "Full_Stree" = ' + "'" + tOS + "' " + '"'
and when I print the value of query variable I get this output:
'" "Full_Stree" = \'ALLENDALE RD\' "'
The slashes are causing my query to fail. I also tried using a modulus operator to pass the value of the tOS variable, but get the same results:
where = '" "Full_Stree" = \'%s\' "' % (tOS)
print where
'" "Full_Stree" = \'ALLENDALE RD\' "'
How can I get my string concatenated into the correct format, leaving the slashes out of the expression?

What you are seeing is the repr of your string.
>>> s = '" "Full_Stree" = \'ALLENDALE RD\' "'
>>> s # without print console displays the repr
'" "Full_Stree" = \'ALLENDALE RD\' "'
>>> print s # with print the string itself is displayed
" "Full_Stree" = 'ALLENDALE RD' "
Your real problem is the extra quotes at the beginning and end of your where-clause.
This
query = '" "Full_Stree" = ' + "'" + tOS + "' " + '"'
should be
query = '"Full_Stree" = ' + "'" + tOS + "'"
It is more clearly written as
query = """"Full_Stree" = '%s'""" % tOS
The ArcGis docs recommend something more like this
dataset = '/path/to/featureclass/shapefile/or/table'
field = arcpy.AddFieldDelimiters(dataset, 'Full_Stree')
whereclause = "%s = '%s'" % (field, tOS)
arcpy.AddFieldDelimiters makes sure that the field name includes the proper quoting style for the dataset you are using (some use double-quotes and some use square brackets).

Somehow the way I already tried worked out:
where = '" "Full_Stree" = \'%s\' "' % (tOS)
print where
'" "Full_Stree" = \'ALLENDALE RD\' "'

Can't you just use triple quotes?
a=""" "Full_Street" = 'ALLENDALE RD' """
print a
"Full_Street" = 'ALLENDALE RD'

Related

PDF template not merging data properly with pdftk

I'm editing a PDF template with using pdftk
command = ("pdftk " + '"' +
template + '"' +
" fill_form " + '"' +
pathUser + user['mail'] + ".xfdf" + '"' +
" output " + '"' +
pathUser + user['mail'] + ".pdf" + '"' +
" need_appearances")
command = command.replace('/', '\\')
os.system(command)
First I'm writing my data in a .xfdf file
for key, value in user.items():
print(key, value)
fields.append(u"""<field name="%s"><value>%s</value></field>""" % (key, value))
tpl = u"""<?xml version="1.0" encoding="UTF-8"?>
<xfdf xmlns="http://ns.adobe.com/xfdf/" xml:space="preserve">
<fields>
%s
</fields>
</xfdf>""" % "\n".join(fields)
f = open(pathUser + user['mail'] + '.xfdf', 'wb')
f.write(tpl.encode("utf-8"))
f.close()
I fetch the template and as shown above, write the data from the xfdf to pdf but for some reason, only the ime gets written.
Templates get fetched using some basic conditional logic as shown below:
for item in user['predavanja']:
user[acthead + str(actn)] = item
actn += 1
for item in user['radionice']:
user[acthead + str(actn)] = item
actn += 1
for item in user['izlet']:
user[acthead + str(actn)] = item
actn += 1
print(actn)
templates = {}
templates['0'] = "Template/2019/certificate_2019.pdf"
templates['5'] = "Template/2019/certificate_2019_5.pdf"
templates['10'] = "Template/2019/certificate_2019_10.pdf"
templates['15'] = "Template/2019/certificate_2019_15.pdf"
templates['20'] = "Template/2019/certificate_2019_20.pdf"
templates['25'] = "Template/2019/certificate_2019_25.pdf"
templates['30'] = "Template/2019/certificate_2019_30.pdf"
templates['35'] = "Template/2019/certificate_2019_35.pdf"
templates['40'] = "Template/2019/certificate_2019_40.pdf"
templates['45'] = "Template/2019/certificate_2019_45.pdf"
templates['50'] = "Template/2019/certificate_2019_50.pdf"
I'm writing this data
user['id'] = data['recommendations'][0]['role_in_team']['user']['id']
user['ime'] = data['recommendations'][0]['role_in_team']['user']['first_name']
user['prezime'] = data['recommendations'][0]['role_in_team']['user']['last_name']
user['tim'] = data['recommendations'][0]['role_in_team']['team']['short_name']
user['mail'] = data['recommendations'][0]['role_in_team']['user']['estudent_email']
user['puno_ime'] = (data['recommendations'][0]['role_in_team']['user']['first_name'] + ' ' +
data['recommendations'][0]['role_in_team']['user']['last_name'])
user['predavanja'] = predavanja
user['radionice'] = radionice
user['izlet'] = izlet
One note. predavanja, radionice and izlet are lists.
I've tried printing tpl which shows all the data being properly added to the scheme.
Turns out the issue was the naming of the variables since they didn't match the field names in the acroform PDF. So the solution was to rename the variables in the code to match the field names.

TypeError: cannot concatenate 'str' and 'dict' objects error with Bit.ly_API

I'm having a bit a problem. I'm still very new to python. So dict's are very new to me. I know of solve problems as I go and hopefully learn from my mistakes.
So the error is below with 'shortURL' in the first text variable. Just wondering what the issue means and how I could go about solving it? I've had a look online and didn't fully understand why as a lot of issues where looking at string but the bit.ly_api just return a URL.
Thanks for any help :)
print("")
print("Welcome to Sole Retriever Tweet Formulator b0.1")
type = (raw_input('What kind of tweet would you like to do?' + '\n' + '1. Store URL + Website Site' + '\n' + '2.Website URL Only' + '\n'))
if type == ('1'):
store1 = (raw_input('What is the store name hosting the Off-White x Converse Raffle? '))
storeURL = (raw_input('What is the direct URL to the raffle? '))
shortURL = b.shorten(storeURL)
text = ('Woof! ' + store1 + ' raffle is now live for the Off-White x Converse Chuck Taylor!' + '\n' + '\n' + shortURL + '\n' + '\n' + 'For raffle details and where to enter check and filter by "live" -' + '\n' + '\n' + 'https://www.soleretriever.com/off-white-x-converse-chuck-taylor/' + '\n' + '\n' + '#soleretriever #offwhite #converse #sneakers')
print (text)
os.system("echo '%s' | pbcopy" % text)
print ('\n')
print ('Copied to Clipboard')
twitter = Twython(APP_KEY, APP_SECRET, OAUTH_TOKEN, OAUTH_TOKEN_SECRET)
twitter.update_status(status=text)
if type == ('2'):
store = (raw_input('What is the store name hosting the Off-White x Converse Raffle? '))
text = ('Woof! ' + store + ' raffle is now live for the Off-White x Converse Chuck Taylor!' + '\n' + '\n' + 'For raffle details and where to enter check and filter by "live" -' + '\n' + '\n' + 'https://www.soleretriever.com/off-white-x-converse-chuck-taylor/' + '\n' + '\n' + '#soleretriever #offwhite #converse #sneakers')
print (text)
os.system("echo '%s' | pbcopy" % text)
print ('\n')
print ('Copied to Clipboard')
twitter = Twython(APP_KEY, APP_SECRET, OAUTH_TOKEN, OAUTH_TOKEN_SECRET)
twitter.update_status(status=text)
The call to b.shorten(storeURL) will return a dict with key/value pairs, including the url. Python will throw that error when you concatenate a string and dict like you're doing in print(). I'd suggest something like:
response = b.shorten(storeURL)
shortURL = response['url']

python3 string interpolation - two ways, neither one works

Given these 2 different string interpolation stanzas, neither one works. Both return the {n} and %(text) inserts as plain, raw text in the output. What am I missing?
Ive been using the %(string)s method forever in python2, now porting to python 3.
bulk_string = (
"bulk insert SomeDB.dbo.%(lru)s\n" +
"from 'C:\\Someplace\\"
"project\\%(filename)s'\n" +
"with (\n" +
" FIELDTERMINATOR = ',',\n" +
" ROWTERMINATOR = '%(term)s'\n" +
");"
% {
'lru': lru,
'filename': filename,
'term' : "\n"
}
)
and:
bulk_string = (
"bulk insert SomeDB.dbo.{0}\n" +
"from 'C:\\Someplace\\"
"project\\{1}'\n" +
"with (\n" +
" FIELDTERMINATOR = ',',\n" +
" ROWTERMINATOR = '{2}'\n" +
");"
.format(lru, filename, "\n")
)
either format or % apply only to the last string of your added strings. You could use """ (triple quoted strings) or parenthesize the strings (that, you did, but incorrectly):
bulk_string = (
"bulk insert SomeDB.dbo.{0}\n" +
"from 'C:\\Someplace\\"
"project\\{1}'\n" +
"with (\n" +
" FIELDTERMINATOR = ',',\n" +
" ROWTERMINATOR = '{2}'\n" +
");")
.format(lru, filename, "\\n")
or with triple quotes/raw string/automatic format positionning:
bulk_string = r"""bulk insert SomeDB.dbo.{}
from 'C:\Someplace\project\{}'
with (
FIELDTERMINATOR = ',',
ROWTERMINATOR = '{}'
);""".format(lru, filename, "\\n")
Aside: the third parameter of format should be \\n or r\n if you want to generate a literal \n in your code.
Here is the most readable way to do this:
In Python 3, you can use literal string interpolation, or f-strings, see PEP 498 - just be careful with escaping the backslashes properly, especially in front of a curly bracket such as in C:\Someplace\project\\{filename}
lru = "myTable"
filename = "myFile"
rt = "\\n"
bulk_string = f"""bulk insert SomeDB.dbo.{lru}
from 'C:\Someplace\project\\{filename}'
with ( FIELDTERMINATOR = ,
ROWTERMINATOR = '{rt}');"""

Trouble with apostrophe in arcpy search cursor where clause

I've put together a tkinter form and python script for downloading files from an ftp site. The filenames are in the attribute table of a shapefile, as well as an overall Name that the filenames correspond too. In other words I look up a Name such as "CABOT" and download the filename 34092_18.tif. However, if a Name has an apostrophe, such as "O'KEAN", it's giving me trouble. I try to replace the apostrophe, like I've done in previous scripts, but it doesn't download anything....
whereExp = quadField + " = " + "'" + quadName.replace("'", '"') + "'"
quadFields = ["FILENAME"]
c = arcpy.da.SearchCursor(collarlessQuad, quadFields, whereExp)
for row in c:
tifFile = row[0]
tifName = quadName.replace("'", '') + '_' + tifFile
#fullUrl = ftpUrl + tifFile
local_filename = os.path.join(downloadDir, tifName)
lf = open(local_filename, "wb")
ftp.retrbinary('RETR ' + tifFile, lf.write)
lf.close()
Here is an example of a portion of a script that works fine by replacing the apostrophe....
where_clause = quadField + " = " + "'" + quad.replace("'", '"') + "'"
#out_quad = quad.replace("'", "") + ".shp"
arcpy.MakeFeatureLayer_management(quadTable, "quadLayer")
select_out_feature_class = arcpy.SelectLayerByAttribute_management("quadLayer", "NEW_SELECTION", where_clause)

Python non-ascii characters

I have a python file that creates and populates a table in ms sql. The only sticking point is that the code breaks if there are any non-ascii characters or single apostrophes (and there are quite a few of each). Although I can run the replace function to rid the strings of apostrophes, I would prefer to keep them intact. I have also tried converting the data into utf-8, but no luck there either.
Below are th error messages I get:
"'ascii' codec can't encode character u'\2013' in position..." (for non-ascii characters)
and for the single quotes
class 'pyodbc.ProgrammingError'>: ('42000', "[42000] [Microsoft][ODBC SQL Server Driver][SQL Server] Incorrect syntax near 'S, 230 X 90M.; Eligibilty....
When I try to encode string in utf-8, I instead get the following error message:
<type 'exceptions.UnicodeDecodeError'>: ascii' codec can't decode byte 0xe2 in position 219: ordinal not in range(128)
The python code is included below. I believe the point in the code where this break occurs is after the following line: InsertValue = str(row.GetValue(CurrentField['Name'])).
# -*- coding: utf-8 -*-
import pyodbc
import sys
import arcpy
import arcgisscripting
gp = arcgisscripting.create(9.3)
SQL_KEYWORDS = ['PERCENT', 'SELECT', 'INSERT', 'DROP', 'TABLE']
#SourceFGDB = '###'
#SourceTable = '###'
SourceTable = sys.argv[1]
TempInputName = sys.argv[2]
SourceTable2 = sys.argv[3]
#---------------------------------------------------------------------------------------------------------------------
# Target Database Settings
#---------------------------------------------------------------------------------------------------------------------
TargetDatabaseDriver = "{SQL Server}"
TargetDatabaseServer = "###"
TargetDatabaseName = "###"
TargetDatabaseUser = "###"
TargetDatabasePassword = "###"
# Get schema from FGDB table.
# This should be an ordered list of dictionary elements [{'FGDB_Name', 'FGDB_Alias', 'FGDB_Type', FGDB_Width, FGDB_Precision, FGDB_Scale}, {}]
if not gp.Exists(SourceTable):
print ('- The source does not exist.')
sys.exit(102)
#### Should see if it is actually a table type. Could be a Feature Data Set or something...
print(' - Processing Items From : ' + SourceTable)
FieldList = []
Field_List = gp.ListFields(SourceTable)
print(' - Getting number of rows.')
result = gp.GetCount_management(SourceTable)
Number_of_Features = gp.GetCount_management(SourceTable)
print(' - Number of Rows: ' + str(Number_of_Features))
print(' - Getting fields.')
Field_List1 = gp.ListFields(SourceTable, 'Layer')
Field_List2 = gp.ListFields(SourceTable, 'Comments')
Field_List3 = gp.ListFields(SourceTable, 'Category')
Field_List4 = gp.ListFields(SourceTable, 'State')
Field_List5 = gp.ListFields(SourceTable, 'Label')
Field_List6 = gp.ListFields(SourceTable, 'DateUpdate')
Field_List7 = gp.ListFields(SourceTable, 'OBJECTID')
for Current_Field in Field_List1 + Field_List2 + Field_List3 + Field_List4 + Field_List5 + Field_List6 + Field_List7:
print(' - Field Found: ' + Current_Field.Name)
if Current_Field.AliasName in SQL_KEYWORDS:
Target_Name = Current_Field.Name + '_'
else:
Target_Name = Current_Field.Name
print(' - Alias : ' + Current_Field.AliasName)
print(' - Type : ' + Current_Field.Type)
print(' - Length : ' + str(Current_Field.Length))
print(' - Scale : ' + str(Current_Field.Scale))
print(' - Precision: ' + str(Current_Field.Precision))
FieldList.append({'Name': Current_Field.Name, 'AliasName': Current_Field.AliasName, 'Type': Current_Field.Type, 'Length': Current_Field.Length, 'Scale': Current_Field.Scale, 'Precision': Current_Field.Precision, 'Unique': 'UNIQUE', 'Target_Name': Target_Name})
# Create table in SQL Server based on FGDB table schema.
cnxn = pyodbc.connect(r'DRIVER={SQL Server};SERVER=###;DATABASE=###;UID=sql_webenvas;PWD=###')
cursor = cnxn .cursor()
#### DROP the table first?
try:
DropTableSQL = 'DROP TABLE dbo.' + TempInputName + '_Test;'
print DropTableSQL
cursor.execute(DropTableSQL)
dbconnection.commit()
except:
print('WARNING: Can not drop table - may not exist: ' + TempInputName + '_Test')
CreateTableSQL = ('CREATE TABLE ' + TempInputName + '_Test '
' (Layer varchar(500), Comments varchar(5000), State int, Label varchar(500), DateUpdate DATETIME, Category varchar(50), OBJECTID int)')
cursor.execute(CreateTableSQL)
cnxn.commit()
# Cursor through each row in the FGDB table, get values, and insert into the SQL Server Table.
# We got Number_of_Features earlier, just use that.
Number_Processed = 0
print(' - Processing ' + str(Number_of_Features) + ' features.')
rows = gp.SearchCursor(SourceTable)
row = rows.Next()
while row:
if Number_Processed % 10000 == 0:
print(' - Processed ' + str(Number_Processed) + ' of ' + str(Number_of_Features))
InsertSQLFields = 'INSERT INTO ' + TempInputName + '_Test ('
InsertSQLValues = 'VALUES ('
for CurrentField in FieldList:
InsertSQLFields = InsertSQLFields + CurrentField['Target_Name'] + ', '
InsertValue = str(row.GetValue(CurrentField['Name']))
if InsertValue in ['None']:
InsertValue = 'NULL'
# Use an escape quote for the SQL.
InsertValue = InsertValue.replace("'","' '")
if CurrentField['Type'].upper() in ['STRING', 'CHAR', 'TEXT']:
if InsertValue == 'NULL':
InsertSQLValues = InsertSQLValues + "NULL, "
else:
InsertSQLValues = InsertSQLValues + "'" + InsertValue + "', "
elif CurrentField['Type'].upper() in ['GEOMETRY']:
## We're not handling geometry transfers at this time.
if InsertValue == 'NULL':
InsertSQLValues = InsertSQLValues + '0' + ', '
else:
InsertSQLValues = InsertSQLValues + '1' + ', '
else:
InsertSQLValues = InsertSQLValues + InsertValue + ', '
InsertSQLFields = InsertSQLFields[:-2] + ')'
InsertSQLValues = InsertSQLValues[:-2] + ')'
InsertSQL = InsertSQLFields + ' ' + InsertSQLValues
## print InsertSQL
cursor.execute(InsertSQL)
cnxn.commit()
Number_Processed = Number_Processed + 1
row = rows.Next()
print(' - Processed all ' + str(Number_Processed))
del row
del rows
James, I believe the real issue is that your are not using Unicode accross the board. Try to do the following:
Make sure that your input file that you are using to populate the DB is in UTF-8 and that you are reading it with the UTF-8 encoder.
Make sure your DB is actually storing the data as Unicode
When you retrieve data from the file or from the DB or want to manipulate strings (with the + operator for instance) you need to make sure that all parts are proper Unicode. You can NOT use the str() method. You need to use unicode() as Dave pointed out. If you define strings in your code use u'my string' instead of 'my string' (otherwise it is not considered unicode).
Also, please provide us the full stack trace and the exception name.
I'm going to use my psychic debugging skills and say you are trying to str()ify something and getting an error with the ascii codec. What you really should do is to use the utf-8 codec instead like this:
insert_value_uni = unicode(row.GetValue(CurrentField['Name']))
InsertValue = insert_value_uni.encode('utf-8')
Or you can take the view that only ASCII is allowed and use the awesomely named Unicode Hammer
In general you want to convert to unicode on data input, and convert to the desired encoding on output.
So it will be easier to find your problem if you do this. This means changing all strings to unicode, 'INSERT INTO ' to u'INSERT INTO '. (Notice the "u" before the string)
Then when you send the string to be executed convert to the desired encoding, "utf8".
cursor.execute(InsertSQL.encode("utf8")) # Where InsertSQL is unicode
Also, you should add the encoding string to the top of your source code.
This means adding the encoding cookie to one of the first two lines of the file:
#!/usr/bin/python
# -*- coding: <encoding name> -*-
If your pulling data from a file to build your string you can uses codecs.open to auto convert from a specific encoding to unicode on load.
When I converted my str() to unicode, that solved the problem. A simple answer, and I appreciate everyone's help on this.

Categories