We Keep Coding

How can i reduce my sqlite3 code using variables?

I got around 500 lines of sqlite code and i am trying to reduce that number. So i tried making a function with variables.
I tried using this code:
x = "c.execute("
o = "'"
y = "'INSERT INTO "
unio = x+y
tabl = ["d","d_2","n_d","n_d_2"]
val = "VALUES ("
w = ") "
k = "?,?"
mark = 369
ek = 963
def db_up(table,):
aa = y+table+"("+tabl[0] + ", " + tabl[1]+w+val+k+")"+"',"+"(mark,ek,)"
bb = unio+table+"("+tabl[1]+ ", " + tabl[1]+w+val+k+")"+"',"+"(mark,ek,))"
print(aa) # 'INSERT INTO avg_dt(d, d_2) VALUES (?,?)',(mark,ek,)
print(bb) # c.execute('INSERT INTO avg_dt(d_2, d_2) VALUES (?,?)',(mark,ek,))
c.execute(str(aa)) # no succes
c.execute(aa) # no success
bb # no success
When i run the "c.execute(aa)" line. it throws me this error:
sqlite3.OperationalError: near "'INSERT INTO avg_dt(d, d_2) VALUES (?,?)'": syntax error
So...how can i make a sqlite3 code using variables a functions?
Thanks for taking the time ;)

I can't post a comment with my current rep, so let me try to provide a full answer.
Your approach seems to be that you have too much SQL code, which you consider a problem, so you're trying to "compress" the code by reducing common statements into shorter variable names. There are a number of reasons you should avoid doing this, and a number of better ways to achieve a similar result.
First, let's talk about why this is a bad idea:
Your code doesn't become more maintainable. Interspersing cryptic variable names like x, o, y, or unio doesn't make the code any easier to read, even for the author, assuming a few days have passed since you wrote it.
Using this kind of method doesn't make your code any more performant, and most likely makes it less performant: your program now has to worry about allocating and reallocating memory when doing string interpolation or concatenation, which takes cycles.
Doing string interpolation or concatenation in SQL should be done with extreme caution: this is essentially a homebrew version of prepared statements, which are usually authored by people with loads of experience doing SQL programming. Making this on your own risks your program being targetted by SQL injection (or at least column/value type mismatches).
Now, let's talk about mitigating this issue for you:
Lots of SQL code need not be unmanageable: typically, if you must maintain large amounts of raw SQL in your project, you dump that into a separate SQL file which you can then either execute directly from the database CLI or run from your program via a database driver.
ORMs are (usually) your friend: with the exception of exotic or outstandingly performance-sensitive SQL queries, modern ORMs can get rid of most raw SQL code in your program. The more naturally programmatic structure of ORM code also means you can break it up into different delegate functions to avoid code reuse, to an extent.
Please feel free to add details to your question; as it stands, it's not totally clear whether your concerns can be addressed with this answer.

Getting exception "openpyxl.utils.exceptions.IllegalCharacterError"

I am extracting data from an Oracle 11g Database using python and writing it to an Excel file. During extraction, I'm using a python list of tuples (each tuple indicates each row in dataset) and the openpyxl module to write the data into Excel. It's working fine for some datasets but for some, it's throwing the exception:
openpyxl.utils.exceptions.IllegalCharacterError
This is the solution I've already tried:
Openpyxl.utils.exceptions.IllegalcharacterError
Here is my Code:
for i in range(0,len(list)):
for j in range(0,len(header)):
worksheet_ntn.cell(row = i+2, column = j+1).value = list[i][j]
Here is the error message:
raise IllegalCharacterError
openpyxl.utils.exceptions.IllegalCharacterError

I did get this error because of some hex charactres in some of my strings.
'Suport\x1f_01'
The encode\decode solutions mess with the accente words too
So...
i resolve this with repr()
value = repr(value)
That give a safe representation, with quotation marks
And then i remove the first and last charactres
value = repr(value)[1:-1]
Now you can safe insert value on your cell

The exception tells you everything you need to know: you must replace the characters that cause the exception. This can be done using re.sub() but, seeing as only you can decide what you want to replace them with — spaces, empty strings, etc. — only you can do this.

SQLite3 (Python): execute() command not working with string parameters

I'm writing code that takes a user-inputted string and binds it to a SQLite command through a parameter.
I've gotten this functionality to work when the string is already embedded in the query (see first example under "these lines do work"), and this binding method works with integers just fine (see second example).
testname = "\"EJtheDJ\"" #this would be user-inputted, but I get the same issue when I initialize it this way
#this line doesn't work
c.execute("select lastfm from usernames where discordname=?", (testname,))
#these lines do work
#c.execute("select lastfm from usernames where discordname=\"EJtheDJ\"")
#c.execute("select integer1 from test where integer2=?", (num,)) #num=3
print(c.fetchone()) #prints out "None"
My syntax seems 100% correct according to all the tutorials and forum threads I've looked at. I don't normally make posts here (in fact this is my first time), but I'm completely stumped on this and felt it was my only option. Any help would be massively appreciated. I'm running this on a CentOS server with Python 3.6, if that helps.
Edit: Turns out I just needed to do away with the escaped quotes. testname = EJtheDJ works perfectly.

what is the matter with the select_string wrapped in py_sqlite3?

I can select a record in sqlitemanager (firefox plugin).Please see the attatchment of the selected record.The database is here , http://pan.baidu.com/s/1eQ04tlo .
You can download it and test the sql statement.
select capital.代码 as code,quote.close as close, quote.close*capital.股份总数/10000 as market_cap,
balance.负债合计/balance.资产总计 as debt_ratio,
profit.归属于母公司所有者的净利润/balance.'所有者权益(或股东权益)合计' as roe,
profit.归属于母公司所有者的净利润/balance.'负债和所有者权益(或股东权益)总计' as roa ,
quote.close*capital.股份总数/profit.归属于母公司所有者的净利润 as pe,
quote.close*capital.股份总数/balance.资产总计 as pb
from profile,capital,quote,balance,profit
where capital.代码=300001
and profile.代码=capital.代码
and profile.代码=quote.code
and profile.代码=balance.代码
and profile.代码=profit.代码
and quote.date="20140523"
and balance.报告日期="20131231"
and profit.报告日期="20131231";
When i wrap it in python code ,i can't get the selected record,What is the matter?
# -*- coding: utf-8 -*-
import os,re,sqlite3
db_name='china.sqlite'
target_dir='e:\\workspace\\data\\'
con = sqlite3.connect(target_dir+db_name)
cur=con.cursor()
select_str='''select capital.代码 as code,quote.close as close, quote.close*capital.股份总数/10000 as market_cap,
balance.负债合计/balance.资产总计 as debt_ratio,
profit.归属于母公司所有者的净利润/balance.'所有者权益(或股东权益)合计' as roe,
profit.归属于母公司所有者的净利润/balance.'负债和所有者权益(或股东权益)总计' as roa ,
quote.close*capital.股份总数/profit.归属于母公司所有者的净利润 as pe,
quote.close*capital.股份总数/balance.资产总计 as pb
from profile,capital,quote,balance,profit
where capital.代码=300001
and profile.代码=capital.代码
and profile.代码=quote.code
and profile.代码=balance.代码
and profile.代码=profit.代码
and quote.date='20140523'
and balance.报告日期='20131231'
and profit.报告日期='20131231';'''
cur.execute(select_str)
print(cur.fetchall())
it's so strange for me to understand that that code1 can run,code2 can not run.
code1
import os,re,sqlite3
db_name='china.sqlite'
target_dir='e:\\workspace\\data\\'
con = sqlite3.connect(target_dir+db_name)
cur=con.cursor()
select_str='''select capital.代码 as code,quote.close as close, quote.close*capital.股份总数/10000 as market_cap,
balance.负债合计/balance.资产总计 as debt_ratio,
profit.归属于母公司所有者的净利润/balance.'所有者权益(或股东权益)合计' as roe,
profit.归属于母公司所有者的净利润/balance.'负债和所有者权益(或股东权益)总计' as roa ,
quote.close*capital.股份总数/profit.归属于母公司所有者的净利润 as pe,
quote.close*capital.股份总数/balance.资产总计 as pb
from profile,capital,quote,balance,profit
where capital.代码='300001'
and profile.代码='300001'
and quote.code='300001'
and balance.代码='300001'
and profit.代码='300001'
and quote.date='20140523'
and balance.报告日期='20131231'
and profit.报告日期='20131231';'''
cur.execute(select_str)
print(cur.fetchall())
code2
import os,re,sqlite3
db_name='china.sqlite'
target_dir='e:\\workspace\\data\\'
con = sqlite3.connect(target_dir+db_name)
cur=con.cursor()
select_str='''select capital.代码 as code,quote.close as close, quote.close*capital.股份总数/10000 as market_cap,
balance.负债合计/balance.资产总计 as debt_ratio,
profit.归属于母公司所有者的净利润/balance.'所有者权益(或股东权益)合计' as roe,
profit.归属于母公司所有者的净利润/balance.'负债和所有者权益(或股东权益)总计' as roa ,
quote.close*capital.股份总数/profit.归属于母公司所有者的净利润 as pe,
quote.close*capital.股份总数/balance.资产总计 as pb
from profile,capital,quote,balance,profit
where capital.代码=300001
and profile.代码=capital.代码
and profile.代码=quote.code
and profile.代码=balance.代码
and profile.代码=profit.代码
and quote.date='20140523'
and balance.报告日期='20131231'
and profit.报告日期='20131231';'''
cur.execute(select_str)
print(cur.fetchall())
In my point of view ,cod1 is equal to code2 logically ,there is no difference between them.
Is there difference between
where capital.代码='300001'
and profile.代码='300001'
and quote.code='300001'
and balance.代码='300001'
and profit.代码='300001'
and
where capital.代码=300001
and profile.代码=capital.代码
and profile.代码=quote.code
and profile.代码=balance.代码
and profile.代码=profit.代码
??
when i quote the code and get the right result ,so the reason of problem remain unknown.

I haven't downloaded the database since it's about 250 megabytes, and that's far too much to download to answer a simple question. If you want people to download the database, I suggest creating a small database with just a few records, the ones in your question.
Now for the actual answer: when you put quotes around the number 300001, you're searching for a string. When you omit the quotes, you're searching for an integer. The SQLite manual says that integers will never compare equal to strings (SQLite calls it a TEXT value):
An INTEGER or REAL value is less than any TEXT or BLOB value.
So your WHERE clause is never comparing equal when you search for an integer, because the actual value is a string. That's why you have to put quotes around "300001" is working.
UPDATE: The original version of my answer said "the actual value is an integer", which was backwards. After looking at the SQLite3 database that the OP posted, I see that the "代码" fields (for those who don't read Chinese, that translates roughly as "source code") are defined as TEXT fields in his schema. So I've edited my answer to swap integers and strings.
UPDATE 2: It seems, from the screenshots you've posted of sqlitemanager, that it is converting the integer values in your query to strings, because it knows that the SQL schema says that the 代码 field is a TEXT field and so it is correcting your query for you. (Which it should not do -- the number 1 is NOT the same as the string "1" and the one should NOT be converted into the other unless the user specifically asks for that conversion, but that's a rant about bad software design, which doesn't help answer your question.) So this "feature" of sqlitemanager (that's really a bug) is hiding the problem in your query, but the pysqlite3 library does not auto-convert data types and so it is exposing the problem.
Regardless of whether sqlitemanager's behavior is the right thing or not, the solution to your problem is very simple: make sure that when you're searching for a value in a TEXT field such as 代码, that you give it a search value that is a text string, enclosed in quotes. That will work everywhere, in both Python and in sqlitemanager, and so that's the way you want to write your queries.

replace the strings with variables in python

The following statement works as expected and myreservation object is created.
myreservation = conn.run_instances(amiid, placement='us-east-1a', key_name='n15a',instance_type=my_instance_type,security_groups=['N-1-0-1-AutogenByAWSMP-'])
But the following does not work. I get an error in the script later because myreservation object is not created.
myreservation = ("conn.run_instances('%s', placement='us-east-1a', key_name='n15a',instance_type='%s',security_groups=['%s'])" % (amiid, my_instance_type, my_security_groups))
I need to replace the security group with variable because that will make it easy for me to change it later.
error is:
The security group '%s' does not exist
when I print that statement and run it at python console, it works as expected
And the following does not work either:
myreservation = conn.run_instances('%s', placement='us-east-1a', key_name='nov15a',instance_type='%s',security_groups=['%s']) % (amiid, my_instance_type, my_security_groups)

your second version doesn't work because you put "conn.run_instances" into the string which probably is not what you want. hard to tell from your very small code example but I'd guess:
myreservation = conn.run_instances(amiid, placement='us-east-1a', key_name='n15a',instance_type=my_instance_type,security_groups=[my_security_groups])
is what you are looking for (assuming my_security_groups is correctly set)
edit:
Why does %s does not work?
'%s' works fine if you use it correctly.
in your third example:
myreservation = conn.run_instances('%s', placement='us-east-1a', key_name='nov15a',instance_type='%s',security_groups=['%s']) % (amiid, my_instance_type, my_security_groups)
you try to use a single replacement tuple for three different strings. this does not work, you'd have to replace each string individually. something like this would probably work, but it doesn't really make sense to use string replacements if you're replacing the whole string
myreservation = conn.run_instances('%s'%amiid, placement='us-east-1a', key_name='nov15a',instance_type='%s'%my_instance_type,security_groups=['%s'my_security_groups])