Question: how do I insert a datetime value into MS SQL server, given the code below?
Context:
I have a 2-D list (i.e., a list of lists) in Python that I'd like to upload to a table in Microsoft SQL Server 2008. For this project I am using Python's pymssql package. Each value in each list is a string except for the very first element, which is a datetime value.
Here is how my code reads:
import pymssql
db_connect = pymssql.connect( # these are just generic names
server = server_name,
user = db_usr,
password = db_pwd,
database = db_name
)
my_cursor = db_connect.cursor()
for individual_list in list_of_lists:
# the first value in the paranthesis should be datetime
my_cursor.execute("INSERT INTO [DB_Table_Name] VALUES (%s, %s, %s, %s, %s, %s, %s, %s)", tuple(individual_list))
db_connect.commit()
The python interpreter is having a tough time inserting my datetime values. I understand that currently I have %s and that it is a string formatter, but I'm unsure what I should use for datetime, which is what the database's first column is formatted as.
The "list of lists" looks like this (after each list is converted into a tuple):
[(datetime.datetime(2012, 4, 1), '1', '4.1', 'hip', 'A1', 'J. Smith', 'B123', 'XYZ'),...]
Here is an illustration of what the table should look like:
+-----------+------+------+--------+-------+-----------+---------+---------+
| date | step | data | type | ID | contact | notif. | program |
+-----------+------+------+--------+-------+-----------+---------+---------+
|2012-04-01 | 1 | 4.1 | hip | A1 | J. Smith | B123 | XYZ |
|2012-09-05 | 2 | 5.1 | hip | A9 | B. Armst | B123 | ABC |
|2012-01-16 | 5 | 9.0 | horray | C6 | F. Bayes | P995 | XYZ |
+-----------+------+------+--------+-------+-----------+---------+---------+
Thank you in advance.
I would try formatting the date time to "yyyymmdd hh:mm:ss" before inserting. With what you are doing SQL will be parsing the string so I would also build the entire string and then insert the string as a variable. See below
for individual_list in list_of_lists:
# the first value in the parentheses should be datetime
date_time = individual_list[0].strftime("%Y%m%d %H:%M:%S")
insert_str = "INSERT INTO [DB_Table_Name] VALUES (" + str(date_time) + "),(" + str(individual_list[1]) + ");"
print insert_str
my_cursor.execute(insert_str)
db_connect.commit()
I apologize for the crude python but SQL should like that insert statement as long as all the fields match up. If not you may want to specify what fields those values go to in your insert statement.
Let me know if that works.
Related
so I know how to use %s but it doesn't appear to work for a column name.
my aim here is to get a column name (a roll number here) and use that to find information (how many days they attended)
roll=input("enter roll no.: ")
c1.execute("select sum(%s) from attendance", ("" + roll + "",))
a=c1.fetchall()
the table looks like:
date | 11b1 | 11b2 | 11b3 |......| 11b45 |
2020-12-01 | 1 | 0 | 1 |......| 1 |
2020-12-02 | 1 | 1 | 1 |......| 0 |
2020-12-03 | 0 | 1 | 1 |......| 1 |
this doesn't work and seems to give me a random value
so how do I write that middle code? also why does the original code not give errors but still give an arbitrary seeming number?
I will assume you mean the columns_names;
According toPython 3.8
By using f string¹
roll = input("enter roll name.: ")
a = c1.execute(f"select sum({roll}) from attendance").fetchall()
The names of MySQL schema objects - tables, columns etc - can be interpolated using string formatting, by surrounding the placeholder with backticks ('`', ASCII 0x96). See MySQL Identifiers.
Using backticks prevents errors if the column name contains a space, or matches a keyword or reserved word.
However backticks do not protect against SQL injection. As the programmer, it's your responsibility to make sure that any column names coming from outside your program (for example, from user input) are verified as matching the column names in the table.
colname = 'roll'
sql = f"""SELECT sum(`{colname}`) FROM attendance"""
mycursor.execute(sql)
For values in INSERT or UPDATE statements, or WHERE clauses, DB-API parameter substitution should always be used.
colname = 'roll'
colvalue = 42
sql = f"""SELECT sum(`{colname}`) FROM attendance WHERE colvalue = %s"""
mycursor.execute(sql, (colvalue,))
I am trying to store some tables I create in my code in an RDS instance using psycopg2. The script runs without issue and I can see the table being stored correctly in the DB. However, if I try to retrieve the query, I only see the columns, but no data:
import pandas as pd
import psycopg2
test=pd.DataFrame({'A':[1,1],'B':[2,2]})
#connect is a function to connect to the RDS instance
connection= connect()
cursor=connection.cursor()
query='CREATE TABLE test (A varchar NOT NULL,B varchar NOT NULL);'
cursor.execute(query)
connection.commit()
cursor.close()
connection.close()
This script runs without issues and, printing out file_check from the following script:
connection=connect()
# check if file already exists in SQL
sql = """
SELECT "table_name","column_name", "data_type", "table_schema"
FROM INFORMATION_SCHEMA.COLUMNS
WHERE "table_schema" = 'public'
ORDER BY table_name
"""
file_check=pd.read_sql(sql, con=connection)
connection.close()
I get:
table_name column_name data_type table_schema
0 test a character varying public
1 test b character varying public
which looks good.
Running the following however:
read='select * from public.test'
df=pd.read_sql(read,con=connection)
returns:
Empty DataFrame
Columns: [a, b]
Index: []
Anybody have any idea why this is happening? I cannot seem to get around this
Erm, your first script has a test_tbl dataframe, but it's never referred to after it's defined.
You'll need to
test_tbl.to_sql("test", connection)
or similar to actually write it.
A minimal example:
$ createdb so63284022
$ python
>>> import sqlalchemy as sa
>>> import pandas as pd
>>> test = pd.DataFrame({'A':[1,1],'B':[2,2], 'C': ['yes', 'hello']})
>>> engine = sa.create_engine("postgres://localhost/so63284022")
>>> with engine.connect() as connection:
... test.to_sql("test", connection)
...
>>>
$ psql so63284022
so63284022=# select * from test;
index | A | B | C
-------+---+---+-------
0 | 1 | 2 | yes
1 | 1 | 2 | hello
(2 rows)
so63284022=# \d+ test
Table "public.test"
Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
--------+--------+-----------+----------+---------+----------+--------------+-------------
index | bigint | | | | plain | |
A | bigint | | | | plain | |
B | bigint | | | | plain | |
C | text | | | | extended | |
Indexes:
"ix_test_index" btree (index)
Access method: heap
so63284022=#
I was able to solve this:
As it was pointed out by #AKX, I was only creating the table structure, but I was not filling in the table.
I now import import psycopg2.extras as well and, after this:
query='CREATE TABLE test (A varchar NOT NULL,B varchar NOT NULL);'
cursor.execute(query)
I add something like:
update_query='INSERT INTO test(A, B) VALUES(%s,%s) ON CONFLICT DO NOTHING'
psycopg2.extras.execute_batch(cursor, update_query, test.values)
cursor.close()
connection.close()
My table is now correctly filled after checking with pd.read_sql
Consider a table users of 6 rows
+_______________________+
| userid | name |
+-----------------------+
| 1 | john |
| 2 | steve |
| 3 | joe |
| 4 | jason |
| 5 | abraham |
| 6 | leonard |
+-----------------------+
I am using the below SQL query:
SELECT userid,name FROM users where userid IN (2,3,4,5);
which returns 4 rows -
| 2 | steve |
| 3 | joe |
| 4 | jason |
| 5 | abraham |
Pymysql equivalent code is as below:
def get_username(user_ids):
data=[]
conn = init_db()
cur = conn.cursor(pymysql.cursors.DictCursor)
cur.executemany("SELECT userid,name from users WHERE userid IN (%s)",user_ids)
rows=cur.fetchall()
for row in rows:
data.append([row['userid'],row['name']])
cur.close()
conn.close()
return data
user_ids=[2,3,4,5]
get_usernames(user_ids)
This code just returns the last row [[5,abraham]] . How can I fetch all those rows?.
That's the (partly documented) behaviour of .executemany():
Help on method executemany in module pymysql.cursors:
executemany(self, query, args) method of pymysql.cursors.Cursor instance
Run several data against one query
:param query: query to execute on server
:param args: Sequence of sequences or mappings. It is used as parameter.
:return: Number of rows affected, if any.
This method improves performance on multiple-row INSERT and
REPLACE. Otherwise it is equivalent to looping over args with
execute().
So what you want here is cursor.execute() - but then, you have a bit more work to build your SQL query:
user_ids = (2, 3, 4, 5)
placeholders = ", ".join(["%s"] * len(user_ids))
sql = "SELECT userid,name from users WHERE userid IN ({})".format(placeholders)
cursor.execute(sql, user_ids)
data = list(cursor)
Note that cursors are iterables, so you don't need to explicitely call cursor.fetchall() then iterate on the result, you can iterate directly on the cursor. Also note that if you want a list of (id, name) tuples, using a DictCursor is just a double waste of CPU cycles (once for building the dicts and once for rebuilding tuples out of them), you could just use a default cursor and return list(cursor) instead.
My first guess is that is something related SELECT statement.
May you try this way of generating the query?
def get_username(user_ids):
data=[]
conn = init_db()
cur = conn.cursor(pymysql.cursors.DictCursor)
cur.executemany("SELECT userid,name from users WHERE userid IN "+"("+','.join(str(e) for e in user_ids)+")")
rows=cur.fetchall()
for row in rows:
data.append([row['userid'],row['name']])
cur.close()
conn.close()
return data
user_ids=[2,3,4,5]
get_usernames(user_ids)
I'm using petl and trying to figure out how to insert a value into a specific row.
I have a table that looks like this:
+----------------+---------+------------+
| Cambridge Data | IRR | Price List |
+================+=========+============+
| '3/31/1989' | '4.37%' | |
+----------------+---------+------------+
| '4/30/1989' | '5.35%' | |
+----------------+---------+------------+
I want to set the price list to 100 on the row where Cambridge Data is 4/30/1989. This is what I have so far:
def insert_initial_price(self, table):
import petl as etl
initial_price_row = etl.select(table, 'Cambridge Data', lambda v: v == '3/31/1989')
That selects the row I need to insert 100 into, but i'm unsure how to insert it. petl doesn't seem to have an "insert value" function.
I would advice not to use select.
To update the value of a field use convert.
See the docs with many examples: https://petl.readthedocs.io/en/stable/transform.html#petl.transform.conversions.convert
I have not tested it, but this should solve it:
import petl as etl
table2 = etl.convert(
table,
'Price List',
100,
where = lambda rec: rec["Cambridge Data"] == '4/30/1989',
)
I'm using mysql connector 1.0.9. and Python 3.2
This query fails due to a syntax error (mysql.connector throws ProgrammingError, the specific MySQL error is just "there is a syntax error to the right of "%(IP)s AND DATE_SUB(NOW(), INTERVAL 1 HOUR) < accessed":
SELECT COUNT(*) FROM bad_ip_logins WHERE IP = %(IP)s AND DATE_SUB(NOW(), INTERVAL 1 HOUR) < accessed
But if I quote the variable IP, it works:
SELECT COUNT(*) FROM bad_ip_logins WHERE IP = '%(IP)s' AND DATE_SUB(NOW(), INTERVAL 1 HOUR) < accessed
In context:
IP = 1249764151 # IP converted to an int
conn = mysql.connector.connect(db_params)
curs = conn.cursor()
query = "SELECT COUNT(*) FROM bad_ip_logins WHERE IP = %(IP)s AND DATE_SUB(NOW(), INTERVAL 1 HOUR) < accessed"
params = {'IP', IP}
curs.execute(query, params)
My understanding is that you never have to quote variables for a prepared statement (and this is true for every other query in my code, even ones that access the IP variable on this table). Why do I need to quote it in this single instance, and nowhere else?
If this isn't doing a prepared statement I'd be interested in hearing about that as well. I wasn't able to inject anything with this - was it just quoting it in such a way as to prevent that?
If it matters, this is the table description:
+----------+------------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+----------+------------------+------+-----+---------+-------+
| IP | int(10) unsigned | YES | | NULL | |
| user_id | int(11) | YES | | NULL | |
| accessed | datetime | YES | | NULL | |
+----------+------------------+------+-----+---------+-------+
Do not use string interpolation. Leave the SQL parameter to the database adapter:
cursor.execute('''\
SELECT COUNT(*) FROM bad_ip_logins WHERE IP = %s AND DATE_SUB(NOW(), INTERVAL 1 HOUR) < accessed''', (ip,))
Here, we pass the parameter ip in to the execute() call as a separate parameter (in a tuple, to make it a sequence), and the database adapter will take care of proper quoting, filling in the %s placeholder.