I scrape a list of elements from the site and extract two values from it. The text and the href. I cannot figure out how to add these to the mysql DB in side a loop. I tried execute inside the loop, without using the list they are appeneded to, that failed. So, I tried executemany but I think my format might be incorrect. I saw examples where a list of tuples is fed to executemany, I don't know how to do that in this case.
for name in name_eles:
names_list.append(name.text)
n_li = name.get_attribute('href')
names_links.append(n_li)
sql = "INSERT INTO profiles(company, coprofile) VALUES (%s,%s)"
val = [(name for name in name_list),(n_li for n_li in names_links)]
cursor.executemany(sql,val)
This is the error I get.
MySQLdb._exceptions.ProgrammingError: not all arguments converted during bytes formatting
You have to pass a list of tuples in the executemany arguments.
You can simply try this:
val = list(zip(name_list, names_links))
You can read more about it here
Related
I am trying to use pyodbc to update an existing MS Access database table with a very long multiline string. The string is actually a csv that has been turned into a string.
The query I am trying to use to update the table is as follows:
query = """
UPDATE Stuff
SET Results = '{}'
WHERE AnalyteName =
'{}'
""".format(df, analytename)
The full printed statement looks as follows:
UPDATE Stuff
SET Results =
'col a,col b,col c,...,col z,
Row 1,a1,b1,c1,
...,...,...,...,
Row 3000,a3000,b3000,c3000'
WHERE AnalyteName = 'Serotonin'
However this does not seem to be working, and I keep getting the following error:
pyodbc.ProgrammingError: ('42000', '[42000] [Microsoft][ODBC Microsoft Access Driver] Syntax error in UPDATE statement. (-3503) (SQLExecDirectW)')
Which I assume is due to the format of the csv string I am trying to use to update the table with.
I have tried using INSERT and inserting a new row with the csv string and other relevant information and that seems to work. However, I need to use UPDATE as I will eventually be adding other csv strings to these columns. This leads me to believe that there is A) Something is wrong with the syntax of my UPDATE query (I am new to SQL syntax) or B) I am missing something from the documentation regarding UPDATE queries.
Is executing an UPDATE query like this possible? If so, where am I going wrong?
It would be determined by the table's field type.
For large amounts of text you'd need a blob field in your database table.
A blob field will store binary info so using blob will not 'see' illegal characters.
Answering my own question in case anyone else wants to use this.
It turns out what I was missing was brackets around the table column fields from my UPDATE statement. My final code looked something like this.
csv = df.to_csv(index=False)
name = 'some_name'
query = """
UPDATE Stuff
SET
[Results] = ?
WHERE
[AnalyteName] = ?
"""
self.cursor.execute(query, (csv, name))
I've seen several other posts here where brackets were not around the column names. However, since this is MS Access, I believe they were required for this query, or rather this specific query since it included a very long strong in the SET statement.
I welcome anyone else here to provide a more efficient method of performing this task or someone else who can provide more insight into why this is what worked for me.
I have been working with an SQL database through the mySQL library in Python. I have recently found that when I try searching for a string in my database, it is not correctly returning the output I expect. I think this may be due to my variable not being properly inserted into my SQL command.
code = data['code']
sql = "SELECT 200 FROM mytable WHERE mycolumn = '%s';"
query.execute(sql, teamCode)
print(str(query.fetchall()))
My problem is that printing query.fetchall() prints an empty list ([]) instead of the expected [[200,]] which means the program thinks the code value it is using does not exist in the SQL database, which it does.
The parameters in an execute call need to be a sequence, like a tuple. Try:
query.excute(sql, (teamCode,))
That turns it into a one-element tuple. BTW, did you really mean "code" there?
Is there a way to return all strings from a specific column in a MySQL database?
Note: I want to save those strings in a list, I'm also using pymysql.
Those are the steps you need to follow (in words as well since you do not provide any code either):
Establish a connection with the database
Create a cursor
With the cursor you created execute a query which should be type of "SELECT column_that_you_want FROM table_you_want"
The cursor now will hold the results.
You can add the results in a list via a loop for example. Typically those results will be a one item tuple.
I was trying to query a database based on some pre selected items and ran into a weird situation. I started with pre selecting some parameters that I would like use as filter in a query from one of the tables in the database:
MX_noaa_numbers = list(Events_df[Events_df['flareclass'].str.contains('M|X')].noaanumber.unique())
Which produces a list such as:
[11583,11611,11771,11777,11778,11865,12253,11967,11968,...,12673]
But when I tried to obtain the results using:
session.query(ActiveRegion).filter(sql.or_(ActiveRegion.noaa_number1.in_(MX_noaa_numbers),
ActiveRegion.noaa_number2.in_(MX_noaa_numbers),
ActiveRegion.noaa_number3.in_(MX_noaa_numbers))).all()
it returns me an empty list. However if I print MX_noaa_numbers and copy the output inside the in_() statement substituting the object name (MX_noaa_numbers) I actually get the results as I should. Am I missing something or I actually ran into some weird error?
Thanks!
This question already has answers here:
imploding a list for use in a python MySQLDB IN clause
(8 answers)
Closed 1 year ago.
I want to insert a list in my database but I can't.
Here is an example of what I need:
variable_1 = "HELLO"
variable_2 = "ADIOS"
list = [variable_1,variable_2]
INSERT INTO table VALUES ('%s') % list
Can something like this be done? Can I insert a list as a value?
When I try it, an error says that is because of an error in MySQL syntax
The answer to your original question is: No, you can't insert a list like that.
However, with some tweaking, you could make that code work by using %r and passing in a tuple:
variable_1 = "HELLO"
variable_2 = "ADIOS"
varlist = [variable_1, variable_2]
print "INSERT INTO table VALUES %r;" % (tuple(varlist),)
Unfortunately, that style of variable insertion leaves your code vulnerable to SQL injection attacks.
Instead, we recommend using Python's DB API and building a customized query string with multiple question marks for the data to be inserted:
variable_1 = "HELLO"
variable_2 = "ADIOS"
varlist = [variable_1,variable_2]
var_string = ', '.join('?' * len(varlist))
query_string = 'INSERT INTO table VALUES (%s);' % var_string
cursor.execute(query_string, varlist)
The example at the beginning of the SQLite3 docs shows how to pass arguments using the question marks and it explains why they are necessary (essentially, it assures correct quoting of your variables).
Your question is not clear.
Do you want to insert the list as a comma-delimited text string into a single column in the database? Or do you want to insert each element into a separate column? Either is possible, but the technique is different.
Insert comma-delimited list into one column:
conn.execute('INSERT INTO table (ColName) VALUES (?);', [','.join(list)])
Insert into separate columns:
params = ['?' for item in list]
sql = 'INSERT INTO table (Col1, Col2. . .) VALUES (%s);' % ','.join(params)
conn.execute(sql, list)
both assuming you have established a connection name conn.
A few other suggestions:
Try to avoid INSERT statements that do not list the names and order of the columns you're inserting into. That kind of statement leads to very fragile code; it breaks if you add, delete, or move columns around in your table.
If you're inserting a comma-separted list into a single-field, that generally violates principals of database design and you should use a separate table with one value per record.
If you're inserting into separate fields and they have names like Word1 and Word2, that is likewise an indication that you should be using a separate table instead.
Never use direct string substitution to create SQL statements. It will break if one of the values is, for example o'clock. It also opens you to attacks by people using SQL injection techniques.
You can use json.dumps to convert a list to json and write the json to db.
For example:
insert table example_table(column_name) values(json.dumps(your_list))