After saving some data in a variable with cursor.fetchall(), it looks as follows:
mylist = [('abc1',), ('abc2',)] this is apparently a list.
That is not the issue.
The problem is that the following doesn't work:
if 'abc1' in mylist
it can't find 'abc1'. Is there a way in Python to do it easily or do I have to use a loop?
fetchall() returns a row list, i.e., a list containing rows.
Each row is a tuple containing column values. There is a tuple even when there is only one column.
To check whether a row is in the row list, you have to check for the row instead of the column value alone:
if ('abc1',) in mylist
This is problem with using select * statement.
Instead use select col1,col2 from table_name
Below code might help
sql = "select col1,col2,col3 from table_name"
cursor.execute(sql) # initialize cursor in your way
input_dict = dict( (row[0],(row[1],row[2])) for row in cursor.fetchall() )
Related
When I run a SQLAlchemy query to get values from a SQLAlchemy.String() column;
query = db.session.query(self.hosts_class.cluster.distinct())
for row in query.all()
Iterating and printing the result object results in output like;
('sometext',)
If I browse the row object I can see that row contains a key, value pair of "0": "sometext". What's the correct way to access this value from row?
row is a single-element tuple, so you could access by index:
for row in query:
print(row[0])
or by unpacking the tuple
for row, in query: # <- note the comma
print(row)
I am defining a function here and making a query.
def fetch(temp_pass,temp_accno):
cur.execute('''SELECT id, name, acc_no, ph_no,address, email,balance
FROM accounts
WHERE id = %s and acc_no = %s''',
(str(temp_pass), str(temp_accno)));
row = cur.fetchall()
print(row[2])
In this row should be a list of length 7 but when I run print(row[2])
it gives me error that list index out of range.
This is the error I get
File "Accounting.py", line 13, in fetch
print(row[2])
IndexError: list index out of range
row = cur.fetchall() won't give you a row but a list of rows, so row is not a row at all and row[2] is the third row in the list, not the third field in a row. If you want only a row use cur.fetchone().
Note the the query might return several rows and it is not clear what you want to do in that case so I won't deal with it here. cur.fetchone() will give you only one row anyway.
row[2] returns the 3rd row of the list of rows. row[0][2] returns the 3rd column of the 1st row.
You could run a snippet like this to visualize what gets returned:
cur.execute(...)
for row in cur:
print(row)
I'm trying to import the results of a complex SQL query into a pandas dataframe. My query requires me to create several temporary tables since the final result table I want includes some aggregates.
My code looks like this:
cnxn = pyodbc.connect(r'DRIVER=foo;SERVER=bar;etc')
cursor = cnxn.cursor()
cursor.execute('SQL QUERY HERE')
cursor.execute('SECONDARY SQL QUERY HERE')
...
df = pd.DataFrame(cursor.fetchall(),columns = [desc[0] for desc in cursor.description])
I get an error that tells me shapes aren't matching:
ValueError: Shape of passed values is (1,900000),indices imply (5,900000)
And indeed, the result of all the SQL queries should be a table with 5 columns rather than 1. I've run the SQL query using Microsoft SQL Server Management Studio and it works and returns the 5 column table that I want. I've tried to not pass any column names into the dataframe and printed out the head of the dataframe and found that pandas has put all the information in 5 columns into 1. The values in each row is a list of 5 values separated by commas, but pandas treats the entire list as 1 column. Why is pandas doing this? I've also tried going the pd.read_sql route but I still get the same error.
EDIT:
I have done some more debugging, taking the comments into account. The issue doesn't appear to stem from the fact that my query is nested. I tried a simple (one line) query to return a 3 column table and I still got the same error. Printing out fetchall() looks like this:
[(str1,str2,str3,datetime.date(stuff),datetime.date(stuff)),
(str1,str2,str3,datetime.date(stuff),datetime.date(stuff)),...]
Use pd.DataFrame.from_records instead:
df = pd.DataFrame.from_records(cursor.fetchall(),
columns = [desc[0] for desc in cursor.description])
Simply adjust the pd.DataFrame() call as right now cursor.fetchall() returns one-length list of tuples. Use tuple() or list to map child elements into their own columns:
df = pd.DataFrame([tuple(row) for row in cur.fetchall()],
columns = [desc[0] for desc in cursor.description])
I have searched the docs and SO and could not find anything to resolve my issue. I am trying to call a select from my sqlite database and add it to a dictionary with the columns as keys. When I do this it returns a row for each column/key. It is has 14 columns and if I only have 4 rows it repeats for each one. This was the first attempt
columns = [desc[0] for desc in cursor.description]
results = []
for row in r:
Summary = {}
items = zip(columns, row)
for (items, values) in items:
Summary[items] = row
results.append(Summary)
Then I also tried the row_factory as given in the docs. That didn't work. My end goal is to be able to print out to a text file verticly by using
for x in results:
print x[name]
print x[email]
etc
Any help is appreciated
You are creating your dictionary incorrectly. Use:
for row in r:
summary = dict(zip(columns, row))
results.append(summary)
instead.
Your code sets the whole row sequence as the value for each key in Summary, instead of the individual column value, then appending that same dictionary to the results list for each column key..
I'm attempting to write the results of a regression back to MySQL, but am having problems iterating through the fitted values and getting the NaNs to write as null values. Originally, I did the iteration this way:
for i in dataframe:
cur = cnx.cursor()
query = ("UPDATE Regression_Data.Input SET FITTEDVALUES="+(dataframe['yhat'].__str__())+" where timecount="+(datafrane['timecount'].__str__())+";")
cur.execute(query)
cnx.commit()
cur.close()
.....which SQL thew back to me by saying:
"mysql.connector.errors.ProgrammingError: 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'NaN'
So, I've been trying to filter out the NaNs by only asking Python to commit when yhat does not equal NaN:
for i in dataframe:
if cleandf['yhat']>(-1000):
cur = cnx.cursor()
query = ("UPDATE Regression_Data.Input SET FITTEDVALUES="+(dataframe['yhat'].__str__())+" where timecount="+(datafrane['timecount'].__str__())+";")
cur.execute(query)
cnx.commit()
cur.close()
But then I get this:
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
So, I try to get around it with this in my above syntax:
if cleandf['yhat'][i]>(-1000):
but then get this:
ValueError: Can only tuple-index with a MultiIndex
And then tried adding itterows() to both as in:
for i in dataframe.iterrows():
if cleandf['yhat'][i]>(-1000):
but get the same problems as above.
I'm not sure what I'm doing wrong here, but assume it's something with iterating in Pandas DataFrames. But, even if I got the iteration right, I would want to write Nulls into SQL where the NaN appeared.
So, how do you think I should do this?
I don't have a complete answer, but perhaps I have some tips that might help. I believe you are thinking of your dataframe as an object similar to a SQL record set.
for i in dataframe
This will iterate over the column name strings in the dataframe. i will take on column names, not rows.
dataframe['yhat']
This returns an entire column (pandas.Series, which is a numpy.ndarray), not a single value. Therefore:
dataframe['yhat'].__str__()
will give a string representation of an entire column that is useful for humans to read. It is certainly not a single value that can be converted to string for your query.
if cleandf['yhat']>(-1000)
This gives an error, because again, cleandf['yhat'] is an entire array of values, not just a single value. Think of it as an entire column, not the value from a single row.
if cleandf['yhat'][i]>(-1000):
This is getting closer, but you really want i to be an integer here, not another column name.
for i in dataframe.iterrows():
if cleandf['yhat'][i]>(-1000):
Using iterrows seems like the right thing for you. However, i takes on the value of each row, not an integer that can index into a column (cleandf['yhat'] is a full column).
Also, note that pandas has better ways to check for missing values than relying on a huge negative number. Try something like this:
non_missing_index = pandas.isnull(dataframe['yhat'])
cleandf = dataframe[non_missing_index]
for row in cleandf.iterrows():
row_index, row_values = row
query = ("UPDATE Regression_Data.Input SET FITTEDVALUES="+(row_values['yhat'].__str__())+" where timecount="+(row_values['timecount'].__str__())+";")
execute_my_query(query)
You can implement execute_my_query better than I can, I expect. However, this solution is not quite what you want. You really want to iterate over all rows and do two types of inserts. Try this:
for row in dataframe.iterrows():
row_index, row_values = row
if pandas.isnull(row_values['yhat']):
pass # populate the 'null' insert query here
else:
query = ("UPDATE Regression_Data.Input SET FITTEDVALUES="+(row_values['yhat'].__str__())+" where timecount="+(row_values['timecount'].__str__())+";")
execute_my_query(query)
Hope it helps.