I have a function (remote database, without models in my app), I am making a request to it. Is it possible to get the column names using query rather than execute?
session = Session(bind=engine)
data = session.query(func.schema.func_name())
I am getting an array of strings with values, how do I get the keys? I want to generate a dict.
When I make a request with an execute, the dictionary is generated fine.
data = session.execute("select * from schema.func_name()")
result = [dict(row) for row in data]
You can do something like:
keys = session.execute("select * from schema.func_name()").keys()
Or try accessing it after the query:
data = session.query(func.schema.func_name()).all()
data[0].keys()
You can also use: data.column_descriptions
Documention:
https://docs.sqlalchemy.org/en/14/orm/query.html
Related
I want to store the JSON I get from an API, but only get the JSON of the last loop. How to get the lists dynamic? Also I need to use the last query (Pandas) but it's not working.
Last how to make an API to :
List latest forecast for each location for every day.
List average the_temp of last 3 forecasts for each location for every day.
Get the top n locations based on each available metric where n is a parameter given in the API call.
import requests
import json
import sqlite3
import pandas as pd #library for data frame
print(sqlite3.sqlite_version)
for x in range(20,28): # i need to get LONDjson/BERLjson/SANjson lists dynamic to store bot 7 jsons from each urls
r = requests.get('https://www.metaweather.com/api/location/44418/2021/4/'+str(x)+'/') #GET request from the source url
LONDjson=r.json() #JSON object of the result
r2 = requests.get('https://www.metaweather.com//api/location/2487956/2021/4/'+str(x)+'/')
SANjson=r2.json()
r3 = requests.get('https://www.metaweather.com//api/location/638242/2021/4/'+str(x)+'/')
BERLjson=r3.json()
conn= sqlite3.connect('D:\weatherdb.db') #create db in path
cursor = conn.cursor()
#import pprint
#pprint.pprint(LONDjson)
cursor.executescript('''
DROP TABLE IF EXISTS LONDjson;
DROP TABLE IF EXISTS SANjson;
DROP TABLE IF EXISTS BERLjson;
CREATE TABLE LONDjson (id int, data json);
''');
for LOND in LONDjson:
cursor.execute("insert into LONDjson values (?, ?)",
[LOND['id'], json.dumps(LOND)])
conn.commit()
z=cursor.execute('''select json_extract(data, '$.id', '$.the_temp', '$.weather_state_name', '$.applicable_date' ) from LONDjson;
''').fetchall() #query the data
hint: in your initial for loop you are not storing the results of api call. you are storing in variable but that is just getting re-written each loop.
a common solution for this starting with empty list that you append to. where perhaps if storing mutliple variables you are storing a dictionary as elements of list
example
results = []
for x in range(10):
results.append(
{
'x': x,
'x_sqaured': x*x,
'abs_x': abs(x)
}
)
print(results)
It looks like there's at least two things that can be improved in the data manipulation part of your code.
Using an array to store the retrieved data
LONDjson = []
SANjson = []
BERLjson = []
for x in range(20,28):
r = requests.get('https://www.metaweather.com/api/location/44418/2021/4/'+str(x)+'/')
LONDjson.append(r.json())
r2 = requests.get('https://www.metaweather.com//api/location/2487956/2021/4/'+str(x)+'/')
SANjson.append(r2.json())
r3 = requests.get('https://www.metaweather.com//api/location/638242/2021/4/'+str(x)+'/')
BERLjson.append(r3.json())
Retrieving the data from the array
# The retrieved data is a dictionary inside a list with only one entry
for LOND in LONDjson:
print(LOND[0]['id'])
Hope this helps you out.
Am trying to use petl library to build an ETL process that copied data between two tables. The table contain a unique slug field on the destination. For that, I wrote my script so It would identify duplicate slugs and convert them with by appending ID to the slug value.
table = etl.fromdb(source_con, 'SELECT * FROM user')
# get whatever remains as duplicates
duplicates = etl.duplicates(table, 'slug')
for dup in [i for i in duplicates.values('id')]:
table = etl.convert(
table,
'slug',
lambda v, row: '{}-{}'.format(slugify_unicode(v), str(row.id).encode('hex')),
where=lambda row: row.id == dup,
pass_row=True
)
The above did not work as expected, it seems like the table object remains with duplicate values after the loop.
Anyone can advise?
Thanks
Im using psycopg2 to access postgres database using the below query. In order to return a dictionary from the executed query, im using DictCursor in my cursor but still my output is a list and not a dictonary.
Here is the program and output below.
import psycopg2.extras
try:
conn = psycopg2.connect("user='postgres' host='localhost' password='postgres'",
)
except:
print "I am unable to connect to the database"
cur = conn.cursor(cursor_factory=psycopg2.extras.DictCursor)
cur.execute("""SELECT datname from pg_database""")
rows = cur.fetchall()
print "\nShow me the databases:\n"
print rows
Output:-
[['template1'], ['template0'], ['postgres'], ['iip'], ['test'], ['test_postgres'], ['testdb']]
It looks like a list, smells like a list, but it's a DictRow.
rows = cur.fetchall()
for row in rows :
print(type(row))
#>>> <class 'psycopg2.extras.DictRow'>
This means that you can still use the column names as keys to access the data :
rows = cur.fetchall()
print([row['datname'] for row in rows])
This class inherits directly from the builtinlist and add all the needed methods to implement a dictionary logic, but it doesn't change the representation __repr__ or __str__, so the output is the same as a list.
class DictRow(list):
"""A row object that allow by-column-name access to data."""
fetchall() packs all the queried rows in a list without specifying the exact type.
Btw, maybe you are looking for this kind of cursor : RealDictCursor ?
For those who came where because they really like the easy reference of the dictionary for column:value record representation, the answer by PRMoureu which notes that the DictRow has all the usual dictionary logic means that you can iterate over the DictRow with .items() and get the key:value pairs.
rows = cur.fetchall()
row_dict = [{k:v for k, v in record.items()} for record in rows]
Will turn your list of DictRow records into a list of dict records.
I have the following code that allows me to retrieve the first keyspace:
def Query(str):
auth_provider = PlainTextAuthProvider(username='admin', password='root')
cluster = Cluster(['hostname'], auth_provider=auth_provider)
session = cluster.connect('system')
rows = session.execute(str)
keyspaces = []
row_list = list(rows)
for x in range(len(row_list)):
return row_list[0]
#app.route('/keyspaces')
def all():
return Query('select json * from schema_keyspaces')
I would like not only get all the keyspaces, but also their attributes and that in JSON document, how I can proceed ?
Thanks,
Instead of a loop that only runs once, you need to collect all the elements
rows = session.execute(str)
return jsonify(list(rows))
Note that you should ideally not be creating a new cassandra connection for each query you need to make, but that's unrelated to the current problem
I'm using this below in views.py to get app
from django.db import connection
def test(request):
cursor = connection.cursor()
sql = """
SELECT x , n
from table1 limit 10
"""
cursor.execute(sql)
rows = cursor.fetchall()
# df1 = pd.read_sql_query(sql,cursor) <==> not working )
# df1.columns = cursor.keys() <==> not working )
return render(request, 'app/test.html',{ "row" : rows,})
I am able to print row and got a list of something like this below in test.html
row((x1,yvalue1),(x2,yvalue2) , .... ))
But what I'm trying to do is to get all data with its column name that I fetched and put into dataframe , hopefully to use something like this below :
http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_sql_query.html#pandas.read_sql_query
I think aus_lacy is a bit off in his solution - first you have to convert the QuerySet to a string containing the SQL backing the QuerySet
from django.db import connection
query = str(ModelToRetrive.objects.all().query)
df = pandas.read_sql_query(query, connection)
Also there is a less memory efficient but still valid solution:
df = DataFrame(list(ModelToRetrive.objects.values('id','some_attribute_1','some_attribute_2')))
You need to use Django's built in QuerySet API. More information on it can be seen here. Once you create a QuerySet you can then use pandas read_sql_query method to construct the data frame. The simplest way to construct a QuerySet is simply query the entire database which can be done like so:
db_query = YourModel.objects.all()
You can use filters which are passed in as args when querying the database to create different QuerySet objects depending on what your needs are.
Then using pandas you could do something like:
d_frame = pandas.read_sql_query(db_query, other_args...)