I am new to python and mongo db, i have a dict called data and i wanna use the keys to query the database called db.
def update_db(data, db):
for key in data:
x=db.find({'label':key})
for a in x:
print a
I get the error message that find() does not exist.
Anyone can give me input on my problem ?
There is no find method in database object. You should search documents in some collection, not in database. Database has collections like SQL database has tables. Collections have documents, like SQL tables have rows of data. E.g. if you have users collection:
def update_db(data, db):
for key in data:
users = db.users
matchedUsers = users.find({'label':key})
for user in matchedUsers:
print user
Future reading PyMongo tutorial
Related
I'm very new to py programming; please excuse if its a silly question.
I need to perform two actions
Extract json file from a Rest API (which will have a very large number of Key/Value pairs)
Pass the extract in a tabular format to a SQL Server
I have written a sample function consisting of only 03 parameters which are passed on to a SQL Server on my system.
How will this function change if there are unknown number of parameters as in the case of a json extract?
def InsertJsonsql(Name, City, Age):
connection = pyodbc.connect('Driver={SQL Server};'
'Server=IN2367403W1\SQLEXPRESS;' #IN2367403W1\SQLEXPRESS : use server name
'Database=TestDB;'
'Trusted_Connection=yes;')
cursor = connection.cursor()
json_insert_query = """
INSERT INTO TestDB.dbo.Person (Name, City, Age) VALUES ('{}', '{}', '{}')
""".format(Name, City, Age)
cursor.execute(json_insert_query)
connection.commit()
print("Record inserted successfully into Person table")
As mentioned from #shimo in the comments a relational database like mysql or Windows SQL are fixed when created. So you can't add new columns to handle unknown number of parameters.
As far as I know you have 2 options:
1.) Use a NoSQL database like MongoDB. There the data is saved as JSON and therefore it can handle any number of key-value pairs. This is the safest and best method.
2.) If you have to use a relational database you may have to create a column in your table that can handle a long String and save your data as JSON String in this column. This is not a good solution and you should rather use a NoSQL database.
I'm connecting to an Oracle database from sqlalchemy and I want to know when the tables in the database were created. I can access this information through the sql developer application so I know that it is stored somewhere, but I don't know if its possible to get this information from sqlalchemy.
Also if its not possible, how should I be getting it?
SqlAlchemy doesn't provide anything to help you get that information. You have to query the database yourself.
something like:
with engine.begin() as c:
result = c.execute("""
SELECT created
FROM dba_objects
WHERE object_name = <<your table name>>
AND object_type = 'TABLE'
""")
I have been given a task to migrate data from SQL SERVER to Oracle database. The structure of those two databases is diferent and each table has at least 20 columns and there are at least 500 tables to migrate. I need your recomendation to choose good tools for this task. My initial choice for this task is sqlalchemy. There structure of databases is already there and is different. I can't edit tables.
Lets see some simplified examples. Lets say that I have a tables Person and PersonAdress in SQL Server. It has fields like name, name2, surname and keeps person_address_id as foreign key, birth_date.
In Oracle database I have a model PERSON_DATA and PERSON_ADDRES where I have fields like NAME, NAME2, SUERNAME, DATE_OF_BIRTH. The tablePERSON_ADRESSkeeps thePERSON_DATA_IDasforeign key`.
So there are two problems:
1)Name of fieelds can be different(birth_date in SQL Server and DATE_OF_BIRTH in ORACLE) but represents the same concept.
2)Data that represent the same concept may be in different tables(person_address_id is a foreign key in table Person in SQL_SERVER where PERSON_ADRESS keeps the PERSON_DATA_ID as `foreign key in Oracle).
I though about using from automap_base from sqlalchemy.ext.automap to query models. But i dont know it it is the best idea. This is my pseudo code snippet. Maybe It would help to show what I have tried to do so far.. Please note that this is more pseudo-code.
from sqlalchemy import create_engine, MetaData
from sqlalchemy.ext.automap import automap_base
class PersonModel:
# DO i need this model? It suppose to represent Person table in
# SQL SERVER
def __init__(self, name, name2, birth_date, ...):
self.name = name
self.name2 = name2
...
class PERSON_DATAModel:
# Do I need this model? It supose to represent table in Oracle
def __init__(self, NAME, NAME2, DATE_OF_BIRTH, ...):
self.NAME = NAME
self.NAME2 = NAME2
metadata = MetaData()
sql_server_engine = create_engine('..')
metadata.reflect(sql_server_engine)
BaseSQLSERVER= automap_base(metadata=metadata)
BaseSQLSERVER.prepare()
metadata.reflect(sql_server_engine, only=['Person'])
BaseSQLSERVER.prepare()
Person = BaseSQLSERVER.classes.DaneOsMain
# The same what is above for Oracle
oracle_server_engine = create_engine('..')
# and so on..
# to get
PERSON_DATA = BaseOracle.classes.PERSON_DATA
session = Session(sql_server_engine)
query_data = sesion.query(Person)...
PersonTable = []
for data in query_data:
e = PersonModel(**query_data.__dict__)
PersonTable.append(e)
# I have to push data from PersonTable somehow to PERSON_DATA table
Do you know good paterns to solve this problem? I would like to know good deisgn patern to solve this problem and useful python tools for this.
If this is a one-time conversion it might be easier to:
1) Create a conversion Stored Procedure in SQL Server that will output the various tables as csv flat files in the format the Oracle tables are expecting. You can name the files as the destination table in Oracle.
2) Write a python script to loop through each file in the folder you have outputed the files in from 1. Bulk Load each flat file into destination table from Oracle.
For 2, you will need to disable the constraints and also provide error checking in case the file fails to load. In the script or the flat file naming convention you can include a record count for sanity checking.
Currently I am trying to retrieve JSON data from an API and store it in a database. I am able to retrieve the JSON data as a list, and I am able to connect to and query the DB2 Database. My issue is that I can not figure out how to generate an INSERT statement for the data retrieved from the API. The application is only for short term personal use, so SQL Injection attacks are not a concern. So overall I need to generate an sql insert statement from a list. My current code is below, with the api url and info changed.
import ibm_db
import requests
ibm_db_conn = ibm_db.connect("DATABASE=node1;HOSTNAME=100.100.100.100;PORT=50000;PROTOCOL=TCPIP;UID=username;PWD=password;", "", "")
api_request = requests.get("http://api-url/resource?api_key=123456",
auth=('user#api.com', 'password'))
api_code = api_request.status_code
api_data = api_request.json()
print(api_code)
print(api_data)
Depends on the format of the Json returned, and on what your table looks like. My first thought, though, is to use Python's json module:
import json
#...
#...
api_data = json.loads(api_request.json())
Now, you have a Python object you can access like normal:
api_data["key"][2]
for instance. You can itterate, slice, or do whatever else to extract the data you want. Say your json represented rows to be inserted:
query = "INSERT INTO <table> VALUES\n"
i = 0
for row in api_data:
query += "(%s)" %([i for i in row])
if i < len(api_data)-1: query += ",\n"
i += 1
Again, this will vary greatly depending on the format of your table and JSON, but that's the general idea I'd start with.
I need to perform a raw sql on multiple tables. I then render the result set. For one table I would do:
sql = "select * from my_table"
results = my_table.objects.raw(sql)
For multiple tables I am doing:
sql = "select * from my_table, my_other_table where ...."
results = big_model.objects.raw(sql)
But, do I really need to create a table/model/class big_model, which contains all fields that I may need? I will never actually store any data in this "table".
ADDED:
I have a table my_users. I have a table my_listings. These are defined in Models.py. The table my_listings has a foreign key to my_users, indicating who created the listing.
The SQL is
"select user_name, listing_text from my_listings, my_users where my_users.id = my_listings.my_user_id".
I want this SQL to generate a result set that I can use to render my page in django.
The question is: Do I have to create a model that contains the fields user_name and listing_text? Or is there some better way that still uses raw SQL (select, from, where)? Of course, my actual queries are more complicated than this example. (The models that I define in models.py become actual tables in the database hence the use of the model/table term. Not sure how else to refer to them, sorry.) I use raw sql because I found that python table references only work with simple data models.
This works. Don't know why it didn't before :( From Dennis Baker's comment:
You do NOT need to have a model with all the fields in it, you just need the first model and fields from that. You do need to have the fields with unique names and as far as I know you should use "tablename.field as fieldname" to make sure you have all unique fields. I've done some fairly complex queries with 5+ tables this way and always tie them back to a single model. –
2 . Another solution is to use a cursor. However, a cursor has to be changed from a list of tuples to a list of dictionaries. I'm sure there are cleaner ways using iterators, but this function works. It takes a string, which is the raw sql query, and returns a list which can be rendered and used in a template.
from django.db import connection, transaction
def sql_select(sql):
cursor = connection.cursor()
cursor.execute(sql)
results = cursor.fetchall()
list = []
i = 0
for row in results:
dict = {}
field = 0
while True:
try:
dict[cursor.description[field][0]] = str(results[i][field])
field = field +1
except IndexError as e:
break
i = i + 1
list.append(dict)
return list
you do not need a model that includes the fields that you want to return from your raw sql. If you happen to have a model that actually has the fields that you want to return from your raw sql then you can map your raw sql output to this model, otherwise you can use cursors to go around models altogether.