I'm trying to get the values of all databases existing in mongodb, iterate over all databases and collections for than print it documents. I can to print the document passing the collection as a variable, but can`t do it iterating over all databases and collections (as the value of variable). Someone knows if pymongo supports to do it dynamically passing as value and not passing the collection and the database as the variable itself??
client = MongoClient('mongodb://localhost:27017/')
names = client.database_names()
for dbName in names:
print(dbName)
db = client.dbName
collectionNames = client[dbName].collection_names()
for colecao in collectionNames:
print(colecao)
cursor = db.colecao # choosing the collection you need
print(cursor)
cursor2 = cursor.find() # get documents
for document in cursor2:
pprint(document)
The database names and collection names print normally, but the print cursor returns:
"Collection(Database(MongoClient(host=['localhost:27017'], document_class=dict, tz_aware=False, connect=True), u'dbName'), u'colecao')"
It goes with the name of variables.
Instead of
client.dbName
use
client.get_database(dbName)
and instead of
cursor = db.colecao
use
cursor = db.get_collection(colecao)
Related
Currently i'm executing stored procedure that way:
engine = sqlalchemy.create_engine(self.getSql_conn_url())
query = "exec sp_getVariablesList #City = '{0}', #Station='{1}'".format(City, Station)
self.Variables = pd.read_sql_query(query, engine)
but at How set ARITHABORT ON at sqlalchemy was correctly noticed that that make that open to SQL injection. I tried different ways but without success. So how should I pass parameters to the MSSQL stored procedure to eliminate the risk of SQL injection? That can be with sqlalchemy or any other way.
Write your SQL command text using the "named" paramstyle, wrap it in a SQLAlchemy text() object, and pass the parameter values as a dict:
import pandas as pd
import sqlalchemy as sa
connection_uri = "mssql+pyodbc://#mssqlLocal64"
engine = sa.create_engine(connection_uri)
# SQL command text using "named" paramstyle
sql = """
SET NOCOUNT ON;
SET ARITHABORT ON;
EXEC dbo.breakfast #name = :name_param, #food = :food_param;
"""
# parameter values
param_values = {"name_param": "Gord", "food_param": "bacon"}
# execute query wrapped in SQLAlchemy text() object
df = pd.read_sql_query(sa.text(sql), engine, params=param_values)
print(df)
"""
column1
0 Gord likes bacon for breakfast.
"""
I would like to switch on the fast_executemany option for the pyODBC driver while using SQLAlchemy to insert rows to a table. By default it is of and the code runs really slow... Could anyone suggest how to do this?
Edits:
I am using pyODBC 4.0.21 and SQLAlchemy 1.1.13 and a simplified sample of the code I am using are presented below.
import sqlalchemy as sa
def InsertIntoDB(self, tablename, colnames, data, create = False):
"""
Inserts data into given db table
Args:
tablename - name of db table with dbname
colnames - column names to insert to
data - a list of tuples, a tuple per row
"""
# reflect table into a sqlalchemy object
meta = sa.MetaData(bind=self.engine)
reflected_table = sa.Table(tablename, meta, autoload=True)
# prepare an input object for sa.connection.execute
execute_inp = []
for i in data:
execute_inp.append(dict(zip(colnames, i)))
# Insert values
self.connection.execute(reflected_table.insert(),execute_inp)
Try this for pyodbc
crsr = cnxn.cursor()
crsr.fast_executemany = True
Starting with version 1.3, SQLAlchemy has directly supported fast_executemany, e.g.,
engine = create_engine(connection_uri, fast_executemany=True)
I have a list of items which I like to store in my firebird database.
Thus far I made the following code
Sens=278.3
DSens=1.2
Fc10=3.8
Bw10=60.0
Fc20=4.2
Bw20=90.0
ResultArray = (Sens,DSens,Fc10,Bw10,Fc20,Bw20,t6,t20,Nel,Nsub)
con = fdb.connect(dsn="192.168.0.2:/database/us-database/usdb.gdb", user="sysdba", password="#########")
cur = con.cursor()
InsertStatement="insert into Tosh_Probe (TestResults ) Values (?)"
cur.execute(InsertStatement, (ResultArray,))
con.commit()
In here the TestResult field is blob field in my database.
This gives a TypeError (???)
What is the correct syntax to store these values into a blob
An other option I tried is to write the list of items into a StringIO, and store that in the database. Now a new entry is made in the database but no data is added to the blob field
Here is the code for adding the fields to the StringIO
ResultArray = StringIO.StringIO()
ResultArray.write = Sens
ResultArray.write = DSens
#ResultArray.close #tried with and without this line but with the same result
I've tested this with Python 3.5.1 and FDB 1.6. The following variants of writing all work (into a blob sub_type text):
import fdb
import io
con = fdb.connect(dsn='localhost:testdatabase', user='sysdba', password='masterkey')
cur = con.cursor()
statement = "insert into blob_test2 (text_blob) values (?)"
cur.execute(statement, ("test blob as string",))
cur.execute(statement, (io.StringIO("test blob as StringIO"),))
streamwrites = io.StringIO()
streamwrites.write("streamed write1,")
streamwrites.write("streamed write2,")
streamwrites.seek(0)
cur.execute(statement, (streamwrites,))
con.commit()
con.close()
The major differences with your code in the case of the the writes to StringIO are:
Use of write(...) instead of write = ...
Use of seek(0) to position the stream at the start, otherwise you read nothing, as the stream is positioned after the last write.
I haven't tried binary IO, but I expect that to work in a similar fashion.
I'm retrieving a json file from online that I want to insert into my database using peewee. The problem is that some of the rows may already exist in my database. The solution should be to either ignore or replace the duplicate rows.
The InsertQuery function supports adding multiple rows, but I cannot figure out how to either suppress errors that the instance already exists or to replace the existing instance.
Starting with an empty database test, I run the following code
from peewee import *
from peewee import InsertQuery
database = MySQLDatabase('test', **{'password': 'researchServer', 'user': 'root'})
class BaseModel(Model):
class Meta:
database = database
class Image(BaseModel):
url = CharField(unique=True)
database.connect()
database.create_tables([Image])
images= [{'url': 'one'}, {'url':'two'}]
try:
image_entry = InsertQuery(Image, rows=images)
image_entry.execute()
except:
print 'error'
This produces no errors and successfully adds 'one' and 'two' to my table.
If I then run,
images= [{'url':'three'}, {'url': 'one'}, {'url':'four'}]
try:
image_entry = InsertQuery(Image, rows=images)
image_entry.execute()
except:
print 'error'
The execute function throws an error and neither 'three' or 'four' get added to the database.
I suppose one solution would be to check each row before adding it to the database, but this seems like it would be more inefficient.
You can use on_conflict() or upsert() on your InsertQuery.
on_conflict() will add an SQL ON CONFLICT clause with the given argument but only works with SQLite. upsert() basically turns the query into a REPLACE INTO on MySQL. You still need to call execute() after.
Image.insert_many(images).upsert(True).execute()
peewee doc upsert
I haven't been able to find a solution in peewee, but here's one that I wrote for SQLAlchemy.
from sqlalchemy import Table, Column, Integer, String, MetaData, ForeignKey
from sqlalchemy import create_engine
#Make the table
metadata = MetaData()
image = Table('image', metadata,
Column('url', String(250), primary_key=True))
db = create_engine('mysql://root:researchServer#localhost/test3')
metadata.create_all(db)
conn = engine.connect()
#Insert the first set of rows
images = [{'url': 'one'}, {'url': 'two'}]
inserter = db.image.insert()
conn.execute(inserter, images)
#Insert the second set of rows with some duplicates
images = [{'url': 'three'}, {'url': 'one'}, {'url':'four'}]
try:
inserter = db.image.insert().prefix_with("IGNORE")
conn.execute(inserter, images)
except:
print 'error'
The key is using the 'prefix_with' method to add the 'ignore' to the SQL expression. I was greatly helped by SQLAlchemy INSERT IGNORE
so right now I am querying an existing collection within mongoDB for some documents that all have the tag: "_t" : "SkeletonJoints". Once I have these documents, I want to insert it into a NEW collection that is created to hold only documents of these types with the username (e.g. username_kinectdata).
So here is my code:
#Import modules needed
import os, pymongo, json
from datetime import datetime
conn = None
db = None
isConnected = False
#try connecting with mongodb server
try:
conn = pymongo.MongoClient()
db = conn.emmersiv #connect to the emmersiv db
print db.collection_names() #print the collection of files within emmersiv db
print "Connected to the MongoDB server"
isConnected = True
except:
print "Connection Failed..."
#get all collections in a list and then remove non user data
allUsers = db.collection_names()
'''Filtering Kinect Data By Username'''
for users in allUsers:
coll = pymongo.collection.Collection(db, users.lower())
print "Currently looking at", users.lower(), " to filter kinect data"
#find all skeletal data
#kinectData = coll.find({"_t": "SkeletonJoints"})
newColl = users.lower() + "_kinectData" #name of the new collection to be made
#try to create and insert all kinect data into a new collection
try:
for line in coll.find({'_t': 'SkeletonJoints'}):
print line
jsonObj = json.loads(line) #convert to JSON?
if jsonObj is not None:
#create collection
db.create_collection(newColl)
#and insert JSON documents
coll.insert(jsonObj)
print "Insertion finished for ", newColl
print "No Insertion for ", newColl
except pymongo.errors.CollectionInvalid:
print 'Collection ', newColl, ' already exists'
except pymongo.errors.OperationFailure:
print "----> OP insertion failed"
except pymongo.errors.InvalidName:
print "----> Invalid insertion Name"
except:
print "----> WTF? ", traceback.print_exc()
So my problem is when I try insert, there is nothing being inserted. I don't really understand why this doesn't work. I am trying to iterate through the cursor.....
Thank you for your help!
No need to convert to JSON: PyMongo reads BSON from MongoDB and converts to Python dicts, and when you pass it a Python dict PyMongo converts it to BSON and sends it to MongoDB. JSON is never involved.
No need to call create_collection, MongoDB creates a collection when you insert into it for the first time.
Your statement, for line in coll.find({'_t': 'SkeletonJoints'}), will retrieve each document from the current collection that has a field "_t" with the value "SkeletonJoints", so I hypothesize that no such documents exist? Have you tried the following in the mongo shell?:
> use emmersiv
> db.MY_COLLECTION.find({_t: "SkeletonJoints"})
I expect that if you do this query (replacing "MY_COLLECTION" with the name of an actual collection) you'll get no documents in the Mongo shell, either.