so right now I am querying an existing collection within mongoDB for some documents that all have the tag: "_t" : "SkeletonJoints". Once I have these documents, I want to insert it into a NEW collection that is created to hold only documents of these types with the username (e.g. username_kinectdata).
So here is my code:
#Import modules needed
import os, pymongo, json
from datetime import datetime
conn = None
db = None
isConnected = False
#try connecting with mongodb server
try:
conn = pymongo.MongoClient()
db = conn.emmersiv #connect to the emmersiv db
print db.collection_names() #print the collection of files within emmersiv db
print "Connected to the MongoDB server"
isConnected = True
except:
print "Connection Failed..."
#get all collections in a list and then remove non user data
allUsers = db.collection_names()
'''Filtering Kinect Data By Username'''
for users in allUsers:
coll = pymongo.collection.Collection(db, users.lower())
print "Currently looking at", users.lower(), " to filter kinect data"
#find all skeletal data
#kinectData = coll.find({"_t": "SkeletonJoints"})
newColl = users.lower() + "_kinectData" #name of the new collection to be made
#try to create and insert all kinect data into a new collection
try:
for line in coll.find({'_t': 'SkeletonJoints'}):
print line
jsonObj = json.loads(line) #convert to JSON?
if jsonObj is not None:
#create collection
db.create_collection(newColl)
#and insert JSON documents
coll.insert(jsonObj)
print "Insertion finished for ", newColl
print "No Insertion for ", newColl
except pymongo.errors.CollectionInvalid:
print 'Collection ', newColl, ' already exists'
except pymongo.errors.OperationFailure:
print "----> OP insertion failed"
except pymongo.errors.InvalidName:
print "----> Invalid insertion Name"
except:
print "----> WTF? ", traceback.print_exc()
So my problem is when I try insert, there is nothing being inserted. I don't really understand why this doesn't work. I am trying to iterate through the cursor.....
Thank you for your help!
No need to convert to JSON: PyMongo reads BSON from MongoDB and converts to Python dicts, and when you pass it a Python dict PyMongo converts it to BSON and sends it to MongoDB. JSON is never involved.
No need to call create_collection, MongoDB creates a collection when you insert into it for the first time.
Your statement, for line in coll.find({'_t': 'SkeletonJoints'}), will retrieve each document from the current collection that has a field "_t" with the value "SkeletonJoints", so I hypothesize that no such documents exist? Have you tried the following in the mongo shell?:
> use emmersiv
> db.MY_COLLECTION.find({_t: "SkeletonJoints"})
I expect that if you do this query (replacing "MY_COLLECTION" with the name of an actual collection) you'll get no documents in the Mongo shell, either.
Related
I am trying to build an automated report. Most of the data is being pulled through web scraping before I start my shift. The program then enters a loop where it checks for data every five minutes using the function provided below. There is one data set that I have been unable to scrape so I set up a excel workbook that I can paste the data in to then insert the data in to my database.
Using a cli query SELECT * FROM TableName it returned 5160 rows in set (0.089 sec) at the same time as the python program was getting an empty list.
def checkData():
try:
db.c.execute('SELECT * FROM TableName')
except mariadb.Error as error:
printc(f"Error: {error}")
data = db.c.fetchall()
rcount = db.c.rowcount
logging.debug(f'No data found: {data} num rows: {rcount}')
print(data)
if len(data)>0:
return True
else:
return False
Logging output
2019-06-28 09:00:20,817 - DEBUG - No data found: [] num rows: 0
When I execute only the checkData function it returns True if there is data. Earlier in the program there is a db.c.execute('DELETE FROM TableName') statement. Is it possible that the database is caching old responses and what would be the best way to get it to inform me when it is no longer empty?
Additional Information:
Include Statement
import mysql.connector as mariadb
db is an object of
class DatabaseConnection():
def __init__(self):
self.url = 'host'
self.passwd = 'password'
self.db_conn = mariadb.connect(user='user', password=self.passwd, host=self.url, database='DatabaseName')
self.c = self.db_conn.cursor()
The rowcount works as expected (tried it myself, even after the fetch_all). So, your query is most likely returning nothing.
Are you sure you have a valid connection to your database in your python script?
Assign the result from the query to a variable and use the fetchall() method on that variable.
def checkData():
try:
result = db.c.execute('SELECT * FROM TableName')
except mariadb.Error as error:
printc(f"Error: {error}")
data = result.fetchall()
I'm trying to get the values of all databases existing in mongodb, iterate over all databases and collections for than print it documents. I can to print the document passing the collection as a variable, but can`t do it iterating over all databases and collections (as the value of variable). Someone knows if pymongo supports to do it dynamically passing as value and not passing the collection and the database as the variable itself??
client = MongoClient('mongodb://localhost:27017/')
names = client.database_names()
for dbName in names:
print(dbName)
db = client.dbName
collectionNames = client[dbName].collection_names()
for colecao in collectionNames:
print(colecao)
cursor = db.colecao # choosing the collection you need
print(cursor)
cursor2 = cursor.find() # get documents
for document in cursor2:
pprint(document)
The database names and collection names print normally, but the print cursor returns:
"Collection(Database(MongoClient(host=['localhost:27017'], document_class=dict, tz_aware=False, connect=True), u'dbName'), u'colecao')"
It goes with the name of variables.
Instead of
client.dbName
use
client.get_database(dbName)
and instead of
cursor = db.colecao
use
cursor = db.get_collection(colecao)
I'm fetching some data from an API on regular interval and wants to store the JSON data into database to access and use later.
From API, I get data in this sample each time:
'{"data": {"cursor": null, "files": {"nodes": [{u'code': u'BOPhmYQg5Vm', u'date': 1482244678,u'counts': 2, u'id': u'1409492981312099686'}, {u'code': u'g5VmBOPhmYQ', u'date': 1482244678,u'counts': 5, u'id': u'1209968614094929813'}]}}}'
I can json_data = json.loads(above_data) and then fetch nodes as nodes_data = json_data["data"]["files"]["nodes"] which gives a list of nodes.
I want to store this nodes data into DB column data = Column(db.Text) of Text type. Each time there are going to be 10-15 values in nodes list.
How do I store? There are multiple nodes and I need it in a way that in future I can append/add more nodes to already available data column in my db.
While I would like to do json.loads(db_data_col) so that I get valid json and can loop over all of nodes to get internal data and use later.
I'm confused on how to store in db and access later in valid json format.
Edit 1: Using Sqlite for testing. Can use PostgresSQL in future. Text type of column is main point.
If you are using Django 1.8 you can create your own model field that can store a json. This class will make sure that you have the right JSON format as well.
import json
from django.db import models
class JsonField(models.TextField):
"""
Stores json-able python objects as json.
"""
def get_db_prep_value(self, value, connection, prepared=False):
try:
return json.dumps(value)
except TypeError:
BAD_DATA.error(
"cannot serialize %s to store in a JsonField", str(value)
)
return ""
def from_db_value(self, value, expression, connection, context):
if value == "":
return None
try:
return json.loads(value)
except TypeError:
BAD_DATA.error("cannot load dictionary field -- type error")
return None
I found a way to store JSON data into DB. Since I'm accessing nodes from remote service which returns a list of nodes on every request, I need to build proper json to store/retrieve from db.
Say API returned json text as : '{"cursor": null, "nodes" = [{"name": "Test1", "value: 1}, {"name": "Test2", "value: 2}, ...]}'
So, first we need to access nodes list as:
data = json.loads(api_data)
nodes = data['nodes']
Now for 1st entry into DB column we need to do following:
str_data = json.dumps({"nodes": nodes})
So, str_data would return a valid string/buffer, which we can store into DB with a "nodes" key.
For 2nd or successive entries into DB column, we will do following:
# get data string from DB column and load into json
db_data = json.loads(db_col_data)
# get new/latest 'nodes' data from api as explained above
# append this data to 'db_data' json as
latest_data = db_data["nodes"] + new_api_nodes
# now add this data back to column after json.dumps()
db_col_data = json.dumps(latest_data)
# add to DB col and DB commit
It is a proper way to load/dump data from DB while adding/removing json and keeping proper format.
Thanks!
I have a list of items which I like to store in my firebird database.
Thus far I made the following code
Sens=278.3
DSens=1.2
Fc10=3.8
Bw10=60.0
Fc20=4.2
Bw20=90.0
ResultArray = (Sens,DSens,Fc10,Bw10,Fc20,Bw20,t6,t20,Nel,Nsub)
con = fdb.connect(dsn="192.168.0.2:/database/us-database/usdb.gdb", user="sysdba", password="#########")
cur = con.cursor()
InsertStatement="insert into Tosh_Probe (TestResults ) Values (?)"
cur.execute(InsertStatement, (ResultArray,))
con.commit()
In here the TestResult field is blob field in my database.
This gives a TypeError (???)
What is the correct syntax to store these values into a blob
An other option I tried is to write the list of items into a StringIO, and store that in the database. Now a new entry is made in the database but no data is added to the blob field
Here is the code for adding the fields to the StringIO
ResultArray = StringIO.StringIO()
ResultArray.write = Sens
ResultArray.write = DSens
#ResultArray.close #tried with and without this line but with the same result
I've tested this with Python 3.5.1 and FDB 1.6. The following variants of writing all work (into a blob sub_type text):
import fdb
import io
con = fdb.connect(dsn='localhost:testdatabase', user='sysdba', password='masterkey')
cur = con.cursor()
statement = "insert into blob_test2 (text_blob) values (?)"
cur.execute(statement, ("test blob as string",))
cur.execute(statement, (io.StringIO("test blob as StringIO"),))
streamwrites = io.StringIO()
streamwrites.write("streamed write1,")
streamwrites.write("streamed write2,")
streamwrites.seek(0)
cur.execute(statement, (streamwrites,))
con.commit()
con.close()
The major differences with your code in the case of the the writes to StringIO are:
Use of write(...) instead of write = ...
Use of seek(0) to position the stream at the start, otherwise you read nothing, as the stream is positioned after the last write.
I haven't tried binary IO, but I expect that to work in a similar fashion.
I'm new to python and even newer to sql, but I have been trying to make this work all morning and I cannot seem to figure it out. I have a table setup called POSTS(id). I'm trying to check if a number is already in the id column, but it doesn't matter what number postid is because data never returns none even if postid is set to a number not in the database.
import sqlite3 as lite
import sys
con = lite.connect('post.db')
postid = '52642'
cur = con.cursor()
cur.execute("select id from POSTS where id=?", (postid,))
data = cur.fetchall()
if data is None:
print ('not found')
else:
print ('found')
The statement
data = cur.fetchall()
will assign to data an empty list, not None, when nothing is found.
So just change your check to:
if not data:
instead of:
if data is None: