We Keep Coding

import data from salesforce to databricks

I am trying to import data from salesforce to databricks using simple_salesforce . It is working fine with objects having less fields but started failing if my object has more fields.
Below is what I am trying
sf = Salesforce (
username = username,
password = password,
security_token = security_token,
domain="test"
)
df = pd.DataFrame(sf.query("Here I am passing all fields they are around 1000 in select query" from "+table)['records']).dropna(axis='columns', how='all').drop(['attributes'],axis=1)
Error
Error Code 414. Response content: <h1>Bad Message 414</h1><pre>reason: URI Too Long</pre>

SOQL can handle queries up to 100K characters. Have you hit that limit?
Can you cut it into 2 queries? SELECT Id, ExternalId__c, Afield__c, Bfield__c, ... say till "M" and then SELECT Id, ExternalId__c, Mfield__c, Nfield__c, ... ?
Or split them other way, maybe more and less important ones? You could even hide certain fields from the integration user (remove the checkboxes in Profile / Permission Set) if you can think of something that doesn't have to be synchronised.

Fetch from one database, Insert/Update into another using SQLAlchemy

We have data in a Snowflake cloud database that we would like to move into an Oracle database. As we would like to work toward refreshing the Oracle database regularly, I am trying to use SQLAlchemy to automate this.
I would like to do this using Core because my team is all experienced with SQL, but I am the only one with Python experience. I think it would be easier to tweak the data pulls if we just pass SQL strings. Plus the Snowflake db has some columns with JSON that seems easier to parse using direct SQL since I do not see JSON in the SnowflakeDialect.
I have established connections to both databases and am able to do select queries from both. I have also manually created the tables in our Oracle db so that the keys and datatypes match what I am pulling from Snowflake. When I try to insert, though, my Jupyter notebook just continuously says "Executing Cell" and hangs. Any thoughts on how to proceed or how to get the notebook to tell me where the hangup is?
from sqlalchemy import create_engine,pool,MetaData,text
from snowflake.sqlalchemy import URL
import pandas as pd
eng_sf = create_engine(URL( #engine for snowflake
account = 'account'
user = 'user'
password = 'password'
database = 'database'
schema = 'schema'
warehouse = 'warehouse'
role = 'role'
timezone = 'timezone'
))
eng_o = create_engine("oracle+cx_oracle://{}[{}]:{}#{}".format('user','proxy','password','database'),poolclass=pool.NullPool) #engine for oracle
meta_o = MetaData()
meta_o.reflect(bind=eng_o)
person_o = meta_o['bb_lms_person'] # other oracle tables follow this example
meta_sf = MetaData()
meta_sf.reflect(bind=eng_sf,only=['person']) # other snowflake tables as well, but for simplicity, let's look at one
person_sf = meta_sf.tables['person']
person_query = """
SELECT ID
,EMAIL
,STAGE:student_id::STRING as STUDENT_ID
,ROW_INSERTED_TIME
,ROW_UPDATED_TIME
,ROW_DELETED_TIME
FROM cdm_lms.PERSON
"""
with eng_sf.begin() as connection:
result = connection.execute(text(person_query)).fetchall() # this snippet runs and returns result as expected
with eng_o.begin() as connection:
connection.execute(person_o.insert(),result) # this is a coinflip, sometimes it runs, sometimes it just hangs 5ever
eng_sf.dispose()
eng_o.dispose()
I've checked the typical offenders. The keys for both person_o and the result are all lowercase and match. Any guidance would be appreciated.

use the metadata for the table. the fTable_Stage update or inserted as fluent functions and assign values to lambda variables. This is very safe because only metadata field variables can be used in the lambda. I am updating three fields:LateProbabilityDNN, Sentiment_Polarity, Sentiment_Subjectivity
engine = create_engine("mssql+pyodbc:///?odbc_connect=%s" % params)
connection=engine.connect()
metadata=MetaData()
Session = sessionmaker(bind = engine)
session = Session()
fTable_Stage=Table('fTable_Stage', metadata,autoload=True,autoload_with=engine)
stmt=fTable_Stage.update().where(fTable_Stage.c.KeyID==keyID).values(\
LateProbabilityDNN=round(float(late_proba),2),\
Sentiment_Polarity=round(my_valance.sentiment.polarity,2),\
Sentiment_Subjectivity= round(my_valance.sentiment.subjectivity,2)\
)
connection.execute(stmt)

Python MySQLdb Query Returns "None"

I'm running the following code with web.py to get mysql data and print it out on a webpage. For some reason, when I query the webpage, I get just the word "None". Any idea what the problem is? There is some data in the MySQL database, the credentials are valid, and the code is running without any errors.
import web
import MySQLdb
import json
import collections
db=MySQLdb.connect(user="someuser",
passwd="somepass",db="somedatabase")
urls = (
'/', 'index',
'/test', 'test'
)
class index:
def GET(self):
c=db.cursor()
c.execute("""
SELECT email, password
FROM users
""")
rows = c.fetchall()

I think your code fragment may be either incomplete or misformatted, but assuming it is not, the GET method has no return statement, which means that any invocation would return None, which in turn would be rendered as the string
"None"

Cross Server Select In SQLAlchemy

Is it possible to make SQLAlchemy do cross server joins?
If I try to run something like
engine = create_engine('mssql+pyodbc://SERVER/Database')
query = sql.text('SELECT TOP 10 * FROM [dbo].[Table]')
with engine.begin() as connection:
data = connection.execute(query).fetchall()
It works as I'd expect. If I change the query to select from [OtherServer].[OtherDatabase].[dbo].[Table] I get an error message "Login failed for user 'NT AUTHORITY\\ANONYMOUS LOGON"

Looks like there's an issue with how you authenticate to SQL server.
I believe you can connect using the current Windows user, the URI syntax is then mssql+pyodbc://SERVER/Database?trusted_connection=yes (I have never tested this, but give it a try).
Another option is to create a SQL server login (ie. a username/password that is defined within SQL server, NOT a Windows user) and use the SQL server login when you connect.
The database URI then becomes: mssql+pyodbc://username:password#SERVER/Database.

mssql+pyodbc://SERVER/Database?trusted_connection=yes threw an error when I tried to it. It did point me in the right direction though.
from sqlalchemy import create_engine, sql
import urllib
string = "DRIVER={SQL SERVER};SERVER=server;DATABASE=db;TRUSTED_CONNECTION=YES"
params = urllib.quote_plus(string)
engine = create_engine('mssql+pyodbc:///?odbc_connect={0}'.format(params))
query = sql.text('SELECT TOP 10 * FROM [CrossServer].[datbase].[dbo].[Table]')
with engine.begin() as connection:
data = connection.execute(query).fetchall()

It's quite complicated if you suppose to alter different servers through one connection.
But if you need to perform a query to a different server under different credentials you should add linked server first with sp_addlinkedserver. Then it should be added credentials to the linked server with sp_addlinkedsrvlogin. Have you tried this?

how to fetch specific collection from mongodb in django

I have clouded the mongodb collection for my project in Amazon server.
I am new to mongodb queries. When I connect and look in 'robomongo' tool, I can see that there two databases A and B. I want to access the one collection named 'wl_c' under B in view function in django and convert to JSON data.
I do not know how to do even though I tried,
from pymongo import Connection
server = '000.00.000.00'
port = 00000
conn = Connection(server,port)
def mongo(request):
mdb = conn.events.polls_post.find({})
data = json.dumps(mdb)
return HttpResponse(data, mimetype="application/json")
got
Type error mdb is not json serializable

find({}) returns cursor. You need to get the items. Either cast to list or iterate over the result.
Something like:
mdb = conn.events.polls_post.find({})
mdb_list = list(mdb)
json.dumps(mdb_list)
Look here