I want to use a remote Elasticsearch server for my website.
I have used elastic.co/ cloud service to create a remote Elasticsearch server.
I can connect/ping remote Elasticsearch server using the following command (it is scrubbed of sensitive info):
curl -u username:password https://55555555555bb0c30d1cba4e9e6.us-central1.gcp.cloud.es.io:9243
After tying this command into terminal, I receive the following response:
{
"name" : "instance-0000000001",
"cluster_name" : "555555555555",
"cluster_uuid" : "55555555555",
"version" : {
"number" : "7.10.2",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "555555555555555555555555",
"build_date" : "2021-01-13T00:42:12.435326Z",
"build_snapshot" : false,
"lucene_version" : "8.7.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
I am using the following Elasticsearch python client to integrate Elasticsearch with my website: https://elasticsearch-py.readthedocs.io/en/v7.10.1/index.html#
To initialize a local connection, I import the Elasticsearch class and initialize an instance of that class by pasting in the local url of my Elasticsearch server.
>>> from elasticsearch import Elasticsearch
>>> es = Elasticsearch('http://localhost:9200')
>>> es
<Elasticsearch([{'host': 'localhost', 'port': 9200}])>
Now I want to connect to my remote server using the same Elasticsearch class. To do this, I need to format the initialization of the Elasticsearch object with the info to connect to my remote elasticsearch server.
This is where I am having some trouble. The docstring for the Elasticsearch class is very opaque. In short, it is asking me to create a custom connection object and I have not been able to figure it out how to do it.
The documentation for the Elasticsearch object is a [little bit better][1] but still does not give an example involving a username and password.
I have the ability to create an API key, but I am not sure how to use it. An answer involving either an API key or answering how to connect using my username and password, would be very helpful.
You need to connect using TLS/SSL and Authentication as described in the documentation.
In your case you should use something like this.
from elasticsearch import Elasticsearch
es = Elasticsearch(
['5555555555bb0c30d1cba4e9e6.us-central1.gcp.cloud.es.io'],
http_auth=('username', 'password'),
scheme="https",
port=9243,
)
Related
I'am having a trouble to connect the Python API to Elasticsearch.
The Elasticsearch cluster is in azure cloud environment.
This is what I tried:
from elasticsearch import Elasticsearch
es = Elasticsearch(
"https://machine_name.kb.westeurope.azure.elastic-cloud.com:9200/",
ca_certs="/path/to/http_ca.crt",
api_key=("api_id", "api_key")
)
However i can't ping the 'es' client. In fact the test return a None Object.
if not es.ping():
print("No connection")
else:
print(es.ping())
The code abose print "No connection".
Can you please tell me what is wrong with my code ?
There is another method using the Cloud ID. Where i can find the Cloud ID ?
Please Help, Thank you so much.
Tldr;
You are pinging Kibana's domain name not Elasticsearch. The domain name is wrong. You should take the Elasticsearch domain name.
To Solve
In your url you need to change machine_name.kb.westeurope to machine_name.es.westeurope
from elasticsearch import Elasticsearch
es = Elasticsearch(
"https://machine_name.es.westeurope.azure.elastic-cloud.com:9200/",
ca_certs="/path/to/http_ca.crt",
api_key=("api_id", "api_key")
)
I'm trying to connect my Databricks cluster to an existing SQL Server database using python. I will like to leverage the integrated authentication method. Getting error com.microsoft.sqlserver.jdbc.SQLServerException: This driver is not configured for integrated authentication.
jdbcHostname = "sampledb-dev.database.windows.net"
jdbcPort= 1433
jdbcDatabase = "sampledb-dev"
jdbcUrl = "jdbc:sqlserver://{0}:{1}; database={2}".format(jdbcHostname, jdbcPort, jdbcDatabase)
connectionProperties={
"integratedSecurity" : "true",
"driver" : "com.microsoft.sqlserver.jdbc.SQLServerDriver"
}
print(jdbcUrl)
query ="(SELECT * FROM TABLE1.Domain)"
domains = spark.read.jdbc(url = jdbcUrl, table = query, properties = connectionProperties)
display(domains)
You can't use integratedSecurity=true with an Azure PaaS database. IntegratedSecurity is an on-premise construct.
You need to use authentication=ActiveDirectoryIntegrated or authentication=ActiveDirectoryPassword, please see JDBC docs here:
https://learn.microsoft.com/en-us/sql/connect/jdbc/connecting-using-azure-active-directory-authentication?view=sql-server-ver15
You will also need your account to be user with appropriate permissions to that database which is synch'd to Azure AD. If you use multi-factor authentication, then that's not supported for JDBC and your admin will need to provide you with a non-MFA enabled account. You'll know if this is the case because you will get a WSTrust error when trying to connect.
I have 3 databases in my MongoDB server. I am using pymongo to do some scripting with Python3.
I want to use the latest versions and practices. Once I open the client and pick the database, the API for pymongo.MongoClient.['mydatabase'].authenticate is deprecated.
https://api.mongodb.com/python/current/api/pymongo/database.html
Authentication prior to picking the database (while dialing the client) doesn't seem to flow down toward the database. Not just for pymongo, but also when I use mongo shell. So I have a feeling this is the issue.
script.py
import pymongo
from pymongo import MongoClient
u = getUser() # function which prompts for username
p = getPassword() # getpass.getpass('Password')
uri = formatUri(u, p) # formats 'mongodb://%s:%s#%s'.format(user, password, host)
client = MongoClient(uri)
db = client['mydb']
col = db.mycollection
for doc in col.find():
print(doc)
I get the error that I am not authorized for the database. I know my account works in shell but I have to dial the client first then use the db and then auth.
Here's a mongo shell example:
$ mongo
MongoDB shell version: v3.4.10
Connecting to: mongodb://127.0.0.1:port
MongoDB server version: v3.4.10
> use mydb
switched to mydb
> db.auth("user", "pass")
1
Any idea how I can either auth after picking the database or once I use the db it remembers the context I dialed with?
You seem to be missing some concepts here so I'll basically answer as a "guide" to what you should be doing instead. So "authentication' is not really something you do "after" connection, but rather you need to be "looking in the right place" when you actually attempt to authenticate.
We can start this by essentially following the process outlined in Enable Auth from the core documentation, but specifically altered because you want to be running this "test" under your own user account and local directory.
Revision Steps - Straight from Documentation
So first would would want to pick a local working directory and make a path for the database storage files underneath that. On *nix based systems you can do something like:
mkdir -p scratch/data/db
cd scratch
Then we want to startup a separate MongoDB instance without any other options. Making sure the port does not conflict with any other running instance:
mongod --port 37017 --dbpath data/db
In a new terminal or command line window, you can then connect to the shell:
mongo --port 37017
You always want at least one account with administrative privileges to at least "create accounts" and alter them in case you get in trouble, so create one:
use admin
db.createUser(
{
user: "admin",
pwd: "admin",
roles: [{ role: "userAdminAnyDatabase", db: "admin" }]
}
)
Now exit the shell and close the existing mongod instance running in the other terminal or command prompt and then start it again using --auth:
mongod --auth --port 37017 --dbpath data/db
Specific User - Make sure you follow these
Now you actually want to create a user that will be "used by your application". So these steps are important to ensure you get it right.
Log into a shell using your "adminstrative user":
mongo -u admin -p admin --port 37017 --authenticationDatabase 'admin'
You can alternately do the db.auth() method as shown in the question, but as noted this must be authorised on the "admin" namespace.
The next thing you want to do is create a user with access to "mydb" as a namespace with the readWrite role. For kicks, we are also going to let this user have the readAnyDatabase allowing them to "list" all databases namespaces, if not actually being able to do anything else with them.
IMPORTANT: You create ALL your users in the "admin" namespace. And this will be very important in future releases:
use admin
db.createUser(
{
"user": "myuser",
"pwd": "password",
"roles": [
{ "role": "readWrite", "db": "mydb" },
"readAnyDatabase"
]
}
)
Just for additional output, let's look at the current created users:
db.getUsers()
[
{
"_id" : "admin.admin",
"user" : "admin",
"db" : "admin",
"roles" : [
{
"role" : "userAdminAnyDatabase",
"db" : "admin"
}
]
},
{
"_id" : "admin.myuser",
"user" : "myuser",
"db" : "admin",
"roles" : [
{
"role" : "readWrite",
"db" : "mydb"
},
{
"role" : "readAnyDatabase",
"db" : "admin"
}
]
}
]
See how these have expanded in naming, and particularly the values assigned to the various "db" keys on each user. This should give you a little more insight into how MongoDB looks this up and why.
Python Connection
Finally we just want to connect from python. So presuming you have python and pymongo installed already, then it's just a simple listing to verify:
import pymongo
from pymongo import MongoClient
client = MongoClient('mongodb://myuser:password#localhost:37017');
db = client['mydb']
col = db.test
col.remove()
col.insert_one({ "a": 1 })
for doc in col.find():
print(doc)
Which shows the document created and listed without problem:
{u'a': 1, u'_id': ObjectId('5a08e5e0760108251722a737')}
Note that we don't actually need to make any mention of "admin" here, because this is the default where the driver "expects the accounts to be" and where you really "should" be doing it.
But I did it the wrong way
So let's say you originally got all confused and created the user under "mydb" instead:
use mydb
db.createUser({ "user": "bert", "pwd": "password", "roles": ["readWrite"] })
If you go look in "admin" that user is not there. But if you look on "mydb":
use mydb
db.getUsers()
[
{
"_id" : "mydb.bert",
"user" : "bert",
"db" : "mydb",
"roles" : [
{
"role" : "readWrite",
"db" : "mydb"
}
]
}
]
So you can see where the actual user data is now kept and how it has been recorded.
The simple case here is you "must" tell MongoDB where to obtain the authentication from for this user:
client = MongoClient('mongodb://bert:password#localhost:37017/mydb');
See how we add "mydb" on to the connection string. This is how it's done.
This is actually "in progress" to be made consistent with ALL drivers in how connections are made and where authentication happens as well as where you select the database. But there are basic rules:
If no other database namespace is provided with connection details for authentication credentials, then "admin" is taken to be the default.
Where there is a database namespace provided on the connection string, this will be used for authentication and this is the actual intent of the database namespace on the connection string.
Though other drivers "presently" differ in the role of the database namespace on the connection string, the usage is being changed to be consistent with all drivers that "using" a database namespace is in fact an API call, rather than being assigned from the connection string.
So where you need to authenticate depends on "where you created the user". But you should really be noting that "admin" is the place where you "should" be doing this instead of anywhere else.
Deprecation of Authenticate after connect
Whilst all drivers actually do have a similar method to authenticate(), which is used much like the shell example in the question, this method is now considered DEPRECATED as is mentioned throughout the content of the answer it is "intended" that you actually store your users in the "admin" namespace:
"Changed in version 3.5: Deprecated. Authenticating multiple users conflicts with support for logical sessions in MongoDB 3.6. To authenticate as multiple users, create multiple instances of MongoClient."
This is why the whole answer here is based on NOT using that method as you are meant to creating new connection instances, or using the "sessions" functionality available from MongoDB 3.6 instead.
I have 3 databases in my MongoDB server. I am using pymongo to do some scripting with Python3.
I want to use the latest versions and practices. Once I open the client and pick the database, the API for pymongo.MongoClient.['mydatabase'].authenticate is deprecated.
https://api.mongodb.com/python/current/api/pymongo/database.html
Authentication prior to picking the database (while dialing the client) doesn't seem to flow down toward the database. Not just for pymongo, but also when I use mongo shell. So I have a feeling this is the issue.
script.py
import pymongo
from pymongo import MongoClient
u = getUser() # function which prompts for username
p = getPassword() # getpass.getpass('Password')
uri = formatUri(u, p) # formats 'mongodb://%s:%s#%s'.format(user, password, host)
client = MongoClient(uri)
db = client['mydb']
col = db.mycollection
for doc in col.find():
print(doc)
I get the error that I am not authorized for the database. I know my account works in shell but I have to dial the client first then use the db and then auth.
Here's a mongo shell example:
$ mongo
MongoDB shell version: v3.4.10
Connecting to: mongodb://127.0.0.1:port
MongoDB server version: v3.4.10
> use mydb
switched to mydb
> db.auth("user", "pass")
1
Any idea how I can either auth after picking the database or once I use the db it remembers the context I dialed with?
You seem to be missing some concepts here so I'll basically answer as a "guide" to what you should be doing instead. So "authentication' is not really something you do "after" connection, but rather you need to be "looking in the right place" when you actually attempt to authenticate.
We can start this by essentially following the process outlined in Enable Auth from the core documentation, but specifically altered because you want to be running this "test" under your own user account and local directory.
Revision Steps - Straight from Documentation
So first would would want to pick a local working directory and make a path for the database storage files underneath that. On *nix based systems you can do something like:
mkdir -p scratch/data/db
cd scratch
Then we want to startup a separate MongoDB instance without any other options. Making sure the port does not conflict with any other running instance:
mongod --port 37017 --dbpath data/db
In a new terminal or command line window, you can then connect to the shell:
mongo --port 37017
You always want at least one account with administrative privileges to at least "create accounts" and alter them in case you get in trouble, so create one:
use admin
db.createUser(
{
user: "admin",
pwd: "admin",
roles: [{ role: "userAdminAnyDatabase", db: "admin" }]
}
)
Now exit the shell and close the existing mongod instance running in the other terminal or command prompt and then start it again using --auth:
mongod --auth --port 37017 --dbpath data/db
Specific User - Make sure you follow these
Now you actually want to create a user that will be "used by your application". So these steps are important to ensure you get it right.
Log into a shell using your "adminstrative user":
mongo -u admin -p admin --port 37017 --authenticationDatabase 'admin'
You can alternately do the db.auth() method as shown in the question, but as noted this must be authorised on the "admin" namespace.
The next thing you want to do is create a user with access to "mydb" as a namespace with the readWrite role. For kicks, we are also going to let this user have the readAnyDatabase allowing them to "list" all databases namespaces, if not actually being able to do anything else with them.
IMPORTANT: You create ALL your users in the "admin" namespace. And this will be very important in future releases:
use admin
db.createUser(
{
"user": "myuser",
"pwd": "password",
"roles": [
{ "role": "readWrite", "db": "mydb" },
"readAnyDatabase"
]
}
)
Just for additional output, let's look at the current created users:
db.getUsers()
[
{
"_id" : "admin.admin",
"user" : "admin",
"db" : "admin",
"roles" : [
{
"role" : "userAdminAnyDatabase",
"db" : "admin"
}
]
},
{
"_id" : "admin.myuser",
"user" : "myuser",
"db" : "admin",
"roles" : [
{
"role" : "readWrite",
"db" : "mydb"
},
{
"role" : "readAnyDatabase",
"db" : "admin"
}
]
}
]
See how these have expanded in naming, and particularly the values assigned to the various "db" keys on each user. This should give you a little more insight into how MongoDB looks this up and why.
Python Connection
Finally we just want to connect from python. So presuming you have python and pymongo installed already, then it's just a simple listing to verify:
import pymongo
from pymongo import MongoClient
client = MongoClient('mongodb://myuser:password#localhost:37017');
db = client['mydb']
col = db.test
col.remove()
col.insert_one({ "a": 1 })
for doc in col.find():
print(doc)
Which shows the document created and listed without problem:
{u'a': 1, u'_id': ObjectId('5a08e5e0760108251722a737')}
Note that we don't actually need to make any mention of "admin" here, because this is the default where the driver "expects the accounts to be" and where you really "should" be doing it.
But I did it the wrong way
So let's say you originally got all confused and created the user under "mydb" instead:
use mydb
db.createUser({ "user": "bert", "pwd": "password", "roles": ["readWrite"] })
If you go look in "admin" that user is not there. But if you look on "mydb":
use mydb
db.getUsers()
[
{
"_id" : "mydb.bert",
"user" : "bert",
"db" : "mydb",
"roles" : [
{
"role" : "readWrite",
"db" : "mydb"
}
]
}
]
So you can see where the actual user data is now kept and how it has been recorded.
The simple case here is you "must" tell MongoDB where to obtain the authentication from for this user:
client = MongoClient('mongodb://bert:password#localhost:37017/mydb');
See how we add "mydb" on to the connection string. This is how it's done.
This is actually "in progress" to be made consistent with ALL drivers in how connections are made and where authentication happens as well as where you select the database. But there are basic rules:
If no other database namespace is provided with connection details for authentication credentials, then "admin" is taken to be the default.
Where there is a database namespace provided on the connection string, this will be used for authentication and this is the actual intent of the database namespace on the connection string.
Though other drivers "presently" differ in the role of the database namespace on the connection string, the usage is being changed to be consistent with all drivers that "using" a database namespace is in fact an API call, rather than being assigned from the connection string.
So where you need to authenticate depends on "where you created the user". But you should really be noting that "admin" is the place where you "should" be doing this instead of anywhere else.
Deprecation of Authenticate after connect
Whilst all drivers actually do have a similar method to authenticate(), which is used much like the shell example in the question, this method is now considered DEPRECATED as is mentioned throughout the content of the answer it is "intended" that you actually store your users in the "admin" namespace:
"Changed in version 3.5: Deprecated. Authenticating multiple users conflicts with support for logical sessions in MongoDB 3.6. To authenticate as multiple users, create multiple instances of MongoClient."
This is why the whole answer here is based on NOT using that method as you are meant to creating new connection instances, or using the "sessions" functionality available from MongoDB 3.6 instead.
I am actually trying to develop a simple Node.js client, which can be used to query my Hive table on the Cosmos table .
I first tried to use this implementation for Hive using Node.js :
https://github.com/wdavidw/node-thrift-hive. My code is the following :
var router = express.Router();
var request = require('request');
var _ = require('lodash');
var hive = require('thrift-hive');
// Client connection
var client = hive.createClient({
version: '0.7.1-cdh3u3',
server: '130.206.80.46',
port: 10000,
timeout: 1000
});
// Execute call
router.route('/')
.get(function(req, res, next) {
client.execute('use default', function(err) {
// Query call
client.query('select * from interserre')
.on('row', function(database) {
console.log(database);
})
.on('error', function(err) {
console.log(err.message);
client.end();
})
.on('end', function() {
client.end();
});
});
})
module.exports = router;
I get this error:
events.js:85
throw er; // Unhandled 'error' event
^
Error: connect ECONNREFUSED
at exports._errnoException (util.js:746:11)
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1010:19)
After that I tried to use the Python example (provided by the documentation just to run a simple test.
http://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/BigData_Analysis_-_User_and_Programmer_Guide#Python
I couldn't even connect to the server.
Could not connect to 130.206.80.46:10000
I think the problem come from the fact that we don't provide any information that allow us to connect to the Cosmos portal. With Thrift there is no way to put your credentials. The server can't identify you're a real Cosmos user and reject your connection.
The python was just a test. I'm actually doing a Hackaton and was planning to use Node.js to query the data that I have. Where do you think the problem come from? The module? The Cosmos Portal?
We are currently adding to Cygnus a suite of Hive client examples for both HiveServer1 and HiveServe2 written in Java, Python and Node.js. This is something appearing in Cygnus 0.9.0, in the meantime you can check for the current status of the work (clients for Java and Python) in the develop branch.
Regarding Node.js, there are libraries allowing connecting to HiveServer1 (node-thrift-hive, node-hive) in a non authenticated way as you have experienced (technically speaking, they only work with a NOSASL configuration of Hive server). Regarding HiveServer2 occurs the same with libraries such as jsh2.