How to run Hive commands on hadoop using python - python

all, I am trying to use Python to run Hive commands on the Hadoop Edge node. I went through all related questions on this website. But I still cannot solve it.
Currently, I can run hive after I ssh the server using terminal:
ssh user-admin#abc-hadoop-edge01.endor.lan
Then type hive, I am able to run hive commands.
However, I cannot run hive using python. I use pyhs2. My codes are as follows:
import pyhs2
conn = pyhs2.connect('abc-hadoop-edge01.endor.lan', port = 10000, user = 'user-admin', password = '125438a', database='default')
cursor = conn.cursor()
conn.close()
The error is: TTransportException: TTranspo...:10000
Is there anyone who knows how to solve this ?
BTW: I use Mac.

Related

ORA-12154: TNS:could not resolve the connect identifier specified while executing python code from windows cmd

Getting TNS Error while executing a python code from Windows Command Prompt.
The python code connects to Oracle schema and perform few analysis.
The code works properly in jupyter but while executing the code in command prompt getting below error
ORA-12154: TNS:could not resolve the connect identifier specified
Below are the starting lines of my code.
import cx_Oracle
def test_func():
try:
con = cx_Oracle.connect('USER_ID/PASSWORD#host:Port/service')
Note: in above example given the syntax which I am using to connect to Oracle schema.
Kindly let me know, is any additional step is to be done?

Execute mysql commands in linux terminal using Python

I would like to automate the setup of a database and I would like to do it by using a python script to execute some commands in a linux terminal.
But I cannot see any way of executing commands in the terminal after connection to mysql database.
Below you can see a part of the script:
from time import sleep
from os import system
print("Setting up the database...\n")
system("sudo mysql -u root")
sleep(2)
This starts mysql, and after this none of the commands I try to execute from python get executed.
For example, I would like to run commands like these:
system("CREATE DATABASE mydatabase;")
system("CREATE USER 'user'#'localhost' IDENTIFIED BY '[PASSWORD]';")
system("GRANT ALL PRIVILEGES ON mydatabase.* TO 'user'#'localhost';")
Is this possible?
Edit 1: Potential solution as suggested by #AndHeFallen:
In python I create an sql file named db.sql, then run this in the terminal:
with open("db.sql","w") as db:
db.write("CREATE DATABASE mydatabase;\n")
db.write("CREATE USER 'user'#'localhost' IDENTIFIED BY '[PASSWORD]';\n")
db.write("GRANT ALL PRIVILEGES ON mydatabase.* TO 'user'#'localhost';\n")
system("sudo mysql -u root < db.sql")
I don't think it's possible to do that. If you want to interact with your MySQL database with a Python script, you'll need to connect directly to your db with an api like MySQLdb. (I may be wrong but the way you want to do may cause security issues)

HIVE not connecting with Python?

I have installed Hadoop and HIVE on windows 10 by following tutorials,
https://exitcondition.com/install-hadoop-windows/ & https://www.youtube.com/watch?v=npyRXkMhrgk respectively.
Both Hadoop and HIVE are running on my machine, I have been able to put files in HDFS and run queries in HIVE, but when I try to connect HIVE with python it gives different errors. Such as
from pyhive import hive
hive.Connection(host='localhost',port=10000,auth='NOSASL')
it gives following error:
TTransportException: TSocket read 0 bytes
I have tried impala as well but it did not work.
How can I connect python with hive, is it possible on windows 10 or should I shift to linux?
Pyhive had issues with auth = NOSASL in past.. not sure whether it got fixed .
Try hdfs3 python lib
conda install hdfs3
from hdfs3 import HDFileSystem
hdfs=HDFileSystem(host='localhost',port=9000)
More info available here..
https://medium.com/#arush.xtremelife/connecting-hadoop-hdfs-with-python-267234bb68a2

How to run a single process multiple times in parallel using a UNIX shell script?

I am working on one of the MPP databases and would like to run a single SQL query using multiple sessions in python or UNIX shell script. Can somebody share your thoughts on spawning a SQL in python/UNIX utility. Any inputs would be appreciated. Thank you.
Code :-
for i in {1..$n}
do
( sh run_sql.sh test.sql touchstone_test & )
done
For python you could download the MySQLdb module. MySQLdb is an interface for connecting to a MySQL database server from Python. It implements the Python Database API v2.0 and is built on top of the MySQL C API. More info.
Depending on scale, you can choose either option.
If you have small tasks to accomplish, running a shell script is fine. Note that you can also pipe the query to the mysql CLI client, e.g.
mysql_cmd="mysql -h<host> -u<user> -p<pwd> <db>"
echo "SELECT name, id FROM myobjects WHERE ...." | $mysql_cmd
For a larger scale project, I would go with Python and the mysqldb interface that was mentioned already.

Python and Connecting to MySQL over SSH

I am trying to connect to a MySQL database on someone else's "machine". When I use Navicat for MySQL, I have no problem connecting to it. I am trying to do the same with Python so that I do not have to use the GUI interface. I know all my info below is correct (even though I swap fake info) -- can anyone spot where I went wrong? The error I get is OperationalError: (2005, "Unknown MySQL server host 'FTP_hostname' (0)")
My code (using paramiko for the SSH):
import MySQLdb
import paramiko
import time
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect('SSH_hostname', 22, username='me', password='pswrd')
time.sleep(1)
db = MySQLdb.connect(host="FTP_hostname",
user="root",
passwd="pswrd2",
db="MyDB")
cur = db.cursor()
Again, I put all this into Navicat and connect no problem. Hoping you can help! Thanks!
MySQL, like most databases, by default runs locally and disallows access from outside networks. As such, you cannot connect to it from an external computer.
Navicat, being a software explicitely for remote administration of databases, likely connects via SSH and tunnels the MySQL connection over it. That way it can act as if the database was installed locally, and for the database it looks as if it was accessed locally.
You could try to do the same by creating a tunnel using Paramiko; see also this question.
If you still in need of connecting to a remote MySQL db via SSH I have used a library named sshtunnel, that wraps ands simplifies the use of paramiko (a dependency of the sshtunnel).
You can check my answer in another similar question with some sample code to use it.
db = MySQLdb.connect(host="FTP_hostname",
Would the host not need to be 127.0.0.1 (localhost) as the tunnel is making the MySQL DB local to the machine that the python script is running on?

Categories