I'm setting up an application on OpenShift. It requires a Python script to run every hour and extract data from an online server into a MySQL database on OpenShift. I've been filling this database for some time now by running the Python script locally on my computer and using the port-forward technique. This works like a charm. Underneath is the port-forwarding info that's displayed while doing this.
Service Local OpenShift
------- -------------- ---- -------------------
httpd 127.0.0.1:8080 => 127.12.248.131:8080
mysql 127.0.0.1:3306 => 127.12.248.130:3306
node 127.0.0.1:8081 => 127.12.248.129:8080
Press CTRL-C to terminate port forwarding
And the variables I use in my local script...
host='127.0.0.1'
user='user_placeholder'
passwd='password_placeholder'
db='3v3'
port=3306
Of course I'd like the script to run automatically on the server of OpenShift so I don't have to do so myself. After a quick search I stumbled upon the cron method. It's possible to just put a python script in a map to make it run every hour. I set up some environment variables to access my database, just like I did in the local script. However, the script can't seem to connect to the MySQL database when I tail it. I've even printed the environment variables out and they're exactly the same as the ones I used to succesfully port-forward.
host=os.environ['OPENSHIFT_EXTMYSQL_DB_HOST'] # prints '127.12.248.130'
user=os.environ['OPENSHIFT_EXTMYSQL_DB_USERNAME'] # prints 'user_placeholder'
passwd=os.environ['OPENSHIFT_EXTMYSQL_DB_PASSWORD'] # prints 'password_placeholder'
db=os.environ['OPENSHIFT_EXTMYSQL_DB_NAME'] # prints '3v3'
port=int(os.environ['OPENSHIFT_EXTMYSQL_DB_PORT']) # prints 3306
The error message:
File "/var/lib/openshift/57c57f1889f5cfd9bb00006b/app-root/runtime/repo/.openshift/cron/minutely/test.py",
line 36, in
db = MySQLdb.connect(host=os.environ['OPENSHIFT_EXTMYSQL_DB_HOST'],
user=os.environ['OPENSHIFT_EXTMYSQL_DB_USERNAME'],
passwd=os.environ['OPENSHIFT_EXTMYSQL_DB_PASSWORD'],
db=os.environ['OPENSHIFT_EXTMYSQL_DB_NAME'],
port=int(os.environ['OPENSHIFT_EXTMYSQL_DB_PORT']))
File "/opt/rh/python27/root/usr/lib64/python2.7/site-packages/MySQLdb/__init__.py",
line 81, in Connect
return Connection(*args, **kwargs)
File "/opt/rh/python27/root/usr/lib64/python2.7/site-packages/MySQLdb/connections.py",
line 187, in init
super(Connection, self).init(*args, **kwargs2)
_mysql_exceptions.OperationalError: (2003, "Can't connect to MySQL server on '127.12.248.130' (113)")
I honestly don't know where this could go wrong. I know that it's not much of a lead but if anyone thinks to know what the problem could be, I'd be glad to hear.
Related
Currently, you have to manually connect to a remote database to extract info via an openvpn connection openvpn-gui.exe to extract the info and disconnect after each extraction job.
Connection is authenticated by a config.ovpn file stored locally.
Is there a way to automate the (connect > extract data > disconnect) process?
Managed to solve this issue...
Log in to your openvpn server domain via browser (e.g. https://12.345.678.999/)
Download Connection Profile "Yourself (autologin profile)". File usually named "client.ovpn"
IMPT! File holds user id and password credentials. Although file is kept on local machine, credentials file can be copied if pc is hacked/stolen
Paste "client.ovpn" file in "C:\Program Files\OpenVPN\config"
From openvpn-gui.exe desktop icon, import file and direct it to "client.ovpn"
Create 2 notepad files with the following commands and save as batch file (xxx.bat)
ovpn_connect.bat
"C:\Program Files\OpenVPN\bin\openvpn-gui.exe" --command connect client.ovpn
ovpn_disconnect.bat
"C:\Program Files\OpenVPN\bin\openvpn-gui.exe" --command disconnect client.ovpn
Follow website instruction to allow .bat file to be run with admin rights.
https://www.howtogeek.com/124087/how-to-create-a-shortcut-that-lets-a-standard-user-run-an-application-as-administrator/
Insert code into python script and run as per usual
import subprocess, time
# Connect to OpenVPN
subprocess.call([r'filepath\ovpn_connect.bat'])
time.sleep(15) # adjust your connection time
print("Connect OpenVPN")
# Disconnect from OpenVPN
subprocess.call([r'filepath\ovpn_disconnect.bat'])
print("Disconnect OpenVPN")
Hello Dear StackOverflow friends,
I'm receiving an strange error when connecting to a managed MySQL instance (DigitalOcean). The connection works on my Dev computer (Windows 8.1 machine), but not on the Prod server (CentOS 8, SELinux in permissive mode). The connection also works with MySQL Workbench.
I've done pip freeze on both mentioned environments and both results are mysql-connector-python==8.0.19 which I find very strange. I've made sure to run my tests with the venv activated.
The managed MySQL 8.x instance is set up to allow connections from both my droplet and my Dev IP address. I've also tried this without the firewall enabled. The managed instance requires the usage of an SSL enabled connection, so a CA Certificate is provided (I've applied chmod 777 over it for now to make sure that's not the cause of the problem).
I've checked the documentation of the library I'm using and it's compatible with MySQL 8.
It is also worth noting I've also tried the solution in this question about it.
The code is the following. Works as expected in Windows.
import datetime
import mysql.connector
from mysql.connector.constants import ClientFlag
dbconn_host = '<sanitized>'
dbconn_port = '<sanitized>'
dbconn_user = '<sanitized>'
dbconn_passwd = '<sanitized>'
dbconn_database = '<sanitized>'
cnx = mysql.connector.connect(
host=dbconn_host,
port=dbconn_port,
user=dbconn_user,
passwd=dbconn_passwd,
database=dbconn_database,
client_flags=ClientFlag.SSL,
ssl_ca='.\\ca_certificate.crt', # When running on prod server I change it to a proper Linux path
# auth_plugin='caching_sha2_password' # Trying another solution I had it changed to mysql_native_password
)
cur_a = cnx.cursor(buffered=True)
query_sel = (
"SELECT * FROM datasources"
)
cur_a.execute(query_sel)
for w in cur_a:
print(w[0])
This is the stack trace I receive in Linux.
(venv) [root#<sanitized> <sanitized>]# python -i conn-test.py
Traceback (most recent call last):
File "conn-test.py", line 12, in <module>
cnx = mysql.connector.connect(
File "/var/<sanitized>/venv/lib/python3.8/site-packages/mysql/connector/__init__.py", line 219, in connect
return MySQLConnection(*args, **kwargs)
File "/var/<sanitized>/venv/lib/python3.8/site-packages/mysql/connector/connection.py", line 104, in __init__
self.connect(**kwargs)
File "/var/<sanitized>/venv/lib/python3.8/site-packages/mysql/connector/abstracts.py", line 960, in connect
self._open_connection()
File "/var/<sanitized>/venv/lib/python3.8/site-packages/mysql/connector/connection.py", line 290, in _open_connection
self._do_auth(self._user, self._password,
File "/var/<sanitized>/venv/lib/python3.8/site-packages/mysql/connector/connection.py", line 212, in _do_auth
self._auth_switch_request(username, password)
File "/var/<sanitized>/venv/lib/python3.8/site-packages/mysql/connector/connection.py", line 256, in _auth_switch_request
raise errors.get_exception(packet)
mysql.connector.errors.DatabaseError: 1251: Client does not support authentication protocol requested by server; consider upgrading MySQL client
>>>
What do you think could be the issue here?
The magic of StackOverflow, is when you post a question that you find the solution in a few minutes. Two things happened:
Half the time I didn't have network connectivity to the MySQL database.
So I ran all kinds of tests before I could even ping the server, then I realized I should run all tests again, but I didn't start with the basics (I did all tests with patches applied, instead of trying a "vanilla" connection first, so to speak).
The solution is I commented out client_flags=ClientFlag.SSL, but left the CA Certificate enabled and the connection worked as expected in the Prod server.
I am trying to connect with Mysql server using mentioned below python code
import mysql.connector
mydb = mysql.connector.connect(
host = "127.0.0.1",
port = 5000,
user = "user id",
password = "password"
)
print(mydb)
But while running this code to test whether I have been connected with MySQL or not, I am facing the error which I am not able to understand.
Traceback (most recent call last):
File "C:\Users\varul.jain\Desktop\Test Phase\Mysql\mydb_test.py", line 7, in <module>
password = "root"
File "C:\Users\varul.jain\AppData\Local\Programs\Python\Python36\lib\site-packages\mysql\connector\__init__.py", line 179, in connect
return MySQLConnection(*args, **kwargs)
File "C:\Users\varul.jain\AppData\Local\Programs\Python\Python36\lib\site-packages\mysql\connector\connection.py", line 95, in __init__
self.connect(**kwargs)
File "C:\Users\varul.jain\AppData\Local\Programs\Python\Python36\lib\site-packages\mysql\connector\abstracts.py", line 716, in connect
self._open_connection()
File "C:\Users\varul.jain\AppData\Local\Programs\Python\Python36\lib\site-packages\mysql\connector\connection.py", line 210, in _open_connection
self._ssl)
File "C:\Users\varul.jain\AppData\Local\Programs\Python\Python36\lib\site-packages\mysql\connector\connection.py", line 142, in _do_auth
auth_plugin=self._auth_plugin)
File "C:\Users\varul.jain\AppData\Local\Programs\Python\Python36\lib\site-packages\mysql\connector\protocol.py", line 102, in make_auth
auth_data, ssl_enabled)
File "C:\Users\varul.jain\AppData\Local\Programs\Python\Python36\lib\site-packages\mysql\connector\protocol.py", line 58, in _auth_response
auth = get_auth_plugin(auth_plugin)(
File "C:\Users\varul.jain\AppData\Local\Programs\Python\Python36\lib\site-packages\mysql\connector\authentication.py", line 191, in get_auth_plugin
"Authentication plugin '{0}' is not supported".format(plugin_name))
mysql.connector.errors.NotSupportedError: Authentication plugin 'caching_sha2_password' is not supported
Note: I have initialized the default as 5000
for the testing purpose, I have initialized the port 5000, user - root and password - root
but the authorization the default password is not available as per mentioned above error
Is there any way to check the user id and password to cross verify and update in python code accordingly?
suggestions will be helpful
Follow these steps:
You must install MySQL Server (https://dev.mysql.com/downloads/installer/)
from this install MySQL Server.
Install mysql-connector-python (in your python environment)
Use this code :
Open database connection
import mysql.connector
mydb = mysql.connector.connect(host="127.0.0.1", port="3306", user="root", password="root", auth_plugin="mysql_native_password")
print(mydb)
As per SQL documentation for python library.you need to specify the auth plugin as follows:-
conn = mysql.connector.connect(user='root', password='password', host='127.0.0.1',port=5000, database='test', auth_plugin='mysql_native_password')
print(conn)
I come across a similar problem on my centOS VPS using webmin + python3.
At first I think it might due to connection package , so I tried:
'python3 -m pip install mysql-connector' , in python it will use 'import mysql.connector'
'pip3 install PyMySQL' , in python it use 'import PyMySQL'
both dont solve the problem. Then I found there are a couples of other connection method from a website (which I dont remember its address now), which detaily explain pro and cos of each, according to it, I finally choose to use mysqlclient.
'sudo pip3 install mysqlclient' , I use this because it is more versatile and have support for python3.
All above does not solve my problem, I still face connection error.
I tried a couple of random test and accidentially with root and the skip-grant-tables option in config file, I was able to log in mysql with SSH python code, but that is not the reason , cos a normal newly created user still cannot login no matter what hosts I am accessing in the code, localhost, 127.0.0.1, domains, ....
So I revoke the skip-grant-tables option cos thats risky without protection (just for test).
Lately, from default already exist user 'root' in MySQL Server, I found it has 4 entries, so I have try to create 4 individual similar entries record in webmin -> server -> MySQL server -> User Permission.
these records have same user name and less privilege right but similar host content assigned. they are 'local' , '127.0.0.1' , 'myservername.vps.provider.ca' , '::1'
(I do not know what does the last one means)
And, Bingo, after above 4 entries added, the new user is able to log in with password authenthicate provided via python3 code.
(remember the first time, you have to set the MySQL Admin 'root' with password in MySQL Server page, so that password Login authenthication feature will work, and that password need not same as root password of system)
Thats all, this take me 6 hours. hope it can help someone using webmin as well. Please give positive vote if it helps you in anyway. thanks
I have a frontend server that is reachable over the internet, and a database server that is only available in the local network where the frontend and database server are both in.
I need fabric to create a new database on the database server, but as the database server is not available on the internet, I need to "proxy" through the frontend server to call tasks on the database server.
How can I do that?
I searched for the answer for a few hours, but of course I only found it after asking about it here on stackoverflow.
The solution is to set the frontend server which is available through the internet as the gateway, either using the --gateway|-g flag in the command line, or by setting env.gateway.
I use this in combination with the env.roledefs property and fabric.api.roles to execute some tasks on the database server.
The solution roughly looks like this:
from fabric.api import task, env, roles
env.gateway = 'frontend.server'
env.hosts = ['frontend.server']
env.roledefs = {'db': ['database.server']}
#task
#roles('db')
def create_database():
""" Run on the database server. """
run(... mysql create database query stuff ...)
#task
def who_am_i():
""" Run on the frontend server. """
run('who am i')
Basically I can use ZODB fine. However the ZEO tutorials are all very confusing.
From my understanding you start a server by going into my directory and punchign into the command prompt
python runzeo.py -C zeo.config
Where my zeo.config file is as follows
<zeo>
address localhost:8090
</zeo>
<filestorage>
path C:\\Anaconda\\Lib\\site-packages\\ZEO\\var\\tmp\\Data.fs
</filestorage>
<eventlog>
<logfile>
path C:\\Anaconda\\Lib\\site-packages\\ZEO\\var\\tmp\\zeo.log
format %(asctime)s %(message)s
</logfile>
</eventlog>
When I run it the log file is filled with
2014-07-02T14:49:15 (1948) opening storage '1' using FileStorage
2014-07-02T14:49:15 StorageServer created RW with storages: 1:RW:C:\\Anaconda\\Lib\\site-packages\\ZEO\\var\\tmp\\Data.fs
2014-07-02T14:49:15 (1948) listening on ('localhost', 8090)
Now when I try to get a client to add some random stuff to the database with prints after every line to see how its going:
from ZEO.ClientStorage import ClientStorage
from ZODB import DB
import transaction
print "starting"
storage=ClientStorage(('localhost',8090))
print "storage opened"
db=DB(storage)
conn=db.open()
print "connection opened"
root=conn.root()
print "established connection"
root['letters']=['a','b','c']
print "added values"
transaction.commit()
print "transaction done"
root.close()
print "closed"
My code only prints "starting", no error messages are thrown, so Im assuming that its getting stuck on the storage = ClientStorange(('localhost',8090)) line, my Data.fs file remains unchanged. I have no idea what is wrong and I have consulted all the tutorials.
I'm on Windows using Python 2.7 and installed ZEO / ZODB from pip so I assume they are all up to date versions is that helps.
Any help or pointers to different object orientated databases (with multiple proccess access) would be appreciated.
Thanks everyone
Found the answer to my own question. Seems there is a bug with the implementation of using the localhost in windows. (Running server and client on same machine)
The source code needs an edit:
I have the same problem (can't connect to ZEO Server) using ZODB/ZEO 4.0 with Python 2.7.6 on Windows.
The proposed solution (changing line 446 of ZEO/zrpc/client.py) works for me, so why not incorporate the patch into the 4.0 release too?
- socket.getaddrinfo(host or 'localhost', port)
+ socket.getaddrinfo(host or 'localhost', port, 0, socket.SOCK_STREAM)"
From https://bugs.launchpad.net/zodb/+bug/1004513