How do I set a timeout length for my connection with Splunk through Python with the splunklib.binding class?
From the Splunk docs (http://docs.splunk.com/DocumentationStatic/PythonSDK/1.0/binding.html)
splunklib.binding.handler(key_file=None, cert_file=None, timeout=None)
where
timeout (integer or “None”) – The request time-out period, in seconds (optional)
import splunklib.client as client
import splunklib.results as results
import splunklib.binding as binding
port = '8089'
host = 'foo.splunkcloud.com'
username=self.SPLUNKUSERNAME,
password=self.SPLUNKPASSWORD,
connectionHanlder = binding.handler(timeout=4)
service = client.connect(host=host, port=port, password=password, usesrname=username,handler=connectionHanlder)
Related
I am posting the question and the answer, I found, as well, incase it would help someone. The following were my minimum requirements:
1. Client machine is Windows 10 and remote server is Linux
2. Connect to remote server via SSH through HTTP Proxy
3. HTTP Proxy uses Basic Authentication
4. Run commands on remote server and display output
The purpose of the script was to login to the remote server, run a bash script (check.sh) present on the server and display the result. The Bash script simply runs a list of commands displaying the overall health of the server.
There have been numerous discussions, here, on how to implement HTTP Proxy or running remote commands using Paramiko. However, I could not find the combination of both.
from urllib.parse import urlparse
from http.client import HTTPConnection
import paramiko
from base64 import b64encode
# host details
host = "remote-server-IP"
port = 22
# proxy connection & socket definition
proxy_url = "http://uname001:passw0rd123#HTTP-proxy-server-IP:proxy-port"
url = urlparse(proxy_url)
http_con = HTTPConnection(url.hostname, url.port)
auth = b64encode(bytes(url.username + ':' + url.password,"utf-8")).decode("ascii")
headers = { 'Proxy-Authorization' : 'Basic %s' % auth }
http_con.set_tunnel(host, port, headers)
http_con.connect()
sock = http_con.sock
# ssh connection
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
try:
ssh.connect(hostname=host, port=port, username='remote-server-uname', password='remote-server-pwd', sock=sock)
except paramiko.SSHException:
print("Connection Failed")
quit()
stdin,stdout,stderr = ssh.exec_command("./check")
for line in stdout.readlines():
print(line.strip())
ssh.close()
I would welcome any suggestions to the code as I am a network analyst and not a coder but keen to learn and improve.
I do not think that your proxy code is correct.
For a working proxy code, see How to ssh over http proxy in Python?, particularly the answer by #tintin.
As it seems that you need to authenticate to the proxy, after the CONNECT command, add a Proxy-Authorization header like:
Proxy-Authorization: Basic <credentials>
where the <credentials> is base-64 encoded string username:password.
cmd_connect = "CONNECT {}:{} HTTP/1.1\r\nProxy-Authorization: Basic <credentials>\r\n\r\n".format(*target)
I'm a little over my head when it comes to this SSH thing. Basically I am trying to access a friends server through and SSH tunnel using twisted conch. He has given me the following information:
MONGO_HOST = "ip address"
MONGO_DB = "server name"
MONGO_USER = "user name"
MONGO_PASS = "server password"
I was able to get this information to work using the python library motor.motor_asyncio (I need this be async compatible in order to use with other libraries) but for reasons that I can get into if necessary, will not work on the raspberry pi that I plan on running this program on.
Long story short, I was wondering if anyone could help me with some sample code to access my friends server using the information given above with twisted.conch.
I looked on the twisted.conch readthedocs, but the example needs more information than I can provide (I think) and is WAY over my head in terms of networking/SSH/etc.
Thanks in advance. I am willing to put in the work, but I need to know where to look.
Here is my relevant bit of code so far:
from motor.motor_asyncio import AsyncIOMotorClient
from sshtunnel import SSHTunnelForwarder
MONGO_HOST = "host address"
MONGO_DB = "server name"
MONGO_USER = "username"
MONGO_PASS = "password"
server = SSHTunnelForwarder(
MONGO_HOST,
ssh_username=MONGO_USER,
ssh_password=MONGO_PASS,
remote_bind_address=('address', gate),
local_bind_address=('address', gate)
)
server.start()
client = AsyncIOMotorClient('address', gate)
db = client.server_name
You can forward ports with Conch like this:
rom twisted.internet.defer import Deferred
from twisted.conch.scripts import conch
from twisted.conch.scripts.conch import ClientOptions, SSHConnection
from twisted.conch.client.direct import connect
from twisted.conch.client.default import SSHUserAuthClient, verifyHostKey
from twisted.internet.task import react
def main(reactor):
# authenticate as this user to SSH
user = "sshusername"
# the SSH server address
host = "127.0.0.1"
# the SSH server port number
port = 22
# a local port number to listen on and forward
localListenPort = 12345
# an address to forward to from the remote system
remoteForwardHost = "127.0.0.1"
# and the port number to forward to
remoteForwardPort = 22
conch.options = ClientOptions()
conch.options.parseOptions([
# don't ask the server for a shell
"--noshell",
# set up the local port forward
"--localforward={}:{}:{}".format(
localListenPort,
remoteForwardHost,
remoteForwardPort,
),
# specify the hostname for host key checking
host,
])
# don't stop when the last forwarded connection stops
conch.stopConnection = lambda: None
# do normal-ish authentication - agent, keys, passwords
userAuthObj = SSHUserAuthClient(user, conch.options, SSHConnection())
# create a Deferred that will tell `react` when to stop the reactor
runningIndefinitely = Deferred()
# establish the connection
connecting = connect(
host,
port,
conch.options,
verifyHostKey,
userAuthObj,
)
# only forward errors so the reactor will run forever unless the
# connection attempt fails. note this does not set up reconnection for a
# connection that succeeds and then fails later.
connecting.addErrback(runningIndefinitely.errback)
return runningIndefinitely
# run the reactor, call the main function with it, passing no other args
react(main, [])
Some of the APIs are weird because they are CLI focused. You don't have to do it this way but port forwarding is most easily accessible with these APIs rather than the APIs that are more focused on programmatic use.
I want to backup the VM on the server through the Python API interface, and I backup the VM via ovirt-engine. How do I backup using Ovirtsdk4?
I've tried a number of times, still errors
import logging
import ovirtsdk4 as sdk
logging.basicConfig(level=logging.DEBUG, filename='example.log')
def get_connection():
# Create the connection to the server:
return sdk.Connection(
url='https://172.32.42.75/ovirt-engine/api',
username='admin#internal',
password='password',
ca_file='/etc/pki/ovirt-engine/ca.pem',
debug=True,
log=logging.getLogger(),
)
if __name__ == "__main__":
# Set VM name
vm_name = 'myvm'
# Create a connection to the server:
connection = get_connection()
# Get a reference to the root service:
system_service = connection.system_service()
# Get the reference to the "vms" service:
vms_service = system_service.vms_service()
# Locate VM service
try:
vm = vms_service.list(search="name=%s" % vm_name, all_content=True)[0]
ovf_filename = "%s.ovf" % vm.id
with open(ovf_filename, "wb") as ovf_file:
ovf_file.write(vm.initialization.configuration.data)
finally:
# Close the connection to the server:
connection.close()
I hope the VM that will do the backup can run properly using the Python API oVirt.error of ovirtsdk4
I think error might be because of older curl. Please update curl, libcurl and NSS libraries and try again.
Ref: curl: (35) SSL connect error
Is it possible to have the Python requests library resolve a consul domain name with a SRV record and utilize the correct IP address and port when making the request?
For example, given that I have serviceA running with the IP address 172.18.0.5 on port 8080 and this service is registered with consul. And given that DNS for the host is set to use consul to resolve queries. Can I make a request like:
requests.get('http://serviceA.service.consul')
and have it be equivalent to the request:
requests.get('http://172.18.0.5:8080')
I ended up writing a patch for requests that would do this using this answer. I had to make some changes due to updates to the requests library. This patch works with requests version 2.11.1.
I used the dnspython library to resolve the SRV records and it expects the IP address and port that consul is listening for DNS requests on to be available as the environment variable CONSUL_DNS_IP_PORT. To use the patch import the requests_use_srv_records function from whatever module the patch is in and then call it. It will only attempt to use consul SRV records for hosts that end with .service.consul, other hosts will be resolved regularly.
Here's the patch:
# Python Imports
import os
from socket import error as SocketError, timeout as SocketTimeout
# 3rd Party Imports
from dns import resolver
from requests.packages.urllib3.connection import HTTPConnection
from requests.packages.urllib3.exceptions import (NewConnectionError,
ConnectTimeoutError)
from requests.packages.urllib3.util import connection
def resolve_srv_record(host):
consul_dns_ip_port = os.environ.get('CONSUL_DNS_IP_PORT',
'172.17.0.1:53')
consul_dns_ip, consul_dns_port = consul_dns_ip_port.split(':')
res = resolver.Resolver()
res.port = consul_dns_port
res.nameservers = [consul_dns_ip]
ans = resolver.query(host, 'SRV')
return ans.response.additional[0].items[0].address, ans[0].port
def patched_new_conn(self):
if self.host.endswith('.service.consul'):
hostname, port = resolve_srv_record(self.host)
else:
hostname = self.host
port = self.port
extra_kw = {}
if self.source_address:
extra_kw['source_address'] = self.source_address
if self.socket_options:
extra_kw['socket_options'] = self.socket_options
try:
conn = connection.create_connection((hostname, port),
self.timeout,
**extra_kw)
except SocketTimeout as e:
raise ConnectTimeoutError(
self, "Connection to %s timed out. (connect timeout=%s)" %
(self.host, self.timeout))
except SocketError as e:
raise NewConnectionError(
self, "Failed to establish a new connection: %s" % e)
return conn
def requests_use_srv_records():
HTTPConnection._new_conn = patched_new_conn
No, you can't unless you rewrite requests.
SRV record is design to find a service.
In this case, you already indicate to use http. So client will only query A or AAAA record for serviceA.service.consul.
In the code below, is the pipeline timeout 2 seconds?
client = redis.StrictRedis(host=host, port=port, db=0, socket_timeout=2)
pipe = client.pipeline(transaction=False)
for name in namelist:
key = "%s-%s-%s-%s" % (key_sub1, key_sub2, name, key_sub3)
pipe.smembers(key)
pipe.execute()
In the redis, there are a lot of members in the set "key". It always return the error as below with the code last:
error Error while reading from socket: ('timed out',)
If I modify the socket_timeout value to 10, it returns ok.
Doesn't the param "socket_timeout" mean connection timeout? But it looks like response timeout.
The redis-py version is 2.6.7.
I asked andymccurdy , the author of redis-py, on github and the answer is as below:
If you're using redis-py<=2.9.1, socket_timeout is both the timeout
for socket connection and the timeout for reading/writing to the
socket. I pushed a change recently (465e74d) that introduces a new
option, socket_connect_timeout. This allows you to specify different
timeout values for socket.connect() differently from
socket.send/socket.recv(). This change will be included in 2.10 which
is set to be released later this week.
The redis-py version is 2.6.7, so it's both the timeout for socket connection and the timeout for reading/writing to the socket.
It is not connection timeout, it is operation timeout. Internally the socket_timeout argument on StrictRedis() will be passed to the socket's settimeout method.
See here for details: https://docs.python.org/2/library/socket.html#socket.socket.settimeout