I am trying to use the a meteor ddp client to use the data from a meteor app in my python script. IT is a script that uses the Tor proxy API called stem. This is how my tor communicator looks like which works if ran separately:
Tor communicator (taken from the tor tutorial page with minor alterations):
import socket
import socks
import stem.process
import requests
from stem.util import term
from requestData import requestData
SOCKS_PORT = 7000
# Set socks proxy and wrap the urllib module
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, '127.0.0.1', SOCKS_PORT)
socket.socket = socks.socksocket
# Perform DNS resolution through the socket
def getaddrinfo(*args):
return [(socket.AF_INET, socket.SOCK_STREAM, 6, '', (args[0], args[1]))]
socket.getaddrinfo = getaddrinfo
def query(itemId):
"""
Uses requests to fetch a site using SocksiPy for Tor over the SOCKS_PORT.
"""
try:
return requestData(itemId)
except:
return "Unable to get data"
# Start an instance of Tor configured to only exit through Russia. This prints
# Tor's bootstrap information as it starts. Note that this likely will not
# work if you have another Tor instance running.
def print_bootstrap_lines(line):
if "Bootstrapped " in line:
print(line)
print(term.format("Starting Tor:\n", term.Attr.BOLD))
tor_process = stem.process.launch_tor_with_config(
config = {
'SocksPort': str(SOCKS_PORT),
'ExitNodes': '{ru}',
},
init_msg_handler = print_bootstrap_lines,
)
tor_process.kill() # stops tor
The above script is being ran from this script:
import Communicator
from MeteorClient import MeteorClient
client = MeteorClient('ws://127.0.0.1:3000/websocket')
client.connect()
def subscription_callback(error):
if error:
print(error)
client.subscribe('accounts', callback=subscription_callback)
all_posts = client.find('accounts')
print(all_posts)
Communicator.query("190aqe41vbewh7367f2hf27521")
But it is then giving me this result:
[1mStarting Tor:
[0m
May 10 13:21:45.000 [notice] Bootstrapped 0%: Starting
May 10 13:21:45.000 [notice] Bootstrapped 80%: Connecting to the Tor network
May 10 13:21:46.000 [notice] Bootstrapped 85%: Finishing handshake with first hop
May 10 13:21:46.000 [notice] Bootstrapped 90%: Establishing a Tor circuit
May 10 13:21:47.000 [notice] Bootstrapped 100%: Done
Traceback (most recent call last):
File "C:\Python34\lib\site-packages\socks.py", line 663, in connect
_BaseSocket.connect(self, proxy_addr)
ConnectionRefusedError: [WinError 10061] No connection could be made because the target machine actively refused it
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\gatsu\My Documents\LiClipse Workspace\TorCommunicator\MeteorDDP.py", line 5, in <module>
client.connect()
File "C:\Python34\lib\site-packages\python_meteor-0.1.6-py3.4.egg\MeteorClient.py", line 55, in connect
File "C:\Python34\lib\site-packages\python_ddp-0.1.5-py3.4.egg\DDPClient.py", line 119, in connect
File "C:\Python34\lib\site-packages\ws4py-0.3.4-py3.4.egg\ws4py\client\__init__.py", line 209, in connect
File "C:\Python34\lib\site-packages\socks.py", line 674, in connect
raise ProxyConnectionError(msg, error)
socks.ProxyConnectionError: Error connecting to SOCKS5 proxy 127.0.0.1:7000: [WinError 10061] No connection could be made because the target machine actively refused it
I solved this by importing the Communicator after I had done my Meteor stuff, right before I call a method in the Communicator.
from MeteorClient import MeteorClient
client = MeteorClient('ws://127.0.0.1:3000/websocket')
client.connect()
def subscription_callback(error):
if error:
print(error)
client.subscribe('accounts', callback=subscription_callback)
all_posts = client.find('accounts')
print(all_posts)
import Communicator
Communicator.query("190aqe41vbewh7367f2hf27521")
Related
I'm trying to create a chat between client and server written in Python, using SSL protocols with mutual authentication (i.e: server authenticates client and client authenticates server using certificates). My host machine is being used as the server, and my laptop is the client.
When attempting to connect to my host ip, I keep getting this error on my laptop:
Traceback (most recent call last):
File "/home/icarus/Codes/RealtimeChat/Chat.py", line 88, in <module>
main()
File "/home/icarus/Codes/RealtimeChat/Chat.py", line 75, in main
connection(ip, port, SSLSock)
File "/home/icarus/Codes/RealtimeChat/Chat.py", line 35, in connection
sock.connect((ip, port))
File "/usr/lib/python3.10/ssl.py", line 1375, in connect
self._real_connect(addr, False)
File "/usr/lib/python3.10/ssl.py", line 1362, in _real_connect
super().connect(addr)
ConnectionRefusedError: [Errno 111] Connection refused
And in the server - which was supposed to print a message saying that a connection was refused - nothing happens, it keeps listening for connections as if nothing happened
Connection function on client side:
def connection(ip, port, sock):
try:
sock.connect((ip, port))
print(f"Connected with {ip}")
except Exception as e:
print("Connection failed: ", e)
sock.close()
Server side:
def acceptConnection(self):
while True:
con, senderIP = self.sock.accept()
# Attempting to wrap connection with SSL socket
try:
SSLSock = self.getSSLSocket(con)
# If exception occurs, close socket and continue listening
except Exception as e:
print("Connection refused: ", e)
con.close()
continue
print(f"{senderIP} connected to the server")
# Adding connection to clients list
self.clients.append(SSLSock)
# Initializing thread to receive and communicate messages
# to all clients
threading.Thread(target=self.clientCommunication, args=(SSLSock, ), daemon=True).start()
This is the main function on my server:
def main():
serverIP = "127.0.0.1"
port = int(input("Port to listen for connections: "))
server = Server()
server.bindSocket(serverIP, port)
server.socketListen(2)
server.acceptConnection()
Everything works fine when I connect from my localhost (e.g I open a server on my host machine on one terminal and use another one on the same machine to connect to it). Both machines have the required certificates to authenticate each other, so I don't think that's the problem. Also, without the SSL implementation, the connection between this two different computers was refused by the server
I've tried using sock.bind('', port) on server side, disabling my firewall, used telnet 127.0.0.1 54321 (on my host machine) to check if the connection was working on the specified port (and it is), and also on the client machine (which showed that the connection was refused). I also tried running both scripts with admin privileges (sudo), but it also didn't work. Any suggestions?
I found what was wrong: I was trying to connect to my public IP address (which I found by searching for "What is my ip" on Google), but instead what should be done is to connect to the private IP address (I guess that's the correct name), and you can see yours using ifconfig on Linux and Mac and ipconfig on Windows using a terminal. By doing this, I could connect two computers that are on my network to my desktop server, I still haven't tested for computers in different networks, but the problem has been solved.
I am writing a webscraper using Python and Beutifulsoup.
It was not long before my IP got blocked. I now need to rotate my IP so that I can connect to the website and scrape the required data.
I mostly followed tutorials and git repo documentation:
https://medium.com/#abhishek_menon/anonymous-web-crawling-in-10-minutes-edff782decdc
https://github.com/baatout/tor-ip-rotation-python-example
I am just following the tutorials line by line, not 100% certain if I am doing the right things.
I have set the torrc file to:
# This file was generated by Tor; if you edit it, comments will not be preserved
# The old torrc file was renamed to torrc.orig.1, and Tor will ignore it
ClientOnionAuthDir /Users/user/Library/Application Support/TorBrowser-Data/Tor/onion-auth
DataDirectory /Users/user/Library/Application Support/TorBrowser-Data/Tor
GeoIPFile /Applications/Tor Browser.app/Contents/Resources/TorBrowser/Tor/geoip
GeoIPv6File /Applications/Tor Browser.app/Contents/Resources/TorBrowser/Tor/geoip6
ControlPort 9051
HashedControlPassword my_hashed_password
CookieAuthentication 1
The my_hashed_password I got by running tor --hash-password my_password.
I went on to create a config file in the directory where privoxy is installed with the following content:
forward-socks5 / 127.0.0.1:9050 .
Every time I change something in these two files I run a short script to restart the services and call privoxy to check everything is ok:
brew services restart tor
brew services restart privoxy
privoxy
When I run a test script:
import time
from urllib.request import ProxyHandler, build_opener, install_opener, Request, urlopen
from stem import Signal
from stem.control import Controller
class TorHandler:
def __init__(self):
self.headers = {
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.73.11 (KHTML, like Gecko) Version/7.0.1 Safari/537.73.11'}
def open_url(self, url):
# communicate with TOR via a local proxy (privoxy)
def _set_url_proxy():
proxy_support = ProxyHandler({'http': '127.0.0.1:8118'})
opener = build_opener(proxy_support)
install_opener(opener)
_set_url_proxy()
request = Request(url, None, self.headers)
return urlopen(request).read().decode('utf-8')
#staticmethod
def renew_connection():
__TOR_password__ = 'my_password'
__TOR_hashed_password__ = 'my_hashed_password'
with Controller.from_port(port=9051) as controller:
controller.authenticate(password=__TOR_password__)
controller.signal(Signal.NEWNYM)
controller.close()
if __name__ == '__main__':
wait_time = 2
number_of_ip_rotations = 3
tor_handler = TorHandler()
ip = tor_handler.open_url('http://icanhazip.com/')
print('My first IP: {}'.format(ip))
# Cycle through the specified number of IP addresses via TOR
for i in range(0, number_of_ip_rotations):
old_ip = ip
seconds = 0
tor_handler.renew_connection()
# Loop until the 'new' IP address is different than the 'old' IP address,
# It may take the TOR network some time to effect a different IP address
while ip == old_ip:
time.sleep(wait_time)
seconds += wait_time
print('{} seconds elapsed awaiting a different IP address.'.format(seconds))
ip = tor_handler.open_url('http://icanhazip.com/')
print('My new IP: {}'.format(ip))
Note: I have tried both TOR_password and TOR_hashed_password.
I receive the following output:
"/Users/code/venv/bin/python" "/Users/code/proxy_rotation.py"
My first IP: 185.220.101.16
Traceback (most recent call last):
File "/Users/code/venv/lib/python3.8/site-packages/stem/socket.py", line 535, in _make_socket
control_socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/code/proxy_rotation.py", line 48, in <module>
tor_handler.renew_connection()
File "/Users/code/proxy_rotation.py", line 29, in renew_connection
with Controller.from_port(port=9051) as controller:
File "/Users/code/venv/lib/python3.8/site-packages/stem/control.py", line 1033, in from_port
control_port = stem.socket.ControlPort(address, port)
File "/Users/code/venv/lib/python3.8/site-packages/stem/socket.py", line 503, in __init__
self.connect()
File "/Users/code/venv/lib/python3.8/site-packages/stem/socket.py", line 172, in connect
self._socket = self._make_socket()
File "/Users/code/venv/lib/python3.8/site-packages/stem/socket.py", line 538, in _make_socket
raise stem.SocketError(exc)
stem.SocketError: [Errno 61] Connection refused
Process finished with exit code 1
I would appreciate some assistance in:
Resolving this issue and use Tor to rotate my IP
Any suggestions on a better way to rotate IPs with Python on a local host. Ill place it in a docker container when I have this working
Thank you
It looks like my Tor was not started.
When I manually start it as an application I can connect and request a website.
Ill have to look at how I can automatically start Tor without having to start it from my applications
I am building one grpc based application with secure authentication, is there any issue with grpc python client or i am missing anything ? because on documentation it don't say much things.
My environment is following:
OS: "Ubuntu 18.04.5 LTS"
python 3.7
openssl V 1.1.1-1ubuntu2.1~18.04.13
grpcio==1.39.0
grpcio-tools==1.39.0
protobuf==3.17.3
I am always getting SSL_ERROR_SSL: CERTIFICATE_VERIFY_FAILED
E0901 13:16:05.996420843 13512 ssl_transport_security.cc:1468] Handshake failed with fatal error SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED.
E0901 13:16:06.495380631 13512 ssl_transport_security.cc:1468] Handshake failed with fatal error SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED.
E0901 13:16:06.953751870 13512 ssl_transport_security.cc:1468] Handshake failed with fatal error SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED.
E0901 13:16:07.407166253 13512 ssl_transport_security.cc:1468] Handshake failed with fatal error SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED.
Traceback (most recent call last):
File "grpc-client.py", line 21, in <module>
main()
File "grpc-client.py", line 11, in main
response = stub.ApiEndpoint(request)
File "/home/ggarg/.local/lib/python3.7/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/home/ggarg/.local/lib/python3.7/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "failed to connect to all addresses"
my proto file is:
syntax = "proto3";
service Api{
rpc ApiEndpoint(ApiRequest) returns (ApiResponse);
}
message ApiRequest{
string name = 1;
string message = 2;
}
message ApiResponse{
string reply = 1;
}
server code is:
import grpc
from concurrent import futures
import time
import api_pb2
import api_pb2_grpc
import os
class ChatBox(api_pb2_grpc.ApiServicer):
def ApiEndpoint(self, request, context):
response = api_pb2.ApiResponse()
response.reply = "Hi {}, myself {} , Thanks for this message : {}".format(
request.name, os.getenv("POD_NAME"), request.message)
return response
if __name__ == '__main__':
# create a gRPC server
server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
with open('cert/server.key', 'rb') as f:
private_key = f.read()
with open('cert/server.crt', 'rb') as f:
certificate_chain = f.read()
server_credentials = grpc.ssl_server_credentials(
((private_key, certificate_chain), ))
# add the servier created above tp the server
api_pb2_grpc.add_ApiServicer_to_server(ChatBox(), server)
# listen on port 50051
print('Starting server. Listening on port 50051.')
server.add_secure_port('[::]:50051', server_credentials)
server.start()
# since server.start() will not block,
# a sleep-loop is added to keep alive
try:
while True:
time.sleep(86400)
except KeyboardInterrupt:
server.stop(0)
and client code is:
import grpc
import api_pb2_grpc
import api_pb2
import time
def main():
request = api_pb2.ApiRequest(
name="timus",
message="You are awesome")
response = stub.ApiEndpoint(request)
print(response)
if __name__ == '__main__':
with open('cert/server.crt', 'rb') as f:
creds = grpc.ssl_channel_credentials(f.read())
#channel = grpc.secure_channel('www.timus.com:443', creds)
channel = grpc.secure_channel('0.0.0.0:50051', creds)
stub = api_pb2_grpc.ApiStub(channel)
while True:
main()
time.sleep(2)
I used following openssl command to generate crt.
openssl req -newkey rsa:4096 -nodes -sha512 -x509 -days 3650 -nodes -out server.crt -keyout server.key
I pasted above code as a example. for simplicity i am using same certificate on client and server. Initially i used ca/server/client certificate differently but I was having this issue and just for testing i used server certificate on both client and server side. but still getting same issue ? did anyone encounter similar issue ? thanks in advance.
It seems you are trying to assign the server certificate as the client's trust certificate. That will require the certificate to be a self-signed certificate. You will have to check that.
Even if it is self-signed, it is not the usual way for us to establish TLS connections. That can leave your applications to man-in-the-middle attack. You will want to have your client set a trust certificate that is issued by a CA, and have the CA sign the server's certificate. That can make sure there is a trustworthy third-party making sure the authentication is good.
You will also make sure the syntax is correct(sorry I am not that familiar with the Python syntax). We allow certs, root certs in both client and server for mutual TLS, but for single TLS, you only need to set certs for server, and root certs for client.
I am trying to connect to a remote server and list files inside a path using the below code:
import pysftp
myHostname = "myhostname.com"
myPort = <someportnumber>
myUsername = "<valid username>"
myPassword = "<valid password>"
cnopts = pysftp.CnOpts()
cnopts.hostkeys = None
with pysftp.Connection(host=myHostname, username=myUsername, password=myPassword, cnopts= cnopts, port=myPort) as sftp:
print("Connection succesfully established ... ")
sftp.chdir('/logs/dev')
# Obtain structure of the remote directory
directory_structure = sftp.listdir_attr()
# Print data
for attr in directory_structure:
print(attr.filename, attr)
But when I am running it, It is unable to establish a connection. It's throwing below exception:
Traceback (most recent call last):
File "c:/Users/611841191/Documents/SFTP File Download/SFTPFileDownload.py", line 11, in <module>
with pysftp.Connection(host=myHostname, username=myUsername, password=myPassword, cnopts= cnopts, port=myPort) as sftp:
File "C:\Users\611841191\AppData\Roaming\Python\Python38\site-packages\pysftp\__init__.py", line 140, in __init__
self._start_transport(host, port)
File "C:\Users\611841191\AppData\Roaming\Python\Python38\site-packages\pysftp\__init__.py", line 176, in _start_transport
self._transport = paramiko.Transport((host, port))
File "C:\Users\611841191\AppData\Roaming\Python\Python38\site-packages\paramiko\transport.py", line 415, in __init__
raise SSHException(
paramiko.ssh_exception.SSHException: Unable to connect to <myhostname.com>: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time or established connection failed because connected host has failed to respond
Can anyone help me out with why this exception is being thrown because I tried with other remote servers and the code seems to be working fine for them? The code is throwing exception for one particular remote server only.
If your code cannot connect somewhere using some protocol (SFTP in this case), the first thing to test is, whether you can connect using the same protocol using any existing GUI/commandline client from the same machine that runs your code.
If that does not work either, you do not have a programming question, but a simple network connectivity problem.
If you need a help resolving the problem, please go to Super User or Server Fault.
I am using python 2.7.2 on a windows 7 machine.
my code:
from multiprocessing import Process
def dummy_ftp_server(local_interface, username, password, homedir, perms):
from pyftpdlib.authorizers import DummyAuthorizer
from pyftpdlib.handlers import FTPHandler
from pyftpdlib.servers import FTPServer
authorizer = DummyAuthorizer()
authorizer.add_user(username, password, homedir, perm=perms)
handler = FTPHandler
handler.authorizer = authorizer
server = FTPServer((local_interface, 21), handler)
server.serve_forever()
process = Process(target=dummy_ftp_server, args=('127.0.0.1', 'user', 'pass', "C:/path/test", 'elradfmw'))
process.start()
when trying to run the ftp server using multiprocessing.process i get this error
Traceback (most recent call last):
File "C:\path\test_ftp.py", line 18, in dummy_ftp_server
server = FTPServer((local_interface, 21), handler)
File "C:\Python27\lib\site-packages\pyftpdlib\servers.py", line 145, in init
self._af = self.bind_af_unspecified(address_or_socket)
File "C:\Python27\lib\site-packages\pyftpdlib\ioloop.py", line 733, in bind_af_unspecified
raise socket.error(err)
error: [Errno 10048] Only one usage of each socket address (protocol/network address/port) is normally permitted
if i run it without using multiprocessing.process by using
dummy_ftp_server('127.0.0.1', 'user', 'pass', "C:/path/test", 'elradfmw')
it works fine.
on a linux machine i dont have this problem.
You are trying to run multiple ftp servers on the same ip address with the same port.
That is what the error is telling you.
It could be you already have something running on 127.0.0.1:21
check in a console with netstat -an