We Keep Coding

Configure SOCK5 proxy in Libtorrent session (Using Python)

I have been trying to configure proxy for a libtorent session; went through the documentation yet could not figure out a solution.
I tried the approach mentioned here. Did not work for me. This is the error I got-
r = lt.proxy_settings()
AttributeError: module 'libtorrent' has no attribute 'proxy_settings'. Did you mean: 'pe_settings'?
I tried to look for pe_settings() is the documentation but could not find anything like that. Being hopeful, I decided to do as the error message suggests, hoping it would lead me to some other error(from where i can pick up). Hence, I changed lt.proxy_settings() to lt.pe_settings(). The error that I get now is-
ses.set_dht_proxy(r)
Boost.Python.ArgumentError: Python argument types in
session.set_dht_proxy(session, pe_settings)
did not match C++ signature:
set_dht_proxy(libtorrent::session {lvalue}, libtorrent::aux::proxy_settings)
Mismatch in argument while trying to call the setters.
I also tried to use settings_pack. That did not work as well.
Here is what the session configuration looks like-
ses = lt.session()
ses.listen_on(6881, 6891)
r = lt.pe_settings()
r.proxy_hostnames = True
r.proxy_peer_connections = True
r.hostname = self.proxy_ip
r.username = ""
r.password = ""
r.port = self.proxy_port
r.type = lt.proxy_type_t().socks5_pw
#print lt.proxy_type().socks5_pw
#ses.set_dht_proxy(r)
#ses.set_peer_proxy(r)
#ses.set_tracker_proxy(r)
#ses.set_web_seed_proxy(r)
ses.set_proxy(r)
t = ses.settings()
t.force_proxy = True
t.proxy_hostnames = True
t.proxy_peer_connections = True
#t.proxy_tracker_connections = True
t.anonymous_mode = True
ses.set_settings(t)
print (ses.get_settings())
#ses.dht_proxy()
ses.peer_proxy()
#ses.tracker_proxy()
ses.web_seed_proxy()
ses.proxy()
ses.set_settings(t)
Any suggestions/ comments/ insights will be highly appreciated! Thanks!
Versions and platforms-
1. python 3.9
2. libtorrent 2.0.7 (installed it using vcpkg dependency manager)
3. mac os- Monterey
Vcpkg dependency manager installation steps
settings_pack documentation

you configure the proxy via the same settings_pack all other session-wide settings are configured via. Specifically you want to set:
proxy_hostname
proxy_port
proxy_type
Maybe proxy_username and proxy_password (docs)

adding fake_useragent to people_also_ask module

I want to scrape google 'people also ask questions/answer'. I am doing it successfully with the following module.
pip install people_also_ask
The problem is the library is configured such that no one can send many requests to google. I want to send 1000 requests per day and to achieve that I have to add fake_useragent to module. I tried a lot but when I try to add fake user agent to header it gives error. I am not a pro so I must have done wrong myself. Can anyone help me add fake_useragent to module(people_also_ask). here is working code to get question/answer.
from encodings import utf_8
import people_also_ask as paa
from fake_useragent import UserAgent
ua = UserAgent()
while True:
input("Please make sure the queries are in \\query.txt file.\npress Enter to continue...")
try:
query_file = open("query.txt","r")
queries = query_file.readlines()
query_file.close()
break
except:
print("Error with the query.txt file...")
for query in queries:
res_file = open("result.csv","a",encoding="utf_8")
try:
query = query.replace("\n","")
except:
pass
print(f'Searching for "{query}"')
questions = paa.get_related_questions(query, 14)
questions.insert(0,query)
print("\n________________________\n")
main_q = True
for i in questions:
i = i.split('?')[0]
try:
answer = str(paa.get_answer(i)['response'])
if answer[-1].isdigit():
answer = answer[:-11]
print(f"Question:{i}?")
except Exception as e:
print(e)
print(f"Answer:{answer}")
if main_q:
a = ""
b = ""
main_q = False
else:
a = "<h2>"
b = "</h2>"
res_file.writelines(str(f'{a}{i}?{b},"<p>{answer}</p>",'))
print("______________________")
print("______________________")
res_file.writelines("\n")
res_file.close()
print("\nSearch Complete.")
input("Press any key to Exit!")

This is against Google's terms of service, and the wishes of the people_also_ask package. This answer is for educational purposes only.
You asked why fake_useragent is prevented from working. It's not prevented from working, but the people_also_ask package simply isn't implementing any calls to make use of any fake_useragent methods. You can't just import a package and expect another package to start using it. You manually have to make packages work together.
To do that, you have to have some idea of how the 2 packages work. Have a look at the source code and you will see you can make them work together very easily. Just substitute the constant header in people_also_ask with one generated by fake_useragent before you request any data.
paa.google.HEADERS = {'User-Agent': ua.random} # replace the HEADER with a randomised HEADER from fake_useragent
questions = paa.get_related_questions(query, 14)
and
paa.google.HEADERS = {'User-Agent': ua.random} # replace the HEADER with a randomised HEADER from fake_useragent
answer = str(paa.get_answer(i)['response'])
NOTE:
Not all user agents will work. Google doesn't give related questions depending on the user agent. It is not the fault of either the fake_useragent, or the people_also_ask package.
In order to alleviate this issue somewhat, make sure you call ua.update() and you can also use PR #122 of fake_useragents to only select a subset of the newest user agents which are more likely to work, though you will still get a few missed queries. There is a reason the people_also_ask package didn't bypass or work-around this limitation from google

getting http debug info

I'm going through Dive into Python3. When I get to the chapter on http web services section 14.4, I can't seem to duplicate the following output in the python3 shell. Here's what the sample code looks like:
from http.client import HTTPConnection
HTTPConnection.debuglevel = 1
from urllib.request import urlopen
response = urlopen('http://diveintopython3.org/examples/feed.xml')
send: b'GET /examples/feed.xml HTTP/1.1
Host: diveintopython3.org
Accept-Encoding: identity
User-Agent: Python-urllib/3.1'
Connection: close
reply: 'HTTP/1.1 200 OK'
…further debugging information omitted…
When I enter this in ipython3, the final command gives no output. So why am I not getting the debug info in the example? After entering the above code, response.debuglevel == 0. I'm using python3.5.2.

The final command should not give any output, what you probably want is:
print(response.read())

I know this is an old question, bit I thought I would answer to help those who might still be seeing this question.
Ever since Python version 3.5.2 (release ~June 2016) the http.client.HTTPConnection.debuglevel is entirely ignored in favor of the debuglevel constructor argument for urllib.request.HTTPHandler.
This is due to this change that sets the value of http.client.HTTPConnection.debuglevel to whatever is set in urllib.request.HTTPHandler's constructor argument debuglevel, on this line.
A PR has been opened to fix this, but in the mean time you can either do one of two things:
You can use the constructor argument for HTTPHandler and HTTPSHandler (as demonstrated in this SO answer):
import urllib.request
handler = urllib.request.HTTPHandler(debuglevel=10)
opener = urllib.request.build_opener(handler)
content = opener.open('http://stackoverflow.com').read()
print(content[0:120])
You can also monkey patch the __init__ methods of HTTPHandler and HTTPSHandler to respect the global values like so:
https_old_init = urllib.request.HTTPSHandler.__init__
def https_new_init(self, debuglevel=None, context=None, check_hostname=None):
debuglevel = debuglevel if debuglevel is not None else http.client.HTTPSConnection.debuglevel
https_old_init(self, debuglevel, context, check_hostname)
urllib.request.HTTPSHandler.__init__ = https_new_init
http_old_init = urllib.request.HTTPHandler.__init__
def http_new_init(self, debuglevel=None):
debuglevel = debuglevel if debuglevel is not None else http.client.HTTPSConnection.debuglevel
http_old_init(self, debuglevel)
urllib.request.HTTPHandler.__init__ = http_new_init
Note: I don't recommend setting the debuglevel in HTTPHandler's as a method argument default value because the default values for method arguments get evaluated at function definition evaluation time, which, for HTTPHandler's constructor, is when the module urllib.request is imported.

Configuring eulexistdb with python bringing errors in django setting module

I have following code written in python in order to communicate with ExistDB using eulexistdb module.
from eulexistdb import db
class TryExist:
def __init__(self):
self.db = db.ExistDB(server_url="http://localhost:8899/exist")
def get_data(self, query):
result = list()
qresult = self.db.executeQuery(query)
hits = self.db.getHits(qresult)
for i in range(hits):
result.append(str(self.db.retrieve(qresult, i)))
return result
query = '''
let $x:= doc("/db/sample/books.xml")
return $x/bookstore/book/author/text()
'''
a = TryExist()
response = a.get_data(query)
print response
I am amazed that this code runs fine in Aptana Studio 3 giving me the output I want, but when running from other IDE or using command "python.exe myfile.py" brings following error:
django.core.exceptions.ImproperlyConfigured: Requested setting EXISTDB_TIMEOUT, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.
I used my own localsetting.py to solve the problem using following code:
import os
# must be set before importing anything from django
os.environ['DJANGO_SETTINGS_MODULE'] = 'localsettings'
... writing link for existdb here...
Then I get error as:
django.core.exceptions.ImproperlyConfigured: The SECRET_KEY setting must not be empty.
How do I configure the setting in Django to suit for ExistDB? Help me here please..

Never Mind. I found the answer with little research from this site. What I did was created a localsetting.py file with following configurations.
EXISTDB_SERVER_USER = 'user'
EXISTDB_SERVER_PASSWORD = 'admin'
EXISTDB_SERVER_URL = "http://localhost:8899/exist"
EXISTDB_ROOT_COLLECTION = "/db"
and in my main file myfile.py I used :
from localsettings import EXISTDB_SERVER_URL
import os
os.environ['DJANGO_SETTINGS_MODULE'] = 'localsettings.py'
and In the class TryExist I changed in __ init __() as:
def __init__(self):
self.db = db.ExistDB(server_url=EXISTDB_SERVER_URL)
PS: Using only os.environ['DJANGO_SETTINGS_MODULE'] = 'localsettings' brings the django.core.exceptions.ImproperlyConfigured: The SECRET_KEY setting must not be empty..

The reason your code works in an IDE but not at the command line is probably that you have a difference in what Python environments are used to run your code.
I've done a couple of tests:
Virtualenv with eulexistdb installed but not Django. eulexistdb tries to load django.conf but fails and so does not try to get its configuration from a Django configuration. Ultimately, your code runs without error.
Virtualenv with 'eulexistdb*and* Django:eulexistdbtries to loaddjango.conf` and succeed. I then tries to get is configuration from the Django configuration but fails. I get the same error you describe in your question.
To prevent the error in the presence of a Django installation, the problem can be fixed by adding a Django configuration like you did in your accepted self-answer. But if the code you are writing does not otherwise use Django, that's a bit of a roundabout way to get your code to run. The most direct way to fix the problem is to simply add a timeout parameter to the code that creates the ExistDB instance:
self.db = db.ExistDB(
server_url="http://localhost:8080/exist", timeout=None)
If you do this, then there won't be any error. Setting the timeout to None leaves the default behavior in place but prevents eulexistdb from looking for a Django configuration.

Need help using M2Crypto.Engine to access USB Token

I am using M2Crypto-0.20.2. I want to use engine_pkcs11 from the OpenSC project and the Aladdin PKI client for token based authentication making xmlrpc calls over ssl.
from M2Crypto import Engine
Engine.load_dynamic()
dynamic = Engine.Engine('dynamic')
# Load the engine_pkcs from the OpenSC project
dynamic.ctrl_cmd_string("SO_PATH", "/usr/local/ssl/lib/engines/engine_pkcs11.so")
Engine.cleanup()
Engine.load_dynamic()
# Load the Aladdin PKI Client
aladdin = Engine.Engine('dynamic')
aladdin.ctrl_cmd_string("SO_PATH", "/usr/lib/libeTPkcs11.so")
key = aladdin.load_private_key("PIN","password")
This is the error I receive:
key = pkcs.load_private_key("PIN","eT0ken")
File "/usr/local/lib/python2.4/site-packages/M2Crypto/Engine.py", line 70, in load_private_key
return self._engine_load_key(m2.engine_load_private_key, name, pin)
File "/usr/local/lib/python2.4/site-packages/M2Crypto/Engine.py", line 60, in _engine_load_key
raise EngineError(Err.get_error())
M2Crypto.Engine.EngineError: 23730:error:26096075:engine routines:ENGINE_load_private_key:not initialised:eng_pkey.c:112:
For load_private_key(), what should be passed as the first argument? The M2Crypto documentation does not explain it.
I don't get any errors loading the engines, but I'm not sure if I'm loading them correctly. It seems like the engine ID has to be a specific name but I don't find that list anywhere. 'dynamic' is working for me.
Any help would be appreciated!

Found !!!!
Yes, exactly the way where I came from.
So, actually the ENGINE_init() is not implemented in M2Crypto.Engine. So, only one solution: patching!!! (very small...) so I've created a new Engine method (in Engine.py)
def engine_initz(self):
"""Return engine name"""
return m2.engine_initz(self._ptr)
Why engine_initz ? because engine_init is already define in SWIG/_engine.i,:
void engine_init(PyObject *engine_err) {
Py_INCREF(engine_err);
_engine_err = engine_err;
}
I don't really know what is done, so I've prefered creating a new one... So I've just added the following to SWIG/_engine.i:
%rename(engine_initz) ENGINE_init;
extern int ENGINE_init(ENGINE *);
And recompile the __m2crypto.so, now just add a "pkcs11.engine_initz()" before launching the private key, and it works.....

I don't know what and why the engine_init code present in current M2Crypto is supposed to do. Exposing ENGINE_init() as engine_init2 with the following patch to M2Crypto helps:
Index: SWIG/_engine.i
===================================================================
--- SWIG/_engine.i (revision 719)
+++ SWIG/_engine.i (working copy)
## -44,6 +44,9 ##
%rename(engine_free) ENGINE_free;
extern int ENGINE_free(ENGINE *);
+%rename(engine_init2) ENGINE_init;
+extern int ENGINE_init(ENGINE *);
+
/*
* Engine id/name functions
*/
After this, the following code takes me further (but urllib does not fully work for me currently):
import sys, os, time, cgi, urllib, urlparse
from M2Crypto import m2urllib2 as urllib2
from M2Crypto import m2, SSL, Engine
# load dynamic engine
e = Engine.load_dynamic_engine("pkcs11", "/Users/martin/prefix/lib/engines/engine_pkcs11.so")
pk = Engine.Engine("pkcs11")
pk.ctrl_cmd_string("MODULE_PATH", "/Library/OpenSC/lib/opensc-pkcs11.so")
m2.engine_init2(m2.engine_by_id("pkcs11")) # This makes the trick
cert = e.load_certificate("slot_01-id_01")
key = e.load_private_key("slot_01-id_01", sys.argv[1])
ctx = SSL.Context("sslv23")
ctx.set_cipher_list("HIGH:!aNULL:!eNULL:#STRENGTH")
ctx.set_session_id_ctx("foobar")
m2.ssl_ctx_use_x509(ctx.ctx, cert.x509)
m2.ssl_ctx_use_pkey_privkey(ctx.ctx, key.pkey)
opener = urllib2.build_opener(ctx)
urllib2.install_opener(opener)

Looking at the pastebin link Becky provided, I believe it translates to something like this in the new API:
from M2Crypto import Engine, m2
dynamic = Engine.load_dynamic_engine("pkcs11", "/Users/martin/prefix/lib/engines/engine_pkcs11.so")
pkcs11 = Engine.Engine("pkcs11")
pkcs11.ctrl_cmd_string("MODULE_PATH", "/Library/OpenSC/lib/opensc-pkcs11.so")
r = pkcs11.ctrl_cmd_string("PIN", sys.argv[1])
key = pkcs11.load_private_key("id_01")
So I am betting that if you substitute "/Users/martin/prefix/lib/engines/engine_pkcs11.so" with "/usr/local/ssl/lib/engines/engine_pkcs11.so" and "/Library/OpenSC/lib/opensc-pkcs11.so" with "/usr/lib/libeTPkcs11.so" you might get it to work with Aladdin.

That is exactly the code I've tried. But It ended with the following error:
Traceback (most recent call last):
File "prog9.py", line 13, in <module>
key = pkcs11.load_private_key("id_45")
File "/usr/lib/pymodules/python2.5/M2Crypto/Engine.py", line 70, in load_private_key
return self._engine_load_key(m2.engine_load_private_key, name, pin)
File "/usr/lib/pymodules/python2.5/M2Crypto/Engine.py", line 60, in _engine_load_key
raise EngineError(Err.get_error())
M2Crypto.Engine.EngineError: 11814:error:26096075:engine outines:ENGINE_load_private_key:not initialised:eng_pkey.c:112:
I'm using OpenSC PKCS11 lib, not aladdin lib. But I don't think the problem is closed.

I tried the code that Heikki suggested (minus one line) and got the same error as Erlo. For load_private_key(), how do I know what to put in for the argument?
dynamic = Engine.load_dynamic_engine("pkcs11", "/usr/local/ssl/lib/engines/engine_pkcs11.so")
# m2.engine_free(dynamic) this line gave me an error TypeError: in method 'engine_free', argument 1 of type 'ENGINE *'
pkcs11 = Engine.Engine("pkcs11")
pkcs11.ctrl_cmd_string("MODULE_PATH", "/usr/lib/libeTPkcs11.so")
r = pkcs11.ctrl_cmd_string("PIN", "password")
key = pkcs11.load_private_key("id_01")

I think the problem is not really the "load_private_key()". It's like something is missing between "MODULE_PATH" definition and the load_private_key() call. What happen if you remplace "/usr/lib/libeTPkcs11.so" by a wrong path ? In my case I have no error related to this.
I've run "pcscd" in foreground with high debug level, there is no call to smartcard during the python execution... So definitly, I don't understand what's wrong...
The equivalent in "openssl" is using "-pre" command. The "-pre" (by opposite to the "-post") are command sent to the engine before loading. Perhaps we need to call a methode which "load" the engine after all "ctrl_cmd_string" calls ?? ...
Lost :-/