I am new to python and elasticsearch and i am trying to run a sample application. But my elasticsearch connection shows some error. I have installed elasticsearch and started the server. In the application there is a code like this
ES_URL = ['thrift://127.0.0.1:9200']
connection = pyes.ES(ES_URL)
index = connection.indices.get_alias('staging-asset')
when .get_alias is called, it shows
Client error: timed out
Traceback (most recent call last):
File "/Users/shahanahamza/.virtualenvs/cms/lib/python2.7/site-
packages/pyes/connection.py", line 151, in _client_call
return getattr(conn.client, attr)(*args, **kwargs)
File "/Users/shahanahamza/.virtualenvs/cms/lib/python2.7/site-
packages/pyes/pyesthrift/Rest.py", line 41, in execute
return self.recv_execute()
File "/Users/shahanahamza/.virtualenvs/cms/lib/python2.7/site-
packages/pyes/pyesthrift/Rest.py", line 52, in recv_execute
(fname, mtype, rseqid) = self._iprot.readMessageBegin()
File "/Users/shahanahamza/.virtualenvs/cms/lib/python2.7/site-
packages/thrift/protocol/TBinaryProtocol.py", line 126, in
readMessageBegin
sz = self.readI32()
File "/Users/shahanahamza/.virtualenvs/cms/lib/python2.7/site-
packages/thrift/protocol/TBinaryProtocol.py", line 206, in readI32
buff = self.trans.readAll(4)
File "/Users/shahanahamza/.virtualenvs/cms/lib/python2.7/site-
packages/thrift/transport/TTransport.py", line 58, in readAll
chunk = self.read(sz - have)
File "/Users/shahanahamza/.virtualenvs/cms/lib/python2.7/site-
packages/thrift/transport/TTransport.py", line 159, in read
self.__rbuf = StringIO(self.__trans.read(max(sz, self.__rbuf_size)))
File "/Users/shahanahamza/.virtualenvs/cms/lib/python2.7/site-
packages/thrift/transport/TSocket.py", line 103, in read
buff = self.handle.recv(sz)
timeout: timed out
Please help me to figure out the issue. Thanks in advance.
As mentioned in the comment by #omri , you might need to increase default threshold timeout-
pyes.es.ES(server, timeout=30, bulk_size=500,max_retries=10)
you can read more about here
There can also be other reasons for this timeouts. Check this and see if its still happening.
Check your mappings also
Related
I am using MongoDB atlas for my discord bot but recently ran into an error. On the hosting (Heroku) everything works without errors, I first updated all the modules, but the error has not disappeared.
I am using motor as a driver to work with MongoDB atlas.
Checked the database connection URL, everything is correct.
Python version 3.10(on Heroku too)
Traceback (most recent call last):
File "D:\Проекты\AkainuBot\main.py", line 23, in <module>
mongo = AsyncIOMotorClient(
File "D:\Python\lib\site-packages\motor\core.py", line 159, in __init__
delegate = self.__delegate_class__(*args, **kwargs)
File "D:\Python\lib\site-packages\pymongo\mongo_client.py", line 718, in __init__
self.__options = options = ClientOptions(
File "D:\Python\lib\site-packages\pymongo\client_options.py", line 165, in __init__
self.__pool_options = _parse_pool_options(options)
File "D:\Python\lib\site-packages\pymongo\client_options.py", line 132, in _parse_pool_options
ssl_context, ssl_match_hostname = _parse_ssl_options(options)
File "D:\Python\lib\site-packages\pymongo\client_options.py", line 98, in _parse_ssl_options
ctx = get_ssl_context(
File "D:\Python\lib\site-packages\pymongo\ssl_support.py", line 159, in get_ssl_context
ctx.load_verify_locations(certifi.where())
File "D:\Python\lib\site-packages\pymongo\pyopenssl_context.py", line 276, in load_verify_locations
self._callback_data.trusted_ca_certs = _load_trusted_ca_certs(cafile)
File "D:\Python\lib\site-packages\pymongo\ocsp_support.py", line 79, in _load_trusted_ca_certs
_load_pem_x509_certificate(cert_data, backend))
File "D:\Python\lib\site-packages\cryptography\x509\base.py", line 436, in load_pem_x509_certificate
return rust_x509.load_pem_x509_certificate(data)
ValueError: error parsing asn1 value: ParseError { kind: InvalidValue, location: ["RawCertificate::tbs_cert", "TbsCertificate::serial"] }
I've solved my problem getting the parsing asn1 value error by adding the ssl_cert_reqs option on the MongoClient (and obviously importing ssl).
dbClient = pymongo.MongoClient(uri, ssl_cert_reqs=ssl.CERT_NONE)
I have written a python script that successfully queries the trend deepsecurity api calls when ran locally on my machine.
I've been tasked with running the script in an aws lambda so that it is automated and can be scheduled.
The script is following the examples in the api reference and calls the legacy api successfully. However when I attempt to query using the computers api It blows up on the line : computers_api = deepsecurity.ComputersApi(deepsecurity.ApiClient(configuration))
def get_computer_status_api():
# Include computer status information in the returned Computer objects
#expand = deepsecurity.Expand(deepsecurity.Expand.computer_status)
expand = deepsecurity.Expand()
expand.add(deepsecurity.Expand.security_updates)
expand.add(deepsecurity.Expand.computer_status)
expand.add(deepsecurity.Expand.anti_malware)
# Set Any Required Values
computers_api = deepsecurity.ComputersApi(deepsecurity.ApiClient(configuration))
try:
computers = computers_api.list_computers(api_version, expand=expand.list(), overrides=False)
print("Querying ComputersApi...")
api_response_str=str(computers)
computer_count = len(computers.computers)
print(str(computer_count) + " Computers listed in Trend")
...
the error I get is :
[ERROR] OSError: [Errno 38] Function not implemented
Traceback (most recent call last):
File "/var/task/handler.py", line 782, in main
get_computer_status_api()
File "/var/task/handler.py", line 307, in get_computer_status_api
computers_api = deepsecurity.ComputersApi(deepsecurity.ApiClient(configuration))
File "/var/task/deepsecurity/api_client.py", line 69, in __init__
self.pool = ThreadPool()
File "/var/lang/lib/python3.8/multiprocessing/pool.py", line 925, in __init__
Pool.__init__(self, processes, initializer, initargs)
File "/var/lang/lib/python3.8/multiprocessing/pool.py", line 196, in __init__
self._change_notifier = self._ctx.SimpleQueue()
File "/var/lang/lib/python3.8/multiprocessing/context.py", line 113, in SimpleQueue
return SimpleQueue(ctx=self.get_context())
File "/var/lang/lib/python3.8/multiprocessing/queues.py", line 336, in __init__
self._rlock = ctx.Lock()
File "/var/lang/lib/python3.8/multiprocessing/context.py", line 68, in Lock
return Lock(ctx=self.get_context())
File "/var/lang/lib/python3.8/multiprocessing/synchronize.py", line 162, in __init__
SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
File "/var/lang/lib/python3.8/multiprocessing/synchronize.py", line 57, in __init__
sl = self._semlock = _multiprocessing.SemLock(
searching for this error it implies that I can't use the deepsecurity api in a lambda because lambdas don't support multiprocessing.
Looking for either confirmation that this is the case or suggestions for what I can change to get this working.
Trend support ticket suggested posting to here.
Resolved the issue by changing the python version from 3.8 to 3.7 in the lambda. The script now runs successfully
I am trying to read emails using the poplib library but getting an error - poplib.error_proto: -ERR EOF. The initial connection seems to be successful and I am able to get the response for the getwelcome() function but after that, the server seems to be closing the connection. Adding the code below, nothing special just the dummy code -
import poplib
client = poplib.POP3_SSL(server_name, port)
client.user(username)
client.pass_(password)
welcome = client.getwelcome()
print(welcome)
messages = len(client.list()[1]) # throws error - poplib.error_proto: -ERR EOF
Traceback (most recent call last):
File "email-pop.py", line 68, in <module>
fetch_all_mails()
File "email-pop.py", line 36, in fetch_all_mails
numMessages = len(pop_conn.list()[1])
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/poplib.py", line 243, in list
return self._longcmd('LIST')
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/poplib.py", line 186, in _longcmd
return self._getlongresp()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/poplib.py", line 162, in _getlongresp
resp = self._getresp()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/poplib.py", line 152, in _getresp
resp, o = self._getline()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/poplib.py", line 136, in _getline
if not line: raise error_proto('-ERR EOF')
poplib.error_proto: -ERR EOF
Can't seem to find what could be the issue, any help would be appreciated.
Thanks
I am trying to check if a certain dataset exists in bigquery using the Google Api Client in Python. It always worked untill the last update where I got this strange error I don't know how to fix:
Traceback (most recent call last):
File "/root/miniconda/lib/python2.7/site-packages/dsUtils/bq_utils.py", line 106, in _get
resp = bq_service.datasets().get(projectId=self.project_id, datasetId=self.id).execute(num_retries=2)
File "/root/miniconda/lib/python2.7/site-packages/oauth2client/util.py", line 140, in positional_wrapper
return wrapped(*args, **kwargs)
File "/root/miniconda/lib/python2.7/site-packages/googleapiclient/http.py", line 755, in execute
method=str(self.method), body=self.body, headers=self.headers)
File "/root/miniconda/lib/python2.7/site-packages/googleapiclient/http.py", line 93, in _retry_request
resp, content = http.request(uri, method, *args, **kwargs)
File "/root/miniconda/lib/python2.7/site-packages/oauth2client/client.py", line 598, in new_request
self._refresh(request_orig)
File "/root/miniconda/lib/python2.7/site-packages/oauth2client/client.py", line 864, in _refresh
self._do_refresh_request(http_request)
File "/root/miniconda/lib/python2.7/site-packages/oauth2client/client.py", line 891, in _do_refresh_request
body = self._generate_refresh_request_body()
File "/root/miniconda/lib/python2.7/site-packages/oauth2client/client.py", line 1597, in _generate_refresh_req
uest_body
assertion = self._generate_assertion()
File "/root/miniconda/lib/python2.7/site-packages/oauth2client/service_account.py", line 263, in _generate_ass
ertion
key_id=self._private_key_id)
File "/root/miniconda/lib/python2.7/site-packages/oauth2client/crypt.py", line 97, in make_signed_jwt
signature = signer.sign(signing_input)
File "/root/miniconda/lib/python2.7/site-packages/oauth2client/_pycrypto_crypt.py", line 101, in sign
return PKCS1_v1_5.new(self._key).sign(SHA256.new(message))
File "/root/miniconda/lib/python2.7/site-packages/Crypto/Signature/PKCS1_v1_5.py", line 112, in sign
m = self._key.decrypt(em)
File "/root/miniconda/lib/python2.7/site-packages/Crypto/PublicKey/RSA.py", line 174, in decrypt
return pubkey.pubkey.decrypt(self, ciphertext)
File "/root/miniconda/lib/python2.7/site-packages/Crypto/PublicKey/pubkey.py", line 93, in decrypt
plaintext=self._decrypt(ciphertext)
File "/root/miniconda/lib/python2.7/site-packages/Crypto/PublicKey/RSA.py", line 235, in _decrypt
r = getRandomRange(1, self.key.n-1, randfunc=self._randfunc)
File "/root/miniconda/lib/python2.7/site-packages/Crypto/Util/number.py", line 123, in getRandomRange
value = getRandomInteger(bits, randfunc)
File "/root/miniconda/lib/python2.7/site-packages/Crypto/Util/number.py", line 104, in getRandomInteger
S = randfunc(N>>3)
File "/root/miniconda/lib/python2.7/site-packages/Crypto/Random/_UserFriendlyRNG.py", line 202, in read
return self._singleton.read(bytes)
File "/root/miniconda/lib/python2.7/site-packages/Crypto/Random/_UserFriendlyRNG.py", line 178, in read
return _UserFriendlyRNG.read(self, bytes)
File "/root/miniconda/lib/python2.7/site-packages/Crypto/Random/_UserFriendlyRNG.py", line 137, in read
self._check_pid()
File "/root/miniconda/lib/python2.7/site-packages/Crypto/Random/_UserFriendlyRNG.py", line 153, in _check_pid
raise AssertionError("PID check failed. RNG must be re-initialized after fork(). Hint: Try Random.atfork()")
AssertionError: PID check failed. RNG must be re-initialized after fork(). Hint: Try Random.atfork()
Is someone understanding what is hapening?
Note that I also get this error with other bricks like GCStorage.
Note also that I use the following command to load my Google credentials:
from oauth2client.client import GoogleCredentials
def get_credentials(credentials_path): #my json credentials path
logger.info('Getting credentials...')
try:
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = credentials_path
credentials = GoogleCredentials.get_application_default()
return credentials
except Exception as e:
raise e
So if anyone know a better way to load my google credentials using my json service account file, and which would avoid the error, please tell me.
It looks like the error is in the PyCrypto module, which appears to be used under the hood by Google's OAuth2 implementation. If your code is calling os.fork() at some point, you may need to call Crypto.Random.atfork() afterward in both the parent and child process in order to update the module's internal state.
See here for PyCrypto docs; search for "atfork" for more info:
https://github.com/dlitz/pycrypto
This question and answer might also be relevant:
PyCrypto : AssertionError("PID check failed. RNG must be re-initialized after fork(). Hint: Try Random.atfork()")
So initially I connect to a server with:
soapclient = Client(url, transport=WindowsHttpAuthenticated(username=user, password=passwd))
and this works fine because when I print it it shows all of its properties correctly.
Then I run:
request = self.soapclient.factory.create('ns3:CRRequest')
request['ChangeRequestId'] = '601904'
response = self.soapclient.service.GetChangeRequestById([request])
And then I get an error on the response line which reads:
Traceback (most recent call last):
File "C:\Users\nlujan\Desktop\prism_source.py", line 63, in <module>
CR = prism.getChangeRequestById('601904')
File "C:\Users\nlujan\Desktop\prism_source.py", line 36, in getChangeRequestById
response = self.soapclient.service.GetChangeRequestById([request])
File "build\bdist.win32\egg\suds\client.py", line 542, in __call__
return client.invoke(args, kwargs)
File "build\bdist.win32\egg\suds\client.py", line 602, in invoke
File "build\bdist.win32\egg\suds\client.py", line 643, in send
result = self.succeeded(binding, reply.message)
File "build\bdist.win32\egg\suds\client.py", line 678, in succeeded
reply, result = binding.get_reply(self.method, reply)
File "build\bdist.win32\egg\suds\bindings\binding.py", line 149, in get_reply
soapenv.promotePrefixes()
AttributeError: 'NoneType' object has no attribute 'promotePrefixes'
I had someone else run my script and it works fine on their computer but I keep getting an error in mine. I've installed several different versions of python to see if that's the problem but with no luck. Does anyone see what could be the problem?
Try to add an endpoint address
client = Client('url', location = "https://someurl/services")