When trying to connect to a database with this python code to test the connection.
import boto3
s3 = boto3.resource('s3')
for b in s3.buckets.all():
print b.name
I am getting this error thrown at me.
Traceback (most recent call last):
File "boto3_test.py", line 4, in <module>
for b in s3.buckets.all():
File "/usr/local/lib/python2.7/dist-packages/boto3/resources/collection.py", line 83, in __iter__
for page in self.pages():
File "/usr/local/lib/python2.7/dist-packages/boto3/resources/collection.py", line 161, in pages
pages = [getattr(client, self._py_operation_name)(**params)]
File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 310, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 385, in _make_api_call
operation_model, request_dict)
File "/usr/local/lib/python2.7/dist-packages/botocore/endpoint.py", line 111, in make_request
return self._send_request(request_dict, operation_model)
File "/usr/local/lib/python2.7/dist-packages/botocore/endpoint.py", line 140, in _send_request
success_response, exception):
File "/usr/local/lib/python2.7/dist-packages/botocore/endpoint.py", line 213, in _needs_retry
caught_exception=caught_exception)
File "/usr/local/lib/python2.7/dist-packages/botocore/hooks.py", line 226, in emit
return self._emit(event_name, kwargs)
File "/usr/local/lib/python2.7/dist-packages/botocore/hooks.py", line 209, in _emit
response = handler(**kwargs)
File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 183, in __call__
if self._checker(attempts, response, caught_exception):
File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 250, in __call__
caught_exception)
File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 273, in _should_retry
return self._checker(attempt_number, response, caught_exception)
File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 313, in __call__
caught_exception)
File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 222, in __call__
return self._check_caught_exception(attempt_number, caught_exception)
File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 355, in _check_caught_exception
raise caught_exception
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://s3.us-east-5.amazonaws.com/"
The strange part is that the connection was working earlier in the day and then at it stopped working the same thing happened to me yesterday at the same time so I am assuming it is a network configuration issue. This question is the only one that is similar the solution the original poster stated was to unset both HTTP_PROXY and HTTPS_PROXY which I attempted but it did not work.
You have configured AWS env incorrectly. Check your ~/.aws/config file. You must have configured it incorrectly as us-east-5 which is an invalid region. Change it to us-east-1. Or run aws configure command again and specify the correct region.
[default]
region = us-east-1
We got this issue (with status code 500 returned) out of the blue.
After some research we found out that there was some infra update on AWS which caused the service to be down.
You can find the current down\up services per zone and upcoming down time here
Got the same boto3 error.
In my case a config.lock and credentials.lock were created behind the scenes:
Removing them solved the issue.
Related
I have a simple Google App Engine web app that runs fine in my local development environment (including being able to post to the datastore), but when deployed to Google App Engine I get the following error messages in the console.
Note that the second traceback is abridged - above it is a call to the put method of the datastore.Client() instance.
I have tried re-deploying and have tried googling the error messages but no luck so far. Is there somewhere I need to tell GAE that my app should have permission to the datastore?
Any pointers much appreciated.
Traceback (most recent call last):
File "/env/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
return callable_(*args, **kwargs)
File "/env/lib/python3.7/site-packages/grpc/_channel.py", line 923, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/env/lib/python3.7/site-packages/grpc/_channel.py", line 826, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
File "/layers/google.python.pip/pip/google/cloud/datastore/client.py", line 565, in put
self.put_multi(entities=[entity], retry=retry, timeout=timeout)
File "/layers/google.python.pip/pip/google/cloud/datastore/client.py", line 605, in put_multi
current.commit(retry=retry, timeout=timeout)
File "/layers/google.python.pip/pip/google/cloud/datastore/batch.py", line 293, in commit
self._commit(retry=retry, timeout=timeout)
File "/layers/google.python.pip/pip/google/cloud/datastore/batch.py", line 257, in _commit
commit_response_pb = self._client._datastore_api.commit(
File "/layers/google.python.pip/pip/google/cloud/datastore_v1/gapic/datastore_client.py", line 645, in commit
return self._inner_api_calls["commit"](
File "/layers/google.python.pip/pip/google/api_core/gapic_v1/method.py", line 145, in __call__
return wrapped_func(*args, **kwargs)
File "/layers/google.python.pip/pip/google/api_core/retry.py", line 281, in retry_wrapped_func
return retry_target(
File "/layers/google.python.pip/pip/google/api_core/retry.py", line 184, in retry_target
return target()
File "/layers/google.python.pip/pip/google/api_core/timeout.py", line 214, in func_with_timeout
return func(*args, **kwargs)
File "/layers/google.python.pip/pip/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "<string>", line 3, in raise_from
google.api_core.exceptions.PermissionDenied: 403 Missing or insufficient permissions.
Your account that runs the app (probably your App Engine service account) should have the required permissions for the Datastore. Please give roles/datastore.user as described here
I am running a spark application in Amazon EMR Cluster and since a few days ago, I am getting the following error whenever I try reading a file from S3 using pandas. I have added bootstrap actions to install pandas, fsspec and s3fs.
Code:
import pandas as pd
df = pd.read_csv(s3_path)
Error Log:
Traceback (most recent call last):
File "spark.py", line 84, in <module>
df=pd.read_csv('s3://<bucketname>/<filename>.csv')
File "/usr/local/lib64/python3.7/site-packages/pandas/io/parsers.py", line 686, in read_csv
return _read(filepath_or_buffer, kwds)
File "/usr/local/lib64/python3.7/site-packages/pandas/io/parsers.py", line 435, in _read
filepath_or_buffer, encoding, compression
File "/usr/local/lib64/python3.7/site-packages/pandas/io/common.py", line 222, in get_filepath_or_buffer
filepath_or_buffer, mode=mode or "rb", **(storage_options or {})
File "/usr/local/lib/python3.7/site-packages/fsspec/core.py", line 133, in open
out = self.__enter__()
File "/usr/local/lib/python3.7/site-packages/fsspec/core.py", line 101, in __enter__
f = self.fs.open(self.path, mode=mode)
File "/usr/local/lib/python3.7/site-packages/fsspec/spec.py", line 844, in open
**kwargs
File "/usr/local/lib/python3.7/site-packages/s3fs/core.py", line 394, in _open
autocommit=autocommit, requester_pays=requester_pays)
File "/usr/local/lib/python3.7/site-packages/s3fs/core.py", line 1276, in __init__
cache_type=cache_type)
File "/usr/local/lib/python3.7/site-packages/fsspec/spec.py", line 1134, in __init__
self.details = fs.info(path)
File "/usr/local/lib/python3.7/site-packages/s3fs/core.py", line 719, in info
return sync(self.loop, self._info, path, bucket, key, kwargs, version_id)
File "/usr/local/lib/python3.7/site-packages/fsspec/asyn.py", line 51, in sync
raise exc.with_traceback(tb)
File "/usr/local/lib/python3.7/site-packages/fsspec/asyn.py", line 35, in f
result[0] = await future
File "/usr/local/lib/python3.7/site-packages/s3fs/core.py", line 660, in _info
Key=key, **version_id_kw(version_id), **self.req_kw)
File "/usr/local/lib/python3.7/site-packages/s3fs/core.py", line 214, in _call_s3
raise translate_boto_error(err)
File "/usr/local/lib/python3.7/site-packages/s3fs/core.py", line 207, in _call_s3
return await method(**additional_kwargs)
File "/usr/local/lib/python3.7/site-packages/aiobotocore/client.py", line 121, in _make_api_call
operation_model, request_dict, request_context)
File "/usr/local/lib/python3.7/site-packages/aiobotocore/client.py", line 140, in _make_request
return await self._endpoint.make_request(operation_model, request_dict)
File "/usr/local/lib/python3.7/site-packages/aiobotocore/endpoint.py", line 90, in _send_request
exception):
File "/usr/local/lib/python3.7/site-packages/aiobotocore/endpoint.py", line 199, in _needs_retry
caught_exception=caught_exception, request_dict=request_dict)
File "/usr/local/lib/python3.7/site-packages/aiobotocore/hooks.py", line 29, in _emit
response = handler(**kwargs)
File "/usr/local/lib/python3.7/site-packages/botocore/utils.py", line 1225, in redirect_from_error
new_region = self.get_bucket_region(bucket, response)
File "/usr/local/lib/python3.7/site-packages/botocore/utils.py", line 1283, in get_bucket_region
headers = response['ResponseMetadata']['HTTPHeaders']
TypeError: 'coroutine' object is not subscriptable
sys:1: RuntimeWarning: coroutine 'AioBaseClient._make_api_call' was never awaited
Could there be an issue with the s3fs as this and pandas seems to be the only packages that received updates, but I couldn't find anything in the pandas' changelogs related to this?
Dask/s3fs team has acknowledged this to be a bug. This Github issue suggests that the aiobotocore is unable to get the region_name for the S3 bucket.
If you are facing the same issue, either consider downgrading s3fs to 0.4.2 or else try setting the environment variable AWS_DEFAULT_REGION as a workaround.
Edit: It has been fixed in the latest release of aiobotocore=1.1.1. Upgrade your aiobotocore and s3fs if you are facing the same issue.
It is full log
File "/home/workspace/viv_data_handler/s3manager/s3_downloader.py", line 18, in download_file
self.resource.meta.client.download_file(bucket, key, dest)
File "/usr/local/lib/python3.5/site-packages/boto3/s3/inject.py", line 172, in download_file
extra_args=ExtraArgs, callback=Callback)
File "/usr/local/lib/python3.5/site-packages/boto3/s3/transfer.py", line 307, in download_file
future.result()
File "/usr/local/lib/python3.5/site-packages/s3transfer/futures.py", line 73, in result
return self._coordinator.result()
File "/usr/local/lib/python3.5/site-packages/s3transfer/futures.py", line 233, in result
raise self._exception
File "/usr/local/lib/python3.5/site-packages/s3transfer/tasks.py", line 255, in _main
self._submit(transfer_future=transfer_future, **kwargs)
File "/usr/local/lib/python3.5/site-packages/s3transfer/download.py", line 353, in _submit
**transfer_future.meta.call_args.extra_args
File "/usr/local/lib/python3.5/site-packages/botocore/client.py", line 314, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.5/site-packages/botocore/client.py", line 599, in _make_api_call
operation_model, request_dict)
File "/usr/local/lib/python3.5/site-packages/botocore/endpoint.py", line 148, in make_request
return self._send_request(request_dict, operation_model)
File "/usr/local/lib/python3.5/site-packages/botocore/endpoint.py", line 177, in _send_request
success_response, exception):
File "/usr/local/lib/python3.5/site-packages/botocore/endpoint.py", line 273, in _needs_retry
caught_exception=caught_exception, request_dict=request_dict)
File "/usr/local/lib/python3.5/site-packages/botocore/hooks.py", line 227, in emit
return self._emit(event_name, kwargs)
File "/usr/local/lib/python3.5/site-packages/botocore/hooks.py", line 210, in _emit
response = handler(**kwargs)
File "/usr/local/lib/python3.5/site-packages/botocore/retryhandler.py", line 183, in __call__
if self._checker(attempts, response, caught_exception):
File "/usr/local/lib/python3.5/site-packages/botocore/retryhandler.py", line 251, in __call__
caught_exception)
File "/usr/local/lib/python3.5/site-packages/botocore/retryhandler.py", line 277, in _should_retry
return self._checker(attempt_number, response, caught_exception)
File "/usr/local/lib/python3.5/site-packages/botocore/retryhandler.py", line 317, in __call__
caught_exception)
File "/usr/local/lib/python3.5/site-packages/botocore/retryhandler.py", line 223, in __call__
attempt_number, caught_exception)
File "/usr/local/lib/python3.5/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
raise caught_exception
File "/usr/local/lib/python3.5/site-packages/botocore/endpoint.py", line 222, in _get_response
proxies=self.proxies, timeout=self.timeout)
File "/usr/local/lib/python3.5/site-packages/botocore/vendored/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.5/site-packages/botocore/vendored/requests/adapters.py", line 431, in send
raise SSLError(e, request=request)
botocore.vendored.requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:719)
Hello I am developing a program using S3 and I have a problem
botocore.vendored.requests.exceptions.SSLError error occurred when I try download data using boto3 from S3
It works when another server use the same code. and It works when use aws cli on the same server(boto3 failed server).
I don't know how to solve it.
please help me. thank you.
For any boto3.Session.client or boto3.Session.resource object, you can pass the verify=False parameter to skip SSL certificate verification. For example, when you create the client or resource, do this:
import boto3
boto3_session = boto3.Session(profile_name=some_profile_you_configured)
s3_resource = boto3_session.resource(service_name='s3', verify=False)
s3_client = boto3_session.client(service_name='s3', verify=False)
[insert your code here using the s3_resource or s3_client objects]
The boto3 documentation is your best friend here.
I have configured the awscli in my linux server. I have set the credential file. Jenkins is also installed in it.
I have a python code for placing a file from the linux server to AWS s3 bucket.
When I run the script in the linux server it works fine. I get the correct output.
But when I run the same script for a job in jenkins execute shell, it throws the following error:
"File "/usr/lib/python2.7/site-packages/boto3/resources/factory.py", line 520, in do_action
response = action(self, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/boto3/resources/action.py", line 83, in __call__
response = getattr(parent.meta.client, operation_name)(**params)
File "/usr/lib/python2.7/site-packages/botocore/client.py", line 312, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/lib/python2.7/site-packages/botocore/client.py", line 588, in _make_api_call
operation_model, request_dict)
File "/usr/lib/python2.7/site-packages/botocore/endpoint.py", line 141, in make_request
return self._send_request(request_dict, operation_model)
File "/usr/lib/python2.7/site-packages/botocore/endpoint.py", line 166, in _send_request
request = self.create_request(request_dict, operation_model)
File "/usr/lib/python2.7/site-packages/botocore/endpoint.py", line 150, in create_request
operation_name=operation_model.name)
File "/usr/lib/python2.7/site-packages/botocore/hooks.py", line 227, in emit
return self._emit(event_name, kwargs)
File "/usr/lib/python2.7/site-packages/botocore/hooks.py", line 210, in _emit
response = handler(**kwargs)
File "/usr/lib/python2.7/site-packages/botocore/signers.py", line 90, in handler
return self.sign(operation_name, request)
File "/usr/lib/python2.7/site-packages/botocore/signers.py", line 154, in sign
auth.add_auth(request)
File "/usr/lib/python2.7/site-packages/botocore/auth.py", line 408, in add_auth
super(S3SigV4Auth, self).add_auth(request)
File "/usr/lib/python2.7/site-packages/botocore/auth.py", line 340, in add_auth
raise NoCredentialsError
botocore.exceptions.NoCredentialsError: Unable to locate credentials
Build step 'Execute shell' marked build as failure
Finished: FAILURE"
Any solution????
I'm trying to run two instances of OpenERP on the same machine, but whenever I launch the second one it crashes. It is the same user that is launching the two instances of OpenERP and in the same postgresDB but, I launch each instance of OpenERP with different users in the database. Here is the log file I have no idea what is going on here:
Traceback (most recent call last):
File "/home/user/lib/python2.7/Werkzeug-0.9.4-py2.7.egg/werkzeug/serving.py", line 177, in run_wsgi
execute(self.server.app)
File "/home/user/lib/python2.7/Werkzeug-0.9.4-py2.7.egg/werkzeug/serving.py", line 165, in execute
application_iter = app(environ, start_response)
File "/home/user/openerp-8.0-acmlpsc/openerp8/openerp/service/server.py", line 273, in app
return self.app(e, s)
File "/home/user/openerp-8.0-acmlpsc/openerp8/openerp/service/wsgi_server.py", line 215, in application
return application_unproxied(environ, start_response)
File "/home/user/openerp-8.0-acmlpsc/openerp8/openerp/service/wsgi_server.py", line 201, in application_unproxied
result = handler(environ, start_response)
File "/home/user/openerp-8.0-acmlpsc/openerp8/openerp/http.py", line 1097, in __call__
return self.dispatch(environ, start_response)
File "/home/user/openerp-8.0-acmlpsc/openerp8/openerp/http.py", line 1074, in __call__
return self.app(environ, start_wrapped)
File "/home/user/lib/python2.7/Werkzeug-0.9.4-py2.7.egg/werkzeug/wsgi.py", line 579, in __call__
return self.app(environ, start_response)
File "/home/user/openerp-8.0-acmlpsc/openerp8/openerp/http.py", line 1226, in dispatch
ir_http = request.registry['ir.http']
File "/home/user/openerp-8.0-acmlpsc/openerp8/openerp/http.py", line 220, in registry
return openerp.modules.registry.RegistryManager.get(self.db) if self.db else None
File "/home/user/openerp-8.0-acmlpsc/openerp8/openerp/modules/registry.py", line 269, in get
update_module)
File "/home/user/openerp-8.0-acmlpsc/openerp8/openerp/modules/registry.py", line 299, in new
openerp.modules.load_modules(registry._db, force_demo, status, update_module)
File "/home/user/openerp-8.0-acmlpsc/openerp8/openerp/modules/loading.py", line 335, in load_modules
force, status, report, loaded_modules, update_module)
File "/home/user/openerp-8.0-acmlpsc/openerp8/openerp/modules/loading.py", line 241, in load_marked_modules
graph.add_modules(cr, module_list, force)
File "/home/user/openerp-8.0-acmlpsc/openerp8/openerp/modules/graph.py", line 99, in add_modules
info = openerp.modules.module.load_information_from_description_file(module)
File "/home/user/openerp-8.0-acmlpsc/openerp8/openerp/modules/module.py", line 180, in load_information_from_description_file
terp_file = opj(mod_path, '__openerp__.py')
File "/usr/local/lib/python2.7/posixpath.py", line 77, in join
elif path == '' or path.endswith('/'):
AttributeError: 'bool' object has no attribute 'endswith'
I hope you can help me.
This may be due to port clash. Try running the instances on different ports, say one on 8069 and the other on 8070.
Try opening the instances in separate Icognito Window (while using chrome), or Private Browsing Window(while using mozilla), because there may be case of one instance overwriting the sessions of another openerp instance.
If this doesn't solve your problem, then I suggest you to pull the latest code of OpenERP8, as still it is not stable.
Hope this helps, do update me with your feedback..