I have a deployment package in the following structure:
my-project.zip
--- my-project.py
------ lambda_handler()
Then I define the handler path in configuration file
my-project.lambda_handler
Get the error:
'handler' missing on module
Can not understand that
There are some issues occurring this error.
Issue#1:
The very first issue you’re gonna run into is if you name the file incorrectly, you get this error:
Unable to import module 'lambda_function': No module named lambda_function
If you name the function incorrectly you get this error:
Handler 'handler' missing on module 'lambda_function_file': 'module'
object has no attribute 'handler'
On the dashboard, make sure the handler field is entered as function_filename.actual_function_name and make sure they match up in your deployment package.
If only the messages were a bit more instructive that would have been a simpler step.
Resource Link:
No lambda_function?
Issue#2:
adrian_praja has solved the issue in aws forum. He answered the following
I belive your index.js should contain
exports.createThumbnailHandler = function(event, context) {}
Issue#3:
Solution: Correctly specify the method call
This happens when the specification of the method called by node.js is incorrect in Lambda's setting.
Please review the specification of the method to call.
In the case of the above error message, I attempted to call the handler method of index.js, but the corresponding method could not be found.
The processing to call is set with "Handler" on the configuration tab.
Below is an example of setting to call the handler method of index.js.
Resource Link:
http://qiita.com/kazuqqfp/items/ac8d93918d0030b31aad
AWS Lambda Function is returning Handler 'handler' missing on module 'index'
I had this issue and had to make sure I had a function called handler in my file, e.g.:
# this just takes whatever is sent to the api gateway and sends it back
def handler(event, context):
try:
return response(event, 200)
except Exception as e:
return response('Error' + e.message, 400)
def response(message, status_code):
return message
Related
I'm trying to get S3 hook in Apache Airflow using the Connection object.
It looks like this:
class S3ConnectionHandler:
def __init__():
# values are read from configuration class, which loads from env. variables
self._s3 = Connection(
conn_type="s3",
conn_id=config.AWS_CONN_ID,
login=config.AWS_ACCESS_KEY_ID,
password=config.AWS_SECRET_ACCESS_KEY,
extra=json.dumps({"region_name": config.AWS_DEFAULT_REGION}),
)
#property
def s3(self) -> Connection:
return get_live_connection(self.logger, self._s3)
#property
def s3_hook(self) -> S3Hook:
return self.s3.get_hook()
I get an error:
Broken DAG: [...] Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/connection.py", line 282, in get_hook
return hook_class(**{conn_id_param: self.conn_id})
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/amazon/aws/hooks/base_aws.py", line 354, in __init__
raise AirflowException('Either client_type or resource_type must be provided.')
airflow.exceptions.AirflowException: Either client_type or resource_type must be provided.
Why does this happen? From what I understand the S3Hook calls the constructor from the parent class, AwsHook, and passes the client_type as "s3" string. How can I fix this?
I took this configuration for hook from here.
EDIT: I even get the same error when directly creating the S3 hook:
#property
def s3_hook(self) -> S3Hook:
#return self.s3.get_hook()
return S3Hook(
aws_conn_id=config.AWS_CONN_ID,
region_name=self.config.AWS_DEFAULT_REGION,
client_type="s3",
config={"aws_access_key_id": self.config.AWS_ACCESS_KEY_ID, "aws_secret_access_key": self.config.AWS_SECRET_ACCESS_KEY}
)
``
If youre using Airflow 2
please refer to the new documentation - it can be kind of tricky as most of the google searches redirect you to the old ones.
In my case I was using the AwsHook and had to switch to AwsBaseHook as it seems to be the only and correct one for version 2. I've had to switch the import path as well, now aws stuff isnt on contrib anymore its under providers
And as you can see on the new documentation you can pass either client_type ou resource_type as a AwsBaseHook parameter, depending on the one you want to use. Once you do that your problem should be solved
No other answers worked, I couldn't get around this. I ended up using boto3 library directly, which also gave me more low-level flexibility that Airflow hooks lacked.
First of all , I suggest that you create a S3 connection , for this you must go the path Admin >> Connections
After that and assuming that you want to load a file into S3 Bucket, you can code :
def load_csv_S3():
# Send to S3
hook = S3Hook(aws_conn_id="s3_conn")
hook.load_file(
filename='/write_your_path_file/filename.csv',
key='filename.csv',
bucket_name="BUCKET_NAME",
replace=True,
)
Finally, you can check all the functions of S3Hook 👉 HERE
What has worked for me, in case it helps someone, in my answer to a similar post: https://stackoverflow.com/a/73652781/4187360
Let's say I want to display my own 404 & 500 pages, I've found 2 possibilities so far:
1: Using cherrypy.config.update
def error_page_404(status, message, traceback, version):
return ('Error 404 Page not found')
def error_page_500(status, message, traceback, version):
return ('Error:')
cherrypy.config.update({'error_page.404': error_page_404, 'error_page.500': error_page_500})
Using _cp_config:
from cherrypy import _cperror
def handle_error():
cherrypy.response.status = 500
cherrypy.log("handle_error() called. Alarm!", "WEBAPP")
cherrypy.response.body = ['Sorry, an error occured. The admin has been notified']
error = _cperror.format_exc()
def error_page(status, message, traceback, version):
cherrypy.log("error_page() called. Probably not very important.", "WEBAPP")
return "Sorry, an error occured."
class Root:
_cp_config = {
'error_page.default': error_page,
'request.error_response': handle_error
}
but is there a difference or a suggestion which is preferable to use?
request.error_response allows you to set a handler for processing of some unexpected errors, like your own exceptions raised from HTTP handlers.
The callable that you'll set for this option will receive no arguments at all and you'll have to inspect sys.exc_info() for the details, to find out what happened.
You'll also have to set cherrypy.response.status and cherrypy.response.body by yourself, explicitly in your error handler.
If you want to modify the error response for HTTP error codes (when instances of cherrypy.HTTPError are raised, like raise cherrypy.NotFound), you can use error_page.default (catch-all) or error_page.404 (error-specific) for handling those errors.
error_page options support both file path and callable values. In case of using a file path, the HTML template file can use the following substitution patterns: %(status)s, %(message)s, %(traceback)s, and %(version)s.
If you opt-in to using a function, it'll receive those as arguments (callback(status, message, traceback, version)). The return value of this callable is then used HTTP response payload.
As you can see, these approaches have different implications and different levels of flexibility and usability. Choose whatever works for you. Internally, the default request.error_response uses error_page settings to figure out what to return. So if you redefine request.error_response, it'll not use error_page.* settings unless you explicitly make it do so.
See the docstring with some explanation here.
I have functions in a local_code.py file that I would like to pass to workers through dask. I've seen answers to questions on here saying that this can be done using the upload_file() function, but I can't seem to get it working because I'm still getting a ModuleNotFoundError.
The relevant part of the code is as follows.
from dask.distributed import Client
from dask_jobqueue import SLURMCluster
from local_code import *
helper_file = '/absolute/path/to/local_code.py'
def main():
with SLURMCluster(**slurm_params) as cluster:
cluster.scale(n_workers)
with Client(cluster) as client:
client.upload_file(helper_file)
mapping = client.map(myfunc, data)
client.gather(mapping)
if __name__ == '__main__':
main()
Note, myfunc is imported from local_code, and there's no error importing it to map. The function myfunc also depends on other functions that are defined in local_code.
With this code, I'm still getting this error
distributed.protocol.pickle - INFO - Failed to deserialize b'\x80\x04\x95+\x00\x00\x00\x00\x00\x00\x00\x8c\x11local_code\x94\x8c\x$
Traceback (most recent call last):
File "/home/gallagher.r/.local/lib/python3.7/site-packages/distributed/protocol/pickle.py", line 61, in loads
return pickle.loads(x)
ModuleNotFoundError: No module named 'local_code'
Using upload_file() seems so straightforward that I'm not sure what I'm doing wrong. I must have it in the wrong place or not be understanding correctly what is passed to it.
I'd appreciate any help with this. Please let me know if you need any other information or if there's anything else I can supply from the error file.
The upload_file method only uploads the file to the currently available workers. If a worker arrives after you call upload_file then that worker won't have the provided file.
If your situation the easiest thing to do is probably to wait until all of the workers arrive before you call upload file
cluster.scale(n)
with Client(cluster) as client:
client.wait_for_workers(n)
client.upload_file(...)
Another option when you have workers going in/out is to use the Client.register_worker_callbacks to hook into whenever a new worker is registered/added. The one caveat is you will need to serialize your file(s) in the callback partial:
fname = ...
with open(fname, 'rb') as f:
data = f.read()
client.register_worker_callbacks(
setup=functools.partial(
_worker_upload, data=data, fname=fname,
)
)
def _worker_upload(dask_worker, *, data, fname):
dask_worker.loop.add_callback(
callback=dask_worker.upload_file,
comm=None, # not used
filename=fname,
data=data,
load=True)
This will also upload the file the first time the callback is registered so you can avoid calling client.upload_file entirely.
I'm trying to get gae-sessions working, but am having an issue.
I've logged inside of appengine_config.py and inside of webapp_add_wsgi_middleware and it is being called.
I've logged inside the libs __init__.py and it's running __init__, __call__ and my_start_response inside SessionMiddleware without issue.
My problem is that the second I try to actually use the session with:
session = get_current_session()
I get this:
session = get_current_session()
File "gaesessions\__init__.py", line 38, in get_current_session
return _tls.current_session
AttributeError: 'thread._local' object has no attribute 'current_session'
Well the problem seems to be that I can only call get_current_session within a get request handler. If I try to call a function in another module from the get request handler, that itself requests the session, I get the above error.
I have a unit test that's failing in an assertion that passes in another test in the same test case class.
Here's the passing test:
def test_home(self):
c = Client()
resp = c.get('/')
self.assertEqual(resp.status_code, 200)
self.assertTrue('a_formset' in resp.context)
Here's the failing test:
def test_number_initial_number_of_forms(self):
c = Client()
resp = c.get('/')
self.assertEqual(resp.context['a_formset'].total_form_count(), 1)
In the second test, I get the error TypeError: 'NoneType' object has no attribute '__getitem__'.
If I execute the second test as
def test_number_initial_number_of_forms(self):
c = Client()
resp = c.get('/')
self.assertTrue('a_formset' in resp.context)
self.assertEqual(resp.context['a_formset'].total_form_count(), 1)
I get the error TypeError: argument of type 'NoneType' is not iterable. I've confirmed via print statements in the second test that the response.content contains the page I expect to get, that the status code is correct, and that the template is correct. But the response's context is consistently None in the second test.
I'm running my Django unit tests through the standard "python manage.py test ..." interface, so I don't believe I'm running into the "context is empty from the shell" issue.
What's going on with this?
Edit:
If I add print type(resp.context['a_formset']) to each test, for the working test I get <class 'django.forms.formsets.AFormFormSet'>. For the non-working test, I get TypeError: 'NoneType' object has no attribute '__getitem__' again.
It's because you ran into some error, exited the shell and restarted it.
But you forgot to start environment...
from django.test.utils import setup_test_environment
>>> setup_test_environment()
That was my problem. Hope it works...
Today I run into the same issue. The second test gets same page has nothing in response.context
I made a research and found that
1) test client uses signals to populate context,
2) my view method is not called for the second test
I turned on a debugger and found that the guilty one is 'Cache middleware'. Knowing that I found this ticket and this SO question (the latter has a solution).
So, in short: the second request is served from cache, not from a view, thus a view is not executed and test-client doesn't get the signal and have no ability to populate context.
I can not disable cache middleware for my project, so I added next hack-lines into my settings:
if 'test' in sys.argv:
CACHE_MIDDLEWARE_SECONDS = 0
Hope this helps someone
You can also clear cache manually by calling cache.clear() inside a test method:
from django.core.cache import cache
import pytest
class TestPostView:
#pytest.mark.django_db(transaction=True)
def test_index_post(self, client, post):
cache.clear()
response = client.get('/')