I am trying to write unit test for some of Python classes I have created. I have created a class for wrapping s3 functionalities, and in that class I'm initializing boto3 s3 client.
class S3_Client:
def __init__(self, bucket_name):
self.s3 = boto3.client("s3", aws_access_key_id=e_config["aws_access_key"], aws_secret_access_key=e_config["aws_secret_key"])
self.bucket_name = bucket_name
def fetch(self, key):
response = self.s3.get_object(Bucket=self.bucket_name, Key=key)
return self.__prepare_file_info(response, key) # return formatted response
I would like to test method fetch with mocked response from self.s3.get_object. This is my test class:
import unittest
from .aws_s3_service import S3_Client # class I want to test
import boto3
from botocore.stub import Stubber
class TestS3_Client(unittest.TestCase):
def setUp(self):
self.client = boto3.client('s3')
self.stubber = Stubber(self.client)
def test_fetch(self):
get_object_response = {...} # hardcoded response
self.stubber.add_response('get_object', get_object_response, {"Bucket": "test_bucket", "Key": "path/to/file/test_file.txt"})
with self.stubber:
client = S3_Client("test_bucket")
result = client.fetch("path/to/file/test_file.txt")
The stubber is not actually injected into S3_Client, a real call to S3 is made. How do I inject the stubber? Any help is appreciated, thanks.
You need to make S3_Client accept a client object in a constructor argument.
In this way in your tests you can create a client, stub it, then inject it to S3_Client as a parameter.
If you don't like having to always create that client outside of the class, you can make it an optional argument, and create an instance in __init__ if none was passed:
class S3_Client:
def __init__(self, bucket_name, s3=None):
if s3 is None:
self.s3 = boto3.client("s3", aws_access_key_id=e_config["aws_access_key"], aws_secret_access_key=e_config["aws_secret_key"])
else:
self.s3 = s3
self.bucket_name = bucket_name
...
In the code of your test you would then say: client = S3_Client("test_bucket", self.client).
Related
I'm trying to add metrics to external services with aioprometheus in an app built with FastAPI. Here is a simplified example of what I'm trying to achieve.
Say I have a wrapper App class as such:
from aioprometheus import Registry, Counter, Histogram
from fastapi import FastAPI
class App:
def __init__(self, ...):
self.registry = Registry()
self.counter = Counter(
name="counts", doc="request counts"
)
self.latency = Histogram(
name="latency",
doc="request latency",
buckets=[0.1, 0.5, 1, 1.5, 2]
)
self.app = FastAPI()
self._metrics()
def _metrics(self):
# Counter metrics
#self.app.middleware("http")
async def counter_metrics(request, call_next):
response = await call_next(request)
self.counter.inc(
{"path": str(request.url.path), "status": response.status_code}
)
return response
# Latency metrics
#self.app.middleware("http")
async def latency_metrics(request, call_next):
start = time.time()
response = await call_next(request)
total_duration = time.time() - start
self.latency.observe(
{"path": str(request.url.path)}, total_duration
)
return response
#self.app.on_event("startup")
async def startup():
self.app.include_router(some_router(...))
self.registry.register(self.counter)
self.registry.register(self.latency)
Basically, I have Registry, Counter, and Histogram initiated. In _metrics, I have Counter and Histogram specific logics that are later added to Registry. This will do its magic and catch the metrics when an endpoint in some_router is called (this is good! I would want to keep this, as well as having the external service metrics).
However, say I call an external service from some_router as such:
from fastapi import APIRouter
def some_router():
router = APIRouter()
#router.get("/some_router")
async def some_router():
response = await external_service()
return response
return router
In this case, how would I add metrics specifically to external_service, i.e., Latency of this specific external service?
As per the documentation, you would need to attach your metrics to the app instance using the generic app.state attribute (see the implementation of Starlette's State class as well), so they can easily be accessed in the route handler—as metrics are often created in a different module than where they are used (as in your case). Thus, you could use the following in your App class, after instantiating the metrics:
self.app.state.registry = registry
self.app.state.counter = counter
self.app.state.latency = latency
In your routers module, you could get the app instance using the Request object, as described here and here, and then use it to get the metrics instances (as shown below), which will let you add metrics to your external_service:
from fastapi import Request
...
#router.get("/some_router")
async def some_router(request: Request):
registry = request.app.state.registry
counter = request.app.state.counter
latency = request.app.state.latency
response = await external_service()
return response
I'm trying to write a test for my lambda handler, which uses boto3 to download a file from a bucket and store it locally:
s3_resource = boto3.resource('s3')
TEMP_FILE = '/tmp/file.csv'
def lambda_handler(event, context):
bucket_name = event['detail']['bucket']['name']
file_name = event['detail']['object']['key']
s3_resource.Bucket(bucket_name).download_file(Key=file_name, Filename=TEMP_FILE)
Because I don't want to actually interact with s3 in my test, I created a dummy s3_upload_event to pass into the function call. I also used moto to create a mocked s3 bucket and put some dummy test_data in it, as well as a mocked iam user with s3 permissions:
TEST_BUCKET = "test_bucket_name"
S3_TEST_FILE_KEY = 'path/to/test.csv'
#pytest.fixture
def s3_upload_event():
return {"detail":{"bucket":{"name": TEST_BUCKET}, "object": {"key": S3_TEST_FILE_KEY}}}
#pytest.fixture
def context():
return object()
#pytest.fixture
def test_data():
return b'col_1,col_2\n1,2\n3,4\n'
#pytest.fixture
#mock_iam
def mock_user(user_name="test-user"):
# create user
client = boto3.client("iam", region_name="us-west-2")
client.create_user(UserName=user_name)
# create and attach policy
policy_document = {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": ["s3:*", "s3-object-lambda:*"],
"Resource": "*"
}]
}
policy_arn = client.create_policy(
PolicyName="test",
PolicyDocument=json.dumps(policy_document))["policy"]["Arn"]
client.attach_user_policy(UserName=user_name, PolicyArn=policy_arn)
# Return access keys
yield client.create_access_key(UserName=user_name)["AccessKey"]
#pytest.fixture
#mock_s3
def mock_s3(test_data, mock_user):
s3 = boto3.client(
"s3",
region_name="us-west-2",
aws_access_key_id=mock_user["AccessKeyId"],
aws_secret_access_key=mock_user["SecretAccessKey"])
s3.create_bucket(Bucket=TEST_BUCKET)
s3.put_object(Bucket=TEST_BUCKET, Key=S3_TEST_FILE_KEY, Body=test_data)
yield s3
I inject those mocked fixtures to my test as follows:
class TestLambdaHandler:
def test_lambda_handler(self, mock_user, mock_s3, s3_upload_event, context):
response = lambda_handler(event = s3_upload_event, context = context)
assert response["statusCode"] == 200
But when I run the test botocore throws an exception when it reaches this line of the code: s3_resource.Bucket(bucket_name).download_file(Key=file_name, Filename=TEMP_FILE):
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden
When I googled this error, it seems that this has to do with missing IAM permissions. The PolicyDocument I'm using for my mocked user is the same as the actual policy I'm using in the code, so I don't see why it would be able to download the file in real life but fail in the test. Is there anything I'm missing in my mocked user?
I managed to get the mocks working after re-reading the setup instructions for Moto (http://docs.getmoto.org/en/latest/docs/getting_started.html), there were a few issues with the code above.
Mocking the iam credentials and user was not necessary, but I did need to mock AWS credentials and pass them to the s3 mock.
I changed the decorator (#s3_mock) to the context manager pattern on the s3 mock. My understanding is that you should not use the decorator for a custom fixture that's applying the mocking.
Re-named my mock from mock_s3 to just s3 to avoid confusion with the moto-vended fixture.
Updated the s3 fixture to create the mocked client only
Updated fixtures:
#pytest.fixture(scope='function')
def aws_credentials():
"""Mocked AWS Credentials for moto."""
os.environ['AWS_ACCESS_KEY_ID'] = 'testing'
os.environ['AWS_SECRET_ACCESS_KEY'] = 'testing'
os.environ['AWS_SECURITY_TOKEN'] = 'testing'
os.environ['AWS_SESSION_TOKEN'] = 'testing'
#pytest.fixture(scope='function')
def s3(aws_credentials):
with mock_s3():
yield boto3.client('s3', region_name="us-west-2")
I then updated the test to take in the s3 fixture and created the test "bucket" and put the test "object" as part of the test setup.
VALID_TEST_FILE_KEY = "path/to/test.csv"
VALID_DATA = "tests/data/valid_data.csv"
TEST_BUCKET = "test_bucket_name"
def test_lambda_handler(self, s3_upload_valid_file, context, s3):
#arrange
self._create_bucket_and_add_file(s3, VALID_DATA, VALID_TEST_FILE_KEY)
#act
response = lambda_handler(event = s3_upload_valid_file, context = context)
assert response["statusCode"] == 200
### Helper Method for Test ###
def _create_bucket_and_add_file(self, s3, data_file, key):
with open(data_file, 'r') as f:
test_data = f.read()
s3.create_bucket(Bucket=TEST_BUCKET, CreateBucketConfiguration={'LocationConstraint': "us-west-2"})
s3.put_object(Bucket=TEST_BUCKET, Key=key, Body=test_data)
I have code that uses zeep to create a soap client. My server does not return the wsdl file but i have it locally.
The sycronous version works looks like this:
import uuid
from os import path
import structlog
import zeep
logger = structlog.get_logger(__name__)
class SyncClient(object):
def __init__(self, ip_address: str):
self.ip_address = ip_address
self.port = 8080
self.soap_client = None
self.corrupt_timeseries_files = []
self.id = uuid.uuid4()
def connect_soap_client(self):
this_files_dir = path.dirname(path.realpath(__file__))
wsdl = 'file://{}'.format(path.join(this_files_dir, 'SOAPInterface.wsdl'))
transport = zeep.Transport(timeout=5, operation_timeout=3)
client = zeep.Client(wsdl, transport=transport)
location = "http://{}:{}".format(self.ip_address, str(self.port))
self.soap_client = client.create_service("{urn:webservices}SOAPInterface", location)
Then the asyc Client looks like this:
class AsyncClient(object):
def __init__(self, ip_address: str):
self.ip_address = ip_address
self.port = 8080
self.soap_client: zeep.client.Client = None
self.corrupt_timeseries_files = []
self.id = uuid.uuid4()
def connect_soap_client(self):
this_files_dir = path.dirname(path.realpath(__file__))
wsdl = 'file://{}'.format(path.join(this_files_dir, 'SOAPInterface.wsdl'))
transport = zeep.transports.AsyncTransport(timeout=5, wsdl_client=wsdl, operation_timeout=3)
client = zeep.AsyncClient(wsdl, transport=transport)
location = "http://{}:{}".format(self.ip_address, str(self.port))
self.soap_client = client.create_service("{urn:webservices}SOAPInterface", location)
I have seen that the documentation of zeep states that the file loading is syncronous. But I don't get how I could create a async client when I have a local file...
Error message when i run my code in tests:
httpx.UnsupportedProtocol: Unsupported URL protocol 'file'
After debugging my way through the zeep and httpx source, I have found that the solution is actually quite simple:
Don't specify file://{path}, just specify {path}. Then the WSDL loads fine.
I want to upload a image from front end to google storage using javascript ajax functionality. I need a presigned url that the server would generate which would provide authentication to my frontend to upload a blob.
How can I generate a presigned url when using my local machine.
Previously for aws s3 I would do :
pp = s3.generate_presigned_post(
Bucket=settings.S3_BUCKET_NAME,
Key='folder1/' + file_name,
ExpiresIn=20 # seconds
)
When generating a signed url for a user to just view a file stored on google storage I do :
bucket = settings.CLIENT.bucket(settings.BUCKET_NAME)
blob_name = 'folder/img1.jpg'
blob = bucket.blob(blob_name)
url = blob.generate_signed_url(
version='v4',
expiration=datetime.timedelta(minutes=1),
method='GET')
Spent 100$ on google support and 2 weeks of my time to finally find a solution.
client = storage.Client() # works on app engine standard without any credentials requirements
But if you want to use generate_signed_url() function then you need service account Json key.
Every app engine standard has a default service account. ( You can find it in IAM/service account). Create a key for that default sv account and download the key ('sv_key.json') in json format. Store that key in your Django project right next to app.yaml file. Then do the following :
from google.cloud import storage
CLIENT = storage.Client.from_service_account_json('sv_key.json')
bucket = CLIENT.bucket('bucket_name_1')
blob = bucket.blob('img1.jpg') # name of file to be saved/uploaded to storage
pp = blob.generate_signed_url(
version='v4',
expiration=datetime.timedelta(minutes=1),
method='POST')
This will work on your local machine and GAE standard. WHen you deploy your app to GAE, sv_key.json also gets deployed with Django project and hence it works.
Hope it helps you.
Editing my answer as I didn't understand the problem you were facing.
Taking a look at the comments thread in the question, as #Nick Shebanov stated, there's one possibility to accomplish what are you trying to when using GAE with flex environment.
I have been trying to do the same with GAE Standard environment with no luck so far. At this point, I would recommend opening a feature request at the public issue tracker so this gets somehow implemented.
Create a service account private key and store it in SecretManager (SM).
In settings.py retrieve that key from SecretManager and store it in a constant - SV_ACCOUNT_KEY
Override Client() class func from_service_account_json() to take json key content instead of a path to json file. This way we dont have to have a json file in our file system (locally, cloudbuild or in GAE). we can just get private key contents from SM anytime anywhere.
settings.py
secret = SecretManager()
SV_ACCOUNT_KEY = secret.access_secret_data('SV_ACCOUNT_KEY')
signed_url_mixin.py
import datetime
import json
from django.conf import settings
from google.cloud.storage.client import Client
from google.oauth2 import service_account
class CustomClient(Client):
#classmethod
def from_service_account_json(cls, json_credentials_path, *args, **kwargs):
"""
Copying everything from base func (from_service_account_json).
Instead of passing json file for private key, we pass the private key
json contents directly (since we cannot save a file on GAE).
Since its not properly written, we cannot just overwrite a class or a
func, we have to rewrite this entire func.
"""
if "credentials" in kwargs:
raise TypeError("credentials must not be in keyword arguments")
credentials_info = json.loads(json_credentials_path)
credentials = service_account.Credentials.from_service_account_info(
credentials_info
)
if cls._SET_PROJECT:
if "project" not in kwargs:
kwargs["project"] = credentials_info.get("project_id")
kwargs["credentials"] = credentials
return cls(*args, **kwargs)
class _SignedUrlMixin:
bucket_name = settings.BUCKET_NAME
CLIENT = CustomClient.from_service_account_json(settings.SV_ACCOUNT_KEY)
exp_min = 4 # expire minutes
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.bucket = self.CLIENT.bucket(self.bucket_name)
def _signed_url(self, file_name, method):
blob = self.bucket.blob(file_name)
signed_url = blob.generate_signed_url(
version='v4',
expiration=datetime.timedelta(minutes=self.exp_min),
method=method
)
return signed_url
class GetSignedUrlMixin(_SignedUrlMixin):
"""
A GET url to view file on CS
"""
def get_signed_url(self, file_name):
"""
:param file_name: name of file to be retrieved from CS.
xyz/f1.pdf
:return: GET signed url
"""
method = 'GET'
return self._signed_url(file_name, method)
class PutSignedUrlMixin(_SignedUrlMixin):
"""
A PUT url to make a put req to upload a file to CS
"""
def put_signed_url(self, file_name):
"""
:file_name: xyz/f1.pdf
"""
method = 'PUT'
return self._signed_url(file_name, method)
Sorry for the noob question about classes. I'm trying to assign a soap client to a variable inside a class function and then access that variable in other class functions. I don't have any arguments to pass to the setup_client() function.
In the following example code, how do I make self.client accessible outside setup_client() so that I can use it in use_client(), and ditto making self.response available outside use_client()
class soap_call(self):
def __init__(self):
# What goes here?
self.client = # what?
self.response = # what?
def setup_client(self):
credentials = {'username': 'stuff', 'password': 'stuff'}
url = 'stuff'
t = HttpAuthenticated(**credentials)
self.client = suds.client.Client(url, transport=t)
def use_client(self):
self.response = self.client.service.whatever
print self.response
I quickly realized that if I add an optional client argument (self, client=None) to the class definition and include self.client = client, then I get a None type error when trying to use it in my functions.
I realize I just have a lack of understanding of classes. I've done some general reading up on classes but haven't come across any specific examples that describe what I'm dealing with.
I'd go with None in both cases because logically speaking, none of them exists at the time of object instantiation. It also allows you to do some sanity checking of your logic, e.g.
class SoapCall(object):
def __init__(self):
self.client = None
self.response = None
def setup_client(self):
credentials = {'username': 'stuff', 'password': 'stuff'}
url = 'stuff'
t = HttpAuthenticated(**credentials)
if self.client is None:
self.client = suds.client.Client(url, transport=t)
def use_client(self):
if self.client is None:
self.client = self.setup_client()
self.response = self.client.service.whatever
print self.response
It's fine to leave the client unspecified when you first create the instance, but then you need to be sure to call setup_client before you call use_client.