My pyramid app permits users to, from www.domain.com to create new html pages in an amazon s3 bucket (call it "testbucket"). right now when the page is created, the user gets redirected to it at https://s3.amazonaws.com/testbucket/(some uuid). I want it so that it redirects them to www.subdomain.domain.com/(someuuid) where that html is stored. Right now this line is in my view callable (views.py)
return HTTPFound(location="https://s3.amazonaws.com/testbucket/%(uuid)s" % {'uuid':uuid})
Ive read (http://carltonbale.com/how-to-alias-a-domain-name-or-sub-domain-to-amazon-s3/) that in order to do this I need to create a bucket on my amazon s3 account called subdomain.domain.com and then cname it to s3.amazon.com. (What is the extra . for?). What then should I put in my return HTTPFound call? Should it be:
return HTTPFound(location="https://s3.amazonaws.com/subdomain.domain.com/%(uuid)s" % {'uuid':uuid})
That doesn't make any sense to me. What should it be instead?
If you create the CNAME record (the trailing dot is a quirk of how DNS works; it means that the domain is fully-qualified), then you don't need to mention s3.amazonaws.com. Instead, you can redirect to http://subdomain.domain.com/your_object_here.
Note, however, that Amazon does not have an SSL certificate for your domain, so if you want to access it over SSL you will need to connect to s3.amazonaws.com (or bucket_name_here.s3.amazonaws.com).
As far as I know, S3 does not currently implement a means for supplying your own SSL certificate (and even if it did, it would presumably need to use a feature called Server Name Indication, which is not yet universally supported).
Related
I'm creating an Amazon Managed Airflow (MWAA) using CDK with the setting of webserver_access_mode='PRIVATE_ONLY'. In this mode, AWS creates a VPC interface endpoint and binds an IP address, from the selected VPC private subnets, to them as explained here: https://docs.aws.amazon.com/mwaa/latest/userguide/configuring-networking.html
Now, I want to use those IPs to add a listener to an existing load balancer that I can then use to connect to a VPN, but this doesn't seem to be available as an output attribute/property of aws_cdk.aws_mwaa.CfnEnvironment: https://docs.aws.amazon.com/cdk/api/v1/python/aws_cdk.aws_mwaa/CfnEnvironment.html#aws_cdk.aws_mwaa.CfnEnvironment.NetworkConfigurationProperty
My question is, is there a way to obtain those IPs associated with the aws_cdk.aws_mwaa.CfnEnvironment? Right now I am looking up the results manually after the deployment with CDK and creating the listener but I would prefer to fully automate it in the same CDK construct.
I struggled with this same problem for some time. In the end I used a Custom Resource in my CFN template, passing it the URL of the MWAA webserver. In the Python code associated with the Custom Resource (Lambda) I do a socket.gethostbyname_ex() call, passing the URL as an argument. This call will return a tuple that that you'll have to parse to extract the endpoint addresses.
I made good use of the crhelper libraries (https://aws.amazon.com/blogs/infrastructure-and-automation/aws-cloudformation-custom-resource-creation-with-python-aws-lambda-and-crhelper/), which made things a lot easier.
In the end, I used a lambda function to resolve the webserver URL and register the IP addresses to the target group. The approach is described in the following AWS blog post: https://aws.amazon.com/blogs/networking-and-content-delivery/hostname-as-target-for-network-load-balancers/
The implementation of the lambda function is also available through the following AWS sample code: https://github.com/aws-samples/hostname-as-target-for-elastic-load-balancer
I am wondering if there is a way to obtain the hostname of a Django application when running tests. That is, I would like the tests to pass both locally and when run at the staging server. Hence a need to know http://localhost:<port> vs. http://staging.example.com is needed because some tests query particular URLs.
I found answers on how to do it inside templates, but that does not help since there is no response object to check the hostname.
How can one find out the hostname outside the views/templates? Is it stored in Django settings somewhere?
Why do you need to know the hostname? Tests can run just fine without it, if you use the test client. You do not need to know anything about the system they're running on.
You can also mark tests with a tag and then have the CI system run the tests including that tag.
And finally there is the LiveServerTestCase:
LiveServerTestCase does basically the same as TransactionTestCase with one extra feature: it launches a live Django server in the background on setup, and shuts it down on teardown. This allows the use of automated test clients other than the Django dummy client such as, for example, the Selenium client, to execute a series of functional tests inside a browser and simulate a real user’s actions.
The live server listens on localhost and binds to port 0 which uses a free port assigned by the operating system. The server’s URL can be accessed with self.live_server_url during the tests.
Additional information from comments:
You can test if the URL of an image file is present in your response by testing for the MEDIA_URL:
self.assertContains(response, f'{settings.MEDIA_URL}/default-avatar.svg')
You can test for the existence of an upload in various ways, but the easiest one is to check if there's a file object associated with the FileField. It will throw ValueError if there is not.
I am making AutoRecovery alarm via API and i want to make my code work in any system by just calling the python script. but as every one has unique AWS account id and region can also be different, so i want to fetch account id and region dynamically.
get-caller-identity is used for account id. but i ain't able to get how to use this function basically!
To identify these attributes of a particular EC2 instance from code running on the EC2 instance itself, use the Instance Identity Document.
Use any HTTP client utility to fetch this URL:
http://169.254.169.254/latest/dynamic/instance-identity/document
You can test this from the command line with curl.
This returns a JSON document containing -- among other things -- instanceId, accountId, and region for the instance.
The IP address 169.254.169.254 is always the same for every instance in every region. See also What's special about 169.254.169.254 IP address for AWS?
While STS GetCallerIdentity can be used to identify certain attributes of the credentials currently in use, it should not be necessary.
I am trying to upload a blob to azure blob storage with python sdk. I want to pass the MD5 hash for validation on the server side after upload.
Here's the code:
blob_service.put_block_blob_from_path(
container_name='container_name',
blob_name='upload_dir/'+object_name,
file_path=object_name,
content_md5=object_md5Hash
)
But I get this error:
AzureHttpError: The MD5 value specified in the request did not match with the MD5 value calculated by the server.
The file is ~200mb and the error throws instantly. Does not upload the file. So I suspect that it may be comparing the supplied hash with perhaps the hash of the first chunk or something.
Any ideas?
This is sort of an SDK bug in that we should throw a better error message rather than hitting the service, but validating the content of a large upload that has to be chunked simply doesn't work. x_ms_blob_content_md5 will store the md5 but the service will not validate it. That is something you could do on download though. content_md5 is validated by the server for the body of a particular request but since there's more than one with chunked blobs it will never work.
So, if the blob is small enough (below BLOB_MAX_DATA_SIZE) to be put in a single request, content_md5 will work fine. Otherwise I'd simply recommend using HTTPS and storing MD5 in x_ms_blob_content_md5 if you think you might want to download with HTTP and validate it on download. HTTPS already provides validation for things like bit flips on the wire so using it for upload/download will do a lot. If you can't upload/download with HTTPS for one reason or another you can consider chunking the blob yourself using the put block and put block list APIs.
FYI: In future versions we do intend to add automatic MD5 calculation for both single put and chunked operations in the library itself which will fully solve this. For the next version, we will add an improved error message if content_md5 is specified for a chunked download.
I reviewed the source code of the function put_block_blob_from_path of the Azure Blob Storage SDK. It explained the case in the function comment, please see the content below and refer to https://github.com/Azure/azure-storage-python/blob/master/azure/storage/blob/blobservice.py.
content_md5:
Optional. An MD5 hash of the blob content. This hash is used to
verify the integrity of the blob during transport. When this header
is specified, the storage service checks the hash that has arrived
with the one that was sent. If the two hashes do not match, the
operation will fail with error code 400 (Bad Request).
I think there're two things going on here.
Bug in SDK - I believe you have discovered a bug in the SDK. I looked at the source code for this function on Github and what I found is that when a large blob is uploaded in chunks, the SDK is first trying to create an empty block blob. With block blobs, this is not required. When it creates the empty block blob, it does not send any data. But you're setting content-md5 and the SDK compares the content-md5 you sent with the content-md5 of empty content and because they don't match, you get an error.
To fix the issue in the interim, please modify the source code in blobservice.py and comment out the following lines of code:
self.put_blob(
container_name,
blob_name,
None,
'BlockBlob',
content_encoding,
content_language,
content_md5,
cache_control,
x_ms_blob_content_type,
x_ms_blob_content_encoding,
x_ms_blob_content_language,
x_ms_blob_content_md5,
x_ms_blob_cache_control,
x_ms_meta_name_values,
x_ms_lease_id,
)
I have created a new issue on Github for this: https://github.com/Azure/azure-storage-python/issues/99.
Incorrect Usage - I noticed that you're passing the md5 hash of the file in content_md5 parameter. This will not work for you. You should actually pass md5 hash in x_ms_blob_content_md5 parameter. So your call should be:
blob_service.put_block_blob_from_path(
container_name='container_name',
blob_name='upload_dir/'+object_name,
file_path=object_name,
x_ms_blob_content_md5=object_md5Hash
)
I'm sending a callback URL to a remote widely API over which I have no control.
I've written my callback view and it's properly named (say, myapp_callback) in my urls.py, so all that I have to do is to call reverse('myapp_callback'), right? That's what it says in the manual.
Well, not so much. The result is /myapp/callback. Where's my protocol and hostname? The remote service I'm sending these API calls to has no idea. How can I detect it while maybe behind an Apache reverse proxy?
I'm working around this problem by putting the full URL into the settings file, but I'd love to provide a more "turnkey" solution.
Try out the request.build_absolute_uri(reverse('myapp_callback')).
Returns the absolute URI form of location. If no location is provided, the location will be set to request.get_full_path().
If the location is already an absolute URI, it will not be altered. Otherwise the absolute URI is built using the server variables available in this request.
Example: "http://example.com/music/bands/the_beatles/?print=true"