Access files from AWS S3 using pre-signed URLs in Python
Generate pre-signed URLs for S3 objects in Python
Note: For non-members, this article can be accessed at https://dock2learn.com/tech/how-to-generate-presigned-urls-for-s3-objects-using-python/
Introduction:
Let’s assume that you need to share files from your AWS S3 bucket(private) without providing AWS access to a user. How would you do that? Well, we have pre-signed URLs that are shortly lived, which can be shared and used to access the content shared.
Pre-signed URLs:
What is a pre-signed URL?
A pre-signed URL gives you temporary access to the object identified in the URL, provided that the creator of the pre-signed URL has permissions to access that object.
That is, if you receive a pre-signed URL to upload an object, you can upload the object only if the creator of the pre-signed URL has the necessary permissions to upload that object. Same applies for download as well.
We will see how to generate pre-signed URLs for S3 bucket programmatically using python and boto3.
When we say, the creator of the presigned URL should have access what does it mean?
It means, the URL generator should have a aws access with right credentials(may be in a lambda)and to achieve this, we could expose a REST API to the customer to request for a URL based on the upload/download operation. This ensures the user need not be provided with the AWS credentials.
The pre-signed URL will expire based on the expiry value configured while generating it. We shall look at it shortly.
A high-level design:
In the above design, a user requests the URL from the UI(could be a web portal) via a REST API based on the operation required. This hits the API gateway which triggers a lambda. The lambda executes the code to generate the pre-signed URL for the requested S3 bucket and key location.
The most prevalent operations are but not limited to upload/download objects to and from S3 buckets which are performed using
- put_object
- get_ object.
Let’s look at the code which goes in the lambda
1. Generating pre-signed URL for download
Please note that the awssession token is an optional parameter. This may be required if your organization is providing credentials that expire. If you are using your personal account and do not have any configuration for session expiry they may not be required.
The ‘get_object’ specifies the URL is being generated for a download operation. The bucket name and object should be passed as part of the params dictionary.
2. Generating pre-signed URL for upload
We use the same create presigned url with put_object method to create a presigned URL for uploading a file.
Output:
https://experiment.s3.amazonaws.com/3eb426c2-8b03-4df0-ac8b-2818d7/0185-43b6-4e82-90-f80ddeb/Dockerfile.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAYBAO6D35IL7EVWWQ%2F20201203%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20201203T160918Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Security-Token=FwoGZXIvYXdzEEkaDBMMvCnwF6gDfBdJJyKNAkWcPF2UGsEKr%2BhTII0%2Fea0BnYVEfD%2B715iFASMgmQajg3D%2FcOYV5y975wzaUDCTESEt8VKQjUWb3sayTlDp6Hs2LuPoII92n%2FmOFZPxvBqU43FBYndpIVmOVg1vKno%2Bj7pYaoYzSdpAEIVv4yb5Bg%2BBiMT2x3E7GG771%2Fq4mi1jWF8lqf6QkyTT9qpLEiFQDxSGH47nT%2BzcoNmpHLLdHSTMDJsmDpiHiuvczQdSroBR6I9%2BksN7Lm3k1cKM1XAw3JFN%2BKmyAX%2BDuR5cGpNUWQehl1OxXHZx7%2F9BLsWgF6Nf7vj5vLc5e%2BvcuUwsmSliKsQvnnXM2zvzT2LK5wBm3qeKcxiyX1994U9jt%2BemKJOQpP4FMiv8IKTD07lWgWZN8uYrU2II4O5YUE8hhiRGCIch8v4b3mfPoVp%2B7OYKVkVn&X-Amz-Signature=812952fe5fadf0174ea95a2eadaa88c7526613b29c76e865d4548952fa55dc64This is a sample pre-signed URL output.
Try accessing the presigned URL either through browser or programmatically.
Oops:
We have hit a roadblock. The URL throws a signature does not match error.
Message: The request signature we calculated does not match the signature you provided. Check your key and signing method.
Why?
The reason for this is, it’s not recommended to use generate_presigned_url with put_object parameter to generate pre-signed URLs for uploading files though it wouldn’t throw any error while generating.
I had deliberately used it here because I had run into this issue and wanted to share this learning.
Please refer to this github link for more information about this.
Let’s move to the recommended solution. While generating URLs for upload, it’s always better to use generate_presigned_post method as this includes the proper header information and other parameters required for the URL.
3. Pre-signed URL post
Output: A sample presigned URL
https://experiment.s3.amazonaws.com/3eb426c2-8b03-4df0-ac8b-2818d7/0185-43b6-4e82-90-f80ddeb/Dockerfile.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAYBAO6D35IL7EVWWQ%2F20201203%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20201203T160918Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Security-Token=FwoGZXIvYXdzEEkaDBMMvCnwF6gDfBdJJyKNAkWcPF2UGsEKr%2BhTII0%2Fea0BnYVEfD%2B715iFASMgmQajg3D%2FcOYV5y975wzaUDCTESEt8VKQjUWb3sayTlDp6Hs2LuPoII92n%2FmOFZPxvBqU43FBYndpIVmOVg1vKno%2Bj7pYaoYzSdpAEIVv4yb5Bg%2BBiMT2x3E7GG771%2Fq4mi1jWF8lqf6QkyTT9qpLEiFQDxSGH47nT%2BzcoNmpHLLdHSTMDJsmDpiHiuvczQdSroBR6I9%2BksN7Lm3k1cKM1XAw3JFN%2BKmyAX%2BDuR5cGpNUWQehl1OxXHZx7%2F9BLsWgF6Nf7vj5vLc5e%2BvcuUwsmSliKsQvnnXM2zvzT2LK5wBm3qeKcxiyX1994U9jt%2BemKJOQpP4FMiv8IKTD07lWgWZN8uYrU2II4O5YUE8hhiRGCIch8v4b3mfPoVp%2B7OYKVkVn&X-Amz-Signature=812952fe5fadf0174ea95a2eadaa88c7526613b29c76e865d4548952fa55dc64Caveats:
- If the server-side encryption of S3 is set to KMS, you may need to set the signature version to v4 while creating the boto3 object.
- Boto3 by default supports signature v4. However for S3, the objects should explicitly set the signature version to v4 in case of KMS.
- Not setting the signature to v4 may result in 403 error while trying to access the URL though you have the right permissions.
Setting signature version explicitly:
s3_client=boto3.client("s3",config=Config(signature_version='s3v4'))Summary:
- Pre-signed URLs could be used to provide temporary access to users without providing
awsaccess to users - URLs could be generated to upload and download files
