AWS S3 Client¶
A thread-safe singleton client for interacting with AWS S3 storage, implementing the StorageClient interface. The client provides both synchronous and asynchronous methods for interacting with S3, with automatic cleanup of test resources when used in test environments.
Note
This client is designed to be used as a singleton within your application. Multiple instances will return the same underlying client.
Configuration¶
The S3Client is configured using environment variables. All parameters are required.
Variable |
Type |
Default |
Description |
---|---|---|---|
S3_AWS_REGION |
string |
None |
AWS region name (e.g., ‘us-east-1’) |
S3_AWS_ACCESS_KEY_ID |
string |
None |
AWS access key ID with S3 permissions |
S3_AWS_SECRET_ACCESS_KEY |
string |
None |
AWS secret access key |
Example .env
file:
S3_AWS_REGION=us-east-1
S3_AWS_ACCESS_KEY_ID=your-access-key-id
S3_AWS_SECRET_ACCESS_KEY=your-secret-access-key
Basic Usage¶
Initialization¶
The S3Client is configured using environment variables. Make sure these are set before initializing the client:
from prs_commons import S3Client
# Initialize with environment variables
s3 = S3Client()
# The client is now ready to use
Testing¶
When using the S3Client in tests, it automatically handles cleanup of test files. The test suite includes fixtures that:
Clean up test files after each test
Handle both synchronous and asynchronous test methods
Provide proper error handling and reporting
To use in tests, simply extend from the test class and use the provided fixtures.
File Operations¶
Uploading Files¶
# Basic upload
result = s3.upload_file(
file_path="local_file.txt",
bucket="my-bucket",
key="path/in/s3/file.txt"
)
# Upload with additional S3 parameters
result = s3.upload_file(
file_path="local_file.txt",
bucket="my-bucket",
key="path/in/s3/file.txt",
ExtraArgs={
'ContentType': 'text/plain',
'Metadata': {'author': 'user'}
}
)
Downloading Files¶
# Basic download
try:
success = s3.download_file(
bucket="my-bucket",
key="path/in/s3/file.txt",
file_path="local_file.txt"
)
if success:
print("File downloaded successfully")
except FileNotFoundError as e:
print(f"File not found: {e}")
except PermissionError as e:
print(f"Permission denied: {e}")
except ClientError as e:
print(f"S3 error: {e}")
# Download as base64-encoded string
try:
file_data = s3.download_as_base64(
bucket="my-bucket",
key="documents/report.pdf"
)
print(f"Downloaded file (base64, {len(file_data)} bytes)")
except FileNotFoundError as e:
print(f"File not found: {e}")
except PermissionError as e:
print(f"Permission denied: {e}")
except ClientError as e:
print(f"S3 error: {e}")
Deleting Files¶
result = s3.delete_object(
bucket="my-bucket",
key="path/in/s3/file.txt"
)
Pre-signed URLs¶
Generate Upload URL¶
- generate_upload_url(bucket: str, key: str, expiration: int = 3600, **kwargs: Any) str | None
Generate a pre-signed URL for uploading a file to S3.
This is a convenience wrapper around
generate_presigned_url()
for uploads. The content type will be automatically detected from the file extension if not provided.- Parameters:
- Returns:
Pre-signed URL as a string, or None if credentials are invalid
- Return type:
str | None
Common Parameters
ContentType
(str, optional): The content type of the file (e.g., ‘image/jpeg’). If not provided, it will be automatically detected from the file extension.ACL
(str, optional): Access control for the file. Defaults to the bucket’s ACL. Common values: ‘private’, ‘public-read’, ‘public-read-write’, ‘authenticated-read’Metadata
(dict, optional): Dictionary of metadata to store with the object. Keys will be prefixed with ‘x-amz-meta-’ when stored in S3.Other parameters supported by boto3’s generate_presigned_url for ‘put_object’ operation
Example
# Generate URL for uploading a text file with metadata url = s3.generate_upload_url( bucket='my-bucket', key='documents/report.txt', ContentType='text/plain', Metadata={ 'author': 'user@example.com', 'description': 'Quarterly report Q2 2023' }, expiration=7200 # 2 hours )
# Use the URL to upload a file with a PUT request # import requests # with open(‘file.txt’, ‘rb’) as f: # response = requests.put(url, data=f)
Generate Download URL¶
# Generate a pre-signed URL for file download
download_url = s3.generate_download_url(
bucket="my-bucket",
key="downloads/file.txt",
ResponseContentType="application/octet-stream",
expiration=3600 # URL expires in 1 hour (default)
)
print(f"Download URL: {download_url}")
Error Handling¶
The S3 client raises the following exceptions:
ClientError
: For AWS service errorsNoCredentialsError
: When AWS credentials are not foundRuntimeError
: For client initialization errorsFileNotFoundError
: When the requested file doesn’t existPermissionError
: When there are permission issues
Example error handling:
from botocore.exceptions import ClientError, NoCredentialsError
try:
# Your S3 operations here
pass
except FileNotFoundError as e:
print(f"File not found: {e}")
except PermissionError as e:
print(f"Permission denied: {e}")
except NoCredentialsError:
print("AWS credentials not found")
except ClientError as e:
error_code = e.response.get('Error', {}).get('Code')
if error_code == 'NoSuchBucket':
print("Bucket does not exist")
else:
print(f"AWS error: {e}")
except Exception as e:
print(f"Unexpected error: {e}")
from botocore.exceptions import ClientError, NoCredentialsError
try:
s3.upload_file("nonexistent.txt", "my-bucket", "file.txt")
except FileNotFoundError as e:
print(f"Local file not found: {e}")
except NoCredentialsError:
print("AWS credentials not found")
except ClientError as e:
print(f"AWS error: {e.response['Error']['Message']}")
except Exception as e:
print(f"Unexpected error: {e}")
API Reference¶
- class prs_commons.aws.s3_client.S3Client[source]¶
Bases:
StorageClient
Thread-safe singleton client for AWS S3 operations.
This client provides a simple interface to interact with AWS S3 using boto3. It implements the singleton pattern to ensure only one instance exists.
- property client: BaseClient¶
Get the boto3 S3 client instance.
- Returns:
The boto3 S3 client instance.
- Raises:
RuntimeError – If the S3 client cannot be initialized.
- async upload_file(file_path: str, bucket: str, key: str, **kwargs: Any) Dict[str, Any] [source]¶
Upload a file to an S3 bucket.
- Parameters:
file_path – Path to the file to upload
bucket – Target S3 bucket name
key – S3 object key (path in the bucket)
**kwargs – Additional arguments to pass to boto3 upload_file
- Returns:
Dictionary containing status and operation details
- Return type:
S3Response
- Raises:
ClientError – If the upload fails
- async download_file(bucket: str, key: str, file_path: str, **kwargs: Any) bool [source]¶
Download a file from an S3 bucket to the local filesystem.
- Parameters:
bucket – Source S3 bucket name
key – S3 object key (path in the bucket)
file_path – Local filesystem path where the file will be saved. Must include the target filename and extension. Example: ‘/path/to/destination/filename.ext’ The parent directory must exist and be writable.
**kwargs – Additional arguments to pass to boto3 download_file
- Returns:
True if download was successful
- Return type:
- Raises:
FileNotFoundError – If the file doesn’t exist in S3 or local path is invalid
PermissionError – If there are permission issues with S3 or local filesystem
botocore.exceptions.ClientError – For other S3-specific errors
IOError – If there are issues writing to the local filesystem
Example
>>> s3 = S3Client() >>> success = s3.download_file( ... bucket='my-bucket', ... key='folder/file.txt', ... file_path='/local/path/to/save/file.txt' ... ) >>> if success: ... print("File downloaded successfully")
- async delete_object(bucket: str, key: str) Dict[str, Any] [source]¶
Delete an object from an S3 bucket.
- Parameters:
bucket – S3 bucket name
key – S3 object key to delete
- Returns:
Dictionary containing status and operation details
- Return type:
S3Response
- generate_presigned_url(bucket: str, key: str, operation: str = 'get_object', expiration: int = 3600, **kwargs: Any) str | None [source]¶
Generate a pre-signed URL for an S3 object.
- Parameters:
bucket – S3 bucket name
key – S3 object key (path in the bucket)
operation – The S3 operation to allow with this URL. Common values: ‘get_object’, ‘put_object’, ‘delete_object’
expiration – Time in seconds until the URL expires (default: 1 hour)
**kwargs – Additional parameters to pass to the S3 operation
- Returns:
The pre-signed URL as a string, or None if credentials are invalid
- Return type:
Optional[str]
Example
# Generate a URL to upload a file >>> upload_url = s3.generate_presigned_url( … bucket=’my-bucket’, … key=’uploads/file.txt’, … operation=’put_object’, … ContentType=’text/plain’ … )
# Generate a URL to download a file >>> download_url = s3.generate_presigned_url( … bucket=’my-bucket’, … key=’downloads/file.txt’, … operation=’get_object’, … ResponseContentType=’application/octet-stream’ … )
- generate_upload_url(bucket: str, key: str, expiration: int = 3600, **kwargs: Any) str | None [source]¶
Generate a pre-signed URL for uploading a file to S3.
This is a convenience wrapper around generate_presigned_url for uploads.
- Parameters:
bucket – S3 bucket name
key – S3 object key where the file will be stored
expiration – Time in seconds until the URL expires (default: 1 hour)
**kwargs – Additional parameters to pass to the S3 put_object operation Common parameters: - ContentType: The content type of the file (e.g., ‘image/jpeg’) - ACL: Access control for the file (e.g., ‘private’, ‘public-read’) - Metadata: Dictionary of metadata to store with the object
- Returns:
The pre-signed URL as a string, or None if credentials are invalid
- Return type:
Optional[str]
Example
>>> upload_url = s3.generate_upload_url( ... bucket='my-bucket', ... key='uploads/file.jpg', ... ContentType='image/jpeg', ... ACL='private', ... Metadata={ ... 'custom': 'value' ... } ... )
- generate_download_url(bucket: str, key: str, expiration: int = 3600, **kwargs: Any) str | None [source]¶
Generate a pre-signed URL for downloading a file from S3.
This is a convenience wrapper around generate_presigned_url for downloads.
- Parameters:
bucket – S3 bucket name
key – S3 object key of the file to download
expiration – Time in seconds until the URL expires (default: 1 hour)
**kwargs – Additional parameters to pass to the S3 get_object operation
- Returns:
The pre-signed URL as a string, or None if credentials are invalid
- Return type:
Optional[str]
Example
>>> download_url = s3.generate_download_url( ... bucket='my-bucket', ... key='downloads/file.txt', ... ResponseContentType='application/pdf', ... ResponseContentDisposition='attachment; filename=report.pdf' ... )
- async download_as_base64(bucket: str, key: str, check_exists: bool = True, **kwargs: Any) str | None [source]¶
Download a file from S3 and return its contents as a base64-encoded string.
This method is useful when you need to work with the file contents directly in memory without saving to disk, such as when sending files in API responses or processing file contents in memory.
- Parameters:
bucket – Name of the S3 bucket
key – S3 object key (path in the bucket)
check_exists – If True, verify the file exists before downloading
**kwargs – Additional arguments to pass to boto3 download_fileobj
- Returns:
Base64-encoded string of the file contents
- Raises:
FileNotFoundError – If the file doesn’t exist
PermissionError – If there are permission issues
botocore.exceptions.ClientError – For S3-specific errors
Exception – For any other unexpected errors
Example
>>> # Download a file as base64 >>> file_data = s3.download_as_base64( ... bucket='my-bucket', ... key='documents/report.pdf', ... ) >>> if file_data: ... # Use the base64 data (e.g., embed in HTML, send in API response) ... print(f"File size: {len(file_data)} bytes")
File Operations¶
Base64 Download¶
- async S3Client.download_as_base64(bucket: str, key: str, check_exists: bool = True, **kwargs: Any) str | None [source]
Download a file from S3 and return its contents as a base64-encoded string.
This method is useful when you need to work with the file contents directly in memory without saving to disk, such as when sending files in API responses or processing file contents in memory.
- Parameters:
bucket – Name of the S3 bucket
key – S3 object key (path in the bucket)
check_exists – If True, verify the file exists before downloading
**kwargs – Additional arguments to pass to boto3 download_fileobj
- Returns:
Base64-encoded string of the file contents
- Raises:
FileNotFoundError – If the file doesn’t exist
PermissionError – If there are permission issues
botocore.exceptions.ClientError – For S3-specific errors
Exception – For any other unexpected errors
Example
>>> # Download a file as base64 >>> file_data = s3.download_as_base64( ... bucket='my-bucket', ... key='documents/report.pdf', ... ) >>> if file_data: ... # Use the base64 data (e.g., embed in HTML, send in API response) ... print(f"File size: {len(file_data)} bytes")
Pre-signed URLs¶
The S3 client provides methods to generate pre-signed URLs for secure, time-limited access to S3 objects:
- S3Client.generate_presigned_url(bucket: str, key: str, operation: str = 'get_object', expiration: int = 3600, **kwargs: Any) str | None [source]
Generate a pre-signed URL for an S3 object.
- Parameters:
bucket – S3 bucket name
key – S3 object key (path in the bucket)
operation – The S3 operation to allow with this URL. Common values: ‘get_object’, ‘put_object’, ‘delete_object’
expiration – Time in seconds until the URL expires (default: 1 hour)
**kwargs – Additional parameters to pass to the S3 operation
- Returns:
The pre-signed URL as a string, or None if credentials are invalid
- Return type:
Optional[str]
Example
# Generate a URL to upload a file >>> upload_url = s3.generate_presigned_url( … bucket=’my-bucket’, … key=’uploads/file.txt’, … operation=’put_object’, … ContentType=’text/plain’ … )
# Generate a URL to download a file >>> download_url = s3.generate_presigned_url( … bucket=’my-bucket’, … key=’downloads/file.txt’, … operation=’get_object’, … ResponseContentType=’application/octet-stream’ … )
- S3Client.generate_upload_url(bucket: str, key: str, expiration: int = 3600, **kwargs: Any) str | None [source]
Generate a pre-signed URL for uploading a file to S3.
This is a convenience wrapper around generate_presigned_url for uploads.
- Parameters:
bucket – S3 bucket name
key – S3 object key where the file will be stored
expiration – Time in seconds until the URL expires (default: 1 hour)
**kwargs – Additional parameters to pass to the S3 put_object operation Common parameters: - ContentType: The content type of the file (e.g., ‘image/jpeg’) - ACL: Access control for the file (e.g., ‘private’, ‘public-read’) - Metadata: Dictionary of metadata to store with the object
- Returns:
The pre-signed URL as a string, or None if credentials are invalid
- Return type:
Optional[str]
Example
>>> upload_url = s3.generate_upload_url( ... bucket='my-bucket', ... key='uploads/file.jpg', ... ContentType='image/jpeg', ... ACL='private', ... Metadata={ ... 'custom': 'value' ... } ... )
- S3Client.generate_download_url(bucket: str, key: str, expiration: int = 3600, **kwargs: Any) str | None [source]
Generate a pre-signed URL for downloading a file from S3.
This is a convenience wrapper around generate_presigned_url for downloads.
- Parameters:
bucket – S3 bucket name
key – S3 object key of the file to download
expiration – Time in seconds until the URL expires (default: 1 hour)
**kwargs – Additional parameters to pass to the S3 get_object operation
- Returns:
The pre-signed URL as a string, or None if credentials are invalid
- Return type:
Optional[str]
Example
>>> download_url = s3.generate_download_url( ... bucket='my-bucket', ... key='downloads/file.txt', ... ResponseContentType='application/pdf', ... ResponseContentDisposition='attachment; filename=report.pdf' ... )
Error Handling¶
All methods return a dictionary with the following structure:
{
"status": "success" | "error",
"bucket": "bucket-name",
"key": "object-key",
# Only present if status is "error"
"error": "error-message",
# Additional fields may be present depending on the operation
}
Thread Safety¶
The S3Client is implemented as a thread-safe singleton. Multiple threads can safely use the same instance.
Dependencies¶
boto3 >= 1.28.0
botocore >= 1.31.0