AWS S3 Client

A thread-safe singleton client for interacting with AWS S3 storage, implementing the StorageClient interface. The client provides both synchronous and asynchronous methods for interacting with S3, with automatic cleanup of test resources when used in test environments.

Note

This client is designed to be used as a singleton within your application. Multiple instances will return the same underlying client.

Configuration

The S3Client is configured using environment variables. All parameters are required.

Variable	Type	Default	Description
REGION	string	None	AWS region name (e.g., ‘us-east-1’)
BUCKET_ACCESS_KEY_ID	string	None	AWS access key ID with S3 permissions
BUCKET_SECRET_ACCESS_KEY	string	None	AWS secret access key
BUCKET_NAME	string	None	Default bucket name (optional, can be passed to methods)

Example .env file:

REGION=us-east-1
BUCKET_ACCESS_KEY_ID=your-access-key-id
BUCKET_SECRET_ACCESS_KEY=your-secret-access-key
BUCKET_NAME=my-default-bucket

Basic Usage

Initialization

The S3Client is configured using environment variables. Make sure these are set before initializing the client:

from prs_commons import S3Client

# Initialize with environment variables
s3 = S3Client()

# The client is now ready to use

Testing

When using the S3Client in tests, it automatically handles cleanup of test files. The test suite includes fixtures that:

• Clean up test files after each test

• Handle both synchronous and asynchronous test methods

• Provide proper error handling and reporting

To use in tests, simply extend from the test class and use the provided fixtures.

File Operations

Uploading Files

# Basic upload
result = s3.upload_file(
    file_path="local_file.txt",
    bucket="my-bucket",
    key="path/in/s3/file.txt"
)

# Upload with additional S3 parameters
result = s3.upload_file(
    file_path="local_file.txt",
    bucket="my-bucket",
    key="path/in/s3/file.txt",
    ExtraArgs={
        'ContentType': 'text/plain',
        'Metadata': {'author': 'user'}
    }
)

Downloading Files

# Basic download
try:
    success = s3.download_file(
        bucket="my-bucket",
        key="path/in/s3/file.txt",
        file_path="local_file.txt"
    )
    if success:
        print("File downloaded successfully")
except FileNotFoundError as e:
    print(f"File not found: {e}")
except PermissionError as e:
    print(f"Permission denied: {e}")
except ClientError as e:
    print(f"S3 error: {e}")

# Download as base64-encoded string
try:
    file_data = s3.download_as_base64(
        bucket="my-bucket",
        key="documents/report.pdf"
    )
    print(f"Downloaded file (base64, {len(file_data)} bytes)")
except FileNotFoundError as e:
    print(f"File not found: {e}")
except PermissionError as e:
    print(f"Permission denied: {e}")
except ClientError as e:
    print(f"S3 error: {e}")

Deleting Files

result = s3.delete_object(
    bucket="my-bucket",
    key="path/in/s3/file.txt"
)

Pre-signed URLs

Generate Upload URL

generate_upload_url(bucket: str, key: str, expiration: int = 3600, **kwargs: Any) → str | None

Generate a pre-signed URL for uploading a file to S3.

This is a convenience wrapper around generate_presigned_url() for uploads. The content type will be automatically detected from the file extension if not provided.

Parameters:

• bucket (str) – S3 bucket name

• key (str) – S3 object key where the file will be stored

• expiration (int) – Time in seconds until the URL expires (default: 3600)

• **kwargs – Additional parameters to pass to the S3 put_object operation

Returns:

Pre-signed URL as a string, or None if credentials are invalid

Return type:

str | None

Common Parameters

• ContentType (str, optional): The content type of the file (e.g., ‘image/jpeg’). If not provided, it will be automatically detected from the file extension.

• ACL (str, optional): Access control for the file. Defaults to the bucket’s ACL. Common values: ‘private’, ‘public-read’, ‘public-read-write’, ‘authenticated-read’

• Metadata (dict, optional): Dictionary of metadata to store with the object. Keys will be prefixed with ‘x-amz-meta-’ when stored in S3.

• Other parameters supported by boto3’s generate_presigned_url for ‘put_object’ operation

Example

# Generate URL for uploading a text file with metadata
url = s3.generate_upload_url(
    bucket='my-bucket',
    key='documents/report.txt',
    ContentType='text/plain',
    Metadata={
        'author': 'user@example.com',
        'description': 'Quarterly report Q2 2023'
    },
    expiration=7200  # 2 hours
)

# Use the URL to upload a file with a PUT request # import requests # with open(‘file.txt’, ‘rb’) as f: # response = requests.put(url, data=f)

Generate Download URL

# Generate a pre-signed URL for file download
download_url = s3.generate_download_url(
    bucket="my-bucket",
    key="downloads/file.txt",
    ResponseContentType="application/octet-stream",
    expiration=3600  # URL expires in 1 hour (default)
)
print(f"Download URL: {download_url}")

Error Handling

The S3 client raises the following exceptions:

• ClientError: For AWS service errors

• NoCredentialsError: When AWS credentials are not found

• RuntimeError: For client initialization errors

• FileNotFoundError: When the requested file doesn’t exist

• PermissionError: When there are permission issues

Example error handling:

from botocore.exceptions import ClientError, NoCredentialsError

try:
    # Your S3 operations here
    pass
except FileNotFoundError as e:
    print(f"File not found: {e}")
except PermissionError as e:
    print(f"Permission denied: {e}")
except NoCredentialsError:
    print("AWS credentials not found")
except ClientError as e:
    error_code = e.response.get('Error', {}).get('Code')
    if error_code == 'NoSuchBucket':
        print("Bucket does not exist")
    else:
        print(f"AWS error: {e}")
except Exception as e:
    print(f"Unexpected error: {e}")

from botocore.exceptions import ClientError, NoCredentialsError

try:
    s3.upload_file("nonexistent.txt", "my-bucket", "file.txt")
except FileNotFoundError as e:
    print(f"Local file not found: {e}")
except NoCredentialsError:
    print("AWS credentials not found")
except ClientError as e:
    print(f"AWS error: {e.response['Error']['Message']}")
except Exception as e:
    print(f"Unexpected error: {e}")

API Reference

class prs_commons.aws.s3_client.S3Client

Bases: StorageClient

Thread-safe singleton client for AWS S3 operations.

This client provides a simple interface to interact with AWS S3 using boto3. It implements the singleton pattern to ensure only one instance exists.

__init__() → None

Initialize the S3 client with configuration from environment variables.

property client: BaseClient

Get the boto3 S3 client instance.

Returns:

The boto3 S3 client instance.

Raises:

RuntimeError – If the S3 client cannot be initialized.

async upload_file(file_path: str, bucket: str, key: str, **kwargs: Any) → Dict[str, Any]

Upload a file to an S3 bucket.

Parameters:

• file_path – Path to the file to upload

• bucket – Target S3 bucket name

• key – S3 object key (path in the bucket)

• **kwargs – Additional arguments to pass to boto3 upload_file

Returns:

Dictionary containing status and operation details

Return type:

S3Response

Raises:

ClientError – If the upload fails

async download_file(bucket: str, key: str, file_path: str, **kwargs: Any) → bool

Download a file from an S3 bucket to the local filesystem.

Parameters:

• bucket – Source S3 bucket name

• key – S3 object key (path in the bucket)

• file_path – Local filesystem path where the file will be saved. Must include the target filename and extension. Example: ‘/path/to/destination/filename.ext’ The parent directory must exist and be writable.

• **kwargs – Additional arguments to pass to boto3 download_file

Returns:

True if download was successful

Return type:

bool

Raises:

• FileNotFoundError – If the file doesn’t exist in S3 or local path is invalid

• PermissionError – If there are permission issues with S3 or local filesystem

• botocore.exceptions.ClientError – For other S3-specific errors

• IOError – If there are issues writing to the local filesystem

Example

>>> s3 = S3Client()
>>> success = s3.download_file(
...     bucket='my-bucket',
...     key='folder/file.txt',
...     file_path='/local/path/to/save/file.txt'
... )
>>> if success:
...     print("File downloaded successfully")

async delete_object(bucket: str, key: str) → Dict[str, Any]

Delete an object from an S3 bucket.

Parameters:

• bucket – S3 bucket name

• key – S3 object key to delete

Returns:

Dictionary containing status and operation details

Return type:

S3Response

generate_presigned_url(bucket: str, key: str, operation: str = 'get_object', expiration: int = 3600, **kwargs: Any) → str | None

Generate a pre-signed URL for an S3 object.

Parameters:

• bucket – S3 bucket name

• key – S3 object key (path in the bucket)

• operation – The S3 operation to allow with this URL. Common values: ‘get_object’, ‘put_object’, ‘delete_object’

• expiration – Time in seconds until the URL expires (default: 1 hour)

• **kwargs – Additional parameters to pass to the S3 operation

Returns:

The pre-signed URL as a string, or None if credentials are invalid

Return type:

Optional[str]

Example

# Generate a URL to upload a file >>> upload_url = s3.generate_presigned_url( … bucket=’my-bucket’, … key=’uploads/file.txt’, … operation=’put_object’, … ContentType=’text/plain’ … )

# Generate a URL to download a file >>> download_url = s3.generate_presigned_url( … bucket=’my-bucket’, … key=’downloads/file.txt’, … operation=’get_object’, … ResponseContentType=’application/octet-stream’ … )

generate_upload_url(bucket: str, key: str, expiration: int = 3600, **kwargs: Any) → str | None

Generate a pre-signed URL for uploading a file to S3.

This is a convenience wrapper around generate_presigned_url for uploads.

Parameters:

• bucket – S3 bucket name

• key – S3 object key where the file will be stored

• expiration – Time in seconds until the URL expires (default: 1 hour)

• **kwargs – Additional parameters to pass to the S3 put_object operation Common parameters: - ContentType: The content type of the file (e.g., ‘image/jpeg’) - ACL: Access control for the file (e.g., ‘private’, ‘public-read’) - Metadata: Dictionary of metadata to store with the object

Returns:

The pre-signed URL as a string, or None if credentials are invalid

Return type:

Optional[str]

Example

>>> upload_url = s3.generate_upload_url(
...     bucket='my-bucket',
...     key='uploads/file.jpg',
...     ContentType='image/jpeg',
...     ACL='private',
...     Metadata={
...         'custom': 'value'
...     }
... )

generate_download_url(bucket: str, key: str, expiration: int = 3600, **kwargs: Any) → str | None

Generate a pre-signed URL for downloading a file from S3.

This is a convenience wrapper around generate_presigned_url for downloads.

Parameters:

• bucket – S3 bucket name

• key – S3 object key of the file to download

• expiration – Time in seconds until the URL expires (default: 1 hour)

• **kwargs – Additional parameters to pass to the S3 get_object operation

Returns:

The pre-signed URL as a string, or None if credentials are invalid

Return type:

Optional[str]

Example

>>> download_url = s3.generate_download_url(
...     bucket='my-bucket',
...     key='downloads/file.txt',
...     ResponseContentType='application/pdf',
...     ResponseContentDisposition='attachment; filename=report.pdf'
... )

async download_as_base64(bucket: str, key: str, check_exists: bool = True, **kwargs: Any) → str | None

Download a file from S3 and return its contents as a base64-encoded string.

This method is useful when you need to work with the file contents directly in memory without saving to disk, such as when sending files in API responses or processing file contents in memory.

Parameters:

• bucket – Name of the S3 bucket

• key – S3 object key (path in the bucket)

• check_exists – If True, verify the file exists before downloading

• **kwargs – Additional arguments to pass to boto3 download_fileobj

Returns:

Base64-encoded string of the file contents

Raises:

• FileNotFoundError – If the file doesn’t exist

• PermissionError – If there are permission issues

• botocore.exceptions.ClientError – For S3-specific errors

• Exception – For any other unexpected errors

Example

>>> # Download a file as base64
>>> file_data = s3.download_as_base64(
...     bucket='my-bucket',
...     key='documents/report.pdf',
... )
>>> if file_data:
...     # Use the base64 data (e.g., embed in HTML, send in API response)
...     print(f"File size: {len(file_data)} bytes")

File Operations

Base64 Download

async S3Client.download_as_base64(bucket: str, key: str, check_exists: bool = True, **kwargs: Any) → str | None

Download a file from S3 and return its contents as a base64-encoded string.

This method is useful when you need to work with the file contents directly in memory without saving to disk, such as when sending files in API responses or processing file contents in memory.

Parameters:

• bucket – Name of the S3 bucket

• key – S3 object key (path in the bucket)

• check_exists – If True, verify the file exists before downloading

• **kwargs – Additional arguments to pass to boto3 download_fileobj

Returns:

Base64-encoded string of the file contents

Raises:

• FileNotFoundError – If the file doesn’t exist

• PermissionError – If there are permission issues

• botocore.exceptions.ClientError – For S3-specific errors

• Exception – For any other unexpected errors

Example

>>> # Download a file as base64
>>> file_data = s3.download_as_base64(
...     bucket='my-bucket',
...     key='documents/report.pdf',
... )
>>> if file_data:
...     # Use the base64 data (e.g., embed in HTML, send in API response)
...     print(f"File size: {len(file_data)} bytes")

Pre-signed URLs

The S3 client provides methods to generate pre-signed URLs for secure, time-limited access to S3 objects:

S3Client.generate_presigned_url(bucket: str, key: str, operation: str = 'get_object', expiration: int = 3600, **kwargs: Any) → str | None

Generate a pre-signed URL for an S3 object.

Parameters:

• bucket – S3 bucket name

• key – S3 object key (path in the bucket)

• operation – The S3 operation to allow with this URL. Common values: ‘get_object’, ‘put_object’, ‘delete_object’

• expiration – Time in seconds until the URL expires (default: 1 hour)

• **kwargs – Additional parameters to pass to the S3 operation

Returns:

The pre-signed URL as a string, or None if credentials are invalid

Return type:

Optional[str]

Example

# Generate a URL to upload a file >>> upload_url = s3.generate_presigned_url( … bucket=’my-bucket’, … key=’uploads/file.txt’, … operation=’put_object’, … ContentType=’text/plain’ … )

# Generate a URL to download a file >>> download_url = s3.generate_presigned_url( … bucket=’my-bucket’, … key=’downloads/file.txt’, … operation=’get_object’, … ResponseContentType=’application/octet-stream’ … )

S3Client.generate_upload_url(bucket: str, key: str, expiration: int = 3600, **kwargs: Any) → str | None

Generate a pre-signed URL for uploading a file to S3.

This is a convenience wrapper around generate_presigned_url for uploads.

Parameters:

• bucket – S3 bucket name

• key – S3 object key where the file will be stored

• expiration – Time in seconds until the URL expires (default: 1 hour)

• **kwargs – Additional parameters to pass to the S3 put_object operation Common parameters: - ContentType: The content type of the file (e.g., ‘image/jpeg’) - ACL: Access control for the file (e.g., ‘private’, ‘public-read’) - Metadata: Dictionary of metadata to store with the object

Returns:

The pre-signed URL as a string, or None if credentials are invalid

Return type:

Optional[str]

Example

>>> upload_url = s3.generate_upload_url(
...     bucket='my-bucket',
...     key='uploads/file.jpg',
...     ContentType='image/jpeg',
...     ACL='private',
...     Metadata={
...         'custom': 'value'
...     }
... )

S3Client.generate_download_url(bucket: str, key: str, expiration: int = 3600, **kwargs: Any) → str | None

Generate a pre-signed URL for downloading a file from S3.

This is a convenience wrapper around generate_presigned_url for downloads.

Parameters:

• bucket – S3 bucket name

• key – S3 object key of the file to download

• expiration – Time in seconds until the URL expires (default: 1 hour)

• **kwargs – Additional parameters to pass to the S3 get_object operation

Returns:

The pre-signed URL as a string, or None if credentials are invalid

Return type:

Optional[str]

Example

>>> download_url = s3.generate_download_url(
...     bucket='my-bucket',
...     key='downloads/file.txt',
...     ResponseContentType='application/pdf',
...     ResponseContentDisposition='attachment; filename=report.pdf'
... )

Error Handling

All methods return a dictionary with the following structure:

{
    "status": "success" | "error",
    "bucket": "bucket-name",
    "key": "object-key",
    # Only present if status is "error"
    "error": "error-message",
    # Additional fields may be present depending on the operation
}

Thread Safety

The S3Client is implemented as a thread-safe singleton. Multiple threads can safely use the same instance.

Dependencies

• boto3 >= 1.28.0

• botocore >= 1.31.0