Add boto3 as s3 backend option

README.md

@@ -52,7 +52,7 @@ Install `django-pgclone` with:
 
 After this, add `pgclone` to the `INSTALLED_APPS` setting of your Django project.
 
-**Note**  Install the AWS CLI to enable the S3 storage backend. Use `pip install awscli` or follow the [installation guide here](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html).
+**Note**  Install the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) to enable the S3 storage backend. Alternatively, install the optional S3 extra (`pip install django-pgclone[s3]`) and set `PGCLONE_S3_BACKEND = "boto3"`.
 
 ## Contributing Guide
 

docs/installation.md

@@ -8,4 +8,4 @@ After this, add `pgclone` to the `INSTALLED_APPS` setting of your Django project
 
 !!! note
 
-    Install the AWS CLI to enable the S3 storage backend. Use `pip install awscli` or follow the [installation guide here](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html).
+    Install the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) to enable the S3 storage backend. Alternatively, install the optional S3 extra (`pip install django-pgclone[s3]`) and set `PGCLONE_S3_BACKEND = "boto3"`.

docs/settings.md

@@ -80,7 +80,7 @@ The hooks to run by default for restores before swapping happens.
 
 ## PGCLONE_S3_CONFIG
 
-The environment variable overrides when using the AWS CLI. Only applicable when using the S3 storage backend.
+The AWS credentials and region configuration for the S3 storage backend. Applies to both the boto3 and AWS CLI backends.
 
 For example:
 
@@ -95,6 +95,20 @@ PGCLONE_S3_CONFIG = {
 
 **Default**: `{}`
 
+## PGCLONE_S3_BACKEND
+
+The S3 backend to use. Must be `"boto3"` or `"awscli"`.
+
+When unset, the AWS CLI backend is used.
+
+For example:
+
+```python
+PGCLONE_S3_BACKEND = "boto3"
+```
+
+**Default**: `"awscli"`
+
 ## PGCLONE_S3_ENDPOINT_URL
 
 The S3 endpoint url to send requests to if using a non-standard AWS endpoint or an S3 service other than AWS (such as DigitalOcean Spaces or self-hosting an endpoint directly within your private VPC). Only applicable when using the S3 storage backend.

docs/storage.md

@@ -8,15 +8,27 @@ S3 storage is enabled by configuring a path that starts with `s3://`. A bucket a
 
 When using S3, dumps and restores are streamed, reducing the memory consumption required for large databases.
 
-In order to use the S3 storage backend, one must additionally install the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html). Earlier versions of the CLI can be installed with `pip install awscli`.
+### AWS CLI (default)
+
+Install the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html). Earlier versions of the CLI can be installed with `pip install awscli`.
+
+This is the default backend and requires no additional configuration.
 
 !!! warning
 
     Installing the AWS CLI with pip can cause dependency issues in projects that depend on later versions of colorama or docutils. It is recommended to manually install the CLI using the [installation instructions](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) if the pip version doesn't work.
 
+### boto3 (alternative)
+
+Install the optional S3 extra:
+
+    pip install django-pgclone[s3]
+
+This installs `boto3`. No external binaries are required. Set `settings.PGCLONE_S3_BACKEND = "boto3"` to use it.
+
 ## Configuring the S3 backend
 
-The AWS CLI can be configured by environment variables. Inject custom environment variables by configuring `settings.PGCLONE_S3_CONFIG`. Here we override the AWS credentials and region:
+S3 credentials and region can be configured with `settings.PGCLONE_S3_CONFIG`. Here we override the AWS credentials and region:
 
 ```python
 
@@ -29,10 +41,20 @@ PGCLONE_S3_CONFIG = {
 }
 ```
 
-See [this guide](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html) for all environment variables that can be used with the AWS CLI.
+When using the boto3 backend, unset keys fall through to boto3's default credential chain (environment variables, IAM roles, profiles, etc.). When using the AWS CLI backend, these values are passed as environment variables to the `aws` subprocess.
 
-If using a non-standard AWS endpoint url or a non-AWS S3 provider, the endpoint url must be specified. Unfortunately, AWS CLI does not provide an environmental variable for this purpose. Instead, use the `settings.PGCLONE_S3_ENDPOINT_URL` setting, which will override the AWS CLI commands with the `--endpoint-url` option. `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` must still be specified in `settings.PGCLONE_S3_CONFIG`.
+See [this guide](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html) for all environment variables that can be used with S3.
+
+If using a non-standard AWS endpoint url or a non-AWS S3 provider, the endpoint url must be specified with `settings.PGCLONE_S3_ENDPOINT_URL`. `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` must still be specified in `settings.PGCLONE_S3_CONFIG` when not using IAM roles or other automatic credential providers.
 
 ```python
 PGCLONE_S3_ENDPOINT_URL = "https://endpoint.example.com"
 ```
+
+## S3 backend selection
+
+By default, `django-pgclone` uses the AWS CLI backend. Override this behavior with `settings.PGCLONE_S3_BACKEND`:
+
+```python
+PGCLONE_S3_BACKEND = "boto3"  # or "awscli"
+```

pgclone/dump_cmd.py

@@ -53,13 +53,12 @@ def _dump(
     # Note - do note format {db_dump_url} with an `f` string.
     # It will be formatted later when running the command.
     pg_dump_cmd_fmt = "pg_dump -Fc --no-acl --no-owner {db_dump_url} " + exclude_args
-    pg_dump_cmd_fmt += " " + storage_client.pg_dump(file_path)
 
     anon_pg_dump_cmd = pg_dump_cmd_fmt.format(db_dump_url="<DB_URL>")
     logging.success_msg(f"Creating DB copy with cmd: {anon_pg_dump_cmd}")
 
     pg_dump_cmd = pg_dump_cmd_fmt.format(db_dump_url=db.url(dump_db))
-    run.shell(pg_dump_cmd, env=storage_client.env, pipefail=True)
+    storage_client.run_pg_dump(pg_dump_cmd, file_path)
 
     logging.success_msg(f'Database "{database}" successfully dumped to "{dump_key}"')
 

pgclone/restore_cmd.py

@@ -110,13 +110,12 @@ def _remote_restore(
 
     logging.success_msg(f'Running pg_restore on "{dump_key}"')
     pg_restore_cmd = f"pg_restore --verbose --no-acl --no-owner -d {db.url(temp_db)}"
-    pg_restore_cmd = storage_client.pg_restore(file_path) + " " + pg_restore_cmd
 
     # When restoring, we need to ignore errors because there are certain
     # errors we cannot get around when pg restoring some DBs (like Aurora).
     # In the future, we may parse the output of the pg_restore command to see
     # if an unexpected error happened.
-    run.shell(pg_restore_cmd, env=storage_client.env, ignore_errors=True)
+    storage_client.run_pg_restore(pg_restore_cmd, file_path)
 
     return dump_key
 

pgclone/settings.py

@@ -26,6 +26,18 @@ def s3_endpoint_url() -> str | None:
     return getattr(settings, "PGCLONE_S3_ENDPOINT_URL", None)
 
 
+def s3_backend() -> str:
+    backend = getattr(settings, "PGCLONE_S3_BACKEND", None)
+    if backend is not None:
+        if backend not in ("boto3", "awscli"):
+            raise exceptions.RuntimeError(
+                'Invalid PGCLONE_S3_BACKEND setting. Must be "boto3" or "awscli".'
+            )
+        return backend
+
+    return "awscli"
+
+
 def storage_location() -> str:
     location = getattr(settings, "PGCLONE_STORAGE_LOCATION", ".pgclone")
     if not location.endswith("/"):  # pragma: no cover