Tools for OVH S3 backup relying on versioned bucket.
1.9K
Provides efficient utility to set up and manage a cost-effective backup strategy for S3 compatible object stores.
This provides:
Combined with asynchronous replication, it allows a close to zero Recovery Point Objective for the given retention period.
Usage: s3-tooling [COMMAND]
S3 tooling
--disable-metrics Disable metrics if specified. By default metrics are enabled.
--verbose Enable debug logs. By default verbose logs are disabled.
Commands:
help Display help information about the specified command.
undo-deletes Undo deletes
reset-versions Delete all versions older than the provided period,
effectively reseting the bucket as it was on the
provided period
list-deleted-objects List deleted objects. An optional provided period can
be allowed as a filter. This allows auditing the
result of 'undo-deletes'
purge-bucket Purges the bucket, effectively deleting all versions of
objects marked as deleted for long enough.
copy Copy all data of the source bucket into the destination
bucket. All data is copied to the destination
regardless whether it is present or not in the
destination.
copy-versions Copy all versions of the source bucket into the
destination bucket. All data is copied to the
destination regardless whether it is present or not
in the destination.For each object only the lastest
applicable version is copied.For a version to be
applicable it needs to be older than the provided
period.
sync Synchronizes all data of the source bucket into the
destination bucket. Data is copied into the
destination bucket if, and only if it is not already
present in the destination.No file is deleted from
the destination.This is slower than a copy as for
each object the destination is checked for existence.
propagate-deletes List data on the destination bucket and deletes all
data not present in the source.
list-versions List versions of an object
condense-history Condense history
Usage: s3-tooling undo-deletes [--api-key=<apiKey>] [--bucket=<bucket>]
[--concurrency=<concurrency>]
[--period-to-undo=<period>] [--region=<region>]
[--secret-key=<secretKey>] [--url=<url>]
Undo deletes
--api-key=<apiKey> API key to access the bucket
--bucket=<bucket> Bucket where to undo deletes
--concurrency=<concurrency>
Count of objects to revert in parallel
--dry-run If specified, only logs the objects to restore. No
deletion of the deletion markers is attempted.
--lenient Shall the command stop on the first encountered
error, or proceed traitements and report error
last. Defaults to being not being lenient.
--period-to-undo=<period>
Will revert all deletion markers not older than the
provided period
--prefix=<prefix> Prefix of the objects to be managed. Defaults to
none.
--region=<region> Region of the bucket
--secret-key=<secretKey>
Secret key to access the bucket
--url=<url> URL where to undo deletes
Usage: s3-tooling reset-versions [--api-key=<apiKey>] [--bucket=<bucket>]
[--concurrency=<concurrency>]
[--period-to-reset=<period>]
[--region=<region>] [--secret-key=<secretKey>]
[--url=<url>]
Delete all versions newer than the provided period, effectively reseting the
bucket as it was on the provided period
--api-key=<apiKey> API key to access the bucket
--bucket=<bucket> Bucket where to undo deletes
--concurrency=<concurrency>
Count of objects to reset in parallel
--dry-run If specified, only logs the versions that will be
deleted. No deletes is performed.
--lenient Shall the command stop on the first encountered
error, or proceed traitements and report error
last. Defaults to being not being lenient.
--period-to-reset=<period>
Will revert all versions not older than the provided
period
--prefix=<prefix> Prefix of the objects to be managed. Defaults to
none.
--region=<region> Region of the bucket
--secret-key=<secretKey>
Secret key to access the bucket
--url=<url> URL where to undo deletes
Usage: s3-tooling list-deleted-objects [--api-key=<apiKey>] [--bucket=<bucket>]
[--period=<period>] [--region=<region>]
[--secret-key=<secretKey>] [--url=<url>]
List deleted objects. An optional provided period can be allowed as a filter.
This allows auditing the result of 'undo-deletes'.By default it lists all
deleted objects.
--api-key=<apiKey> API key to access the bucket
--bucket=<bucket> Bucket where to undo deletes
--lenient Shall the command stop on the first encountered
error, or proceed traitements and report error
last. Defaults to being not being lenient.
--period=<period> Will revert all deletion markers not older than the
provided period
--prefix=<prefix> Prefix of the objects to be managed. Defaults to
none.
--region=<region> Region of the bucket
--secret-key=<secretKey>
Secret key to access the bucket
--url=<url> URL where to undo deletes
Usage: s3-tooling purge-bucket [--api-key=<apiKey>] [--bucket=<bucket>]
[--concurrency=<concurrency>]
[--period-to-purge=<period>] [--region=<region>]
[--secret-key=<secretKey>] [--url=<url>]
Purges the bucket, effectively deleting all versions of objects marked as
deleted for long enough.
--api-key=<apiKey> API key to access the bucket
--bucket=<bucket> Bucket where to undo deletes
--concurrency=<concurrency>
Count of objects to purge in parallel
--lenient Shall the command stop on the first encountered
error, or proceed traitements and report error
last. Defaults to being not being lenient.
--period-to-purge=<period>
Will specify a horizon from which to purge data
--prefix=<prefix> Prefix of the objects to be managed. Defaults to
none.
--region=<region> Region of the bucket
--secret-key=<secretKey>
Secret key to access the bucket
--url=<url> URL where to undo deletes
Usage: s3-tooling copy [--concurrency=<concurrency>]
[--destination-api-key=<destinationApiKey>]
[--destination-bucket=<destinationBucket>]
[--destination-region=<destinationRegion>]
[--destination-secret-key=<destinationKey>]
[--destination-url=<destinationUrl>]
[--source-api-key=<sourceApiKey>]
[--source-bucket=<sourceBucket>]
[--source-region=<sourceRegion>]
[--source-secret-key=<secretKey>]
[--source-url=<sourceUrl>]
Copy all data of the source bucket into the destination bucket. All data is
copied to the destination regardless whether it is present or not in the
destination.
--concurrency=<concurrency>
Count of objects to copy in parallel
--destination-api-key=<destinationApiKey>
API key to access the destination bucket
--destination-bucket=<destinationBucket>
Destination bucket to take the data from
--destination-region=<destinationRegion>
Region of the destination bucket
--destination-secret-key=<destinationKey>
Secret key to access the destination bucket
--destination-url=<destinationUrl>
Destination URL where to take the data from
--lenient Shall the command stop on the first encountered
error, or proceed traitements and report error
last. Defaults to being not being lenient.
--prefix=<prefix> Prefix of the objects to be managed. Defaults to
none.
--source-api-key=<sourceApiKey>
API key to access the source bucket
--source-bucket=<sourceBucket>
Source bucket to take the data from
--source-region=<sourceRegion>
Region of the source bucket
--source-secret-key=<secretKey>
Secret key to access the source bucket
--source-url=<sourceUrl>
Source URL where to take the data from
Usage: s3-tooling copy-versions [--concurrency=<concurrency>]
[--destination-api-key=<destinationApiKey>]
[--destination-bucket=<destinationBucket>]
[--destination-region=<destinationRegion>]
[--destination-secret-key=<destinationKey>]
[--destination-url=<destinationUrl>]
[--period-to-undo=<period>]
[--source-api-key=<sourceApiKey>]
[--source-bucket=<sourceBucket>]
[--source-region=<sourceRegion>]
[--source-secret-key=<secretKey>]
[--source-url=<sourceUrl>]
Copy all versions of the source bucket into the destination bucket. All data is
copied to the destination regardless whether it is present or not in the
destination.For each object only the lastest applicable version is copied.For a
version to be applicable it needs to be older than the provided period.
--concurrency=<concurrency>
Count of objects to copy in parallel
--destination-api-key=<destinationApiKey>
API key to access the destination bucket
--destination-bucket=<destinationBucket>
Destination bucket to take the data from
--destination-region=<destinationRegion>
Region of the destination bucket
--destination-secret-key=<destinationKey>
Secret key to access the destination bucket
--destination-url=<destinationUrl>
Destination URL where to take the data from
--lenient Shall the command stop on the first encountered
error, or proceed traitements and report error
last. Defaults to being not being lenient.
--period-to-undo=<period>
Will copy all versions anterior to the given period
--prefix=<prefix> Prefix of the objects to be managed. Defaults to
none.
--source-api-key=<sourceApiKey>
API key to access the source bucket
--source-bucket=<sourceBucket>
Source bucket to take the data from
--source-region=<sourceRegion>
Region of the source bucket
--source-secret-key=<secretKey>
Secret key to access the source bucket
--source-url=<sourceUrl>
Source URL where to take the data from
Usage: s3-tooling sync [--concurrency=<concurrency>]
[--destination-api-key=<destinationApiKey>]
[--destination-bucket=<destinationBucket>]
[--destination-region=<destinationRegion>]
[--destination-secret-key=<destinationKey>]
[--destination-url=<destinationUrl>]
[--source-api-key=<sourceApiKey>]
[--source-bucket=<sourceBucket>]
[--source-region=<sourceRegion>]
[--source-secret-key=<secretKey>]
[--source-url=<sourceUrl>]
Synchronizes all data of the source bucket into the destination bucket. Data is
copied into the destination bucket if, and only if it is not already present in
the destination.No file is deleted from the destination.This is slower than a
copy as for each object the destination is checked for existence.
--concurrency=<concurrency>
Count of objects to sync in parallel
--destination-api-key=<destinationApiKey>
API key to access the destination bucket
--destination-bucket=<destinationBucket>
Destination bucket to take the data from
--destination-region=<destinationRegion>
Region of the destination bucket
--destination-secret-key=<destinationKey>
Secret key to access the destination bucket
--destination-url=<destinationUrl>
Destination URL where to take the data from
--dry-run If specified, only logs the objects present in the source and
not in the destination. No copy is performed.
--lenient Shall the command stop on the first encountered
error, or proceed traitements and report error
last. Defaults to being not being lenient.
--prefix=<prefix> Prefix of the objects to be managed. Defaults to
none.
--source-api-key=<sourceApiKey>
API key to access the source bucket
--source-bucket=<sourceBucket>
Source bucket to take the data from
--source-region=<sourceRegion>
Region of the source bucket
--source-secret-key=<secretKey>
Secret key to access the source bucket
--source-url=<sourceUrl>
Source URL where to take the data from
Usage: s3-tooling propagate-deletes [--concurrency=<concurrency>]
[--destination-api-key=<destinationApiKey>]
[--destination-bucket=<destinationBucket>]
[--destination-region=<destinationRegion>]
[--destination-secret-key=<destinationKey>]
[--destination-url=<destinationUrl>]
[--source-api-key=<sourceApiKey>]
[--source-bucket=<sourceBucket>]
[--source-region=<sourceRegion>]
[--source-secret-key=<secretKey>]
[--source-url=<sourceUrl>]
List data on the destination bucket and deletes all data not present in the
source.
--concurrency=<concurrency>
Count of deletes to propagate in parallel
--destination-api-key=<destinationApiKey>
API key to access the destination bucket
--destination-bucket=<destinationBucket>
Destination bucket to take the data from
--destination-region=<destinationRegion>
Region of the destination bucket
--destination-secret-key=<destinationKey>
Secret key to access the destination bucket
--destination-url=<destinationUrl>
Destination URL where to take the data from
--dry-run If specified, only logs the objects present in the
destination and not in the source. No delete is performed.
--lenient Shall the command stop on the first encountered
error, or proceed traitements and report error
last. Defaults to being not being lenient.
--prefix=<prefix> Prefix of the objects to be managed. Defaults to
none.
--source-api-key=<sourceApiKey>
API key to access the source bucket
--source-bucket=<sourceBucket>
Source bucket to take the data from
--source-region=<sourceRegion>
Region of the source bucket
--source-secret-key=<secretKey>
Secret key to access the source bucket
--source-url=<sourceUrl>
Source URL where to take the data from
Usage: s3-tooling list-versions [--disable-metrics]... [--verbose]...
[--api-key=<apiKey>] [--bucket=<bucket>]
[--key=<key>] [--region=<region>]
[--secret-key=<secretKey>] [--url=<url>]
List versions of an object
--api-key=<apiKey> API key to access the bucket
--bucket=<bucket> Bucket where to undo deletes
--disable-metrics Disable metrics if specified. By default metrics are
enabled.
--key=<key> key of the object
--region=<region> Region of the bucket
--secret-key=<secretKey>
Secret key to access the bucket
--url=<url> URL where to undo deletes
--verbose Enable debug logs. By default verbose logs are
disabled.
Usage: s3-tooling condense-history [--dry-run] [--lenient]
[--enable-metrics]... [--verbose]...
[--api-key=<apiKey>] [--bucket=<bucket>]
[--concurrency=<concurrency>]
[--prefix=<prefix>] [--region=<region>]
[--secret-key=<secretKey>] [--url=<url>]
Condense history
--api-key=<apiKey> API key to access the bucket
--bucket=<bucket> Bucket to compact
--concurrency=<concurrency>
Count of objects to compact in parallel
--dry-run If specified, only logs the object versions to
delete. No deletion is attempted.
--enable-metrics Enable metrics if specified. By default metrics are
disabled.
--force If specified, force the run of the tool and skip the
disclaimer. Meant for automated runs.
--lenient Shall the command stop on the first encountered
error, or proceed traitements and report error
last. Defaults to being not being lenient.
--prefix=<prefix> Prefix of the objects to be managed. Defaults to
none.
--region=<region> Region of the bucket
--secret-key=<secretKey>
Secret key to access the bucket
--url=<url> URL where to compact
--verbose Enable debug logs. By default verbose logs are
disabled.
In order to ease usage the default value for all of the above is a set of environment variable:
S3_CONCURRENCY
S3_PERIOD
S3_PREFIX
S3_BUCKET
S3_URL
S3_API_KEY
S3_SECRET_KEY
S3_REGION
S3_SOURCE_BUCKET
S3_SOURCE_URL
S3_SOURCE_API_KEY
S3_SOURCE_SECRET_KEY
S3_SOURCE_REGION
S3_DESTINATION_BUCKET
S3_DESTINATION_URL
S3_DESTINATION_API_KEY
S3_DESTINATION_SECRET_KEY
S3_DESTINATION_REGION
Compile this project:
mvn clean install
And run it:
alias s3-tooling='java -cp target/s3-tooling-1.0-SNAPSHOT-jar-with-dependencies.jar com.linagora.james.s3.tooling.Main'
s3-tooling help
OpenJDK JRE 21 needs to be installed as a dependency.
This can be used if you do not have a Java environment locally:
docker build -t s3-tooling .
docker run --rm s3-tooling help
Content type
Image
Digest
sha256:b4a839b21…
Size
273.6 MB
Last updated
over 2 years ago
docker pull chibenwa/s3-tooling