twindb_backup.destination package¶
Submodules¶
twindb_backup.destination.base_destination module¶
Module defines Base destination class and destination exception(s).
- class twindb_backup.destination.base_destination.BaseDestination(remote_path)[source]¶
Bases:
object
Base destination class.
- Parameters
remote_path (str) – Common path where all backups are stored. The remote path is specific to the actual destination class. For example, for Amazon S3 it will be something like
s3://bucket_with_backups
- abstract delete(path)[source]¶
Delete object from the destination
- Parameters
path – Relative path to the file to delete
- abstract get_stream(copy)[source]¶
Get a PIPE handler with content of the backup copy streamed from the destination.
- Parameters
copy (BaseCopy) – Backup copy
- Returns
Standard output.
- list_files(prefix=None, recursive=False, pattern=None, files_only=False)[source]¶
Get list of file by prefix.
- Parameters
prefix (str) – Path
recursive (bool) – Recursive return list of files
pattern (str) – files must match with this regexp if specified
files_only (bool) – If True don’t list directories
- Returns
List of files
- Return type
list
- abstract read(filepath)[source]¶
Read content of a file path from destination.
- Parameters
filepath (str) – Relative path to file.
- Returns
Content of the file.
- Return type
str
twindb_backup.destination.exceptions module¶
Module for destination exceptions.
- exception twindb_backup.destination.exceptions.DestinationError[source]¶
Bases:
twindb_backup.exceptions.TwinDBBackupError
General destination error
- exception twindb_backup.destination.exceptions.FileNotFound[source]¶
Bases:
twindb_backup.destination.exceptions.DestinationError
File doesn’t exist on destination
- exception twindb_backup.destination.exceptions.GCSDestinationError[source]¶
Bases:
twindb_backup.destination.exceptions.DestinationError
GCS destination errors
- exception twindb_backup.destination.exceptions.S3DestinationError[source]¶
Bases:
twindb_backup.destination.exceptions.DestinationError
S3 destination errors
- exception twindb_backup.destination.exceptions.SshDestinationError[source]¶
Bases:
twindb_backup.destination.exceptions.DestinationError
SSH destination errors
twindb_backup.destination.local module¶
Module defines Local destination.
- class twindb_backup.destination.local.Local(path=None)[source]¶
Bases:
twindb_backup.destination.base_destination.BaseDestination
Local destination class.
- delete(path)[source]¶
Delete object from the destination
- Parameters
path – Relative path to the file to delete
- get_stream(copy)[source]¶
Get a PIPE handler with content of the backup copy streamed from the destination
- Parameters
copy (BaseCopy) – Backup copy
- Returns
- property path¶
Root path on local file system where local backup copies are stored.
- read(filepath)[source]¶
Read content of a file path from destination.
- Parameters
filepath (str) – Relative path to file.
- Returns
Content of the file.
- Return type
str
twindb_backup.destination.s3 module¶
Module for S3 destination.
- class twindb_backup.destination.s3.S3(**kwargs)[source]¶
Bases:
twindb_backup.destination.base_destination.BaseDestination
S3 destination class.
- Parameters
kwargs – Keyword arguments.
bucket - S3 bucket name.
aws_access_key_id - AWS key id.
aws_secret_access_key - AWS secret key.
aws_default_region - AWS default region.
hostname - Hostname of a host where a backup is taken from.
- property bucket¶
S3 bucket name.
- create_bucket()[source]¶
Creates the bucket in s3 that will store the backups.
- Raises
S3DestinationError – if failed to create the bucket.
- delete(path)[source]¶
Deletes an S3 object.
- Parameters
path (str) – Key of S3 object.
- Raises
S3DestinationError – if failed to delete object.
- delete_all_objects()[source]¶
Delete all objects from S3 bucket.
- Raises
S3DestinationError – if failed to delete objects from the bucket.
- delete_bucket(force=False)[source]¶
Delete the bucket in s3 that was storing the backups.
- Parameters
force (bool) – If the bucket is non-empty then delete the objects before deleting the bucket.
- Raises
S3DestinationError – if failed to delete the bucket.
- get_stream(copy)[source]¶
Get a PIPE handler with content of the backup copy streamed from the destination.
- Parameters
copy (BaseCopy) – Backup copy
- Returns
Stream with backup copy
- Return type
generator
- Raises
S3DestinationError – if failed to stream a backup copy.
- static get_transfer_config()[source]¶
Build Transfer config
- Returns
Transfer config
- Return type
boto3.s3.transfer.TransferConfig
- list_files(prefix=None, recursive=False, pattern=None, files_only=False)[source]¶
List files in the destination that have common prefix.
- Parameters
prefix (str) – Common prefix. May include the bucket name. (e.g.
s3://my_bucket/foo/
) or simply a prefix in the bucket (e.g.foo/
).recursive – Does nothing for this class.
pattern (str) – files must match with this regexp if specified.
files_only – Does nothing for this class.
- Returns
sorted list of file names.
- Returns
Full S3 url in form
s3://bucket/path/to/file
.- Return type
list(str)
- Raises
S3DestinationError – if failed to list files.
- read(filepath)[source]¶
Read content of filepath and return it as a string.
- Parameters
filepath – Path in S3 bucket.
- Returns
Content of the file.
- Return type
str
- Raises
FileNotFound – If filepath doesn’t exist.
- save(handler, filepath)[source]¶
Read from handler and save it to Amazon S3
- Parameters
filepath – save backup copy in a file with this name
handler – stdout handler from backup source
- static setup_s3_client()[source]¶
Creates an authenticated s3 client.
- Returns
S3 client instance.
- Return type
botocore.client.BaseClient
Share S3 file and return public link
- Parameters
s3_url (str) – S3 url
- Returns
Public url
- Return type
str
- Raises
S3DestinationError – if failed to share object.
- static validate_client_response(response)[source]¶
- Validates the response returned by the client. Raises an exception
if the response code is not 200 or 204
- Parameters
response (dict) – The response that needs to be validated.
- Raises
S3DestinationError – if response from S3 is invalid.
twindb_backup.destination.ssh module¶
Module for SSH destination.
- class twindb_backup.destination.ssh.Ssh(remote_path, **kwargs)[source]¶
Bases:
twindb_backup.destination.base_destination.BaseDestination
The SSH destination class represents a destination backup storage with running SSH demon.
- Parameters
remote_path (str) – Path to store backups.
kwargs (dict) – Keyword arguments. See below.
hostname (str): Hostname of the host where backup is taken from.
ssh_host (str): Hostname for SSH connection. Default
127.0.0.1
.ssh_user (str): Username for SSH connection. Default
root
.ssh_port (int): TCP port for SSH connection. Default 22.
- ssh_key (str): File with an rsa/dsa key for SSH authentication.
Default
/root/.ssh/id_rsa
.
- delete(path)[source]¶
Delete file by path. The path is a relative to the
self.remote_path
.- Parameters
path (str) – Path to a remote file.
- ensure_tcp_port_listening(port, wait_timeout=10, wait=True)[source]¶
Check that tcp port is open and ready to accept connections. Keep checking up to
wait_timeout
seconds.- Parameters
port (int) – TCP port that is supposed to be listening.
wait_timeout (int) – Wait this many seconds until the port is ready.
wait (bool) – Wait a wait_timeout of seconds until the TCP port become available. If set to False the method will return after the first check.
- Returns
True
if the TCP port is listening.- Return type
bool
- execute_command(cmd, quiet=False, background=False)[source]¶
Execute ssh command on the remote destination.
- Parameters
cmd (str) – Command to execute.
quiet (bool) – If
True
don’t print errors.background (bool) – If
True
don’t wait until the command exits.
- Returns
stdin, stdout and stderr handlers.
- Return type
tuple
- get_stream(copy)[source]¶
Get a PIPE handler with content of the backup copy streamed from the destination.
- Parameters
copy (BaseCopy) – Backup copy.
- Returns
Standard output.
- Return type
file
- property host¶
- Returns
IP address of the destination.
- Return type
str
- netcat(command, port=9990)[source]¶
Run
netcat
on the destination pipe it to a given command:ncat -l <port> --recv-only | <command>
- Parameters
command (str) – Command that would accept
netcat
’s output.port (int) – TCP port to run
netcat
on. Default 9999.
- property port¶
- Returns
TCP port of the destination.
- Return type
int
- read(filepath)[source]¶
Read content of a file path from destination.
- Parameters
filepath (str) – Relative path to file.
- Returns
Content of the file.
- Return type
str
- save(handler, filepath)[source]¶
Read from the handler and save it on the remote ssh server in a file
filepath
.- Parameters
filepath (str) – Relative path to a file to store the backup copy.
handler (file) – Stream with content of the backup.
- property user¶
- Returns
SSH user.
- Return type
str