twindb_backup.destination package

Submodules

twindb_backup.destination.base_destination module

Module defines Base destination class and destination exception(s).

class twindb_backup.destination.base_destination.BaseDestination(remote_path)

Bases: object

Base destination class.

Parameters:remote_path (str) – Common path where all backups are stored. The remote path is specific to the actual destination class. For example, for Amazon S3 it will be something like s3://bucket_with_backups

delete(path)

Delete object from the destination

Parameters:path – Relative path to the file to delete

get_stream(copy)

Get a PIPE handler with content of the backup copy streamed from the destination.

Parameters:copy (BaseCopy) – Backup copy Returns:Standard output.

list_files(prefix=None, recursive=False, pattern=None, files_only=False)

Get list of file by prefix.

Parameters:

• prefix (str) – Path
• recursive (bool) – Recursive return list of files
• pattern (str) – files must match with this regexp if specified
• files_only (bool) – If True don’t list directories

Returns:

List of files

Return type:

list

read(filepath)

Read content of a file path from destination.

Parameters:filepath (str) – Relative path to file. Returns:Content of the file. Return type:str

save(handler, filepath)

Save a stream given as handler to filepath.

Parameters:

• handler (file) – Incoming stream.
• filepath (str) – Save stream as this name.

write(content, filepath)

Write content to a file.

Parameters:

• content (str) – Content to write to the file.
• filepath (str) – Relative path to file.

twindb_backup.destination.exceptions module

Module for destination exceptions.

exception twindb_backup.destination.exceptions.DestinationError

Bases: twindb_backup.exceptions.TwinDBBackupError

General destination error

exception twindb_backup.destination.exceptions.FileNotFound

Bases: twindb_backup.destination.exceptions.DestinationError

File doesn’t exist on destination

exception twindb_backup.destination.exceptions.GCSDestinationError

Bases: twindb_backup.destination.exceptions.DestinationError

GCS destination errors

exception twindb_backup.destination.exceptions.S3DestinationError

Bases: twindb_backup.destination.exceptions.DestinationError

S3 destination errors

exception twindb_backup.destination.exceptions.SshDestinationError

Bases: twindb_backup.destination.exceptions.DestinationError

SSH destination errors

twindb_backup.destination.local module

Module defines Local destination.

class twindb_backup.destination.local.Local(path=None)

Bases: twindb_backup.destination.base_destination.BaseDestination

Local destination class.

delete(path)

Delete object from the destination

Parameters:path – Relative path to the file to delete

get_stream(copy)

Get a PIPE handler with content of the backup copy streamed from the destination

Parameters:copy (BaseCopy) – Backup copy Returns:

path

Root path on local file system where local backup copies are stored.

read(filepath)

Read content of a file path from destination.

Parameters:filepath (str) – Relative path to file. Returns:Content of the file. Return type:str

save(handler, filepath)

Read from handler and save it on local storage

Parameters:

• filepath – store backup copy as this name
• handler – Input stream

write(content, filepath)

Write content to a file.

Parameters:

• content (str) – Content to write to the file.
• filepath (str) – Relative path to file.

twindb_backup.destination.s3 module

Module for S3 destination.

class twindb_backup.destination.s3.S3(**kwargs)

Bases: twindb_backup.destination.base_destination.BaseDestination

S3 destination class.

Parameters:kwargs – Keyword arguments.

• bucket - S3 bucket name.
• aws_access_key_id - AWS key id.
• aws_secret_access_key - AWS secret key.
• aws_default_region - AWS default region.
• hostname - Hostname of a host where a backup is taken from.

bucket

S3 bucket name.

create_bucket()

Creates the bucket in s3 that will store the backups.

Raises:S3DestinationError – if failed to create the bucket.

delete(path)

Deletes an S3 object.

Parameters:path (str) – Key of S3 object. Raises:S3DestinationError – if failed to delete object.

delete_all_objects()

Delete all objects from S3 bucket.

Raises:S3DestinationError – if failed to delete objects from the bucket.

delete_bucket(force=False)

Delete the bucket in s3 that was storing the backups.

Parameters:force (bool) – If the bucket is non-empty then delete the objects before deleting the bucket. Raises:S3DestinationError – if failed to delete the bucket.

get_stream(copy)

Get a PIPE handler with content of the backup copy streamed from the destination.

Parameters:copy (BaseCopy) – Backup copy Returns:Stream with backup copy Return type:generator Raises:S3DestinationError – if failed to stream a backup copy.

static get_transfer_config()

Build Transfer config

Returns:Transfer config Return type:boto3.s3.transfer.TransferConfig

list_files(prefix=None, recursive=False, pattern=None, files_only=False)

List files in the destination that have common prefix.

Parameters:

• prefix (str) – Common prefix. May include the bucket name. (e.g. s3://my_bucket/foo/) or simply a prefix in the bucket (e.g. foo/).
• recursive – Does nothing for this class.
• pattern (str) – files must match with this regexp if specified.
• files_only – Does nothing for this class.

Returns:

sorted list of file names.

Returns:

Full S3 url in form s3://bucket/path/to/file.

Return type:

list(str)

Raises:

S3DestinationError – if failed to list files.

read(filepath)

Read content of filepath and return it as a string.

Parameters:filepath – Path in S3 bucket. Returns:Content of the file. Return type:str Raises:FileNotFound – If filepath doesn’t exist.

save(handler, filepath)

Read from handler and save it to Amazon S3

Parameters:

• filepath – save backup copy in a file with this name
• handler – stdout handler from backup source

static setup_s3_client()

Creates an authenticated s3 client.

Returns:S3 client instance. Return type:botocore.client.BaseClient

share(s3_url)

Share S3 file and return public link

Parameters:s3_url (str) – S3 url Returns:Public url Return type:str Raises:S3DestinationError – if failed to share object.

static validate_client_response(response)

Validates the response returned by the client. Raises an exception

if the response code is not 200 or 204

Parameters:response (dict) – The response that needs to be validated. Raises:S3DestinationError – if response from S3 is invalid.

write(content, filepath)

Write content to a file.

Parameters:

• content (str) – Content to write to the file.
• filepath (str) – Relative path to file.

class twindb_backup.destination.s3.S3FileAccess

Bases: object

Access modes for S3 files

private = 'private'

public_read = 'public-read'

twindb_backup.destination.ssh module

Module for SSH destination.

class twindb_backup.destination.ssh.Ssh(remote_path, **kwargs)

Bases: twindb_backup.destination.base_destination.BaseDestination

The SSH destination class represents a destination backup storage with running SSH demon.

Parameters:

• remote_path (str) – Path to store backups.
• kwargs (dict) – Keyword arguments. See below.

• hostname (str): Hostname of the host where backup is taken from.
• ssh_host (str): Hostname for SSH connection. Default 127.0.0.1.
• ssh_user (str): Username for SSH connection. Default root.
• ssh_port (int): TCP port for SSH connection. Default 22.

• ssh_key (str): File with an rsa/dsa key for SSH authentication.

  Default /root/.ssh/id_rsa.

client

Returns:SSH client. Return type:SshClient

delete(path)

Delete file by path. The path is a relative to the self.remote_path.

Parameters:path (str) – Path to a remote file.

ensure_tcp_port_listening(port, wait_timeout=10)

Check that tcp port is open and ready to accept connections. Keep checking up to wait_timeout seconds.

Parameters:

• port (int) – TCP port that is supposed to be listening.
• wait_timeout (int) – Wait this many seconds until the port is ready.

Returns:

True if the TCP port is listening.

Return type:

bool

execute_command(cmd, quiet=False, background=False)

Execute ssh command on the remote destination.

Parameters:

• cmd (str) – Command to execute.
• quiet (bool) – If True don’t print errors.
• background (bool) – If True don’t wait until the command exits.

Returns:

stdin, stdout and stderr handlers.

Return type:

tuple

get_stream(copy)

Get a PIPE handler with content of the backup copy streamed from the destination.

Parameters:copy (BaseCopy) – Backup copy. Returns:Standard output. Return type:file

host

Returns:IP address of the destination. Return type:str

netcat(command, port=9990)

Run netcat on the destination pipe it to a given command:

ncat -l <port> --recv-only | <command>

Parameters:

• command (str) – Command that would accept netcat’s output.
• port (int) – TCP port to run netcat on. Default 9999.

port

Returns:TCP port of the destination. Return type:int

read(filepath)

Read content of a file path from destination.

Parameters:filepath (str) – Relative path to file. Returns:Content of the file. Return type:str

save(handler, filepath)

Read from the handler and save it on the remote ssh server in a file filepath.

Parameters:

• filepath (str) – Relative path to a file to store the backup copy.
• handler (file) – Stream with content of the backup.

user

Returns:SSH user. Return type:str

write(content, filepath)

Write content to a file.

Parameters:

• content (str) – Content to write to the file.
• filepath (str) – Relative path to file.

Module contents