twindb_backup.destination package

Submodules

twindb_backup.destination.base_destination module

Module defines Base destination class and destination exception(s).

class twindb_backup.destination.base_destination.BaseDestination(remote_path)

Bases: object

Base destination class.

Parameters

remote_path (str) – Common path where all backups are stored. The remote path is specific to the actual destination class. For example, for Amazon S3 it will be something like s3://bucket_with_backups

abstract delete(path)

Delete object from the destination

Parameters

path – Relative path to the file to delete

abstract get_stream(copy)

Get a PIPE handler with content of the backup copy streamed from the destination.

Parameters

copy (BaseCopy) – Backup copy

Returns

Standard output.

list_files(prefix=None, recursive=False, pattern=None, files_only=False)

Get list of file by prefix.

Parameters

• prefix (str) – Path

• recursive (bool) – Recursive return list of files

• pattern (str) – files must match with this regexp if specified

• files_only (bool) – If True don’t list directories

Returns

List of files

Return type

list

abstract read(filepath)

Read content of a file path from destination.

Parameters

filepath (str) – Relative path to file.

Returns

Content of the file.

Return type

str

abstract save(handler, filepath)

Save a stream given as handler to filepath.

Parameters

• handler (file) – Incoming stream.

• filepath (str) – Save stream as this name.

abstract write(content, filepath)

Write content to a file.

Parameters

• content (str) – Content to write to the file.

• filepath (str) – Relative path to file.

twindb_backup.destination.exceptions module

Module for destination exceptions.

exception twindb_backup.destination.exceptions.AzureBlobDestinationError

Bases: twindb_backup.destination.exceptions.DestinationError

Azure-blob destination errors

exception twindb_backup.destination.exceptions.DestinationError

Bases: twindb_backup.exceptions.TwinDBBackupError

General destination error

exception twindb_backup.destination.exceptions.FileNotFound

Bases: twindb_backup.destination.exceptions.DestinationError

File doesn’t exist on destination

exception twindb_backup.destination.exceptions.GCSDestinationError

Bases: twindb_backup.destination.exceptions.DestinationError

GCS destination errors

exception twindb_backup.destination.exceptions.S3DestinationError

Bases: twindb_backup.destination.exceptions.DestinationError

S3 destination errors

exception twindb_backup.destination.exceptions.SshDestinationError

Bases: twindb_backup.destination.exceptions.DestinationError

SSH destination errors

twindb_backup.destination.local module

Module defines Local destination.

class twindb_backup.destination.local.Local(path=None)

Bases: twindb_backup.destination.base_destination.BaseDestination

Local destination class.

delete(path)

Delete object from the destination

Parameters

path – Relative path to the file to delete

get_stream(copy)

Get a PIPE handler with content of the backup copy streamed from the destination

Parameters

copy (BaseCopy) – Backup copy

Returns

property path

Root path on local file system where local backup copies are stored.

read(filepath)

Read content of a file path from destination.

Parameters

filepath (str) – Relative path to file.

Returns

Content of the file.

Return type

str

save(handler, filepath)

Read from handler and save it on local storage

Parameters

• filepath – store backup copy as this name

• handler – Input stream

write(content, filepath)

Write content to a file.

Parameters

• content (str) – Content to write to the file.

• filepath (str) – Relative path to file.

twindb_backup.destination.s3 module

Module for S3 destination.

class twindb_backup.destination.s3.S3(**kwargs)

Bases: twindb_backup.destination.base_destination.BaseDestination

S3 destination class.

Parameters

kwargs – Keyword arguments.

• bucket - S3 bucket name.

• aws_access_key_id - AWS key id.

• aws_secret_access_key - AWS secret key.

• aws_default_region - AWS default region.

• hostname - Hostname of a host where a backup is taken from.

property bucket

S3 bucket name.

create_bucket()

Creates the bucket in s3 that will store the backups.

Raises

S3DestinationError – if failed to create the bucket.

delete(path)

Deletes an S3 object.

Parameters

path (str) – Key of S3 object.

Raises

S3DestinationError – if failed to delete object.

delete_all_objects()

Delete all objects from S3 bucket.

Raises

S3DestinationError – if failed to delete objects from the bucket.

delete_bucket(force=False)

Delete the bucket in s3 that was storing the backups.

Parameters

force (bool) – If the bucket is non-empty then delete the objects before deleting the bucket.

Raises

S3DestinationError – if failed to delete the bucket.

get_stream(copy)

Get a PIPE handler with content of the backup copy streamed from the destination.

Parameters

copy (BaseCopy) – Backup copy

Returns

Stream with backup copy

Return type

generator

Raises

S3DestinationError – if failed to stream a backup copy.

static get_transfer_config()

Build Transfer config

Returns

Transfer config

Return type

boto3.s3.transfer.TransferConfig

list_files(prefix=None, recursive=False, pattern=None, files_only=False)

List files in the destination that have common prefix.

Parameters

• prefix (str) – Common prefix. May include the bucket name. (e.g. s3://my_bucket/foo/) or simply a prefix in the bucket (e.g. foo/).

• recursive – Does nothing for this class.

• pattern (str) – files must match with this regexp if specified.

• files_only – Does nothing for this class.

Returns

sorted list of file names.

Returns

Full S3 url in form s3://bucket/path/to/file.

Return type

list(str)

Raises

S3DestinationError – if failed to list files.

read(filepath)

Read content of filepath and return it as a string.

Parameters

filepath – Path in S3 bucket.

Returns

Content of the file.

Return type

str

Raises

FileNotFound – If filepath doesn’t exist.

save(handler, filepath)

Read from handler and save it to Amazon S3

Parameters

• filepath – save backup copy in a file with this name

• handler – stdout handler from backup source

static setup_s3_client()

Creates an authenticated s3 client.

Returns

S3 client instance.

Return type

botocore.client.BaseClient

share(s3_url)

Share S3 file and return public link

Parameters

s3_url (str) – S3 url

Returns

Public url

Return type

str

Raises

S3DestinationError – if failed to share object.

static validate_client_response(response)

Validates the response returned by the client. Raises an exception

if the response code is not 200 or 204

Parameters

response (dict) – The response that needs to be validated.

Raises

S3DestinationError – if response from S3 is invalid.

write(content, filepath)

Write content to a file.

Parameters

• content (str) – Content to write to the file.

• filepath (str) – Relative path to file.

class twindb_backup.destination.s3.S3FileAccess

Bases: object

Access modes for S3 files

private = 'private'

public_read = 'public-read'

twindb_backup.destination.ssh module

Module for SSH destination.

class twindb_backup.destination.ssh.Ssh(remote_path, **kwargs)

Bases: twindb_backup.destination.base_destination.BaseDestination

The SSH destination class represents a destination backup storage with running SSH demon.

Parameters

• remote_path (str) – Path to store backups.

• kwargs (dict) – Keyword arguments. See below.

• hostname (str): Hostname of the host where backup is taken from.

• ssh_host (str): Hostname for SSH connection. Default 127.0.0.1.

• ssh_user (str): Username for SSH connection. Default root.

• ssh_port (int): TCP port for SSH connection. Default 22.

• ssh_key (str): File with an rsa/dsa key for SSH authentication.

  Default /root/.ssh/id_rsa.

property client

Returns

SSH client.

Return type

SshClient

delete(path)

Delete file by path. The path is a relative to the self.remote_path.

Parameters

path (str) – Path to a remote file.

ensure_tcp_port_listening(port, wait_timeout=10, wait=True)

Check that tcp port is open and ready to accept connections. Keep checking up to wait_timeout seconds.

Parameters

• port (int) – TCP port that is supposed to be listening.

• wait_timeout (int) – Wait this many seconds until the port is ready.

• wait (bool) – Wait a wait_timeout of seconds until the TCP port become available. If set to False the method will return after the first check.

Returns

True if the TCP port is listening.

Return type

bool

execute_command(cmd, quiet=False, background=False)

Execute ssh command on the remote destination.

Parameters

• cmd (str) – Command to execute.

• quiet (bool) – If True don’t print errors.

• background (bool) – If True don’t wait until the command exits.

Returns

stdin, stdout and stderr handlers.

Return type

tuple

get_stream(copy)

Get a PIPE handler with content of the backup copy streamed from the destination.

Parameters

copy (BaseCopy) – Backup copy.

Returns

Standard output.

Return type

file

property host

Returns

IP address of the destination.

Return type

str

netcat(command, port=9990)

Run netcat on the destination pipe it to a given command:

ncat -l <port> --recv-only | <command>

Parameters

• command (str) – Command that would accept netcat’s output.

• port (int) – TCP port to run netcat on. Default 9999.

property port

Returns

TCP port of the destination.

Return type

int

read(filepath)

Read content of a file path from destination.

Parameters

filepath (str) – Relative path to file.

Returns

Content of the file.

Return type

str

save(handler, filepath)

Read from the handler and save it on the remote ssh server in a file filepath.

Parameters

• filepath (str) – Relative path to a file to store the backup copy.

• handler (file) – Stream with content of the backup.

property user

Returns

SSH user.

Return type

str

write(content, filepath)

Write content to a file.

Parameters

• content (str) – Content to write to the file.

• filepath (str) – Relative path to file.

Module contents