twindb_backup.destination package

Submodules

twindb_backup.destination.base_destination module

Module defines Base destination class and destination exception(s).

class twindb_backup.destination.base_destination.BaseDestination(remote_path)

Bases: object

Base destination class.

Parameters

remote_path (str) – Common path where all backups are stored. The remote path is specific to the actual destination class. For example, for Amazon S3 it will be something like s3://bucket_with_backups

abstract delete(path)

Delete object from the destination

Parameters

path – Relative path to the file to delete

abstract get_stream(copy)

Get a PIPE handler with content of the backup copy streamed from the destination.

Parameters

copy (BaseCopy) – Backup copy

Returns

Standard output.

list_files(prefix=None, recursive=False, pattern=None, files_only=False)

Get list of file by prefix.

Parameters

• prefix (str) – Path

• recursive (bool) – Recursive return list of files

• pattern (str) – files must match with this regexp if specified

• files_only (bool) – If True don’t list directories

Returns

List of files

Return type

list

abstract read(filepath)

Read content of a file path from destination.

Parameters

filepath (str) – Relative path to file.

Returns

Content of the file.

Return type

str

abstract save(handler, filepath)

Save a stream given as handler to filepath.

Parameters

• handler (file) – Incoming stream.

• filepath (str) – Save stream as this name.

abstract write(content, filepath)

Write content to a file.

Parameters

• content (str) – Content to write to the file.

• filepath (str) – Relative path to file.

twindb_backup.destination.exceptions module

Module for destination exceptions.

exception twindb_backup.destination.exceptions.DestinationError

Bases: twindb_backup.exceptions.TwinDBBackupError

General destination error

exception twindb_backup.destination.exceptions.FileNotFound

Bases: twindb_backup.destination.exceptions.DestinationError

File doesn’t exist on destination

exception twindb_backup.destination.exceptions.GCSDestinationError

Bases: twindb_backup.destination.exceptions.DestinationError

GCS destination errors

exception twindb_backup.destination.exceptions.S3DestinationError

Bases: twindb_backup.destination.exceptions.DestinationError

S3 destination errors

exception twindb_backup.destination.exceptions.SshDestinationError

Bases: twindb_backup.destination.exceptions.DestinationError

SSH destination errors

twindb_backup.destination.local module

Module defines Local destination.

class twindb_backup.destination.local.Local(path=None)

Bases: twindb_backup.destination.base_destination.BaseDestination

Local destination class.

delete(path)

Delete object from the destination

Parameters

path – Relative path to the file to delete

get_stream(copy)

Get a PIPE handler with content of the backup copy streamed from the destination

Parameters

copy (BaseCopy) – Backup copy

Returns

property path

Root path on local file system where local backup copies are stored.

read(filepath)

Read content of a file path from destination.

Parameters

filepath (str) – Relative path to file.

Returns

Content of the file.

Return type

str

save(handler, filepath)

Read from handler and save it on local storage

Parameters

• filepath – store backup copy as this name

• handler – Input stream

write(content, filepath)

Write content to a file.

Parameters

• content (str) – Content to write to the file.

• filepath (str) – Relative path to file.

twindb_backup.destination.s3 module

Module for S3 destination.

class twindb_backup.destination.s3.S3(**kwargs)

Bases: twindb_backup.destination.base_destination.BaseDestination

S3 destination class.

Parameters

kwargs – Keyword arguments.

• bucket - S3 bucket name.

• aws_access_key_id - AWS key id.

• aws_secret_access_key - AWS secret key.

• aws_default_region - AWS default region.

• hostname - Hostname of a host where a backup is taken from.

property bucket

S3 bucket name.

create_bucket()

Creates the bucket in s3 that will store the backups.

Raises

S3DestinationError – if failed to create the bucket.

delete(path)

Deletes an S3 object.

Parameters

path (str) – Key of S3 object.

Raises

S3DestinationError – if failed to delete object.

delete_all_objects()

Delete all objects from S3 bucket.

Raises

S3DestinationError – if failed to delete objects from the bucket.

delete_bucket(force=False)

Delete the bucket in s3 that was storing the backups.

Parameters

force (bool) – If the bucket is non-empty then delete the objects before deleting the bucket.

Raises

S3DestinationError – if failed to delete the bucket.

get_stream(copy)

Get a PIPE handler with content of the backup copy streamed from the destination.

Parameters

copy (BaseCopy) – Backup copy

Returns

Stream with backup copy

Return type

generator

Raises

S3DestinationError – if failed to stream a backup copy.

static get_transfer_config()

Build Transfer config

Returns

Transfer config

Return type

boto3.s3.transfer.TransferConfig

list_files(prefix=None, recursive=False, pattern=None, files_only=False)

List files in the destination that have common prefix.

Parameters

• prefix (str) – Common prefix. May include the bucket name. (e.g. s3://my_bucket/foo/) or simply a prefix in the bucket (e.g. foo/).

• recursive – Does nothing for this class.

• pattern (str) – files must match with this regexp if specified.

• files_only – Does nothing for this class.

Returns

sorted list of file names.

Returns

Full S3 url in form s3://bucket/path/to/file.

Return type

list(str)

Raises

S3DestinationError – if failed to list files.

read(filepath)

Read content of filepath and return it as a string.

Parameters

filepath – Path in S3 bucket.

Returns

Content of the file.

Return type

str

Raises

FileNotFound – If filepath doesn’t exist.

save(handler, filepath)

Read from handler and save it to Amazon S3

Parameters

• filepath – save backup copy in a file with this name

• handler – stdout handler from backup source

static setup_s3_client()

Creates an authenticated s3 client.

Returns

S3 client instance.

Return type

botocore.client.BaseClient

share(s3_url)

Share S3 file and return public link

Parameters

s3_url (str) – S3 url

Returns

Public url

Return type

str

Raises

S3DestinationError – if failed to share object.

static validate_client_response(response)

Validates the response returned by the client. Raises an exception

if the response code is not 200 or 204

Parameters

response (dict) – The response that needs to be validated.

Raises

S3DestinationError – if response from S3 is invalid.

write(content, filepath)

Write content to a file.

Parameters

• content (str) – Content to write to the file.

• filepath (str) – Relative path to file.

class twindb_backup.destination.s3.S3FileAccess

Bases: object

Access modes for S3 files

private = 'private'

public_read = 'public-read'

twindb_backup.destination.ssh module

Module for SSH destination.

class twindb_backup.destination.ssh.Ssh(remote_path, **kwargs)

Bases: twindb_backup.destination.base_destination.BaseDestination

The SSH destination class represents a destination backup storage with running SSH demon.

Parameters

• remote_path (str) – Path to store backups.

• kwargs (dict) – Keyword arguments. See below.

• hostname (str): Hostname of the host where backup is taken from.

• ssh_host (str): Hostname for SSH connection. Default 127.0.0.1.

• ssh_user (str): Username for SSH connection. Default root.

• ssh_port (int): TCP port for SSH connection. Default 22.

• ssh_key (str): File with an rsa/dsa key for SSH authentication.

  Default /root/.ssh/id_rsa.

property client

Returns

SSH client.

Return type

SshClient

delete(path)

Delete file by path. The path is a relative to the self.remote_path.

Parameters

path (str) – Path to a remote file.

ensure_tcp_port_listening(port, wait_timeout=10)

Check that tcp port is open and ready to accept connections. Keep checking up to wait_timeout seconds.

Parameters

• port (int) – TCP port that is supposed to be listening.

• wait_timeout (int) – Wait this many seconds until the port is ready.

Returns

True if the TCP port is listening.

Return type

bool

execute_command(cmd, quiet=False, background=False)

Execute ssh command on the remote destination.

Parameters

• cmd (str) – Command to execute.

• quiet (bool) – If True don’t print errors.

• background (bool) – If True don’t wait until the command exits.

Returns

stdin, stdout and stderr handlers.

Return type

tuple

get_stream(copy)

Get a PIPE handler with content of the backup copy streamed from the destination.

Parameters

copy (BaseCopy) – Backup copy.

Returns

Standard output.

Return type

file

property host

Returns

IP address of the destination.

Return type

str

netcat(command, port=9990)

Run netcat on the destination pipe it to a given command:

ncat -l <port> --recv-only | <command>

Parameters

• command (str) – Command that would accept netcat’s output.

• port (int) – TCP port to run netcat on. Default 9999.

property port

Returns

TCP port of the destination.

Return type

int

read(filepath)

Read content of a file path from destination.

Parameters

filepath (str) – Relative path to file.

Returns

Content of the file.

Return type

str

save(handler, filepath)

Read from the handler and save it on the remote ssh server in a file filepath.

Parameters

• filepath (str) – Relative path to a file to store the backup copy.

• handler (file) – Stream with content of the backup.

property user

Returns

SSH user.

Return type

str

write(content, filepath)

Write content to a file.

Parameters

• content (str) – Content to write to the file.

• filepath (str) – Relative path to file.

Module contents