twindb_backup.destination package

Submodules

twindb_backup.destination.azblob module

Module for Azure-blob destination.

class twindb_backup.destination.azblob.AzureBlob(remote_path: AnyStr, connection_string: AnyStr, can_do_overwrites: bool = False, cpu_cap: int = 2, max_mem_bytes: int = 536870912, default_protocol: Optional[AnyStr] = None, default_host_name: Optional[AnyStr] = None, default_container_name: Optional[AnyStr] = None, default_interval: Optional[AnyStr] = None, default_media_type: Optional[AnyStr] = None, default_fname_prefix: Optional[AnyStr] = None)[source]

Bases: twindb_backup.destination.base_destination.BaseDestination

property can_overwrite
property connection_manager
property connection_string

An Azure specific authentication string for accessing the target backup destination host

property default_container_name

The default container (aka bucket) name that we default to if a relative path string omits the reference

property default_fname_prefix
property default_host_name

The default host server name directory that we default to if a relative path string omits the reference

property default_interval

The default backup interval directory that we default to if a relative path string omits the reference

property default_media_type
property default_protocol
delete(path: AnyStr)[source]

Delete object from the destination

the general form for the path object should conform to the following:

[azure:/]/[bucket or container name]/[server name]/[update interval]/[query language]/<file name>

NOTE: The protocol type (the left-most component of the example above) is technically optional,

as it should always be an azure storage type; but if passed we will check to confirm that it is indeed for azure-blob storage, so including it ensures proper sanity checking.

– If path defines a new absolute path string then it must contain all parts defined above,

with the option to omit those components wrapped in square brackets, E.G.: [some component]

where:

[components inside square brackets] => optional <objects inside chevrons> => required

such that:

optional components that are not provided should be substituted with an ellipsis (the triple period => …)

E.G.:

…://foo/…/hourly/mysql/bar-that.foos.gz

Note:

Where optional path components are omitted, we assume that the context of the called AzureBlob instance should be used to fill in the gaps.

– If path is given as a relative path string then you may also use the ellipsis as defined for absolute paths,

with the added option to use .. for relative directory hierarchy referencing. The one caveat is that

E.G.:

../../daily/mysql/relative-foo.bar.gz

or

../../../some_different_host/…/mysql where:

The component signals that we wish to use the given default interval this object was initialized with.

Parameters

path – A string or bytes object; The path to the file (blob) to delete. Can be relative or absolute.

get_stream(copy: Union[str, twindb_backup.copy.mysql_copy.MySQLCopy])[source]

Get a PIPE handler with content of the backup copy streamed from the destination.

Parameters

copy (BaseCopy) – Backup copy

Returns

Standard output.

property max_bytes_per_pipe_message
property max_system_memory_usage
read(filepath: str, bytes_per_chunk: Optional[int] = None) bytes[source]

Read content from destination at the end of given filepath.

Parameters
  • filepath (str) – REQUIRED; a str object; Relative path to destination file that we will read from.

  • bytes_per_chunk (int) – OPTIONAL; DEFAULT = self.max_bytes_per_pipe_message; an int value; This parameter dictates the max chunk size (in bytes) that should be passed into the pipe for any single chunk.

Returns

Content of the file.

Return type

bytes

save(handler, filepath)[source]

Save a stream given as handler to filepath.

Parameters
  • handler (file) – Incoming stream.

  • filepath (str) – Save stream as this name.

write(content: Union[AnyStr, io.BufferedIOBase], filepath: AnyStr)[source]

Write content to a file.

Parameters
  • content (str, bytes, or subclass of BufferedIOBase object) – Content to write to the file.

  • filepath (str or bytes object) – Relative path to file.

exception twindb_backup.destination.azblob.AzureBlobClientError[source]

Bases: twindb_backup.destination.exceptions.AzureBlobDestinationError

blob_name: str = ''
container_name: str = ''
exception twindb_backup.destination.azblob.AzureBlobInitError[source]

Bases: twindb_backup.destination.exceptions.AzureBlobDestinationError

exception twindb_backup.destination.azblob.AzureBlobPathParseError[source]

Bases: twindb_backup.destination.exceptions.AzureBlobDestinationError

exception twindb_backup.destination.azblob.AzureBlobReadError[source]

Bases: twindb_backup.destination.exceptions.AzureBlobDestinationError

blob_name: str = ''
blob_path: str = ''

The path string which lead to this exception

blob_properties: azure.storage.blob._models.BlobProperties = None
chunk_byte_range: Tuple[int, int] = (-1, -1)

The [start,end) bytes defining the chunk where this exception occurs (if chunking used) else set to (-1,-1)

container_name: str = ''
exception twindb_backup.destination.azblob.AzureBlobWriteError[source]

Bases: twindb_backup.destination.exceptions.AzureBlobDestinationError

blob_name: str = ''
blob_path: str = ''

The path string which lead to this exception

blob_properties: azure.storage.blob._models.BlobProperties = None
container_name: str = ''
content_type = None
exception twindb_backup.destination.azblob.AzureClientIterationError[source]

Bases: twindb_backup.destination.exceptions.AzureBlobDestinationError

exception twindb_backup.destination.azblob.AzureClientManagerError[source]

Bases: twindb_backup.destination.exceptions.AzureBlobDestinationError

aggregated_traceback: List[str]
err_map: Dict[str, List[Dict[str, str]]]
class twindb_backup.destination.azblob.ClientWrapper(name: Optional[str] = None, props: Optional[azure.storage.blob._models.ContainerProperties] = None)[source]

Bases: object

The ContainerWrapper class exists to simplify the process of ensuring that a container’s name is accessible from mixed types of inputs.

property name: str
twindb_backup.destination.azblob.GC_TOGGLE_DEPTH = 0

GC_TOGGLE_DEPTH is used as a reference counter for managing when the _gc_toggle function should call gc.enable().

twindb_backup.destination.azblob.MAX_SYS_MEM_USE = 536870912

MAX_PIPE_CHUNK_BYTES is a conservatively safe upper bound on the number of bytes we send through multiprocessing.connections.Connection objects.

This boundary will be derived for the current machine’s OS at runtime.

Per the official Python 3.9.6 documentation:

send(obj)
    Send an object to the other end of the connection which should be read using recv().

    The object must be picklable. Very large pickles (approximately 32 MiB+, though it depends on the OS)
    may raise a ValueError exception.

For source documentation on send(obj) see:

twindb_backup.destination.azblob.client_generator(conn_str, container: Optional[Union[str, twindb_backup.destination.azblob.ClientWrapper, azure.storage.blob._models.ContainerProperties, Iterable[Union[str, twindb_backup.destination.azblob.ClientWrapper, azure.storage.blob._models.ContainerProperties]]]] = None, prefix: Optional[str] = None, blob: Optional[Union[str, twindb_backup.destination.azblob.ClientWrapper, azure.storage.blob._models.ContainerProperties, Iterable[Union[str, twindb_backup.destination.azblob.ClientWrapper, azure.storage.blob._models.ContainerProperties]]]] = None, recurse: bool = False) Generator[Union[str, azure.storage.blob._blob_service_client.BlobServiceClient, azure.storage.blob._container_client.ContainerClient, azure.storage.blob._blob_client.BlobClient], None, None][source]
twindb_backup.destination.azblob.flatten_client_iters(clients: List[Union[azure.storage.blob._container_client.ContainerClient, List[azure.storage.blob._blob_client.BlobClient]]])[source]

twindb_backup.destination.base_destination module

Module defines Base destination class and destination exception(s).

class twindb_backup.destination.base_destination.BaseDestination(remote_path)[source]

Bases: object

Base destination class.

Parameters

remote_path (str) – Common path where all backups are stored. The remote path is specific to the actual destination class. For example, for Amazon S3 it will be something like s3://bucket_with_backups

abstract delete(path)[source]

Delete object from the destination

Parameters

path – Relative path to the file to delete

abstract get_stream(copy)[source]

Get a PIPE handler with content of the backup copy streamed from the destination.

Parameters

copy (BaseCopy) – Backup copy

Returns

Standard output.

list_files(prefix=None, recursive=False, pattern=None, files_only=False)[source]

Get list of file by prefix.

Parameters
  • prefix (str) – Path

  • recursive (bool) – Recursive return list of files

  • pattern (str) – files must match with this regexp if specified

  • files_only (bool) – If True don’t list directories

Returns

List of files

Return type

list

abstract read(filepath)[source]

Read content of a file path from destination.

Parameters

filepath (str) – Relative path to file.

Returns

Content of the file.

Return type

str

abstract save(handler, filepath)[source]

Save a stream given as handler to filepath.

Parameters
  • handler (file) – Incoming stream.

  • filepath (str) – Save stream as this name.

abstract write(content, filepath)[source]

Write content to a file.

Parameters
  • content (str) – Content to write to the file.

  • filepath (str) – Relative path to file.

twindb_backup.destination.exceptions module

Module for destination exceptions.

exception twindb_backup.destination.exceptions.AzureBlobDestinationError[source]

Bases: twindb_backup.destination.exceptions.DestinationError

Azure-blob destination errors

exception twindb_backup.destination.exceptions.DestinationError[source]

Bases: twindb_backup.exceptions.TwinDBBackupError

General destination error

exception twindb_backup.destination.exceptions.FileNotFound[source]

Bases: twindb_backup.destination.exceptions.DestinationError

File doesn’t exist on destination

exception twindb_backup.destination.exceptions.GCSDestinationError[source]

Bases: twindb_backup.destination.exceptions.DestinationError

GCS destination errors

exception twindb_backup.destination.exceptions.S3DestinationError[source]

Bases: twindb_backup.destination.exceptions.DestinationError

S3 destination errors

exception twindb_backup.destination.exceptions.SshDestinationError[source]

Bases: twindb_backup.destination.exceptions.DestinationError

SSH destination errors

twindb_backup.destination.gcs module

Module for GCS destination.

class twindb_backup.destination.gcs.GCS(**kwargs)[source]

Bases: twindb_backup.destination.base_destination.BaseDestination

GCS destination class.

Parameters

kwargs – Keyword arguments.

  • bucket - (required) GCS bucket name.

  • gc_credentials_file - (required) GC credentials json filepath.

  • chunk_size - when storing a stream use this a a chunk size.

    The stream will be stored a set of chunks of this size on the GS.

property bucket

GCS bucket name.

create_bucket()[source]

Creates the bucket in gcs that will store the backups.

Raises
delete(path)[source]

Delete object from the destination

Parameters

path – Relative path to the file to delete

delete_bucket(force=False)[source]

Delete the bucket in gcs that was storing the backups.

Parameters

force (bool) – If the bucket is non-empty then delete the objects before deleting the bucket.

Raises

GCSDestinationError – if failed to delete the bucket.

get_stream(copy)[source]

Get a PIPE handler with content of the backup copy streamed from the destination.

Parameters

copy (BaseCopy) – Backup copy

Returns

Standard output.

read(filepath)[source]

Read content from a file.

Parameters

filepath (str) – relative path to a file with status.

Returns

Content of the file.

Return type

str

Raises

FileNotFound – if filepath doesn’t exist on the destination.

save(handler, filepath)[source]

Read from handler and save it to GCS

Parameters
  • handler (file) – stdout handler from backup source

  • filepath (str) – save backup copy in a file with this name

write(content, filepath)[source]

Write a string passed in content to a filepath on the destination.

Parameters
  • content (str) – String to write.

  • filepath (str) – Relative file path on the destination.

twindb_backup.destination.local module

Module defines Local destination.

class twindb_backup.destination.local.Local(path=None)[source]

Bases: twindb_backup.destination.base_destination.BaseDestination

Local destination class.

delete(path)[source]

Delete object from the destination

Parameters

path – Relative path to the file to delete

get_stream(copy)[source]

Get a PIPE handler with content of the backup copy streamed from the destination

Parameters

copy (BaseCopy) – Backup copy

Returns

property path

Root path on local file system where local backup copies are stored.

read(filepath)[source]

Read content of a file path from destination.

Parameters

filepath (str) – Relative path to file.

Returns

Content of the file.

Return type

str

save(handler, filepath)[source]

Read from handler and save it on local storage

Parameters
  • filepath – store backup copy as this name

  • handler – Input stream

write(content, filepath)[source]

Write content to a file.

Parameters
  • content (str) – Content to write to the file.

  • filepath (str) – Relative path to file.

twindb_backup.destination.s3 module

Module for S3 destination.

class twindb_backup.destination.s3.S3(**kwargs)[source]

Bases: twindb_backup.destination.base_destination.BaseDestination

S3 destination class.

Parameters

kwargs – Keyword arguments.

  • bucket - S3 bucket name.

  • aws_access_key_id - AWS key id.

  • aws_secret_access_key - AWS secret key.

  • aws_default_region - AWS default region.

  • hostname - Hostname of a host where a backup is taken from.

property bucket

S3 bucket name.

create_bucket()[source]

Creates the bucket in s3 that will store the backups.

Raises

S3DestinationError – if failed to create the bucket.

delete(path)[source]

Deletes an S3 object.

Parameters

path (str) – Key of S3 object.

Raises

S3DestinationError – if failed to delete object.

delete_all_objects()[source]

Delete all objects from S3 bucket.

Raises

S3DestinationError – if failed to delete objects from the bucket.

delete_bucket(force=False)[source]

Delete the bucket in s3 that was storing the backups.

Parameters

force (bool) – If the bucket is non-empty then delete the objects before deleting the bucket.

Raises

S3DestinationError – if failed to delete the bucket.

get_stream(copy)[source]

Get a PIPE handler with content of the backup copy streamed from the destination.

Parameters

copy (BaseCopy) – Backup copy

Returns

Stream with backup copy

Return type

generator

Raises

S3DestinationError – if failed to stream a backup copy.

static get_transfer_config()[source]

Build Transfer config

Returns

Transfer config

Return type

boto3.s3.transfer.TransferConfig

list_files(prefix=None, recursive=False, pattern=None, files_only=False)[source]

List files in the destination that have common prefix.

Parameters
  • prefix (str) – Common prefix. May include the bucket name. (e.g. s3://my_bucket/foo/) or simply a prefix in the bucket (e.g. foo/).

  • recursive – Does nothing for this class.

  • pattern (str) – files must match with this regexp if specified.

  • files_only – Does nothing for this class.

Returns

sorted list of file names.

Returns

Full S3 url in form s3://bucket/path/to/file.

Return type

list(str)

Raises

S3DestinationError – if failed to list files.

read(filepath)[source]

Read content of filepath and return it as a string.

Parameters

filepath – Path in S3 bucket.

Returns

Content of the file.

Return type

str

Raises

FileNotFound – If filepath doesn’t exist.

save(handler, filepath)[source]

Read from handler and save it to Amazon S3

Parameters
  • filepath – save backup copy in a file with this name

  • handler – stdout handler from backup source

static setup_s3_client()[source]

Creates an authenticated s3 client.

Returns

S3 client instance.

Return type

botocore.client.BaseClient

share(s3_url)[source]

Share S3 file and return public link

Parameters

s3_url (str) – S3 url

Returns

Public url

Return type

str

Raises

S3DestinationError – if failed to share object.

static validate_client_response(response)[source]
Validates the response returned by the client. Raises an exception

if the response code is not 200 or 204

Parameters

response (dict) – The response that needs to be validated.

Raises

S3DestinationError – if response from S3 is invalid.

write(content, filepath)[source]

Write content to a file.

Parameters
  • content (str) – Content to write to the file.

  • filepath (str) – Relative path to file.

class twindb_backup.destination.s3.S3FileAccess[source]

Bases: object

Access modes for S3 files

private = 'private'
public_read = 'public-read'

twindb_backup.destination.ssh module

Module for SSH destination.

class twindb_backup.destination.ssh.Ssh(remote_path, **kwargs)[source]

Bases: twindb_backup.destination.base_destination.BaseDestination

The SSH destination class represents a destination backup storage with running SSH demon.

Parameters
  • remote_path (str) – Path to store backups.

  • kwargs (dict) – Keyword arguments. See below.

  • hostname (str): Hostname of the host where backup is taken from.

  • ssh_host (str): Hostname for SSH connection. Default 127.0.0.1.

  • ssh_user (str): Username for SSH connection. Default root.

  • ssh_port (int): TCP port for SSH connection. Default 22.

  • ssh_key (str): File with an rsa/dsa key for SSH authentication.

    Default /root/.ssh/id_rsa.

property client
Returns

SSH client.

Return type

SshClient

delete(path)[source]

Delete file by path. The path is a relative to the self.remote_path.

Parameters

path (str) – Path to a remote file.

ensure_tcp_port_listening(port, wait_timeout=10, wait=True)[source]

Check that tcp port is open and ready to accept connections. Keep checking up to wait_timeout seconds.

Parameters
  • port (int) – TCP port that is supposed to be listening.

  • wait_timeout (int) – Wait this many seconds until the port is ready.

  • wait (bool) – Wait a wait_timeout of seconds until the TCP port become available. If set to False the method will return after the first check.

Returns

True if the TCP port is listening.

Return type

bool

execute_command(cmd, quiet=False, background=False)[source]

Execute ssh command on the remote destination.

Parameters
  • cmd (str) – Command to execute.

  • quiet (bool) – If True don’t print errors.

  • background (bool) – If True don’t wait until the command exits.

Returns

stdin, stdout and stderr handlers.

Return type

tuple

get_stream(copy)[source]

Get a PIPE handler with content of the backup copy streamed from the destination.

Parameters

copy (BaseCopy) – Backup copy.

Returns

Standard output.

Return type

file

property host
Returns

IP address of the destination.

Return type

str

netcat(command, port=9990)[source]

Run netcat on the destination pipe it to a given command:

ncat -l <port> --recv-only | <command>
Parameters
  • command (str) – Command that would accept netcat’s output.

  • port (int) – TCP port to run netcat on. Default 9999.

property port
Returns

TCP port of the destination.

Return type

int

read(filepath)[source]

Read content of a file path from destination.

Parameters

filepath (str) – Relative path to file.

Returns

Content of the file.

Return type

str

save(handler, filepath)[source]

Read from the handler and save it on the remote ssh server in a file filepath.

Parameters
  • filepath (str) – Relative path to a file to store the backup copy.

  • handler (file) – Stream with content of the backup.

property user
Returns

SSH user.

Return type

str

write(content, filepath)[source]

Write content to a file.

Parameters
  • content (str) – Content to write to the file.

  • filepath (str) – Relative path to file.

Module contents