twindb_backup.destination package¶
Submodules¶
twindb_backup.destination.azblob module¶
Module for Azure-blob destination.
- class twindb_backup.destination.azblob.AzureBlob(remote_path: AnyStr, connection_string: AnyStr, can_do_overwrites: bool = False, cpu_cap: int = 2, max_mem_bytes: int = 536870912, default_protocol: Optional[AnyStr] = None, default_host_name: Optional[AnyStr] = None, default_container_name: Optional[AnyStr] = None, default_interval: Optional[AnyStr] = None, default_media_type: Optional[AnyStr] = None, default_fname_prefix: Optional[AnyStr] = None)[source]¶
Bases:
twindb_backup.destination.base_destination.BaseDestination
- property can_overwrite¶
- property connection_manager¶
- property connection_string¶
An Azure specific authentication string for accessing the target backup destination host
- property default_container_name¶
The default container (aka bucket) name that we default to if a relative path string omits the reference
- property default_fname_prefix¶
- property default_host_name¶
The default host server name directory that we default to if a relative path string omits the reference
- property default_interval¶
The default backup interval directory that we default to if a relative path string omits the reference
- property default_media_type¶
- property default_protocol¶
- delete(path: AnyStr)[source]¶
Delete object from the destination
- the general form for the path object should conform to the following:
[azure:/]/[bucket or container name]/[server name]/[update interval]/[query language]/<file name>
- NOTE: The protocol type (the left-most component of the example above) is technically optional,
as it should always be an azure storage type; but if passed we will check to confirm that it is indeed for azure-blob storage, so including it ensures proper sanity checking.
- – If path defines a new absolute path string then it must contain all parts defined above,
with the option to omit those components wrapped in square brackets, E.G.: [some component]
- where:
[components inside square brackets] => optional <objects inside chevrons> => required
- such that:
optional components that are not provided should be substituted with an ellipsis (the triple period => …)
- E.G.:
…://foo/…/hourly/mysql/bar-that.foos.gz
- Note:
Where optional path components are omitted, we assume that the context of the called AzureBlob instance should be used to fill in the gaps.
- – If path is given as a relative path string then you may also use the ellipsis as defined for absolute paths,
with the added option to use .. for relative directory hierarchy referencing. The one caveat is that
- E.G.:
../../daily/mysql/relative-foo.bar.gz
- or
../../../some_different_host/…/mysql where:
The … component signals that we wish to use the given default interval this object was initialized with.
- Parameters
path – A string or bytes object; The path to the file (blob) to delete. Can be relative or absolute.
- get_stream(copy: Union[str, twindb_backup.copy.mysql_copy.MySQLCopy])[source]¶
Get a PIPE handler with content of the backup copy streamed from the destination.
- Parameters
copy (BaseCopy) – Backup copy
- Returns
Standard output.
- property max_bytes_per_pipe_message¶
- property max_system_memory_usage¶
- read(filepath: str, bytes_per_chunk: Optional[int] = None) bytes [source]¶
Read content from destination at the end of given filepath.
- Parameters
filepath (str) – REQUIRED; a str object; Relative path to destination file that we will read from.
bytes_per_chunk (int) – OPTIONAL; DEFAULT = self.max_bytes_per_pipe_message; an int value; This parameter dictates the max chunk size (in bytes) that should be passed into the pipe for any single chunk.
- Returns
Content of the file.
- Return type
bytes
- exception twindb_backup.destination.azblob.AzureBlobClientError[source]¶
Bases:
twindb_backup.destination.exceptions.AzureBlobDestinationError
- blob_name: str = ''¶
- container_name: str = ''¶
- exception twindb_backup.destination.azblob.AzureBlobInitError[source]¶
Bases:
twindb_backup.destination.exceptions.AzureBlobDestinationError
- exception twindb_backup.destination.azblob.AzureBlobPathParseError[source]¶
Bases:
twindb_backup.destination.exceptions.AzureBlobDestinationError
- exception twindb_backup.destination.azblob.AzureBlobReadError[source]¶
Bases:
twindb_backup.destination.exceptions.AzureBlobDestinationError
- blob_name: str = ''¶
- blob_path: str = ''¶
The path string which lead to this exception
- blob_properties: azure.storage.blob._models.BlobProperties = None¶
- chunk_byte_range: Tuple[int, int] = (-1, -1)¶
The [start,end) bytes defining the chunk where this exception occurs (if chunking used) else set to (-1,-1)
- container_name: str = ''¶
- exception twindb_backup.destination.azblob.AzureBlobWriteError[source]¶
Bases:
twindb_backup.destination.exceptions.AzureBlobDestinationError
- blob_name: str = ''¶
- blob_path: str = ''¶
The path string which lead to this exception
- blob_properties: azure.storage.blob._models.BlobProperties = None¶
- container_name: str = ''¶
- content_type = None¶
- exception twindb_backup.destination.azblob.AzureClientIterationError[source]¶
Bases:
twindb_backup.destination.exceptions.AzureBlobDestinationError
- exception twindb_backup.destination.azblob.AzureClientManagerError[source]¶
Bases:
twindb_backup.destination.exceptions.AzureBlobDestinationError
- aggregated_traceback: List[str]¶
- err_map: Dict[str, List[Dict[str, str]]]¶
- class twindb_backup.destination.azblob.ClientWrapper(name: Optional[str] = None, props: Optional[azure.storage.blob._models.ContainerProperties] = None)[source]¶
Bases:
object
The ContainerWrapper class exists to simplify the process of ensuring that a container’s name is accessible from mixed types of inputs.
- property name: str¶
- twindb_backup.destination.azblob.GC_TOGGLE_DEPTH = 0¶
GC_TOGGLE_DEPTH is used as a reference counter for managing when the _gc_toggle function should call gc.enable().
- twindb_backup.destination.azblob.MAX_SYS_MEM_USE = 536870912¶
MAX_PIPE_CHUNK_BYTES is a conservatively safe upper bound on the number of bytes we send through multiprocessing.connections.Connection objects.
This boundary will be derived for the current machine’s OS at runtime.
Per the official Python 3.9.6 documentation:
send(obj) Send an object to the other end of the connection which should be read using recv(). The object must be picklable. Very large pickles (approximately 32 MiB+, though it depends on the OS) may raise a ValueError exception.
For source documentation on send(obj) see:
- twindb_backup.destination.azblob.client_generator(conn_str, container: Optional[Union[str, twindb_backup.destination.azblob.ClientWrapper, azure.storage.blob._models.ContainerProperties, Iterable[Union[str, twindb_backup.destination.azblob.ClientWrapper, azure.storage.blob._models.ContainerProperties]]]] = None, prefix: Optional[str] = None, blob: Optional[Union[str, twindb_backup.destination.azblob.ClientWrapper, azure.storage.blob._models.ContainerProperties, Iterable[Union[str, twindb_backup.destination.azblob.ClientWrapper, azure.storage.blob._models.ContainerProperties]]]] = None, recurse: bool = False) Generator[Union[str, azure.storage.blob._blob_service_client.BlobServiceClient, azure.storage.blob._container_client.ContainerClient, azure.storage.blob._blob_client.BlobClient], None, None] [source]¶
twindb_backup.destination.base_destination module¶
Module defines Base destination class and destination exception(s).
- class twindb_backup.destination.base_destination.BaseDestination(remote_path)[source]¶
Bases:
object
Base destination class.
- Parameters
remote_path (str) – Common path where all backups are stored. The remote path is specific to the actual destination class. For example, for Amazon S3 it will be something like
s3://bucket_with_backups
- abstract delete(path)[source]¶
Delete object from the destination
- Parameters
path – Relative path to the file to delete
- abstract get_stream(copy)[source]¶
Get a PIPE handler with content of the backup copy streamed from the destination.
- Parameters
copy (BaseCopy) – Backup copy
- Returns
Standard output.
- list_files(prefix=None, recursive=False, pattern=None, files_only=False)[source]¶
Get list of file by prefix.
- Parameters
prefix (str) – Path
recursive (bool) – Recursive return list of files
pattern (str) – files must match with this regexp if specified
files_only (bool) – If True don’t list directories
- Returns
List of files
- Return type
list
- abstract read(filepath)[source]¶
Read content of a file path from destination.
- Parameters
filepath (str) – Relative path to file.
- Returns
Content of the file.
- Return type
str
twindb_backup.destination.exceptions module¶
Module for destination exceptions.
- exception twindb_backup.destination.exceptions.AzureBlobDestinationError[source]¶
Bases:
twindb_backup.destination.exceptions.DestinationError
Azure-blob destination errors
- exception twindb_backup.destination.exceptions.DestinationError[source]¶
Bases:
twindb_backup.exceptions.TwinDBBackupError
General destination error
- exception twindb_backup.destination.exceptions.FileNotFound[source]¶
Bases:
twindb_backup.destination.exceptions.DestinationError
File doesn’t exist on destination
- exception twindb_backup.destination.exceptions.GCSDestinationError[source]¶
Bases:
twindb_backup.destination.exceptions.DestinationError
GCS destination errors
- exception twindb_backup.destination.exceptions.S3DestinationError[source]¶
Bases:
twindb_backup.destination.exceptions.DestinationError
S3 destination errors
- exception twindb_backup.destination.exceptions.SshDestinationError[source]¶
Bases:
twindb_backup.destination.exceptions.DestinationError
SSH destination errors
twindb_backup.destination.gcs module¶
Module for GCS destination.
- class twindb_backup.destination.gcs.GCS(**kwargs)[source]¶
Bases:
twindb_backup.destination.base_destination.BaseDestination
GCS destination class.
- Parameters
kwargs – Keyword arguments.
bucket - (required) GCS bucket name.
gc_credentials_file - (required) GC credentials json filepath.
- chunk_size - when storing a stream use this a a chunk size.
The stream will be stored a set of chunks of this size on the GS.
- property bucket¶
GCS bucket name.
- create_bucket()[source]¶
Creates the bucket in gcs that will store the backups.
- Raises
GCSDestinationError – if failed to create the bucket.
GCSDestinationError – If authentication error.
- delete(path)[source]¶
Delete object from the destination
- Parameters
path – Relative path to the file to delete
- delete_bucket(force=False)[source]¶
Delete the bucket in gcs that was storing the backups.
- Parameters
force (bool) – If the bucket is non-empty then delete the objects before deleting the bucket.
- Raises
GCSDestinationError – if failed to delete the bucket.
- get_stream(copy)[source]¶
Get a PIPE handler with content of the backup copy streamed from the destination.
- Parameters
copy (BaseCopy) – Backup copy
- Returns
Standard output.
- read(filepath)[source]¶
Read content from a file.
- Parameters
filepath (str) – relative path to a file with status.
- Returns
Content of the file.
- Return type
str
- Raises
FileNotFound – if filepath doesn’t exist on the destination.
twindb_backup.destination.local module¶
Module defines Local destination.
- class twindb_backup.destination.local.Local(path=None)[source]¶
Bases:
twindb_backup.destination.base_destination.BaseDestination
Local destination class.
- delete(path)[source]¶
Delete object from the destination
- Parameters
path – Relative path to the file to delete
- get_stream(copy)[source]¶
Get a PIPE handler with content of the backup copy streamed from the destination
- Parameters
copy (BaseCopy) – Backup copy
- Returns
- property path¶
Root path on local file system where local backup copies are stored.
- read(filepath)[source]¶
Read content of a file path from destination.
- Parameters
filepath (str) – Relative path to file.
- Returns
Content of the file.
- Return type
str
twindb_backup.destination.s3 module¶
Module for S3 destination.
- class twindb_backup.destination.s3.S3(**kwargs)[source]¶
Bases:
twindb_backup.destination.base_destination.BaseDestination
S3 destination class.
- Parameters
kwargs – Keyword arguments.
bucket - S3 bucket name.
aws_access_key_id - AWS key id.
aws_secret_access_key - AWS secret key.
aws_default_region - AWS default region.
hostname - Hostname of a host where a backup is taken from.
- property bucket¶
S3 bucket name.
- create_bucket()[source]¶
Creates the bucket in s3 that will store the backups.
- Raises
S3DestinationError – if failed to create the bucket.
- delete(path)[source]¶
Deletes an S3 object.
- Parameters
path (str) – Key of S3 object.
- Raises
S3DestinationError – if failed to delete object.
- delete_all_objects()[source]¶
Delete all objects from S3 bucket.
- Raises
S3DestinationError – if failed to delete objects from the bucket.
- delete_bucket(force=False)[source]¶
Delete the bucket in s3 that was storing the backups.
- Parameters
force (bool) – If the bucket is non-empty then delete the objects before deleting the bucket.
- Raises
S3DestinationError – if failed to delete the bucket.
- get_stream(copy)[source]¶
Get a PIPE handler with content of the backup copy streamed from the destination.
- Parameters
copy (BaseCopy) – Backup copy
- Returns
Stream with backup copy
- Return type
generator
- Raises
S3DestinationError – if failed to stream a backup copy.
- static get_transfer_config()[source]¶
Build Transfer config
- Returns
Transfer config
- Return type
boto3.s3.transfer.TransferConfig
- list_files(prefix=None, recursive=False, pattern=None, files_only=False)[source]¶
List files in the destination that have common prefix.
- Parameters
prefix (str) – Common prefix. May include the bucket name. (e.g.
s3://my_bucket/foo/
) or simply a prefix in the bucket (e.g.foo/
).recursive – Does nothing for this class.
pattern (str) – files must match with this regexp if specified.
files_only – Does nothing for this class.
- Returns
sorted list of file names.
- Returns
Full S3 url in form
s3://bucket/path/to/file
.- Return type
list(str)
- Raises
S3DestinationError – if failed to list files.
- read(filepath)[source]¶
Read content of filepath and return it as a string.
- Parameters
filepath – Path in S3 bucket.
- Returns
Content of the file.
- Return type
str
- Raises
FileNotFound – If filepath doesn’t exist.
- save(handler, filepath)[source]¶
Read from handler and save it to Amazon S3
- Parameters
filepath – save backup copy in a file with this name
handler – stdout handler from backup source
- static setup_s3_client()[source]¶
Creates an authenticated s3 client.
- Returns
S3 client instance.
- Return type
botocore.client.BaseClient
Share S3 file and return public link
- Parameters
s3_url (str) – S3 url
- Returns
Public url
- Return type
str
- Raises
S3DestinationError – if failed to share object.
- static validate_client_response(response)[source]¶
- Validates the response returned by the client. Raises an exception
if the response code is not 200 or 204
- Parameters
response (dict) – The response that needs to be validated.
- Raises
S3DestinationError – if response from S3 is invalid.
twindb_backup.destination.ssh module¶
Module for SSH destination.
- class twindb_backup.destination.ssh.Ssh(remote_path, **kwargs)[source]¶
Bases:
twindb_backup.destination.base_destination.BaseDestination
The SSH destination class represents a destination backup storage with running SSH demon.
- Parameters
remote_path (str) – Path to store backups.
kwargs (dict) – Keyword arguments. See below.
hostname (str): Hostname of the host where backup is taken from.
ssh_host (str): Hostname for SSH connection. Default
127.0.0.1
.ssh_user (str): Username for SSH connection. Default
root
.ssh_port (int): TCP port for SSH connection. Default 22.
- ssh_key (str): File with an rsa/dsa key for SSH authentication.
Default
/root/.ssh/id_rsa
.
- delete(path)[source]¶
Delete file by path. The path is a relative to the
self.remote_path
.- Parameters
path (str) – Path to a remote file.
- ensure_tcp_port_listening(port, wait_timeout=10, wait=True)[source]¶
Check that tcp port is open and ready to accept connections. Keep checking up to
wait_timeout
seconds.- Parameters
port (int) – TCP port that is supposed to be listening.
wait_timeout (int) – Wait this many seconds until the port is ready.
wait (bool) – Wait a wait_timeout of seconds until the TCP port become available. If set to False the method will return after the first check.
- Returns
True
if the TCP port is listening.- Return type
bool
- execute_command(cmd, quiet=False, background=False)[source]¶
Execute ssh command on the remote destination.
- Parameters
cmd (str) – Command to execute.
quiet (bool) – If
True
don’t print errors.background (bool) – If
True
don’t wait until the command exits.
- Returns
stdin, stdout and stderr handlers.
- Return type
tuple
- get_stream(copy)[source]¶
Get a PIPE handler with content of the backup copy streamed from the destination.
- Parameters
copy (BaseCopy) – Backup copy.
- Returns
Standard output.
- Return type
file
- property host¶
- Returns
IP address of the destination.
- Return type
str
- netcat(command, port=9990)[source]¶
Run
netcat
on the destination pipe it to a given command:ncat -l <port> --recv-only | <command>
- Parameters
command (str) – Command that would accept
netcat
’s output.port (int) – TCP port to run
netcat
on. Default 9999.
- property port¶
- Returns
TCP port of the destination.
- Return type
int
- read(filepath)[source]¶
Read content of a file path from destination.
- Parameters
filepath (str) – Relative path to file.
- Returns
Content of the file.
- Return type
str
- save(handler, filepath)[source]¶
Read from the handler and save it on the remote ssh server in a file
filepath
.- Parameters
filepath (str) – Relative path to a file to store the backup copy.
handler (file) – Stream with content of the backup.
- property user¶
- Returns
SSH user.
- Return type
str