twindb_backup.destination package

Submodules

twindb_backup.destination.base_destination module

Module defines Base destination class and destination exception(s).

class twindb_backup.destination.base_destination.BaseDestination(remote_path)[source]

Bases: object

Base destination class.

Parameters:remote_path (str) – Common path where all backups are stored. The remote path is specific to the actual destination class. For example, for Amazon S3 it will be something like s3://bucket_with_backups
delete(path)[source]

Delete object from the destination

Parameters:path – Relative path to the file to delete
get_stream(copy)[source]

Get a PIPE handler with content of the backup copy streamed from the destination.

Parameters:copy (BaseCopy) – Backup copy
Returns:Standard output.
list_files(prefix=None, recursive=False, pattern=None, files_only=False)[source]

Get list of file by prefix.

Parameters:
  • prefix (str) – Path
  • recursive (bool) – Recursive return list of files
  • pattern (str) – files must match with this regexp if specified
  • files_only (bool) – If True don’t list directories
Returns:

List of files

Return type:

list

read(filepath)[source]

Read content of a file path from destination.

Parameters:filepath (str) – Relative path to file.
Returns:Content of the file.
Return type:str
save(handler, filepath)[source]

Save a stream given as handler to filepath.

Parameters:
  • handler (file) – Incoming stream.
  • filepath (str) – Save stream as this name.
write(content, filepath)[source]

Write content to a file.

Parameters:
  • content (str) – Content to write to the file.
  • filepath (str) – Relative path to file.

twindb_backup.destination.exceptions module

Module for destination exceptions

exception twindb_backup.destination.exceptions.DestinationError[source]

Bases: twindb_backup.exceptions.TwinDBBackupError

General destination error

exception twindb_backup.destination.exceptions.FileNotFound[source]

Bases: twindb_backup.destination.exceptions.DestinationError

File doesn’t exist on destination

exception twindb_backup.destination.exceptions.GCSDestinationError[source]

Bases: twindb_backup.destination.exceptions.DestinationError

GCS destination errors

exception twindb_backup.destination.exceptions.S3DestinationError[source]

Bases: twindb_backup.destination.exceptions.DestinationError

S3 destination errors

exception twindb_backup.destination.exceptions.SshDestinationError[source]

Bases: twindb_backup.destination.exceptions.DestinationError

SSH destination errors

twindb_backup.destination.local module

Module defines Local destination.

class twindb_backup.destination.local.Local(path=None)[source]

Bases: twindb_backup.destination.base_destination.BaseDestination

Local destination class.

delete(path)[source]

Delete object from the destination

Parameters:path – Relative path to the file to delete
get_stream(copy)[source]

Get a PIPE handler with content of the backup copy streamed from the destination

Parameters:copy (BaseCopy) – Backup copy
Returns:
path

Root path on local file system where local backup copies are stored.

read(filepath)[source]

Read content of a file path from destination.

Parameters:filepath (str) – Relative path to file.
Returns:Content of the file.
Return type:str
save(handler, filepath)[source]

Read from handler and save it on local storage

Parameters:
  • filepath – store backup copy as this name
  • handler – Input stream
write(content, filepath)[source]

Write content to a file.

Parameters:
  • content (str) – Content to write to the file.
  • filepath (str) – Relative path to file.

twindb_backup.destination.s3 module

Module for S3 destination.

class twindb_backup.destination.s3.S3(**kwargs)[source]

Bases: twindb_backup.destination.base_destination.BaseDestination

S3 destination class.

Parameters:kwargs – Keyword arguments.
  • bucket - S3 bucket name.
  • aws_access_key_id - AWS key id.
  • aws_secret_access_key - AWS secret key.
  • aws_default_region - AWS default region.
  • hostname - Hostname of a host where a backup is taken from.
bucket

S3 bucket name.

create_bucket()[source]

Creates the bucket in s3 that will store the backups.

Raises:S3DestinationError – if failed to create the bucket.
delete(path)[source]

Deletes an S3 object.

Parameters:path (str) – Key of S3 object.
Raises:S3DestinationError – if failed to delete object.
delete_all_objects()[source]

Delete all objects from S3 bucket.

Raises:S3DestinationError – if failed to delete objects from the bucket.
delete_bucket(force=False)[source]

Delete the bucket in s3 that was storing the backups.

Parameters:force (bool) – If the bucket is non-empty then delete the objects before deleting the bucket.
Raises:S3DestinationError – if failed to delete the bucket.
get_stream(**kwds)[source]

Get a PIPE handler with content of the backup copy streamed from the destination.

Parameters:copy (BaseCopy) – Backup copy
Returns:Stream with backup copy
Return type:generator
Raises:S3DestinationError – if failed to stream a backup copy.
static get_transfer_config()[source]

Build Transfer config

Returns:Transfer config
Return type:boto3.s3.transfer.TransferConfig
list_files(prefix=None, recursive=False, pattern=None, files_only=False)[source]

List files in the destination that have common prefix.

Parameters:
  • prefix (str) – Common prefix. May include the bucket name. (e.g. s3://my_bucket/foo/) or simply a prefix in the bucket (e.g. foo/).
  • recursive – Does nothing for this class.
  • pattern (str) – files must match with this regexp if specified.
  • files_only – Does nothing for this class.
Returns:

sorted list of file names.

Returns:

Full S3 url in form s3://bucket/path/to/file.

Return type:

list(str)

Raises:

S3DestinationError – if failed to list files.

read(filepath)[source]

Read content of filepath and return it as a string.

Parameters:filepath – Path in S3 bucket.
Returns:Content of the file.
Return type:str
Raises:FileNotFound – If filepath doesn’t exist.
save(handler, filepath)[source]

Read from handler and save it to Amazon S3

Parameters:
  • filepath – save backup copy in a file with this name
  • handler – stdout handler from backup source
static setup_s3_client()[source]

Creates an authenticated s3 client.

Returns:S3 client instance.
Return type:botocore.client.BaseClient
share(s3_url)[source]

Share S3 file and return public link

Parameters:s3_url (str) – S3 url
Returns:Public url
Return type:str
Raises:S3DestinationError – if failed to share object.
static validate_client_response(response)[source]
Validates the response returned by the client. Raises an exception
if the response code is not 200 or 204
Parameters:response (dict) – The response that needs to be validated.
Raises:S3DestinationError – if response from S3 is invalid.
write(content, filepath)[source]

Write content to a file.

Parameters:
  • content (str) – Content to write to the file.
  • filepath (str) – Relative path to file.
class twindb_backup.destination.s3.S3FileAccess[source]

Bases: object

Access modes for S3 files

private = 'private'
public_read = 'public-read'

twindb_backup.destination.ssh module

Module for SSH destination.

class twindb_backup.destination.ssh.Ssh(remote_path, **kwargs)[source]

Bases: twindb_backup.destination.base_destination.BaseDestination

The SSH destination class represents a destination backup storage with running SSH demon.

Parameters:
  • remote_path (str) – Path to store backups.
  • kwargs (dict) – Keyword arguments. See below.
  • hostname (str): Hostname of the host where backup is taken from.
  • ssh_host (str): Hostname for SSH connection. Default 127.0.0.1.
  • ssh_user (str): Username for SSH connection. Default root.
  • ssh_port (int): TCP port for SSH connection. Default 22.
  • ssh_key (str): File with an rsa/dsa key for SSH authentication.
    Default /root/.ssh/id_rsa.
client
Returns:SSH client.
Return type:SshClient
delete(path)[source]

Delete file by path. The path is a relative to the self.remote_path.

Parameters:path (str) – Path to a remote file.
ensure_tcp_port_listening(port, wait_timeout=10)[source]

Check that tcp port is open and ready to accept connections. Keep checking up to wait_timeout seconds.

Parameters:
  • port (int) – TCP port that is supposed to be listening.
  • wait_timeout (int) – Wait this many seconds until the port is ready.
Returns:

True if the TCP port is listening.

Return type:

bool

execute_command(cmd, quiet=False, background=False)[source]

Execute ssh command on the remote destination.

Parameters:
  • cmd (str) – Command to execute.
  • quiet (bool) – If True don’t print errors.
  • background (bool) – If True don’t wait until the command exits.
Returns:

stdin, stdout and stderr handlers.

Return type:

tuple

get_stream(**kwds)[source]

Get a PIPE handler with content of the backup copy streamed from the destination.

Parameters:copy (BaseCopy) – Backup copy.
Returns:Standard output.
Return type:file
host
Returns:IP address of the destination.
Return type:str
netcat(command, port=9990)[source]

Run netcat on the destination pipe it to a given command:

ncat -l <port> --recv-only | <command>
Parameters:
  • command (str) – Command that would accept netcat’s output.
  • port (int) – TCP port to run netcat on. Default 9999.
port
Returns:TCP port of the destination.
Return type:int
read(filepath)[source]

Read content of a file path from destination.

Parameters:filepath (str) – Relative path to file.
Returns:Content of the file.
Return type:str
save(handler, filepath)[source]

Read from the handler and save it on the remote ssh server in a file filepath.

Parameters:
  • filepath (str) – Relative path to a file to store the backup copy.
  • handler (file) – Stream with content of the backup.
user
Returns:SSH user.
Return type:str
write(content, filepath)[source]

Write content to a file.

Parameters:
  • content (str) – Content to write to the file.
  • filepath (str) – Relative path to file.

Module contents