twindb_backup package

Subpackages

Submodules

twindb_backup.backup module

Module that parses config file, builds a modifiers chain and fires backup jobs.

twindb_backup.backup.backup_binlogs(run_type, config)[source]

Copy MySQL binlog files to the backup destination.

Parameters
twindb_backup.backup.backup_everything(run_type, twindb_config, binlogs_only=False)[source]

Run backup job

Parameters

  • run_type (str) – hourly, daily, etc

  • twindb_config (TwinDBBackupConfig) – ConfigParser instance

  • binlogs_only (bool) – If True copy only MySQL binary logs.

twindb_backup.backup.backup_files(run_type, config: twindb_backup.configuration.TwinDBBackupConfig)[source]

Backup local directories

Parameters
twindb_backup.backup.backup_mysql(run_type, config)[source]

Take backup of local MySQL instance

Parameters
twindb_backup.backup.binlogs_to_backup(cursor, last_binlog=None)[source]

Finds list of binlogs to copy. It will return the binlogs from the last to the current one (excluding it). If binlog are not enabled in the server the function will return empty list.

Parameters

  • cursor – MySQL cursor

  • last_binlog – Name of the last copied binlog.

Returns

list of binlogs to backup.

Return type

list

twindb_backup.backup.run_backup_job(twindb_config, run_type, lock_file='/var/run/twindb-backup.lock', binlogs_only=False)[source]

Grab a lock waiting up to allowed timeout and start backup jobs

Parameters

  • twindb_config (TwinDBBackupConfig) – Tool configuration

  • run_type (str) – Run type

  • lock_file (str) – File used as a lock

  • binlogs_only (bool) – If True copy only binlogs.

twindb_backup.backup.set_open_files_limit()[source]

Detect maximum supported number of open file and set it

twindb_backup.backup.timeout(seconds)[source]

Implement timeout

Parameters

seconds (int) – timeout in seconds

twindb_backup.cli module

Entry points for twindb-backup tool

twindb_backup.clone module

Module defines clone feature

twindb_backup.clone.clone_mysql(cfg: twindb_backup.configuration.TwinDBBackupConfig, source: str, destination: str, replication_user: str, replication_password: str, netcat_port=9990, compress=False)[source]

Clone a MySQL instance from the remote server on source to the remote server on the destination.

Parameters

  • cfg (TwinDBBackupConfig) – TwinDB Backup tool config.

  • source (str) – A host:port couple e.g. master:3306, the donor.

  • destination (str) – A host:port couple e.g. slave:3306, the recipient.

  • replication_user (str) – Username for the replication.

  • replication_password (str) – Password the replication.

  • netcat_port (int) – A beginning of the TCP ports range to be used by netcat for the stream transfer.

  • compress (bool) – Whether compress the stream or not. gzip will be used if yes.

twindb_backup.clone.detect_xbstream(cfg: twindb_backup.configuration.TwinDBBackupConfig, mysql_client: twindb_backup.source.mysql_source.MySQLClient) → str[source]

Guess what xbtream tool should be used.

If a user specifies the xbstream via the config, it will be used. Otherwise, use appropriate tool for the MySQL flavor.

Parameters
Returns

String, a path to xbstream that will be run on the destination to accept and extract XtraBackup stream.

Return type

str

twindb_backup.clone.get_dst(cfg: twindb_backup.configuration.TwinDBBackupConfig, destination: str) → twindb_backup.destination.ssh.Ssh[source]

Prepare destination object.

Parameters

  • cfg (TwinDBBackupConfig) – TwinDB Backup config.

  • destination (str) – A host:port couple e.g. slave:3306, the recipient.

Returns

Destination Ssh object.

Return type

Ssh

twindb_backup.clone.get_src(cfg: twindb_backup.configuration.TwinDBBackupConfig, mysql_client: twindb_backup.source.mysql_source.MySQLClient, source: str) → Union[twindb_backup.source.remote_mariadb_source.RemoteMariaDBSource, twindb_backup.source.remote_mysql_source.RemoteMySQLSource][source]

Prepare source object.

Parameters

  • cfg (TwinDBBackupConfig) – TwinDB Backup config.

  • mysql_client (MySQLClient) – MySQL client connected to the source.

  • source (str) – A host:port couple e.g. slave:3306, the recipient.

Returns

Source object

Return type

RemoteMariaDBSource, RemoteMySQLSource

twindb_backup.clone.get_src_by_vendor(vendor: twindb_backup.source.mysql_source.MySQLFlavor, ssh_host: str, ssh_user: str, ssh_key_path: str, mysql_connect_info: twindb_backup.source.mysql_source.MySQLConnectInfo, run_type: str)[source]
twindb_backup.clone.step_clone_mysql_config(source: Union[twindb_backup.source.remote_mariadb_source.RemoteMariaDBSource, twindb_backup.source.remote_mysql_source.RemoteMySQLSource], destination: twindb_backup.destination.ssh.Ssh)[source]

Copy MySQL config from the source MySQL server to the destination server.

twindb_backup.clone.step_clone_source(source: Union[twindb_backup.source.remote_mariadb_source.RemoteMariaDBSource, twindb_backup.source.remote_mysql_source.RemoteMySQLSource], destination_hostname: str, netcat_port: int, compress_flag: bool)[source]

Start cloning the source to the destination.

Parameters

  • source – An instance with the source class that will be used as a donor.

  • destination_hostname – A hostname where the clone will be set up.

  • netcat_port – A TCP port for the netcat transfer.

  • compress_flag – Whether compress the stream or not.

twindb_backup.clone.step_configure_replication(destination: Union[twindb_backup.source.remote_mariadb_source.RemoteMariaDBSource, twindb_backup.source.remote_mysql_source.RemoteMySQLSource], master_data: twindb_backup.source.mysql_source.MySQLMasterInfo)[source]

Configure replication on the remote MySQL server.

Parameters

  • destination – MySQL server, a replica.

  • master_data – Replication parameters: master hostname, user, password, etc.

twindb_backup.clone.step_ensure_empty_directory(destination: twindb_backup.destination.ssh.Ssh, path: str)[source]

Check if a directory on the remote server is empty. If not raise an exception.

Parameters

  • destination – Remote server.

  • path – Path to the directory to check.

Raises

OperationError – If the directory is not empty.

twindb_backup.clone.step_run_remote_netcat(compress: bool, datadir, dst, netcat_port, xbstream_path)[source]

Run netcat with xbstream on a remote server via SSH. The function will find a free TCP port for the transfer. It will start looking from a given netcat_port up to max TCP port 65535. The function will yield the found port number.

Parameters

  • compress (bool) – Compress the stream or not.

  • datadir (str) – The directory on the remote destination server where to uncompress the stream.

  • dst –

  • netcat_port –

  • xbstream_path –

Returns

twindb_backup.clone.step_start_mysql_service(dst)[source]

Start MySQL service

Parameters

dst (Ssh) – Destination server

twindb_backup.exceptions module

Module that describes exceptions of twindb_backup module.

exception twindb_backup.exceptions.LockWaitTimeoutError[source]

Bases: twindb_backup.exceptions.TwinDBBackupError

Timeout expired while waiting for a lock

exception twindb_backup.exceptions.OperationError[source]

Bases: twindb_backup.exceptions.TwinDBBackupError

Exceptions that prevent normal TwinDB Backup operation

exception twindb_backup.exceptions.TwinDBBackupError[source]

Bases: Exception

Catch-all exceptions

exception twindb_backup.exceptions.TwinDBBackupInternalError[source]

Bases: twindb_backup.exceptions.TwinDBBackupError

Internal errors in the tool itself

twindb_backup.export module

Module to process export

twindb_backup.export.export_info(cfg, data, category, measure_type)[source]

Export data to service

Parameters

  • cfg (TwinDBBackupConfig) – Config file

  • data – Data

  • category – Category of data

  • measure_type – Type of measure

  • category – Category

twindb_backup.ls module

Module that works with list of backup copies

twindb_backup.ls.list_available_backups(twindb_config, copy_type=None)[source]

Print known backup copies on a destination specified in the configuration.

Parameters

  • twindb_config (TwinDBBackupConfig) – tool configuration

  • copy_type (files|mysql) – Limit list to specific type of backups.

twindb_backup.restore module

Module that restores backup copies.

twindb_backup.restore.gen_grastate(path, version, uuid, seqno)[source]

Generate and save grastate file.

Parameters

  • path – Path to grastate file.

  • version – Galera version from grastate.dat.

  • uuid – UUID from grastate.dat.

  • seqno – seqno from grastate.dat.

twindb_backup.restore.get_free_memory()[source]

Return size of available memory in bytes. It calculates it as a half of available memory.

Returns

Size of free memory in bytes

Return type

int

twindb_backup.restore.get_my_cnf(status, key)[source]

Get MySQL config from the status.

Parameters

  • status (MySQLStatus) – Backup status.

  • key (str) – Backup name.

Returns

Content of my.cnf or None if not found

Return type

str

twindb_backup.restore.restore_from_file(twindb_config, copy, dst_dir)[source]

Restore a directory from a backup copy in the directory

Parameters

  • twindb_config (TwinDBBackupConfig) – tool configuration

  • copy (BaseCopy) – Instance of BaseCopy or and inheriting classes.

  • dst_dir (str) – Path to destination directory. Must exist and be empty.

twindb_backup.restore.restore_from_mysql(twindb_config, copy: twindb_backup.copy.mysql_copy.MySQLCopy, dst_dir, tmp_dir=None, cache=None, hostname=None)[source]

Restore MySQL datadir in a given directory

Parameters

  • twindb_config (TwinDBBackupConfig) – tool configuration

  • copy (MySQLCopy) – Backup copy instance.

  • dst_dir (str) – Destination directory. Must exist and be empty.

  • tmp_dir (str) – Path to temp directory

  • cache (Cache) – Local cache object.

  • hostname (str) – Hostname

twindb_backup.restore.restore_from_mysql_full(stream, dst_dir, config, redo_only=False, xtrabackup_binary='xtrabackup', xbstream_binary='xbstream')[source]

Restore MySQL datadir from a backup copy

Parameters

  • stream – Generator that provides backup copy

  • dst_dir (str) – Path to destination directory. Must exist and be empty.

  • config (TwinDBBackupConfig) – Tool configuration.

  • redo_only (bool) – True if the function has to do final apply of the redo log. For example, if you restore backup from a full copy it should be False. If you restore from incremental copy and you restore base full copy redo_only should be True.

  • xtrabackup_binary – path to xtrabackup binary.

  • xbstream_binary – Path to xbstream binary

Returns

If success, return True

Return type

bool

twindb_backup.restore.restore_from_mysql_incremental(stream, dst_dir, config, tmp_dir=None, xtrabackup_binary='xtrabackup', xbstream_binary='xbstream')[source]

Restore MySQL datadir from an incremental copy.

Parameters

  • stream – Generator that provides backup copy

  • dst_dir (str) – Path to destination directory. Must exist and be empty.

  • config (TwinDBBackupConfig) – Tool configuration.

  • tmp_dir (str) – Path to temp dir

  • xtrabackup_binary – Path to xtrabackup binary.

  • xbstream_binary – Path to xbstream binary

Returns

If success, return True

Return type

bool

twindb_backup.restore.update_grastate(dst_dir, status, key)[source]

If xtrabackup_galera_info exists in the destination directory then parse it and generate grastate.dat file.

Parameters

  • dst_dir (str) – Path to destination directory.

  • status (dict) – Backup status

  • key (str) – Backup name

twindb_backup.share module

Module that works with sharing backups

twindb_backup.share.share(twindb_config, s3_url)[source]

Function for generate make public file and get public url

Parameters

  • twindb_config (TwinDBBackupConfig) – tool configuration

  • s3_url (str) – S3 url to file

Raise

TwinDBBackupError

twindb_backup.util module

Module with helper functions

twindb_backup.util.empty_dir(path)[source]

Remove all files are directories in path

Parameters

path (str) – Path to directory to be emptied.

twindb_backup.util.ensure_empty(path)[source]

Check if a given directory is empty and exit if not.

Parameters

path (str) – path to directory

twindb_backup.util.kill_children()[source]

Kill child process

twindb_backup.util.mkdir_p(path, mode=511)[source]

Emulate mkdir -p. Create a directory named path with numeric mode mode. The default mode is 0777 (octal)

Parameters

  • path (str) – Directory path.

  • mode (int) – Directory permissions. The default mode is 0777 (octal)

twindb_backup.util.my_cnfs(common_paths=None)[source]

Start reading a root my.cnf file given in common paths and parse included files.

Parameters

common_paths (list) – list of my.cnf files to start parsing from.

Returns

list of all included my.cnf files

Return type

list

twindb_backup.util.normalize_b64_data(coding)[source]

Normalize base64 key. See http://bit.ly/2vxIAnC for details.

Parameters

coding – Encoded data

Returns

Normalized encoded data

twindb_backup.util.run_command(command, ok_non_zero=False)[source]

Run shell command locally

Parameters

  • command (list) – Command to run

  • ok_non_zero (bool) – Don’t consider non-zero exit code as an error.

Returns

file object with stdout as generator to use with with

twindb_backup.util.split_host_port(host_port)[source]

Splits a string of host and port separated by a semicolon.

Parameters

host_port – host or host:port. Allowed values are like 10.20.31.1:3306 or just 10.20.31.1

Returns

a tuple with host and port. If only address is specified it’ll return (address, None). If host_port is None it will return (None, None)

Return type

tuple

twindb_backup.verify module

Module that verify backup copies.

twindb_backup.verify.edit_backup_my_cnf(dst_path)[source]

Removed options from config(besides MySQL 5.7.8)

twindb_backup.verify.verify_mysql_backup(twindb_config, dst_path, backup_file, hostname=None)[source]

Restore mysql backup and measure time

Parameters

  • hostname –

  • backup_file –

  • dst_path –

  • twindb_config (TwinDBBackupConfig) – tool configuration

Module contents

TwinDB Backup module.

The module is a core of twindb-backup tool. It includes backup and restore functionality. The module takes a backup from something defined in a source class and saves the backup copy in something defined in a destination class.

The source class inherits from BaseSource() from twindb_backup.source.base_source.py. The source class must define get_stream() method that yields a file object that is used for next classes. Typical classes are FileSource() to backup files and directories, MySQLSource() to backup MySQL.

The destination class inherits from BaseDestination(). This is where you store backups. The destination class must define save() method that takes an input stream and saves it somewhere. Examples of the destination class are S3(), Ssh().

There are modifier classes. The modifier class sits in the middle between the source and the destination and does something with a stream before the stream is saved. The modifier class may save a local copy (KeepLocal()) or encrypt the stream or else. The modifier class inherits Modifier()

The backup process may be depicted as a chain of modifiers with the source in the head and the destination in the tail.

+--------+    +------------+    +------------+    +-------------+
| source | -- | modifier 1 | -- | modifier 2 | -- | destination |
+--------+    +------------+    +------------+    +-------------+
twindb_backup.DestTypes

alias of twindb_backup.DestinationTypes

class twindb_backup.LessThanFilter(exclusive_maximum, name='')[source]

Bases: logging.Filter

Filters out log messages of a lower level.

filter(record)[source]

Determine if the specified record is to be logged.

Is the specified record to be logged? Returns 0 for no, nonzero for yes. If deemed appropriate, the record may be modified in-place.

class twindb_backup.QueryTypes(mysql)

Bases: tuple

mysql

Alias for field number 0

twindb_backup.delete_local_files(dir_backups, keep_copies)[source]

Deletes local backup copies based on given retention number.

Parameters

  • dir_backups (str) – directory with backup copies

  • keep_copies (int) – how many to keep

Returns

None

twindb_backup.get_files_to_delete(all_files, keep_copies)[source]

If you give it a list of files and number of how many you’d like to keep the function will return files that need to be deleted

Parameters

  • all_files (list) – list of strings

  • keep_copies (int) – number of copied to keep

Returns

list of strings (files) to delete

Return type

list

twindb_backup.get_timeout(run_type)[source]

Get timeout for a each run type - daily, hourly etc

Parameters

run_type (str) – Run type

Returns

Number of seconds the tool allowed to wait until other instances finish

Return type

int

twindb_backup.save_measures(start_time, end_time, log_path='/var/log/twindb-backup-measures.log')[source]

Save backup measures to log file

twindb_backup.setup_logging(logger, debug=False)[source]

Configures logging for the module