twindb_backup package¶

Subpackages¶

Submodules¶

twindb_backup.backup module¶

Module that parses config file, builds a modifiers chain and fires backup jobs.

twindb_backup.backup.backup_binlogs(run_type, config)[source]¶

Copy MySQL binlog files to the backup destination.

Parameters:	run_type (str) – Run type config (TwinDBBackupConfig) – Tool configuration

twindb_backup.backup.backup_everything(run_type, twindb_config, binlogs_only=False)[source]¶

Run backup job

Parameters:	run_type (str) – hourly, daily, etc twindb_config (TwinDBBackupConfig) – ConfigParser instance binlogs_only (bool) – If True copy only MySQL binary logs.

twindb_backup.backup.backup_files(run_type, config)[source]¶

Backup local directories

Parameters:	run_type (str) – Run type config (TwinDBBackupConfig) – Configuration

twindb_backup.backup.backup_mysql(run_type, config)[source]¶

Take backup of local MySQL instance

Parameters:	run_type (str) – Run type config (TwinDBBackupConfig) – Tool configuration

twindb_backup.backup.binlogs_to_backup(cursor, last_binlog=None)[source]¶

Finds list of binlogs to copy. It will return the binlogs from the last to the current one (excluding it). If binlog are not enabled in the server the function will return empty list.

Parameters:	cursor – MySQL cursor last_binlog – Name of the last copied binlog.
Returns:	list of binlogs to backup.
Return type:	list

twindb_backup.backup.run_backup_job(twindb_config, run_type, lock_file='/var/run/twindb-backup.lock', binlogs_only=False)[source]¶

Grab a lock waiting up to allowed timeout and start backup jobs

Parameters:	twindb_config (TwinDBBackupConfig) – Tool configuration run_type (str) – Run type lock_file (str) – File used as a lock binlogs_only (bool) – If True copy only binlogs.

twindb_backup.backup.set_open_files_limit()[source]¶: Detect maximum supported number of open file and set it

twindb_backup.backup.timeout(seconds)[source]¶

Implement timeout

Parameters:	seconds (int) – timeout in seconds

twindb_backup.cli module¶

Entry points for twindb-backup tool

twindb_backup.clone module¶

Module defines clone feature

twindb_backup.clone.clone_mysql(cfg, source, destination, replication_user, replication_password, netcat_port=9990, compress=False)[source]¶

Clone mysql backup of remote machine and stream it to slave

Parameters:	cfg (TwinDBBackupConfig) – TwinDB Backup tool config

twindb_backup.exceptions module¶

Module that describes exceptions of twindb_backup module.

exception twindb_backup.exceptions.LockWaitTimeoutError[source]¶

Bases: twindb_backup.exceptions.TwinDBBackupError

Class that describes exception of lock wait timeout

exception twindb_backup.exceptions.OperationError[source]¶

Bases: twindb_backup.exceptions.TwinDBBackupError

High level exceptions of twindb_backup package

exception twindb_backup.exceptions.TwinDBBackupError[source]¶

Bases: Exception

Catch-all exceptions

exception twindb_backup.exceptions.TwinDBBackupInternalError[source]¶

Bases: twindb_backup.exceptions.TwinDBBackupError

Internal errors in the tool itself

twindb_backup.export module¶

Module to process export

twindb_backup.export.export_info(cfg, data, category, measure_type)[source]¶

Export data to service

Parameters:	cfg (TwinDBBackupConfig) – Config file data – Data category – Category of data measure_type – Type of measure category – Category

twindb_backup.ls module¶

Module that works with list of backup copies

twindb_backup.ls.list_available_backups(twindb_config, copy_type=None)[source]¶

Print known backup copies on a destination specified in the configuration.

Parameters:	twindb_config (TwinDBBackupConfig) – tool configuration copy_type (files\|mysql) – Limit list to specific type of backups.

twindb_backup.restore module¶

Module that restores backup copies.

twindb_backup.restore.gen_grastate(path, version, uuid, seqno)[source]¶

Generate and save grastate file.

Parameters:	path – Path to grastate file. version – Galera version from grastate.dat. uuid – UUID from grastate.dat. seqno – seqno from grastate.dat.

twindb_backup.restore.get_my_cnf(status, key)[source]¶

Get MySQL config from the status.

Parameters:	status (MySQLStatus) – Backup status. key (str) – Backup name.
Returns:	Content of my.cnf or None if not found
Return type:	str

twindb_backup.restore.restore_from_file(twindb_config, copy, dst_dir)[source]¶

Restore a directory from a backup copy in the directory

Parameters:	twindb_config (TwinDBBackupConfig) – tool configuration copy (BaseCopy) – Instance of BaseCopy or and inheriting classes. dst_dir (str) – Path to destination directory. Must exist and be empty.

twindb_backup.restore.restore_from_mysql(twindb_config, copy, dst_dir, tmp_dir=None, cache=None, hostname=None)[source]¶

Restore MySQL datadir in a given directory

Parameters:	twindb_config (TwinDBBackupConfig) – tool configuration copy (MySQLCopy) – Backup copy instance. dst_dir (str) – Destination directory. Must exist and be empty. tmp_dir (str) – Path to temp directory cache (Cache) – Local cache object. hostname (str) – Hostname

twindb_backup.restore.restore_from_mysql_full(stream, dst_dir, config, redo_only=False, xtrabackup_binary='/opt/twindb-backup/embedded/bin/xtrabackup', xbstream_binary='/opt/twindb-backup/embedded/bin/xbstream')[source]¶

Restore MySQL datadir from a backup copy

Parameters:	stream – Generator that provides backup copy dst_dir (str) – Path to destination directory. Must exist and be empty. config (TwinDBBackupConfig) – Tool configuration. redo_only (bool) – True if the function has to do final apply of the redo log. For example, if you restore backup from a full copy it should be False. If you restore from incremental copy and you restore base full copy redo_only should be True. xtrabackup_binary – path to xtrabackup binary. xbstream_binary – Path to xbstream binary
Returns:	If success, return True
Return type:	bool

twindb_backup.restore.restore_from_mysql_incremental(stream, dst_dir, config, tmp_dir=None, xtrabackup_binary='/opt/twindb-backup/embedded/bin/xtrabackup', xbstream_binary='/opt/twindb-backup/embedded/bin/xbstream')[source]¶

Restore MySQL datadir from an incremental copy.

Parameters:	stream – Generator that provides backup copy dst_dir (str) – Path to destination directory. Must exist and be empty. config (TwinDBBackupConfig) – Tool configuration. tmp_dir (str) – Path to temp dir xtrabackup_binary – Path to xtrabackup binary. xbstream_binary – Path to xbstream binary
Returns:	If success, return True
Return type:	bool

twindb_backup.restore.update_grastate(dst_dir, status, key)[source]¶

If xtrabackup_galera_info exists in the destination directory then parse it and generate grastate.dat file.

Parameters:	dst_dir (str) – Path to destination directory. status (dict) – Backup status key (str) – Backup name

twindb_backup.share module¶

Module that works with sharing backups

twindb_backup.share.share(twindb_config, s3_url)[source]¶

Function for generate make public file and get public url

Parameters:	twindb_config (TwinDBBackupConfig) – tool configuration s3_url (str) – S3 url to file
Raise:	TwinDBBackupError

twindb_backup.util module¶

Module with helper functions

twindb_backup.util.empty_dir(path)[source]¶

Remove all files are directories in path

Parameters:	path (str) – Path to directory to be emptied.

twindb_backup.util.ensure_empty(path)[source]¶

Check if a given directory is empty and exit if not.

Parameters:	path (str) – path to directory

twindb_backup.util.kill_children()[source]¶: Kill child process

twindb_backup.util.mkdir_p(path, mode=511)[source]¶

Emulate mkdir -p. Create a directory named path with numeric mode mode. The default mode is 0777 (octal)

Parameters:	path (str) – Directory path. mode (int) – Directory permissions. The default mode is 0777 (octal)

twindb_backup.util.my_cnfs(common_paths=None)[source]¶

Start reading a root my.cnf file given in common paths and parse included files.

Parameters:	common_paths (list) – list of my.cnf files to start parsing from.
Returns:	list of all included my.cnf files
Return type:	list

twindb_backup.util.normalize_b64_data(coding)[source]¶

Normalize base64 key. See http://bit.ly/2vxIAnC for details.

Parameters:	coding – Encoded data
Returns:	Normalized encoded data

twindb_backup.util.run_command(command, ok_non_zero=False)[source]¶

Run shell command locally

Parameters:	command (list) – Command to run ok_non_zero (bool) – Don’t consider non-zero exit code as an error.
Returns:	file object with stdout as generator to use with `with`

twindb_backup.util.split_host_port(host_port)[source]¶

Splits a string of host and port separated by a semicolon.

Parameters:	host_port – host or host:port. Allowed values are like 10.20.31.1:3306 or just 10.20.31.1
Returns:	a tuple with host and port. If only address is specified it’ll return (address, None). If host_port is None it will return (None, None)
Return type:	tuple

twindb_backup.verify module¶

Module that verify backup copies.

twindb_backup.verify.edit_backup_my_cnf(dst_path)[source]¶: Removed options from config(besides MySQL 5.7.8)

twindb_backup.verify.verify_mysql_backup(twindb_config, dst_path, backup_file, hostname=None)[source]¶

Restore mysql backup and measure time

Parameters:	hostname – backup_file – dst_path – twindb_config (TwinDBBackupConfig) – tool configuration

Module contents¶

TwinDB Backup module.

The module is a core of twindb-backup tool. It includes backup and restore functionality. The module takes a backup from something defined in a source class and saves the backup copy in something defined in a destination class.

The source class inherits from BaseSource() from twindb_backup.source.base_source.py. The source class must define get_stream() method that yields a file object that is used for next classes. Typical classes are FileSource() to backup files and directories, MySQLSource() to backup MySQL.

The destination class inherits from BaseDestination(). This is where you store backups. The destination class must define save() method that takes an input stream and saves it somewhere. Examples of the destination class are S3(), Ssh().

There are modifier classes. The modifier class sits in the middle between the source and the destination and does something with a stream before the stream is saved. The modifier class may save a local copy (KeepLocal()) or encrypt the stream or else. The modifier class inherits Modifier()

The backup process may be depicted as a chain of modifiers with the source in the head and the destination in the tail.

+--------+    +------------+    +------------+    +-------------+
| source | -- | modifier 1 | -- | modifier 2 | -- | destination |
+--------+    +------------+    +------------+    +-------------+

class twindb_backup.LessThanFilter(exclusive_maximum, name='')[source]¶

Bases: logging.Filter

Filters out log messages of a lower level.

filter(record)[source]¶

Determine if the specified record is to be logged.

Is the specified record to be logged? Returns 0 for no, nonzero for yes. If deemed appropriate, the record may be modified in-place.

twindb_backup.delete_local_files(dir_backups, keep_copies)[source]¶

Deletes local backup copies based on given retention number.

Parameters:	dir_backups (str) – directory with backup copies keep_copies (int) – how many to keep
Returns:	None

twindb_backup.get_files_to_delete(all_files, keep_copies)[source]¶

If you give it a list of files and number of how many you’d like to keep the function will return files that need to be deleted

Parameters:	all_files (list) – list of strings keep_copies (int) – number of copied to keep
Returns:	list of strings (files) to delete
Return type:	list

twindb_backup.get_timeout(run_type)[source]¶

Get timeout for a each run type - daily, hourly etc

Parameters:	run_type (str) – Run type
Returns:	Number of seconds the tool allowed to wait until other instances finish
Return type:	int

twindb_backup.save_measures(start_time, end_time, log_path='/var/log/twindb-backup-measures.log')[source]¶: Save backup measures to log file

twindb_backup.setup_logging(logger, debug=False)[source]¶: Configures logging for the module