twindb_backup package

Subpackages

Submodules

twindb_backup.backup module

Module that parses config file, builds a modifiers chain and fires backup jobs.

twindb_backup.backup.backup_binlogs(run_type, config)[source]

Copy MySQL binlog files to the backup destination.

Parameters:
twindb_backup.backup.backup_everything(run_type, twindb_config, binlogs_only=False)[source]

Run backup job

Parameters:
  • run_type (str) – hourly, daily, etc
  • twindb_config (TwinDBBackupConfig) – ConfigParser instance
  • binlogs_only (bool) – If True copy only MySQL binary logs.
twindb_backup.backup.backup_files(run_type, config)[source]

Backup local directories

Parameters:
twindb_backup.backup.backup_mysql(run_type, config)[source]

Take backup of local MySQL instance

Parameters:
twindb_backup.backup.binlogs_to_backup(cursor, last_binlog=None)[source]

Finds list of binlogs to copy. It will return the binlogs from the last to the current one (excluding it). If binlog are not enabled in the server the function will return empty list.

Parameters:
  • cursor – MySQL cursor
  • last_binlog – Name of the last copied binlog.
Returns:

list of binlogs to backup.

Return type:

list

twindb_backup.backup.run_backup_job(twindb_config, run_type, lock_file='/var/run/twindb-backup.lock', binlogs_only=False)[source]

Grab a lock waiting up to allowed timeout and start backup jobs

Parameters:
  • twindb_config (TwinDBBackupConfig) – Tool configuration
  • run_type (str) – Run type
  • lock_file (str) – File used as a lock
  • binlogs_only (bool) – If True copy only binlogs.
twindb_backup.backup.set_open_files_limit()[source]

Detect maximum supported number of open file and set it

twindb_backup.backup.timeout(*args, **kwds)[source]

Implement timeout

Parameters:seconds (int) – timeout in seconds

twindb_backup.cli module

Entry points for twindb-backup tool

twindb_backup.clone module

Module defines clone feature

twindb_backup.clone.clone_mysql(cfg, source, destination, replication_user, replication_password, netcat_port=9990, compress=False)[source]

Clone mysql backup of remote machine and stream it to slave

Parameters:cfg (TwinDBBackupConfig) – TwinDB Backup tool config

twindb_backup.exceptions module

Module that describes exceptions of twindb_backup module

exception twindb_backup.exceptions.LockWaitTimeoutError[source]

Bases: twindb_backup.exceptions.TwinDBBackupError

Class that describes exception of lock wait timeout

exception twindb_backup.exceptions.OperationError[source]

Bases: twindb_backup.exceptions.TwinDBBackupError

High level exceptions of twindb_backup package

exception twindb_backup.exceptions.TwinDBBackupError[source]

Bases: exceptions.Exception

Catch-all exceptions

exception twindb_backup.exceptions.TwinDBBackupInternalError[source]

Bases: twindb_backup.exceptions.TwinDBBackupError

Internal errors in the tool itself

twindb_backup.export module

Module to process export

twindb_backup.export.export_info(cfg, data, category, measure_type)[source]

Export data to service

Parameters:
  • cfg (TwinDBBackupConfig) – Config file
  • data – Data
  • category – Category of data
  • measure_type – Type of measure
  • category – Category

twindb_backup.ls module

Module that works with list of backup copies

twindb_backup.ls.list_available_backups(twindb_config, copy_type=None)[source]

Print known backup copies on a destination specified in the configuration.

Parameters:
  • twindb_config (TwinDBBackupConfig) – tool configuration
  • copy_type (files|mysql) – Limit list to specific type of backups.

twindb_backup.restore module

Module that restores backup copies.

twindb_backup.restore.gen_grastate(path, version, uuid, seqno)[source]

Generate and save grastate file.

Parameters:
  • path – Path to grastate file.
  • version – Galera version from grastate.dat.
  • uuid – UUID from grastate.dat.
  • seqno – seqno from grastate.dat.
twindb_backup.restore.get_my_cnf(status, key)[source]

Get MySQL config from the status.

Parameters:
  • status (MySQLStatus) – Backup status.
  • key (str) – Backup name.
Returns:

Content of my.cnf or None if not found

Return type:

str

twindb_backup.restore.restore_from_file(twindb_config, copy, dst_dir)[source]

Restore a directory from a backup copy in the directory

Parameters:
  • twindb_config (TwinDBBackupConfig) – tool configuration
  • copy (BaseCopy) – Instance of BaseCopy or and inheriting classes.
  • dst_dir (str) – Path to destination directory. Must exist and be empty.
twindb_backup.restore.restore_from_mysql(twindb_config, copy, dst_dir, tmp_dir=None, cache=None, hostname=None)[source]

Restore MySQL datadir in a given directory

Parameters:
  • twindb_config (TwinDBBackupConfig) – tool configuration
  • copy (MySQLCopy) – Backup copy instance.
  • dst_dir (str) – Destination directory. Must exist and be empty.
  • tmp_dir (str) – Path to temp directory
  • cache (Cache) – Local cache object.
  • hostname (str) – Hostname
twindb_backup.restore.restore_from_mysql_full(stream, dst_dir, config, redo_only=False, xtrabackup_binary='/opt/twindb-backup/embedded/bin/xtrabackup', xbstream_binary='/opt/twindb-backup/embedded/bin/xbstream')[source]

Restore MySQL datadir from a backup copy

Parameters:
  • stream – Generator that provides backup copy
  • dst_dir (str) – Path to destination directory. Must exist and be empty.
  • config (TwinDBBackupConfig) – Tool configuration.
  • redo_only (bool) – True if the function has to do final apply of the redo log. For example, if you restore backup from a full copy it should be False. If you restore from incremental copy and you restore base full copy redo_only should be True.
  • xtrabackup_binary – path to xtrabackup binary.
  • xbstream_binary – Path to xbstream binary
Returns:

If success, return True

Return type:

bool

twindb_backup.restore.restore_from_mysql_incremental(stream, dst_dir, config, tmp_dir=None, xtrabackup_binary='/opt/twindb-backup/embedded/bin/xtrabackup', xbstream_binary='/opt/twindb-backup/embedded/bin/xbstream')[source]

Restore MySQL datadir from an incremental copy.

Parameters:
  • stream – Generator that provides backup copy
  • dst_dir (str) – Path to destination directory. Must exist and be empty.
  • config (TwinDBBackupConfig) – Tool configuration.
  • tmp_dir (str) – Path to temp dir
  • xtrabackup_binary – Path to xtrabackup binary.
  • xbstream_binary – Path to xbstream binary
Returns:

If success, return True

Return type:

bool

twindb_backup.restore.update_grastate(dst_dir, status, key)[source]

If xtrabackup_galera_info exists in the destination directory then parse it and generate grastate.dat file.

Parameters:
  • dst_dir (str) – Path to destination directory.
  • status (dict) – Backup status
  • key (str) – Backup name

twindb_backup.share module

Module that works with sharing backups

twindb_backup.share.share(twindb_config, s3_url)[source]

Function for generate make public file and get public url

Parameters:
  • twindb_config (TwinDBBackupConfig) – tool configuration
  • s3_url (str) – S3 url to file
Raise:

TwinDBBackupError

twindb_backup.util module

Module with helper functions

twindb_backup.util.empty_dir(path)[source]

Remove all files are directories in path

Parameters:path (str) – Path to directory to be emptied.
twindb_backup.util.ensure_empty(path)[source]

Check if a given directory is empty and exit if not.

Parameters:path (str) – path to directory
twindb_backup.util.kill_children()[source]

Kill child process

twindb_backup.util.mkdir_p(path, mode=511)[source]

Emulate mkdir -p. Create a directory named path with numeric mode mode. The default mode is 0777 (octal)

Parameters:
  • path (str) – Directory path.
  • mode (int) – Directory permissions. The default mode is 0777 (octal)
twindb_backup.util.my_cnfs(common_paths=None)[source]

Start reading a root my.cnf file given in common paths and parse included files.

Parameters:common_paths (list) – list of my.cnf files to start parsing from.
Returns:list of all included my.cnf files
Return type:list
twindb_backup.util.normalize_b64_data(coding)[source]

Normalize base64 key. See http://bit.ly/2vxIAnC for details.

Parameters:coding – Encoded data
Returns:Normalized encoded data
twindb_backup.util.run_command(*args, **kwds)[source]

Run shell command locally

Parameters:
  • command (list) – Command to run
  • ok_non_zero (bool) – Don’t consider non-zero exit code as an error.
Returns:

file object with stdout as generator to use with with

twindb_backup.util.split_host_port(host_port)[source]

Splits a string of host and port separated by a semicolon.

Parameters:host_port – host or host:port. Allowed values are like 10.20.31.1:3306 or just 10.20.31.1
Returns:a tuple with host and port. If only address is specified it’ll return (address, None). If host_port is None it will return (None, None)
Return type:tuple

twindb_backup.verify module

Module that verify backup copies.

twindb_backup.verify.edit_backup_my_cnf(dst_path)[source]

Removed options from config(besides MySQL 5.7.8)

twindb_backup.verify.verify_mysql_backup(twindb_config, dst_path, backup_file, hostname=None)[source]

Restore mysql backup and measure time

Parameters:
  • hostname
  • backup_file
  • dst_path
  • twindb_config (TwinDBBackupConfig) – tool configuration

Module contents

TwinDB Backup module.

The module is a core of twindb-backup tool. It includes backup and restore functionality. The module takes a backup from something defined in a source class and saves the backup copy in something defined in a destination class.

The source class inherits from BaseSource() from twindb_backup.source.base_source.py. The source class must define get_stream() method that yields a file object that is used for next classes. Typical classes are FileSource() to backup files and directories, MySQLSource() to backup MySQL.

The destination class inherits from BaseDestination(). This is where you store backups. The destination class must define save() method that takes an input stream and saves it somewhere. Examples of the destination class are S3(), Ssh().

There are modifier classes. The modifier class sits in the middle between the source and the destination and does something with a stream before the stream is saved. The modifier class may save a local copy (KeepLocal()) or encrypt the stream or else. The modifier class inherits Modifier()

The backup process may be depicted as a chain of modifiers with the source in the head and the destination in the tail.

+--------+    +------------+    +------------+    +-------------+
| source | -- | modifier 1 | -- | modifier 2 | -- | destination |
+--------+    +------------+    +------------+    +-------------+
class twindb_backup.LessThanFilter(exclusive_maximum, name='')[source]

Bases: logging.Filter

Filters out log messages of a lower level.

filter(record)[source]

Determine if the specified record is to be logged.

Is the specified record to be logged? Returns 0 for no, nonzero for yes. If deemed appropriate, the record may be modified in-place.

twindb_backup.delete_local_files(dir_backups, keep_copies)[source]

Deletes local backup copies based on given retention number.

Parameters:
  • dir_backups (str) – directory with backup copies
  • keep_copies (int) – how many to keep
Returns:

None

twindb_backup.get_files_to_delete(all_files, keep_copies)[source]

If you give it a list of files and number of how many you’d like to keep the function will return files that need to be deleted

Parameters:
  • all_files (list) – list of strings
  • keep_copies (int) – number of copied to keep
Returns:

list of strings (files) to delete

Return type:

list

twindb_backup.get_timeout(run_type)[source]

Get timeout for a each run type - daily, hourly etc

Parameters:run_type (str) – Run type
Returns:Number of seconds the tool allowed to wait until other instances finish
Return type:int
twindb_backup.save_measures(start_time, end_time, log_path='/var/log/twindb-backup-measures.log')[source]

Save backup measures to log file

twindb_backup.setup_logging(logger, debug=False)[source]

Configures logging for the module