twindb_backup package



twindb_backup.backup module

Module that parses config file, builds a modifiers chain and fires backup jobs.

twindb_backup.backup.backup_binlogs(run_type, config)[source]

Copy MySQL binlog files to the backup destination.

twindb_backup.backup.backup_everything(run_type, twindb_config, binlogs_only=False)[source]

Run backup job

  • run_type (str) – hourly, daily, etc
  • twindb_config (TwinDBBackupConfig) – ConfigParser instance
  • binlogs_only (bool) – If True copy only MySQL binary logs.
twindb_backup.backup.backup_files(run_type, config)[source]

Backup local directories

twindb_backup.backup.backup_mysql(run_type, config)[source]

Take backup of local MySQL instance

twindb_backup.backup.binlogs_to_backup(cursor, last_binlog=None)[source]

Finds list of binlogs to copy. It will return the binlogs from the last to the current one (excluding it).

  • cursor – MySQL cursor
  • last_binlog – Name of the last copied binlog.

list of binlogs to backup.

Return type:


twindb_backup.backup.run_backup_job(twindb_config, run_type, lock_file='/var/run/twindb-backup.lock', binlogs_only=False)[source]

Grab a lock waiting up to allowed timeout and start backup jobs

  • twindb_config (TwinDBBackupConfig) – Tool configuration
  • run_type (str) – Run type
  • lock_file (str) – File used as a lock
  • binlogs_only (bool) – If True copy only binlogs.

Detect maximum supported number of open file and set it

twindb_backup.backup.timeout(*args, **kwds)[source]

Implement timeout

Parameters:seconds (int) – timeout in seconds

twindb_backup.cli module

Entry points for twindb-backup tool

twindb_backup.clone module

Module defines clone feature

twindb_backup.clone.clone_mysql(cfg, source, destination, replication_user, replication_password, netcat_port=9990, compress=False)[source]

Clone mysql backup of remote machine and stream it to slave

Parameters:cfg (TwinDBBackupConfig) – TwinDB Backup tool config

twindb_backup.configuration module

Module to process configuration file.

class twindb_backup.configuration.TwinDBBackupConfig(config_file='/etc/twindb/twindb-backup.cfg')[source]

Bases: object

Class represents TwinDB Backup configuration


Directories to backup


FLag to backup MySQL or not

Parameters:backup_source (str) – Hostname of the host where backup is taken from.
Returns:Backup destination instance
Return type:BaseDestination

Read config and return export transport instance

Returns:Instance of export transport, if it is set
Return type:BaseExporter
Raise:ConfigurationError, if transport isn’t implemented

GPG configuration.


If specified a local path where the tool will keep an additional local backup copy.

Returns:Local MySQL source configuration.
Return type:MySQLConfig
Returns:Remote retention policy.
Return type:RetentionPolicy
Returns:Local retention policy.
Return type:RetentionPolicy

Run intervals config. When to run or not the backup.

Returns:Configuration with data on whether to run the backup tool now.
Return type:RunIntervals

Amazon S3 configuration

Returns:Remote SSH configuration.
Return type:SSHConfig

twindb_backup.exceptions module

Module that describes exceptions of twindb_backup module

exception twindb_backup.exceptions.LockWaitTimeoutError[source]

Bases: exceptions.Exception

Class that describes exception of lock wait timeout

exception twindb_backup.exceptions.OperationError[source]

Bases: exceptions.Exception

High level exceptions of twindb_backup package

twindb_backup.export module

Module to process export

twindb_backup.export.export_info(cfg, data, category, measure_type)[source]

Export data to service

  • cfg (TwinDBBackupConfig) – Config file
  • data – Data
  • category – Category of data
  • measure_type – Type of measure
  • category – Category module

Module that works with list of backup copies, copy_type=None)[source]

Print known backup copies on a destination specified in the configuration.

  • twindb_config (TwinDBBackupConfig) – tool configuration
  • copy_type (files|mysql) – Limit list to specific type of backups.

twindb_backup.restore module

Module that restores backup copies.

twindb_backup.restore.gen_grastate(path, version, uuid, seqno)[source]

Generate and save grastate file.

  • path – Path to grastate file.
  • version – Galera version from grastate.dat.
  • uuid – UUID from grastate.dat.
  • seqno – seqno from grastate.dat.
twindb_backup.restore.get_my_cnf(status, key)[source]

Get MySQL config from the status.

  • status (MySQLStatus) – Backup status.
  • key (str) – Backup name.

Content of my.cnf or None if not found

Return type:


twindb_backup.restore.restore_from_file(twindb_config, copy, dst_dir)[source]

Restore a directory from a backup copy in the directory

  • twindb_config (TwinDBBackupConfig) – tool configuration
  • copy (BaseCopy) – Instance of BaseCopy or and inheriting classes.
  • dst_dir (str) – Path to destination directory. Must exist and be empty.
twindb_backup.restore.restore_from_mysql(twindb_config, copy, dst_dir, tmp_dir=None, cache=None, hostname=None)[source]

Restore MySQL datadir in a given directory

  • twindb_config (TwinDBBackupConfig) – tool configuration
  • copy (MySQLCopy) – Backup copy instance.
  • dst_dir (str) – Destination directory. Must exist and be empty.
  • tmp_dir (str) – Path to temp directory
  • cache (Cache) – Local cache object.
  • hostname (str) – Hostname
twindb_backup.restore.restore_from_mysql_full(stream, dst_dir, config, redo_only=False, xtrabackup_binary='/opt/twindb-backup/embedded/bin/xtrabackup', xbstream_binary='/opt/twindb-backup/embedded/bin/xbstream')[source]

Restore MySQL datadir from a backup copy

  • stream – Generator that provides backup copy
  • dst_dir (str) – Path to destination directory. Must exist and be empty.
  • config (TwinDBBackupConfig) – Tool configuration.
  • redo_only (bool) – True if the function has to do final apply of the redo log. For example, if you restore backup from a full copy it should be False. If you restore from incremental copy and you restore base full copy redo_only should be True.
  • xtrabackup_binary – path to xtrabackup binary.
  • xbstream_binary – Path to xbstream binary

If success, return True

Return type:


twindb_backup.restore.restore_from_mysql_incremental(stream, dst_dir, config, tmp_dir=None, xtrabackup_binary='/opt/twindb-backup/embedded/bin/xtrabackup', xbstream_binary='/opt/twindb-backup/embedded/bin/xbstream')[source]

Restore MySQL datadir from an incremental copy.

  • stream – Generator that provides backup copy
  • dst_dir (str) – Path to destination directory. Must exist and be empty.
  • config (TwinDBBackupConfig) – Tool configuration.
  • tmp_dir (str) – Path to temp dir
  • xtrabackup_binary – Path to xtrabackup binary.
  • xbstream_binary – Path to xbstream binary

If success, return True

Return type:


twindb_backup.restore.update_grastate(dst_dir, status, key)[source]

If xtrabackup_galera_info exists in the destination directory then parse it and generate grastate.dat file.

  • dst_dir (str) – Path to destination directory.
  • status (dict) – Backup status
  • key (str) – Backup name

twindb_backup.share module

Module that works with sharing backups

twindb_backup.share.share(twindb_config, s3_url)[source]

Function for generate make public file and get public url

  • twindb_config (TwinDBBackupConfig) – tool configuration
  • s3_url (str) – S3 url to file


twindb_backup.util module

Module with helper functions


Remove all files are directories in path

Parameters:path (str) – Path to directory to be emptied.

Check if a given directory is empty and exit if not.

Parameters:path (str) – path to directory

Backup copy includes hostname where the backup was taken from. The function extracts the hostname from the backup name.

Parameters:backup_copy (str) – Backup copy name.
Returns:Hostname where the backup was taken from.
Return type:str

Backup copy includes hostname where the backup was taken from. The function extracts the run type from the backup name.

Parameters:backup_copy (str) – Backup copy name.
Returns:Run type.
Return type:str

Kill child process

twindb_backup.util.mkdir_p(path, mode=511)[source]

Emulate mkdir -p. Create a directory named path with numeric mode mode. The default mode is 0777 (octal)

  • path (str) – Directory path.
  • mode (int) – Directory permissions. The default mode is 0777 (octal)

Start reading a root my.cnf file given in common paths and parse included files.

Parameters:common_paths (list) – list of my.cnf files to start parsing from.
Returns:list of all included my.cnf files
Return type:list

Normalize base64 key. See for details.

Parameters:coding – Encoded data
Returns:Normalized encoded data
twindb_backup.util.run_command(*args, **kwds)[source]

Run shell command locally

  • command (list) – Command to run
  • ok_non_zero (bool) – Don’t consider non-zero exit code as an error.

file object with stdout as generator to use with with


Splits a string of host and port separated by a semicolon.

Parameters:host_port – host or host:port. Allowed values are like or just
Returns:a tuple with host and port. If only address is specified it’ll return (address, None). If host_port is None it will return (None, None)
Return type:tuple

twindb_backup.verify module

Module that verify backup copies.


Removed options from config(besides MySQL 5.7.8)

twindb_backup.verify.verify_mysql_backup(twindb_config, dst_path, backup_copy, hostname=None)[source]

Restore mysql backup and measure time

  • hostname
  • backup_copy
  • dst_path
  • twindb_config (TwinDBBackupConfig) – tool configuration

Module contents

TwinDB Backup module.

The module is a core of twindb-backup tool. It includes backup and restore functionality. The module takes a backup from something defined in a source class and saves the backup copy in something defined in a destination class.

The source class inherits from BaseSource() from The source class must define get_stream() method that yields a file object that is used for next classes. Typical classes are FileSource() to backup files and directories, MySQLSource() to backup MySQL.

The destination class inherits from BaseDestination(). This is where you store backups. The destination class must define save() method that takes an input stream and saves it somewhere. Examples of the destination class are S3(), Ssh().

There are modifier classes. The modifier class sits in the middle between the source and the destination and does something with a stream before the stream is saved. The modifier class may save a local copy (KeepLocal()) or encrypt the stream or else. The modifier class inherits Modifier()

The backup process may be depicted as a chain of modifiers with the source in the head and the destination in the tail.

+--------+    +------------+    +------------+    +-------------+
| source | -- | modifier 1 | -- | modifier 2 | -- | destination |
+--------+    +------------+    +------------+    +-------------+
class twindb_backup.LessThanFilter(exclusive_maximum, name='')[source]

Bases: logging.Filter

Filters out log messages of a lower level.


Determine if the specified record is to be logged.

Is the specified record to be logged? Returns 0 for no, nonzero for yes. If deemed appropriate, the record may be modified in-place.

exception twindb_backup.TwinDBBackupError[source]

Bases: exceptions.Exception

Class for script errors

twindb_backup.delete_local_files(dir_backups, keep_copies)[source]

Deletes local backup copies based on given retention number.

  • dir_backups (str) – directory with backup copies
  • keep_copies (int) – how many to keep


twindb_backup.get_files_to_delete(all_files, keep_copies)[source]

If you give it a list of files and number of how many you’d like to keep the function will return files that need to be deleted

  • all_files (list) – list of strings
  • keep_copies (int) – number of copied to keep

list of strings (files) to delete

Return type:



Get timeout for a each run type - daily, hourly etc

Parameters:run_type (str) – Run type
Returns:Number of seconds the tool allowed to wait until other instances finish
Return type:int
twindb_backup.save_measures(start_time, end_time, log_path='/var/log/twindb-backup-measures.log')[source]

Save backup measures to log file

twindb_backup.setup_logging(logger, debug=False)[source]

Configures logging for the module