Automating DB Snapshots at Amazon RDS

Amazon RDS is a relational database service by Amazon Web Services. RDS offers an easy to administer environment and supports many known relational database engines such as PostgreSQL, MySQL and MariaDB.

The database is a crucial part for a company and it is very important to maintain the integrity of the data.
For this reason, you should consider creating a good backup strategy from day one.

RDS provides two backup methods: Automated backups and user-initiated DB snapshots. Automated backups are initiated during the creation of an RDS instance. You set the backup window and the retention period for the backups and you’re ready.

Although automated backups seem attractive because they are easy to manage, they have some constraints.

  1. Retention: There is a 35-day retention period limit. After exceeding that limit, the snapshot is then deleted.

  2. Deleted database: If you accidentally delete a database for any reason, automated backups are going to be removed too.

  3. Disaster Recovery: Automated backups can only be restored from within the same region.
    If you have a DR strategy, you might want to move the snapshots between multiple regions.

These 3 constraints can be resolved with the DB snapshots.

  1. DB snapshots can be retained for as long as you wish.
  2. DB snapshots are not removed if you accidentally delete the database.
  3. DB snapshots can be moved from one region to another without any constraint.

Amazon Web Services gives us three ways to take manual DB Snapshots: the management console,
the awscli command line utility and the various Amazon SDKs.

Lets see how we can capture & delete DB snapshots from Amazon RDS with boto3, the Python SDK.
The benefit of using the SDK is that we can write extensible tools that fit our needs.

Source code:

Our utility comprises a wrapper of the boto’s RDS interface and uses click module which allows us to create simple command line utilities.

Lets go through the code to see some of the core calls to the RDS API.

Here’s how we connect to boto’s client for RDS.

boto3.client('rds')

When we run the create operation we create a basic naming convention for our snapshots.

snapshot = "{0}-{1}-{2}".format("mysnapshot", db_instance, timestamp) # mysnapshot-database-instance-1-YYYY-MM-DD

Then we call the API to create our snapshot:

create_db_snapshot(DBSnapshotIdentifier=snapshot, DBInstanceIdentifier=db_instance)

If we want to know the availability status of the snapshot we run describe_db_snapshots:

describe_db_snapshots(DBSnapshotIdentifier=snapshot)['DBSnapshots'][0]['Status']

We could also call the delete operation to remove a captured snapshot:

delete_db_snapshot(DBSnapshotIdentifier=snapshot)

If you have multiple RDS instances you might want to list them before you perform any action.
This is how we get a list with the available RDS instances:

describe_db_instances()['DBInstances']

Now that we have looked at what is going on under the hood, lets move onto some basic examples.

Get the help page with the available operations:

dbsnap.py --help

Output:

Usage: dbsnap.py [OPTIONS] COMMAND [ARGS]...

  dbsnap is a snapshot management tool for Amazon RDS.

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  create     Creates a new database snapshot
  delete     Deletes the specified snapshot
  instances  Returns the available database instances
Get a list with all your database instances:

dbsnap.py instances

Output:

Database Instances:
- database-instance-1
- database-instance-2
- database-instance-3
Create a new DB snapshot for database-instance-1:

dbsnap.py create --db-instance database-instance-1

Output:

Creating a new snapshot from database-instance-1 instance...
Snapshot status: available

If you check at your RDS dashboard now, you will see a new snapshot in the form of:
mysnapshot-database-instance-1-YYYY-MM-DD

Delete the DB snapshot that we created earlier:

dbsnap.py delete --db-snapshot mysnapshot-database-instance-1-YYYY-MM-DD

Output:

Snapshot: mysnapshot-database-instance-1-YYYY-MM-DD has been deleted

Based on the code and the examples you can extend the tool as you wish. You could add logging capabilities, run it daily to get DB snapshots for all your instances or add functionalities to delete the snapshots after a period of time.

It’s worth mentioning that during the backup window, I/O freeze may occur on Single AZ (Availability Zone)
database instances. You can avoid that freeze by enabling the Multi-AZ deployments which provides high availability.

To conclude, automated backups are very handy and if they are used in conjunction with automated DB snapshots,
they can give a very reliable and stable backup strategy.

I hope you enjoyed the post.

Further information:
* DB Instance Backups
* Best Practices for Amazon RDS
* Pinterest recently open-sourced a collection of its in-house MySQL utilities. It is a good source to get some ideas.

4 Comments

  1. Liam Duncan

    08 Dec 2015 12:03:31

    If you are not a company or business what would you want it for?

    • Pavlos

      10 Mar 2016 13:45:26

      RDS provides a flexible environment where you can manage replications and backups with just some clicks. If an individual does not want to maintain his own database infrastructure then RDS seems a good solution but it comes a small amount of fee depending of the usage.

  2. Marlon Olaya

    12 Feb 2016 19:07:31

    Don’t I need some api key, and api secret key for the connection to the rds?
    I mean, I am trying to connect to the rds module and only one of the 30 instances take a snapshot.

    Don’t I Need the api key, ans the api secret key?, How can I configure it in the boto3.client(‘rds’)
    Can I create a boto session and after use the session to create a client connection? Something like this:
    session = boto.session(apikey,apisecretkey,region)
    client = session.client(‘rds’)
    It is possible?

    • Pavlos

      10 Mar 2016 13:46:53

      Here is a guide that can help you configure your environment.

Leave a Comment

Stay tuned:

* we promise that we won´t spam you, never.