How to Sync Local Directories to Azure Blob Storage Using AzCopy

Azure Blob Storage is a cloud-based object storage solution provided by Microsoft Azure. It allows you to store unstructured data like text, images, and video. To synchronize your local directories with Azure Blob Storage, you can use a powerful tool called AzCopy, which is a command-line utility designed for high-performance data transfer.

What is AzCopy?

AzCopy is a cross-platform tool for managing Azure Storage. It allows you to upload, download, and manage blobs in Azure Storage accounts. AzCopy supports transferring data between your local machine and Azure Blob Storage, as well as between different storage accounts. This guide will walk you through the steps of syncing local directories to Azure Blob Storage using AzCopy.

Prerequisites

• Azure Subscription
• Azure Storage Account with Blob Storage enabled
• AzCopy tool installed on your machine (available for Windows, Linux, and macOS)
• Access to the local directory you want to sync
• A SAS (Shared Access Signature) token or an Azure Storage account key for authentication

Installing AzCopy

Before you can use AzCopy, you need to install it on your system. Here’s how to do it on various platforms:

Windows

To install AzCopy on Windows, download the MSI installer from the official Microsoft Azure website. After downloading, run the installer and follow the on-screen instructions.

Linux

On Linux, you can install AzCopy using the package manager. For example, on Ubuntu, use the following commands:

sudo apt-get update
sudo apt-get install wget
wget https://aka.ms/downloadazcopy-v10-linux
tar -xvf downloadazcopy-v10-linux
sudo ./install.sh

macOS

On macOS, you can use Homebrew to install AzCopy:

brew install azcopy

Authenticating AzCopy

Once AzCopy is installed, you must authenticate to Azure Storage. There are several methods available for authentication, but two of the most common are using a SAS token or an Azure Storage account key.

Using SAS Token

A SAS token grants restricted access to your Azure Storage resources. You can generate a SAS token from the Azure Portal. Once you have the SAS token, you can authenticate using the following command:

azcopy login --sas-token "Your_SAS_Token"

Using Account Key

Alternatively, you can authenticate with the Azure Storage account key. The command to do this is as follows:

azcopy login --account-name "Your_Account_Name" --account-key "Your_Account_Key"

Syncing Local Directories to Azure Blob Storage

Once you’ve authenticated, you can start syncing your local directory with Azure Blob Storage using the sync command. This command compares the contents of a local directory with a destination blob container and uploads any new or modified files.

Basic Sync Command

The basic syntax of the azcopy sync command is:

azcopy sync   --recursive

Where:

• is the path to the local directory you want to sync.
• is the URL to your Azure Blob container. The URL will be in the following format: https://.blob.core.windows.net/
• The --recursive flag tells AzCopy to include all subdirectories and files within the specified directory.

Syncing with Advanced Options

There are additional options you can use with the azcopy sync command. For example, you can exclude files based on patterns or only upload files that are newer than those already in the destination container.

Excluding Specific Files

To exclude specific files or patterns during the sync operation, use the --exclude-pattern option:

azcopy sync   --recursive --exclude-pattern "*.log"

Syncing Only Newer Files

To sync only files that are newer than the ones in the destination, use the --overwrite option with the ifSourceNewer value:

azcopy sync   --recursive --overwrite ifSourceNewer

Monitoring the Sync Process

During the sync process, AzCopy provides real-time status updates. You can monitor the progress of the sync operation directly in the terminal. AzCopy also logs errors and successes, so you can track which files were uploaded, skipped, or failed.

Syncing to Azure Blob Storage with Filters

In some cases, you may want to sync only specific types of files, such as images or documents. You can achieve this by using filters to target specific file extensions. For example, to sync only .jpg and .png files, you can use the following command:

azcopy sync   --recursive --include-pattern "*.jpg;*.png"

Conclusion

Syncing local directories to Azure Blob Storage with AzCopy is an efficient way to manage cloud storage. By using the azcopy sync command, you can quickly upload large volumes of files, with options for filtering, excluding specific files, and handling only newer versions of files. With AzCopy, you can automate the process of syncing data to the cloud, ensuring your local directories and Azure Blob Storage remain up to date.