Developing Cloud Applications with Windows Azure Storage: Blobs

Navigating blob container hierarchies

As suggested earlier, a rough analogy of blob storage is your file system. Actually, when developing your cloud-deployable software, your file system may be used in some circumstances as an adequate on-premise substitute for blob storage (without a few of the advanced features such as snapshots and shared access signatures). You may even consider implementing a provider model in your software to facilitate this kind of convenient on-premise abstraction.

In your file system, files are placed within directories, and those directories are stored within other directories to create an extensive organizational hierarchy. All directories can be traversed back to a single root directory that houses the entire tree structure. Blob containers are like directories that live within the root directory of blob storage, but the analogy begins to weaken at this point because blob containers may not be embedded within other blob containers. If that were the end of the story, you would be left with a very flat file system. Fortunately, this is not the case. The Windows Azure client library provides support for accessing blobs by using a simulation of a nested file system, thus allowing directory-style navigation over delimiters used in your blob names, such as the slash character.

To see how to navigate flat blob storage as if it were hierarchical, you’ll first create a set of blobs in a container that uses a path delimiter. In this case, you will use the default delimiter of a slash (/).

The following code creates a container called demo and then populates this container with eight blobs named FileA, FileB, Dir1/FileC, Dir1/FileD, Dir1/Dir2/FileE, Dir3/FileF, Dir3/FileG, and Dir4/FileH by uploading an empty string as the content of each blob. The UseFlatBlobListing property of an instance of the BlobRequestOptions class is used as a parameter to control whether the container is navigated. You set this property to true when you want each blob in the container to be navigated without regard to the delimiter, and to false when you want navigation to behave as if the container were a file system style directory.

public static void DirectoryHierarchies(CloudStorageAccount account) {
    Console.Clear();
    CloudBlobClient client = account.CreateCloudBlobClient();
    Console.WriteLine("Default delimiter={0}", client.DefaultDelimiter /* settable */);
    Console.WriteLine();

    // Create the virtual directory
    const String virtualDirName = "demo";
    CloudBlobContainer virtualDir =
    client.GetContainerReference(virtualDirName).EnsureExists(true);

    // Create some file entries under the virtual directory
    String[] virtualFiles = new String[] {
                            "FileA", "FileB", // Avoid  $&+,/:=?@ in blob names
                            "Dir1/FileC", "Dir1/FileD", "Dir1/Dir2/FileE",
                            "Dir3/FileF", "Dir3/FileG",
                            "Dir4/FileH"
                    };
    foreach (String file in virtualFiles) {
        virtualDir.GetBlockBlobReference("Root/" + file).UploadText(String.Empty);
    }

    // Show the blobs in the virtual directory container
    ShowContainerBlobs(virtualDir);   // Same as UseFlatBlobListing = false
    Console.WriteLine();
    ShowContainerBlobs(virtualDir, true);

    // CloudBlobDirectory (derived from IListBlobItem) is for traversing
    // and accessing blobs with names structured in a directory hierarchy.
    CloudBlobDirectory root = virtualDir.GetDirectoryReference("Root");
    WalkBlobDirHierarchy(root, 0);

    // Show just the blobs under Dir1
    Console.WriteLine();
    String subdir = virtualDir.Name + "/Root/Dir1/";
    foreach (var file in client.ListBlobs(subdir))
        Console.WriteLine(file.Uri);
}

private static void ShowContainerBlobs(CloudBlobContainer container,
     Boolean useFlatBlobListing = false, BlobListingDetails details = BlobListingDetails.None,
          BlobRequestOptions options = null, OperationContext operationContext = null) {
    Console.WriteLine("Container: " + container.Name);
    for (BlobResultSegment brs = null; brs.HasMore(); ) {
        brs = container.ListBlobsSegmented(null, useFlatBlobListing, details, 1000,
              brs.SafeContinuationToken(), options, operationContext);
        foreach (var blob in brs.Results) Console.WriteLine("   " + blob.Uri);
    }
}

private static void WalkBlobDirHierarchy(CloudBlobDirectory dir, Int32 indent) {
    // Get all the entries in the root directory
    IListBlobItem[] entries = dir.ListBlobs().ToArray();
    String spaces = new String(' ', indent * 3);

    Console.WriteLine(spaces + dir.Prefix + " entries:");
    foreach (var entry in entries.OfType<ICloudBlob>())
        Console.WriteLine(spaces + "   " + entry.Name);

     foreach (var entry in entries.OfType<CloudBlobDirectory>()) {
        String[] segments = entry.Uri.Segments;
        CloudBlobDirectory subdir = dir.GetSubdirectoryReference(segments[segments.Length - 1]);
        WalkBlobDirHierarchy(subdir, indent + 1); // Recursive call
    }
}

private static void ShowContainer(CloudBlobContainer container, Boolean showBlobs) {
    Console.WriteLine("Blob container={0}", container);

    BlobContainerPermissions permissions = container.GetPermissions();
    String[] meanings = new String[] {
                            "no public access",
                            "anonymous clients can read container & blob data",
                            "anonymous readers can read blob data only"
                    };
    Console.WriteLine("Container's public access={0} ({1})",
       permissions.PublicAccess, meanings[(Int32)permissions.PublicAccess]);

    // Show collection of access policies; each consists of name & SharedAccesssPolicy
    // A SharedAccesssBlobPolicy contains:
    //    SharedAccessPermissions enum (None, Read, Write, Delete, List) &
    //    SharedAccessStartTime/SharedAccessExpireTime
    Console.WriteLine("   Shared access policies:");
    foreach (var policy in permissions.SharedAccessPolicies) {
        Console.WriteLine("   {0}={1}", policy.Key, policy.Value);
    }

    container.FetchAttributes();
    Console.WriteLine("   Attributes: Name={0}, Uri={1}", container.Name, container.Uri);
    Console.WriteLine("   Properties: LastModified={0}, ETag={1},",
         container.Properties.LastModified, container.Properties.ETag);
    ShowMetadata(container.Metadata);

    if (showBlobs)
        foreach (ICloudBlob blob in container.ListBlobs())
            ShowBlob(blob);
}

private static void ShowBlob(ICloudBlob blob) {
    // A blob has attributes: Uri, Snapshot DateTime?, Properties & Metadata
    // The CloudBlob Uri/SnapshotTime/Properties/Metadata properties return these
    // You can set the properties & metadata; not the Uri or snapshot time
    Console.WriteLine("Blob Uri={0}, Snapshot time={1}", blob.Uri, blob.SnapshotTime);
    BlobProperties bp = blob.Properties;
    Console.WriteLine("BlobType={0}, CacheControl={1}, Encoding={2}, Language={3},
         MD5={4}, ContentType={5}, LastModified={6}, Length={7}, ETag={8}",
       bp.BlobType, bp.CacheControl, bp.ContentEncoding, bp.ContentLanguage,
            bp.ContentMD5, bp.ContentType, bp.LastModified, bp.Length, bp.ETag);
    ShowMetadata(blob.Metadata);
}

private static void ShowMetadata(IDictionary<String, String> metadata) {
    foreach (var kvp in metadata)
        Console.WriteLine("{0}={1}", kvp.Key, kvp.Value);
}

Executing this code produces the following results.

Default delimiter=/

Container: demo, UseFlatBlobListing: False
   http://azureinsiders.blob.core.windows.net/demo/Dir1/
   http://azureinsiders.blob.core.windows.net/demo/Dir3/
   http://azureinsiders.blob.core.windows.net/demo/Dir4/
   http://azureinsiders.blob.core.windows.net/demo/FileA
   http://azureinsiders.blob.core.windows.net/demo/FileB

Container: demo, UseFlatBlobListing: True
   http://azureinsiders.blob.core.windows.net/demo/Dir1/Dir2/FileE
   http://azureinsiders.blob.core.windows.net/demo/Dir1/FileC
   http://azureinsiders.blob.core.windows.net/demo/Dir1/FileD
   http://azureinsiders.blob.core.windows.net/demo/Dir3/FileF
   http://azureinsiders.blob.core.windows.net/demo/Dir3/FileG
   http://azureinsiders.blob.core.windows.net/demo/Dir4/FileH
   http://azureinsiders.blob.core.windows.net/demo/FileA
   http://azureinsiders.blob.core.windows.net/demo/FileB

demo entries:
   FileA
   FileB
   Dir1 entries:
      FileC
      FileD
      Dir2 entries:
         FileE
   Dir3 entries:
      FileF
      FileG
   Dir4 entries:
      FileH

http://azureinsiders.blob.core.windows.net/demo/Dir1/Dir2/FileE
http://azureinsiders.blob.core.windows.net/demo/Dir1/FileC
http://azureinsiders.blob.core.windows.net/demo/Dir1/FileD

After printing the delimiter being used, the blob container named demo is iterated by using the UseFlatBlobListing property of an instance of BlobRequestOptions set to false. This option suppresses the iterator’s descent into the blob names beyond the first occurrence of the delimiter character, providing you with a high-level listing of all of the simulated directories in the root of the container. The next section of code performs the same operation, with the UseFlatBlobListing property of an instance of BlobRequestOptions set to true. You’ll see more on this class later in this chapter. With this option set, the container’s ListBlobsSegmented method recursively returns the subdirectories in each directory (using the segmented technique described in Chapter 4, “Accessing Windows Azure data storage”), providing a flattened view of the blobs in the container.

Occasionally, because of business requirements, you may have to traverse all of the blobs in a container as if they were files in a file system tree. The next section of code calls the WalkBlobDirHierarchy routine, which recursively calls itself to list the contents of each segment of the delimited blob names. The CloudBlobDirectory class (which derives from CloudBlob) provides the abstraction of a blob directory. You traverse the entire tree by calling the GetSubdirectory method on each directory to retrieve a list of subdirectories and then use that list to recursively call back into the WalkBlobDirHierarchy routine.

In some situations, it may be desirable to locate all blobs that are contained in a single simulated directory structure. This can be accomplished using the ListBlobsWithPrefix method of your instance of CloudBlobClient, as shown in the preceding section of the code.