Design and Implement an Azure Storage Strategy

  • 3/11/2015
Contents

Objective 4.5: Monitor storage

Azure Storage has a built-in analytics feature called Azure Storage Analytics used for collecting metrics and logging storage request activity. You enable Storage Analytics Metrics to collect aggregate transaction and capacity data, and you enable Storage Analytics Logging to capture successful and failed request attempts to your storage account. This section covers how to enable monitoring and logging, control logging levels, set retention policies, and analyze the logs.

NOTE: Storage Analytics Availability

At the time of this writing, Storage Analytics is not available for Azure Files.

This objective covers how to:

  • Enable monitoring and logging
  • Set retention policies and logging levels
  • Analyze logs

Configuring storage metrics

Storage Analytics metrics provide insight into transactions and capacity for your storage accounts. You can think of them as the equivalent of Windows Performance Monitor counters. By default, storage metrics are not enabled, but you can enable them through the management portal, using Windows PowerShell, or by calling the management API directly.

When you configure storage metrics for a storage account, tables are generated to store the output of metrics collection. You determine the level of metrics collection for transactions and the retention level for each service—Blob, Table, and Queue.

Transaction metrics record request access to each service for the storage account. You specify the interval for metric collection (hourly or by minute). In addition, there are two levels of metrics collection:

  • Service level These metrics include aggregate statistics for all requests, aggregated at the specified interval. Even if no requests are made to the service, an aggregate entry is created for the interval, indicating no requests for that period.
  • API level These metrics record every request to each service only if a request is made within the hour interval.

NOTE: Metrics Collected

All requests are included in the metrics collected, including any requests made by Storage Analytics.

Capacity metrics are only recorded for the Blob service for the account. Metrics include total storage in bytes, the container count, and the object count (committed and uncommitted).

Table 4-1 summarizes the tables automatically created for the storage account when Storage Analytics metrics are enabled.

TABLE 4-1 Storage metrics tables

METRICS

TABLE NAMES

Hourly metrics

$MetricsHourPrimaryTransactionsBlob

$MetricsHourPrimaryTransactionsTable

$MetricsHourPrimaryTransactionsQueue

Minute metrics (cannot set through the management portal)

$MetricsMinutePrimaryTransactionsBlob

$MetricsMinutePrimaryTransactionsTable

$MetricsMinutePrimaryTransactionsQueue

Capacity (only for the Blob service)

$MetricsCapacityBlob

MORE INFO: Storage Analytics Metrics Table Schema

For additional details on the transaction and capacity metrics collected, see http://msdn.microsoft.com/en-us/library/azure/hh343264.aspx.

Retention can be configured for each service in the storage account. By default, Storage Analytics will not delete any metrics data. When the shared 20-terabyte limit is reached, new data cannot be written until space is freed. This limit is independent of the storage limit of the account. You can specify a retention period from 0 to 365 days. Metrics data is automatically deleted when the retention period is reached for the entry.

When metrics are disabled, existing metrics that have been collected are persisted up to their retention policy.

MORE INFO: Storage Metrics

For more information about enabling and working with storage metrics, see http://msdn.microsoft.com/en-us/library/azure/dn782843.aspx.

Configuring storage metrics and retention (existing portal)

To enable storage metrics and associated retention levels for Blob, Table, and Queue services in the existing management portal, follow these steps:

  1. Navigate to the Configure tab for your storage account in the management portal accessed via https://manage.windowsazure.com.
  2. If this storage account uses blobs, set the metrics level for blobs to Minimal. Set retention according to your retention policy.
  3. If this storage account uses tables, set the metrics level for tables to Minimal. Set retention according to your retention policy.
  4. If this storage account uses queues, set the metrics level for queues to Minimal. Set retention according to your retention policy.
  5. Click Save to commit the settings.

NOTE: Choosing a Metrics Level

Minimal metrics yield enough information to provide a picture of the overall usage and health of the storage account services. Verbose metrics provide more insight at the API level, allowing for deeper analysis of activities and issues, which is helpful for troubleshooting.

Configuring storage metrics and retention (Preview portal)

To enable storage metrics and associated retention levels for Blob, Table, and Queue services in the Preview portal, follow these steps:

  1. Navigate to the management portal accessed via https://portal.azure.com.
  2. Click Browse on the command bar.
  3. Select Storage from the Filter By drop-down list.
  4. Select your storage account from the list on the Storage blade.
  5. Scroll down to the Usage section, and click the Capacity Past Week check box.
  6. On the Metric blade, click Diagnostics on the command bar.
  7. Click the On button under Status. This shows the options for metrics and logging.
  8. If this storage account uses blobs, select Blob Aggregate Metrics to enable service level metrics. Select Blob Per API Metrics for API level metrics.
  9. If this storage account uses tables, select Table Aggregate Metrics to enable service level metrics. Select Table Per API Metrics for API level metrics.
  10. If this storage account uses queues, select Queue Aggregate Metrics to enable service level metrics. Select Queue Per API Metrics for API level metrics.
  11. Provide a value for retention according to your retention policy. Through the Preview portal, this will apply to all services. It will also apply to Storage Analytics Logging if that is enabled. Select one of the available retention settings from the drop-down list, or enter a number from 0 to 365.

Configuring storage metrics and retention using Windows PowerShell

To enable storage metrics and associated retention levels using Windows PowerShell, use the Set-AzureStorageMetricsProperty cmdlet.

To enable service level metrics collected by minute for blobs, tables, and queues with unlimited retention, run the following:

Set-AzureStorageServiceMetricsProperty –MetricsType Minute –ServiceType Blob –

MetricsLevel Service –RetentionDays 0
Set-AzureStorageServiceMetricsProperty –MetricsType Minute –ServiceType Table –
MetricsLevel Service –RetentionDays 0
Set-AzureStorageServiceMetricsProperty –MetricsType Minute –ServiceType Queue –
MetricsLevel Service –RetentionDays 0

To enable service and API level metrics collected hourly for blobs, tables, and queues with 90 days of retention, run the following:

Set-AzureStorageServiceMetricsProperty –MetricsType Hour –ServiceType Blob –MetricsLevel
ServiceAndApi –RetentionDays 90
Set-AzureStorageServiceMetricsProperty –MetricsType Hour –ServiceType Table –MetricsLevel ServiceAndApi –RetentionDays 90
Set-AzureStorageServiceMetricsProperty –MetricsType Hour –ServiceType Queue –
MetricsLevel ServiceAndApi –RetentionDays 90

To disable the collection of metrics, run the following:

Set-AzureStorageServiceMetricsProperty–ServiceType Blob –MetricsLevel None
Set-AzureStorageServiceMetricsProperty–ServiceType Table –MetricsLevel None
Set-AzureStorageServiceMetricsProperty–ServiceType Queue –MetricsLevel None

Analyzing storage metrics

Storage Analytics metrics are collected in tables as discussed in the previous section. You can access the tables directly to analyze metrics, but you can also review metrics in both Azure management portals. This section discusses various ways to access metrics and review or analyze them.

MORE INFO: Storage Monitoring, Diagnosing, and Troubleshooting

For more details on how to work with storage metrics and logs, see http://azure.microsoft.com/en-us/documentation/articles/storage-monitoring-diagnosing-troubleshooting.

Monitoring metrics (existing portal)

To monitor metrics in the existing portal, complete the following steps:

  1. Navigate to the Monitor tab for your storage account in the management portal accessed via https://manage.windowsazure.com.
  2. Click Add Metric to choose metrics to monitor in the management portal.
  3. In the Choose Metrics dialog box, select from the list of metrics for blobs, tables, or queues.
  4. Click the check mark to commit the settings.

  5. You may choose up to six metrics to show in the monitoring graph, as shown in Figure 4-8.

    Figure 4-8

    FIGURE 4-8 Monitoring metrics from the existing management portal

  6. Select absolute or relative display of metrics from the appropriate drop-down list (RELATIVE is indicated in Figure 4-8).
  7. Select the time range to display from the appropriate drop-down list (6 HOURS is indicated in Figure 4-8). Select 6 hours, 24 hours, or 7 days.

Monitoring metrics (Preview portal)

At the time of this writing, the Preview portal features for monitoring metrics is limited to some predefined metrics, including total requests, total egress, average latency, and availability (see Figure 4-9). Click each box to see a Metric blade that provides additional detail.

Figure 4-9

FIGURE 4-9 Monitoring overview from the Preview portal

To monitor the metrics available in the Preview portal, complete the following steps:

  1. Navigate to the management portal accessed via https://portal.azure.com.
  2. Click Browse on the command bar.
  3. Select Storage from the Filter By drop-down list.
  4. Select your storage account from the list on the Storage blade.
  5. Scroll down to the Monitor section, and view the monitoring boxes summarizing statistics. You’ll see TotalRequests, TotalEgress, AverageE2ELatency, and AvailabilityToday by default.
  6. Click each metric box to view additional details for each metric. You’ll see metrics for blobs, tables, and queues if all three metrics are being collected.

NOTE: Customizing the Monitoring Blade

You can customize which boxes appear in the Monitoring area of the Preview portal, and you can adjust the size of each box to control how much detail is shown at a glance without drilling into the metrics blade.

Configuring Storage Analytics Logging

Storage Analytics Logging provides details about successful and failed requests to each storage service that has activity across the account’s blobs, tables, and queues. By default, storage logging is not enabled, but you can enable it through the management portal, by using Windows PowerShell, or by calling the management API directly.

When you configure Storage Analytics Logging for a storage account, a blob container named $logs is automatically created to store the output of the logs. You choose which services you want to log for the storage account. You can log any or all of the Blob, Table, or Queue services. You can also choose which type of requests to log: read, write, or delete. Logs are created only for those services that have activity, so you will not be charged if you enable logging for a service that has no requests. The logs are stored as block blobs as requests are logged and are periodically committed so that they are available as blobs.

NOTE: Deleting the Log Container

After Storage Analytics has been enabled, the log container cannot be deleted; however, the contents of the log container can be deleted.

Retention can be configured for each service in the storage account. By default, Storage Analytics will not delete any logging data. When the shared 20-terabyte limit is reached, new data cannot be written until space is freed. This limit is independent of the storage limit of the account. You can specify a retention period from 0 to 365 days. Logging data is automatically deleted when the retention period is reached for the entry.

NOTE: Duplicate Logs

Duplicate log entries may be present within the same hour. You can use the RequestId and operation number to uniquely identify an entry to filter duplicates.

MORE INFO: Storage Logging

For more information about enabling and working with Azure storage logging, see http://msdn.microsoft.com/en-us/library/azure/dn782843.aspx and http://msdn.microsoft.com/en-us/library/azure/hh343262.aspx.

Configuring storage logging and retention (existing portal)

To enable storage logging and associated retention levels for Blob, Table, and Queue services in the existing portal, follow these steps:

  1. Navigate to the Configure tab for your storage account in the management portal accessed via https://manage.windowsazure.com. Scroll to the logging section.
  2. If this storage account uses blobs, select Read, Write, and Delete requests to log all activity. Set retention according to your retention policy.
  3. If this storage account uses tables, select Read, Write, and Delete requests to log all activity. Set retention according to your retention policy.
  4. If this storage account uses queues, check Read, Write, and Delete requests to log all activity. Set retention according to your retention policy.
  5. Click Save to commit the settings.

Configuring storage logging and retention (Preview portal)

To enable storage logging and associated retention levels for Blob, Table, and Queue services in the Preview portal, follow these steps:

  1. Navigate to the management portal accessed via https://portal.azure.com.
  2. Click Browse on the command bar.
  3. Select Storage from the Filter By drop-down list.
  4. Select your storage account from the list on the Storage blade.
  5. Scroll down to the Usage section, and select the Capacity Past Week check box.
  6. On the Metric blade, click Diagnostics on the command bar.
  7. Click the On button under Status. This shows the options for enabling monitoring features.
  8. If this storage account uses blobs, select Blob Logs to log all activity.
  9. If this storage account uses tables, select Table Logs to log all activity.
  10. If this storage account uses queues, select Queue Logs to log all activity.
  11. Provide a value for retention according to your retention policy. Through the Preview portal, this will apply to all services. It will also apply to Storage Analytics Metrics if that is enabled. Select one of the available retention settings from the drop-down list, or enter a number from 0 to 365.

NOTE: Controlling Logged Activities

From the Preview portal, when you enable or disable logging for each service, you enable read, write, and delete logging. To log only specific activities, use Windows PowerShell cmdlets.

Configuring storage logging and retention using Windows PowerShell

To enable storage metrics and associated retention levels using Windows PowerShell, use the Set-AzureStorageServiceLoggingProperty cmdlet.

To enable logging for read, write, and delete actions with retention of 30 days, run the following:

Set-AzureStorageServiceLoggingProperty –ServiceType Blob –LoggingOperations
read,write,delete –RetentionDays 30
Set-AzureStorageServiceLoggingProperty –ServiceType Table –LoggingOperations
read,write,delete –RetentionDays 30
Set-AzureStorageServiceLoggingProperty –ServiceType Queue –LoggingOperations
read,write,delete –RetentionDays 30

To disable the collection of metrics, run the following:

Set-AzureStorageServiceLoggingProperty –ServiceType Blob –LoggingOperations none
Set-AzureStorageServiceLoggingProperty –ServiceType Table –LoggingOperations none
Set-AzureStorageServiceLoggingProperty –ServiceType Queue –LoggingOperations none

Enabling client-side logging

You can enable client-side logging using Microsoft Azure storage libraries to log activity from client applications to your storage accounts. For information on the .NET Storage Client Library, see http://msdn.microsoft.com/en-us/library/azure/dn782839.aspx. For information on the Storage SDK for Java, see http://msdn.microsoft.com/en-us/library/azure/dn782844.aspx.

Analyzing storage logs

Logs are stored as block blobs in delimited text format. When you access the container, you can download logs for review and analysis using any tool compatible with that format. Within the logs, you’ll find entries for authenticated and anonymous requests, as listed in Table 4-3.

TABLE 4-3 Authenticated and anonymous logs

Request Type

Logged Requests

Authenticated requests
  • Successful requests
  • Failed requests such as timeouts, authorization, throttling issues, and other errors
  • Requests that use an SAS
  • Requests for analytics data

Anonymous requests
  • Successful requests
  • Server errors
  • Timeouts for client or server
  • Failed GET requests with error code 304 (Not Modified)

Logs include status messages and operation logs. Status message columns include those shown in Table 4-4. Some status messages are also reported with storage metrics data. There are many operation logs for the Blob, Table, and Queue services.

MORE INFO: Status Messages and Operation Logs

For a detailed list of specific logs and log format specifics, see http://msdn.microsoft.com/en-us/library/azure/hh343260.aspx and http://msdn.microsoft.com/en-us/library/hh343259.aspx.

TABLE 4-4 Information included in logged status messages

Column

Description

Status Message

Indicates a value for the type of status message, indicating type of success or failure

Description

Describes the status, including any HTTP verbs or status codes

Billable

Indicates whether the request was billable

Availability

Indicates whether the request is included in the availability calculation for storage metrics

Finding your logs

When storage logging is configured, log data is saved to blobs in the $logs container created for your storage account. You can’t see this container by listing containers, but you can navigate directly to the container to access, view, or download the logs.

To view analytics logs produced for a storage account, do the following:

  1. Using a storage browsing tool, navigate to the $logs container within the storage account you have enabled Storage Analytics Logging for using this convention: https://<accountname>.blob.core.windows.net/$logs.
  2. View the list of log files with the convention <servicetype>/YYYY/MM/DD/HHMM/<counter>.log.
  3. Select the log file you want to review, and download it using the storage browsing tool.

MORE INFO: Log Metadata

The blob name for each log file does not provide an indication of the time range for the logs. You can search this information in the blob metadata using storage browsing tools or Windows PowerShell.

Accessing logs with Windows PowerShell

Using Windows PowerShell, you can access logs with the Get-AzureStorageBlob cmdlet and then filter logs by filename and metadata. The following example illustrates how to filter a list of write logs for Blob storage entries on a particular date during the ninth hour:

Get-AzureStorageBlob -Container '$logs' |
where {
  $_.Name -match 'blob/2014/12/01/09' -and
  $_.ICloudBlob.Metadata.LogType -match 'write'
} |
foreach {
  "{0} {1} {2} {3}" –f $_.Name,
  $_.ICloudBlob.Metadata.StartTime,
  $_.ICloudBlob.Metadata.EndTime,
  $_.ICloudBlob.Metadata.LogType
}

Downloading logs

To review and analyze your logs, first download them to your local machine. You can do this with storage browsing tools, programmatically, or with AzCopy.

MORE INFO: AZCopy

AzCopy is part of the Azure SDK. You can download the latest version directly from http://aka.ms/AzCopy.

Viewing logs with Microsoft Excel

Storage logs are recorded in a delimited format so that you can use any compatible tool to view logs. To view logs data in Excel, follow these steps:

  1. Open Excel, and on the Data menu, click From Text.
  2. Find the log file and click Import.
  3. During import, select Delimited format. Select Semicolon as the only delimiter, and Double-Quote (“) as the text qualifier.

Analyzing logs

After you load your logs into a viewer like Excel, you can analyze and gather information such as the following:

  • Number of requests from a specific IP range
  • Which tables or containers are being accessed and the frequency of those requests
  • Which user issued a request, in particular, any requests of concern
  • Slow requests
  • How many times a particular blob is being accessed with an SAS URL
  • Details to assist in investigating network errors

MORE INFO: Log Analysis

You can run the Azure HDInsight Log Analysis Toolkit (LAT) for a deeper analysis of your storage logs. For more information, see https://hadoopsdk.codeplex.com/releases/view/117906.

THOUGHT EXPERIMENT: Proactive and reactive diagnostics

experiment.jpg

In this thought experiment, apply what you’ve learned about this objective. You can find answers to these questions in the “Answers” section at the end of this chapter.

Your application is now live, and you are planning how you will monitor the overall health of your storage account resources.

  1. What features would you use to give you a proactive, early warning of problems with storage services?
  2. What features could you use to diagnose an issue retroactively so that you can perform root cause analysis and fix the problem?

Objective summary

  • Storage Analytics metrics provide the equivalent of Windows Performance Monitor counters for storage services.
  • You can determine which services to collect metrics for (Blob, Table, or Queue), whether to collect metrics for the service or API level, and whether to collect metrics by the minute or hour.
  • Capacity metrics are only applicable to the Blob service.
  • Storage Analytics Logging provides details about the success or failure of requests to storage services.
  • Storage logs are stored in blob services for the account, in the $logs container for the service.
  • You can specify up to 365 days for retention of storage metrics or logs, or you can set retention to 0 to retain metrics indefinitely. Metrics and logs are removed automatically from storage when the retention period expires.
  • Storage metrics can be viewed in the management portal. Storage logs can be downloaded and viewed in a reporting tool such as Excel.

Objective review

Answer the following questions to test your knowledge of the information in this objective. You can find the answers to these questions and explanations of why each answer choice is correct or incorrect in the “Answers” section at the end of this chapter.

  1. Which statements are true of Storage Analytics Metrics? (Choose all that apply.)

    1. Capacity metrics are recorded only for blobs.
    2. You can set hourly or by minute metrics through the management portal.
    3. By default, metrics are retained for one year.
    4. If you disable metrics, existing metrics are deleted from storage.

  2. Which statements are true of Storage Analytics Logging? (Choose all that apply.)

    1. Logs are stored in the same storage account where they are enabled and are measured as part of your storage quota.
    2. Logs can have duplicate entries.
    3. Logs cannot be deleted.
    4. You can log all read, write, and delete requests to blobs, queues, and tables in a storage account.

  3. Which of the following are captured by Storage Analytics Logging? (Choose all that apply.)

    1. Successful requests for authenticated calls only
    2. Failed requests for authenticated calls only
    3. Server errors
    4. Requests using SAS URIs
Save to your account

This chapter is from the book