Let's rokk! [Tudor Cret's blog]

April 18, 2011

Azure Diagnostics

Filed under: Windows Azure — Tudor Cret @ 2:49 pm
Tags: ,

Diagnostics and monitoring for services and applications that don’t run in the cloud doesn’t require a special attention since you have physical  access to the production server, more or less, it depends on how you’ve decided to host them. But in the cloud there are several challenges with diagnostics like:

  • Many instances
  • They move around
  • Massive amount of data
  • Can’t remote desktop in
  • No remote tools (yet)

So Microsoft have implemented a monitoring agent (MonAgentHost.exe) which runs on each instance in the cloud. The agent is started automatically by default. The listener is wired up in the app/web.config, like any other TraceListener. More you need to define a storage account connection string for this listener. In a nutshell it works in 5 steps:

image

  1. 1.Role instance starts and by default the monitoring agent is started too.
  2. 2.Diagnostic Monitor Starts
  3. 3.Monitor must be configured at start time or remotely any time using service configuration file.
  4. 4.Monitor starts to buffer data locally. The user might set a quota, too.
  5. 5.User initiates transfer to storage. The transfer can be scheduled  or on demand. I recommend a scheduled transfer. In this way diagnostics storage is up to date with live locally data.
  6. image

 

Below is the table with items that can be monitored in the cloud, if they are activated by default or not and the storage destination type.

Data Source

Default

Destination

Trace Logs

Enabled

Azure Table

Diagnostic Infrastructure Logs

Enabled

Azure Table

IIS Logs

Enabled

Blob

Performance Counters

Disabled

Azure Table

Windows Event Logs

Disabled

Azure Table

IIS Failed Request Logs

Disabled

Blob

Crash Dumps

Disabled

Blob

Arbitrary Files

Disabled

Blob

Implementing Azure Diagnostics:

First Diagnostic Agent is loaded as an Azure module the ServiceDefinition.csdef:

image

Then module expects a connection string:

image

Attention, a production connection string must be https! Like this:

<Setting name="Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString" value="DefaultEndpointsProtocol=https;
AccountName=YOURACCOUNT;AccountKey=YOURKEY" />

The common pattern to configure diagnostics suppose:

  • Get Config
    • From default
    • Current running
  • Make a change to the config
    • If changed within the instance it will affect only that instance. Don’t forget to start the agent immediately
    • If changed from outside for all roles then
      • change the central file
      • agent notices a change and reloads
      • affects all instances of the role
  • Start the Diagnostics agent with the new configuration

The code:

Method 1:

  • I’ve set up a transfer to the storage every one minute
  • Also I’ve added “System” and “Application” transfer from Event Viewer
Code Snippet
  1. //—————————————-method 1—————————-//
  2.                 //get config
  3.                 CloudStorageAccount storageAcc = CloudStorageAccount.Parse(RoleEnvironment.GetConfigurationSettingValue("Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString"));
  4.                 RoleInstanceDiagnosticManager ridm = storageAcc.CreateRoleInstanceDiagnosticManager(RoleEnvironment.DeploymentId,
  5.                     RoleEnvironment.CurrentRoleInstance.Role.Name, RoleEnvironment.CurrentRoleInstance.Id);
  6.                 DiagnosticMonitorConfiguration dmc = ridm.GetCurrentConfiguration();
  7.  
  8.                 //change config
  9.                 // Transfer logs to storage every minute
  10.                 dmc.Logs.ScheduledTransferPeriod = tsTenSeconds;
  11.                 dmc.DiagnosticInfrastructureLogs.ScheduledTransferPeriod = tsTenSeconds;
  12.                 dmc.Directories.ScheduledTransferPeriod = tsTenSeconds;
  13.                 dmc.WindowsEventLog.ScheduledTransferPeriod = tsTenSeconds;
  14.                                 
  15.                 dmc.WindowsEventLog.DataSources.Add("System!*");
  16.                 dmc.WindowsEventLog.DataSources.Add("Application!*");
  17.                 dmc.Logs.ScheduledTransferLogLevelFilter = LogLevel.Information;
  18.  
  19.                 ridm.SetCurrentConfiguration(dmc);
  20.                 //—————————————-end method 1—————————-//

Method 2:

  • I’ve loaded the default configuration and I’ve set up a scheduled transfer
Code Snippet
  1. //—————————————-method 2—————————-//
  2.                 // Start up the diagnostic manager with the given configuration
  3.                 DiagnosticMonitorConfiguration dmc = DiagnosticMonitor.GetDefaultInitialConfiguration();
  4.                 dmc.Logs.ScheduledTransferPeriod = tsTenSeconds;
  5.                 // Transfer verbose, critical, etc. logs
  6.                 dmc.Logs.ScheduledTransferLogLevelFilter = LogLevel.Verbose;
  7.                 DiagnosticMonitor.Start("Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString", dmc);
  8.                 //—————————————-end method 2—————————-//

Method 3:

  • I’ve created a custom listener that writes log entries in a dedicated Azure table
Code Snippet
  1. //—————————————-method 3—————————-//
  2.                 CloudStorageAccount storageAcc = CloudStorageAccount.Parse(RoleEnvironment.GetConfigurationSettingValue("Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString"));
  3.                 CloudTableClient tableStorage = storageAcc.CreateCloudTableClient();
  4.                 tableStorage.CreateTableIfNotExist(TableStorageTraceListener.DIAGNOSTICS_TABLE);
  5.                 AzureDiagnostics.TableStorageTraceListener listener =
  6.                  new AzureDiagnostics.TableStorageTraceListener("Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString")
  7.                  {
  8.                      Name = "TableStorageTraceListener"
  9.                  };
  10.                 System.Diagnostics.Trace.Listeners.Add(listener);
  11.                 System.Diagnostics.Trace.AutoFlush = true;
  12.                 Trace.Listeners.Add(new DiagnosticMonitorTraceListener());
  13.                 //—————————————-end method 3—————————-//

Full code available here.

Visualizing the data

You might use Azure Storage Explorer to explore diagnostics storage account or you can use Cerebrata’s Azure Diagnostics Manager which is a dedicated tool for diagnostics management. It will display a more friendly UI than the Storage Explorer:

image

Advertisements

Blog at WordPress.com.