Category Archives: #SQL

SCOM: Agents not submitting performance data

Sometimes you might have a situation where all of your agents are showing as healthy in the console but when you try and draw a performance report data is missing.

The below SQL query which has been developed by my colleague Gerrie Louw will identify any agent that has not submitted performance data in the past 4 hours. It does so by checking the following performance counters:

Processor > % Processor Time
LogicalDisk > % Free Space > C:
Memory > Available MBytes

Note: You will probably have to change the DisplayName_ and IsVirtualNode for your OperationsManager database.

if object_id(‘tempdb..#temptable’) IS NOT NULL
DROP TABLE #temptable

SELECT     distinct bmetarget.Name into #temptable
FROM        OperationsManager.dbo.BaseManagedEntity AS BMESource WITH (nolock) INNER JOIN
OperationsManager.dbo.Relationship AS R WITH (nolock) ON
R.SourceEntityId = BMESource.BaseManagedEntityId INNER JOIN
OperationsManager.dbo.BaseManagedEntity AS BMETarget WITH (nolock) ON
R.TargetEntityId = BMETarget.BaseManagedEntityId inner join mtv_computer d on bmetarget.name=d.[DisplayName_55270A70_AC47_C853_C617_236B0CFF9B4C]
and d.IsVirtualNode_E817D034_02E8_294C_3509_01CA25481689 is null
WHERE     (bmetarget.fullname like ‘Microsoft.Windows.Computer%’)

if object_id(‘tempdb..#healthstate’) IS NOT NULL
DROP TABLE #healthstate

select  megv.path, megv.ismanaged, megv.isavailable, megv.healthstate into #healthstate
from managedentitygenericview as megv with (nolock) inner join managedtypeview as mtv with (nolock)
on megv.monitoringclassid=mtv.id
where mtv.name =’microsoft.systemcenter.agent’

if object_id(‘tempdb..#perfcpudata’) IS NOT NULL
DROP TABLE #perfcpudata

select Path, ‘CPU’ as ‘Cat’ into #perfcpudata
from PerformanceDataAllView pdv with (NOLOCK)
inner join PerformanceCounterView pcv on pdv.performancesourceinternalid = pcv.performancesourceinternalid
inner join BaseManagedEntity bme on pcv.ManagedEntityId = bme.BaseManagedEntityId
where (TimeSampled < GETUTCDATE() AND TimeSampled > DATEADD(MINUTE,-240, GETUTCDATE()))
and objectname =’Processor’ and countername=’% Processor Time’

if object_id(‘tempdb..#perfmemdata’) IS NOT NULL
DROP TABLE #perfmemdata

select Path,’Memory’ as ‘Cat’ into #perfmemdata
from PerformanceDataAllView pdv with (NOLOCK)
inner join PerformanceCounterView pcv on pdv.performancesourceinternalid = pcv.performancesourceinternalid
inner join BaseManagedEntity bme on pcv.ManagedEntityId = bme.BaseManagedEntityId
where (TimeSampled < GETUTCDATE() AND TimeSampled > DATEADD(MINUTE,-240, GETUTCDATE()))
and objectname =’Memory’ and countername=’Available MBytes’

if object_id(‘tempdb..#perfdiskdata’) IS NOT NULL
DROP TABLE #perfdiskdata

select Path,’Disk’ as ‘Cat’ into #perfdiskdata
from PerformanceDataAllView pdv with (NOLOCK)
inner join PerformanceCounterView pcv on pdv.performancesourceinternalid = pcv.performancesourceinternalid
inner join BaseManagedEntity bme on pcv.ManagedEntityId = bme.BaseManagedEntityId
where (TimeSampled < GETUTCDATE() AND TimeSampled > DATEADD(MINUTE,-240, GETUTCDATE()))
and objectname =’LogicalDisk’ and countername=’% Free Space’ and instancename=’C:’

if object_id(‘tempdb..#temptable1′) IS NOT NULL
DROP TABLE #temptable1
create table #temptable1 (
name nvarchar(250),
cat nvarchar(20),
val nvarchar(2)
)
insert into #temptable1
select name, ‘CPU’ as ‘cat’, ’1′ as ‘val’
from #temptable where name not in
(select path from #perfcpudata)

insert into #temptable1
select name, ‘Memory’ as ‘cat’, ’1′ as ‘val’
from #temptable where name not in
(select path from #perfmemdata)

insert into #temptable1
select name, ‘Disk’ as ‘cat’, ’1′ as ‘val’
from #temptable where name not in
(select path from #perfdiskdata)

if object_id(‘tempdb..#output’) IS NOT NULL
DROP TABLE #output
create table #output (
name nvarchar(250),
cpu nvarchar(2),
memory nvarchar(2),
disk nvarchar(2)
)

insert into #output
select distinct tt.name ,’0′,’0′,’0′
from #temptable1 as tt, #healthstate as hs
where tt.name=hs.path collate SQL_Latin1_General_CP1_CI_AS
and hs.isavailable=1
and hs.ismanaged=1
and hs.healthstate is not null

update #output set cpu=1 where #output.name in (select name from #temptable1 where #temptable1.name=#output.name and #temptable1.cat=’CPU’)
update #output set memory=1 where #output.name in (select name from #temptable1 where #temptable1.name=#output.name and #temptable1.cat=’Memory’)
update #output set disk=1 where #output.name in (select name from #temptable1 where #temptable1.name=#output.name and #temptable1.cat=’Disk’)

select * from #output

You can use this query to build a report such as the one sampled below:

No Perf Report edited

Loading

SCOM: Maintenence mode investigation query

An oldie but a goodie, the below SQL script will help with investigating if a server was placed into maintenance mode and if so by whom. Just change %computer:% to the name of the specific device you are investigating and the StartDateTime as required.

USE OperationsManagerDW
go
SELECT DISTINCT
MngE.DisplayName AS ServerName, MM.StartDateTime, MM.EndDateTime, MM.PlannedMaintenanceInd AS [PlannedMaintenance_1Yes-0No],
MM.DWLastModifiedDateTime, MMH.Comment, MMH.UserId
FROM OperationsManagerDW.dbo.vMaintenanceMode AS MM INNER JOIN
OperationsManagerDW.dbo.vMaintenanceModeHistory AS MMH ON MMH.MaintenanceModeRowId = MM.MaintenanceModeRowId INNER JOIN
OperationsManagerDW.dbo.vManagedEntity AS MngE ON MngE.ManagedEntityRowId = MM.ManagedEntityRowId
WHERE     
MngE.FullName LIKE ‘%computer:%’ AND (MM.StartDateTime > ‘2014-03-01’)
ORDER BY MM.DWLastModifiedDateTime

Loading

SCOM: Upgrade to Operations Manager 2012 R2 may result in Data Warehouse synchronization failures

Brian McDermott highlighted an issue to watch out for when upgrading to SCOM 2012 R2 where you may get Data Warehouse synchronization failure errors after the upgrade.

The article can be found here with solid reasoning as to the cause and solution:

Please note that the Event ID 31565 noted above is a very generic error and you should only run the SQL below if the description identifies that it is the problem with the TFSWorkItemID column.

Error below:

Log Name:      Operations Manager
Source:        Health Service Modules
Date:
Event ID:      31565
Task Category: Data Warehouse
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      OMMS.domain.com
Description:
Failed to deploy Data Warehouse component. The operation will be retried.
Exception ‘DeploymentException’: Failed to perform Data Warehouse component deployment operation: Install; Component: DataSet, Id: ‘0d698dff-9b7e-24d1-8a74-4657b86a59f8′, Management Pack Version-dependent Id: ’29a3dd22-8645-bae5-e255-9b56bf0b12a8′; Target: DataSet, Id: ’23ee52b1-51fb-469b-ab18-e6b4be37ab35’. Batch ordinal: 3; Exception: Sql execution failed. Error 207, Level 16, State 1, Procedure vAlertDetail, Line 18, Message: Invalid column name ‘TfsWorkItemId’.

This issue can be fixed with the below SQL query, as always BACKUP your databases and proceed at your own risk:
USE OperationsManagerDW
 
DECLARE @GuidString NVARCHAR(50)
SELECT @GuidString = DatasetId FROM StandardDataset
WHERE SchemaName = ‘Alert’
 
— update all tables that were already created
DECLARE
   @StandardDatasetTableMapRowId int
  ,@Statement nvarchar(max)
  ,@SchemaName sysname
  ,@TableNameSuffix sysname
  ,@BaseTableName sysname
  ,@FullTableName sysname
 
SET @StandardDatasetTableMapRowId = 0
 
WHILE EXISTS (SELECT *
              FROM StandardDatasetTableMap tm
              WHERE (tm.StandardDatasetTableMapRowId > @StandardDatasetTableMapRowId)
                AND (tm.DatasetId = @GuidString)
             )
BEGIN
  SELECT TOP 1
     @StandardDatasetTableMapRowId = tm.StandardDatasetTableMapRowId
    ,@SchemaName = sd.SchemaName
    ,@TableNameSuffix = tm.TableNameSuffix
    ,@BaseTableName = sdas.BaseTableName
  FROM StandardDatasetTableMap tm
          JOIN StandardDataset sd ON (tm.DatasetId = sd.DatasetId)
          JOIN StandardDatasetAggregationStorage sdas ON (sdas.DatasetId = tm.DatasetId) AND (sdas.AggregationTypeId = tm.AggregationTypeId)
  WHERE (tm.StandardDatasetTableMapRowId > @StandardDatasetTableMapRowId)
    AND (tm.DatasetId = @GUIDString)
    AND (sdas.TableTag = ‘detail’)
    AND (sdas.DependentTableInd = 1)
  ORDER BY tm.StandardDatasetTableMapRowId
 
  SET @FullTableName = @BaseTableName + ‘_’ + @TableNameSuffix
 
  IF NOT EXISTS (SELECT * FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = @FullTableName AND TABLE_SCHEMA = @SchemaName
    AND COLUMN_NAME = N’TfsWorkItemId’)
  BEGIN
    SET @Statement = ‘ALTER TABLE ‘ + QUOTENAME(@SchemaName) + ‘.’ + QUOTENAME(@FullTableName) + ‘ ADD TfsWorkItemId nvarchar(256) NULL’
    EXECUTE (@Statement)
  END
 
  IF NOT EXISTS (SELECT * FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = @FullTableName AND TABLE_SCHEMA = @SchemaName
    AND COLUMN_NAME = N’TfsWorkItemOwner’)
  BEGIN
    SET @Statement = ‘ALTER TABLE ‘ + QUOTENAME(@SchemaName) + ‘.’ + QUOTENAME(@FullTableName) + ‘ ADD TfsWorkItemOwner nvarchar(256) NULL’
    EXECUTE (@Statement)
  END
END
 
— alter cover views
EXEC StandardDatasetBuildCoverView@GUIDString, 0
GO

 

Loading

SCOM: Windows 2008 also running monitors for Windows 2003, Orphaned class

Now here’s something you certainly don;t see every day. It all started when I was asked to investigate a flood of memory alerts for a particular server at one of my customers. When I opened Health Explorer I noticed the following:

The server was running the monitors for Windows 2008 and Windows 2003.

HExpl double monitors
As it turns out the server had been recently re-installed from 2003 to 2008 with the same name, without the agent being uninstalled or being removed from the console. This caused a bit of confusion in the back-end. A quick look at the Windows Server 2003 Operating System Inventory showed another server which was “Upgraded” i in the same fashion:

2003 OS edt
What’s happened here is the class for Windows Server 2003 Operating System is still being loaded by the agent, and this is causing all of the related rules and monitors to load as well. In the past when I’ve come across this particular issue I’ve been able to solve it with the remove-disabledmonitoringobject powershell cmdlet. 
All that you need to do is override the discovery rule in question to false for your object (In this case “Discover Windows Server 2003 Operating System) and then open OpsMgr Shell and run remove-disabledmonitoringobject. After a short delay the offending objects are removed.

However in this case the above did not work, eventually I deleted the agent from the console, waited for grooming to run (you can force it if you are in a hurry), cleared the local agent cache and then approved the agent. Now only the correct objects are being discovered.

Loading

SCOM: Updated SQL Management Pack 6.4.1.0

An updated SQL management pack has been release which fixes the issues being experienced with version 6.4.0.0

SQL Server Management Pack version 6.4.1.0 includes the following changes:

  • Fixed CPU Utilization Monitor
  • Fixed SQL Server seed discovery for WoW64 environments
  • Alert severity of Average Wait Time Monitor was changed to Warning, added consecutive sampling to reduce noise, threshold was changed to 250
  • Alert severity of SQL Re-Compilation monitor was changed to Warning, threshold was changed to 25. The monitor was disabled by default.
  • Minor fixes

Kevin Holman has written a great article on the correct way to configure this management pack which is definitely worth the read.

Loading

SCOM: New SQL Management Pack 6.4.0.0

As of 2013-09-06 a new management pack has been released for SQL:

2013-09-12 Note: Issues are currently being reported with with this version of the SQL MP: Error ID 11052 “

  • Module was unable to convert parameter to a double value Original parameter:
    ‘$Data/Property[@Name=’CPUUsage’]$’ Parameter after $Data replacement: ‘-1.#IND’
    Error: 0x80020005 Details”

Download it here

Below is the new feature information from the site / guide, there are several new rules and monitors which offer a deeper look into your SQL environments:

      The Management pack for SQL Server provides the capabilities for Operations Manager 2007 R2 and Operations Manager 2012 to discover SQL Server 2005, 2008, 2008 R2, and SQL Server 2012. It monitors SQL Server components such as database engine instances, databases, and SQL Server agents.
      The monitoring provided by this management pack includes performance, availability, and configuration monitoring, performance data collection, and default thresholds. You can integrate the monitoring of SQL Server components into your service-oriented monitoring scenarios.
      In addition to health monitoring capabilities, this management pack includes dashboard views, diagram views and extensive knowledge with embedded inline tasks, and views that enable near real-time diagnosis and resolution of detected issues.

Important Prerequisite Notes:

      Clusters: In order to ensure that all monitoring works correctly for clustered instances of SQL Server ensure that your OpsMgr agents on the physical nodes of the cluster are running either OpsMgr 2007 R2 or OpsMgr 2007 SP1 with the most recent cumulative update for OpsMgr 2007 SP1 applied or OpsMgr 2012.

Feature Summary:

      The following list gives an overview of the features of the SQL Server management pack. Refer to the SQL Server management pack guide for more detail.

New features:

      • New Dashboard for SQL Server 2012 DB
      • New Monitors and Rules – only for SQL 2008 and SQL 2012
        • Collect DB Active Connections count
        • Collect DB Active Requests count
        • Collect DB Active Sessions count
        • Collect DB Active Transactions count
        • Collect DB Engine Thread count
        • Thread Count monitor
        • Transaction Log Free Space (%) monitor
        • Transaction Log Free Space (%) collection
        • Collect DB Engine CPU Utilization (%)
        • CPU Utilization (%) monitor for DB engine
        • Buffer Cache Hit Ratio monitor
        • Collect DB Engine Page Life Expectancy (s)
        • Page Life Expectancy monitor
        • Collect DB Disk Read Latency (ms)
        • Collect DB Disk Write Latency (ms)
        • Disk Read Latency monitor
        • Disk Write Latency monitor
        • Collect DB Transactions per second count
        • Collect DB Engine Average Wait Time (ms)
        • Average Wait Time monitor
        • Collect DB Engine Stolen Server Memory (MB)
        • Stolen Server Memory monitor
        • Collect DB Allocated Free Space (MB)
        • Collect DB Used Space (MB)
        • Collect DB Disk Free Space (MB)
        • SQL Re-Compilation monitor
      • Run As configuration changes to support Low privilege for SQL Server 2012 Cluster

Additional features:

    • AlwaysOn Monitoring
      • Automatically discover and monitor availability groups, availability replicas, and availability databases for hundreds of computers.
      • Health roll-up from availability database to availability replicas.
      • Detailed knowledge with every critical health state to enable faster resolution to a problem.
    • Seamless integration with Policy based management (PBM)
      • Auto-discover custom PBM polices targeting AlwaysOn and database components.
      • Rollup of health of policy execution within the SQL monitoring pack under extended health.
    • Support for Mirroring and Replication Monitoring (only applicable to SQL Server 2008 and 2008 R2 version of management pack)
      • Discover mirroring databases, witness, and mirroring group.
      • Monitor database mirror state, database mirror witness state, and mirroring partners’ state.
      • Custom diagram view to visually represent the primary and the mirrored databases.
      • Approximately twenty rules to detect replication events.
    • Improved Freespace monitoring with mount point support
    • Support for Enterprise, Standard and Express editions of SQL Server 2005, 2008, 2008 R2, and 2012 and 32bit, 64bit and ia64 architectures.
    • Support for both simple and complex SQL Server configurations such as clustered installations, multiple instances and 32bit roles running on a 64bit OS. For full details on supported configurations refer to the guide included with the management pack.
    • Discovery and monitoring of SQL Server roles such as DB Engine, Reporting Services, Analysis Services, Integrations Services.
    • Discovery of SQL Server components such as databases, the SQL Agent and SQL jobs.
    • Views covering areas such as database free space, SQL Server related performance, SQL Server related alerts, and lists of the various SQL Server roles and components which are discovered and their related state.
    • Discovery and basic monitoring for SQL Server Reporting Services and Integration Services.
    • Reports for longer-term analysis of common problem areas related to SQL Server such as SQL Server lock analysis and top deadlocked databases, SQL Server service pack levels across discovered roles, user connection activity. Likewise the generic reports from the Microsoft Generic Report Library can be used against the roles and components discovered by the SQL MPs to review availability and performance across many systems and over longer periods of time.
    • Role and component specific tasks which provide access to common tools, triage information, or corrective actions without needing to leave the Operations Console in most cases.
    • Monitoring of databases covers database status, database free space, log shipping monitoring for both the source and destination, and transaction log free space.
    • Monitoring of key SQL Server related services.
    • Monitoring for persistent SPID blocking.
    • Monitoring of numerous SQL Server events and performance data points. Alerts bring the issue to your attention and provide knowledge on the impact and possible resolutions.
    • A low-privilege configuration for discovery and monitoring that eliminates the need for SQL Server sysadmin, dbo, db_owner, and box admin privileges

Loading

SQL 2008 – Relocating log files

I needed to move our Data Warehouse and ACS database log files to a new drive, the following is a relatively simple method to move a SQL log file to a new location without needing to detach your database.

ALTER DATABASE DBNAME SET SINGLE_USER WITH ROLLBACK IMMEDIATE;
ALTER DATABASE DBNAME SET OFFLINE;
ALTER DATABASE DBNAME MODIFY FILE ( Name = Log File Name, Filename = ‘Drive Letter:\Log file location.ldf‘);
At this point copy the log file to the new location
ALTER DATABASE DBNAME SET ONLINE;
ALTER DATABASE DBNAME SET MULTI_USER;

The Log file name and file location can be found in the properties of your database under the File Tab.
This method could also be used to move your database file.

Loading

SCOM issue discovering SQL Instances / Databases

 

There appears to be an issue when trying to discover certain SQL 2008 instances with SCOM. Which is related to MOF data not being registered in the WMI repository correctly when SQL is installed.

The error that you will see in your Operations Manager event log is:

DiscoverSQL2008DBEngineDiscovery.vbs : The Query ‘select * from SqlService where SQLServiceType =’1” did not return any valid instances. Please check to see if this is a valid WMI Query.. Invalid class

I’ve had success with the below solution:

From an elevated command prompt, browse to the folder “C:\Program Files (x86)\Microsoft SQL Server\100\Shared”

Run mofcomp sqlmgmproviderxpsp2up.mof

Restart your SCOM Agent on the server.

Loading

Moving your OperationsManager database – SCOM 2012

Recently I had to move our OperationsManager database to a different server as part of testing a disaster recovery plan. While the Microsoft procedure for this (http://technet.microsoft.com/en-us/library/hh278848.aspx) is quite in-depth there is something to be aware of if you upgraded your environment from SCOM 2007.

Step 8 will ask you you to update the table dbo. MT_Microsoft$SystemCenter$ManagementGroup in your OperationsManager database however if you upgraded this table does not exist. Instead you will need to update the old SCOM 2007 management group table dbo.MT_ManagementGroup.

Step below:

Update the operational database with the new database server name.

  1. Open SQL Server Management Studio.
  2. Expand Databases, OperationsManager, and Tables.
  3. Right-click dbo. MT_Microsoft$SystemCenter$ManagementGroup (or dbo.MT_ManagementGroup if you upgraded from SCOM 2007), and then click Edit Top 200 Rows.
  4. Change the value in the SQLServerName_6B1D1BE8_EBB4_B425_08DC_2385C5930B04 column to reflect the name of the new SQL Server-based computer.
  5. Save the change.

Loading