Here is an interesting issue I came across. I did not encounter this too often but still, it’s very interesting and it was very fun to troubleshoot! 😀
In case you get the same error I suggest opening a case with Microsoft Support because it will be easier. Still, let me tell the story just for fun.
Some Alerts were not appearing in Reports and when we checked these were definitely not written to the Data Warehouse database – the AlertStage table for example where it ends up in the first place when Alerts are synched from the operational database to the data warehouse database. Ok then, so first thing to check is the Operations Manager event log on all Management Servers in case you have OM 2012 because any Management Server from the All Management Server Resource Pool can get the role of Data Warehouse Synchronization Server *OR* if you are on OM 2007 still, then *only* the Root Management Server will have this role.
Interestingly enough, we were seeing error events 31551 in the Operations Manager event log on the Server which had the Data Warehouse Synchronization Server role:
- please keep in mind that the error event ID (31551) is pretty general and it is not the event ID that matters, it is its description that tells us if we have the same issue or not
Failed to store data in the Data Warehouse. The operation will be retried.
Exception ‘InvalidOperationException’: The given value of type String from the data source cannot be converted to type nvarchar of the specified target column.
So what does this actually mean?!
Well the way that the Alert data is inserted into the Data Warehouse when synchronized from the operational database is by using SQL Bulk Insert because of the amount of data needed to be inserted. Well when Bulk Insert is used the class used will check the values of the data being inserted against the database tables field types and limitations. It will perform a check in this case so that all the values of an Alert (Item) will “fit” into the fields of the AlertStage table of the data warehouse database. If any of the fields (like Alert Title) are bigger than the MAX value of the table (AlertStage) then it will fail with that error message. The AlertStage Title field has NVARCHAR(255) as maximum length and if we have an Alert which has a title bigger than 255, then we will encounter this issue – as of course any other field which does not fit – like the CustomProperty fields of an Alert.
Here is a screenshot as FYI for the field types and definitions of the AlertStage table from the data warehouse database:
So now the question is … how do we figure out *which* of the Alerts has a longer string *than* *which* property (field) ?! Well because we are doing a Bulk Insert here from the .NET SQL Class we cannot see this by tracing the SQL Server there … it never ends up trying to insert this into SQL because it fails locally in the code on the Management Server which is trying to do this action. Right … so how to figure this one out?!
Debugging my friends, the answer to everything, hehe! 😀
Ok so we need to look at … what process here? Well as you know (or will find out now) the Management Servers as well as the Agents have the “core” service called HealthService.exe. This process (service) however, does *NOT* do the heavy-lifting. It will spawn MonitoringHost.exe processes which will do the work for it after loading the necessary modules (DLL’s) in them which are needed for the specific tasks they are assigned to do.
So which MonitoringHost.exe process will it be which runs the Data Warehouse Synchronization workflow? Well this would have to be a MonitoringHost.exe process which is running under the Data Warehouse Writer Account would it not? 😉
- in case you have the same account used for more roles or something you can use Process Explorer to check out the loaded DLLs in the MonitoringHost.exe processes and find out which has the Microsoft.EnterpriseManagement.HealthService.Modules.DataWarehouse.dll DLL loaded
Because of the fact that the process is *NOT* crashing when throwing this Exception – as it is handled – we cannot simply get a memory dump of that process. So we either attach the debugger (WinDbg) live to the environment *OR* we can use another nifty tool which is called Time Travel Tracing to get the “dynamic” memory dump of that process so that we can “step” through the process execution. We could also use a tool like DebugDiag which can be set to write a memory dump of a process on a custom .NET Exception like in this example.
Either way, whatever method we will use here, we end up in the “same” scenario where we need to set up a breakpoint of the Exception we are interested in (if not using DebugDiag which writes the memory dump on exactly that Exception). So I will continue with this further on with the example of attaching to the MonitoringHost.exe process in question directly in the live environment.
Let’s start shall we? 😀 We are in the debugger now and let’s make sure we have loaded the correct SOS .NET Debugger Extension version: .cordll -ve -u -l
- make sure the correct version of SOS is loaded: http://msdn.microsoft.com/en-us/library/windows/hardware/ff540665(v=vs.85).aspx
Now we can continue by creating the breakpoint to stop when hitting the exception we are interested in which in this case is System.InvalidOperationException: !StopOnException -create System.InvalidOperationException 1
And then we enter g in the debugger so that the process continues execution until we hit the breakpoint (Exception). When it stops and the debugger breaks in, we use !PrintException to see the exception details:
0:008> !PrintException
Exception object: 0000000201f1dc38
Exception type: System.InvalidOperationException
Message: The given value of type String from the data source cannot be converted to type nvarchar of the specified target column.
InnerException: System.InvalidOperationException, Use !PrintException 0000000201f1d6d8 to see more.
StackTrace (generated):
SP IP Function
00000000075FC7A0 000007FEF338D5A8 system_data_ni!System.Data.SqlClient.SqlBulkCopy.ConvertValue(System.Object, System.Data.SqlClient._SqlMetaData)+0x19ec38
00000000075FED10 000007FEF31EDD39 system_data_ni!System.Data.SqlClient.SqlBulkCopy.WriteToServerInternal()+0x989
00000000075FEE00 000007FEF31EE263 system_data_ni!System.Data.SqlClient.SqlBulkCopy.WriteRowSourceToServer(Int32)+0x463
00000000075FEED0 000007FEF31EE945 system_data_ni!System.Data.SqlClient.SqlBulkCopy.WriteToServer(System.Data.IDataReader)+0x125
00000000075FEF40 000007FF0018DA89 microsoft_enterprisemanagement_datawarehouse_dataaccess!Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.Commands.DataWarehouseSqlBulkInsertCommand.Execute()+0x99
StackTraceString: <none>
HResult: 80131509
As we can see we have a nested Exception here, so let’s have a look at this as well:
0:008> !PrintException 0000000201f1d6d8
Exception object: 0000000201f1d6d8
Exception type: System.InvalidOperationException
Message: String or binary data would be truncated.
InnerException: <none>
StackTrace (generated):
SP IP Function
00000000075FEBB0 000007FEF31EEF5A system_data_ni!System.Data.SqlClient.SqlBulkCopy.ConvertValue(System.Object, System.Data.SqlClient._SqlMetaData)+0x5ea
StackTraceString: <none>
HResult: 80131509
Ok, so this is interning – now how do we find out which table field is affected here? It is going to be a System.Data.SqlClient._SqlMetaData object and should be in the Managed Heap here of this Thread we stopped on. Thus we can use !DumpStackObjects to see all the objects:
0:008> !DumpStackObjects
OS Thread Id: 0x3cb4 (8)
RSP/REG Object Name
[…SNIPPED…]
00000000075FA218 00000001c01beca0 System.Data.SqlClient.TdsParserStateObject
00000000075FA370 00000001c01beb90 System.Data.SqlClient.TdsParser
00000000075FA490 0000000201f1dc38 System.InvalidOperationException
00000000075FA8B0 0000000201f12240 System.Data.SqlClient._SqlMetaData
00000000075FA8C8 0000000201f1d400 System.Int32[]
00000000075FA8F8 0000000201f1d2c0 System.Object[] (System.Object[])
00000000075FABD0 0000000201ee0948 System.Data.SqlClient.SqlBulkCopy
00000000075FABF0 0000000201ee0be8 System.Data.SqlClient.SqlBulkCopyColumnMappingCollection
00000000075FAE00 0000000201ee0948 System.Data.SqlClient.SqlBulkCopy
00000000075FAE20 0000000201ee0be8 System.Data.SqlClient.SqlBulkCopyColumnMappingCollection
00000000075FB128 0000000201f1dc38 System.InvalidOperationException
00000000075FB410 00000001bfe40488 System.String
00000000075FB4D0 0000000201ee0948 System.Data.SqlClient.SqlBulkCopy
00000000075FB4F0 0000000201ee0be8 System.Data.SqlClient.SqlBulkCopyColumnMappingCollection
00000000075FB880 0000000201f1eaf0 System.Environment+ResourceHelper+GetResourceStringUserData
00000000075FBA70 0000000201f1eb68 System.Runtime.CompilerServices.RuntimeHelpers+CleanupCode
00000000075FBA78 0000000201f1eaf0 System.Environment+ResourceHelper+GetResourceStringUserData
00000000075FBAA0 0000000201f1eb68 System.Runtime.CompilerServices.RuntimeHelpers+CleanupCode
00000000075FBAB0 0000000201f1eb68 System.Runtime.CompilerServices.RuntimeHelpers+CleanupCode
00000000075FBB08 0000000201f1eb28 System.Runtime.CompilerServices.RuntimeHelpers+TryCode
00000000075FBB10 0000000201f1eaf0 System.Environment+ResourceHelper+GetResourceStringUserData
[…SNIPPED…]
Ok, from that big list, let’s grab the object ID of the object we are interested in (System.Data.SqlClient._SqlMetaData) – there will be more entries, but it should have the same Object ID: !DumpObj 0000000201f12240
0:008> !DumpObj 0000000201f12240
Name: System.Data.SqlClient._SqlMetaData
MethodTable: 000007fef2d98b90
EEClass: 000007fef2c36e60
Size: 224(0xe0) bytes
File: C:\Windows\Microsoft.Net\assembly\GAC_64\System.Data\v4.0_4.0.0.0__b77a5c561934e089\System.Data.dll
Fields:
MT Field Offset Type VT Attr Value Name
000007fef2d91c50 4001bc9 80 System.Int32 1 instance 12 type
000007fef880c158 4001bca 8c System.Byte 1 instance 231 tdsType
000007fef880c158 4001bcb 8d System.Byte 1 instance 255 precision
000007fef880c158 4001bcc 8e System.Byte 1 instance 255 scale
000007fef880c7d8 4001bcd 84 System.Int32 1 instance 512 length
000007fef2d925a0 4001bce 8 …ient.SqlCollation 0 instance 0000000201f139c0 collation
000007fef880c7d8 4001bcf 88 System.Int32 1 instance 1251 codePage
0000000000000000 4001bd0 10 0 instance 000000017fe6eb28 encoding
000007fef880d608 4001bd1 8f System.Boolean 1 instance 1 isNullable
000007fef880d608 4001bd2 90 System.Boolean 1 instance 0 isMultiValued
000007fef88068f0 4001bd3 18 System.String 0 instance 0000000000000000 udtDatabaseName
000007fef88068f0 4001bd4 20 System.String 0 instance 0000000000000000 udtSchemaName
000007fef88068f0 4001bd5 28 System.String 0 instance 0000000000000000 udtTypeName
000007fef88068f0 4001bd6 30 System.String 0 instance 0000000000000000 udtAssemblyQualifiedName
000007fef8808278 4001bd7 38 System.Type 0 instance 0000000000000000 udtType
000007fef88068f0 4001bd8 40 System.String 0 instance 0000000000000000 xmlSchemaCollectionDatabase
000007fef88068f0 4001bd9 48 System.String 0 instance 0000000000000000 xmlSchemaCollectionOwningSchema
000007fef88068f0 4001bda 50 System.String 0 instance 0000000000000000 xmlSchemaCollectionName
000007fef2d928b8 4001bdb 58 …qlClient.MetaType 0 instance 000000017fe701c0 metaType
000007fef88068f0 4001bdc 60 System.String 0 instance 0000000000000000 structuredTypeDatabaseName
000007fef88068f0 4001bdd 68 System.String 0 instance 0000000000000000 structuredTypeSchemaName
000007fef88068f0 4001bde 70 System.String 0 instance 0000000000000000 structuredTypeName
0000000000000000 4001bdf 78 0 instance 0000000000000000 structuredFields
000007fef88068f0 4001be0 98 System.String 0 instance 0000000201f139f0 column
000007fef88068f0 4001be1 a0 System.String 0 instance 0000000000000000 baseColumn
000007fef32c3cb0 4001be2 b0 …ultiPartTableName 1 instance 0000000201f122f0 multiPartTableName
000007fef880c7d8 4001be3 94 System.Int32 1 instance 25 ordinal
000007fef880c158 4001be4 91 System.Byte 1 instance 2 updatability
000007fef880c158 4001be5 a8 System.Byte 1 instance 0 tableNum
000007fef880d608 4001be6 a9 System.Boolean 1 instance 0 isDifferentName
000007fef880d608 4001be7 aa System.Boolean 1 instance 0 isKey
000007fef880d608 4001be8 ab System.Boolean 1 instance 0 isHidden
000007fef880d608 4001be9 ac System.Boolean 1 instance 0 isExpression
000007fef880d608 4001bea ad System.Boolean 1 instance 0 isIdentity
000007fef880d608 4001beb ae System.Boolean 1 instance 0 isColumnSet
000007fef880c158 4001bec af System.Byte 1 instance 0 op
000007fef88142d0 4001bed 92 System.UInt16 1 instance 0 operand
Ok, so using the column field of this object we can see that it is the CustomProperty9 column which we know has a max 255 character limitation – here we need to dump the pointer address of the object + the offset where the column table is found:
0:008> !DumpObj poi(0000000201f12240+98)
Name: System.String
MethodTable: 000007fef88068f0
EEClass: 000007fef838ed78
Size: 50(0x32) bytes
File: C:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
String: CustomField9
Fields:
MT Field Offset Type VT Attr Value Name
000007fef880c7d8 4000103 8 System.Int32 1 instance 12 m_stringLength
000007fef880b318 4000104 c System.Char 1 instance 43 m_firstChar
000007fef88068f0 4000105 10 System.String 0 shared static Empty
>> Domain:Value 00000000003e9470:00000001bfe40488 <<
So now we need to get a list of Alerts which the process is trying to insert and check the CustomProperty9 of all of them and find which is bigger than 255. We need to dump the object which is having all the alerts which is System.Data.SqlClient.SqlBulkCopy – just like before !DumpStackObjects and look for the object address:
0:008> !DumpStackObjects
OS Thread Id: 0x3cb4 (8)
RSP/REG Object Name
00000000075F5908 0000000201f1dc38 System.InvalidOperationException
00000000075F59A8 0000000201f1dc38 System.InvalidOperationException
00000000075F5A30 0000000201ee0080 Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.DataWarehouseCommandExecutionContext
00000000075F5A38 0000000201f1dc38 System.InvalidOperationException
00000000075F5A48 0000000201ee0018 Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.DataWarehouseSingleCommand+ExecutionStep
00000000075F5A50 0000000201ee0098 System.Collections.Generic.Dictionary`2[[Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.DataWarehouseConnectionDescriptor, Microsoft.EnterpriseManagement.DataWarehouse.DataAccess],[Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.DataWarehouseConnection, Microsoft.EnterpriseManagement.DataWarehouse.DataAccess]]
00000000075F5C90 0000000201f1dc38 System.InvalidOperationException
00000000075F60B0 00000002000b13d8 Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.Commands.DataWarehouseSqlBulkInsertCommand
00000000075F60C8 0000000201ee0018 Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.DataWarehouseSingleCommand+ExecutionStep
00000000075F60D0 0000000201ee0098 System.Collections.Generic.Dictionary`2[[Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.DataWarehouseConnectionDescriptor, Microsoft.EnterpriseManagement.DataWarehouse.DataAccess],[Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.DataWarehouseConnection, Microsoft.EnterpriseManagement.DataWarehouse.DataAccess]]
00000000075F63D0 0000000201ee0080 Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.DataWarehouseCommandExecutionContext
00000000075F63E8 0000000201ee0018 Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.DataWarehouseSingleCommand+ExecutionStep
00000000075F63F0 0000000201ee0098 System.Collections.Generic.Dictionary`2[[Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.DataWarehouseConnectionDescriptor, Microsoft.EnterpriseManagement.DataWarehouse.DataAccess],[Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.DataWarehouseConnection, Microsoft.EnterpriseManagement.DataWarehouse.DataAccess]]
00000000075F6600 0000000201ee0080 Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.DataWarehouseCommandExecutionContext
00000000075F6618 0000000201ee0018 Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.DataWarehouseSingleCommand+ExecutionStep
00000000075F6620 0000000201ee0098 System.Collections.Generic.Dictionary`2[[Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.DataWarehouseConnectionDescriptor, Microsoft.EnterpriseManagement.DataWarehouse.DataAccess],[Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.DataWarehouseConnection, Microsoft.EnterpriseManagement.DataWarehouse.DataAccess]]
00000000075F6928 0000000201f1dc38 System.InvalidOperationException
00000000075F6CD0 0000000201ee0080 Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.DataWarehouseCommandExecutionContext
00000000075F6CE8 0000000201ee0018 Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.DataWarehouseSingleCommand+ExecutionStep
00000000075F6CF0 0000000201ee0098 System.Collections.Generic.Dictionary`2[[Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.DataWarehouseConnectionDescriptor, Microsoft.EnterpriseManagement.DataWarehouse.DataAccess],[Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.DataWarehouseConnection, Microsoft.EnterpriseManagement.DataWarehouse.DataAccess]]
00000000075F7640 0000000201f12240 System.Data.SqlClient._SqlMetaData
00000000075F7968 0000000201f1dc38 System.InvalidOperationException
00000000075F7B10 0000000201ee0948 System.Data.SqlClient.SqlBulkCopy
00000000075F7B40 0000000201f1dc38 System.InvalidOperationException
00000000075F7B58 000000014007fa40 System.RuntimeType
[…SNIPPED…]
Ok now let’s have a look at the list shall we – dumping the object first as before using !DumpObj and the object address:
0:008> !DumpObj 0000000201ee0948
<Note: this object has an invalid CLASS field>
Name: System.Data.SqlClient.SqlBulkCopy
MethodTable: 000007fef32c3868
EEClass: 000007fef2c4eae0
Size: 176(0xb0) bytes
File: C:\Windows\Microsoft.Net\assembly\GAC_64\System.Data\v4.0_4.0.0.0__b77a5c561934e089\System.Data.dll
Fields:
MT Field Offset Type VT Attr Value Name
000007fef880c7d8 40016e9 78 System.Int32 1 instance 0 _batchSize
000007fef880d608 40016ea a0 System.Boolean 1 instance 0 _ownConnection
000007fef32a4370 40016eb 7c System.Int32 1 instance 0 _copyOptions
000007fef880c7d8 40016ec 80 System.Int32 1 instance 30 _timeout
000007fef88068f0 40016ed 8 System.String 0 instance 000000013fe4ea28 _destinationTableName
000007fef880c7d8 40016ee 84 System.Int32 1 instance 0 _rowsCopied
000007fef880c7d8 40016ef 88 System.Int32 1 instance 0 _notifyAfter
000007fef880c7d8 40016f0 8c System.Int32 1 instance 0 _rowsUntilNotification
000007fef880d608 40016f1 a1 System.Boolean 1 instance 32 _insideRowsCopiedEvent
000007fef8805a48 40016f2 10 System.Object 0 instance 00000002000b4c10 _rowSource
000007fef2d91038 40016f3 18 …ent.SqlDataReader 0 instance 0000000000000000 _SqlDataReaderRowSource
000007fef32c3a48 40016f4 20 …MappingCollection 0 instance 0000000201ee09f8 _columnMappings
000007fef32c3a48 40016f5 28 …MappingCollection 0 instance 0000000201ee0be8 _localColumnMappings
000007fef2d8fba8 40016f6 30 …ent.SqlConnection 0 instance 0000000201ee0610 _connection
000007fef32c3b70 40016f7 38 …nt.SqlTransaction 0 instance 0000000000000000 _internalTransaction
000007fef32c3b70 40016f8 40 …nt.SqlTransaction 0 instance 0000000000000000 _externalTransaction
000007fef32a4058 40016f9 90 System.Int32 1 instance 1 _rowSourceType
000007fef2d9b880 40016fa 48 System.Data.DataRow 0 instance 74042e00b98a6449 _currentRow
000007fef880c7d8 40016fb 94 System.Int32 1 instance -1738182535 _currentRowLength
000007fef3290ab0 40016fc 98 System.Int32 1 instance 1765434373 _rowState
000007fef880f240 40016fd 50 …tions.IEnumerator 0 instance 693a640598657079 _rowEnumerator
000007fef2d973e8 40016fe 58 …lClient.TdsParser 0 instance 00000001c01beb90 _parser
000007fef2d97ac8 40016ff 60 …ParserStateObject 0 instance 0000000000000000 _stateObj
000007fef8810158 4001700 68 …ections.ArrayList 0 instance 0000000201f18ca8 _sortedColumnMappings
000007fef32c3630 4001701 70 …opiedEventHandler 0 instance 4c4d582f31303032 _rowsCopiedEventHandler
000007fef880c7d8 4001703 9c System.Int32 1 instance 212855 _objectID
000007fef880c7d8 4001702 9d8 System.Int32 1 static 212855 _objectTypeCount
So the _rowSource is what we are interested in – let’s dump this shall we? Again using !DumpObj with the object address + field offset:
0:008> !DumpObj poi(0000000201ee0948+10)
Name: Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.Commands.BulkDataReader
MethodTable: 000007ff002c4338
EEClass: 000007ff002a4d90
Size: 64(0x40) bytes
File: C:\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\Microsoft.EnterpriseManagement.DataWarehouse.DataAccess.dll
Fields:
MT Field Offset Type VT Attr Value Name
000007fef880d608 40000e5 10 System.Boolean 1 instance 0 isClosed
000007ff002c3b70 40000e6 8 …DataReaderAdaptor 0 instance 00000002000b1480 dataAdaptor
000007fef8828658 40000e7 18 System.Guid 1 instance 00000002000b4c28 dataSetId
000007fef8828658 40000e8 28 System.Guid 1 instance 00000002000b4c38 managementGroupId
Ok what now? Now we need to look at the dataAdapter, like before 😉 But wait … we used poi() before – well, we’ll use it twice this time:
0:008> !DumpObj poi(poi(0000000201ee0948+10)+8)
Name: Microsoft.EnterpriseManagement.HealthService.Modules.DataWarehouse.AlertDataReaderAdaptor
MethodTable: 000007ff002cfa98
EEClass: 000007ff00301318
Size: 48(0x30) bytes
File: C:\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\Microsoft.EnterpriseManagement.HealthService.Modules.DataWarehouse.dll
Fields:
MT Field Offset Type VT Attr Value Name
000007fef2d9a548 4000035 8 …em.Data.DataTable 0 instance 00000002000b1928 schemaTable
000007fef880adf8 4000036 10 System.Object[] 0 instance 00000002001d89e8 dataItems
000007fef880c7d8 4000037 18 System.Int32 1 instance 0 currentIndex
000007fef880c7d8 4000038 1c System.Int32 1 instance 0 startIndex
000007fef880c7d8 4000039 20 System.Int32 1 instance 1000 itemsToProcessCount
We are interested in the dataItems object – but wait … it is an array as we can see from the [] characters in System.Object[] so we can use a nifty other command called !DumpArray here – again with triple pointer function:
0:008> !DumpArray poi(poi(poi(0000000201ee0948+10)+8)+10)
Name: Microsoft.EnterpriseManagement.HealthService.DataItemBase[]
MethodTable: 000007fef880adf8
EEClass: 000007fef838fc68
Size: 8032(0x1f60) bytes
Array: Rank 1, Number of elements 1000, Type CLASS
Element Methodtable: 000007ff000474b8
[0] 0000000140b851a0
[1] 0000000140b8a950
[2] 0000000140b8d420
[3] 0000000140b8feb8
[4] 0000000140b92950
[5] 0000000140b94eb0
[6] 0000000140b97428
[7] 0000000140b999a0
[8] 0000000140b9aba8
[9] 0000000140b9d6b8
[10] 0000000140ba0150
[11] 0000000140ba2be8
[12] 0000000140ba5648
[…SNIPPED…]
Ok interesting, so this is an Array which is holding items of type Microsoft.EnterpriseManagement.HealthService.DataItemBase but when we dump the array we only get the address of each such object, so we would need to dump each of them using !DumpObj and check them out – wow! … really? there are a looot of them … Ok, let’s worry about that a little later and now just dump one of the items (first one as example) just to see what we need to check there for each one:
0:008> !DumpObj 0000000140b851a0
Name: Microsoft.EnterpriseManagement.Mom.AlertSubscriptionModule.DataItemAlertSubscription
MethodTable: 000007ff002cdc70
EEClass: 000007ff002fe738
Size: 568(0x238) bytes
File: C:\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\Microsoft.Mom.AlertSubscriptionDataSourceModule.dll
Fields:
MT Field Offset Type VT Attr Value Name
000007fef88296c8 400001a 18 System.DateTime 1 instance 0000000140b851b8 dateCreated
000007fef8828658 400001b 20 System.Guid 1 instance 0000000140b851c0 sourceHealthServiceId
000007fef88068f0 400001c 8 System.String 0 instance 00000001400a54d8 itemXml
000007fef88068f0 400001d 10 System.String 0 instance 0000000140b862b8 dataTypeName
000007fef8828658 4000019 8 System.Guid 1 static 000000023fe46128 localHealthServiceId
000007ff002cd6f8 400009b 118 System.Int32 1 instance 0 subscriptionType
000007ff002cd850 400009c 11c System.Int32 1 instance 0 subscriptionProperty
000007fef8828658 400009d 138 System.Guid 1 instance 0000000140b852d8 id
000007fef88068f0 400009e 30 System.String 0 instance 0000000140b864b8 name
000007fef88068f0 400009f 38 System.String 0 instance 0000000140b86520 description
000007fef8828658 40000a0 148 System.Guid 1 instance 0000000140b852e8 baseManagedEntityId
000007fef8828658 40000a1 158 System.Guid 1 instance 0000000140b852f8 problemId
000007fef880d608 40000a2 130 System.Boolean 1 instance 0 createdByMonitor
000007fef8828658 40000a3 168 System.Guid 1 instance 0000000140b85308 workflowId
000007fef8827af0 40000a4 120 System.UInt32 1 instance 0 resolutionState
000007ff002cd9a8 40000a5 124 System.Int32 1 instance 2 priority
000007ff002cdb00 40000a6 128 System.Int32 1 instance 1 severity
000007fef88068f0 40000a7 40 System.String 0 instance 0000000140b86778 category
000007fef88068f0 40000a8 48 System.String 0 instance 0000000000000000 owner
000007fef88068f0 40000a9 50 System.String 0 instance 0000000000000000 resolvedBy
000007fef88296c8 40000aa 178 System.DateTime 1 instance 0000000140b85318 timeRaised
000007fef88296c8 40000ab 180 System.DateTime 1 instance 0000000140b85320 timeAdded
000007fef88296c8 40000ac 188 System.DateTime 1 instance 0000000140b85328 lastModified
000007fef88296c8 40000ad 190 System.DateTime 1 instance 0000000140b85330 lastModifiedExceptRepeatCount
000007fef88068f0 40000ae 58 System.String 0 instance 0000000140b86f70 lastModifiedBy
000007fef88296c8 40000af 198 System.DateTime 1 instance 0000000140b85338 timeResolved
000007fef88296c8 40000b0 1a0 System.DateTime 1 instance 0000000140b85340 timeResolutionStateLastModified
000007fef88296c8 40000b1 1a8 System.DateTime 1 instance 0000000140b85348 timeResolutionStateLastModifiedInDB
000007fef88068f0 40000b2 60 System.String 0 instance 0000000140b86f98 customField1
000007fef88068f0 40000b3 68 System.String 0 instance 0000000140b86fe0 customField2
000007fef88068f0 40000b4 70 System.String 0 instance 0000000140b87020 customField3
000007fef88068f0 40000b5 78 System.String 0 instance 0000000140b87060 customField4
000007fef88068f0 40000b6 80 System.String 0 instance 0000000140b870a0 customField5
000007fef88068f0 40000b7 88 System.String 0 instance 0000000140b870f8 customField6
000007fef88068f0 40000b8 90 System.String 0 instance 0000000140b87128 customField7
000007fef88068f0 40000b9 98 System.String 0 instance 0000000140b87160 customField8
000007fef88068f0 40000ba a0 System.String 0 instance 0000000140b871a0 customField9
000007fef88068f0 40000bb a8 System.String 0 instance 0000000140b873c0 customField10
000007fef88068f0 40000bc b0 System.String 0 instance 0000000000000000 ticketId
000007fef88068f0 40000bd b8 System.String 0 instance 0000000140b87438 context
000007fef88296c8 40000be 1b0 System.DateTime 1 instance 0000000140b85350 lastModifiedByNonConnector
000007fef8828658 40000bf 1b8 System.Guid 1 instance 0000000140b85358 connectorId
000007fef880c7d8 40000c0 12c System.Int32 1 instance 0 repeatCount
000007fef8828658 40000c1 1c8 System.Guid 1 instance 0000000140b85368 alertStringId
000007fef88068f0 40000c2 c0 System.String 0 instance 0000000140b88848 alertParams
000007fef88068f0 40000c3 c8 System.String 0 instance 0000000000000000 siteName
000007fef88068f0 40000c4 d0 System.String 0 instance 0000000140b88b30 baseManagedEntityFullName
000007fef88068f0 40000c5 d8 System.String 0 instance 0000000000000000 baseManagedEntityPath
000007fef88068f0 40000c6 e0 System.String 0 instance 0000000140b88bb8 baseManagedEntityName
000007fef88068f0 40000c7 e8 System.String 0 instance 0000000140b88bf8 baseManagedEntityDisplayName
000007fef88068f0 40000c8 f0 System.String 0 instance 0000000140b88c30 resolutionStateName
000007fef88068f0 40000c9 f8 System.String 0 instance 0000000000000000 timeZone
000007fef88068f0 40000ca 100 System.String 0 instance 0000000140b88c50 languageCode
000007fef88296c8 40000cb 1d8 System.DateTime 1 instance 0000000140b85378 timeRaisedLocal
000007fef88296c8 40000cc 1e0 System.DateTime 1 instance 0000000140b85380 timeAddedLocal
000007fef88296c8 40000cd 1e8 System.DateTime 1 instance 0000000140b85388 lastModifiedLocal
000007fef88296c8 40000ce 1f0 System.DateTime 1 instance 0000000140b85390 lastModifiedExceptRepeatCountLocal
000007fef88296c8 40000cf 1f8 System.DateTime 1 instance 0000000140b85398 timeResolvedLocal
000007fef88296c8 40000d0 200 System.DateTime 1 instance 0000000140b853a0 timeResolutionStateLastModifiedLocal
000007fef88296c8 40000d1 208 System.DateTime 1 instance 0000000140b853a8 timeResolutionStateLastModifiedInDBLocal
000007fef88296c8 40000d2 210 System.DateTime 1 instance 0000000140b853b0 lastModifiedByNonConnectorLocal
000007fef88292a0 40000d3 218 System.TimeSpan 1 instance 0000000140b853b8 queryExecutionTimeSpan
000007fef88296c8 40000d4 220 System.DateTime 1 instance 0000000140b853c0 dataItemCreateTime
000007fef88296c8 40000d5 228 System.DateTime 1 instance 0000000140b853c8 dataItemCreateTimeLocal
000007fef88068f0 40000d6 108 System.String 0 instance 0000000000000000 tfsWorkItemId
000007fef88068f0 40000d7 110 System.String 0 instance 0000000000000000 tfsWorkItemOwner
Alright, so we know that the CustomField9 which we are interested in had offset a0 – so now we can dump it directly:
0:008> !DumpObj poi(0000000140b851a0+a0)
Name: System.String
MethodTable: 000007fef88068f0
EEClass: 000007fef838ed78
Size: 540(0x21c) bytes
File: C:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
String: SOME_STRING_HERE_WON’T_TELL_YOU_THE_VALUE_HA!
Fields:
MT Field Offset Type VT Attr Value Name
000007fef880c7d8 4000103 8 System.Int32 1 instance 257 m_stringLength
000007fef880b318 4000104 c System.Char 1 instance 57 m_firstChar
000007fef88068f0 4000105 10 System.String 0 shared static Empty
>> Domain:Value 00000000003e9470:00000001bfe40488 <<
Cool, so now we know what we want and need to dump here – so what’s next? Well we have this nifty .foreach and .shell command to use the Windows CMD Find command on the output as we can see in these examples WinDbg Scripting 😀
- because the list all object addresses command will also output other crap when it gets to object index and/or Alert DataItems with EMPTY CustomProperty9 fields, we need to exclude “Invalid parameter” from the output
0:008> .shell -ci “.foreach (obj { !DumpArray poi(poi(poi(0000000201ee0948+10)+8)+10) }) { !DumpObj poi(${obj}+a0) }” FIND /V “Invalid parameter”
<Note: this object has an invalid CLASS field>
Invalid object
Integrated managed debugging does not support enumeration of symbols.
See http://dbg/managed.htm for more details.
<Note: this object has an invalid CLASS field>
Invalid object
Name: System.String
MethodTable: 000007fef88068f0
EEClass: 000007fef838ed78
Size: 540(0x21c) bytes
File: C:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
String: SOME_STRING_HERE_WON’T_TELL_YOU_THE_VALUE_HA!
Fields:
MT Field Offset Type VT Attr Value Name
000007fef880c7d8 4000103 8 System.Int32 1 instance 257 m_stringLength
000007fef880b318 4000104 c System.Char 1 instance 57 m_firstChar
000007fef88068f0 4000105 10 System.String 0 shared static Empty
>> Domain:Value 00000000003e9470:00000001bfe40488 <<
Name: System.String
MethodTable: 000007fef88068f0
EEClass: 000007fef838ed78
Size: 540(0x21c) bytes
File: C:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
String: SOME_STRING_HERE_WON’T_TELL_YOU_THE_VALUE_HA!
Fields:
MT Field Offset Type VT Attr Value Name
000007fef880c7d8 4000103 8 System.Int32 1 instance 257 m_stringLength
000007fef880b318 4000104 c System.Char 1 instance 57 m_firstChar
000007fef88068f0 4000105 10 System.String 0 shared static Empty
>> Domain:Value 00000000003e9470:00000001bfe40488 <<
.shell: Process exited
—
You do know the FIND command in CMD right? 😉
Now from the output there check the length of the strings and when you found the one which has a size bigger than 255 that’s your hit there!
Well it turns out that we had a 3rd Party Management Pack there which was using the CustomField9 for … well something whatever … and thus we either remove this MP or we engage the Vendor.
Problem solved! 😀
By the way – there are some cool WinDbg extensions out there which may help you a lot with debugging stuff in general so that you don’t need to use WinDbg scripting. These are more than awesome help for .NET Debugging.
A good example of such an extension would be SOSEx.
Have fun debugging!!! 🙂