Possibilities to create Memory Dumps from crashing processes

Sometimes it happens that during regular operation the Dynamics AX Application Object Server (AOS) is suffering from an abnormal program termination, which is better know as a crash. Users connected with the Dynamics AX Client to the affected AOS Instance are noticing such an issue very soon as they get disconnected from the AOS.

In such a situation maybe trying out a more recent Dynamics AX Kernel Hotfix may help. However if you decide to open a Support Request to get help identifying what concretely causes the AOS to crash, you will get asked to create a Memory Dump of the crash, and if you have a Memory Dump ready by that this will speed up the process a bit.

What is a crash and why do I need to collect a Memory Dump?

At a very top level a process is crashing when it is performing an illegal operation that is not captured / handled by the process itself. This unhandled exception forces the Operating System to terminated the process. Examples for an illegal operation are for example a division by zero or accessing an invalid memory location.

When you are creating a Memory Dump from a crashing process (Crash Dump) at a high level the dump is a snapshot from the process’ memory and register context (together with some more information) at the time of the crash. So the dump shows for example the very last operation being executed that caused the process to crash.

By doing a post mortem analysis of the memory dump it is (usually) possible to identify what the AOS was doing (e. g. was X++ code executed) when the crash happened, so the memory dump is the key to solve a crash situation!

Did the AOS really crash?

Before I go into detail about how to create a Memory Dump I would like to show you however some possibilities to confirm you are definitely running into a crash situation and here the Windows Event Log of the AOS is the key.

Unexpected termination

The most common event you see in a crash situation is the Error event 1000 from Application Error in the Application Event Log. This is definitely a confirmation that an AOS Instance was terminating unexpectedly.

Sometimes there is no entry in the Application Event Log however only in the System Event Log. The Error event 7034 from Service Control Manager is another confirmation that an AOS Instance was terminating unexpectedly.

Forced termination

In contrast to an unexpected termination there is also a forced termination whose effects are however similar not to say the same. The difference to an unexpected termination is that a forced termination is caused by the program logic of the Dynamics AX AOS Kernel itself because some error checking code determined a situation the AOS cannot resolve; however continuation in such a situation would most likely cause some more serious effects up to data corruption, so it is better to terminate at this stage and log this fact to the Application Event Log.

In such a case an Error event 110 from Dynamics Server is logged having the wording “Unexpected situation” in the Description.

Controlled shutdown

We have seen so far confirmations for an unexpected termination and a forced termination, however there exist also proves for a controlled shutdown where the AOS in fact did not crash.

The Information event 108 in the Application Event Log is always logged when the AOS has been requested to stop – in a controlled way. If you see this event someone or something has requested the AOS to stop by intention.

The Information events 7035 and 7036 from Service Control Manager in the System Event Log are another prove of a controlled shutdown of the AOS service. Again someone or something requested to AOS to stop by intention.

What do I need to create a Crash Dump?

A Crash Dump is a Memory Dump that is automatically created when a process is executing an operation (CPU instruction) that causes the process to crash.

Lets assume you have confirmed your AOS is indeed crashing and you need to create a Memory Dump (Crash Dump).

There exist several tools you can use to create a Memory Dump from a crashing process, and even the operating system is providing some built in functionality.

Debug Diagnostic Tool v1.2 (DebugDiag)

Highlight: Once installed the tool runs as a windows service.

To create a dump with the Debug Diagnostic Tool v1.2 (DebugDiag) you need to download and install it first. Download appropriate 32bit or 64bit version depending on Operating System in use.

When you launch the tool you are presented with a Graphical User Interface (GUI) and a Wizard the helps you to create a new Crash Rule.

The advantage of using DebugDiag is that it has a GUI and with the Crash Rule created not only will it attach to the currently running AOS Instances but also AOS Instances started after the creation of the Crash Rule.

Windows Error Reporting (WER)

Highlight: No software is required to be downloaded.

The Windows Error Reporting (WER) that is built into Windows 2008/2008 R2 can be configured to automatically create and store memory dumps from an AOS crash (This cannot be used on Windows Server 2003).

The configuration happens by making some Registry modifications, more concrete creating the following Registry Key:

In this new Registry Key several Registry Values have to be added as well (DumpFolder, DumpCount, DumpType, CustomDumpFlags).

Please note:
Do NOT set the DumpType to 1 or 2, but set the DumpType to 0 and set CustomDumpFlags to 7015 decimal (0x1B67 hexadecimal). Invalid settings will generate dumps without the required information.

ADPlus

To create a dump with ADPlus you first need to download and install the Debugging Tools for Windows and launch adplus.vbs from the command line with the -crash and -pn ax32serv parameters.

After executing the script you will see new command windows showing up – one command window per running AOS Instance. Actually this are no command windows but this is the command line debugger CDB that is attached to the AOS Instance.

It is very important that you don’t close those CDB instances nor log off from the machine because a closing of the CDB instances will also terminate the attached AOS Instance!

Please note:
ADPlus will only attach to the currently running AOS Instances. If you start another AOS Instance afterwards or restart a running AOS Instance you need to execute ADPlus again!

PLEASE NOTE: This tool requires an active logged on user session the whole time. You can lock your user session. However as soon as the user logs off it will terminate the tool and with that will terminate any attached processes.

Crash or High Memory Usage?

As indicated before a (Full) Memory Dump is containing the process’ memory and register context, so usually the file size of the dump file is about the size of the currently allocated virtual memory.

If the file size of the memory dump is about or larger than 1.6 GB and your are not running a 64 bit version of Dynamics AX the chances are very high the crash was caused by a High Memory Usage situation or because of a Memory Leak.

EMEADAXSupport

See more articles from this author