In order to ensure a high quality of each build, we run our Build Verification Tests (BVT) automated suite every Monday, Wednesday and Friday. The goal of a BVT test is to verify the basic functionality of the product. First, let me explain the deployment of this test environment. We use an 8-machine environment, with 3 servers and 5 client machines.
Servers:
1. Domain Controller
2. SQL Server
3. CRM Web Server
Clients:4. Laptop client (used in Online mode)
5. Laptop client (used in Offline mode)
6. Desktop Client
7. Web Client / Data Migration Client
8. No CRM Client
Before each test run, the environment is built out “from scratch” – we start by installing Windows on each machine. Installation of all the components and CRM prerequisites is done without any human interaction by using install scripts and configuration files.
Now let’s move on to the actual tests themselves. Every BVT is automated and the tests mainly fall into two broad categories: UI tests that manipulate the various CRM applications or platform tests that directly call the Microsoft CRM SDK. A UI test can actually drive the CRM application, just as a user would. A few examples of actions that a UI test can do are click a button, type text, and take the CRM Outlook offline. These tests ensure that our application layer is functional. On the other hand, a platform test doesn’t drive any UI.
A platform test verifies that SDK methods work as expected, by calling messages such as Create(businessentity entity) and Delete(string typeName, Guid id). The UI tests are run against all four client types (machines 4 – 7) while the platform tests are run on the machine with no client installed (I bet you were wondering what that machine was for). Each client machine is assigned one CRM user that all tests that run on that box will use. This helps to ensure that one client test will not interfere with a test on another client since most of the UI client tests are run at the same time. Once those finish, the platform tests are run alone. Finally, a few cases that need to execute on the CRM Server are run.
Our entire BVT suite is made up of almost 1300 individual test cases and takes a few hours to run. After each test case is run, a result, either Pass or Fail, along with a test case log (and a screenshot if necessary) is uploaded to a central database to be investigated when the run is completed. On most days, the results of the test run are available by 10 A.M. The results are then investigated by the owner of the test case. Any failure is treated very seriously – a bug is filed, an email is sent out, and the developer must fix the bug by the next BVT run.