After perusing this blog, I realized it lacks talk of how we internally test Microsoft CRM and how we ensure that it is ready for customers to use. This is a topic that I hold close to my heart (and have some domain knowledge of) as I have worked on the Quality Assurance team for over two years. So I finally decided to get my feet wet with blogging and get your feet wet with our testing processes. To get started, let’s go through a list of basic types of testing that we do and I’ll very briefly describe them.
Accessibility: Tests that the product can be driven by the keyboard only, tests the product is usable under high-contrast settings, etc. The goal of accessibility testing is to ensure that Microsoft CRM can be used by users with special needs.
Configuration: Tests that the product can be used on various machine configurations.
Functional: Tests that the features of the product work as they are designed.
International: Tests the product correctly handles different date, time, number formats, tests that the product’s input boxes correctly save and display any Unicode character, etc.
Localization: Tests that all strings are correctly translated in the proper language without truncation, etc.
Performance: Self-explanatory.
Scenario: Tests that an end-to-end business scenario can be executed.
Security: Tests the product is not susceptible to cross site scripting, SQL injection, etc.
Setup: Tests the various components of setup work, such as Install, Upgrade, and Repair.
Since most of what I do is functional testing, this is where I’d like to focus the rest of this post. The main goal of functional testing is to answer this question: Does the product work as expected? Does action X produce result Y? For each feature, we create test plans that document what actions we want to take and what results are expected.
Each action/result combination is a test case. Some example test cases would be to create an account and verify that data is saved or to create a workflow rule and verify that it fires at the correct time. A test suite is a group of test cases. There are three broad categories of test suites within the CRM team – Check-In Test (CIT), Build Verification Test (BVT), and Functional Verification Test (FVT). A Check-In Test Suite would be run before a developer makes a source code change in the product to ensure that the change has not broken an existing feature. Simply put, a CIT suite is a brief sanity check, often aimed specifically at the area of the product being modified, and therefore is not broad and not very deep. On the other hand, Build Verification Tests are aimed at verifying the overall quality of the product. The BVT suite is very broad, but not very deep, meaning that it verifies basic functionality of much of the product. Finally, a FVT is everything else; basically, the kitchen sink of test cases. As such, FVTs are both broad and deep.
As you can imagine, manually running these test suites day in and day out would be incredibly time-consuming, quite tedious, and often repetitive. But, these are some of the exact things that computers are great at doing! Our test team has heavily invested in test automation. By automating our testing, we are easily able to test our application over and over again with little effort. This is especially helpful since the CRM Development team produces a new build each day. The time it takes for each of the suites to run differs greatly. A CIT suite can usually be run in minutes, the BVT suite takes hours, and an FVT suite takes almost two days to complete (running 24 hours a day!).