Recipes by Category

App Distribution (2) Bundle logic, interface and services for distribution. App Logic (37) The Apex programming language, workflow and formulas for logic. Collaboration (6) The Salesforce Chatter collaboration platform. Database (29) Data persistence, reporting and analytics. Integration (33) Web Service APIs and toolkits for integration. Security (9) Platform, application and data security. Tools (4) Force.com tooling User Interface (36) Visualforce MVC and metadata-drive user interfaces. Web Sites (12) Public web sites and apps with optional user registration and login.
Beta Feedback
Cookbook Home » Automated Unit Test Execution

Automated Unit Test Execution

Post by LukeJFreeland  (2012-01-23)

Status: Unverified
Level: intermediate

Problem

Unit tests are great. When you follow best practices and implement them when a feature has been implemented, they are a lifesaver when they fail because something has changed.

The problem is that right now using just the platform, as far as I can tell, executing unit tests is a manual process, which typically involves going to the "Apex Test Execution" page or deploying to a production org. Using apex code and some scheduled jobs, we'll automate the execution of our unit tests, so that we can know much sooner if our changes have broken something.

Solution

Automating unit test execution is conceptually simple. The overall algorithm is

  1. Create a scheduled job that queues the test classes containing the desired unit tests to be executed by inserting "ApexTestQueueItem" objects.
  2. Store the Apex Job Id somewhere so that we can process the test results later. This is necessary because the unit tests are executed asynchronously.
  3. Create one or more other scheduled jobs that periodically check the status of the unit tests, and when complete, email us the results.

First, create a custom object whose object name is "AutomatedTestingQueue". Next, create a "Text, Required, Length 40" custom field whose api name is "AsyncId". This object stores the aysnchronous job id while the unit tests are executed so we can periodically check to see if they're done executing and then process them.

Next, save the following classes

global with sharing class AutomatedTestJobQueuer implements schedulable {
    
    global void execute(SchedulableContext SC) {
        doExecute();
    }
    
    @future (callout=true)
    public static void doExecute(){
        enqueueUnitTests();
    }
    
    public static void createDaily4AMScheduledJob(){
        AutomatedTestJobQueuer atj = new AutomatedTestJobQueuer();  
        string sch = '0 0 4 * * ?';  
        system.schedule('Enqueue Unit Tests 4 AM',sch,atj);
    }

    /* Allows us to externally enqueue our unit tests. For example,
       whenever we check our code into source control, we could
       run our unit tests.
    */
    webservice static void enqueueUnitTests(){      
        enqueueTests();
    }


    // Enqueue all classes beginning with "Test".  
    
    public static void enqueueTests() {
       /* The first thing you need to do is query the classes that contain
         the unit tests you want executed.

         In our org, our test classes are named "Test<Class_Name_Here>"
         so that all the test classes are grouped together in Eclipse.
         Change the where clause as necessary to query the desired classes.
       */
            
       ApexClass[] testClasses = 
         [SELECT Id,
                 Name
            FROM ApexClass 
           WHERE Name LIKE 'Test%'];
              
       Integer testClassCnt = testClasses != null ? testClasses.size() : 0;
        
       system.debug('   enqueueTests::testClassCnt ' + testClassCnt);
            
       if (testClassCnt > 0) {
          /*
             Insertion of the ApexTestQueueItem causes the unit tests to be 
             executed. Since they're asynchronous, the apex async job id
             needs to be stored somewhere so we can process the test results
             when the job is complete.
          */
          ApexTestQueueItem[] queueItems = new List<ApexTestQueueItem>();
            
          for (ApexClass testClass : testClasses) {
              system.debug('   enqueueTests::testClass ' + testClass);
                
              queueItems.add(new ApexTestQueueItem(ApexClassId=testClass.Id));
          }

          insert queueItems;

          // Get the job ID of the first queue item returned. 
    
          ApexTestQueueItem item = 
            [SELECT ParentJobId
               FROM ApexTestQueueItem 
              WHERE Id=:queueItems[0].Id
              LIMIT 1];
            
          AutomatedTestingQueue__c atq = new AutomatedTestingQueue__c(
              AsyncId__c = item.parentjobid
          );

          insert atq;
       }
    }
}
global with sharing class AutomatedTestingJob implements Schedulable {
    
    global void execute(SchedulableContext SC) {
        doExecute();
    }

    // Have to use a future method so the email will be sent out.
    @future (callout=true)
    public static void doExecute(){
        processAsyncResults();
    }
    
    /*
        Schedule String Format: Seconds Minutes Hours Day_of_month Month Day_of_week optional_year
    */
    
    public static void createEvery15MinuteScheduledJobs(){
        AutomatedTestingJob atj = new AutomatedTestingJob();  
        string sch = '0 0 * * * ?';  
        system.schedule('Process Queued Unit Tests Every Top Of The Hour',sch,atj);
        
        sch = '0 15 * * * ?';  
        system.schedule('Process Queued Unit Tests At Each Quarter After',sch,atj);
        
        sch = '0 30 * * * ?';  
        system.schedule('Process Queued Unit Tests At Each Bottom Of The Hour',sch,atj);
        
        sch = '0 45 * * * ?';  
        system.schedule('Process Queued Unit Tests At Each Quarter To The Hour',sch,atj);
    }
    
    public static void processAsyncResults(){
        List<AutomatedTestingQueue__c> queuedTests = 
           [select id,
                   name,
                   AsyncId__c
              from AutomatedTestingQueue__c
             limit 5];
        
        if (queuedTests != null && queuedTests.size() > 0){
            Set<Id> AsyncIds = new Set<Id>();

            for (AutomatedTestingQueue__c queuedJob : queuedTests){
                 AsyncIds.add(queuedJob.AsyncId__c);
            }
            
            List<ApexTestQueueItem> queuedItems = checkClassStatus(AsyncIds);
            
            Map<Id, List<ApexTestQueueItem>> groupedTestsByJob = new Map<Id, List<ApexTestQueueItem>>();

            for (ApexTestQueueItem atqi : queuedItems){
                 if (groupedTestsByJob.containsKey(atqi.ParentJobId) == true){
                     List<ApexTestQueueItem> groupedTests = groupedTestsByJob.get(atqi.ParentJobId);
                     groupedTests.add(atqi);
                 }
                 else{
                     List<ApexTestQueueItem> groupedTests = new List<ApexTestQueueItem>();
                     groupedTests.add(atqi);
                     groupedTestsByJob.put(atqi.ParentJobId, groupedTests);
                 }
            }
            
            Set<Id> completedAsyncIds = getCompletedAsyncJobsIds(groupedTestsByJob);
            
            if (completedAsyncIds != null && completedAsyncIds.size() > 0){
                
                List<ApexTestResult> testResults = checkMethodStatus(completedAsyncIds);
                
                Map<Id, List<ApexTestResult>> groupedTestResultsByJob = new Map<Id, List<ApexTestResult>>();
                
                
                for (ApexTestResult testResult : testResults){
                    if (groupedTestResultsByJob.containsKey(testResult.AsyncApexJobId)){
                        List<ApexTestResult> groupedTestsResults = groupedTestResultsByJob.get(testResult.AsyncApexJobId);
                        groupedTestsResults.add(testResult);
                    }
                    else{
                        List<ApexTestResult> groupedTestsResults = new List<ApexTestResult>();
                        groupedTestsResults.add(testResult);
                        
                        groupedTestResultsByJob.put(testResult.AsyncApexJobId, groupedTestsResults );
                    }
                }

                List<AutomatedTestingQueue__c> queuedTestsToDelete = new List<AutomatedTestingQueue__c>(); 
                
                for (List<ApexTestResult> jobTestResults : groupedTestResultsByJob.values()){
                    sendTestResultEmail(jobTestResults);
                }
                
                for (AutomatedTestingQueue__c queuedTest : queuedTests){
                    for (Id completedAsyncId : completedAsyncIds){
                        if (queuedTest.AsyncId__c == completedAsyncId){
                            queuedTestsToDelete.add(queuedTest);
                            break;
                        }
                    }
                    if (groupedTestsByJob.containsKey(queuedTest.asyncId__c) == false){
                        queuedTestsToDelete.add(queuedTest);
                    }
                }
                
                if (queuedTestsToDelete.size() > 0){
                    delete queuedTestsToDelete;
                }
            }
        }
    }
    
    public static Set<Id> getCompletedAsyncJobsIds(Map<Id, List<ApexTestQueueItem>> groupedTestsByJob){
        Set<Id> completedAsyncJobIds = new Set<Id>();
        
        for (List<ApexTestQueueItem> jobTests : groupedTestsByJob.values()){
            if (jobTests == null || jobTests.size() == 0){
                continue;
            }
            
            Boolean allCompleted = true;
            
            for (ApexTestQueueItem queuedTest : jobTests){
                if (queuedTest.Status != 'Completed' && queuedTest.Status != 'Failed' && queuedTest.Status != 'Aborted'){
                    allCompleted = false;
                    break;
                }
            }
            
            if (allCompleted == true){
                completedAsyncJobIds.add(jobTests[0].ParentJobId);
            }
        }
        
        return completedAsyncJobIds;
    }
    
    private static void sendTestResultEmail(List<ApexTestResult> jobTestResults){
        system.debug(' In sendTestResultEmail');
            
        Messaging.SingleEmailMessage mail = new Messaging.SingleEmailMessage();
        
        String emailAddress = 'Your email address here';
        
        String[] toAddresses = new String[] { emailAddress };
    
        mail.setToAddresses(toAddresses);
        
        String emailSubject = 'Dev Unit Test Results ' + String.valueOf(Date.today()); 
    
        mail.setSubject(emailSubject);

        String testResultEmailbody = getTestResultHtmlEmailBody(jobTestResults);

        mail.setHtmlBody(testResultEmailbody);
        Messaging.sendEmail(new Messaging.Email[] { mail });
        
        system.debug(' sent test results email');
    }
    
    private static String getTestResultHtmlEmailBody(List<ApexTestResult> jobTestResults){
        system.debug(' In getTestResultHtmlEmailBody');
        
        List<ApexTestResult> successTests = new List<ApexTestResult>();
        List<ApexTestResult> failedTests = new List<ApexTestResult>();
        
        for (ApexTestResult jobTestResult : jobTestResults){
            if (jobTestResult.Outcome == 'Pass'){
                successTests.add(jobTestResult);
            }
            else{
                failedTests.add(jobTestResult);
            }
        }
        
        Integer numTestsRun = successTests.size() + failedTests.size();
        Integer numFailures = failedTests.size();
        Integer successNum = numTestsRun - numFailures;
        
        if (successNum < 0){
            successNum = 0;
        }
        
        String testResultBody = '';
        
        // Unfortunately, css has to be inlined because many email service providers now exclude external CSS
        // because it can pose a security risk.
        testResultBody += '<td style="text-align: right;">' + numTestsRun + '';
        testResultBody += '<td style="text-align: right;">' + numFailures + '';
        testResultBody += '<td style="text-align: right;">' + successNum + '';
        
        testResultBody += '
Tests Run:
Failure Count:
Success Count:
'; if (numFailures > 0){ testResultBody += '<div style="margin: 5px 0px; font-weight: bold;">Test Failures</div>'; testResultBody += ''; testResultBody += ''; testResultBody += '<th style="text-align: left; padding-left: 5px;">Test Class</th>'; testResultBody += '<th style="text-align: left; padding-left: 5px;">Unit Test</th>'; testResultBody += '<th style="text-align: left; padding-left: 5px;">Message</th>'; testResultBody += '<th style="text-align: left; padding-left: 5px;">Stack Trace</th>'; testResultBody += '<th style="text-align: left; padding-left: 5px;">Time (Ms)</th>'; testResultBody += ''; for (ApexTestResult testFailure : failedTests){ testResultBody += ''; testResultBody += '<td style="padding: 5px; vertical-align: top;">' + testFailure.ApexClass.Name +''; testResultBody += '<td style="padding: 5px; vertical-align: top;">' + testFailure.MethodName +''; testResultBody += '<td style="padding: 5px; vertical-align: top;">' + testFailure.message +''; testResultBody += '<td style="padding: 5px; vertical-align: top;">' + testFailure.stackTrace +''; testResultBody += '<td style="padding: 5px; vertical-align: top;">' + testFailure.ApexLog.DurationMilliseconds +''; //testResultBody += '<td style="vertical-align: top;">' + testFailure.type_x +''; testResultBody += ''; } testResultBody += '
'; } return testResultBody; } // Get the status and pass rate for each class // whose tests were run by the job. // that correspond to the specified job ID. public static List<ApexTestQueueItem> checkClassStatus(Set<ID> jobIds) { ApexTestQueueItem[] items = [SELECT ApexClass.Name, Status, ExtendedStatus, ParentJobId FROM ApexTestQueueItem WHERE ParentJobId in :jobIds]; for (ApexTestQueueItem item : items) { String extStatus = item.extendedstatus == null ? '' : item.extendedStatus; System.debug(item.ApexClass.Name + ': ' + item.Status + extStatus); } return items; } // Get the result for each test method that was executed. public static List<ApexTestResult> checkMethodStatus(Set<ID> jobIds) { ApexTestResult[] results = [SELECT Outcome, MethodName, Message, StackTrace, AsyncApexJobId, ApexClass.Name, ApexClass.Body, ApexClass.LengthWithoutComments, ApexClass.NamespacePrefix, ApexClass.Status, ApexLogId, ApexLog.DurationMilliseconds, ApexLog.Operation, ApexLog.Request, ApexLog.Status, ApexLog.Location, ApexLog.Application FROM ApexTestResult WHERE AsyncApexJobId in :jobIds]; for (ApexTestResult atr : results) { System.debug(atr.ApexClass.Name + '.' + atr.MethodName + ': ' + atr.Outcome); if (atr.message != null) { System.debug(atr.Message + '\n at ' + atr.StackTrace); } } return results; } }

Now that you've the custom object to store the Async Job Ids and the schedulable classes, it's a simple matter of scheduling the jobs. Note: The defaults used in the classes require 5 scheduled jobs and salesforce imposes a limit of 10, so please make sure that you have enough or you'll have to refactor the code or change the scheduling to get it to work.

Use the following to create a scheduled job that will execute your unit tests daily at 4 AM.

AutomatedTestJobQueuer.createDaily4AMScheduledJob();

Use the following to create four scheduled jobs that effectively will process the test results and email them every 15 minutes, if needed.

AutomatedTestingJob.createEvery15MinuteScheduledJobs();

There's also a webservice method "enqueueUnitTests" that allows you to queue them on demand and have the results emailed to you. I envisioned this to be useful for continuous integration purposes.

For example, if you're using source control, which I really hope you are, and you check in code, this method would be called to enqueue and run your unit tests. Essentially, whenever code is checked in, the unit tests will run in semi real-time and you'll know fairly quickly whether or not you broke something. Since finding and fixing bugs is cheaper and faster when you're already coding somewhere, this'll save you from many future headaches.

I hope this helps and happy coding.

Luke

Share

Recipe Activity - Please Log in to write a comment

Don't all the test methods in non-test classes (or incorrectly named classes) get excluded by this code? Can't add a where clause to the SOQL because the Body field is long text area. So how to ensure such classes are included?

by davcondev  (2013-04-22)

Quick test from Dana.

by dle01252010  (2013-04-16)

by dle01252010  (2013-04-16)

Has anyone successfully implemented this?

by Ryan Francis Roque  (2013-03-08)

Matthew,

When I tried to use your schedule syntax of '0 0/15 * * * ?', I receive the following error:

    "System.StringException: Seconds and minutes must be specified as integers: 0 0/15 * * * ?"

Here's the code I tried to use in execute anonymous to create the scheduled job:


        AutomatedTestingJob atj = new AutomatedTestingJob();  
string sch = '0 0/15 * * * ?';  
system.schedule('Process Queued Unit Tests Every 15 minutes',sch,atj);

Are you sure that is possible?

by LukeJFreeland  (2012-01-31)

Great minds think alike! I took a similar approach with Automated Testing for Force.com. Guessing when the results will be ready for processing is tricky, so I like how you're using a recurring Scheduled Job to do it. One way to keep those within the limited number of Scheduled Jobs is to use the increment syntax for the schedule expression:

ie. '0 0/15 * * * ?' would run every 15 minutes starting at the top of the hour.

Apex Schedulable list a few other powerful expression options that might be of interest.

by Matthew Botos  (2012-01-30)

X

Vote to Verify a Recipe

Verifying a recipe is a way to give feedback to others and broaden your own understanding of the capabilities on Force.com. When you verify a recipe, please make sure the code runs, and the functionality solves the articulated problem as expected.

Please make sure:
  • All the necessary pieces are mentioned
  • You have tested the recipe in practice
  • Have sent any suggestions for improvements to the author

Please Log in to verify a recipe

You have voted to verify this recipe.