Sidu Ponnappa
Apr 24, 2024
Over the past couple weeks, we had the chance to put our AI testing agent through its paces with a Salesforce Implementation Partner. And the results have been quite promising, to say the least.
We started by scoping the engagement tightly. We zeroed in on a single user story from their current sprint — generating unit tests for a recently completed Apex source class.
To make sure our agent had all the context it needed to do its thing, we asked their technical lead to provide some key details in the Jira ticket:
link to the specific user story driving the Apex changes
description of the changes which were made on the source class (important for us to understand the source class better and produce quality outcomes)
any critical edge cases or scenarios they wanted covered by tests
any other context / instructions — including any links like PRD / BRD
contact info for the dev who wrote the code, in case we needed to clarify anything
The goal was to equip our AI agent with the context a human would need to write rock-solid unit tests - no more, no less.
Once deployed, our agent spit out a complete set of unit tests in ~2 minutes.
A Salesforce dev from their team then spent about 30 minutes spot-checking the generated tests and adding some additional assertions specific to their business logic.
And just like that, they had a suite of tests that would have taken 6-8 hours to craft by hand.
We're talking 12-16x faster test creation, with 96% code coverage!
Still have a long way to go. Lots of training, engineering and design are still needed for the agent to tackle complex codebases confidently.
But I think this is the mode of engagement you want if you're working on applied AI tools:
Identify where you can create concrete, quantifiable impact on a real-world project.
Seek tight partnerships with service enterprises. You will need their deep domain understanding and access to real work to do well.
Be pragmatic and rigorous. Real-world outcomes over hype and AI-washing.
Get feedback on how good the code is and if the difference is meaningful enough for the business to pay for it.
The future is already here. It's just not evenly distributed. Yet!