Army strategizes shift to Agile test and evaluation of software

Last fall at the Association of the United States Army’s annual conference, PEO EIS Program Executive Officer Ross Guckert said one of the key pillars in his acquisition organization’s Agile journey was the transformation of how software testing is conducted. Now, one year later, some of those changes are being introduced at the program level as PEO EIS collaborates with the U.S. Army Test and Evaluation Command (ATEC), ASA(ALT) and other stakeholders on new governance for the test and evaluation process.  

In the past, under the DOD Instruction 5000.75 for defense business systems, ATEC typically got involved in testing EIS programs toward the end of the waterfall software development process, conducting a so-called “Super Bowl test” of the final delivered product. The challenge with that approach was that multiple — and occasionally significant — problems were occasionally detected in the software, requiring development teams to take additional time and expense to fix them.  

With the launch of PEO EIS’s Agile journey and the Army’s Digital Odyssey, all new and select legacy programs of record will use the DOD’s 5000.87 Software Acquisition Pathway, which directs independent test organizations to “be integrated early into the planning and development activities throughout the software lifecycle.” Hence, ATEC will be involved early and continuously throughout the software development process, according to Aric Sherwood, director of PEO EIS’s Acquisition Innovation Directorate, and will help development teams shift from a Super Bowl testing approach to an integrated “test-as-you-go” approach. 

Some organizational change will be necessitated by this shift, said Sherwood. “We need to have the right amount of the right type of manpower,” he said. 

PEO EIS currently is conferring with ASA(ALT) and ATEC about the right amount of software testing and how it will be conducted. During the early stages of software development, ATEC may have more of an oversight role, observing testing conducted by vendors and program managers and reviewing data to ensure it looks clean, said Sherwood.  

During regular intervals throughout the development process, ATEC will help programs plan and execute user-based Agile demonstrations based on operationalized scenarios provided by the functional sponsor. This may advance operational testing as early as the first or second planning interval. 

The Army is in the process of updating its official regulation on test and evaluation —a process that could take as long as a year according to ATEC’s G-3/G-5 chief of plans Tom Ehrhart — so in the interim, ATEC and ASA(ALT) have developed a draft guidebook on Agile software test and evaluation with input from the director, operational test and evaluation and the deputy under secretary of the Army for test and evaluation.  

ATEC also is conducting a pilot with five Army “pathfinder” programs, including PEO EIS’s Enterprise Business Systems – Convergence (EBS-C), to develop playbooks on software testing. Those playbooks, which may become available sometime in December, will look at topics like end-user support during all aspects of the Agile process; defining the business process during Business Process Reengineering; how to populate, groom and prioritize program backlogs; how to write stories and acceptance criteria; and how to participate in iteration/sprint activities.  

Much work still needs to be done on understanding the Agile process, and Ehrhart says that ATEC is doing “full-court press” on Agile education right now.  

“The biggest impediment is that people in a lot of leadership positions are very accustomed to traditional T&E,” said Ehrhart at a recent “Scrum of Scrums” meeting with PEO EIS team members. “And when you talk about anything involving the principles of Agile, that’s just not possible with some of the ways we’ve done things that are multi-year efforts.” 

To address these challenges, ATEC has been holding weekly integrated product team meetings since January with industry and the T&E community to discuss Agile software, said Ehrhart, as well as monthly updates with leadership from ATEC and the Deputy Assistant Secretary of the Army for Data, Engineering and Software. 

EBS-C’s acting T&E manager, Dr. Allen Huckabee, said his program was asked by Army leadership to help identify roadblocks in adopting Agile, and EBS-C team members were able to identify “a lot” of them.  

“The transition from legacy tests to the new framework is challenging,” said Huckabee, who joined the EBS-C team in February after serving as the test manager at Global Combat Support System – Army for five years. “How do you implement a plan when it’s still fluid? How do you obtain operational test data from DevSecOps processes?” 

Despite these challenges, Huckabee believes the Army is “making progress” and that ATEC and PEO EIS can potentially obtain operational test data in other ways. He also said the Army is planning to keep its playbook agnostic, so it can be applied across all software programs using the .87 methodology.  

Huckabee is writing the test strategy for EBS-C, which still is in its vendor-selection stage. In the meantime, one of PEO EIS’s newly established programs, the Foreign Military Sales – Army Case Execution System (FMS-ACES), is already looking ahead to its test and evaluation process, according to FMS-ACES product lead Kelly Rutherford. 

“We’re going to establish the relationship with ATEC early to ensure we integrate the best practices of continuous evolvement of capabilities released to the end user,” she said. 

As with all things Agile, PEO EIS leaders know they likely will have room for improvement after the first iteration of Agile test and evaluation.  

“We have strong buy-in from our stakeholders and Army senior leadership to try things, fail, learn and retry,” said Sherwood. “With support like that, we can’t help but succeed."