SAP Concur Overcomes Obstacles after Undergoing AWS Experience-Based Acceleration

Based in Bellevue, Washington, SAP Concur is a division of SAP that provides travel and expense management services to more than 60,000 organizations worldwide, including most of the world’s largest corporations. In August 2018, the DoD awarded SAP Concur a major contract to build new travel and expense software that would drastically reduce travel costs and labor for more than 2 million military personnel. Beyond the $9.3 million initial phase, the contract represented a potential annual revenue of $60–70 million for SAP Concur: a sum larger than revenue from its top seven customers combined. Aware of the security and compliance challenges involved with any government contract, SAP Concur opted to proceed with a shared responsibility model exemplified by AWS GovCloud (US), which gives companies the flexibility to architect secure cloud solutions that comply with the Federal Risk and Authorization Management Program High Baseline.

While SAP Concur was already established in the commercial space, the DoD project involved the arduous task of spinning up a new environment that had to comply with the DoD’s strict and complex regulatory framework. The enormity of the project resulted in analysis paralysis, according to Cameron Etezadi, SAP Concur’s chief architect at the time. “It was overwhelming,” says Etezadi, who is now the head of platform architecture for SAP. “Some teams were reluctant to embrace the changes that a cloud DevOps environment represents.” A lack of communication and collaboration across teams also resulted in bottlenecks and stoppages.

SAP Concur missed its initial deadline of June 2019. Feeling enormous pressure to get back on track and fulfill its obligation to shareholders and the DoD, the company set an aggressive new deadline of November 2019—but something had to change. “We needed help opening the lines of communication and clarifying the technical direction between teams,” says Etezadi. That’s when AWS pitched the idea of an EBA party. “I was reluctant at first,” says Etezadi. “We were already behind schedule, and devoting 3 days of engineering resources to a migration party seemed risky.” However, Etezadi ultimately determined that the EBA party would facilitate the kind of organizational and cultural transformation that SAP Concur needed to become unstuck and fulfill its DoD contract.

In July 2019, SAP Concur brought together a coalition of volunteers that included senior vice presidents and other influential people as well as whole teams from around the world. Roughly 120 people gathered in a single room for 3 day-long sessions to identify blockers and jump-start productivity. “We could see some of the problems and gaps in the company almost immediately,” says Etezadi. “The security people sat at one table; the infrastructure people sat at another, and so forth—it was a visual representation of how siloed our teams were.” Throughout the EBA party, AWS EBA organizers looked for ways to better integrate teams and foster an environment of collaboration. After noticing that one team had neglected to address a ticket during the whole first day, the organizers removed the team’s table, forcing them to sit at other tables and work with other teams.

The overall objective was to see how many applications the teams could deploy on AWS by the end of the EBA party. Now together in one room, the teams could see the challenges others faced in real time, and they brainstormed solutions together. The experience was fun as well as productive: teams celebrated minor successes by ringing a cowbell and major accomplishments by banging a gong, creating an atmosphere of collaboration and friendly competition as teams raced to complete cowbell- or gong-worthy tasks. “The cowbells and gongs may seem silly, but they provided a sense of shared ritual and ceremony that brought us together as one tribe,” says Etezadi. “The feeling of shipping and delivery is what motivates engineers. Being able to celebrate progress and mark it audibly turned out to be super motivating.”

The EBA party sparked enthusiasm that participants brought back to their teams all over the world. “The positive feedback was overwhelming,” says Etezadi. “We had several more teams demand that we rerun the EBA party for them.” To satisfy demand, SAP Concur hosted two more 3-day EBA parties near its development centers in Prague, Czech Republic, and Minneapolis, Minnesota, where organizers had to set up extra space for eager employees who couldn’t fit into the EBA party space but wanted to participate remotely from elsewhere in the building. Across the three EBA parties, SAP Concur teams removed more than 60 blockers and deployed more than 19 services on AWS.

The company finally found the momentum it needed to finish the DoD project. Within 4 months, SAP Concur had migrated more than 700 microservices to AWS. “We came out with a replicable set of skills that we could continue applying to the problem at hand,” says Etezadi. “We started doing daily check-ins as we tracked the project, and we met our contractual dates.” The cloud environment enabled the SAP Concur team to quickly make adjustments as needed. As of March 2021, the DoD is in the beginning stages of rolling out the software to its users.