NASA has overspent on Oracle to the tune of $15m. This is among the many findings of NASA’s Office of Inspector General’s scathing report into the agency’s poor SAM practices. The Inspector General certainly didn’t hold back.
The report stated that SAM practices at NASA expose the Agency to “operational, financial, and cybersecurity risks with management of the software life cycle largely decentralized and ad hoc.”
It was keen to point out where the Agency’s SAM challenges originate from, stating that “efforts to implement an enterprise-wide Software Asset Management program have been hindered by both budget and staffing issues and the complexity and volume of the Agency’s software licensing agreements.”
The report rated NASA’s SAM as “basic,” which is the lowest of the four rating options in the Microsoft’s Software Asset Management Maturity and Optimization Model (launched back in 2008) and adopted from the ISO commission. The report points out that “NASA has not implemented a centralized Software Asset Management tool to discover, inventory, and track license data as required by federal policy,” and that NASA is likely “years away from moving to an enterprise computing model in which IT capabilities, such as software asset management and cybersecurity, are centralized and consolidated. In the meantime, the Agency has yet to embrace key best practices or fully implement federal guidance required to appropriately manage its Software Asset Management program.”
The Agency also does not have consistent processes for legal representation during software contract negotiations and vendor audits, which increases the risks of exposing the Agency to penalties for violations of software license agreements. For example, while the NSSC (NASA Shared Services Center) provides a contract legal sufficiency review (e.g., review of terms and conditions), NASA’s Office of General Counsel is only required to be involved when software contracts exceed a $1 million threshold.
Likewise, legal involvement in vendor audits is ad hoc and most often at the discretion of the Agency organization under audit. In fact, the report points out that on many occasions Agency lawyers are unaware a software audit is even happening unless a vendor claim is submitted to NASA.
Training for software license use and management is also inconsistent across the Agency, with aging web-based training randomly assigned to personnel and a lack of a general software licensing training course available to the entire workforce.
Some of the standout figures from the report:
The one figure that is not stated in the report – because it may be an impossible figure to quantify – is precisely how much of NASA’s software spend has been entirely unbudgeted due to licensing shortfalls found after an audit. The report references a few such cases, such as the Office of Procurement receiving an unanticipated invoice for $415,000, and NASA being forced to reimburse the vendor SUSE $3.8 million after a script found it running unlicensed software.
The Agency does not have a centralized, authoritative database or inventory that tracks what licenses have been purchased, specific licensing agreements, and whether licenses are available for use by others at the Agency. These shortcomings have resulted in NASA spending approximately $15 million over the past 5 years on unused Oracle licenses. The report acknowledges that while this is just one example, there are likely multiple others unknown to the Agency.
The NASA Oracle contract is due for renewal in April 2023. In preparation, the OCIO, along with the Enterprise License Management Team, are gathering requirements and examining ‘how and why’ Oracle licensing became so cumbersome and complex to manage. In parallel to this, the Agency is also reviewing the current and desired licensing environment to quantify the true cost of doing business with Oracle.
The Inspector General has made seven recommendations to NASA’s Chief Information Officer to turn its Software Asset Management program around:
An additional two recommendations were made to NASA Chief Financial Officer to strengthen the financial aspects of the Agency’s SAM program:
Clearly, NASA will fail to meet Executive Order (EO) 14028. Under EO 14028 “Improving the Nation’s Cybersecurity” (issued in May 2021), US federal agencies are required to create a full inventory of their software in order to comply with governmental practice on cybersecurity (read more here). With the Agency so far having failed to implement an enterprise-wide Software Asset Management program, they will clearly miss this deadline. Importantly, given NASA’s role as an engineering, technology, and research and development organization, internally-developed applications are also in scope for EO 14028. Extensive use of open source software (OSS) and self-developed code is particularly vulnerable and must be subjected to rigorous controls to ensure cybersecurity.
In many ways it feels like NASA is being run as a technology startup with a lack of control of innovation, limited process controls, and an unwillingness to put basic cybersecurity controls in place. The difference is that they’re funded by the taxpayer and have a multi-billion dollar budget. Their biggest challenge in implementing the changes required by this report may well be cultural change. The signs aren’t good in that regard, with NASA already disagreeing with the OIG’s requirement that the SAM Manager report directly to the CIO. What sort of message does that send to the teams being asked to change their cultures and working practices? Space is hard, and so is SAM, if you don’t have senior executive backing for your change program.
One of the legends of Mission Control, Gene Kranz, required his teams to be Tough and Competent following the Apollo 1 tragedy, and clearly NASA haven’t been applying that doctrine to SAM. Given that software is woven through everything they do it’s perhaps only a matter of time before they have a mission failure related to poor software asset management.