Practical Software and Systems Measurement Continuous Iterative Development Measurement Framework
The collaborative PSM, NDIA, and INCOSE report provides recommendations for the measurement of continuous iterative developments (CID). The report includes a Practical Software and Systems Measurement (PSM) CID measurement framework detailing common information needs and measures that are effective for evaluating CID approaches. The information needs address the team, product, and enterprise perspectives to provide insight and drive decision-making. The framework also identifies and specifies an initial set of measures that have been identified as being practical measures to address these information needs. This guidance is intended to be used by team, program, and enterprise personnel who are implementing CID approaches, as a reference for common, practical measures that can be utilized.
Parts: The report has now been separated into three parts. Part 1 of this report includes a series of diagrams and an ontology to describe the development approaches and terminology used. It also includes an “Information Category-Measurable Concept-Measures” (ICM) Table detailing potential information needs and measures for CID developments. For the highest priority measures, sample measurement specifications have been developed that detail the identified measures. These are included in a separate Part 2 of the paper, along with a discussion of how to use these measures for enterprise decision making. Part 3 of the paper separately extends the main CID paper with detailed information and guidance on Software Assurance and Technical Debt.
Version 1.05 detailed potential information needs and measures that are common to CID approaches, and an initial set of ten measurement specifications that were prioritized by user surveys as highest value. This Version 2.1 includes added material that has been researched and developed by the CID working group. The new materials include information on measuring:
- Product value (Part 2, section 8.11)
- Enterprise measurement (Part 2, section 9)
- Software assurance (Part 3, section 10)
- Technical debt (Part 3, section 11)
We are currently looking for pilots in each of the services, to verify usage in the community, especially for the product value measure. Please contact Cheryl Jones or Geoff Draper if you are interested in participating.
Phase 3 of the project will begin soon, focused on the areas identified in part 1, section 6, including size and estimation, additional measures, and enhancements based on user pilots.
- Implementing Continuous Iterative Development and Acquisition
Defense Science Board (DSB) released a report in Feb-2018 containing seven recommendations regarding software design and acquisition. Section 868 of NDAA 2019 mandates implementation of these recommendations within 18 months. The Defense Innovation Board (DIB) Software Acquisition and Practices (SWAP) study group has also provided many insightful and largely compatible recommendations.
NDIA, INCOSE and PSM support the DSB and DIB concepts and the opportunities they offer to DoD and the defense industry. In 2018 the NDIA Systems Engineering Division commissioned a working group to study the implementation of the DIB and SWAP recommendations. NDIA offers the working group's recommendations below to OUSD(A&S) and OUSD(R&E) representing a broad “industry perspective” on a path forward.
Executive Summary Brief (pdf)
- Top Five SE Issues
The NDIA Systems Engineering Division determined that an update to the Top Issues in Systems Engineering (SE) report that was issued initially in 2003, updated in 2006 and again in 2010 was needed. The issues related to our defense industry are complex, affecting both the industry organizations as well as the government and military organizations.
A Working Group was formed, inputs were solicited in advance, and a reconciliation meeting with 18 members was held on August 17th and 18th, 2016. Although numerous separate issues were identified, the group found that the bulk of these actually fell into several major issue categories. The detailed results, including the status of the previous 2006 and 2010 SE issues, are described in this report.
- A Path Towards Cyber Resilient and Secure Systems, April 2016
Taking a holistic approach to system security requires bringing together multiple communities with rich histories introduces varying perspectives, terminologies, taxonomies and methodologies. This diversity provides opportunities and challenges for evaluating the security quality system attributes. In this paper, we discuss a path towards system security metrics and measures. Ultimately, we are committed to providing systems that are resistant to attack and resilient when under attack.
System Security Engineering Committee Chairs: Holly Dunlap, Raytheon & Beth Wilson, Raytheon
Developmental Test & Evaluation Committee Chair: Joe Manas, Raytheon
In collaboration with: INCOSE SSE Committee, Trusted Supplier Steering Group, and Mitre AF Cyber Integration
- Guidance for Utilizing Systems Engineering Standards (IEEE 15288.1 and IEEE 15288.2) on Contracts for Defense Projects, July 2015
Recommended guidance on tailoring, acquisition language, conformance, and evaluating compliance.
- Mapping Standards for Integrated Development, v0.7. October 4, 2015
Overview of the NDIA standards mapping tasks, strategy, and approach. Initial draft (v0.7), intended to be an industry asset improved upon with use and extended to include additional standards.
- Mappings between CMMI-DEV v1.3 and IEEE 15288 (Excel spreadsheet)
Initial mappings developed by consensus of a team of industry subject matter experts
- The Business Case for Systems Engineering: Results of the Systems Engineering Effectiveness Survey
This report summarizes the results of a survey that had the goal of quantifying the connection between the application of systems engineering (SE) best practices to projects and the performance of those projects. The survey population consisted of projects and programs executed by system developers reached through the NDIA, the IEEE Aerospace and Electronic Systems Society, and INCOSE. Analysis of survey responses revealed strong statistical relationships between project performance and several categories of specific SE best practices. The survey results show notable differences in the relationship between SE best practices and performance between more challenging and less challenging projects.
- The Business Case for Systems Engineering: Detailed Response Data
This report contains detailed response data from The Effectiveness of Systems Engineering: A Survey. The survey had the goal of quantifying the connection between the application of systems engineering (SE) best practices to projects and programs and the performance of those projects and programs. The survey population consisted of projects and programs executed by system developers reached through the NDIA, the IEEE Aerospace and Electronic Systems Society, and INCOSE. Analysis of survey responses revealed strong statistical relationships between project performance and several categories of SE best practices. The survey results show notable differences in the relationship between SE best practices and performance between more challenging and less challenging projects. The statistical relationship with project performance is quite strong for survey data of this kind when both SE capability and project challenge are considered together.
- Report on Systemic Root Cause Analysis of Program Failures. National Defense Industrial Association Systems Engineering Division in conjunction with Office of Under Secretary of Defense Acquisition, Technology & Logistics, Systems & Software Engineering, Arlington, VA, December 2008.