Systems Engineering Studies

Studies and Publications

2024 Reports

 Moving from Predictive Planning to Empirical Planning for Systems Engineering

Systems Engineering coupled with Agile practices is a key enabler to successful execution in an environment where focus is on incrementally delivering product while providing ongoing management of the technical baseline and incorporation of new information. This approach provides an understanding of the current and target state of the system development from early planning through deployment of the program. This paper provides a perspective on the responsibilities of Systems Engineering functions in a modern engineering environment coupled with techniques used to plan and execute the work which must allow for agility through continuous learning cycles and feedback from users. This supports the reality of program execution where comprehensive specification up front may no longer be desired - or even possible.  

Accelerating Value Delivery in Highly Complex Domains

This article provides an example, based on a missile system, on how to map out an operational value stream, to the nested development value streams that build and deliver the solutions within the operational value stream, to an organizational design and lean-agile operating system that enables optimal execution of those value streams. If you are experiencing challenges in value stream identification, decomposition, and organizational design, use this information to help guide your reasoning process. By sharing this information, we hope to deepen the understanding in our community on how to leverage the power of value streams in the delivery of large solutions in complex domains.

The Intersection of MOSA and Agile to Build Better Systems in DoD

By: NDIA Systems Engineering Division Agile in Defense/ADAPT

This presentation explores how the intersection of MOSA with Agile and DevOps practices help to increase adaptability, shorter delivery schedules, reduce costs through improved quality, and improve transparency of the systems current state. This presentation addresses practices and digital capabilities for delivering value in the shortest sustainable lead time.

2021 Reports

NDIA SED CTO Brief Architecture Committee MOSA 2021(Click to view the brief) 

On September 15th, 2021, The NDIA Systems Engineering Division (SED) Architecture Committee was invited to brief the A&D CTO forum on Modular Open Systems Architecture. The A&D CTO Forum are technical leaders across the U.S. Aerospace and Defense (A&D) industry responsible for the development, enhancement, and dissemination of technology within their respective organizations. During the bi-annual meeting between A&D CTO Forum leaders and guest speakers, the NDIA SED Architecture Committee discussed current, disruptive, and emerging technologies within Aerospace & Defense. 

Principal Attendees and Members of the A&D CTO Forum are:

  • Jim Carlini, Leidos
  • Naveed Hussain, Boeing Research & Technology
  • Ross Niebergall, VP and CTO, L3Harris Technologies
  • Cheryl Paradis, VP and GM, FAST Labs, BAE Systems
  • Mark Russell, CTO, Raytheon Technologies
  • Scott Stapp, Corporate VP and CTO Northrop Grumman
  • Steven Walker, VP and CTO, Lockheed Martin

National Defense September 2021 Issue Systems Engineering Division Published Article

Practical Software and Systems Measurement Continuous Iterative Development Measurement Framework

The collaborative PSM, NDIA, and INCOSE report provides recommendations for the measurement of continuous iterative developments (CID).  The report includes a Practical Software and Systems Measurement (PSM) CID measurement framework detailing common information needs and measures that are effective for evaluating CID approaches. The information needs address the team, product, and enterprise perspectives to provide insight and drive decision-making. The framework also identifies and specifies an initial set of measures that have been identified as being practical measures to address these information needs. This guidance is intended to be used by team, program, and enterprise personnel who are implementing CID approaches, as a reference for common, practical measures that  can be utilized. 

Parts: The report has now been separated into three parts.  Part 1 of this report includes a series of diagrams and an ontology to describe the development approaches and terminology used.  It also includes an “Information Category-Measurable Concept-Measures” (ICM) Table detailing potential information needs and measures for CID  developments. For the highest priority measures, sample measurement specifications have been developed that detail the identified measures. These are included in a  separate Part 2 of the paper, along with a discussion of how to use these measures for enterprise decision making.  Part 3 of the paper separately extends the main CID  paper with detailed information and guidance on Software Assurance and Technical Debt.

Version 1.05 detailed potential information needs and measures that are common to CID approaches, and an initial set of ten measurement specifications that were prioritized by user surveys as highest value. This Version 2.1 includes added material that has been researched and developed by the CID working group. The new materials  include information on measuring:

Continuous Iterative Development (Agile) Measurement Framework

We are currently looking for pilots in each of the services, to verify usage in the community, especially for the product value measure.  Please contact Cheryl Jones or Geoff Draper if you are interested in participating. 

Phase 3 of the project will begin soon, focused on the areas identified in part 1, section 6, including size and estimation, additional measures, and enhancements based on user pilots.

2020 Reports

Modular Open Systems Approach - Considerations Impacting Both Acquirer and Supplier Adoption
 
Systems Engineering Architecture Committee
 
July 1, 2020

The  Systems  Architecture  Committee provided this white paper containing recommendations to
government  and  industry  program  managers  and  acquisition  professionals  regarding  the
implementation  of  the  Modular  Open  Systems  Approach  (MOSA).  The
Architecture Committee consisting of a broad representation of industry, government, and the
Services, studied the various facets of MOSA (past and present) in the context of their professional
experience in order to provide practical guidance regarding the approach.

NDIA MOSA White Paper Final Release NDIA Architecture Committee 2020

2019 Reports

  • Implementing Continuous Iterative Development and Acquisition
    Defense Science Board (DSB) released a report in Feb-2018 containing seven recommendations regarding software design and acquisition.  Section 868 of NDAA 2019 mandates implementation of these recommendations within 18 months.  The Defense Innovation Board (DIB) Software Acquisition and Practices (SWAP) study group has also provided many insightful and largely compatible recommendations.

    NDIA, INCOSE and PSM support the DSB and DIB concepts and the opportunities they offer to DoD and the defense industry.  In 2018 the NDIA Systems Engineering Division commissioned a working group to study the implementation of the DIB and SWAP recommendations.  NDIA offers the working group's  recommendations below to OUSD(A&S) and OUSD(R&E) representing a broad “industry perspective” on a path forward.

    Executive SummaryExecutive Summary Brief (pdf)

Executive Summary Brief (ppt with links) view in presentation mode

Full Study Brief

Transmittal Letter to Dr. Boleng, OUSD (A&S)

2017 Reports

  • Top Five SE Issues
    The NDIA Systems Engineering Division determined that an update to the Top Issues in Systems Engineering (SE) report that was issued initially in 2003, updated in 2006 and again in 2010 was needed. The issues related to our defense industry are complex, affecting both the industry organizations as well as the government and military organizations.

    A Working Group was formed, inputs were solicited in advance, and a reconciliation meeting with 18 members was held on August 17th and 18th, 2016. Although numerous separate issues were identified, the group found that the bulk of these actually fell into several major issue categories. The detailed results, including the status of the previous 2006 and 2010 SE issues, are described in this report.

2016 Reports

  • A Path Towards Cyber Resilient and Secure Systems, April 2016
    Taking a holistic approach to system security requires bringing together multiple communities with rich histories introduces varying perspectives, terminologies, taxonomies and methodologies.  This diversity provides opportunities and challenges for evaluating the security quality system attributes. In this paper, we discuss a path towards system security metrics and measures.  Ultimately, we are committed to providing systems that are resistant to attack and resilient when under attack.

System Security Engineering Committee Chairs: Holly Dunlap, Raytheon & Beth Wilson, Raytheon
Developmental Test & Evaluation Committee Chair: Joe Manas, Raytheon
In collaboration with: INCOSE SSE Committee, Trusted Supplier Steering Group, and Mitre AF Cyber Integration

2015 Reports

  • The Business Case for Systems Engineering: Results of the Systems Engineering Effectiveness Survey
    This report summarizes the results of a survey that had the goal of quantifying the connection between the application of systems engineering (SE) best practices to projects and the performance of those projects. The survey population consisted of projects and programs executed by system developers reached through the NDIA, the IEEE Aerospace and Electronic Systems Society, and INCOSE. Analysis of survey responses revealed strong statistical relationships between project performance and several categories of specific SE best practices. The survey results show notable differences in the relationship between SE best practices and performance between more challenging and less challenging projects.
  • The Business Case for Systems Engineering: Detailed Response Data
    This report contains detailed response data from The Effectiveness of Systems Engineering: A Survey. The survey had the goal of quantifying the connection between the application of systems engineering (SE) best practices to projects and programs and the performance of those projects and programs. The survey population consisted of projects and programs executed by system developers reached through the NDIA, the IEEE Aerospace and Electronic Systems Society, and INCOSE. Analysis of survey responses revealed strong statistical relationships between project performance and several categories of SE best practices. The survey results show notable differences in the relationship between SE best practices and performance between more challenging and less challenging projects. The statistical relationship with project performance is quite strong for survey data of this kind when both SE capability and project challenge are considered together.

  • Report on Systemic Root Cause Analysis of Program Failures. National Defense Industrial Association Systems Engineering Division in conjunction with Office of Under Secretary of Defense Acquisition, Technology & Logistics, Systems &  Software Engineering, Arlington, VA, December 2008.

Systems Engineering

; ;