Department of Defense
High Performance Computing Modernization Program


This Software-Defined Overview briefly discusses the software architecture, or emerging architecture, for Software-Defined Networking (SDN) and other Software-Defined technologies. A software architecture is an architecture that has been adopted by at least one standards development organization (SDO). An emerging architecture is an architecture that has been adopted by at least one software development organization (SWDO) but has not yet been standardized. An SWDO is an organization formed by one or more interested individuals, non-commercial organizations and commercial companies. SDOs are similar to SWDOs in many organizational aspects, but their purposes are quite different. The purpose of an SDO is to develop, publish and maintain standards within a defined scope of responsibility. One purpose of an SWDO is to develop, publish, maintain and/or promote use of software within a defined scope of responsibility, and (for a few SWDOs) another purpose is to function as an SDO for that software. An SWDO may have additional purposes that are not limited to software development. The scope of an SDO or SWDO may be self-defined or it may be defined by a higher level organization.

There are 5 layers in the Internet Protocol suite (also known as TCP/IP) that defines the conceptual model for the Internet. Updating the networking infrastructure at an existing Internet site to include SDN along with associated software architectures, or deploying the networking infrastructure for a new Internet site using SDN and associated software architectures, does not require any changes at the Internet Protocol (IP) or lower layers nor does it create any IP packet format incompatibilities. Consequently, a decades-long coordination effort between and among interconnected sites on the Internet is not needed (in stark contrast to the on-going world-wide IPv4 to Internet Protocol version 6 (IPv6) Internet transition, where such coordination efforts are essential). Such updates and deployments are already taking place at individual sites and can continue to occur in the United States and around the world in an uncoordinated fashion.

Because IPv4 has been the only IP for many years, existing software supports IPv4 (although this may not always be the case for software developed after 2016). Unless otherwise noted, specific software products mentioned in this Overview support IPv4 and have been informally determined, based on users’ experiences and developers’ statements, to also support IPv6. (Such lack of support for IPv6 by SDN software products is rare.) 

An added benefit of such updated or newly-deployed infrastructure is that it will be able to support IPv6, and it is important that SDN support access via IPv6, as this article explains.


What is a Software-Defined technology?

Consider any technology that involves both software and the virtualization of some physical computing, storage or networking/communications device(s). The nature and the name of that technology will change over time. For example, when the concept of a physical server was virtualized, the result might have been called a “Software-Defined Server” rather than a “virtual machine” but it was not to be (until years later).

A Software-Defined technology can also involve both software and the virtualization of data or programs apart from the physical device or devices on which the data is stored or on which the programs run. The article “What you need to know about Software-defined Technology?” provides an introduction to SDN, while the Internet Engineering Task Force (IETF) Request for Comment (RFC) 7426 Software-Defined Networking (SDN): Layers and Architecture Terminology provides a detailed technical description.


What is Virtualization?

The concept of virtualization is older than any of the software architectures or emerging architectures discussed in this Overview. For further details, see this review of virtualization technology evolution. As this article on technology trends observes:

The trend of virtualization is one of the most powerful forces in technology.

Virtualization is an integral part of any Software-Defined technology. Virtualization can be used to implement some (typically, but not necessarily) hardware-based features of that technology in software. Virtualization can be used to create an abstraction layer in software that conceals implementation details of one part of the technology from other parts of that technology, or from whomever or whatever is using that technology. For example, early in the evolution of the Software-Defined Networking Architecture, the control plane was an abstraction layer that concealed implementation details of the forwarding table residing in the data plane from the management plane.

During the last 60 years more and more physical information technology (IT) infrastructure resources have been virtualized. Physical computing resources have been virtualized, including memory, processors, displays, sensors and entire servers. Physical storage resources have been virtualized, including tape and disk drives, channels, storage subsystems and entire storage systems. Physical communications/networking resources have been virtualized, including switches, routers, gateways, firewalls and entire networks. All communications/networking hardware (not just the active resources such as switches, routers and gateways, but also the interconnection resources) can be virtualized. Even data can be virtualized.

More recently, almost all layers of software resources have become virtualized resources, including processor instruction sets, subroutines, applications software, operating systems software, and virtual desktop infrastructure (VDI) software. Complete computing center environments have even become virtualized resources.

Once something has been virtualized, it along with other virtualized resources typically:

(This article compares and contrasts the characteristics of virtual machines and containers.)


What is a Radio Access Network (RAN)?

A RAN is a good example of the progressive virtualization of a concept over time. First came the original physical radio access network (RAN), which soon became a virtual RAN (vRAN). Next, it became a “Software-Defined RAN (SD-RAN)”, then it became a “Cloud RAN (C-RAN)”, and then it became a RAN Edge. It will continue evolving, since SDOs and SWDOs such as the O-RAN Alliance, the Open Networking Foundation (ONF) and an umbrella organization called the Telecom Infra Project are actively developing emerging architectures called Open RANs, using disaggregated software (the term “disaggregated” has many different meanings, as described in this article). This article attempts to explain the differences among the various vRAN and Open RAN emerging architectures.

In a different direction, evolution from a public RAN to a private RAN is also on going.


Software-Defined Topics in this Overview:


Software-Defined Networking Architecture

While the adage timing is everything does not always hold true, it has been true during the evolution of the SDN architecture. While there have been previous efforts to change router and switch architectures, the SDN architecture effort has been the most successful. The SDN architecture as originally envisioned (circa 2007) included:

  • A data plane: Hardware that actually receives and forwards IP packets, that could be physically separated from
  • A control plane: Software in this abstraction layer monitors the forwarding tables residing in the data plane and propagates changes in routing policies to the forwarding tables, and
  • A management plane: Software that maintains the routing policies.

This introduction of an intermediate control plane to propagate changes in the routing policies to multiple forwarding tables was a significant addition during the early evolution of the SDN architecture. The control plane provided the management plane with a single, cohesive abstraction of multiple (possibly dissimilar) forwarding tables in the data planes performing distributed IP packet handling. This enabled the centralized management plane to dynamically determine global network behavior from a single location by maintaining the global routing policies. The separation of mechanism (IP packet handling) from policy (routing policies) is a well-established design principle in computer science.

Early SDN technology (circa 2007)   More recent SDN architecture (circa 2012)

Early SDN technology (circa 2007)                        More recent SDN architecture (circa 2012)


By 2012, the SDN architecture had evolved to include the following elements:

  • A management plane that became part of the Application Layer: Software called Business/Network Applications control the behavior and performance of a network. One way that Applications do this is by using the services of modules to maintain the routing policies. An Application does not offer any APIs to other Applications or to modules. The functionality of Network Applications modules included in different SDN implementations varies widely. The functionality of Business Applications modules is determined by the customer.
  • A control plane that became the Control Layer: Software called Network Service modules in this abstraction layer performs one or more services, providing one or more Application Program Interfaces (APIs), allowing modules in the Application or Control Layers to make use of said services and returning one or more results. Some of the services these modules perform include monitoring the forwarding tables residing in multiple networking devices, and propagating changes in routing policies to those forwarding tables. Even though the Network Service modules that make up a Control Layer are installed on one or more physically or virtually separate SDN controllers, the modules function as though they were installed in a common location. This list of commercial SDN controllers and this list of open source SDN controllers mention several controllers available circa 2018. (Since that time, both lists have grown shorter for reasons described in this article.)
  • A data plane that became the Infrastructure Layer: Upon matching a conditional entry in a forwarding table for a received IP packet, a networking device performs one or more actions, possibly including forwarding the IP packet. The forwarding tables residing in each device are structured to meet that device’s specific requirements. A device may be either physical or virtual. A device manufactured by an original equipment manufacturer may be referred to as a white box. Prior to 2007, the operating system (O/S) for a device was usually provided by the manufacturer of the device. More recently, the O/S for an “open network” (also referred to as an “open networking”) device is sometimes provided by an SWDO rather than by the manufacturer.

The evolution of the SDN architecture did not cease in 2012. If anything, it accelerated! For example, see this SDN Architecture for the Internet-of-Things (IoT) developed in 2014 by the Multinetwork INformation Architecture (MINA) research project. It proposed an extension of the SDN architecture to support the IoT. For additional details about the evolution of the SDN architecture, see this 2014 review.

And this evolution will continue – see the Emerging Software-Defined Everything Architectures topic below. As shown by the number of SDN open source communities in the Points of Contact article, this continuing evolution can lead to confusion when implementing SDN.

In the SDN architecture:

  • The Control and Application Layers communicate by means of “Northbound APIs” (NBIs). RESTful and RESTCONF are some examples. This diagram shows an example of communications between the Control and Application Layers using the RESTful NBI. The ONF initiated the OpenSourceSDN initiative in 2015 (no longer active, but website archived in 2017). NBIs developed by that initiative are available on GitHub.
  • The Control and Infrastructure Layers communicate by means of “Southbound APIs” (SBIs). OpenFlow, P4, or NETCONF are some examples.
  • Data modeling languages such as Yet Another Next Generation (YANG) can facilitate the use of NBIs and SBIs.
  • For communications within the Infrastructure Layer (originally called the Data Plane), several APIs and API implementations have been standardized, including the OpenDataPlane (ODP) and the Data Plane Development Kit (DPDK).

Different SWDOs may use different terminology to describe the elements of their SDN architecture. A Layer in one may be analogous to a Plane or a Tier in another, or even have no analogue. For additional details about terms in the SDN architecture, refer to this glossary. In the future, the scope of the SDN architecture could expand to include security – see the Software-Defined Security Architecture topic below.



Network Functions Virtualization Software Architecture

The realization that physical network function components (PNFCs) (e.g., firewalls, load balancers or other network elements) could be virtualized was the foundation for the “Network Functions Virtualization (NFV)” software architecture developed by the European Telecommunications Standards Institute (ETSI) Industry Specification Group for NFV (ISG for NFV). For further details, see this three part report from 2018 on the NFV software architecture:

Part 1: Foundations of NFV: NFV Infrastructure (NFVI) and VIM,
Part 2: Rising Roles of MANO, LSO, and Assurance, and
Part 3: State of the VNF Ecosystem.

Some industry observers worry that NFV promises too much too soon, as in this Does NFV Stand For Not Financially Viable? article. 



NFV software architecture (circa 2013)


An NFV software architecture includes the following elements:

  • “NFV Infrastructure (NFVI)”: An interconnected collection of IT infrastructure resources including virtual and physical computing devices, storage devices, networking devices, network functions and sensors. This collection need not be collocated. If the collection resides at more than one location (whether physical or virtual), the virtual and physical networking resources interconnecting those locations are included in the NFVI. It is neither assumed nor required that the NFVI uses Software-Defined technologies. In 2019 the Linux Foundation (The LF) and the Groupe Spécial Mobile (GSM) Association (GSMA) SWDO developed an NFVI reference architecture, called Botrange.
  • Virtual/Virtualized Network Function (VNF)” or "Cloud-Native Network Function (CNF)": A collection of VNF Components (VNFCs) and PNFCs implementing an external network service delivery policy for an NFVI. The properties of each resource in the collection are specified in a VNF Descriptor (VNFD). A VNFD is a template that specifies the operational capabilities, deployment requirements and management characteristics of a VNFC or PNFC,
  • NFV MANagement and Orchestration (NFV MANO)” or Cloud NFV includes:
  1. Orchestrator: configures and maintains an internal resource allocation policy, an external network service delivery policy and the VNFDs for an NFVI. For example, when multiple NFVIs are running in physically or virtually separate Docker containers, an orchestrator can be used to manage them, as explained by this articleWhat an Orchestrator is not is also important.
  2. VNF Manager(s): abstraction layer that instantiates, manages, and terminates VNFs or CNFs in accordance with an external network service delivery policy and VNFDs contained in the VNFs and CNFs of the NVFI.
  3. Virtualized Infrastructure Manger(s) (VIM): abstraction layer that manages the IT infrastructure resources of the NFVI in accordance with an internal resource allocation policy.

Approaches to controlling interconnected collections of IT infrastructure resources are proliferating. One example is the emerging Software-Defined Interconnect (SD-IX, sometimes called SDI) architecture. (Do not confuse the emerging SD-IX architecture with an implementation of the emerging SDDC architecture called an SDI mentioned in the Software-Defined Data Center topic below.)

SWDOs and researchers typically employ the use case methodology of systems analysis to identify and clarify implementation requirements and discuss anticipated benefits of specific cases of SDN architecture use or of NFV software architecture, as in this SDN and NFV use cases taxonomy and this NFV use cases taxonomy. Some SWDOs and researchers may also apply the user story methodology.

For further details about terminology in the NFV software architecture, refer to the ETSI Group Specification NFV 003 Terminology for Main Concepts in NFV. In the future, the scope of the NFVI software architecture could expand to include security – see the Software-Defined Security Architecture topic below.


Software-Defined Internet Exchange Point

One network function that can be virtualized is an “Internet Exchange Point (IXP)”. An IXP using SDN and possibly other Software-Defined or emerging architectures is called a “Software-Defined IXP (SDX)” or an industrial-scale SDX (iSDX).


Software-Defined Security Architecture

Software-Defined Security (SDSec)” is an emerging architecture under consideration for adoption by SDOs and SWDOs, and implementation by various companies. As envisioned by the ONF in an SDN context, the scope of Applications could be expanded to include configuring and maintaining an external security policy, while Network Service modules could implement that security policy. As envisioned by ETSI in an NFV context, the scope of the Orchestrator in the NFV MANO element could be expanded to include configuring and maintaining an external security policy for an NFVI, while VNFs could implement that security policy. SDSec is also under consideration as an element of the SDx Infrastructure emerging architecture.

Alternative SDSec software and emerging architectures include Software-Defined Perimeter (SDP) and the more recent Zero Trust Network Access (ZTNA) or Zero Trust Security Model (ZTSM), Software-Defined Protection, and MicroSegmentation (which originated in the context of the Software-Defined Data Center) and related emerging architectures such as NanoSegmentation, which is mentioned in passing by this Gartner blog entry.

Meanwhile, companies continue to develop and market software packages using phrases such as Security Information and Event Management (SIEM), Security Orchestration, Automation, and Response (SOAR), Endpoint Detection and Response (EDR), Extended Detection and Response (XDR), Open XDR, and Network Traffic Analysis (NTA) to describe them, sometimes vaguely.

In SDSec software and emerging architectures, security policy implementation could be separated from security policy management in a manner analogous to the distributed routing policy execution, with centralized maintenance of the routing policies in the SDN architecture.


Other Network Virtualization Architectures

In conjunction with the ongoing evolution of the NFV architecture, many “Network Virtualization (NV)” software and emerging architectures are appearing with abstraction layers that conceal (but do not change) the implementation details of the network resources from the network configuration, management or security software, offering hope that network automation may someday become a reality. Just which implementation details of the network resources are concealed and the functionality of the network configuration, management, or security software vary widely, depending on the specific software or emerging architecture.

The earlier Wide Area Network (WAN) routing technology MultiProtocol Label Switching (MPLS) continues to be widely deployed, sometimes in conjunction with NV software and other “Software-Defined WAN (SD-WAN)” emerging architectures. This article describes some of the benefits that SD-WAN provides. In some WAN NV software and emerging architectures, SD-WAN may even be present together with MPLS rather than replacing it, as this article describes.

SD-WAN is sometimes described as an “overlay-based SD-WAN” (or “tunnel-based SD-WAN”, in contrast to another emerging architecture, the “tunnel-free SD-WAN”). Other SD-WAN emerging architectures include "universal SD-WAN (uSD-WAN)", “open source SD-WAN”, "Secure Access Service Edge (SASE)", in contrast to another emerging architecture, the “Flexible Service Edge (FSE)”, “Intent Based Networking (IBN)”, “Network-as-a-Service (NaaS)”, and “Virtual WAN (VWAN)”. This article compares the SD-WAN and SASE emerging architectures.

Open source software implementations of SD-WAN architectures are also beginning to emerge, and in July 2019 the Metro Ethernet Forum (MEF) SWDO released an SD-WAN Services Standard document, defining standard Edge, Controller and Orchestrator functions. (Note: there are multiple definitions of “edge’ computing.)

Implementations of virtual/virtualized “Evolved Packet Core (vEPC)” and virtual/virtualized “IP Multimedia Subsystem (vIMS)” WAN NV, together with network slicing and dynamic network slicing emerging architectures are offered by members of the “3rd Generation Partnership Project (3GPP)”. Additional early WAN NV emerging architectures are listed in this article.

Originating at the Local Area Network (LAN) level, but not confined to that level, “virtualized Customer Premises Equipment (vCPE)” and "universal CPE (uCPE)" NV emerging architectures are discussed in this SD-WAN and virtualized CPE ReportSD-Branch NV emerging architectures are a further evolution of SD-WAN for various use cases. SD-LAN emerging architectures are an alternative approach. LAN NV architectural elements such as VNFCs and PNFCs may also be found in other emerging architectures such as the Software-Defined Data Center.

Independently of the ongoing evolution of the NFV architecture, virtualization is being incorporated into LAN architectures. Examples include “Virtual LAN (VLAN)”, “Virtual eXtensible LAN (VXLAN)”, “Network Virtualization using Generic Routing Encapsulation (NVGRE) LAN” software architectures, and the many “wireless LAN (WLAN)” emerging architectures.

Emerging architectures for both physical and virtual networking switch fabrics continue to evolve, as part of a quixotic quest for an SD-WAN that is an all-encompassing Fabric for Universal Networking (FUN). Seriously.


IT Infrastructure Virtualization Architectures

Abstraction layers can be introduced anywhere in an IT infrastructure. If an abstraction layer is introduced to conceal implementation details in an IT infrastructure from the customer interface to that IT infrastructure, it may result in a Software-Defined Data Center or a Software-Defined Workspace. If an abstraction layer is introduced to conceal implementation details of some applications in an IT infrastructure from the customer interface to those applications, it may result in a Software-Defined Architecture. If an abstraction layer is introduced to conceal details of some storage resources in an IT infrastructure from the application interface to those storage resources, it may result in Software-Defined Storage. If an abstraction layer is introduced to conceal details about the attributes of data, or details about the storage resources of an IT infrastructure where data resides, from the applications that access that data, it may result in Software-Defined Data.


Software-Defined Data Center

Various SWDOs are developing “Software-Defined Data Center (SDDC)” emerging architectures. Broadly speaking, an SDDC is a data center implemented using virtualization together with one or more Software-Defined software or emerging architectures, and there are multiple nomenclatures describing different SDDCs. SDDC emerging architectures can be called Software-Defined Data Infrastructures (SDDIs), Software-Defined Infrastructures (SDIs), Software-Led Infrastructures (SLIs), or Platform-as-a-Service (PaaS) (among other names) by the SWDOs developing them. 

Multi-cloud using SD-WAN is an SDDC emerging architecture that combines an SD-WAN emerging network virtualization architecture (several were mentioned above in the Other Network Virtualization Architectures section) with cloud computing to create an SDDC.

An SDDC may include a combination of both physical and virtual: discrete, composableconverged and hyper-converged infrastructure (HCI) resources. This article explores the differences between composable and HCI resources. A composable infrastructure can even be built with composable fabric, as described by this article. An SDDC may even include disaggregated HCI resources (the term “disaggregated” has several meanings, as described in this article).

The customer interface is often called “Infrastructure-as-a Service (IaaS)”. An SDDC is implemented using a combination of local IT infrastructure, fog computing, edge computing and a refinement of edge computing called multi-access edge computing (MEC, originally called mobile edge computing) resources. An Open Edge Computing Initiative SWDO is working to advance edge computing technology. An SDDC implemented using a combination of MEC and HCI could be even better. Just to be clear, edge computing and MEC are not the same thing, as this article explains. (Note: there are multiple definitions of “edge’ computing.)

According to this article, based on the IT resources included in an SDDC, an SDDC can be categorized as either:

  1. discrete, physical servers (bare metal) plus other physical IT resources,
  2. virtualized servers (running on hypervisors) plus physical IT resources,
  3. HCI (running on hypervisors), or
  4. HCI (running on hypervisors, but adding containers to the mix of IT resources included, and called an Application-Defined Infrastructure (ADI) by some SWDOs).

This Open Glossary of Edge Computing defines terms used when discussing edge computing. For further details about fog-computing and MEC, see this article. The ETSI ISG for MEC and ISG for NFV are working to enable deploying MEC resources in NFV infrastructures.

An abstraction layer is introduced to conceal various details (which vary with the SWDO) of the internal resource allocation policy and the intrinsic (possibly device-specific) orchestration, management, and provisioning characteristics of the resources in the SDDC.

Different SWDOs may use different terminology to describe their different SDDC emerging architectures.


Software-Defined Workspace/Workplace

Software-Defined Workspace (SDW)” (also called Workspace-as-a-Service or an integrated workspace platform) is an emerging architecture. It can be difficult to differentiate from another emerging architecture, Software-Defined Workplace (also called Workplace-as-a-Service). These emerging architectures employ multiple abstraction layers in order to support the use of any type of application on any platform from anywhere via any network.

Different SWDOs may use different terminology when describing the elements of these emerging architectures.


Software-Defined Architecture

Software-Defined Architecture (SDA)” is an emerging architecture in which an abstraction layer introduces outer APIs (NBIs) and inner APIs (SBIs) for selected applications. The NBIs conceal the implementation details of those applications and their run-time environment from external customers/users. The SBIs conceal these details from other applications. This makes it possible to transparently modify or replace the applications or their environment. The SDA emerging architecture is mentioned in this International Data Corporation article.

SDA is a further evolution of the earlier “Service-Oriented Architecture (SOA)” software architecture. While SDA applications are sometimes called microservices by SWDOs, not every SOA application should be called a microservice for the reasons given in this article. Different SWDOs may use different terminology to describe the abstraction layer of their SWDO emerging architecture.


Software-Defined Storage

Software-Defined Storage (SDS)” emerging architectures separate virtual and physical data storage resources from the software that manages those storage resources, in a manner similar to the separation of the data plane from the management plane in the early evolution of SDN architecture. This article compares and contrasts SDS emerging architectures with the earlier software architectures: Storage Area Networks (SAN) and Network-Attached Storage (NAS). Various SWDOs are developing SDS emerging architectures, such as this initiative by the Storage Networking Industry Association and the Virtual SAN (VSAN) offerings by Cisco Systems, Inc.

An SDS implementation is typically transparently modifiable while on-line and in-use. For example, without disrupting availability or performance, storage capacities can be increased or decreased, or individual storage resources can be repaired, upgraded or replaced.

This article argues that 5 key attributes must be part of any SDS software or emerging architecture. Different SWDOs may use different terminology to describe their SDS emerging architectures which do not always include all 5 attributes.


Software-Defined Data

Could “Software-Defined Data (SDD)” become the next step in the evolution of SDS? Implementations of this emerging architecture by SWDOs, including DataSphere (ceased operation in 2018) and ViPR (originally developed by EMC Corp, now marketed by DELL Inc), are mentioned in this International Data Corporation Community article.

This emerging architecture would use a combination of data virtualization and structural metadata to create an abstraction layer that separates data from some of its attributes. These data attributes can include format, storage media, access protocol (IPv4 or IPv6) and physical locality. In the SDD emerging architecture, data would be accessible at any time; appear to reside on any storage media; and appear to be accessible using any protocol, whatever the actual location of the data, the storage medium it resides on, the protocol used to access it, the intended use of that medium (such as on-line, archival or disaster recovery), or the format of the data on the storage resource where it resides (all of which could transparently and automatically be changed over time in response to data access patterns).


Emerging Software-Defined Everything Architectures

Software-Defined technologies continue to evolve. The “Software-Defined Everything" (SDx, originally called “Software-Defined Anything”) emerging architecture is the basis for the inclusive SDx Infrastructure (SDxI). For further details, refer to this eight-part series of articles. Network performance management (NPM) elements of the SDx architecture are defined in this article and application performance management (APM) elements of the SDx architecture are defined in this article.

Note: Do not confuse the SDx emerging architecture with the software-defined network function SDX.