What Is Non-volatile Memory Express Over Fibre Channel (NVMe Over FC)?
What is non-volatile memory express over Fibre Channel (NVMe over FC)?
Non-volatile memory express over Fibre Channel (NVMe over FC) is a technology specification designed to enable NVMe-based message commands to transfer data and status information between a host computer and a target storage subsystem over a Fibre Channel network fabric. NVMe over FC is implemented through the Fibre Channel-NVMe (FC-NVMe) standard.
Fibre Channel is a fabric transport option for NVMe over fabrics (NVMe-oF), a specification developed by NVM Express Inc., a nonprofit organization with more than 100 member technology companies. Additional NVMe transport options include remote direct memory access (RDMA) over Ethernet and InfiniBand.
NVM Express Inc. published the 1.0 version of the NVMe-oF specification on June 5, 2016. This specification has been superseded by newer versions of the NVMe family of standards. The current version is NVMe 2.0, released in 2021. The most recent update, 2.0d, was released in 2024. Some of the standards in the NVMe family include the NVM Express Base Specification, NVM Command Set Specification, RDMA Transport Specification, TCP (Transmission Control Protocol) Transport Specification and NVMe over PCIe (Peripheral Component Interconnect Express) Transport Specification.
The T11 committee of the International Committee for Information Technology Standards (INCITS) defined a frame format and mapping protocol to apply NVMe-oF to Fibre Channel. The T11 committee finalized the first version of the FC-NVMe standard in August 2017 and submitted it to INCITS for publication. The standard supports the following transport protocols: Fibre Channel, TCP, Ethernet and InfiniBand.
How NVMe over FC works
The FC protocol enables the mapping of upper-layer transport protocols, such as NVMe, Small Computer System Interface (SCSI) and IBM’s proprietary Fibre Connection (FICON). This enables the transfer of data and commands between a data center host computer and a peripheral target storage device or system that might communicate using storage area networks (SANs).
Compared with SCSI and FICON, NVMe has a streamlined register interface and command set, reduces the input/output (I/O) stack’s central processing unit (CPU) overhead, lowers latency and improves performance. NVM Express Inc. developed NVMe for use with fast media, including solid-state drives (SSDs) and other types of flash storage and memory-based technologies.
By contrast, the SCSI command set was designed at a time when slower hard disk drives (HDDs) and tape were the primary storage media, and FICON was created to connect mainframe computer and storage devices.
The FC-NVMe standard simplifies the NVMe command sets into basic FC protocol instructions. Because Fibre Channel is designed for storage traffic, functionality such as discovery, management and end-to-end qualification of equipment is built into the system.
A major distinction between NVMe over Fibre Channel and NVMe is the mechanism for transporting commands. NVMe maps requests and responses to shared memory in the host computer using the PCIe interface protocol. NVMe over FC uses a message-based model to send requests and responses between a host and a target storage device over a network.
Large-scale block flash-based storage environments that use Fibre Channel are the most likely to adopt NVMe over FC. It offers the same structure, predictability and reliability characteristics for NVMe that Fibre Channel uses for SCSI. Plus, NVMe traffic and traditional SCSI-based traffic can run simultaneously on the same FC fabric.
In practice, when a host computer wants to communicate with a storage device, a server launches the process. The FC switch then identifies a start-of-frame packet and launches a session between the user and the target over Fibre Channel. This ensures no frames are lost during transmission.
Infrastructure components that must support NVMe over FC to enable the potential benefits include the storage operating system and network interface cards (NICs). FC storage system vendors must qualify their products with the FC-NVMe protocol. Vendors with host bus adapters that support FC-NVMe include Broadcom and Marvell Technology. Broadcom and Cisco are among the leading FC switch vendors.
Who uses NVMe over FC?
Use cases where NVMe over FC makes sense include the following:
- Financial firms that require high-speed data transfers with low latency.
- Companies that use industrial control systems for capturing activity data and delivering it to a host system for analysis.
- Organizations that use artificial intelligence technology and business analytics.
- Organizations with large data volumes, large-scale storage arrays, and high-volume transaction and analytical processing, and whose requirements include rapid access to data, low latency and minimal data loss.
Advantages and limitations of NVMe over FC
The benefits of NVMe over FC include higher performance, scalability, lower latency and parallel I/O in transferring data to and from SSDs using the NVMe command set. This is in comparison to data transfer with the SCSI command set to and from HDDs or Serial ATA or Serial-Attached SCSI SSDs.
NVMe over FC also improves performance of I/O operations per second, or IOPS; completes faster and more efficient connections; and lowers CPU utilization. Use of traditional protocols also helps during migration to NVMe without the need for major architectural changes to network infrastructures.
One disadvantage of NVMe-based SSDs can be higher cost, but the price of NVMe SSDs is consistent with traditional SSDs. Technical expertise is needed for NVMe networks to accommodate the specialized devices that are used. In addition, NVMe over FC doesn’t support multipath, which establishes multiple paths between CPUs and data storage devices. Workarounds are available to ensure the availability and reliability of NVMe-FC infrastructures.
Ethernet- vs. InfiniBand- vs. FC-based NVMe
Comparing NVMe over FC with Ethernet- or InfiniBand-based NVMe alternatives generally takes into consideration the advantages and disadvantages of the networking technologies. FC fabrics are noted for their lossless data transmission, predictable and consistent performance, and reliability. Large enterprises tend to favor FC storage for mission-critical workloads. But Fibre Channel requires special equipment as well as SAN and other storage networking expertise to operate and can be more costly than Ethernet-based alternatives.
Ethernet is the ubiquitous networking technology, so Ethernet-based NVMe storage products are more plentiful than NVMe-over-FC options. Most storage startups focus on Ethernet-based NVMe and sometimes incorporate proprietary technology to get their products to market faster.
InfiniBand-based NVMe works well for high-performance computing workloads that require exceptionally high bandwidth and low latency. InfiniBand networks are typically used for communication within back-end storage systems rather than host-to-storage communication. Like FC, InfiniBand is a lossless network requiring special hardware. It offers advantages such as flow and congestion control, as well as quality of service.
Unlike FC, both InfiniBand and Ethernet lack a discovery service that enables the automatic addition of nodes to the fabric.
NVMe over FC going forward
NVMe as a protocol continues to grow in popularity and deployment. Use of FC as the transport medium ensures workloads can be handled quickly and efficiently. It also reduces the need for specialized devices, such as smartNICs, and provides a platform that achieves high levels of performance, availability and reliability.
Find out how new command sets enable computational storage with NVMe flash and frameworks to use local memory to process workloads on storage hardware.