Asymmetric Logical Unit Access (ALUA)

ALUA

“A storage controller manages the flow of data between the server and the LUN, assigning two paths, in case one of the paths becomes unavailable.”

An Active controller is available at all times. A passive controller sits idle until the active controller becomes unavailable.

A dictionary definition of asymmetric is “not identical on both sides of a central line”. An Asymmetric Logical Unit Access (ALUA) suggests unequal paths between the server to the LUN.
ALUA is implemented on active/active controllers. There are two types of active active controllers:

• Asymmetric Active Active
• Symmetric Active Active

In an Asymmetric Active/Active storage controller architecture (also known as ALUA compliant devices), there is a path to the LUN via either controller and both controllers are defined as “active”, however only one of the paths is defined as an optimal (direct) path. This controller is also referred to as the preferred controller.

IO requests arriving at the preferred controller are sent directly to the LUN. The other path is unoptimized (indirect) and is available only as a standby path, in case the optimized path becomes unavailable. IO requests arriving at the non-preferred controller are first forwarded to the preferred controller before being sent to the LUN.

In a Symmetric Active/Active storage controller architecture, there are no preferred or primary controllers.

ALUA

• ALUA (Asymmetric Logical Unit Access)
• Is a SCSI standard.
• Typically implemented on mid-range storage arrays.
• The LUN is reachable across both storage processors at the same time
• All controllers are defined as “active”, however only one controller provides an optimal path to the LUN, this is the controller that owns the LUN
• Rebalancing across the controllers as workloads change is a manual task
• ALUA compliance required at the array and at the host multipathing layer
• Multipathing software can query ALUA compliant arrays to load balance and failover

With ALUA, any given LUN is accessible via both storage processors however only one of these storage processors owns the LUN. This owning controller provides the optimal path to the LUN. The other controller provides the unoptimized path to the LUN. Paths for the non-owning storage processor transit data across the internal interconnect architecture of the mid-range arrays.

Terms to consider: Optimized vs. unoptimized paths; Direct vs. indirect path; Storage processor; Owned; asymmetric active-active architecture vs. symmetric active-active architecture.

Reference:

• Hitachi Dynamic Virtual Controller Technology
• What’s That ALUA Exactly?

VMDirectPath I/O

VMDirectPath I/O

VMDirectPath I/O allows guest operating systems to directly access an I/O device, bypassing the virtualization layer. This direct connection frees up CPU cycles and improves performance for VMware ESXi hosts that utilize high-speed I/O devices, such as 10 GbE devices.

A single VM can connect to up to four VMDirectPath PCI/PCIe devices.

The disadvantages of using VMDirectPath I/O on a VM include:

• Unavailability or restrictions of vSphere features such as vMotion and DRS
• The adapter can no longer be used by any other virtual machine on the ESXi host

A known exception to this is when ESXi is running on Cisco Unified Computing Systems (UCS) through Cisco Virtual Machine Fabric Extender (VM-FEX) distributed switches.

DirectPath I/O allows virtual machine access to physical PCI functions on platforms with an I/O Memory Management Unit.

The following features are unavailable for virtual machines configured with DirectPath:

• Hot adding and removing of virtual devices
• Suspend and resume
• Record and replay
• Fault tolerance
• High availability
• DRS (limited availability. The virtual machine can be part of a cluster, but cannot migrate across hosts)
• Snapshots

"VMDirectPath I/O enables a virtual machine to directly connect to a physical device such as a network card or storage adapter."

In the case of networking, instead of using an emulated network device, such as E10000, VMDirectPath I/O enables the virtual machine to bypass the hypervisor and directly access a physical NIC.

Reference:

• VMware VMDirectPath I/O
• vSphere 6.0 Configuration Maximums
• vSphere 6.0 Documentation – DirectPath I/O

Claim Rule

Claim Rules

Multiple Multipath Plugins (MPPs) cannot manage the same storage device. As such, claim rules allow you to designate which MPP is assigned to which storage device. Each claim rule identifies the following parameters:

Vendor/model strings

• Transport, i.e. SATA, IDE, FC
• Adaptor, target, or LUN location
• Device driver

Claim rules are defined within /etc/vmware/esx.conf on each ESX/ESXi host and can be managed via the vSphere CLI.

Multipath policies (Fixed, MRU, RR) can be changed within vSphere, however any claim changes are conducted at the command line.

"The PSA uses claim rules to determine which multipathing module should claim the paths to a particular device and to manage the device. esxcli storage core claimrule manages claim rules.
Claim rule modification commands do not operate on the VMkernel directly. Instead they operate on the configuration file by adding and removing rules."

Claim Rule commands:

 List Claim Rules:

Claim rules are numbered as follows.

• Rules 0 – 100 are reserved for internal use by VMware.
• Rules 101 – 65435 are available for general use. Any third party multipathing plugins installed on your system use claim rules in this range.
• Rules 65436 – 65535 are reserved for internal use by VMware.

When claiming a path, the PSA runs through the rules starting from the lowest number and determines if a path matches the claim rule specification. If the PSA finds a match, it gives the path to the corresponding plugin.

A given path might match several claim rules.

Reference:

• PSA->MPP->SATP/PSP and well..you
• Managing Claim Rules

Fixed PSP

Fixed – VMW_PSP_FIXED

With the Fixed (VMW_PSP_FIXED) path selection policy, the host always uses the preferred path to the LUN when that path is available. If the host cannot access the LUN through the preferred path, it tries one of the alternative paths. The host automatically returns to the previously defined preferred path as soon as it becomes available again.

A preferred path is a setting that NMP honors for devices claimed by the VMW_PSP_FIXED path selection policy.

The first path discovered and claimed by the PSP is set as the preferred path.

This is the default policy for LUNs presented from an Active/Active storage array.

Fixed

• The default policy used with a SAN that is set to Active/Active
• Uses the designated preferred path whenever available
• If the preferred path should fail, another path is used until the preferred path is restored
• Once the preferred path is restored the data moves back onto the preferred path

If you want the host to use a particular preferred path, specify it through the vSphere Web Client, or by using esxcli storage nmp psp fixed deviceconfig set.

"NOTE If the host uses a default preferred path and the path's status turns to Dead, a new path is selected as preferred. However, if you manually designate the preferred path, it will remain preferred even when it becomes inaccessible."

Policy	Active/Active Array	Active/Passive Array
Round Robin	No fail back	Next path in round robin scheduling is selected
Most Recently Used	Administrator action is required to failback after path failure	Administrator action is required to fail back after path failure
Fixed	VMkernel resumes using the preferred path when connectivity is restored	VMkernel attempts to resume by using the preferred path. This action can cause path thrashing or failure when another SP now owns the LUN

When using this policy you can maximize the utilization of your bandwidth to the storage array by designating preferred paths to each LUN through different storage controllers. For optimal performance with these arrays you might also consider the Round Robin path policy.

Reference:

• vSphere 5.5 Storage
• vSphere 5.5 Performance Best Practices Guide

Most Recently Used (MRU) PSP

Most Recently Used (MRU) – VMW_PSP_MRU

“The VMW_PSP_MRU policy selects the first working path, discovered at system boot time. If this path becomes unavailable, the ESXi/ESX host switches to an alternative path and continues to use the new path while it is available.

This is the default policy for LUNs presented from an Active/Passive array.
ESXi/ESX does not return to the previous path if, or when, it returns; it remains on the working path until it, for any reason, fails.”

"If the active path fails, then an alternative path will take over, becoming active. When the original path comes back online, it will now be the alternative path."

MRU

• The ESXi host selects the path that it most recently used
• This is the default used with a SAN that is set to Active/Passive
• With this policy, a path is chosen and continues to be used so long as it does not fail
• If it fails, another path is used, and it continues to be used so long as it does not fail, even if the previous path becomes available again.
• The host does not revert back to the original path when that path becomes available again
• There is no preferred path setting with the MRU policy
• The policy is displayed in the client as the Most Recently Used (VMware) path selection policy

The VMW_PSP_MRU ranking capability allows you to assign ranks to individual paths. To set ranks to individual paths, use the esxcli storage nmp psp generic pathconfig set command.

If you want the host to use a particular preferred path, specify it through the vSphere Web Client, or by using esxcli storage nmp psp generic deviceconfig set.

Policy	Active/Active Array	Active/Passive Array
Round Robin	No fail back	Next path in round robin scheduling is selected
Most Recently Used	Administrator action is required to failback after path failure	Administrator action is required to fail back after path failure
Fixed	VMkernel resumes using the preferred path when connectivity is restored	VMkernel attempts to resume by using the preferred path. This action can cause path thrashing or failure when another SP now owns the LUN

For optimal performance with the arrays for which ESXi defaults to MRU you might also consider the Round Robin path policy.

Reference:

• VMW_PSP_MRU path ranking capabilities in ESXi 5.0
• Mastering VMware vSphere 5.5
• vSphere 5.5 Storage
• Performance Best Practices for VMware vSphere 5.5

Round Robin PSP

Round Robin - VMW_PSP_RR

The ESXi host uses an automatic path selection algorithm rotating through all active paths when connecting to active-passive arrays, or through all available paths when connecting to active-active arrays. On supported arrays multiple paths can be active simultaneously; otherwise, the default is to rotate between the paths.

"This is the only path selection policy that uses more than one path during a data transfer session. Data is divided into multiple paths, and the paths are alternated to send data. Even though data is sent on only one path at a time, this increases the size of the pipe and  allows more data transfer in the same period of time."

"Round Robin rotates the path selection among all available optimized paths and enables basic load balancing across the paths and fabrics."

Round Robin policy provides load balancing by cycling I/O requests through all Active paths, sending a fixed (but configurable) number of I/O requests through each one in turn.

If you want the host to use a particular preferred path, specify it through the vSphere Web Client, or by using esxcli storage nmp psp roundrobin deviceconfig set.

Policy	Active/Active Array	Active/Passive Array
Round Robin	No fail back	Next path in round robin scheduling is selected
Most Recently Used	Administrator action is required to failback after path failure	Administrator action is required to fail back after path failure
Fixed	VMkernel resumes using the preferred path when connectivity is restored	VMkernel attempts to resume by using the preferred path. This action can cause path thrashing or failure when another SP now owns the LUN

Reference:

• vSphere 5.5 Storage
• vSphere 5.5 Performance Best Practices Guide

Path Selection Plug-In (PSP)

PSP

Path Selection Plug-Ins (PSPs) are subplug-ins of the VMware NMP and are responsible for choosing a physical path for I/O requests.

“The VMware NMP assigns a default PSP for each logical device based on the SATP associated with the physical paths for that device."

"The Path Selection Plug-in (PSP) performs the task of selecting which physical path to use for storage transport. The NMP assigns a default PSP from the claim rules based on the SATP associated with the physical device."

Since multiple MPP’s cannot manage the same storage device, claim rules allow you to designate which MPP is assigned to which storage device.

One way to think of PSP is which multipathing solution you are using to load balance.
There are three Path Selection Plug-ins (PSPs) pathing policies included in vSphere:

• Fixed
• VMW_PSP_FIXED
• The host will use a fixed path that is either, set as the preferred path by the administrator, or is the first path discovered by the host during the boot process
• Default for active/active arrays
• MRU
• VMW_PSP_MRU
• The host will use the path that is most recently used (MRU). When a path fails and another one is activated, the host will continue to use this new active path even when the original path comes back up
• Default for active/passive arrays
• Default for ALUA devices
• Round Robin
• VMW_PSP_RR
• The host will use all active paths in a round robin (RR) fashion. It uses an algorithm to iterate through all active paths. The default number of I/Os that are issued to a particular path is 1000 before moving on to the next active/available path

No default array types are listed for this PSP.

Reference:

• vSphere 5.5 Storage
• Configure & Manage Complex Multipathing and PSA Plug-ins
• Multipathing policies in ESXi 5.x and ESXi/ESX 4.x

Storage Array Type Plug-In (SATP)

SATP

Storage Array Type Plug-Ins (SATPs) run in conjunction with the VMware NMP and are responsible for array-specific operations.

ESXi offers a Storage Array Type Plug-in (SATP) for every type of array that VMware supports in the VMware Hardware Compatibility List (HCL). It also provides default SATPs that support non-specific active-active and ALUA storage arrays, and the local SATP for direct-attached devices.

Each SATP accommodates special characteristics of a certain class of storage arrays and can perform the array-specific operations required to detect path state and to activate an inactive path. As a result, the NMP module itself can work with multiple storage arrays without having to be aware of the storage device specifics.

The SATP monitors the health of each physical path and can respond to error messages from the storage array to handle path failover. There are third-party SATPs that the storage vendor can provide to take advantage of unique storage properties.

After the NMP determines which SATP to use for a specific storage device and associates the SATP with the physical paths for that storage device, the SATP implements the tasks that include the following:

• Monitors the health of each physical path.
• Reports changes in the state of each physical path.
• Activates an inactive path
• Performs array-specific actions necessary for storage fail-over, e.g. activates passive paths.

Managing SATPs

The esxcli storage nmp satp list command lists the SATPs that are currently available to the NMP system and displays information about those SATPs:

Reference:

• vSphere 5.5 Storage
• PSA – Deep Dive Part III 

Native Multipathing Plugin (NMP)

Native Multipathing Plugin (NMP)

Native Multipathing Plugin (NMP) is a generic VMkernel Multipathing Plugin (MPP) provided by default from VMware and built into ESX/ESXi. NMP is used when the storage array does not have a third-party MPP solution.

VMware provides a generic Multipathing Plugin (MPP) called Native Multipathing Plugin (NMP).

What does NMP do?

• Manages physical path claiming and unclaiming
• Registers and de-registers logical storage devices
• Associates a set of physical paths with a specific logical storage device, or LUN
• Processes I/O requests to storage devices:
• Selects an optimal physical path for the request (load balance)
• Performs actions necessary to handle failures and request retries
• Supports management tasks such as abort or reset of logical storage devices.

NMP is an extensible module that manages subplugins: Storage Array Type Plugins (SATPs) and Path Selection Plugins (PSPs).

Storage Array Type Plugins (SATPs) is responsible for handling path failover for a given storage array. The appropriate SATP for an array you use will be installed automatically.

Path Selection Plugins (PSPs) determines which physical path is used to issue an I/O request to a storage device. SATPs and PSPs are sub plug-ins within the NMP module.

A Storage Array Type Plugin (SATP) determines how path failover is handled for a specific storage array.

SATPs and PSPs can be built-in and provided by VMware, or can be provided by a third party.

If more multipathing functionality is required, a third party can also provide an MPP to run in addition to, or as a replacement for, the default NMP.

The VMware NMP supports all storage arrays listed on the VMware Hardware Compatibility List (HCL).

You can use esxcli storage nmp to manage devices associated with NMP and to set path policies.

"VMware NMP" namespace:
esxcli storage nmp device list
The list command lists the devices controlled by VMware NMP and shows the SATP and PSP information associated with each device:
“One of the tasks of NMP is to associate physical storage paths with an SATP and associate a PSP that chooses the best available path. NMP provides a path selection algorithm based on the array type.“

VMware NMP Flow of I/O
When a virtual machine issues an I/O request to a storage device managed by the NMP, the following process takes place.

• The NMP calls the PSP assigned to this storage device
• The PSP selects an appropriate physical path on which to issue the I/O
• The NMP issues the I/O request on the path selected by the PSP
• If the I/O operation is successful, the NMP reports its completion
• If the I/O operation reports an error, the NMP calls the appropriate SATP
• The SATP interprets the I/O command errors and, when appropriate, activates the inactive paths
• The PSP is called to select a new path on which to issue the I/O
• If the I/O operation is successful, the NMP reports its completion

Reference:

• What is Pluggable Storage Architecture (PSA) and Native Multipathing (NMP)?
• vSphere 5.5 Storage
• Managing NMP with esxcli storage nmp
• PSA Deep Dive – Part II
• VMware NMP I/O Flow

MPP - Multipathing Plug-in

MPP - Multipathing Plug-in

The top-level plug-in in Pluggable Storage Architecture (PSA) is the Multipathing Plug-in (MPP). The MPP can be either the internal MPP, which is called the Native Multipathing Plug-in (NMP), or a third-party MPP supplied by a storage vendor. In ESXi storage is accessed through a VMware built-in MPP (NMP) or a third-party MPP.

VMware’s Native Multipathing Plug-in is also a MPP.

The process for connecting a storage array to VMware includes:

• Check the VMware Hardware Compatibility List (HCL) to determine if it is a supported array
• Use the built-in NMP to handle multipathing and load balancing, if in the HCL
• Use a supported third-party MPP, if there is need for additional functionality  provided by the MPP

Third-party MPP solutions such as Symantec DMP and EMC PowerPath/VE, replace the behavior of the NMP, SATP, and PSP. It takes control of the path failover and the load-balancing operations for specified storage devices. Third-party MPPs might provide better load-balancing performance than the built-in NMP solution.

The MPP module replaces the NMP, SATP, and PSP.

“The MPP can change the path selection normally handled by the PSP. As a result, it can provide more sophisticated path selection, notable performance increases or other new functionality not present in vSphere by default."

MPPs can coexist with NMP.

A storage path is a possible route from an initiator to a given LUN through which the I/O may travel. A path can be in one of the following states:

Path State	Description
Active	A path via an Active storage controller. The path can be used for I/O. Operational and in use.
Standby	A path via a Passive or Standby storage controller. The path can be used for I/O if active paths fail. Operational but not in use.
Disabled	Path has been administratively disabled, usually by the vSphere Administrator.
Dead	Path cannot be used for I/O.
Unknown	Path is in unknown error state.

Reference:

• VCAP5-DCA Official Cert Guide
• Mastering VMware vSphere 5.5
• SCSI Path Activity State