Archive for June, 2010

Nexus

A Nexus, in the world of Storage Networking is a unique identifier.  It identifies the components involved in a command or sequence of commands.  It is normally made up of the following parts:

Initiator (I), Target (T), LUN (L), and Queue Tag (Q).  This is an ITLQ nexus.

Establishing the Nexus

In parallel SCSI (and most storage systems), everything starts on the host side of the system.  When a host wants move some data (do a READ or WRITE) it will need to know that a storage device exists, and then build and send a command to it.  But there is another SCSI protocol that takes place to connect and issue the command.  That protocol is below.

• Arbitration - initiator ID is placed on the bus (usually #7).
• Selection - target ID is place on the bus and target takes control.  At this point, an I_T nexus exists.
• Message(s) – target asks for bytes to identify the LUN and Q-Tag (if needed).  This is where the full I_T_L_Q nexus is established.
• Command - CDB is sent to the device, which processes the command and controls the flow.
• Data - this is the data movement phase (IN for READ, OUT for WRITE).
• Status - information is sent back to the host (command worked or had an error).
• Message - this tell the host that the command is done.

Once the command is finished, the nexus is “released” and can be used again.  This is because the nexus is only needed while the command is in progress.

Disconnect and Reconnect

The protocol shown above shows a single connection, where everything is done during that connection. But where the nexus really comes into play is when there is a disconnect and reconnect after the command had been issued.  This is one of the biggest benefits of “networked” storage.  While the storage devices are disconnected, the host can connect to another device and start a different command with it.

The important part is that while a command is still in process (no status yet), the nexus remains and is known to the host.  The storage device also can do “slow stuff” while disconnect, such as moving the heads, spinning the platters, that kind of thing.  When the device has the data ready (READ), or has resources to accept more data (WRITE), then it will use the same nexus values (I_T_L_Q) and will reconnect to the original host to continue the command sequence.  This can happen multiple times for a single command, and the nexus remains until the command is complete.

Addresses & Identifiers

Addresses and Identifiers are some of the most important parts of the Storage Networking world.  These are the parts of the system that allow use to identify what entity we are trying to connect to an communicate with. If you look at the definition of “nexus” in the SAN, it is all about the identifiers of the components.

Address vs. Identifier

I just want to be a little picky here, for just a moment about terminology.  Some protocols will use the term “Address”, when it really should be “identifier”.  For example, in networking it is very common to use the phrase “MAC Address” when the MAC is really an identifier.  It normally identifies a particular piece of hardware that is almost always at an endpoint or port to a particular device.  Addresses, if you think about it, are normally used to tell us how to get somewhere, like looking up the address of a building.  But very often there are multiple points-of-entry to a building (or device), so and address only gets us close to where we really want to go.  <Picky mode off>

Why Do We Care?

I will often hear the above question about these addresses and identifiers.  The simple answer, in most cases, is : troubleshooting! Depending on what interface and protocol we are using, we will be looking to identify what pieces of the system are involved when a problem occurs, what the source and destination device is, and what happened to cause a problem.  So if you are asked what devices and pieces of the infrastructure are involved, you need to be able to find that out.

Sample Addresses & Identifiers

Parallel SCSI – Parallel SCSI uses what are called SCSI IDs.  These are simply bits on a bus, in the range of 0 (zero) through 15.  This bits are used during the connection sequence to select, or reselect another device on the bus.

Serial Attached SCSI (SAS) – SAS uses SAS Addresses.  This is a 64-bit address that is used in the connection sequence to start (or continue) communication with another device in the SAS domain.

Fibre Channel (FC) – FC uses Source ID (S_ID) and Destination ID (D_ID) when communicating between devices.  These are in the frame header.

Tagged Command Queuing

Tagged Command Queuing (TCQ) is a function that allows multiple commands to be sent to a SCSI disk drive (includes Fibre Channel drives), and now some SATA drives.  The “twist” when you do command queuing on SATA drives, is that they don’t used the same name – they call it Native Command Queuing (NCQ).  I don’t think this is a good name because it might imply that command queuing is or was native in the SATA world.  But I suppose the name was picked more for sales people and not technically oriented geeks like me (off soapbox now).

Why Command Queuing?

So you may be asking the question: “Why do we do queuing?”  There are really a few main reasons which are:

Speed, efficiency, less stress on the physical parts, which results in longer device lifetime.

How Command Queuing Works

I will try to keep this as simple as possible by using an analogy or two.  Try to think of a disk drive as an elevator, and the sectors (or blocks) where it stores the data are labeled from zero (0) to some maximum number (let’s say 100 to keep it easy). Each command that accesses data will include a “Where” field, which is normally called the Logical Block Address (LBA).  Think of that as the “floor number” of this elevator.

Now imagine that the disk drive is the elevator sitting at Floor 0 (ground floor) and the operating system sends 5 commands to the drive, that all go to different floors and in a different order.  For example

• Command 01 moves data to floor 90 (LBA=90)
• Command 02 moves data from floor 10 (LBA=10)
• Command 03 moves data to floor 80 (LBA=80)
• Command 04 moves data from floor 20 (LBA=20)
• Command 05 moves data to floor 70 (LBA=70)

The doors close and the elevator starts to do its thing (move the commands and data).  Now if the elevator operator (the device firmware) had to do each of these commands in the order they were received, then that would take quite a bit of time and the elevator would have to move quite a distance between each command (first to floor 90, then to floor 10,  then back to 80, and so on).  As you can imagine, that would also add quite a bit of physical movement and is not very efficient.

But now think about how a real elevator works.  These same five commands jump on the elevator at the ground floor and the doors close.  What happens next? The elevator doesn’t care about the sequence the buttons for each floor were pressed, it just cares about which floor is closest to its current location, and what direction it is going.  This simple “elevator algorithm” would result in the following command execution sequence:

• Command 02 moves data from floor 10 (LBA=10)
• Command 04 moves data from floor 20 (LBA=20)
• Command 05 moves data to floor 70 (LBA=70)
• Command 03 moves data to floor 80 (LBA=80)
• Command 01 moves data to floor 90 (LBA=90)

Note that the elevator only moved between floors (moved the head on the drive) in the most efficient manor, and did not bounce back and forth (seek) multiple times.  The result is that the overall processing time will be reduced and there is less physical movement when compared to doing the commands in the original order that they were given.

So its all about speed and efficiency.  If you have any questions, leave a comment.

Incoming search terms for the article:

• tagged command queuing
• tag command queuing
• scsi commands ordered commands
• tagged command
• tagged command queing
• Tagged command queing and native command queing
• tagged commands queuing fibre channel
• tagged scsi
• tagged scsi command
• integrate;profile of identification total control of systematic of network network as programming of command sequence