SAN - Port Buffers and Memory
Published 26 August 2002, updated 07 August 2003
Authors: Jon Tate
SAN - what are buffers and credits anyway?
Buffers and credits
Ports need memory, or buffers, to temporarily store frames as they arrive and
until they are assembled in sequence, and delivered to the upper layer protocol.
The number of buffers, that is the number of frames a port can store, is called its
During fabric login, N_Ports and F_Ports at both ends of a link establish its Buffer
to Buffer Credit (BB_Credit). Each port states the maximum BB_Credit that they
can offer and the lower of the two is used.
In the same way during port login all N_Ports establish End to End Credit
(EE_Credit) with each other.
During data transmission a port should not send more frames than the buffer of
the receiving port can handle before getting an indication from the receiving port
that it has processed a previously sent frame. Two counters are used for that.
BB_Credit_CNT and EE_Credit_CNT, both are initialized to 0 during login.
Each time a port sends a frame it increments BB_Credit_CNT and
EE_Credit_CNT by one. When it receives R_RDY from the adjacent port it
decrements BB_Credit_CNT by one, and when it receives ACK from the
destination port it decrements EE_Credit_CNT by one. Should at any time
BB_Credit_CNT become equal to the BB_Credit or EE_Credit_CNT equal to the
EE_Credit of the receiving port, the transmitting port has to stop sending frames
until the respective count is decremented.
The previous statements are true for Class 2 service. Class 1 is a dedicated
connection, so it does not need to care about BB_Credit and only EE_Credit is
used (EE Flow Control). Class 3 on the other hand is an unacknowledged
service, so it only uses BB_Credit (BB Flow Control), but the mechanism is the
same on all cases.
Here we can see the importance that the number of buffers has in overall
performance. We need enough buffers to make sure the transmitting port can
continue sending frames without stopping in order to use the full bandwidth.
This is particularly true with distance. At 1 Gb/s a frame occupies between about
75 m and 4 km of fiber depending on the size of the data payload. In a 100 km
link we could send many frames before the first one reaches destination. We
need an ACK back to start replenishing EE_Credit or an R_RDY to replenish
For a moment, let us consider frames with 2 KB of data. These occupy
approximately 4 km of fiber. We will be able to send about 25 frames before the
first arrives at the far end of our 100 km link. We will be able to send another 25
before the first R_RDY or ACK is received, so we would need at least 50 buffers
to allow for nonstop transmission at 100 km distance with frames of this size. If
the frame size is reduced, more buffers would be required to allow nonstop
This material has not been submitted to any formal IBM test and is published AS IS. It has not been the subject of rigorous review. IBM assumes no responsibility for its accuracy or completeness. The use of this information or the implementation of any of these techniques is a client responsibility and depends upon the client's ability to evaluate and integrate them into the client's operational environment.
Follow IBM Redbooks
Follow IBM Redbooks