PFC Queue Limits (Queue Depth/Size)

LAN Based Cards Queue Depths

Applicable to the LAN based cards such as the ws-x6708, ws-6724, etc, these cards have hardware queues both ingress and egress, each port has limited buffer space and may require reallocating buffer space between the queues. Using the ws-6724 as an example where the egress queue structure is 1p3q8t, meaning that there is 1 priority queue and 3 standard queues we will reapportion the default allocated buffer space to each queue. Firstly the total amount of space for on egress for a port on this card is 1.2MB (note: MegaBytes) which can be found in the 12.2SR configuration guide under the Configure QoS section, under Module to Queue Type Mappings.

Keep in mind that increasing a queue size (length) can introduce more latency due to more buffering, where smaller buffers can cause drops and therefore lost data (UDP) or retransmits (TCP). TCP drops / windows can be controlled by using WRED (wrr-queue random-detect) instead of the default queue management – tail drop, WRED still drops UDP as the drop management uses COS values and is does not look at the type of packet being carried, so UDP being dropped means lost data (unless application checks for missing data) and TCP will retransmit and will shorten window size.

The commands we use to configure the queue sizes are

priority-queue queue-limit

wrr-queue queue-limit

The priority-queue command is not available for all LAN cards, cards with queue structures, check the configuration guides for the cards you are using.

So we will allocate the following:

PQ – 10%
Q1 – 20%
Q2 – 30%
Q4 – 40%

The values entered are relative values and can be anything, a calculation below will show how this works, but for ease just define these as percentages as follows:

priority-queue queue-limit 10
wrr-queue queue-limit 20 30 40

so to calculate what is actually being assigned, using the PQ as an example

10 / (10+20+30+40) = 0.1

0.1 * 100  = 10% (PQ)

and 10% of 1.2 MB is 120KB

The outcome is 10% or 120KB (KiloBytes) as expected as we used percentages for the inputs to the commands, using relative values and this calculation will give you the queue percentages is you decide to use them.

In IOS to check your work use the following commands:

show queueing interface G1/1
remote command switch show qm port X Y (where x = slot, y = port number)

PFC Commands

show platform hardware pfc mode
- Check to see what mode with regards to installed PFC/DFC cards on Sup and line cards, where a difference in model numbers of PFC/DFC, lowest version/mode will run.

show fabric switching mode
- Show current mode with regards to either centralised or distributed CEF. (dCEF or CEF).

Cat6500-SUP720#show fabric switching-mode

Global switching mode is Compact

dCEF mode is not enforced for system to operate
Fabric module is not  required for system to operate
Modules are allowed to operate in bus mode
Truncated mode is allowed, due to presence of DFC, aCEF720 module
Module Slot     Switching Mode
1                 Crossbar
2                     dCEF
5                     dCEF
6                 Crossbar
9                 Crossbar

PFC: Classification, policing and marking with MQC

The MQC (Modular QoS Command Line) for PFC enabled cards provides classifaction, policing and marking. All queueing is done on the interface, this includes queue depths, WRR bandwidth, WRED.

Policing (PFC3+)

Aggregate policing can be one of the following on either ingress or egress:

• Policer attached to a single interface via MQC using the MQC class  ‘police’ command
• Policer attached to mutiple interfaces via MQC using the MQC class command ‘police aggregate {name}’ which references the global policer command ‘mls qos aggregate-policer {name}’

Microflow policing is used to identify individual flows on ingress only:

• Policer is attached to an interface via MQC using the MQC class command ‘police flow’ and parameters defining the traffic flow mask to be policed by this policer. The traffic flow mask defines how the policer views traffic entering the interface and therefore the policer to consitute a flow. The possible flow masks are:
  • mask src-only
    • Identifies flows using source address only, so only different source IP’s will constitute a new/different flow
  • mask dest-only
    • Identifies flows using the destination IP address, so each ingress packet with a different destination address consitutes a different flow.
  • mask full flow (default)
    • Identifies flows using the source and destination IP, protocol and protocol interfaces to indentify different flows.
  • mask destination-source {tbc (PFC3+)}
  • mask destination-source-interface {tbc(PFC3+)}
  • mask full-interface {tbc(PFC3+)}

Ingress Policing

• Ingress polcing is supported on Layer 2, Layer 3, and SVI interfaces through MQC configuration.
• Microflow and aggregate policing is supported on ingress.

Egress Policing

• Egress polcing is supported on Layer 3, and SVI interfaces through MQC configuration (note: not L2 interfaces).
• Only aggregate policing is supported on Egress.
• Direct remarking using policing command is not supported for traffic that exceeds, this requires the use of  ”policed-dscp-transmit” and the map “qos map dscp policed 24 to dscp 16″, this is an EARL 7 limitation (hardware) which SUP/RSP720′s run.

PFC Egress Flow

The following diagram shows the process of egress showing effecting commands vs. the area of concern.  A 1p3qxt card has been used in the example below, the ‘xt’ in the queueing definition could be 4 or 8 thresholds although is irrelevant for this example as WRED is not discussed here.

The priority queue id is always number 1 in the ‘priority-queue cos-map q-id cos# ‘ command, when using the command ‘show queueing interface’ and looking at the cos-queue mappings the priority queue is shown as queue 4 as shown in the output in the diagram below.

Egress Maps
The internal DSCP value is mapped to a internal CoS value first using the dscp-cos map (‘show mls qos maps’), the internal CoS value is then used to map the frame to a queue, use ‘show queueing interface’ to see this mapping. The (internal) DSCP to CoS map shown in the diagram below is the default, the relevent values for DSCP AF and EF have been highlighted and other values grayed out for clarity.

IP Packet DSCP Egress Value
By default the Internal DSCP value is used to modify the IP ToS (IPP) bits in the underlying IP packets. This is done because the Internal DSCP value may have been modified from when it was originally derived from the incoming packet (derived from DSCP, CoS or IPP – this is based on ‘mls qos trust’ command on ingress).  If this behaviour is not desired then use the global command ‘no mls qos rewrite ip dscp’.  This feature is called the “Disable DSCP Rewrite” feature. This will prevent the PFC/DFC from rewriting the IP Packet IPP on egress therefore leaving the IPP/DSCP value intact being the same as it was on ingress to the switch.

Ethernet Frame 802.1p (CoS) Egress Value

PFC Egress

Using Trusted Ports

A port can be set to trust on ingress for CoS, IPP, or DSCP.
‘mls qos trust cos’
‘mls qos trust ip-precedence’
‘mls qos trust dscp’

using ‘mls qos trust’ will default to trust dscp.

trust cos

If the port receives an ingress Ethernet frame 802.1, as this frame does not have any CoS bits the port applies the ingress port CoS value to the internal DSCP value. The ingress port CoS value is configured using ‘mls qos cos x’

If the port recieved an ingress 802.1q/p frame, the CoS bits in the 801.p bits are used to derive the internal DSCP value.

Both of these mapping use the cos-dscp map.

Deriving Internal DSCP Values from CoS

The internal DSCP values that are derived from a port CoS, IPP use the CoS-DSCP map which can be viewed by using:

'show mls qos maps'

   Cos-dscp map:
        cos:   0  1  2  3  4  5  6  7
       ----------------------------------
       dscp:   0  1  2  3  4  5  6  7

The map can be modified by using the global command ‘mls qos map cos-dscp’ which takes up to 8 DSCP values as arguments, each ones of these maps to a CoS value.

e.g mls qos map cos-dscp 8 16 22 32 35 45 46 47′

which will map DSCP value 8 to CoS 0, this means that a frame arriving or having CoS 0, will have an internal DSCP value of 8 assigned whilst being processed through the switch.

trust ip-precedence

If the port recieves a frame with a valid IP packet with TOS bits, the port will derive a internal DSCP value using a ip-prec-dscp map.

Deriving Internal DSCP Values from ip-precedence

The internal DSCP values that are derived from the IPP value use the ip-prec-dscp map, which can be viewed by using

'show mls qos maps'
 IpPrecedence-dscp map:

     ipprec:   0  1  2  3  4  5  6  7
    ----------------------------------
       dscp:   0  1  2  3  4  5  6  7

The map  can be modified by using the global command ‘mls qos map ip-prec-dscp’ which takes up to 8 DSCP values as arguments, each ones of these maps to a IPP value.

e.g mls qos map ip-prec-dscp 3 4 2 1 4 5 6 7

which will map IPP value 1 to DSCP 4, this means that a frame arriving with IPP value 1, will have an internal DSCP value of 4 assigned whilst being processed through the switch.

trust dscp

The ‘mls qos trust dscp’ enables the switch to used the DSCP value in a IP packet arriving on the switchport. The DSCP value is then mapped directly to the internal DSCP value. No map is required for this as it is a direct copy.

WRED QoS Commands

The following commands are for software based QoS, therefore they are no applicable for PFC based QoS, although they are available on 6500/7600 where the card/port being configured is a OSM/ES20/SIP router based card.

 (Not Supported on hardware switched PFC traffic, but is for software switched traffic)

• random-detect  [dscp-based | prec-based]
• random-detect cos-based
• random-detect flow (Interface Configuration)
• random-detect flow average-depth-factor (Interface Configuration)
• random-detect flow count (Interface Configuration)
• random-detect cos (threshold changes per CoS value)
• random-detect dscp  (threshold changes per dscp value)
• random-detect precedence (threshold changes per prec value)
   
• random-detect exponential-weighting-constant
   
• random-detect discard-class
• random-detect discard-class-based
   
• random-detect aggregate
• random-detect dscp (aggregate)
• random-detect precedence (aggregate)
• random-detect-group (Global Configuration)
  
• random-detect ecn

Commands applied per ASIC

The following commands effect all ports on an ASIC, when you configure one of these commands on a port, all ports connected to that ASIC will have the same command applied. For example on the WS-X6724-SFP card, two ASIC’s are present with one ASIC dedicated to ports 1-12 and the second ASIC dedicated to ports 13-24. So using this example if one of the commands below is configured on port 7, ports 1-12 will be configured with the same command and parameters. All commands shown below are interface commands.

wrr-queue queue-limit
Configures the size of each queue (Queue Depth)

wrr-queue bandwidth (except Gigabit Ethernet LAN ports)
Configures the time/BW used by WRR Scheduling to service each queue (not PQ)

wrr-queue threshold
Configures the tail drop threshold, used when WRED not configured

wrr-queue random-detect
Enables WRED on a queue, which disables tail drop. use the no form of this command to return to tail drop.

wrr-queue random-detect min-threshold
wrr-queue random-detect max-threshold
Enables the minimum and maximum percentages per queue per threshold, where CoS values are assigned to a queue and threshold number using ‘wrr-queue cos-map’

wrr-queue cos-map
Maps the CoS value of a packet to a standard queue. The CoS is derived from the Internal DSCP value using the DSCP-COS map.

priority-queue cos-map
Maps the CoS value of a packet to the PQ. The CoS is derived from the Internal DSCP value using the DSCP-COS map.

rcv-queue random-detect

rcv-queue queue-limit

rcv-queue cos-map

rcv-queue threshold

PFC Hardware QoS Queuing Interface Commands

These interface commands are for PFC QoS Queuing, such as enabling WRED, configuring CoS-to-Q mappings, queue sizes and bandwidth per queue.

mls qos queue-mode mode-dscp (WS-X6708 cards only – other 10Gs?)

rcv-queue random-detect

rcv-queue queue-limit

rcv-queue cos-map
rcv-queue dscp-map

rcv-queue threshold

wrr-queue random-detect
wrr-queue random-detect min-threshold
wrr-queue random-detect max-threshold

wrr-queue queue-limit – Queue Depth
wrr-queue bandwidth – WRR Scheduling
wrr-queue threshold – tail drop

wrr-queue cos-map
priority-queue cos-map

PFC MQC
Classification, policing and marking are done with MQC PFC QoS, see the post at http://ciscoqos.wordpress.com/2009/10/28/pfc-classification-policing-and-marking-with-mqc/