Intel Ivy Bridge

Intel i7-3770 (Ivy Bridge), 3.4 GHz (Turbo Boost off), 22 nm. RAM: 4 GB (Single PC3-12800 10-10-10-28).

• L1 Data cache = 32 KB. 64 B/line, 8-WAY.
• L1 Instruction cache = 32 KB. 8-WAY. 64 B/line
• L2 cache = 256 KB. 64 B/line, 8-WAY
• L3 cache = 8 MB. 64 B/line

• L1 Data Cache Latency = 4 cycles for simple access via pointer
• L1 Data Cache Latency = 5 cycles for access with complex address calculation (size_t n, *p; n = p[n]).
• L2 Cache Latency = 12 cycles
• RAM Latency = 30 cycles + 53 ns

L3 (SLC) cache

The cache latency for reading from different L3 Slices to different Cores with additional ALU OPs between LOADs:

  0   1   2   3   4   5   6   7   8   ALU OPs   

  4   5   5   5   5   5   5   5   5   L1
 12  12  12  12  13  12  12  12  12   L2
            
 30  30  30  31  30  30  30  30  30   L3 core 0,3
 29  29  29  30  29  29  29  29  29   L3 core 1,2
                                
 26  27  26  27  26  27  26  27  26   core-N slice-N
 28  31  30  29  28  29  28  29  28   core-0 slice-1 / core-1 slice-0
 32  33  32  33  32  33  32  34  32   core-0 slice-2
 34  33  34  35  34  33  34  33  34   core-0 slice-3
                                
 32  31  30  31  30  29  30  29  30   core-1 slice-2
 32  33  32  33  32  33  32  33  32   core-1 slice-3

The total L3 iteration latency is always EVEN number, when ALU OPS are included:

• L3 Cache Latency = 29.40 cycles (average latency for cores 1, 2)
• L3 Cache Latency = 30.40 cycles (average latency for cores 0, 3)
• L3 Cache Latency = 26 cycles (the Core reads from nearest L3 Slice)
• L3 Cache Latency = 34.29 cycles (the Core-0 reads from farthest Slice-3)

L3 Latency penalty for reading from different L3 Slices:

Core-0 =##= Slice-0
        || 2c
Core-1 =##= Slice-1
        || 4c
Core-2 =##= Slice-2
        || 2c
Core-3 =##= Slice-3  

To read data from required slice we use the following hash (xor) functions for L3 slice number, from physical address bits [1]:

• L3 Slice Bit #0 = xor (30, 28, 27, 26, 25, 24, 22, 20, 18, 17, 16, 14, 12, 10, 6)
• L3 Slice Bit #1 = xor (31, 29, 28, 26, 24, 23, 22, 21, 20, 19, 17, 15, 13, 11, 7)

Note: L3 cache in Sandy Bridge uses Pseudo-LRU policy for LLC. But LLC replacement policy in Ivy Bridge looks like random replacement policy.

2 MB pages mode (64-bit Linux)

• Data TLB: 32 entries. Miss Penalty = 16 cycles. Parallel miss: 20 cycles per access

  Size        Latency       Increase   Description

  32 K     4                           
  64 K     8                       4   + 8 (L2)        
 128 K    10                       2   
 256 K    11                       1
 512 K    21                      10   + 18 (L3)
   1 M    26                       5
   2 M    28                       2
   4 M    29                       1
   8 M    30                       1
  16 M    30 + 27 ns           27 ns   + 53 ns (RAM)
  32 M    30 + 40 ns           13 ns
  64 M    30 + 47 ns            7 ns
 128 M    38 + 50 ns       8 +  3 ns   + 16 (TLB miss)
 256 M    42 + 52 ns       4 +  2 ns
 512 M    44 + 53 ns       2 +  1 ns
1024 M    45 + 53 ns       1 
2048 M    46 + 53 ns       1 

4 KB pages mode (64-bit Linux)

• Data TLB L1: 64 items. 4-way. Miss penalty = 7 cycles. Parallel miss: 1 cycle per access
• Data TLB L2 (STLB): 512 items. 4-way. Miss penalty = 9 cycles. Parallel miss: 21 cycle per access
• PDE cache = 32 items. Miss penalty = 9 cycles.

  Size        Latency       Increase   Description

  32 K     4                           
  64 K     8                       4   + 8 (L2)        
 128 K    10                       2   
 256 K    14                       4
 512 K    25                      11   + 18 (L3) +7 (L1 TLB miss)
   1 M    31                       6
   2 M    34                       3
   4 M    41                       7   + 9 (L2 TLB miss)
   8 M    44                       3
  16 M    45 + 27 ns       1 + 27 ns   + 53 ns (RAM)
  32 M    46 + 40 ns       1 + 13 ns
  64 M    49 + 47 ns       3 +  7 ns
 128 M    64 + 50 ns      15 +  3 ns   +  9 (PDE cache miss) + 19 (Page walk to L3)
 256 M    69 + 52 ns       5 +  2 ns   + 
 512 M    76 + 53 ns       7 +  1 ns
1024 M    84 + 53 ns      12 
2048 M    94 + 53 ns      10 

MISC

Branch misprediction penalty = 14 cycles.

• 64-bytes range cross penalty = 5 cycles
• 4096-bytes range cross penalty = 28 cycles
• L1 B/W (Parallel Random Read) = 0.54 cycles per one access
• L2->L1 B/W (Parallel Random Read) = 2.5 cycles per cache line
• L2->L1 B/W (Read, 64 bytes step) = 2.2 cycles per cache line
• L2 Write (Write, 64 bytes step) = 6.1 cycles per write (cache line)
• L3->L1 B/W (Parallel Random Read) = 4.8 cycles per cache line
• L3->L1 B/W (Read, 64 bytes step) = 4.9 cycles per cache line
• L3 Write (Write, 64 bytes step) = 9.0 cycles per write (cache line)
• RAM Read B/W (Parallel Random Read) = 9 ns / cache line = 7100 MB/s
• RAM Read B/W (Read, 8 Bytes step) = 12300 MB/s
• RAM Read B/W (Read, 64 Bytes step) = 12500 MB/s
• RAM Read B/W (Read, 64 Bytes step - pointer chasing) = 8800 MB/s
• RAM Write B/W (Write, 4-64 Bytes step) = 5700 MB/s

Links

[1]: Reverse Engineering Intel Last-Level Cache Complex Addressing Using Performance Counters. Maurice, 2015

Ivy Bridge at Wikipedia