Oracle SPARC T5 (SPARC S3 core)

Configuration

Oracle SPARC T5 (S3 core), 3600 MHz, 16 cores, 28 nm, 478 mm2, 8 MB L3, 2013, Sun LDOM: 1 CPU, 8 threads, 8 GB.

Predecessor: SPARC T4 (S3 core), 8 cores, 40 nm, 403 mm2, 4 MB L3, 855 MTr, 2011.

CPU / Core

L1 Data cache = 16 KB. 4-Way, 32-byte lines, write-through (no allocate on write miss). Hardware stride-based prefetcher.
L1 Instruction cache = 16 KB, 4-Way, 32-byte line. Cache miss prefetcher that prefetches up to 3 sequential lines ahead upon a miss.

L2 cache size = 128 KB, 8-WAY. 32-byte lines, Write-back, write-allocate. MESI. Inclusive of L1 caches. Next-line sequential prefetcher.

L3: 16-WAY. 64-byte line, 8 banks, Write-back. MESI. Inclusive of all L2 caches. Bank selection: bits 8:6.
T5: L3 cache size = 8 MB (per 16 cores)
T4: L3 cache size = 4 MB (per 8 cores)

T5: RAM : 4 x 2-channel DDR3-1066
T4: RAM : 2 x 2-channel DDR3-1066

T5: links to another chips: 7 links * 12 lanes * 12.8 Gb/s * 2 directions
T4: links to another chips: 2 Coherent units * 3 links * 14 lanes * 9.4 Gb/s * 2 directions. Frame size = 12*14 = 168 bits. 800M frames per second. Coherent unit : 3 links and 2-channel Memory controller

T5: PCI-E : 2 x x8 3.0
T4: PCI-E : 2 x x8 2.0

SMT8 : modified LRU (Least Recently Used) for thread choice.
Fetch : selects strand and fetches up to 4 instructions to one from 8 Instruction Buffers.
Select/Decode/Rename : selects strand, decodes and renames up to 2 instructions from selected Instruction Buffer. It also assigns instructions to issue slots.
Unified Pick Queue : 40-entry (36-entry out-of-order scheduler). Up to 3 instructions (2 instructions + 1 store data acquisition op) are written into the PQ during the second phase of the Rename3 stage. The PQ is read during the first phase of the pick cycle.
Commit: 128-entry reorder buffer : each cycle attempts to commit instructions from the least-recently-committed thread among the threads which are ready-to-commit.

Out-of-Order (OoO) instruction execution
1 FPU
3 issue slots:
issue slot 0: Alu0 / Alu0->LSU / LSadr / LSU
issue slot 1: Alu1/Branch/FGU/Crypto
issue slot 2: Store data op

Instruction TLB size = 64 items, fully-assoc.
Data TLB size = 128 items, full-assoc. shared by all strands/threads. T4 : 8KB, 64KB, 4MB, 256MB, and 2GB pages.

L1 Data Cache Latency = 5 cycles for simple access via pointer
L1 Data Cache Latency = 6 cycles for access with complex address calculation (size_t n, *p; n = p[n]).

L2 Cache Latency = 19 cycles

L3 Cache Latency = 52 cycles (T5)

RAM Latency = 52 cycles + 143 ns (T5)

Memory Translation

T4: 52-bit virtual and real address spaces (sign-extended from bit51).

3 levels of translation:

TLB
for TLB misses : HW walker tries to load TTE from TSB (located in RAM or L2/L3 cache) to TLB
for TSB misses : interrupt : and software reloads TTE in TSB from some software tables in RAM.

Translation Table Entry (TTE) : 16 bytes : 16-bit context ID, 42 bits VA tag, 43 real page number. Hardware table walker supports only 48-bit real address

Translation Storage Buffer (TSB) - array of TTEs filled by software. It serves as a cache of the Software Translation table. It works as direct-mapped cache (1-way TLB). TSB exists as a normal data structure in memory and therefore may be cached in L2/L3.

The number of TTEs in TSB: from 512 to 16 M entries.

How TSB size is selected? Does it use common TSB for all processes or all threads for same process?

8 KB pages mode (64-bit) : T5 : Linux at Sun LDOM

Data TLB size = 128 items, full-assoc. (shared by all strands/threads). Miss penalty = 34 cycles Parallel miss: 6 cycles per access
translation penalty after 512 MB (1 MB of TSB overflow?) = 527 ns. Parallel miss: 527 ns

  Size        Latency       Increase   Description

  16 K     5                           
  32 K    13                       8   + 14 (L2)         
  64 K    16                       3   
 128 K    20                       4   
 256 K    38                      18   + 33 (L3)
 512 K    45                       7   
   1 M    49                       4   
   2 M    68                      19   + 34 (TLB miss)
   4 M    77                       9	
   8 M    82 +  19 ns     5 +  19 ns   
  16 M    84 +  80 ns     2 +  61 ns   + 143 ns (RAM)
  32 M    85 + 115 ns     1 +  35 ns
  64 M    90 + 131 ns     5 +  16 ns
 128 M   104 + 137 ns    14 +   6 ns   + 33 (TSB TTE reading from L3)
 256 M   111 + 140 ns     7 +   3 ns
 512 M   115 + 142 ns     4 +   2 ns
1024 M   117 + 410 ns     2 + 268 ns   + 527 ns (TSB overflow? software translation)
2048 M   119 + 552 ns     2 + 142 ns

MISC

16 stages integer: 3 Fetch, Select, 2 Decode, 3 Rename, Pick, 4 Issue, Execute, Writeback.
Load pipeline: + 4 cycles
FPU pipeline: + 11 cycles
All FPU operations latency = 11-12 cycles or more. FDIV: 24 (SP) / 37 (DP)
Integer MUL operation latency = 12 cycles : probably it work in float point pipeline

Branch misprediction penalty = 22 cycles (30 cycles in another test)
Branch history table = Half misprediction for 8K branches.

L1 B/W (Parallel Random Read) = 1 cycles per one access

L2->L1 B/W (Parallel Random Read) = 2.0 cycles per cache line (32 bytes)
L2->L1 B/W (Read, 32 bytes step) = 2.2 cycles per cache line (32 bytes)
L2 Write (Write, 8 - 32 bytes step) = 1.75 - 1.0 cycles per write (write-throw via L1)

L3->L1 B/W (Parallel Random Read) = 4.0 cycles per cache line (32 bytes)
L3->L1 B/W (Read, 32 bytes step) = 4.0 cycles per cache line (32 bytes)
L3 Write (Write, 32 bytes step) = 5.4 cycles per write

RAM Read B/W (Parallel Random Read) = 12 ns / read (64 bytes read ?)
RAM Read B/W (Read, 4-16 Bytes step) = 3 GB/s
RAM Read B/W (Read, 32 Bytes step) = 4.7 GB/s
RAM Write B/W (Write, 8 Bytes step, full) = 2.2 GB/s

7-Zip Benchmark

Notes:

7z b -mm=* : MIPS and Effectiveness values are normalized with AMD K8 cpu.


## Sun LDOM, VM Linux-4.11, THP off
# gcc-7 -m64 -O3


7-Zip (a) 16.04 : Copyright (c) 1999-2016 Igor Pavlov : 2016-10-04
p7zip Version 16.04 (locale=C,Utf16=off,HugeFiles=on,64 bits,8 CPUs BE)

freq= 3600
BE
CPU Freq:  3555  3565  3568  3568  3568  3568  3565  3561  3560

RAM size:    8046 MB,  # CPU hardware threads:   8
RAM usage:    225 MB,  # Benchmark threads:      1


Method           Speed Usage    R/U Rating   E/U Effec
                 KiB/s     %   MIPS   MIPS     %     %

CPU                      100   3593   3591   100   100
CPU                      100   3593   3591   100   100
CPU                      100   3602   3593   100   100

LZMA:x1           9476   100   3472   3464    96    96
                 25470   100   2075   2074    58    58
LZMA:x5:mt1       2011   100   2515   2513    70    70
                 23954   100   2023   2020    56    56
LZMA:x5:mt2       3141   203   1936   3924    54   109
                 24049   100   2029   2028    56    56
Deflate:x1       21043   100   2673   2672    74    74
                 85280   100   2654   2650    74    74
Deflate:x5        7691   100   2962   2961    82    82
                 85444   100   2652   2653    74    74
Deflate:x7        2670   100   2961   2958    82    82
                 86126   100   2676   2673    74    74
Deflate64:x5      7271   100   3147   3142    87    87
                 84148   100   2633   2632    73    73
BZip2:x1          3809   100   2302   2302    64    64
                 17711   100   1920   1920    53    53
BZip2:x5          3275   100   2735   2733    76    76
                 14262   100   2796   2799    78    78
BZip2:x5:mt2      4006   196   1703   3344    47    93
                 16844   147   2253   3306    63    92
BZip2:x7           997   100   2585   2584    72    72
                 14382   100   2818   2820    78    78
PPMD:x1           2300   100   2383   2380    66    66
                  1749   100   2061   2060    57    57
PPMD:x5           1585   100   2688   2687    75    75
                  1125   100   2112   2110    59    59
Delta:4         124157   100    764    763    21    21
                130725   100    804    803    22    22
BCJ             708161   100   2903   2901    81    81
                708346   100   2903   2901    81    81
AES256CBC:1      76857   100   1890   1889    52    52
                 79267   100   1949   1948    54    54
AES256CBC:2 

CRC32:1         350172   100   2553   2549    71    71
CRC32:4         733119   100   1636   1636    45    45
CRC32:8        1038139   100   1409   1408    39    39
CRC64           745911   100   1529   1528    42    42
SHA256           99499   100   2031   2030    56    56
SHA1            251078   100   2349   2350    65    65
BLAKE2sp        134075   100   2957   2950    82    82

CPU                      100   3590   3592   100   100
------------------------------------------------------
Tot:                     113   2319   2575    63    72




7-Zip (a) 16.04 : Copyright (c) 1999-2016 Igor Pavlov : 2016-10-04
p7zip Version 16.04 (locale=C,Utf16=off,HugeFiles=on,64 bits,8 CPUs BE)

freq= 3600
BE
CPU Freq:  3563  3565  3568  3568  3568  3567  3567  3568  3568

RAM size:    8046 MB,  # CPU hardware threads:   8
RAM usage:   1802 MB,  # Benchmark threads:      8


Method           Speed Usage    R/U Rating   E/U Effec
                 KiB/s     %   MIPS   MIPS     %     %

CPU                      795    874   6943    24   193
CPU                      792    874   6921    24   192
CPU                      793    874   6930    24   192

LZMA:x1          19191   789    889   7016    25   195
                 63617   793    653   5181    18   144
LZMA:x5:mt1       3939   772    637   4922    18   137
                 61982   792    660   5227    18   145
LZMA:x5:mt2       3902   790    617   4876    17   135
                 62091   794    660   5236    18   145
Deflate:x1       41656   794    666   5289    19   147
                139663   795    546   4340    15   121
Deflate:x5       13853   793    672   5334    19   148
                139917   795    546   4344    15   121
Deflate:x7        4476   794    624   4960    17   138
                140298   791    550   4354    15   121
Deflate64:x5     13180   784    727   5696    20   158
                139328   795    548   4359    15   121
BZip2:x1          6070   776    473   3667    13   102
                 25351   669    411   2748    11    76
BZip2:x5          5192   771    562   4333    16   120
                 24880   788    620   4884    17   136
BZip2:x5:mt2      4929   742    555   4114    15   114
                 25185   791    625   4943    17   137
BZip2:x7          1579   791    518   4093    14   114
                 24839   790    616   4871    17   135
PPMD:x1           3260   795    424   3372    12    94
                  2900   794    430   3416    12    95
PPMD:x5           2589   794    552   4388    15   122
                  2294   792    543   4299    15   119
Delta:4         339474   787    265   2086     7    58
                340696   787    266   2093     7    58
BCJ             736593   787    384   3017    11    84
                746118   782    391   3056    11    85
AES256CBC:1      80539   798    248   1979     7    55
                 81563   798    251   2005     7    56
AES256CBC:2 

CRC32:1         628368   799    572   4575    16   127
CRC32:4        1058708   799    296   2363     8    66
CRC32:8        1114651   798    189   1511     5    42
CRC64           984146   793    254   2016     7    56
SHA256          109282   783    285   2229     8    62
SHA1            251254   795    296   2352     8    65
BLAKE2sp        140965   795    390   3101    11    86

CPU                      796    873   6947    24   193
------------------------------------------------------
Tot:                     787    591   4649    16   129

Links

SPARC T5 at Wikipedia

SPARC T4 at Wikipedia

SPEC Benchmarks

CPU	Cores	Frequency	SPEC_2006		SPEC_2017
CPU	Cores	Frequency	int	fp	int	fp
SPARC T5-1B	16	3600 MHz	441	350
SPARC T7-1	32	4133 MHz	1120	801	114	111