Matched serving measurements
All distribution pairs below are p50 / p95 from n=8 per natural cell or n=4 per capacity-edge cell. They are descriptive small-sample statistics.
Prefill and first output
Arm
Actual input p50
C
TTFO s
Visible TTFT s
Effective prefill tok/s
Engine prefill mean s
Engine queue mean s
EXL3
7,682
1
1.52 / 1.54
3.65 / 4.93
5,060.19 / 5,133.35
1.495
0.000
FP4
7,682
1
0.84 / 0.85
1.99 / 3.18
9,117.68 / 9,184.13
0.830
0.000
EXL3
7,682
2
1.59 / 3.47
3.71 / 9.43
4,832.90 / 4,886.16
1.736
0.033
FP4
7,682
2
1.65 / 3.04
3.52 / 5.22
4,628.54 / 7,999.17
1.652
0.009
EXL3
7,684
4
1.63 / 6.49
6.65 / 12.02
4,236.25 / 4,753.38
2.227
0.484
FP4
7,684
4
4.42 / 5.78
4.97 / 7.91
1,579.55 / 8,531.63
2.088
1.914
EXL3
31,600
1
6.31 / 6.32
7.72 / 10.30
5,011.27 / 5,071.04
6.240
0.000
FP4
31,600
1
3.51 / 3.54
4.12 / 4.52
9,002.46 / 9,089.28
3.487
0.000
EXL3
31,600
2
6.40 / 12.80
8.94 / 24.98
4,939.14 / 4,953.85
7.161
0.034
FP4
31,600
2
6.80 / 9.00
9.54 / 11.01
4,271.20 / 8,753.57
6.038
0.507
EXL3
31,600
4
6.99 / 24.95
26.12 / 46.84
4,313.75 / 4,904.25
9.816
1.460
FP4
31,600
4
17.09 / 19.23
18.60 / 22.19
1,784.13 / 8,672.80
6.462
7.866
EXL3
134,358
1
26.49 / 26.63
27.83 / 29.79
5,057.94 / 5,088.69
26.326
0.000
FP4
134,358
1
17.03 / 17.07
17.49 / 18.33
7,885.65 / 7,929.07
16.954
0.000
EXL3
134,358
2
30.08 / 52.05
53.43 / 59.89
4,374.77 / 4,926.73
35.137
0.034
FP4
134,358
2
33.54 / 34.91
34.40 / 37.71
3,955.46 / 7,907.33
19.667
11.824
EXL3
134,358
4
51.29 / 107.50
76.64 / 189.28
2,546.73 / 4,305.68
51.814
3.962
FP4
134,358
4
68.72 / 70.66
69.20 / 73.26
1,908.00 / 7,892.16
19.601
36.895
EXL3
252,300
1
50.67 / 50.87
52.05 / 53.59
4,969.43 / 5,034.12
50.200
0.000
FP4
252,300
1
33.77 / 33.81
33.90 / 34.65
7,463.28 / 7,528.89
33.626
0.000
EXL3
252,300
2
59.03 / 99.77
100.49 / 101.26
2,761.62 / 4,372.64
76.323
0.135
FP4
252,300
2
67.29 / 68.50
67.69 / 69.29
3,740.22 / 7,433.24
35.639
23.246
EXL3
252,300
4
108.46 / 197.47
195.38 / 200.23
1,615.59 / 4,200.83
122.303
7.320
FP4
252,300
4
68.37 / 136.77
69.18 / 138.34
2,451.30 / 7,450.86
35.499
49.534
Generation and request completion
Arm
Actual input p50
C
Decode tok/s
TPOT ms
E2E s
Aggregate output tok/s
Completion tokens
EXL3
7,682
1
77.67 / 77.73
12.87 / 12.89
4.45 / 5.76
52.42
229.00 / 329.00
FP4
7,682
1
203.31 / 217.50
4.78 / 5.26
2.30 / 3.57
128.88
286.00 / 551.00
EXL3
7,682
2
42.86 / 77.16
23.16 / 24.21
5.47 / 10.33
63.50
170.00 / 373.00
FP4
7,682
2
107.08 / 175.98
9.31 / 12.71
4.12 / 5.56
127.71
271.00 / 399.00
EXL3
7,684
4
28.80 / 59.07
31.17 / 45.08
9.14 / 16.17
79.46
177.00 / 293.00
FP4
7,684
4
113.27 / 192.66
8.81 / 11.71
5.29 / 8.71
115.96
164.00 / 305.00
EXL3
31,600
1
77.38 / 77.60
12.91 / 12.94
8.59 / 11.14
25.32
178.00 / 374.00
FP4
31,600
1
201.49 / 223.19
4.81 / 5.50
4.43 / 4.86
46.57
190.00 / 277.00
EXL3
31,600
2
68.85 / 77.00
14.52 / 58.55
9.66 / 32.29
32.13
190.00 / 534.00
FP4
31,600
2
103.52 / 200.95
9.56 / 10.31
10.26 / 11.87
53.70
197.00 / 525.00
EXL3
31,600
4
11.76 / 65.63
75.77 / 135.47
27.23 / 48.32
31.44
191.00 / 394.00
FP4
31,600
4
110.30 / 190.38
8.99 / 9.65
18.89 / 22.47
63.14
297.00 / 453.00
EXL3
134,358
1
77.80 / 78.60
12.84 / 12.98
28.57 / 30.80
6.86
170.00 / 327.00
FP4
134,358
1
214.18 / 232.46
4.65 / 4.99
17.80 / 18.65
11.41
176.00 / 327.00
EXL3
134,358
2
7.57 / 72.81
79.09 / 193.60
54.35 / 80.05
6.55
159.00 / 295.00
FP4
134,358
2
99.05 / 225.61
9.70 / 10.68
35.15 / 38.05
11.59
157.00 / 374.00
EXL3
134,358
4
3.81 / 65.50
199.84 / 384.88
80.04 / 208.63
8.94
169.00 / 375.00
FP4
134,358
4
101.97 / 199.01
9.80 / 11.13
69.53 / 73.57
11.02
178.00 / 365.00
EXL3
252,300
1
78.72 / 80.42
12.70 / 12.91
52.75 / 54.39
3.29
149.00 / 289.00
FP4
252,300
1
220.91 / 226.52
4.53 / 4.93
34.24 / 34.90
5.16
151.00 / 235.00
EXL3
252,300
2
5.51 / 71.73
64.41 / 182.77
101.34 / 110.27
3.96
166.00 / 243.00
FP4
252,300
2
94.77 / 229.03
9.37 / 10.76
67.94 / 70.03
4.85
151.00 / 217.00
EXL3
252,300
4
4.08 / 65.49
197.81 / 394.40
198.28 / 200.94
4.83
173.00 / 468.00
FP4
252,300
4
92.35 / 209.92
10.23 / 10.92
69.93 / 138.71
6.21
147.00 / 408.00
Observed resource use
Arm
Actual input p50
C
Client / active / queued max
KV use peak %
GPU peak used / min free GiB
Host min available GiB
Cgroup sampled peak GiB
OOM / preempt / refusal
EXL3
7,682
1
1 / 1 / 0
3.66
93.21 / 1.85
63.99
51.56
0 / 0 / 0
FP4
7,682
1
1 / 1 / 0
5.61
94.19 / 0.87
56.67
40.21
0 / 0 / 0
EXL3
7,682
2
2 / 2 / 1
6.10
93.21 / 1.85
63.09
51.57
0 / 0 / 0
FP4
7,682
2
2 / 2 / 0
10.09
94.19 / 0.87
56.70
40.22
0 / 0 / 0
EXL3
7,684
4
4 / 4 / 3
10.98
93.21 / 1.85
63.06
51.57
0 / 0 / 0
FP4
7,684
4
4 / 3 / 2
14.58
94.19 / 0.87
56.98
40.22
0 / 0 / 0
EXL3
31,600
1
1 / 1 / 0
5.79
93.21 / 1.85
63.05
51.57
0 / 0 / 0
FP4
31,600
1
1 / 1 / 0
9.35
94.19 / 0.87
56.53
40.22
0 / 0 / 0
EXL3
31,600
2
2 / 2 / 1
10.37
93.21 / 1.85
63.15
51.57
0 / 0 / 0
FP4
31,600
2
2 / 2 / 1
16.45
94.19 / 0.87
56.77
40.22
0 / 0 / 0
EXL3
31,600
4
4 / 4 / 3
19.51
93.21 / 1.85
63.21
51.58
0 / 0 / 0
FP4
31,600
4
4 / 2 / 3
16.45
94.19 / 0.87
56.65
40.23
0 / 0 / 0
EXL3
134,358
1
1 / 1 / 0
15.24
93.21 / 1.85
60.86
51.60
0 / 0 / 0
FP4
134,358
1
1 / 1 / 0
14.02
94.19 / 0.87
56.90
40.25
0 / 0 / 0
EXL3
134,358
2
2 / 2 / 1
29.27
93.25 / 1.82
58.67
51.60
0 / 0 / 0
FP4
134,358
2
2 / 2 / 1
21.12
94.19 / 0.87
57.04
40.25
0 / 0 / 0
EXL3
134,358
4
4 / 4 / 2
57.01
93.26 / 1.80
57.68
51.62
0 / 0 / 0
FP4
134,358
4
4 / 2 / 3
21.12
94.19 / 0.87
54.66
40.27
0 / 0 / 0
EXL3
252,300
1
1 / 1 / 0
25.91
93.42 / 1.64
57.94
51.75
0 / 0 / 0
FP4
252,300
1
1 / 1 / 0
19.44
94.19 / 0.87
56.32
40.29
0 / 0 / 0
EXL3
252,300
2
2 / 2 / 1
50.61
93.42 / 1.64
60.13
51.82
0 / 0 / 0
FP4
252,300
2
2 / 2 / 1
26.54
94.19 / 0.87
56.32
40.30
0 / 0 / 0
EXL3
252,300
4
4 / 4 / 2
75.30
93.42 / 1.64
59.74
51.89
0 / 0 / 0
FP4
252,300
4
4 / 2 / 3
26.54
94.19 / 0.87
56.38
40.33
0 / 0 / 0
Timing and resource definitions, configuration, exact native references, and limitations are in the finding and summary-table.json. GPU figures are worst single-device samples; maxima are sampled lower bounds. Host available memory is whole-host memory; cgroup memory includes page cache.