Dynamic INT8 quantization (Geron Ch 19, morie.fn grdyq)

Per-tensor scales max|.|/127, INT32 accumulate, dequantize by sx*sw. np.rint and R's round both break ties to even.

Usage

morie_geron_dynamic_quantization(x, w)

Arguments

x, w

Conformable numeric matrices.

Value

List with output, reference, max_abs_error, relative_error, scale_x, scale_w, x_quantized, w_quantized, accumulator_max.

Examples

morie_geron_dynamic_quantization(x = 5L, w = c(1, 2, 3, 4, 5, 6, 7, 8))
#> $output
#>         [,1]     [,2]     [,3]     [,4]     [,5]     [,6]     [,7] [,8]
#> [1,] 5.03937 10.07874 15.11811 20.15748 24.88189 29.92126 34.96063   40
#> 
#> $reference
#>      [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8]
#> [1,]    5   10   15   20   25   30   35   40
#> 
#> $max_abs_error
#> [1] 0.1574803
#> 
#> $relative_error
#> [1] 0.003937008
#> 
#> $scale_x
#> [1] 0.03937008
#> 
#> $scale_w
#> [1] 0.06299213
#> 
#> $x_quantized
#>      [,1]
#> [1,]  127
#> 
#> $w_quantized
#>      [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8]
#> [1,]   16   32   48   64   79   95  111  127
#> 
#> $accumulator_max
#> [1] 16129
#> 
#> $estimate
#>         [,1]     [,2]     [,3]     [,4]     [,5]     [,6]     [,7] [,8]
#> [1,] 5.03937 10.07874 15.11811 20.15748 24.88189 29.92126 34.96063   40
#> 
#> $n
#> [1] 1
#>