ConvNeXt block

Formula: depthwise conv + LayerNorm + 1x1 conv

Usage

Convnx(x, filters = NULL, kernel = 7, expand = 4, layer_scale = 0, seed = 42)

Arguments

x

An H x W matrix for a single channel.

filters

Number of channels; inferred when NULL.

kernel

Depthwise kernel size, odd.

expand

Pointwise expansion factor.

layer_scale

Initial gamma of the residual scaling.

seed

Seed of the deterministic stream.

Value

List with estimate, out, residual_norm, H, W, C, method.

Details

Depthwise k x k convolution, channel-last LayerNorm, a pointwise expansion with GELU, a pointwise projection back, then a residual add scaled by gamma. ConvNeXt initialises the layer scale gamma at a small value; at gamma = 0 the block is EXACTLY the identity, which is the check that the residual path is wired the right way round.

References

Liu et al. (2022), A ConvNet for the 2020s, CVPR 2022:11976-11986.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
Convnx(V)
#> $estimate
#> [1] 4.5
#> 
#> $out
#>      [,1]
#> [1,]    1
#> [2,]    2
#> [3,]    3
#> [4,]    4
#> [5,]    5
#> [6,]    6
#> [7,]    7
#> [8,]    8
#> 
#> $residual_norm
#> [1] 0
#> 
#> $H
#> [1] 8
#> 
#> $W
#> [1] 1
#> 
#> $C
#> [1] 1
#> 
#> $method
#> [1] "ConvNeXt block"
#> 
#> attr(,"class")
#> [1] "morie_rich_result" "list"