ControlNet attachment

Formula: trainable copy of the UNet encoder; zero-conv

Usage

Cncpat(base, condition, zero_conv_weight = 0, seed = 42)

Arguments

base

Base-network feature map, H x W.

condition

Conditioning image of the same shape.

zero_conv_weight

Weight of the zero convolution; 0 at initialisation.

seed

Seed of the deterministic stream.

Value

List with estimate, out, control, delta_norm, is_identity, H, W, method.

Details

The conditioning branch is a copy of the frozen encoder whose output passes through a 1x1 convolution initialised at ZERO before it is added back to the base feature map. At initialisation the sum is therefore exactly the base output: attaching a ControlNet cannot perturb the pretrained model until the zero-convolution has learned a non-zero weight. That identity is the whole design.

References

Zhang, Rao & Agrawala (2023), Adding Conditional Control to Text-to-Image Diffusion Models, ICCV 2023:3836-3847.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
Cncpat(V, V)
#> $estimate
#> [1] 4.5
#> 
#> $out
#>      [,1]
#> [1,]    1
#> [2,]    2
#> [3,]    3
#> [4,]    4
#> [5,]    5
#> [6,]    6
#> [7,]    7
#> [8,]    8
#> 
#> $control
#>               [,1]
#> [1,] -6.972238e-04
#> [2,] -1.663546e-09
#> [3,] -6.767144e-16
#> [4,] -4.310104e-24
#> [5,] -4.311433e-34
#> [6,] -6.781048e-46
#> [7,] -1.677744e-59
#> [8,] -2.428990e-63
#> 
#> $delta_norm
#> [1] 0
#> 
#> $is_identity
#> [1] 1
#> 
#> $H
#> [1] 8
#> 
#> $W
#> [1] 1
#> 
#> $method
#> [1] "ControlNet attachment with a zero convolution"
#> 
#> attr(,"class")
#> [1] "morie_rich_result" "list"