I don't want to overhype it, but this will be the best first paper ever written. @Algomancer is cracked

lol I was playing around with a vision language model, and it thinks this xfinity logo is a hate symbol (see below for full details)

Paper summary: Weight Subcloning TLDR: Weight Subcloning takes a subset of the weights from a larger network and uses it to initialize a smaller network. This speeds up training the smaller network by 4x compared to random initialization. When to use it: If you have a larger model and need to train a smaller version of it, weight subcloning can speed up training the smaller model. How it works: It removes parameters by 1. Removing transformer blocks 2. Using a subset of weights from other layers. 1. It removes blocks from the middle layers of the network. They show that the middle layers have much smaller changes to the inputs and heavily leverage residual connections, more info here: xitter.cf/rosmine_b/status/18056… 2. To get a subset of weights from the remaining blocks (subcloning parameters), it ranks the most important weights by looking at the outputs of each layer on a small set of data. If the original dimension was d, and the new dimension is d', they take the top d' most important weights. Note that they also scale the weights by sqrt(d/d') to maintain the standard deviation of neuron outputs. After removing parameters, they then initialize a new network with the remaining parameters, and train. Loss curves of training with random initialization vs. weight subcloning initialization.