TopoNets

TopoNets: High-Performing Vision and Language models


with Brain-Like Topography


ICLR 2025

The organization of neurons in the brain is highly structured: neurons performing similar functions are located near one another. This “topographic organization” is a fundamental principle of primate brains and plays an important role in shaping the brain’s representations.

Here we introduce TopoLoss, a simple, scalable, and effective method for inducing brain-like topography into leading AI architectures (convolutional networks and transformers) with minimal drop in model performance. The resulting models, TopoNets, are the highest-performing supervised topographic neural networks to date.

pip install topoloss

import torchvision.models as models
from topoloss import TopoLoss, LaplacianPyramid

model = models.resnet18(weights = "DEFAULT")

topo_loss = TopoLoss(
    losses = [
        LaplacianPyramid.from_layer(
            model=model,
            layer = model.fc, ## layer to apply topoloss on
            factor_h=8.0,
            factor_w=8.0,
            scale = 1.0  ## strength, equivalent to "tau" in the paper
        ),
    ],
)
loss = topo_loss.compute(model=model) ## add this to your training loss during training!
loss.backward()

We trained a bunch of NanoGPTs with topoloss applied on the `c_fc` modules. Models with only 20% of the parameters remaining on the on the `c_fc` modules showed no significant drop in performance. τ=0 indicates the baseline model with no topoloss applied and τ=50.0 indicates the model trained with the strongest topoloss.

Studies using fMRI and invasive recordings (for example: Regev et al.) have provided evidence for distinct temporal receptive fields. Based on these results, we wondered if neurons in TopoNets were clustered by their temporal integration windows. We used a new word-swapping method from a recent study to investigate this in TopoNets. Green maps show neurons which information over shorter (exponential) token-windows and red maps show neurons which information over longer (power law) token-windows.


@inproceedings{

deb2025toponets,

title={TopoNets: High Performing Vision and Language Models with Brain-Like Topography},

author={Deb, Mayukh and Deb, Mainak and Murty, N Apurva Ratan},

booktitle={The Thirteenth International Conference on Learning Representations},

year={2025},

url={https://openreview.net/forum?id=THqWPzL00e}
}

}


Keep your files stored safely and securely with the SanDisk 2TB Extreme Portable SSD. With over 69,505 ratings and an impressive 4.6 out of 5 stars, this product has been purchased over 8K+ times in the past month. At only $129.99, this Amazon’s Choice product is a must-have for secure file storage.

Help keep private content private with the included password protection featuring 256-bit AES hardware encryption. Order now for just $129.99 on Amazon!

Help Power Techcratic’s Future – Scan To Support

If Techcratic’s content and insights have helped you, consider giving back by supporting the platform with crypto. Every contribution makes a difference, whether it’s for high-quality content, server maintenance, or future updates. Techcratic is constantly evolving, and your support helps drive that progress.

As a solo operator who wears all the hats, creating content, managing the tech, and running the site, your support allows me to stay focused on delivering valuable resources. Your support keeps everything running smoothly and enables me to continue creating the content you love. I’m deeply grateful for your support, it truly means the world to me! Thank you!