Do VLMs Need Vision Transformers? A controlled study of Transformer, state space, and hybrid vision backbones for frozen vision-language models. Finding: - Under matched settings, VMamba improves localization while staying competitive on open-ended VQA, making SSMs a practical alternative to ViTs. - Detection or segmentation pretraining generally improves VQA and localization, with the largest gains appearing in backbones that need more spatial inductive bias. - Better classification scores and naive scaling do not consistently predict stronger downstream VLM behavior, especially for grounding-sensitive tasks. - Some dense-objective checkpoints fail sharply in localization, but simple interface and connector adjustments recover much more robust behavior. Paper Title: Do VLMs Need Vision Transformers? Evaluating State Space Models as Project: lab-spell.github.io/vlm-ssm-… Link: arxiv.org/abs/2603.19209