VRM: Vision Reasoning Models
It has been fascinating to read about the different applications that are being hooked up to OpenClaw or Hermes or the like. Imagine you have an agent hooked up to the camera feed from your refrigerator. The agent would analyze the different items and signal what’s running low so that you can pick up those items on the way back home in the evening. You get home and realize that you are out of cheese and are mad as to why the agent didn’t put cheese on the list. Well, in many (most?) cases, a proactive reasoning loop is absent in such setups. For instance, the milk can may have occluded the view where the cheese is usually kept. Without this insight, the frustration may recur tomorrow for some other item.
In a similar vein, fulfilling the key premise of Wearables - users should not have to take the phone out of their pocket - rests on providing robust support for reasoning when things fail (owing to, say, for instance, camera pose, motion blur, poor lighting/glare or occlusion). This is particularly important for displayless glasses as a silent failure would result in poor user experience (the user would have to find his/her phone and key in their pincode).
A key limitation of traditional VLMs is “textual drift” (or “visual blindness”). These models are distilled on text-only reasoning datasets and hence are completely disconnected from the actual pixels of the image. On the other hand, VRMs (Visual Reasoning Models) operate either in the pixel or continuous abstract latent space and consequently, fare better than VLMs. To this end, I penned down, along with Bhargav, Aayush, Siddharth and Saurabh, the evolution of reasoning models - refer to the PDF linked below. Feedback welcome!
Thanks Vikas for the feedback on the early versions of the writeup.