Thanks Vivek and Milind for the feedback on the early versions of the writeup.

Thanks for the feedback on the early versions of the writeup.

Download Paper

It has been fascinating to read about the different applications that are being hooked up to OpenClaw or Hermes or the like. Imagine you have an agent hooked up to the camera feed from your refrigerator. The agent would analyze the different items and signal what’s running low so that you can pick up those items on the way back home in the evening. You get home and realize that you are out of cheese and are mad as to why the agent didn’t put cheese on the list. Well, in many (most?) cases, a proactive reasoning loop is absent in such setups. For instance, the milk can may have occluded the view where the cheese is usually kept. Without this insight, the frustration may recur tomorrow for some other item.

In a similar vein, fulfilling the key premise of Wearables - users should not have to take the phone out of their pocket - rests on providing robust support for reasoning when things fail (owing to, say, for instance, camera pose, motion blur, poor lighting/glare or occlusion). This is particularly important for displayless glasses as a silent failure would result in poor user experience (the user would have to find his/her phone and key in their pincode).

A key limitation of traditional VLMs is “textual drift” (or “visual blindness”). These models are distilled on text-only reasoning datasets and hence are completely disconnected from the actual pixels of the image. On the other hand, VRMs (Visual Reasoning Models) operate either in the pixel or continuous abstract latent space and consequently, fare better than VLMs. To this end, I penned down, along with Bhargav, Aayush, Siddharth and Saurabh, the evolution of reasoning models - refer to the PDF linked below. Feedback welcome!

Thanks Vikas for the feedback on the early versions of the writeup.

Download Paper

As LLMs have gotten progressively better — winning IMO gold medal, solving long standing problems in several disciplines, coding agents becoming ubiquitous etc. — there has been a push towards Physical AI. To this end, developing World Models has been a rage nowadays. Recently, Jitendra Malik and Yann LeCun shared a brief historical perspective and some recent work on World Models.

It’s fascinating that the early work can be traced back to the early 1940s and how multi-disciplinary — cross-cutting, for example, psychology, cognitive science and neuroscience, model Predictive Control (MPC) — it has been.

It has been interesting to see the learning evolution — Observation → Imitation → Explore-and-Exploit (/reinforcement) — with my 5 yr and 1.5 yr old. I have come across a few instances of the following:

• Carey and Bartlett’s “Fast Mapping” — a mechanism whereby a child can learn a new word or concept from a single, incidental exposure and retain that knowledge long-term.
• Karmiloff-Smith’s “Representational Redescription” — reorganization of current information leading to a child developing explicit knowledge about a complex topic they were never formally taught.

The above, coupled with self-reflection (which has shown benefits to tasks such as coding), could drive learning acceleration and consequently have material implications for science — for example, discovery on the positive side and security on the negative side.