Google AI estimates body composition with a smartphone camera

Google has developed an AI system that can estimate a person’s body composition using nothing more than a smartphone camera.

Google has created a new technology designed to estimate an individual’s body composition, with a camera as the only essential piece of equipment. At present, it remains a research prototype, but the findings are highly promising and could usher in a new era for healthcare. Although this information can already be obtained through dual-energy X-ray absorptiometry, or DXA, the equipment is expensive, requires specialist facilities and exposes patients to small doses of radiation. It is therefore not a practical option for everyday measurements.

PhotoScan uses 2D images from a smartphone camera

Google’s technology can estimate a person’s body-fat percentage, as well as A/G ratios (“compares fat stored in your trunk [...] with that stored in your hips and thighs”) and VS ratios (“the distinction between internal, high-metabolism fat surrounding your organs, and subcutaneous fat stored just beneath your skin”). To generate these detailed estimates, the AI requires only 2D images captured with a standard smartphone camera. Google pre-trained a deep neural network using data from more than 35,000 UK Biobank participants, before refining the technology with a group of 667 adults.

Results nearly as accurate as dual-energy X-ray absorptiometry

As mentioned above, the results are encouraging. Google acknowledges that dual-energy X-ray absorptiometry delivers the most accurate findings, but says its technology achieves accuracy close to that provided by this specialist equipment. Compared with connected accessories, which are more convenient, this photo-based AI does more than report body-fat percentage by supplying more detailed information.

These details could be valuable in healthcare for predicting insulin resistance in individuals. “Our study demonstrates that optical body-composition estimation from smartphone images is both technically feasible and clinically relevant. Although it is still a research prototype, this approach paves the way for accessible, non-invasive screening for insulin-resistance risk,” Google explains. Data delivered by the AI, called “PhotoScan”, could in particular supplement the everyday physiological information recorded by smartwatches to “build a complete picture” of metabolic risk.

Google’s growing focus on health

Google is presenting PhotoScan shortly after unveiling new AI-powered health features alongside the Google Pixel 11 series and the new Pixel Watch.

Among the health functions due to arrive on eligible Pixel Watch models and the Fitbit Air is a feature that monitors insulin-resistance trends. “A first for Pixel and Fitbit connected devices, Insulin Resistance Trends analyses physiological data over several weeks to track changes in your metabolic health without requiring a single drop of blood,” Google said.