What Google sees in an image

Take two queries on Google Shopping US: “dresses” and “bathroom sinks”. In the filter column, dresses can be filtered by “Sleeve Style” (Long Sleeve, Half Sleeve), “Silhouette”, “Formal Occasion”; sinks by “Installation” (Undermount, Self-rimming, Wall-mount) and “Style” (Vessel, Pedestal, Console). The values of these filters reappear in the /a/ Google puts on photos (Long sleeve, Undermount sink, For prom), and “Sleeve Style” is also a /j/. The /j/ plays the part of the filter, the /a/ that of the checked box.

On the left, Google Shopping filters (United States); on the right, the matching labels Google attaches to photos

Query “dresses”

• 
  Sleeve style/j/6y5h45Long sleeve/a/dr73ssHalf sleeve/a/4x501n
• 
  For prom/a/bckrxlFor homecoming/a/g6jk9w
• 
  Asymmetrical/a/42lnhn

Query “bathroom sinks”

• 
  Sink type/j/4fs9rdUndermount sink/a/8y6cckSelf rimming sink/a/8dv91hWall mount/a/g6jfls
• 
  Vessel sink/a/znv7b6

Until spring 2026, every image generated by Gemini came with 100 labels. These labels are no longer visible. Conversations shared at that time keep them: the 67,499 images we studied come from the public conversations of our Gemini study. Here are fifteen public conversations, each with a link: pick an image, filter the labels by vocabulary, set the threshold to 0.5 to see what the Vision API would keep.

fetch('/_/BardChatUi/data/batchexecute?rpcids=ujx1Bf', {
  method: 'POST',
  headers: {'Content-Type': 'application/x-www-form-urlencoded;charset=UTF-8',
            'X-Same-Domain': '1'},
  body: 'f.req=' + encodeURIComponent(JSON.stringify([[['ujx1Bf',
    JSON.stringify([null, location.pathname.split('/').pop(), [4]]), null, 'generic']]]))
}).then(r => r.text()).then(t => console.log([...new Set(
  t.replaceAll('\\"', '"').match(/\["[^"]+","\/[a-z]\/\w+",[\d.]+\]/g))].join('\n')))

The Vision API, the image analysis service Google sells to developers, uses the same /j/ IDs as Gemini’s images: Table top, Bagged packaged goods, Playing in the snow. Each of these twelve freely licensed Wikimedia Commons photos gets at least one (in orange), on a detected area of the photo or on the whole image.

Googlebot-Image fetches the image, Amarna, Google Search’s image system, gathers everything the models see in it, then Google ties it to a few pages and sorts it into the flags that Google Images filters read. Compared with the 2024 leak, the Google code we had access to adds everything related to AI and safety to what the index keeps about an image: a caption written by Google, its entities, C2PA provenance (the “Content Credentials” that certify an image’s origin), the Google-Extended opt-out (the robots.txt user agent token that keeps content out of Gemini training and grounding), a child safety check. It also adds the frame of the main subject and the source image among the copies.

1. 1

   Googlebot-Image fetches every image

   “All images are crawled using googlebot-images”, the leak says. Crawl status is tracked per cluster of copies, with the reason for any failure. robots.txt rules aimed at Googlebot-Image apply to Google Images, Discover and every Google Search feature that shows images.

   ImageData.robotedAgentscrawlers blocked by robots.txtImageMoosedogCrawlStatecrawl status, per cluster of copies

2. 2

   Amarna gathers everything the models see in the image

   Amarna, the system that stores and analyzes Google Search’s images and videos, keeps for each image its labels, faces and products, the text read in it by three OCR engines side by side, its quality and SafeSearch scores, its embedding, its near-duplicate fingerprint and its likelihood of being generated. Text read on a package also links the photo to the right Shopping product listing.

   OcrSummarytext read in the imageIm2ShopOCRFeaturepackage text linked to Shopping

3. 3

   Google writes a caption and lists the entities

   Every indexed image gets a caption written by a model, with the probability the model gives it, and the list of Knowledge Graph entities recognized, each with its score. The main subject is framed, with a separate bounding box for the face and its own entities. Each label carries two topicality measures and a score of how much it tells about the image; some labels are inherited from visually similar images.

   packed_multiref_captioncaption written by Googlepacked_multiref_entitiesrecognized entitiescutoutframe of the main subjectinformativeness_scorehow much the label tells about the image

4. 4

   Copies are grouped, ranked and dated

   All copies of an image form a ranked group, where rank 1 is the best. Google keeps, for the whole group, the first crawl date and the source image’s address: publishing the original on your site first makes you that source. A stock photo reused by a hundred sites competes for a single spot.

   rankInNeardupClusterrank in the cluster of copiescontentFirstCrawlTimefirst crawl of the contentimage_neardup_sourcesource image among the copies

5. 5

   A fixed number of pages carry the image; the others are “rejected”

   For each image, Google keeps a fixed number of referring pages; on the other pages that show it, the image is said to be “rejected”. It records whether the image is displayed on the page or only linked, keeps the surrounding text, used to scroll visitors straight to the image when they arrive from Google Images, and recognizes a companion image domain, a CDN or a subdomain, when the link holds over time.

   referrerDocidreferring pageisVisibledisplayed or only linkednearbyTextnearby textcodomainStrengthtie to a companion image domain

6. 6

   The Google-Extended opt-out follows the image

   Google-Extended is the robots.txt user agent token that controls whether content is used to train Gemini models and for grounding. For each image, Google keeps that opt-out and what decided it: the robots.txt, or the page that displays the image. It adds a compliance status for the common corpus where Google gathers the content its AI models may use, checked against those same pages. Indexing in Google Images is still governed by Googlebot-Image.

   robots_policy_signalGoogle-Extended opt-outeffective_google_extended_sourcewhat decided: page or robots.txtgcc_image_url_policy_signalscommon corpus compliance

7. 7

   Mustang sorts the image into the filters

   Mustang, the system that serves Google Search’s index, keeps with the image its faces and crops, ready to serve, and the “restricts”: the 86 flags that Google Images filters read.

   PrecomputedRestrictsthe filter flags

Take the query “scandinavian chair”. For Google Images, the query is also a vector: it goes through Starburst’s text encoder, Google’s image embedding model, which places it in the same space as images, then gets compared directly with their vectors. A chair that does not look Scandinavian sits further from the query in that space, even with alt text that says otherwise. Visual content, image quality, its place on the page and clicks each feed into the ranking.

Before a search comes in, Google has already sorted every image into flags: color, type, faces, aspect ratio, size, format, license, SafeSearch. The 2024 leak names these “restricts”, computed before the serving index is built; the Google code we obtained lists them: 86 flags. A filter judges the file itself: its actual dimensions, format, colors and declared license.

When given a photo, AI Mode relies on Google Lens, Google’s visual search, to identify each object, then runs queries on the whole image and on each of its objects: query fan-out applied to images, the “visual search fan-out”, which breaks a single photo into dozens of sub-queries. Here we explain how AI Mode searches for and scores the images in its answers.

“About this image” tells whether an image was created or edited with AI, based on SynthID, the invisible watermark Google DeepMind embeds in the pixels at creation time, and on C2PA metadata (Coalition for Content Provenance and Authenticity), the “Content Credentials” standard. More than 20 billion pieces of content carry the SynthID watermark (November 2025).

Work on an image like a page: what it shows, the file that carries it, the page that displays it and the clicks it gets each count at their own stage.

1. Show what the query and the filters ask for

   Google compares the query with the visual content, through Starburst, its image embedding model, and pamir, its per-query visual score; with Gemini Embedding 2, open to developers, you can measure that closeness between your queries and your images. A sharp packshot, whole product on a neutral or transparent background, returns its packaging type; a shelf returns none. Show in full what the filter asks for: a dress’s length, the sleeve, the rim of a sink. A lifestyle shot adds the occasion (wedding, moving, beach) without taking over the subject: Google keeps it as setting.

2. Publish the original on your own site, first

   Google keeps, for the whole group of copies of an image, the first crawl date and the source image’s address: publishing the original on your site first makes you that source. Google also flags stock photos as such, and a hundred sites publishing the same photo compete for a single spot. A merchant who sends product listings to resellers through XML feeds, or a publisher who syndicates content, should leave a few minutes between going live on its own site and sending the content out: Google then finds the images on its site first.

3. Display the image on the page, large, near the text about it

   Google indexes images in <img> elements and records whether the image is displayed on the page or only linked. Salience and estimated relevance follow the image all the way to web ranking. Search Central advises placing the image near the text about it; Google keeps that text with the image.

4. Make the alt text, the on-page caption and the image say the same thing

   Google writes its own caption for every image and attaches the entities it recognizes; it cross-checks the alt text, computer vision and the page text. An alt text that describes what the image shows reinforces that reading.

5. Serve a file that fits the filters

   Filters read the file: its real dimensions, aspect ratio, format, transparency and license. Serve the original at full size and declare its license in structured data or IPTC for the “Usage rights” filter and the “Licensable” badge.

6. A thumbnail that appeals without clickbait

   NavBoost, Google Search’s click memory, measures how appealing the thumbnail is, query by query: hovers and clicks relative to impressions. An image that draws clicks from unrelated queries gets flagged as a “click magnet”. A visual that reads well at small size and stays true to the query wins on both counts.

7. Keep provenance data, set Google-Extended on the image host

   Provenance checks read the file: a CDN that strips metadata erases C2PA and IPTC, the data that says where the image comes from; an image shrunk too far makes SynthID, Google’s watermark, abstain. For each image, Google keeps the Google-Extended opt-out, the robots.txt user agent token for Gemini training and grounding, with its source: the robots.txt, or the page that displays the image. A robots.txt covers a single host: for a CDN or an image subdomain to follow the site’s rule, set it in their robots.txt too.

8. Restaurants and hotels: name the dishes, get the place photo right

   A complete menu in Google Business Profile gives customers’ photos their names, and dish names entered by the business take precedence over those added by customers. For a place photo, Google prefers a large, landscape image, distinct from the others and true to the name.