RESONEO ·

KScorer: how Google
decides what a search means

A query such as “jaguar speed” can have several intents. The animal? The car? Before choosing an answer, Google weighs them. KScorer makes that call with 56 signals: word meanings, the top web results, clicks and the context of the search.

The question“jaguar speed”Several possible intents
The decisionKScorerEvaluate, rank, select
The answerA preferred interpretationShapes the answer shown

KScorer picks the intent of a search, and with it the answer Google shows

KScorer, for Knowledge Scorer, gives each interpretation of a query a score, ranks them and drops the weakest. The interpretation that wins decides what the searcher gets: a direct answer, a knowledge panel, a local pack or an action from Assistant.

56signals score
each interpretation
12families, from clicks
to distance
10top web results
weigh on the intent chosen

Our discovery

56 signals, each annotated by Google’s engineers

Internal Google schemas list the 56 signals KScorer uses to score an interpretation. Each one carries the engineers’ own comment: what it computes, often with its formula, its family and the stages that use it. The 2024 Content Warehouse leak named only four of them, with no definition.

Three lessons stand out. Clicks predict the score quality raters would give. The top 10 pages help decide: if most of them talk about the big cat, the animal interpretation wins, and the first result weighs ten times more than the tenth. Context then tips the balance: distance, the session, the person.

Family acronyms: UIS, User Interaction Signals (clicks) · AUIS, the same for Google Assistant · REFRES, the entities spotted in the results (most likely WebRef results; WebRef is the system that spots entities in pages) · CLIR, Cross-Language Information Retrieval, searching in other languages · QREF, QRef, the Google system that recognises the entities in the query · PIANNO, a Google Search component that annotates each page, at indexing time, with the interpretations it supports · BST, based salient terms.

  • UISCard utility
    user_interaction_score

    Clicks predict the IS score (Information Satisfaction), the satisfaction rating quality raters give. This score decides whether a card, an answer box such as a knowledge panel, is worth showing on the SERP.

    Engineers’ commentThis is an estimated IS score using user interaction data.
  • UISCard utility
    user_interaction_significant_backoff

    Flags a click score borrowed from similar queries when the exact query has too few impressions.

    Engineers’ commentA binary feature indicates whether the user_interaction_score feature is calculated with user impression data for exact given query or backoff data.
  • UISBefore resultsWith results
    user_interaction_relevance_score

    The interpretation’s relevance, estimated from aggregated interaction logs.

    Engineers’ commentAn estimated relevance score based on query interpretation's aggregated user interaction signals from logs.
  • UISWith results
    user_interaction_confidence

    N / (20 + N): a click score is trusted at 50% after 20 impressions, 83% after 100.

    Engineers’ commentUIS confidence = total_impressions / (20 + total_impressions).
  • UISCard utility
    goodness_imp

    The same rater-score prediction, built from Glue’s satisfaction signals on SERP blocks.

    Engineers’ commentThis is an estimated IS score based on user interaction signals from Glue Goodness.
  • UISCard utility
    goodness_imp_confidence

    Trust in that Glue estimate, with the same 20-impression formula.

    Engineers’ commentThis is goodness imp confidence, defined by UIS confidence = total_impressions / (20 + total_impressions).
  • UISBefore resultsWith results
    parsing_score

    A 0-to-1 score for how well the wording fits known query patterns.

    Engineers’ commentUIS Query-Pattern Score. A number between 0 and 1, based on pattern or ngrams score.
  • REFRESWith results
    refres_question_navfrac

    The question’s entities in the top 10, each result weighted by how navigational it is.

    Engineers’ commentSum of (naviness * refres score of all question entities) over top 10 results.
  • REFRESWith results
    refres_question_rank

    The question’s entities in the top 10, each result weighted 1/rank: position 1 counts ten times more than position 10.

    Engineers’ commentSum of (1/web_rank * refres score of all question entities) over top 10 results.
  • REFRESWith results
    refres_answer_navfrac

    The best answer entity in the top 10, weighted by how navigational each result is.

    Engineers’ commentSum of (naviness * max answer entity refres score) over top 10 results.
  • REFRESWith results
    refres_answer_rank

    The best answer entity in the top 10, each result weighted 1/rank.

    Engineers’ commentSum of (1/web_rank * max answer entity refres score) over top 10 results.
  • REFRESWith results
    refres_rank

    The stronger support, question or answer; the cross-language score replaces it when higher.

    Engineers’ commentMax(refres_question_rank, refres_answer_rank|0). If Max(clir_question_rank, clir_answer_rank|0) exists and is greater, it will be used.
  • REFRESWith results
    refres_navfrac

    The same maximum, with the navigational weighting.

    Engineers’ commentMax(refres_question_navfrac, refres_answer_navfrac|0). If Max(clir_question_navfrac, clir_answer_navfrac|0) exists and is greater, it will be used.
  • REFRESBefore results
    hist_refres_rank

    The previous refres_rank, kept in Feature History: the top 10 of earlier searches weighs in before the new results arrive.

    Engineers’ commentHistorical refres_rank value.
  • SALIENT_TERMS
    pq2t_vs_assistant_ibst_cosine

    The query against the Assistant’s intent-type terms, for Assistant traffic.

    Engineers’ commentCosine similarity between predicted query-to-term model and assistant intent-type-based salient terms. This is intended to be only used for ACE ranking and only populated for assistant traffic.
  • SALIENT_TERMSBefore results
    pq2t_vs_qibst_cosine

    Cosine between the terms predicted for the query and the interpretation’s own terms.

    Engineers’ commentCosine similarity between predicted query-to-term model and interpretation-based salient terms.
  • SALIENT_TERMSWith results
    qibst_result_similarity

    Cosine between the results’ terms and the interpretation’s terms, averaged with rank weights.

    Engineers’ commentRank-weighted average cosine similarity between web and interpretation-based salient terms.
  • SALIENT_TERMSWith results
    qibst_score

    A blend of the two: what the query says and what the results say.

    Engineers’ commentA blend of pq2t_vs_qibst_cosine and qibst_result_similarity.
  • SALIENT_TERMSWith results
    qibst_confidence

    Trust in QIBST, from the volume of data behind it.

    Engineers’ commentQIBST confidence score, estimated based on signal volume.
  • SALIENT_TERMSBefore results
    pq2t_vs_ibst_cosine

    The query against the intent type’s terms: the fallback when the interpretation’s own terms are missing.

    Engineers’ commentCosine similarity between predicted query-to-term model and intent-type-based salient terms. This is intended to be used as a backoff to pq2t_vs_qibst_cosine if it is missing.
  • SALIENT_TERMSWith results
    ibst_z_score

    The intent type’s terms against the query’s, normalised as a z-score.

    Engineers’ commentCosine similarity between salient terms for the intent and those for the query, normalized by Z-score.
  • SALIENT_TERMSWith results
    ibst_confidence

    Trust in IBST, from the volume of data behind it.

    Engineers’ commentIBST confidence score, estimated based on signal volume.
  • SALIENT_TERMSWith results
    sbst_score

    The interpretation against the terms of the session: earlier searches change how a query is read.

    Engineers’ commentCosine similarity between query-based interpretation salient terms and session-based salient terms.
  • ORBITWith results
    orbit_salient_terms_score_no_constraints

    Orbit’s intent model against the results’ salient terms, weighted by rank.

    Engineers’ commentRank-weighted average of dot product of Orbit intent salient terms model and web result salient terms.
  • ORBITWith results
    orbit_score_v2

    The same comparison with Orbit’s V2 models and features of the results.

    Engineers’ commentRank-weighted average of dot product of Orbit V2 intent model and features from web results.
  • ORBITWith results
    orbit_results_trusted

    Are these results reliable enough to suppress an interpretation? Document trust, language and term vectors decide.

    Engineers’ commentOrbit produces this feature to decide whether a result set can be "trusted" for making a suppression decision. This is a function of result document trust, language, presence of salient term vectors.
  • ORBITWith results
    orbit_score

    The best Orbit score across results in the same language and in other languages.

    Engineers’ commentMaximum of orbit_score_v0 and clir_orbit_score_v0.
  • ORBITBefore results
    hist_orbit

    The previous Orbit score, kept in Feature History and usable before the results.

    Engineers’ commentHistorical orbit_salient_terms_score_no_constraints value.
  • GEOWith results
    log_dcr

    The log of the direct click ratio: clicks measure how local a query is.

    Engineers’ commentFloating value in interval (-Inf, 0], representing the logarithm of the direct click ratio (DCR), used to estimate likelihood of a query's local intent.
  • GEOWith results
    local_supports_multiple_geo_results

    1 when the local answer can list several places.

    Engineers’ commentThis feature is equal to 1.0 if local supports multiple geo results. This is a query-level feature.
  • GEOWith results
    user_entity_distance

    Distance in metres from the user to the nearest place in the interpretation.

    Engineers’ commentMinimum distance (in metres) from user's current location to any entity with location signal in intent_query funcall tree.
  • GEOWith results
    cond_log_user_entity_distance

    The same distance, on a log scale.

    Engineers’ commentConditional user-entity distance, in log scale.
  • GEOWith results
    logs_query_prior

    How often local results are shown and clicked for the query’s n-grams.

    Engineers’ commentQuery-level signal that indicates how often local results are shown and clicked for different query n-grams. This signal can be used to estimate the probability of local intent.
  • GEOWith results
    mean_stable_score

    Average quality (stable score) of the top 3 local results shown.

    Engineers’ commentAggregate quality of the answer entities in a Localweb interpretation. The stable score is a result-level signal that is populated for all Localweb entities. This signal is the average stable score of the top 3 local results shown.
  • CLIRWith results
    clir_question_navfrac

    Cross-language topicality of the question’s entities, navigational weighting; a result needs 0.2 to count.

    Engineers’ commentSum of (naviness * cross-language topicality of all question entities) over top 10 results. Note that a result must have at least 0.2 topicality for an entity for it to be included.
  • CLIRWith results
    clir_question_rank

    Cross-language topicality of the question’s entities, weighted 1/rank; 0.2 minimum per result.

    Engineers’ commentSum of (1/web_rank * cross-language topicality of all question entities) over top 10 results. Note that a result must have at least 0.2 topicality for an entity for it to be included.
  • CLIRWith results
    clir_answer_navfrac

    Cross-language topicality of the best answer entity, navigational weighting.

    Engineers’ commentSum of (naviness * max answer entity cross-language topicality) over top 10 results. Note that a result must have at least 0.2 topicality for an entity for it to be included.
  • CLIRWith results
    clir_answer_rank

    Cross-language topicality of the best answer entity, weighted 1/rank.

    Engineers’ commentSum of (1/web_rank * max answer entity cross-language topicality) over top 10 results. Note that a result must have at least 0.2 topicality for an entity for it to be included.
  • CLIRWith results
    clir_pianno_rank

    PIANNO confidence measured on results in other languages, weighted 1/rank.

    Engineers’ commentSum of (1/web_rank * cross-language Pianno confidence) over top 10 results.
  • AUISBefore resultsWith results
    auis_score

    Assistant interactions aggregated over the full interpretation.

    Engineers’ commentAssistant User Interaction Score which is aggregated using full query interpretation.
  • AUISBefore resultsWith results
    auis_confidence

    Q / (k + Q): the more Assistant requests behind the score, the more it is trusted.

    Engineers’ commentAUIS confidence is used to measure the trust of auis_score. Specifically, the more Assistant requests we use to aggregate auis_score, the more confidence we have for auis_score. The definition is query_count / (k + query_count).
  • AUIS
    intent_name_auis_score

    Assistant interactions aggregated by intent name.

    Engineers’ commentAssistant User Interaction Score which is aggregated using intent name.
  • AUIS
    intent_name_auis_score_exp

    The same score, from an experiment.

    Engineers’ commentAssistant User Interaction Score which is aggregated using intent name from exp laelaps.
  • AUIS
    intent_name_pauis_score

    The Assistant score predicted by a model (pAUIS) instead of counted.

    Engineers’ commentAssistant User Interaction Score which is using pAUIS model.
  • MISC
    one_box_interpretation

    1 when the interpretation leads to a OneBox.

    Engineers’ commentThe feature is equal to 1.0 if an interpretation is a OneBoxFuncall. Otherwise, this feature will not be set.
  • MISC
    high_confidence_bit

    The structured query itself marks the interpretation as high confidence.

    Engineers’ commentA score that indicates whether or not an interpretation is of high confidence based on high confidence signal from intent query.
  • MISCCard utility
    kp_intent_prior

    The average IS (Information Satisfaction) rating raters give knowledge panel interpretations (ShowEntity).

    Engineers’ commentThe average IS rating for SingleEntity interpretation's intent function, i.e. the average IS rating for ShowEntity.
  • MISCWith results
    p13n_iq_p_good_score

    A personalised prediction that this interpretation will satisfy this user.

    Engineers’ commentPIPS predicted IQpGood Score for Knowledge Interpretations.
  • MISCWith resultsdeprecated
    web_answers_entity_alignment

    1 when the interpretation shares an entity with the top web answer passage (featured snippet).

    Engineers’ commentWhether an query interpretation has entity alignment with WebAnswers, i.e. whether there is an overlapped entity between the top main WebAnswers passage and the query interpretation.
  • INTENTWith results
    intent_query_name

    The intent’s name, for example ShowEntity, the knowledge panel intent.

    Engineers’ commentQuery intent names, for example ShowEntity, used to distinguish between KP and PP query interpretation. Currently, the value populated in this field is only the top level intent for a recursive query intent. We may consider using concatenated function names in the future.
  • INTENTWith results
    is_implicit_intent

    An implicit intent (ShowEntity, ShowXConcept): the query names something without asking anything about it.

    Engineers’ commentWhether the Query Intent is implicit. The intent names include {ShowEntity, ShowXConcept}.
  • PIANNOWith results
    noclir_pianno_rank

    How strongly the top 10 support the structured interpretation, weighted 1/rank.

    Engineers’ commentSum of (1/web_rank * Pianno confidence) over top 10 results.
  • PIANNOWith results
    pianno_rank

    The better of the two PIANNO scores, same language or other languages.

    Engineers’ commentMax(clir_pianno_rank, noclir_pianno_rank).
  • PARSINGBefore resultsWith results
    lp_lscorer_average_unbiased_pattern_score

    Average score of the known n-gram patterns found in the query.

    Engineers’ commentKnown ngram pattern score / known ngram pattern count.
  • PARSINGBefore resultsWith results
    lp_lscorer_known_ngram_fraction

    Share of the query’s n-gram patterns that are known.

    Engineers’ commentKnown ngram pattern count / all ngram pattern count.
  • QREFBefore resultsWith resultsCard utility
    qref_confidence_score

    A harmonic mean of entity recognition confidences: one doubtful entity drags the whole interpretation down.

    Engineers’ commentHarmonic mean of QRef's confidence in its annotations of all question entities. The confidence of a question entity is the minimum of its own confidence and all implied entities (if applicable). If no confidence scores are found in the intent query proto, this feature will take annotation confidences from the QueryJoin. If none is found, this feature is missing.

Comments quoted from the schemas as written; internal go/ links removed.

01

One search, several interpretations

An intent is what a person wants to achieve. An interpretation is one of the intents Google attributes to the query: how fast the animal runs or how fast the car goes. KScorer decides which interpretation comes first.

An example to follow the reasoning

The queryjaguar speed
Interpretation A

The big cat’s speed

Animal · running · movement

A possible interpretation
Interpretation B

A car’s performance

Brand · model · engine

A possible interpretation

The word “speed” fits both interpretations. The top ten results, the entities recognised and earlier clicks will decide.

To compare these interpretations, Google writes them all in the same form: an intent and its slots to fill. “Who painted Guernica?” becomes a request about the creator of a work, with Guernica in the work’s slot. KScorer then scores each of these structured interpretations.

In the schemas

  • intent_query_namethe intent’s name: ShowEntity for a knowledge panel
  • is_implicit_intentimplicit intent: the query only names something (ShowEntity, ShowXConcept)
  • one_box_interpretation1 when the interpretation leads to a OneBox
  • high_confidence_bitthe interpretation is flagged as high confidence

KScorer’s job in three steps

1 / EVALUATEAssign a score

Measure the estimated relevance of each interpretation using the available evidence.

2 / RANKOrder the interpretations

The best-supported interpretation comes first; a dominant one gets rank 0.

3 / SELECTKeep useful candidates

Interpretations without enough support are dropped before the answer is chosen.

02

The intents Google can recognise are built by its teams

KScorer chooses between interpretations, and each one matches an intent Google already knows. Google keeps an internal catalog of intents for this, the Meaning Schemas, separate from schema.org markup. Each intent describes a type of need, such as finding the actors in a film, and has slots to fill, here the film. Google teams build this catalog by hand, in an internal tool called Intent Lab.

Take these three queries: “the actors in The Dark Knight”, “who stars in The Dark Knight?” and “The Dark Knight cast”. The words change, the request stays the same: the list of the film’s actors. Google files all three under a single intent, CastOfVisualNarrativeWork, “the cast of a visual narrative work”. This intent has one slot to fill: the work. Here, the slot takes the film The Dark Knight, as the Knowledge Graph knows it. For “Inception cast”, the same intent applies, with Inception in the slot: one intent covers every film and every way of asking for its cast.

Three queries

“the actors in The Dark Knight”
“who stars in The Dark Knight?”
“The Dark Knight cast”

One intent

CastOfVisualNarrativeWork

The cast of a work
“Work” slot: The Dark Knight

Adding an intent to the catalog takes four steps, from defining the need to testing.

  1. 01 / DEFINE

    Define the need

    The team first checks what the catalog already covers, then sets what each slot accepts. The “work” slot takes films and series, entity families that the Knowledge Graph groups into collections. That choice draws the intent’s boundary.

  2. 02 / ILLUSTRATE

    Provide examples

    The team gathers queries that express the intent and others that look like it without expressing it. In each one, it marks the words that fill the slot: “The Dark Knight” in “The Dark Knight cast”. Two internal tools, Expand and Query Examples, help find other phrasings and do this marking.

  3. 03 / PREPARE

    Produce KScorer’s signals

    JobRunner, the tool that runs the automated jobs, turns these examples into training data and computes what KScorer needs to score the intent, including its salient terms: the words expected on a page about the subject. Existing signals can also be reused.

  4. 04 / EVALUATE

    Test before launch

    The team checks that the target queries are understood and that the answer shown meets the need. It compares the new version with the old one, measures server load, then has the change reviewed before release.

An intent needs at least 100 examples to be scored

The examples also train two salient-term models. Orbit is the Google system that sorts pages by intent: what they let you do (book, sign in) or their format (tutorial, product review, forum); its model learns the words typical of each intent. KBST is the second one: BST stands for based salient terms, and the K most likely stands for Knowledge, as in KScorer. Several of KScorer’s 56 signals depend on these models, which are rebuilt whenever an intent’s phrasings change.

Nexamples≥100

Below 100 training examples per intent, generating KScorer’s signals fails, and the intent cannot move on to testing.

Three tests before launch

QU_Eval

QU for Query Understanding. The test checks that Google produces the right interpretations and ranks them in the right order.

KE_Eval

KE for Knowledge Engine, the Google systems that run the Knowledge Graph and its answers. The test checks that the answer shown meets the need.

KE_SxS

SxS for side by side. The old and new versions are shown next to each other, to see what concretely changes in the results.

When the final answer is not ready, the team tests with a mock OneBox. The OneBox is the answer box Google shows above the results; the mock version shows which queries the new intent would trigger on, and keeps it from triggering too often once live.

Intents fit together like building blocks. For “Nolan films released after 2010”, Google starts from Christopher Nolan’s filmography, then adds a date constraint (DateRestrict) and a sort order (Sort); a third operator, Intersect, crosses two lists. One intent can also fill another’s slot. With a few blocks, Google understands a compound request without needing an intent for every combination.

In the schemas, a recognised intent is written as a function call, FunctionCall: the function’s name is the intent’s, its Argument entries are the filled slots, and a slot can hold another call (ArgumentValue). KScorer scores and ranks these filled calls.

In the schemas

  • FunctionCall.namethe intent’s name, for example CastOfVisualNarrativeWork
  • FunctionCall.keythe intent’s identifier in the catalog, a MID like those of Knowledge Graph entities
  • FunctionCall.unexplained_tokensquery words that no slot accounts for
  • FunctionCall.ignored_tokensrecognised context words that leave the intent unchanged
  • FunctionCall.sensitivitya sensitive intent, flagged by the intent catalog (Intent Catalog)
  • Argument.keythe slot: the film in “The Dark Knight cast”
  • ArgumentValue.funcalla slot filled by another intent: this is the nesting
  • ArgumentValue.locationa slot that takes a place; others take a person, an amount, a date or a number
  • ArgumentValue.shopping_producta slot that takes a product, for shopping intents
03

The top 10 pages tell Google which intent to choose

To understand a query, Google also looks at the pages it ranks at the top. If the pages that rank for “jaguar speed” mostly talk about big cats, the animal interpretation gains points: each page counts as a vote for the intent it talks about, and Google’s engineers call this step voting. The higher a page ranks, the more its vote counts.

The entities in the question and the answer

An entity is a specific thing the Knowledge Graph knows: a work, a person, a brand, a place. The REFRES signal family (most likely WebRef results; WebRef is the Google system that spots entities in a page) checks whether the top ten results talk about the interpretation’s entities. It gives them two roles. In “who painted Guernica?”, Guernica is the question entity, what the query is about; Pablo Picasso is the answer entity, what the searcher wants.

Question: GuernicaAnswer: Pablo Picasso

Each result weighs according to its rank. A variant weights it by its naviness instead: the likelihood that this result is the site the searcher wants to reach, such as a brand’s official site.

The weight of position

R=∑r=110Err

r is the result’s position, from 1 to 10; Er measures how much this result covers the question’s entities.

Signal name: refres_question_rank. Its variant weighted by naviness is called refres_question_navfrac.

With equal support, a result in first position contributes ten times as much to this sum as a result in tenth position.

How rank changes this calculation

Coefficient applied to each result’s support: 1 ÷ rank

A result’s vote is this coefficient multiplied by how much it covers the question’s entities.

Each vote keeps its source: the web, news, other languages, the person

A result’s support is recorded with its rank, its strength from 0 to 1 and its origin. Google’s main results are only one of seven sources.

  • MAIN_GSRthe main web results
  • NEWSnews results
  • LIVE_RESULTlive results
  • CLIR_GSRresults in other languages
  • PERSONALIZED_RESULTpersonalised results
  • CONTEXTBOOST_RESULTresults boosted by the context
  • SUPPORT_FROM_KSCORERsupport passed on by KScorer itself
  • ResultSupport.rankthe result’s rank, counted from 0
  • ResultSupport.scorestrength of the support, from 0 to 1
  • ResultSupport.estimated_navinesshow navigational the result is
  • ResultSupportProvenance.entity_group_typewhich entity is supported: QUESTION, ANSWER or INTENT

Orbit judges whether the top 10 pages are reliable enough to drop an interpretation

Orbit is the system that sorts every page into intent categories: what it lets you do (book, sign in), its format (tutorial, product review, forum) or its publisher (press, government, merchant). KScorer compares the salient terms the Orbit model links to the intent with those of the top 10 pages. Orbit also judges whether these pages are reliable enough to drop an interpretation, based on trust in the documents, their language and whether their salient terms are available.

Reliable web evidence

A stronger basis for deciding which interpretations to retain or discard.

In the schemas

  • orbit_score_v2Orbit V2 intent model against the results, rank-weighted
  • orbit_scorethe best Orbit score, same language or other languages
  • orbit_results_trustedcan these results justify suppressing an interpretation?
  • hist_orbitthe previous Orbit score, usable before the results
04

Clicks predict what quality raters would say

Google has its results rated by quality raters, human evaluators who follow its guidelines. Their satisfaction score is called IS, for Information Satisfaction. Raters only see a small share of queries; for all the others, KScorer predicts their score from clicks. That is the job of the UIS family, for User Interaction Signals: its main signal, user_interaction_score, is “an estimated IS score using user interaction data”. goodness_imp makes the same estimate from Glue, the memory of interactions with every block of the SERP.

In the US antitrust trial, the plaintiffs describe this score in their findings (Plaintiffs’ Proposed Findings of Fact, ¶ 559-560): each query evaluated by a rater gets an IS score, and Google rolls these scores up into a score out of 100. Google measures it with IS4@5: the fourth version of the score, computed on the top five results. Google trains NavBoost, RankBrain and DeepRank on these scores (testimony of expert Douglas Oard).

In the schemas

  • user_interaction_scoreclicks turned into a predicted IS score
  • goodness_impGlue’s satisfaction signals turned into a predicted IS score
  • user_interaction_relevance_scorerelevance estimated from interaction logs
  • user_interaction_confidenceN / (20 + N): trust grows with impressions
  • user_interaction_significant_backoffscore borrowed from similar queries
  • parsing_scorehow well the wording fits known patterns, 0 to 1

More observations, greater confidence

A click score built on five impressions is not worth one built on five hundred. KScorer weighs it with a confidence factor that grows with the number of impressions.

CUIS=N20+N

N is the number of impressions behind the score.

The factor is called user_interaction_confidence. The relevance score drawn from interactions is user_interaction_relevance_score.

The first impressions count most: 20 impressions give 0.5, 100 give 0.833, then the curve flattens towards 1.

Confidence rises, then levels off

UIS confidence factor from zero to two hundred impressions The curve follows N divided by 20 plus N. It reaches 50% at 20 impressions, 83.3% at 100 and 90.9% at 200. 100%75%50%25%0% 02050100150200 Confidence factorNumber of impressions

Calculated confidence factor50%

The confidence factor weighs the click score; relevance is a separate signal.
20%5 impressions
50%20 impressions
83.3%100 impressions
98%1,000 impressions

On a rare query, the click score is borrowed from similar queries, and user_interaction_significant_backoff flags it: long-tail queries inherit the click behaviour of their more frequent neighbours.

05

Salient terms test the query, the interpretation and the pages against each other

For each subject, Google computes salient terms: the words expected to appear prominently on a relevant page. For the big cat, “predator”, “rainforest”, “prey”; for the car, “engine”, “horsepower”, “0 to 60”. KScorer compares these word lists across the query, the interpretation and the top 10 pages: the interpretation whose terms show up in the results gains points.

The queryExpected terms

The terms the model predicts for the query.

The interpretationA candidate intent

Its own salient terms.

Web resultsObserved terms

The salient terms of the top 10 pages.

The QIBST score (query interpretation-based salient terms, the salient terms specific to one interpretation) makes two comparisons: the query against the interpretation, then the top 10 pages against that same interpretation, with more weight on the first ones.

qibst_score combines pq2t_vs_qibst_cosine, for the query–interpretation relationship, and qibst_result_similarity, for the results–interpretation relationship. In these names, pq2t means the terms the model predicts for the query (predicted query-to-terms).

Do two term lists stress the same words?

Each list of salient terms becomes a vector, with one weight per word. Cosine similarity measures whether two vectors point the same way, that is, whether two lists give weight to the same words.

cos(a,b)=a·b‖a‖‖b‖

Two lists that stress the same words score close to 1; two lists with no word in common score 0.

Three term lists, from intent type to session

IBST

Intent-based salient terms: the terms of the intent type, shared for instance by every cast request.

QIBST

Query interpretation-based salient terms: the terms of this interpretation, with its subject and entities, such as the cast of The Dark Knight.

SBST

Session-based salient terms: the terms of the earlier searches in the session, which change how the next one is read.

When the interpretation’s own terms are missing, the intent type’s terms take over (pq2t_vs_ibst_cosine).

In the schemas

  • ibst_z_scoreintent-type terms against the query, as a z-score
  • sbst_scorethe interpretation against the session’s terms
  • qibst_confidencetrust in QIBST, from the volume of data
  • pq2t_vs_assistant_ibst_cosinethe same comparison for Assistant traffic
06

Recognising the right entities across languages

Understanding “jaguar” means linking the word to the right Knowledge Graph entity: the big cat or the car brand. A more complex request names several, and each one has to be recognised correctly. KScorer measures how sure each recognition is, then checks that the results talk about these entities, in other languages too.

QRef: the least certain entity sets the confidence of the whole interpretation

QRef (Q for query, mirroring WebRef) is the system that spots entities in the query and links them to the Knowledge Graph, each with a confidence from 0 to 1. The qref_confidence_score signal combines these confidences with a harmonic mean, which the lowest value drags down.

H=n∑i=1n1ci

n is the number of entities; ci is the confidence associated with each one.

210.9+10.2≈0.327

Two entities, one recognised at 0.9, the other at 0.2: a simple average would give 0.55, the harmonic mean falls to 0.33. A single doubtful entity weakens the whole interpretation.

In the schemas

  • QRefAnnotation.freebase_midthe MID of the recognised entity, its Knowledge Graph identifier
  • QRefAnnotation.confidence_scoreconfidence in the recognition, from 0 to 1
  • QRefAnnotation.annotated_spanthe words of the query that name it
  • QRefAnnotation.resolution_scorean implicit mention, such as “my hotel”
  • QRefAnnotation.merged_implied_entityentities implied by the mention, merged in

CLIR: pages in other languages vote too

CLIR, for Cross-Language Information Retrieval, means searching across languages: Google also queries the web in languages other than the query’s. For a subject covered mostly in English, English pages thus back the interpretation of a French query. A result counts when the entity weighs enough in its subject, with a topicality of at least 0.2.

topicality≥0.2

clir_question_rank then sums the support of the top ten results, weighted 1/rank, as REFRES does in the query’s language.

Pianno: each page keeps its confidence for an interpretation

Pianno is a Google Search component that annotates pages at indexing time (ANNO most likely for annotation). For each page, it keeps a confidence: how strongly the page supports a query interpretation. KScorer adds up these confidences over the top 10, each divided by the rank. The pianno_rank signal takes the maximum of its standard and multilingual variants.

In the schemas

  • clir_question_rankquestion entities in other languages, weighted 1/rank
  • clir_answer_rankthe best answer entity in other languages
  • clir_pianno_rankPianno confidence on results in other languages
  • pianno_rankthe better Pianno score, any language
07

Context guides the answer

For “pharmacy open”, proximity becomes useful evidence. For a request to Assistant, the history of voice interactions or actions can help clarify the intent. KScorer has signals for both.

Six signals for assessing local intent

Entity proximity

user_entity_distance measures the minimum distance, in metres, between the user and the located entities in the interpretation. A variant transforms this distance onto a logarithmic scale.

Patterns of local searches

logs_query_prior counts how often local results are shown and clicked for groups of words. log_dcr, the logarithm of the direct click ratio (DCR), reads the same intent from clicks.

The quality of the local answer

mean_stable_score averages the stable score, the quality score of each local listing, over the top three local results. Another signal indicates whether multiple geographic results are supported.

Beyond location, p13n_iq_p_good_score predicts whether an interpretation will satisfy this particular user: p13n is short for personalization (a p, thirteen letters, an n), and p_good is most likely the probability that the interpretation is good.

The case of Google Assistant

Interactions suited to action requests

The AUIS family, for Assistant User Interaction Score, does for Google Assistant what UIS does for Search: it scores an interpretation from how Assistant users react. It aggregates these reactions for a complete interpretation, with auis_score, or for an intent type, with intent_name_auis_score. In intent_name_pauis_score, the p marks a prediction: the score comes from a model instead of being counted.

The auis_confidence confidence factor follows the same volume-based logic as UIS.

In the schemas

  • auis_scoreAssistant interactions on the full interpretation
  • intent_name_auis_scorethe same, aggregated by intent name
  • intent_name_pauis_scorethe same, predicted by a model
  • auis_confidenceQ / (k + Q): trust grows with Assistant requests
CAUIS=Qk+Q

Q is the number of Assistant queries considered. k sets how fast confidence rises.

08

KScorer votes twice: before and after the results

Some evidence exists as soon as the query is typed, the rest arrives with the results. KScorer therefore scores interpretations in two rounds, which Google calls votes: a first one on the query alone, then a second one once the documents are in.

  • PRE_RESULTS_VOTING11 signals vote on the query alone
  • PACKER_VOTING41 signals vote while the SERP is assembled
  • CARD_UTILITY6 signals decide whether an answer card, such as a knowledge panel, is worth showing
  • DERIVED_INTENT_RANKINGa fourth use: ranking the derived intents, added alongside the query’s interpretation

Before the results

A first vote on the query alone

Query patterns, recognised entities, salient terms and historical signals provide an initial set of evidence. This first vote is PRE_RESULTS_VOTING.

LanguageEntitiesHistory

With the results

Compare interpretations with the web

The pages found then vote: through their entities (REFRES), through pages in other languages (CLIR), through their intent categories (Orbit) and through their salient terms (QIBST). This second vote, PACKER_VOTING, takes place in the packer, the module that assembles the SERP.

Document supportReliabilityUtility
A memory of past evaluations

Feature History keeps signal values from one search to the next. With hist_orbit and hist_refres_rank, the pages that backed an interpretation in earlier searches already vote in the first round, before the new results are known.

The two votes produce pre_results_ranking_score and post_results_ranking_score. Twiddlers, functions that rework a score after the fact, then adjust them: pre_twiddled_ranking_score and twiddled_ranking_score hold the score before and after adjustment.

Six signals decide whether an answer card is worth showing. A third use, CARD_UTILITY, judges whether a card is useful: cards are the answer blocks Google shows on top of the blue links, such as a knowledge panel or a direct answer. Its six signals say a lot about what Google trusts:
4come from clicks: user_interaction_score, the flag that marks a score borrowed from similar queries, goodness_imp and its confidence.
1comes from raters: kp_intent_prior, the average IS rating of knowledge panel interpretations.
1comes from entity recognition: qref_confidence_score.
09

From score to answer: rank 0 goes to the dominant interpretation

Three values leave KScorer with each interpretation: a score, a rank and a dominance flag.

The KScore

The interpretation’s score after twiddlers. Systems that choose between several answers or actions read it.

k_score

The KScorer rank

A position in the ordering of interpretations. When a dominant interpretation exists, it receives rank 0. The others are ordered from 1 to N.

kscorer_rank

Dominance

The interpretation that clearly wins. It takes rank 0 and opens the way to a faster SERP.

dominant

These values travel beyond Search. In Google Assistant, the Prefulfillment Ranker, which ranks interpretations before the Assistant runs an action or gives an answer, reads them. UnderstandingSignals passes it the KScorer rank along with the 56 signals, filed under the name of the Query Understanding Service, the service that analyses the query (qus_features).

In the schemas

  • predicted_intent_confidencethe confidence predicted for the intent
  • is_intent_from_orbitthe intent was proposed by Orbit
  • has_answer_groupthe interpretation comes with a Search answer
  • is_qus_dominantdominant according to query understanding
  • search_response_is_low_utilitythe Search response is judged of low utility
  • qus_phasethe understanding phase: rewrite, probe query, image…
The available answer matters too. The states has_answer, suppressed and top_ranked_non_suppressed track the presence of an answer, the suppression of an interpretation and the first position among retained interpretations.

A dominant interpretation gets the first block of the SERP sent early

First Result Out sends the first block of the page before the rest is ready. KScorer holds the key: early sending is refused when

  • the top interpretation (QI, for query interpretation) is not dominantINELIGIBLE_TOP_QI_NOT_DOMINANT
  • a competing interpretation dominatesINELIGIBLE_COMPETING_QI_IS_DOMINANT
  • the top interpretation has no click data (UIS)INELIGIBLE_TOP_QI_NO_UIS
  • its KScorer rank is above a thresholdINELIGIBLE_KSCORER_RANK_ABOVE_THRESHOLD
  • its KScorer score and a second score, HTF, are below their thresholdINELIGIBLE_HTF_AND_KSCORER_SCORE_BELOW_THRESHOLD
10

To rank, a page must first match the intent Google picks

Google settles a query’s intent before it ranks pages. A page written for the losing interpretation competes on a SERP built for another intent.

  1. Check which intent the SERP has chosen

    The top 10 pages weigh on the intent chosen, all the more as they rank high, and Feature History keeps that verdict for the next searches. On an ambiguous query, today’s SERP shows the winning interpretation. A page built for the other one targets a query that names it, such as “Jaguar F-Type top speed”.

  2. Make every entity unambiguous

    KScorer combines entity confidences with a harmonic mean: one doubtful entity drags the whole interpretation down. Name brands, models and places the way the Knowledge Graph knows them, and tie them to their entity with structured data (sameAs).

  3. Satisfied clicks keep an answer card on the SERP

    Four of the six signals that decide whether an answer card shows come from clicks, two of them as a predicted rater score. For a brand, the knowledge panel lasts as long as searchers find what they came for: accurate facts on the site, in the Business Profile and in Wikidata.

  4. For local queries, the top three listings set the bar

    mean_stable_score averages the quality of the top three local results, and distance is measured in metres. The local interpretation wins when nearby listings are good: a complete Business Profile, accurate hours and reviews feed that quality.

Behind a SERP that appears in a fraction of a second, Google has already compared several interpretations of the query, and the clicks of earlier searchers have weighed in.

A closer look at the mechanisms

The schemas, field by field

15 schemas and 287 fields from KScorer, to browse here or download as a single file.

File paths9 files · 15 types · 287 fields · 19 values
assistant/grounding/ranker/proto/prefulfillment_signals.proto1 type · 92 fields
assistant/orchestration/understanding/platform/proto/understanding_signals.proto1 type · 10 fields
knowledge/answers/config/meaning_schema_slot_key.proto1 type · 2 fields
knowledge/answers/intent_catalog/meaning_schema_key.proto1 type · 2 fields
knowledge/answers/intent_query/intent_query.proto3 types · 59 fields
knowledge/answers/interpretation_ranking/proto/kscorer_feature.proto4 types · 66 fields · 19 values
logs/proto/knowledge/interpretation/intent_query.proto1 type · 8 fields
nlp/semantic_parsing/proto/qref_annotation.proto1 type · 37 fields
universalsearch/ranking/new_packer/knowledge/voting/matchers/proto/result_support.proto2 types · 11 fields