KScorer picks the intent of a search, and with it the answer Google shows
KScorer, for Knowledge Scorer, gives each interpretation of a query a score, ranks them and drops the weakest. The interpretation that wins decides what the searcher gets: a direct answer, a knowledge panel, a local pack or an action from Assistant.
each interpretation
to distance
weigh on the intent chosen
Our discovery
56 signals, each annotated by Google’s engineers
Internal Google schemas list the 56 signals KScorer uses to score an interpretation. Each one carries the engineers’ own comment: what it computes, often with its formula, its family and the stages that use it. The 2024 Content Warehouse leak named only four of them, with no definition.
Three lessons stand out. Clicks predict the score quality raters would give. The top 10 pages help decide: if most of them talk about the big cat, the animal interpretation wins, and the first result weighs ten times more than the tenth. Context then tips the balance: distance, the session, the person.
Family acronyms: UIS, User Interaction Signals (clicks) · AUIS, the same for Google Assistant · REFRES, the entities spotted in the results (most likely WebRef results; WebRef is the system that spots entities in pages) · CLIR, Cross-Language Information Retrieval, searching in other languages · QREF, QRef, the Google system that recognises the entities in the query · PIANNO, a Google Search component that annotates each page, at indexing time, with the interpretations it supports · BST, based salient terms.
user_interaction_scoreClicks predict the IS score (Information Satisfaction), the satisfaction rating quality raters give. This score decides whether a card, an answer box such as a knowledge panel, is worth showing on the SERP.
Engineers’ commentThis is an estimated IS score using user interaction data.
user_interaction_significant_backoffFlags a click score borrowed from similar queries when the exact query has too few impressions.
Engineers’ commentA binary feature indicates whether the
user_interaction_scorefeature is calculated with user impression data for exact given query or backoff data.user_interaction_relevance_scoreThe interpretation’s relevance, estimated from aggregated interaction logs.
Engineers’ commentAn estimated relevance score based on query interpretation's aggregated user interaction signals from logs.
user_interaction_confidenceN / (20 + N): a click score is trusted at 50% after 20 impressions, 83% after 100.
Engineers’ commentUIS confidence =
total_impressions/ (20 +total_impressions).goodness_impThe same rater-score prediction, built from Glue’s satisfaction signals on SERP blocks.
Engineers’ commentThis is an estimated IS score based on user interaction signals from Glue Goodness.
goodness_imp_confidenceTrust in that Glue estimate, with the same 20-impression formula.
Engineers’ commentThis is goodness imp confidence, defined by UIS confidence =
total_impressions/ (20 +total_impressions).parsing_scoreA 0-to-1 score for how well the wording fits known query patterns.
Engineers’ commentUIS Query-Pattern Score. A number between 0 and 1, based on pattern or ngrams score.
refres_question_navfracThe question’s entities in the top 10, each result weighted by how navigational it is.
Engineers’ commentSum of (naviness * refres score of all question entities) over top 10 results.
refres_question_rankThe question’s entities in the top 10, each result weighted 1/rank: position 1 counts ten times more than position 10.
Engineers’ commentSum of (1/
web_rank* refres score of all question entities) over top 10 results.refres_answer_navfracThe best answer entity in the top 10, weighted by how navigational each result is.
Engineers’ commentSum of (naviness * max answer entity refres score) over top 10 results.
refres_answer_rankThe best answer entity in the top 10, each result weighted 1/rank.
Engineers’ commentSum of (1/
web_rank* max answer entity refres score) over top 10 results.refres_rankThe stronger support, question or answer; the cross-language score replaces it when higher.
Engineers’ commentMax(
refres_question_rank,refres_answer_rank|0). If Max(clir_question_rank,clir_answer_rank|0) exists and is greater, it will be used.refres_navfracThe same maximum, with the navigational weighting.
Engineers’ commentMax(
refres_question_navfrac,refres_answer_navfrac|0). If Max(clir_question_navfrac,clir_answer_navfrac|0) exists and is greater, it will be used.hist_refres_rankThe previous refres_rank, kept in Feature History: the top 10 of earlier searches weighs in before the new results arrive.
Engineers’ commentHistorical
refres_rankvalue.pq2t_vs_assistant_ibst_cosineThe query against the Assistant’s intent-type terms, for Assistant traffic.
Engineers’ commentCosine similarity between predicted query-to-term model and assistant intent-type-based salient terms. This is intended to be only used for ACE ranking and only populated for assistant traffic.
pq2t_vs_qibst_cosineCosine between the terms predicted for the query and the interpretation’s own terms.
Engineers’ commentCosine similarity between predicted query-to-term model and interpretation-based salient terms.
qibst_result_similarityCosine between the results’ terms and the interpretation’s terms, averaged with rank weights.
Engineers’ commentRank-weighted average cosine similarity between web and interpretation-based salient terms.
qibst_scoreA blend of the two: what the query says and what the results say.
Engineers’ commentA blend of
pq2t_vs_qibst_cosineandqibst_result_similarity.qibst_confidenceTrust in QIBST, from the volume of data behind it.
Engineers’ commentQIBST confidence score, estimated based on signal volume.
pq2t_vs_ibst_cosineThe query against the intent type’s terms: the fallback when the interpretation’s own terms are missing.
Engineers’ commentCosine similarity between predicted query-to-term model and intent-type-based salient terms. This is intended to be used as a backoff to
pq2t_vs_qibst_cosineif it is missing.ibst_z_scoreThe intent type’s terms against the query’s, normalised as a z-score.
Engineers’ commentCosine similarity between salient terms for the intent and those for the query, normalized by Z-score.
ibst_confidenceTrust in IBST, from the volume of data behind it.
Engineers’ commentIBST confidence score, estimated based on signal volume.
sbst_scoreThe interpretation against the terms of the session: earlier searches change how a query is read.
Engineers’ commentCosine similarity between query-based interpretation salient terms and session-based salient terms.
orbit_salient_terms_score_no_constraintsOrbit’s intent model against the results’ salient terms, weighted by rank.
Engineers’ commentRank-weighted average of dot product of Orbit intent salient terms model and web result salient terms.
orbit_score_v2The same comparison with Orbit’s V2 models and features of the results.
Engineers’ commentRank-weighted average of dot product of Orbit V2 intent model and features from web results.
orbit_results_trustedAre these results reliable enough to suppress an interpretation? Document trust, language and term vectors decide.
Engineers’ commentOrbit produces this feature to decide whether a result set can be "trusted" for making a suppression decision. This is a function of result document trust, language, presence of salient term vectors.
orbit_scoreThe best Orbit score across results in the same language and in other languages.
Engineers’ commentMaximum of
orbit_score_v0andclir_orbit_score_v0.hist_orbitThe previous Orbit score, kept in Feature History and usable before the results.
Engineers’ commentHistorical
orbit_salient_terms_score_no_constraintsvalue.log_dcrThe log of the direct click ratio: clicks measure how local a query is.
Engineers’ commentFloating value in interval (-Inf, 0], representing the logarithm of the direct click ratio (DCR), used to estimate likelihood of a query's local intent.
local_supports_multiple_geo_results1 when the local answer can list several places.
Engineers’ commentThis feature is equal to 1.0 if local supports multiple geo results. This is a query-level feature.
user_entity_distanceDistance in metres from the user to the nearest place in the interpretation.
Engineers’ commentMinimum distance (in metres) from user's current location to any entity with location signal in
intent_queryfuncall tree.cond_log_user_entity_distanceThe same distance, on a log scale.
Engineers’ commentConditional user-entity distance, in log scale.
logs_query_priorHow often local results are shown and clicked for the query’s n-grams.
Engineers’ commentQuery-level signal that indicates how often local results are shown and clicked for different query n-grams. This signal can be used to estimate the probability of local intent.
mean_stable_scoreAverage quality (stable score) of the top 3 local results shown.
Engineers’ commentAggregate quality of the answer entities in a Localweb interpretation. The stable score is a result-level signal that is populated for all Localweb entities. This signal is the average stable score of the top 3 local results shown.
clir_question_navfracCross-language topicality of the question’s entities, navigational weighting; a result needs 0.2 to count.
Engineers’ commentSum of (naviness * cross-language topicality of all question entities) over top 10 results. Note that a result must have at least 0.2 topicality for an entity for it to be included.
clir_question_rankCross-language topicality of the question’s entities, weighted 1/rank; 0.2 minimum per result.
Engineers’ commentSum of (1/
web_rank* cross-language topicality of all question entities) over top 10 results. Note that a result must have at least 0.2 topicality for an entity for it to be included.clir_answer_navfracCross-language topicality of the best answer entity, navigational weighting.
Engineers’ commentSum of (naviness * max answer entity cross-language topicality) over top 10 results. Note that a result must have at least 0.2 topicality for an entity for it to be included.
clir_answer_rankCross-language topicality of the best answer entity, weighted 1/rank.
Engineers’ commentSum of (1/
web_rank* max answer entity cross-language topicality) over top 10 results. Note that a result must have at least 0.2 topicality for an entity for it to be included.clir_pianno_rankPIANNO confidence measured on results in other languages, weighted 1/rank.
Engineers’ commentSum of (1/
web_rank* cross-language Pianno confidence) over top 10 results.auis_scoreAssistant interactions aggregated over the full interpretation.
Engineers’ commentAssistant User Interaction Score which is aggregated using full query interpretation.
auis_confidenceQ / (k + Q): the more Assistant requests behind the score, the more it is trusted.
Engineers’ commentAUIS confidence is used to measure the trust of
auis_score. Specifically, the more Assistant requests we use to aggregateauis_score, the more confidence we have forauis_score. The definition isquery_count/ (k +query_count).intent_name_auis_scoreAssistant interactions aggregated by intent name.
Engineers’ commentAssistant User Interaction Score which is aggregated using intent name.
intent_name_auis_score_expThe same score, from an experiment.
Engineers’ commentAssistant User Interaction Score which is aggregated using intent name from exp laelaps.
intent_name_pauis_scoreThe Assistant score predicted by a model (pAUIS) instead of counted.
Engineers’ commentAssistant User Interaction Score which is using pAUIS model.
one_box_interpretation1 when the interpretation leads to a OneBox.
Engineers’ commentThe feature is equal to 1.0 if an interpretation is a OneBoxFuncall. Otherwise, this feature will not be set.
high_confidence_bitThe structured query itself marks the interpretation as high confidence.
Engineers’ commentA score that indicates whether or not an interpretation is of high confidence based on high confidence signal from intent query.
kp_intent_priorThe average IS (Information Satisfaction) rating raters give knowledge panel interpretations (ShowEntity).
Engineers’ commentThe average IS rating for SingleEntity interpretation's intent function, i.e. the average IS rating for ShowEntity.
p13n_iq_p_good_scoreA personalised prediction that this interpretation will satisfy this user.
Engineers’ commentPIPS predicted IQpGood Score for Knowledge Interpretations.
web_answers_entity_alignment1 when the interpretation shares an entity with the top web answer passage (featured snippet).
Engineers’ commentWhether an query interpretation has entity alignment with WebAnswers, i.e. whether there is an overlapped entity between the top main WebAnswers passage and the query interpretation.
intent_query_nameThe intent’s name, for example ShowEntity, the knowledge panel intent.
Engineers’ commentQuery intent names, for example ShowEntity, used to distinguish between KP and PP query interpretation. Currently, the value populated in this field is only the top level intent for a recursive query intent. We may consider using concatenated function names in the future.
is_implicit_intentAn implicit intent (ShowEntity, ShowXConcept): the query names something without asking anything about it.
Engineers’ commentWhether the Query Intent is implicit. The intent names include {ShowEntity, ShowXConcept}.
noclir_pianno_rankHow strongly the top 10 support the structured interpretation, weighted 1/rank.
Engineers’ commentSum of (1/
web_rank* Pianno confidence) over top 10 results.pianno_rankThe better of the two PIANNO scores, same language or other languages.
Engineers’ commentMax(
clir_pianno_rank,noclir_pianno_rank).lp_lscorer_average_unbiased_pattern_scoreAverage score of the known n-gram patterns found in the query.
Engineers’ commentKnown ngram pattern score / known ngram pattern count.
lp_lscorer_known_ngram_fractionShare of the query’s n-gram patterns that are known.
Engineers’ commentKnown ngram pattern count / all ngram pattern count.
qref_confidence_scoreA harmonic mean of entity recognition confidences: one doubtful entity drags the whole interpretation down.
Engineers’ commentHarmonic mean of QRef's confidence in its annotations of all question entities. The confidence of a question entity is the minimum of its own confidence and all implied entities (if applicable). If no confidence scores are found in the intent query proto, this feature will take annotation confidences from the QueryJoin. If none is found, this feature is missing.
Comments quoted from the schemas as written; internal go/ links removed.
One search, several interpretations
An intent is what a person wants to achieve. An interpretation is one of the intents Google attributes to the query: how fast the animal runs or how fast the car goes. KScorer decides which interpretation comes first.
An example to follow the reasoning
The big cat’s speed
Animal · running · movement
A possible interpretationA car’s performance
Brand · model · engine
A possible interpretationThe word “speed” fits both interpretations. The top ten results, the entities recognised and earlier clicks will decide.
To compare these interpretations, Google writes them all in the same form: an intent and its slots to fill. “Who painted Guernica?” becomes a request about the creator of a work, with Guernica in the work’s slot. KScorer then scores each of these structured interpretations.
In the schemas
intent_query_namethe intent’s name: ShowEntity for a knowledge panelis_implicit_intentimplicit intent: the query only names something (ShowEntity, ShowXConcept)one_box_interpretation1 when the interpretation leads to a OneBoxhigh_confidence_bitthe interpretation is flagged as high confidence
KScorer’s job in three steps
Measure the estimated relevance of each interpretation using the available evidence.
The best-supported interpretation comes first; a dominant one gets rank 0.
Interpretations without enough support are dropped before the answer is chosen.
The intents Google can recognise are built by its teams
KScorer chooses between interpretations, and each one matches an intent Google already knows. Google keeps an internal catalog of intents for this, the Meaning Schemas, separate from schema.org markup. Each intent describes a type of need, such as finding the actors in a film, and has slots to fill, here the film. Google teams build this catalog by hand, in an internal tool called Intent Lab.
Take these three queries: “the actors in The Dark Knight”, “who stars in The Dark Knight?” and “The Dark Knight cast”. The words change, the request stays the same: the list of the film’s actors. Google files all three under a single intent, CastOfVisualNarrativeWork, “the cast of a visual narrative work”. This intent has one slot to fill: the work. Here, the slot takes the film The Dark Knight, as the Knowledge Graph knows it. For “Inception cast”, the same intent applies, with Inception in the slot: one intent covers every film and every way of asking for its cast.
Three queries
“the actors in The Dark Knight”
“who stars in The Dark Knight?”
“The Dark Knight cast”
One intent
CastOfVisualNarrativeWorkThe cast of a work
“Work” slot: The Dark Knight
Adding an intent to the catalog takes four steps, from defining the need to testing.
- 01 / DEFINE
Define the need
The team first checks what the catalog already covers, then sets what each slot accepts. The “work” slot takes films and series, entity families that the Knowledge Graph groups into collections. That choice draws the intent’s boundary.
- 02 / ILLUSTRATE
Provide examples
The team gathers queries that express the intent and others that look like it without expressing it. In each one, it marks the words that fill the slot: “The Dark Knight” in “The Dark Knight cast”. Two internal tools, Expand and Query Examples, help find other phrasings and do this marking.
- 03 / PREPARE
Produce KScorer’s signals
JobRunner, the tool that runs the automated jobs, turns these examples into training data and computes what KScorer needs to score the intent, including its salient terms: the words expected on a page about the subject. Existing signals can also be reused.
- 04 / EVALUATE
Test before launch
The team checks that the target queries are understood and that the answer shown meets the need. It compares the new version with the old one, measures server load, then has the change reviewed before release.
An intent needs at least 100 examples to be scored
The examples also train two salient-term models. Orbit is the Google system that sorts pages by intent: what they let you do (book, sign in) or their format (tutorial, product review, forum); its model learns the words typical of each intent. KBST is the second one: BST stands for based salient terms, and the K most likely stands for Knowledge, as in KScorer. Several of KScorer’s 56 signals depend on these models, which are rebuilt whenever an intent’s phrasings change.
Below 100 training examples per intent, generating KScorer’s signals fails, and the intent cannot move on to testing.
Three tests before launch
QU_EvalQU for Query Understanding. The test checks that Google produces the right interpretations and ranks them in the right order.
KE_EvalKE for Knowledge Engine, the Google systems that run the Knowledge Graph and its answers. The test checks that the answer shown meets the need.
KE_SxSSxS for side by side. The old and new versions are shown next to each other, to see what concretely changes in the results.
When the final answer is not ready, the team tests with a mock OneBox. The OneBox is the answer box Google shows above the results; the mock version shows which queries the new intent would trigger on, and keeps it from triggering too often once live.
Intents fit together like building blocks. For “Nolan films released after 2010”, Google starts from Christopher Nolan’s filmography, then adds a date constraint (DateRestrict) and a sort order (Sort); a third operator, Intersect, crosses two lists. One intent can also fill another’s slot. With a few blocks, Google understands a compound request without needing an intent for every combination.
In the schemas, a recognised intent is written as a function call, FunctionCall: the function’s name is the intent’s, its Argument entries are the filled slots, and a slot can hold another call (ArgumentValue). KScorer scores and ranks these filled calls.
In the schemas
FunctionCall.namethe intent’s name, for example CastOfVisualNarrativeWorkFunctionCall.keythe intent’s identifier in the catalog, a MID like those of Knowledge Graph entitiesFunctionCall.unexplained_tokensquery words that no slot accounts forFunctionCall.ignored_tokensrecognised context words that leave the intent unchangedFunctionCall.sensitivitya sensitive intent, flagged by the intent catalog (Intent Catalog)Argument.keythe slot: the film in “The Dark Knight cast”ArgumentValue.funcalla slot filled by another intent: this is the nestingArgumentValue.locationa slot that takes a place; others take a person, an amount, a date or a numberArgumentValue.shopping_producta slot that takes a product, for shopping intents
The top 10 pages tell Google which intent to choose
To understand a query, Google also looks at the pages it ranks at the top. If the pages that rank for “jaguar speed” mostly talk about big cats, the animal interpretation gains points: each page counts as a vote for the intent it talks about, and Google’s engineers call this step voting. The higher a page ranks, the more its vote counts.
The entities in the question and the answer
An entity is a specific thing the Knowledge Graph knows: a work, a person, a brand, a place. The REFRES signal family (most likely WebRef results; WebRef is the Google system that spots entities in a page) checks whether the top ten results talk about the interpretation’s entities. It gives them two roles. In “who painted Guernica?”, Guernica is the question entity, what the query is about; Pablo Picasso is the answer entity, what the searcher wants.
Each result weighs according to its rank. A variant weights it by its naviness instead: the likelihood that this result is the site the searcher wants to reach, such as a brand’s official site.
The weight of position
r is the result’s position, from 1 to 10; Er measures how much this result covers the question’s entities.
Signal name: refres_question_rank. Its variant weighted by naviness is called refres_question_navfrac.
With equal support, a result in first position contributes ten times as much to this sum as a result in tenth position.
How rank changes this calculation
Coefficient applied to each result’s support: 1 ÷ rank
Each vote keeps its source: the web, news, other languages, the person
A result’s support is recorded with its rank, its strength from 0 to 1 and its origin. Google’s main results are only one of seven sources.
MAIN_GSRthe main web resultsNEWSnews resultsLIVE_RESULTlive resultsCLIR_GSRresults in other languagesPERSONALIZED_RESULTpersonalised resultsCONTEXTBOOST_RESULTresults boosted by the contextSUPPORT_FROM_KSCORERsupport passed on by KScorer itself
ResultSupport.rankthe result’s rank, counted from 0ResultSupport.scorestrength of the support, from 0 to 1ResultSupport.estimated_navinesshow navigational the result isResultSupportProvenance.entity_group_typewhich entity is supported: QUESTION, ANSWER or INTENT
Orbit judges whether the top 10 pages are reliable enough to drop an interpretation
Orbit is the system that sorts every page into intent categories: what it lets you do (book, sign in), its format (tutorial, product review, forum) or its publisher (press, government, merchant). KScorer compares the salient terms the Orbit model links to the intent with those of the top 10 pages. Orbit also judges whether these pages are reliable enough to drop an interpretation, based on trust in the documents, their language and whether their salient terms are available.
A stronger basis for deciding which interpretations to retain or discard.
In the schemas
orbit_score_v2Orbit V2 intent model against the results, rank-weightedorbit_scorethe best Orbit score, same language or other languagesorbit_results_trustedcan these results justify suppressing an interpretation?hist_orbitthe previous Orbit score, usable before the results
Clicks predict what quality raters would say
Google has its results rated by quality raters, human evaluators who follow its guidelines. Their satisfaction score is called IS, for Information Satisfaction. Raters only see a small share of queries; for all the others, KScorer predicts their score from clicks. That is the job of the UIS family, for User Interaction Signals: its main signal, user_interaction_score, is “an estimated IS score using user interaction data”. goodness_imp makes the same estimate from Glue, the memory of interactions with every block of the SERP.
In the US antitrust trial, the plaintiffs describe this score in their findings (Plaintiffs’ Proposed Findings of Fact, ¶ 559-560): each query evaluated by a rater gets an IS score, and Google rolls these scores up into a score out of 100. Google measures it with IS4@5: the fourth version of the score, computed on the top five results. Google trains NavBoost, RankBrain and DeepRank on these scores (testimony of expert Douglas Oard).
In the schemas
user_interaction_scoreclicks turned into a predicted IS scoregoodness_impGlue’s satisfaction signals turned into a predicted IS scoreuser_interaction_relevance_scorerelevance estimated from interaction logsuser_interaction_confidenceN / (20 + N): trust grows with impressionsuser_interaction_significant_backoffscore borrowed from similar queriesparsing_scorehow well the wording fits known patterns, 0 to 1
More observations, greater confidence
A click score built on five impressions is not worth one built on five hundred. KScorer weighs it with a confidence factor that grows with the number of impressions.
N is the number of impressions behind the score.
The factor is called user_interaction_confidence. The relevance score drawn from interactions is user_interaction_relevance_score.
The first impressions count most: 20 impressions give 0.5, 100 give 0.833, then the curve flattens towards 1.
Confidence rises, then levels off
Calculated confidence factor50%
On a rare query, the click score is borrowed from similar queries, and user_interaction_significant_backoff flags it: long-tail queries inherit the click behaviour of their more frequent neighbours.
Salient terms test the query, the interpretation and the pages against each other
For each subject, Google computes salient terms: the words expected to appear prominently on a relevant page. For the big cat, “predator”, “rainforest”, “prey”; for the car, “engine”, “horsepower”, “0 to 60”. KScorer compares these word lists across the query, the interpretation and the top 10 pages: the interpretation whose terms show up in the results gains points.
The terms the model predicts for the query.
Its own salient terms.
The salient terms of the top 10 pages.
The QIBST score (query interpretation-based salient terms, the salient terms specific to one interpretation) makes two comparisons: the query against the interpretation, then the top 10 pages against that same interpretation, with more weight on the first ones.
qibst_score combines pq2t_vs_qibst_cosine, for the query–interpretation relationship, and qibst_result_similarity, for the results–interpretation relationship. In these names, pq2t means the terms the model predicts for the query (predicted query-to-terms).
Do two term lists stress the same words?
Each list of salient terms becomes a vector, with one weight per word. Cosine similarity measures whether two vectors point the same way, that is, whether two lists give weight to the same words.
Two lists that stress the same words score close to 1; two lists with no word in common score 0.
Three term lists, from intent type to session
Intent-based salient terms: the terms of the intent type, shared for instance by every cast request.
Query interpretation-based salient terms: the terms of this interpretation, with its subject and entities, such as the cast of The Dark Knight.
Session-based salient terms: the terms of the earlier searches in the session, which change how the next one is read.
When the interpretation’s own terms are missing, the intent type’s terms take over (pq2t_vs_ibst_cosine).
In the schemas
ibst_z_scoreintent-type terms against the query, as a z-scoresbst_scorethe interpretation against the session’s termsqibst_confidencetrust in QIBST, from the volume of datapq2t_vs_assistant_ibst_cosinethe same comparison for Assistant traffic
Recognising the right entities across languages
Understanding “jaguar” means linking the word to the right Knowledge Graph entity: the big cat or the car brand. A more complex request names several, and each one has to be recognised correctly. KScorer measures how sure each recognition is, then checks that the results talk about these entities, in other languages too.
QRef: the least certain entity sets the confidence of the whole interpretation
QRef (Q for query, mirroring WebRef) is the system that spots entities in the query and links them to the Knowledge Graph, each with a confidence from 0 to 1. The qref_confidence_score signal combines these confidences with a harmonic mean, which the lowest value drags down.
n is the number of entities; ci is the confidence associated with each one.
Two entities, one recognised at 0.9, the other at 0.2: a simple average would give 0.55, the harmonic mean falls to 0.33. A single doubtful entity weakens the whole interpretation.
In the schemas
QRefAnnotation.freebase_midthe MID of the recognised entity, its Knowledge Graph identifierQRefAnnotation.confidence_scoreconfidence in the recognition, from 0 to 1QRefAnnotation.annotated_spanthe words of the query that name itQRefAnnotation.resolution_scorean implicit mention, such as “my hotel”QRefAnnotation.merged_implied_entityentities implied by the mention, merged in
CLIR: pages in other languages vote too
CLIR, for Cross-Language Information Retrieval, means searching across languages: Google also queries the web in languages other than the query’s. For a subject covered mostly in English, English pages thus back the interpretation of a French query. A result counts when the entity weighs enough in its subject, with a topicality of at least 0.2.
clir_question_rank then sums the support of the top ten results, weighted 1/rank, as REFRES does in the query’s language.
Pianno: each page keeps its confidence for an interpretation
Pianno is a Google Search component that annotates pages at indexing time (ANNO most likely for annotation). For each page, it keeps a confidence: how strongly the page supports a query interpretation. KScorer adds up these confidences over the top 10, each divided by the rank. The pianno_rank signal takes the maximum of its standard and multilingual variants.
In the schemas
clir_question_rankquestion entities in other languages, weighted 1/rankclir_answer_rankthe best answer entity in other languagesclir_pianno_rankPianno confidence on results in other languagespianno_rankthe better Pianno score, any language
Context guides the answer
For “pharmacy open”, proximity becomes useful evidence. For a request to Assistant, the history of voice interactions or actions can help clarify the intent. KScorer has signals for both.
Six signals for assessing local intent
user_entity_distance measures the minimum distance, in metres, between the user and the located entities in the interpretation. A variant transforms this distance onto a logarithmic scale.
logs_query_prior counts how often local results are shown and clicked for groups of words. log_dcr, the logarithm of the direct click ratio (DCR), reads the same intent from clicks.
mean_stable_score averages the stable score, the quality score of each local listing, over the top three local results. Another signal indicates whether multiple geographic results are supported.
Beyond location, p13n_iq_p_good_score predicts whether an interpretation will satisfy this particular user: p13n is short for personalization (a p, thirteen letters, an n), and p_good is most likely the probability that the interpretation is good.
The case of Google Assistant
Interactions suited to action requests
The AUIS family, for Assistant User Interaction Score, does for Google Assistant what UIS does for Search: it scores an interpretation from how Assistant users react. It aggregates these reactions for a complete interpretation, with auis_score, or for an intent type, with intent_name_auis_score. In intent_name_pauis_score, the p marks a prediction: the score comes from a model instead of being counted.
The auis_confidence confidence factor follows the same volume-based logic as UIS.
In the schemas
auis_scoreAssistant interactions on the full interpretationintent_name_auis_scorethe same, aggregated by intent nameintent_name_pauis_scorethe same, predicted by a modelauis_confidenceQ / (k + Q): trust grows with Assistant requests
Q is the number of Assistant queries considered. k sets how fast confidence rises.
KScorer votes twice: before and after the results
Some evidence exists as soon as the query is typed, the rest arrives with the results. KScorer therefore scores interpretations in two rounds, which Google calls votes: a first one on the query alone, then a second one once the documents are in.
PRE_RESULTS_VOTING11 signals vote on the query alonePACKER_VOTING41 signals vote while the SERP is assembledCARD_UTILITY6 signals decide whether an answer card, such as a knowledge panel, is worth showingDERIVED_INTENT_RANKINGa fourth use: ranking the derived intents, added alongside the query’s interpretation
Before the results
A first vote on the query alone
Query patterns, recognised entities, salient terms and historical signals provide an initial set of evidence. This first vote is PRE_RESULTS_VOTING.
With the results
Compare interpretations with the web
The pages found then vote: through their entities (REFRES), through pages in other languages (CLIR), through their intent categories (Orbit) and through their salient terms (QIBST). This second vote, PACKER_VOTING, takes place in the packer, the module that assembles the SERP.
Feature History keeps signal values from one search to the next. With hist_orbit and hist_refres_rank, the pages that backed an interpretation in earlier searches already vote in the first round, before the new results are known.
The two votes produce pre_results_ranking_score and post_results_ranking_score. Twiddlers, functions that rework a score after the fact, then adjust them: pre_twiddled_ranking_score and twiddled_ranking_score hold the score before and after adjustment.
CARD_UTILITY, judges whether a card is useful: cards are the answer blocks Google shows on top of the blue links, such as a knowledge panel or a direct answer. Its six signals say a lot about what Google trusts:user_interaction_score, the flag that marks a score borrowed from similar queries, goodness_imp and its confidence.kp_intent_prior, the average IS rating of knowledge panel interpretations.qref_confidence_score.From score to answer: rank 0 goes to the dominant interpretation
Three values leave KScorer with each interpretation: a score, a rank and a dominance flag.
The KScore
The interpretation’s score after twiddlers. Systems that choose between several answers or actions read it.
k_scoreThe KScorer rank
A position in the ordering of interpretations. When a dominant interpretation exists, it receives rank 0. The others are ordered from 1 to N.
kscorer_rankDominance
The interpretation that clearly wins. It takes rank 0 and opens the way to a faster SERP.
dominantThese values travel beyond Search. In Google Assistant, the Prefulfillment Ranker, which ranks interpretations before the Assistant runs an action or gives an answer, reads them. UnderstandingSignals passes it the KScorer rank along with the 56 signals, filed under the name of the Query Understanding Service, the service that analyses the query (qus_features).
In the schemas
predicted_intent_confidencethe confidence predicted for the intentis_intent_from_orbitthe intent was proposed by Orbithas_answer_groupthe interpretation comes with a Search answeris_qus_dominantdominant according to query understandingsearch_response_is_low_utilitythe Search response is judged of low utilityqus_phasethe understanding phase: rewrite, probe query, image…
has_answer, suppressed and top_ranked_non_suppressed track the presence of an answer, the suppression of an interpretation and the first position among retained interpretations. A dominant interpretation gets the first block of the SERP sent early
First Result Out sends the first block of the page before the rest is ready. KScorer holds the key: early sending is refused when
- the top interpretation (QI, for query interpretation) is not dominant
INELIGIBLE_TOP_ QI_ NOT_ DOMINANT - a competing interpretation dominates
INELIGIBLE_COMPETING_ QI_ IS_ DOMINANT - the top interpretation has no click data (UIS)
INELIGIBLE_TOP_ QI_ NO_ UIS - its KScorer rank is above a threshold
INELIGIBLE_KSCORER_ RANK_ ABOVE_ THRESHOLD - its KScorer score and a second score, HTF, are below their threshold
INELIGIBLE_HTF_ AND_ KSCORER_ SCORE_ BELOW_ THRESHOLD
To rank, a page must first match the intent Google picks
Google settles a query’s intent before it ranks pages. A page written for the losing interpretation competes on a SERP built for another intent.
Check which intent the SERP has chosen
The top 10 pages weigh on the intent chosen, all the more as they rank high, and Feature History keeps that verdict for the next searches. On an ambiguous query, today’s SERP shows the winning interpretation. A page built for the other one targets a query that names it, such as “Jaguar F-Type top speed”.
Make every entity unambiguous
KScorer combines entity confidences with a harmonic mean: one doubtful entity drags the whole interpretation down. Name brands, models and places the way the Knowledge Graph knows them, and tie them to their entity with structured data (
sameAs).Satisfied clicks keep an answer card on the SERP
Four of the six signals that decide whether an answer card shows come from clicks, two of them as a predicted rater score. For a brand, the knowledge panel lasts as long as searchers find what they came for: accurate facts on the site, in the Business Profile and in Wikidata.
For local queries, the top three listings set the bar
mean_stable_scoreaverages the quality of the top three local results, and distance is measured in metres. The local interpretation wins when nearby listings are good: a complete Business Profile, accurate hours and reviews feed that quality.
Behind a SERP that appears in a fraction of a second, Google has already compared several interpretations of the query, and the clicks of earlier searchers have weighed in.
A closer look at the mechanisms
The schemas, field by field
15 schemas and 287 fields from KScorer, to browse here or download as a single file.
File paths9 files · 15 types · 287 fields · 19 values
assistant/grounding/ranker/proto/prefulfillment_signals.proto1 type · 92 fields
Themes: Selection in Assistant
assistant/orchestration/understanding/platform/proto/understanding_signals.proto1 type · 10 fields
Themes: Selection in Assistant
knowledge/answers/config/meaning_schema_slot_key.proto1 type · 2 fields
Themes: Intents and entities
knowledge/answers/intent_catalog/meaning_schema_key.proto1 type · 2 fields
Themes: Intents and entities
knowledge/answers/intent_query/intent_query.proto3 types · 59 fields
knowledge/answers/interpretation_ranking/proto/kscorer_feature.proto4 types · 66 fields · 19 values
Themes: KScorer signals
- The 56 signals
knowledge.answers.interpretation_ranking.InputFeature - A signal’s details
knowledge.answers.interpretation_ranking.KScorerFeatureDetails - The signal families
knowledge.answers.interpretation_ranking.KScorerFeatureDetails.Family - The stages that use signals
knowledge.answers.interpretation_ranking.KScorerFeatureDetails.UsedBy
logs/proto/knowledge/interpretation/intent_query.proto1 type · 8 fields
Themes: Evidence from results
nlp/semantic_parsing/proto/qref_annotation.proto1 type · 37 fields
Themes: Intents and entities