TokenizerType Enum
Definition
Important
Some information relates to prerelease product that may be substantially modified before it’s released. Microsoft makes no warranties, express or implied, with respect to the information provided here.
public enum TokenizerTypetype TokenizerType = - Inheritance
-
TokenizerType
Fields
| Name | Value | Description |
|---|---|---|
| None | 0 | This value indicates that no tokens should be extracted from this property. It is only valid for tokenizer_type to be 'NONE' if getIndexingType() is INDEXING_TYPE_NONE. Portions of this page are modifications based on work created and shared by the Android Open Source Project and used according to terms described in the Creative Commons 2.5 Attribution License. |
| Plain | 1 | Tokenization for plain text. This value indicates that tokens should be extracted from this property based on word breaks. Segments of whitespace and punctuation are not considered tokens. For example, a property with "foo bar. baz." will produce tokens for "foo", "bar" and "baz". The segments " " and "." will not be considered tokens. It is only valid for tokenizer_type to be 'PLAIN' if getIndexingType() is INDEXING_TYPE_EXACT_TERMS or INDEXING_TYPE_PREFIXES. Portions of this page are modifications based on work created and shared by the Android Open Source Project and used according to terms described in the Creative Commons 2.5 Attribution License. |
| Verbatim | 2 | This value indicates that no normalization or segmentation should be applied to string values that are tokenized using this type. Therefore, the output token is equivalent to the raw string value. For example, a property with "Hello, world!" will produce the token "Hello, world!", preserving punctuation and capitalization, and not creating separate tokens between the space. It is only valid for tokenizer_type to be 'VERBATIM' if getIndexingType() is INDEXING_TYPE_EXACT_TERMS or INDEXING_TYPE_PREFIXES. Portions of this page are modifications based on work created and shared by the Android Open Source Project and used according to terms described in the Creative Commons 2.5 Attribution License. |
| Rfc822 | 3 | Tokenization for emails. This value indicates that tokens should be extracted from this property based on email structure. For example, a property with "alex.sav@google.com" will produce tokens for "alex", "sav", "alex.sav", "google", "com", and "alexsav@google.com" It is only valid for tokenizer_type to be 'RFC822' if getIndexingType() is INDEXING_TYPE_EXACT_TERMS or INDEXING_TYPE_PREFIXES. Portions of this page are modifications based on work created and shared by the Android Open Source Project and used according to terms described in the Creative Commons 2.5 Attribution License. |