com.ibm.icu.text
Class Normalizer2

java.lang.Object
  extended by com.ibm.icu.text.Normalizer2
Direct Known Subclasses:
FilteredNormalizer2

public abstract class Normalizer2
extends java.lang.Object

Unicode normalization functionality for standard Unicode normalization or for using custom mapping tables. All instances of this class are unmodifiable/immutable.

The primary functions are to produce a normalized string and to detect whether a string is already normalized. The most commonly used normalization forms are those defined in http://www.unicode.org/unicode/reports/tr15/ However, this API supports additional normalization forms for specialized purposes. For example, NFKC_Casefold is provided via getInstance("nfkc_cf", COMPOSE) and can be used in implementations of UTS #46.

Not only are the standard compose and decompose modes supplied, but additional modes are provided as documented in the Mode enum.

Some of the functions in this class identify normalization boundaries. At a normalization boundary, the portions of the string before it and starting from it do not interact and can be handled independently.

The spanQuickCheckYes() stops at a normalization boundary. When the goal is a normalized string, then the text before the boundary can be copied, and the remainder can be processed with normalizeSecondAndAppend().

The hasBoundaryBefore(), hasBoundaryAfter() and isInert() functions test whether a character is guaranteed to be at a normalization boundary, regardless of context. This is used for moving from one normalization boundary to the next or preceding boundary, and for performing iterative normalization.

Iterative normalization is useful when only a small portion of a longer string needs to be processed. For example, in ICU, iterative normalization is used by the NormalizationTransliterator (to avoid replacing already-normalized text) and ucol_nextSortKeyPart() (to process only the substring for which sort key bytes are computed).

The set of normalization boundaries returned by these functions may not be complete: There may be more boundaries that could be returned. Different functions may return different boundaries.

Author:
Markus W. Scherer
Status:
Draft ICU 4.4.

Nested Class Summary
static class Normalizer2.Mode
          Constants for normalization modes.
 
Constructor Summary
protected Normalizer2()
          Deprecated. This API is ICU internal only.
 
Method Summary
abstract  java.lang.StringBuilder append(java.lang.StringBuilder first, java.lang.CharSequence second)
          Appends the second string to the first string (merging them at the boundary) and returns the first string.
static Normalizer2 getInstance(java.io.InputStream data, java.lang.String name, Normalizer2.Mode mode)
          Returns a Normalizer2 instance which uses the specified data file (an ICU data file if data=null, or else custom binary data) and which composes or decomposes text according to the specified mode.
abstract  boolean hasBoundaryAfter(int c)
          Tests if the character always has a normalization boundary after it, regardless of context.
abstract  boolean hasBoundaryBefore(int c)
          Tests if the character always has a normalization boundary before it, regardless of context.
abstract  boolean isInert(int c)
          Tests if the character is normalization-inert.
abstract  boolean isNormalized(java.lang.CharSequence s)
          Tests if the string is normalized.
 java.lang.String normalize(java.lang.CharSequence src)
          Returns the normalized form of the source string.
abstract  java.lang.Appendable normalize(java.lang.CharSequence src, java.lang.Appendable dest)
          Writes the normalized form of the source string to the destination Appendable and returns the destination Appendable.
abstract  java.lang.StringBuilder normalize(java.lang.CharSequence src, java.lang.StringBuilder dest)
          Writes the normalized form of the source string to the destination string (replacing its contents) and returns the destination string.
abstract  java.lang.StringBuilder normalizeSecondAndAppend(java.lang.StringBuilder first, java.lang.CharSequence second)
          Appends the normalized form of the second string to the first string (merging them at the boundary) and returns the first string.
abstract  Normalizer.QuickCheckResult quickCheck(java.lang.CharSequence s)
          Tests if the string is normalized.
abstract  int spanQuickCheckYes(java.lang.CharSequence s)
          Returns the end of the normalized substring of the input string.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

Normalizer2

protected Normalizer2()
Deprecated. This API is ICU internal only.

Sole constructor. (For invocation by subclass constructors, typically implicit.)

Status:
Internal. This API is ICU internal only.
Method Detail

getInstance

public static Normalizer2 getInstance(java.io.InputStream data,
                                      java.lang.String name,
                                      Normalizer2.Mode mode)
Returns a Normalizer2 instance which uses the specified data file (an ICU data file if data=null, or else custom binary data) and which composes or decomposes text according to the specified mode. Returns an unmodifiable singleton instance. If data!=null, then the binary data is read once and cached using the provided name as the key. If you know or expect the data to be cached already, you can use data!=null for non-ICU data as well.

Parameters:
data - the binary, big-endian normalization (.nrm file) data, or null for ICU data
name - "nfc" or "nfkc" or "nfkc_cf" or name of custom data file
mode - normalization mode (compose or decompose etc.)
Returns:
the requested Normalizer2, if successful
Status:
Draft ICU 4.4.

normalize

public java.lang.String normalize(java.lang.CharSequence src)
Returns the normalized form of the source string.

Parameters:
src - source string
Returns:
normalized src
Status:
Draft ICU 4.4.

normalize

public abstract java.lang.StringBuilder normalize(java.lang.CharSequence src,
                                                  java.lang.StringBuilder dest)
Writes the normalized form of the source string to the destination string (replacing its contents) and returns the destination string. The source and destination strings must be different objects.

Parameters:
src - source string
dest - destination string; its contents is replaced with normalized src
Returns:
dest
Status:
Draft ICU 4.4.

normalize

public abstract java.lang.Appendable normalize(java.lang.CharSequence src,
                                               java.lang.Appendable dest)
Writes the normalized form of the source string to the destination Appendable and returns the destination Appendable. The source and destination strings must be different objects.

Parameters:
src - source string
dest - destination Appendable; gets normalized src appended
Returns:
dest
Status:
Internal. This API is ICU internal only.

normalizeSecondAndAppend

public abstract java.lang.StringBuilder normalizeSecondAndAppend(java.lang.StringBuilder first,
                                                                 java.lang.CharSequence second)
Appends the normalized form of the second string to the first string (merging them at the boundary) and returns the first string. The result is normalized if the first string was normalized. The first and second strings must be different objects.

Parameters:
first - string, should be normalized
second - string, will be normalized
Returns:
first
Status:
Draft ICU 4.4.

append

public abstract java.lang.StringBuilder append(java.lang.StringBuilder first,
                                               java.lang.CharSequence second)
Appends the second string to the first string (merging them at the boundary) and returns the first string. The result is normalized if both the strings were normalized. The first and second strings must be different objects.

Parameters:
first - string, should be normalized
second - string, should be normalized
Returns:
first
Status:
Draft ICU 4.4.

isNormalized

public abstract boolean isNormalized(java.lang.CharSequence s)
Tests if the string is normalized. Internally, in cases where the quickCheck() method would return "maybe" (which is only possible for the two COMPOSE modes) this method resolves to "yes" or "no" to provide a definitive result, at the cost of doing more work in those cases.

Parameters:
s - input string
Returns:
true if s is normalized
Status:
Draft ICU 4.4.

quickCheck

public abstract Normalizer.QuickCheckResult quickCheck(java.lang.CharSequence s)
Tests if the string is normalized. For the two COMPOSE modes, the result could be "maybe" in cases that would take a little more work to resolve definitively. Use spanQuickCheckYes() and normalizeSecondAndAppend() for a faster combination of quick check + normalization, to avoid re-checking the "yes" prefix.

Parameters:
s - input string
Returns:
the quick check result
Status:
Draft ICU 4.4.

spanQuickCheckYes

public abstract int spanQuickCheckYes(java.lang.CharSequence s)
Returns the end of the normalized substring of the input string. In other words, with end=spanQuickCheckYes(s); the substring s.subSequence(0, end) will pass the quick check with a "yes" result.

The returned end index is usually one or more characters before the "no" or "maybe" character: The end index is at a normalization boundary. (See the class documentation for more about normalization boundaries.)

When the goal is a normalized string and most input strings are expected to be normalized already, then call this method, and if it returns a prefix shorter than the input string, copy that prefix and use normalizeSecondAndAppend() for the remainder.

Parameters:
s - input string
Returns:
"yes" span end index
Status:
Draft ICU 4.4.

hasBoundaryBefore

public abstract boolean hasBoundaryBefore(int c)
Tests if the character always has a normalization boundary before it, regardless of context. If true, then the character does not normalization-interact with preceding characters. In other words, a string containing this character can be normalized by processing portions before this character and starting from this character independently. This is used for iterative normalization. See the class documentation for details.

Parameters:
c - character to test
Returns:
true if c has a normalization boundary before it
Status:
Draft ICU 4.4.

hasBoundaryAfter

public abstract boolean hasBoundaryAfter(int c)
Tests if the character always has a normalization boundary after it, regardless of context. If true, then the character does not normalization-interact with following characters. In other words, a string containing this character can be normalized by processing portions up to this character and after this character independently. This is used for iterative normalization. See the class documentation for details.

Note that this operation may be significantly slower than hasBoundaryBefore().

Parameters:
c - character to test
Returns:
true if c has a normalization boundary after it
Status:
Draft ICU 4.4.

isInert

public abstract boolean isInert(int c)
Tests if the character is normalization-inert. If true, then the character does not change, nor normalization-interact with preceding or following characters. In other words, a string containing this character can be normalized by processing portions before this character and after this character independently. This is used for iterative normalization. See the class documentation for details.

Note that this operation may be significantly slower than hasBoundaryBefore().

Parameters:
c - character to test
Returns:
true if c is normalization-inert
Status:
Draft ICU 4.4.


Copyright (c) 2011 IBM Corporation and others.