Package org.apache.sysds.runtime.transform.tokenize.builder

# Class TokenizerBuilderNgram

java.lang.Object
org.apache.sysds.runtime.transform.tokenize.builder.TokenizerBuilder
org.apache.sysds.runtime.transform.tokenize.builder.TokenizerBuilderWhitespaceSplit
org.apache.sysds.runtime.transform.tokenize.builder.TokenizerBuilderNgram

All Implemented Interfaces:
  `Serializable`

public class TokenizerBuilderNgram extends TokenizerBuilderWhitespaceSplit

See Also:
  - Serialized Form

## Field Summary

Fields
Modifier and Type
Field
Description
`int`
`maxGram`
`int`
`minGram`
`org.apache.sysds.runtime.transform.tokenize.builder.TokenizerBuilderNgram.NgramType`
`ngramType`

### Fields inherited from class org.apache.sysds.runtime.transform.tokenize.builder.TokenizerBuilderWhitespaceSplit

`regex`

## Constructor Summary

Constructors
Constructor
Description
`TokenizerBuilderNgram(int[] idCols, int tokenizeCol, org.apache.wink.json4j.JSONObject params)`

## Method Summary

Modifier and Type
Method
Description
`void`
`createInternalRepresentation(FrameBlock in, DocumentRepresentation[] internalRepresentation, int rowStart, int blk)`
`List<Token>`
`splitIntoNgrams(Token token, int minGram, int maxGram)`

### Methods inherited from class org.apache.sysds.runtime.transform.tokenize.builder.TokenizerBuilderWhitespaceSplit

`splitToTokens`

### Methods inherited from class org.apache.sysds.runtime.transform.tokenize.builder.TokenizerBuilder

`createInternalRepresentation, getTasks`

### Methods inherited from class java.lang.Object

`equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

## Field Details

### minGram

public int minGram

### maxGram

public int maxGram

### ngramType

public org.apache.sysds.runtime.transform.tokenize.builder.TokenizerBuilderNgram.NgramType ngramType

## Constructor Details

### TokenizerBuilderNgram

public TokenizerBuilderNgram(int[] idCols, int tokenizeCol, org.apache.wink.json4j.JSONObject params) throws org.apache.wink.json4j.JSONException

Throws:
  `org.apache.wink.json4j.JSONException`

## Method Details

### splitIntoNgrams

public List<Token> splitIntoNgrams(Token token, int minGram, int maxGram)

### createInternalRepresentation

public void createInternalRepresentation(FrameBlock in, DocumentRepresentation[] internalRepresentation, int rowStart, int blk)

Overrides:
  `createInternalRepresentation` in class `TokenizerBuilderWhitespaceSplit`