Package org.apache.sysds.runtime.transform.tokenize

# Class Tokenizer

java.lang.Object
org.apache.sysds.runtime.transform.tokenize.Tokenizer

All Implemented Interfaces:
  `Serializable`

public class Tokenizer extends Object implements Serializable

See Also:
  - Serialized Form

## Field Summary

Fields
Modifier and Type
Field
Description
`static final int`
`TOKENIZE_NUM_BLOCKS`

## Method Summary

Modifier and Type
Method
Description
`void`
`allocateInternalRepresentation(int numDocuments)`
`FrameBlock`
`apply(FrameBlock out, int k)`
`void`
`build(FrameBlock in, int k)`
`List<DependencyTask<?>>`
`getBuildTasks(FrameBlock in)`
`int`
`getMaxNumRows(int inRows)`
`long`
`getNumCols()`
`int`
`getNumRowsEstimate()`
`Types.ValueType[]`
`getSchema()`
`FrameBlock`
`tokenize(FrameBlock in)`
`FrameBlock`
`tokenize(FrameBlock in, int k)`

### Methods inherited from class java.lang.Object

`equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

## Field Details

### TOKENIZE_NUM_BLOCKS

public static final int TOKENIZE_NUM_BLOCKS

## Method Details

### getSchema

public Types.ValueType[] getSchema()

### getMaxNumRows

public int getMaxNumRows(int inRows)

### getNumRowsEstimate

public int getNumRowsEstimate()

### getNumCols

public long getNumCols()

### allocateInternalRepresentation

public void allocateInternalRepresentation(int numDocuments)

### tokenize

public FrameBlock tokenize(FrameBlock in)

### tokenize

public FrameBlock tokenize(FrameBlock in, int k)

### apply

public FrameBlock apply(FrameBlock out, int k)

### getBuildTasks

public List<DependencyTask<?>> getBuildTasks(FrameBlock in)

### build

public void build(FrameBlock in, int k)