Package org.apache.sysds.runtime.instructions.spark.data

# Class PartitionedBroadcast<T extends CacheBlock>

java.lang.Object
org.apache.sysds.runtime.instructions.spark.data.PartitionedBroadcast<T>

All Implemented Interfaces:
  `Serializable`

public class PartitionedBroadcast<T extends CacheBlock> extends Object implements Serializable
This class is a wrapper around an array of broadcasts of partitioned matrix/frame blocks, which is required due to 2GB limitations of Spark's broadcast handling. Without this partitioning of `Broadcast<PartitionedBlock>` into `Broadcast<PartitionedBlock>[]`, we got java.lang.IllegalArgumentException: Size exceeds Integer.MAX_VALUE issue. Despite various jiras, this issue still showed up in Spark 2.1.

See Also:
  - Serialized Form

## Constructor Summary

Constructors
Constructor
Description
`PartitionedBroadcast()`
`PartitionedBroadcast(org.apache.spark.broadcast.Broadcast<PartitionedBlock<T>>[] broadcasts, DataCharacteristics dc)`

## Method Summary

Modifier and Type
Method
Description
`static int`
`computeBlocksPerPartition(long[] dims, int blen)`
`static int`
`computeBlocksPerPartition(long rlen, long clen, long blen)`
`void`
`destroy()`
This method cleanups all underlying broadcasts of a partitioned broadcast, by forward the calls to SparkExecutionContext.cleanupBroadcastVariable.
`T`
`getBlock(int[] ix)`
`T`
`getBlock(int rowIndex, int colIndex)`
`org.apache.spark.broadcast.Broadcast<PartitionedBlock<T>>[]`
`getBroadcasts()`
`DataCharacteristics`
`getDataCharacteristics()`
`long`
`getNumCols()`
`int`
`getNumColumnBlocks()`
`int`
`getNumRowBlocks()`
`long`
`getNumRows()`
`T`
`slice(long rl, long ru, long cl, long cu, T block)`
Utility for slice operations over partitioned matrices, where the index range can cover multiple blocks.

### Methods inherited from class java.lang.Object

`equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

## Constructor Details

### PartitionedBroadcast

public PartitionedBroadcast()

### PartitionedBroadcast

public PartitionedBroadcast(org.apache.spark.broadcast.Broadcast<PartitionedBlock<T>>[] broadcasts, DataCharacteristics dc)

## Method Details

### getBroadcasts

public org.apache.spark.broadcast.Broadcast<PartitionedBlock<T>>[] getBroadcasts()

### getNumRows

public long getNumRows()

### getNumCols

public long getNumCols()

### getNumRowBlocks

public int getNumRowBlocks()

### getNumColumnBlocks

public int getNumColumnBlocks()

### getDataCharacteristics

public DataCharacteristics getDataCharacteristics()

### computeBlocksPerPartition

public static int computeBlocksPerPartition(long rlen, long clen, long blen)

### computeBlocksPerPartition

public static int computeBlocksPerPartition(long[] dims, int blen)

### getBlock

public T getBlock(int rowIndex, int colIndex)

### getBlock

public T getBlock(int[] ix)

### slice

public T slice(long rl, long ru, long cl, long cu, T block)
Utility for slice operations over partitioned matrices, where the index range can cover multiple blocks. The result is always a single result matrix block. All semantics are equivalent to the core matrix block slice operations.

Parameters:
  `rl` - row lower bound
  `ru` - row upper bound
  `cl` - column lower bound
  `cu` - column upper bound
  `block` - block object
Returns:
  block object

### destroy

public void destroy()
This method cleanups all underlying broadcasts of a partitioned broadcast, by forward the calls to SparkExecutionContext.cleanupBroadcastVariable.