Package org.apache.sysds.runtime.instructions.spark.utils

# Class RDDAggregateUtils

java.lang.Object
org.apache.sysds.runtime.instructions.spark.utils.RDDAggregateUtils

public class RDDAggregateUtils extends Object
Collection of utility methods for aggregating binary block rdds. As a general policy always call stable algorithms which maintain corrections over blocks per key. The performance overhead over a simple reducebykey is roughly 7-10% and with that acceptable.

## Constructor Summary

Constructors
Constructor
Description
`RDDAggregateUtils()`

## Method Summary

Modifier and Type
Method
Description
`static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock>`
`aggByKeyStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in, AggregateOperator aop)`
`static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock>`
`aggByKeyStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in, AggregateOperator aop, boolean deepCopyCombiner)`
`static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock>`
`aggByKeyStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in, AggregateOperator aop, int numPartitions, boolean deepCopyCombiner)`
`static MatrixBlock`
`aggStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in, AggregateOperator aop)`
Single block aggregation over pair rdds with corrections for numerical stability.
`static MatrixBlock`
`aggStable(org.apache.spark.api.java.JavaRDD<MatrixBlock> in, AggregateOperator aop)`
Single block aggregation over rdds with corrections for numerical stability.
`static TensorBlock`
`aggStableTensor(org.apache.spark.api.java.JavaPairRDD<TensorIndexes,TensorBlock> in, AggregateOperator aop)`
Single block aggregation over pair rdds with corrections for numerical stability.
`static TensorBlock`
`aggStableTensor(org.apache.spark.api.java.JavaRDD<TensorBlock> in, AggregateOperator aop)`
Single block aggregation over rdds with corrections for numerical stability.
`static double`
`max(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in)`
`static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock>`
`mergeByKey(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in)`
Merges disjoint data of all blocks per key.
`static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock>`
`mergeByKey(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in, boolean deepCopyCombiner)`
Merges disjoint data of all blocks per key.
`static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock>`
`mergeByKey(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in, int numPartitions, boolean deepCopyCombiner)`
Merges disjoint data of all blocks per key.
`static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock>`
`mergeRowsByKey(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,RowMatrixBlock> in)`
Merges disjoint data of all blocks per key.
`static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock>`
`sumByKeyStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in)`
`static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock>`
`sumByKeyStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in, boolean deepCopyCombiner)`
`static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock>`
`sumByKeyStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in, int numPartitions, boolean deepCopyCombiner)`
`static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,Double>`
`sumCellsByKeyStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,Double> in)`
`static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,Double>`
`sumCellsByKeyStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,Double> in, int numParts)`
`static MatrixBlock`
`sumStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in)`
`static MatrixBlock`
`sumStable(org.apache.spark.api.java.JavaRDD<MatrixBlock> in)`

### Methods inherited from class java.lang.Object

`equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

## Constructor Details

### RDDAggregateUtils

public RDDAggregateUtils()

## Method Details

### sumStable

public static MatrixBlock sumStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in)

### sumStable

public static MatrixBlock sumStable(org.apache.spark.api.java.JavaRDD<MatrixBlock> in)

### sumByKeyStable

public static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> sumByKeyStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in)

### sumByKeyStable

public static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> sumByKeyStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in, boolean deepCopyCombiner)

### sumByKeyStable

public static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> sumByKeyStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in, int numPartitions, boolean deepCopyCombiner)

### sumCellsByKeyStable

public static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,Double> sumCellsByKeyStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,Double> in)

### sumCellsByKeyStable

public static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,Double> sumCellsByKeyStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,Double> in, int numParts)

### aggStable

public static MatrixBlock aggStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in, AggregateOperator aop)
Single block aggregation over pair rdds with corrections for numerical stability.

Parameters:
  `in` - matrix as `JavaPairRDD<MatrixIndexes, MatrixBlock>`
  `aop` - aggregate operator
Returns:
  matrix block

### aggStable

public static MatrixBlock aggStable(org.apache.spark.api.java.JavaRDD<MatrixBlock> in, AggregateOperator aop)
Single block aggregation over rdds with corrections for numerical stability.

Parameters:
  `in` - matrix as `JavaRDD<MatrixBlock>`
  `aop` - aggregate operator
Returns:
  matrix block

### aggStableTensor

public static TensorBlock aggStableTensor(org.apache.spark.api.java.JavaPairRDD<TensorIndexes,TensorBlock> in, AggregateOperator aop)
Single block aggregation over pair rdds with corrections for numerical stability.

Parameters:
  `in` - tensor as `JavaPairRDD<TensorIndexes, TensorBlock>`
  `aop` - aggregate operator
Returns:
  tensor block

### aggStableTensor

public static TensorBlock aggStableTensor(org.apache.spark.api.java.JavaRDD<TensorBlock> in, AggregateOperator aop)
Single block aggregation over rdds with corrections for numerical stability.

Parameters:
  `in` - tensor as `JavaRDD<TensorBlock>`
  `aop` - aggregate operator
Returns:
  tensor block

### aggByKeyStable

public static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> aggByKeyStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in, AggregateOperator aop)

### aggByKeyStable

public static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> aggByKeyStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in, AggregateOperator aop, boolean deepCopyCombiner)

### aggByKeyStable

public static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> aggByKeyStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in, AggregateOperator aop, int numPartitions, boolean deepCopyCombiner)

### max

public static double max(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in)

### mergeByKey

public static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> mergeByKey(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in)
Merges disjoint data of all blocks per key. Note: The behavior of this method is undefined for both sparse and dense data if the assumption of disjoint data is violated.

Parameters:
  `in` - matrix as `JavaPairRDD<MatrixIndexes, MatrixBlock>`
Returns:
  matrix as `JavaPairRDD<MatrixIndexes, MatrixBlock>`

### mergeByKey

public static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> mergeByKey(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in, boolean deepCopyCombiner)
Merges disjoint data of all blocks per key. Note: The behavior of this method is undefined for both sparse and dense data if the assumption of disjoint data is violated.

Parameters:
  `in` - matrix as `JavaPairRDD<MatrixIndexes, MatrixBlock>`
  `deepCopyCombiner` - indicator if the createCombiner functions needs to deep copy the input block
Returns:
  matrix as `JavaPairRDD<MatrixIndexes, MatrixBlock>`

### mergeByKey

public static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> mergeByKey(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in, int numPartitions, boolean deepCopyCombiner)
Merges disjoint data of all blocks per key. Note: The behavior of this method is undefined for both sparse and dense data if the assumption of disjoint data is violated.

Parameters:
  `in` - matrix as `JavaPairRDD<MatrixIndexes, MatrixBlock>`
  `numPartitions` - number of output partitions
  `deepCopyCombiner` - indicator if the createCombiner functions needs to deep copy the input block
Returns:
  matrix as `JavaPairRDD<MatrixIndexes, MatrixBlock>`

### mergeRowsByKey

public static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> mergeRowsByKey(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,RowMatrixBlock> in)
Merges disjoint data of all blocks per key. Note: The behavior of this method is undefined for both sparse and dense data if the assumption of disjoint data is violated.

Parameters:
  `in` - matrix as `JavaPairRDD<MatrixIndexes, RowMatrixBlock>`
Returns:
  matrix as `JavaPairRDD<MatrixIndexes, MatrixBlock>`