Package org.apache.sysds.runtime.matrix.data

# Class LibMatrixReorg

java.lang.Object
org.apache.sysds.runtime.matrix.data.LibMatrixReorg

public class LibMatrixReorg extends Object
MB: Library for selected matrix reorg operations including special cases and all combinations of dense and sparse representations. Current list of supported operations: - reshape, - r' (transpose), - rdiag (diagV2M/diagM2V), - rsort (sorting data/indexes) - rmempty (remove empty) - rexpand (outer/table-seq expansion)

## Field Summary

Fields
Modifier and Type
Field
Description
`static long`
`PAR_NUMCELL_THRESHOLD`
`static final int`
`PAR_NUMCELL_THRESHOLD_SORT`
`static final boolean`
`SHALLOW_COPY_REORG`
`static final boolean`
`SPARSE_OUTPUTS_IN_CSR`
`static final boolean`
`TRANSPOSE_IN_PLACE_DENSE_LEGACY`

## Method Summary

Modifier and Type
Method
Description
`static void`
`checkRexpand(MatrixBlock in, boolean ignore)`
Quick check if the input is valid for rexpand, this check does not guarantee that the input is valid for rexpand
`static void`
`copyDenseMtx(MatrixBlock in, MatrixBlock out, int inIdx, int outIdx, int copyLen, boolean isAllocated, boolean copyTotalNonZeros)`
`static void`
`copyMtx(MatrixBlock in, MatrixBlock out, int inStart, int outStart, int copyLen, boolean isAllocated, boolean copyTotalNonZeros)`
`static Future<int[]>`
`countNNZColumns(MatrixBlock in, int k, ExecutorService pool)`
`static List<Future<int[]>>`
`countNNZColumnsFuture(MatrixBlock in, int k, ExecutorService pool)`
`static int[]`
`countNnzPerColumn(MatrixBlock in)`
`static int[]`
`countNnzPerColumn(MatrixBlock in, int rl, int ru)`
`static MatrixBlock`
`diag(MatrixBlock in, MatrixBlock out)`
`static int`
`findSplitIndex(int[] perm)`
Finds a split index for a tensor permutation that allows reduction of the permutation to a 2D matrix transpose.
`static MatrixBlock`
`fusedSeqRexpand(int seqHeight, MatrixBlock A, double w)`
The DML code to activate this function:
`static MatrixBlock`
`fusedSeqRexpand(int seqHeight, MatrixBlock A, double w, MatrixBlock ret, boolean updateClen)`
The DML code to activate this function:
`static MatrixBlock`
`fusedSeqRexpand(int seqHeight, MatrixBlock A, double w, MatrixBlock ret, boolean updateClen, int k)`
The DML code to activate this function:
`static boolean`
`isSupportedReorgOperator(ReorgOperator op)`
`static int[]`
`mergeNnzCounts(int[] cnt, int[] cnt2)`
`static MatrixBlock`
`permute(MatrixBlock in, int[] inDims, int[] perm)`
`static MatrixBlock`
`permute(MatrixBlock in, int[] inDims, int[] perm, int k)`
`static MatrixBlock`
`removeEmptyRows(MatrixBlock in, MatrixBlock ret, boolean emptyReturn, boolean selectNull, boolean[] flags, int rlen2)`
Remove selected rows, based on the boolean array given.
`static MatrixBlock`
`reorg(MatrixBlock in, MatrixBlock out, ReorgOperator op)`
`static MatrixBlock`
`reorgInPlace(MatrixBlock in, ReorgOperator op)`
`static List<IndexedMatrixValue>`
`reshape(IndexedMatrixValue in, DataCharacteristics mcIn, DataCharacteristics mcOut, boolean rowwise, boolean outputEmptyBlocks)`
MR/SPARK reshape interface - for reshape we cannot view blocks independently, and hence, there are different CP and MR interfaces.
`static MatrixBlock`
`reshape(MatrixBlock in, int rows, int cols, boolean rowwise)`
CP reshape operation (single input, single output matrix) NOTE: In contrast to R, the rowwise parameter specifies both the read and write order, with row-wise being the default, while R uses always a column-wise read, rowwise specifying the write order and column-wise being the default.
`static MatrixBlock`
`reshape(MatrixBlock in, MatrixBlock out, int rows, int cols, boolean rowwise)`
CP reshape operation (single input, single output matrix) NOTE: In contrast to R, the rowwise parameter specifies both the read and write order, with row-wise being the default, while R uses always a column-wise read, rowwise specifying the write order and column-wise being the default.
`static MatrixBlock`
`reshape(MatrixBlock in, MatrixBlock out, int rows, int cols, boolean rowwise, int k)`
CP reshape operation (single input, single output matrix) NOTE: In contrast to R, the rowwise parameter specifies both the read and write order, with row-wise being the default, while R uses always a column-wise read, rowwise specifying the write order and column-wise being the default.
`static void`
`rev(IndexedMatrixValue in, long rlen, int blen, ArrayList<IndexedMatrixValue> out)`
`static MatrixBlock`
`rev(MatrixBlock in, MatrixBlock out)`
`static MatrixBlock`
`rev(MatrixBlock in, MatrixBlock out, int k)`
`static void`
`rexpand(IndexedMatrixValue data, double max, boolean rows, boolean cast, boolean ignore, long blen, ArrayList<IndexedMatrixValue> outList)`
MR/Spark rexpand operation (single input, multiple outputs incl empty blocks)
`static MatrixBlock`
`rexpand(MatrixBlock in, MatrixBlock ret, double max, boolean rows, boolean cast, boolean ignore, int k)`
CP rexpand operation (single input, single output), the classic example of this operation is one hot encoding of a column to multiple columns.
`static MatrixBlock`
`rexpand(MatrixBlock in, MatrixBlock ret, int max, boolean rows, boolean cast, boolean ignore, int k)`
CP rexpand operation (single input, single output), the classic example of this operation is one hot encoding of a column to multiple columns.
`static int`
`rexpandSingleRow(int row, double v2, double w, int[] retIx, double[] retVals, boolean updateClen, int maxOutCol)`
`static void`
`rmempty(IndexedMatrixValue data, IndexedMatrixValue offset, boolean rmRows, long len, long blen, ArrayList<IndexedMatrixValue> outList)`
MR rmempty interface - for rmempty we cannot view blocks independently, and hence, there are different CP and MR interfaces.
`static MatrixBlock`
`rmempty(MatrixBlock in, MatrixBlock ret, boolean rows, boolean emptyReturn, MatrixBlock select)`
CP rmempty operation (single input, single output matrix)
`static MatrixBlock`
`rmemptyEarlyAbort(MatrixBlock in, MatrixBlock ret, boolean rows, boolean emptyReturn, MatrixBlock select)`
Handle the early-termination cases of removeEmpty that do not require scanning for empty rows/columns.
`static MatrixBlock`
`rmemptyUnsafe(MatrixBlock in, MatrixBlock ret, boolean rows, boolean emptyReturn, MatrixBlock select)`
`static void`
`roll(IndexedMatrixValue in, long rlen, int blen, int shift, ArrayList<IndexedMatrixValue> out)`
`static MatrixBlock`
`roll(MatrixBlock in, MatrixBlock out, int shift)`
`static MatrixBlock`
`roll(MatrixBlock input, MatrixBlock output, int shift, int numThreads)`
`static MatrixBlock`
`sort(MatrixBlock in, MatrixBlock out, int[] by, boolean desc, boolean ixret)`
`static MatrixBlock`
`sort(MatrixBlock in, MatrixBlock out, int[] by, boolean desc, boolean ixret, int k)`
`static MatrixBlock`
`transpose(MatrixBlock in)`
`static MatrixBlock`
`transpose(MatrixBlock in, int k)`
`static MatrixBlock`
`transpose(MatrixBlock in, int k, boolean allowCSR)`
`static MatrixBlock`
`transpose(MatrixBlock in, MatrixBlock out)`
`static MatrixBlock`
`transpose(MatrixBlock in, MatrixBlock out, int k)`
`static MatrixBlock`
`transpose(MatrixBlock in, MatrixBlock out, int k, boolean allowCSR)`
`static MatrixBlock`
`transposeInPlace(MatrixBlock in, int k)`
`static void`
`transposeInPlaceDenseBrenner(MatrixBlock in, int k)`
Transposes a dense matrix in-place using following cycles based on Brenner's method.
`static boolean`
`transposeInPlaceTensor(MatrixBlock in, int[] shape, int[] perm)`
Performs prime in-place tensor transposition for arbitrary permutations.

### Methods inherited from class java.lang.Object

`equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

## Field Details

### PAR_NUMCELL_THRESHOLD

public static long PAR_NUMCELL_THRESHOLD

### PAR_NUMCELL_THRESHOLD_SORT

public static final int PAR_NUMCELL_THRESHOLD_SORT

See Also:
  - Constant Field Values

### SHALLOW_COPY_REORG

public static final boolean SHALLOW_COPY_REORG

See Also:
  - Constant Field Values

### SPARSE_OUTPUTS_IN_CSR

public static final boolean SPARSE_OUTPUTS_IN_CSR

See Also:
  - Constant Field Values

### TRANSPOSE_IN_PLACE_DENSE_LEGACY

public static final boolean TRANSPOSE_IN_PLACE_DENSE_LEGACY

See Also:
  - Constant Field Values

## Method Details

### isSupportedReorgOperator

public static boolean isSupportedReorgOperator(ReorgOperator op)

### reorg

public static MatrixBlock reorg(MatrixBlock in, MatrixBlock out, ReorgOperator op)

### reorgInPlace

public static MatrixBlock reorgInPlace(MatrixBlock in, ReorgOperator op)

### transpose

public static MatrixBlock transpose(MatrixBlock in)

### transpose

public static MatrixBlock transpose(MatrixBlock in, MatrixBlock out)

### transpose

public static MatrixBlock transpose(MatrixBlock in, int k)

### transpose

public static MatrixBlock transpose(MatrixBlock in, int k, boolean allowCSR)

### transpose

public static MatrixBlock transpose(MatrixBlock in, MatrixBlock out, int k)

### transpose

public static MatrixBlock transpose(MatrixBlock in, MatrixBlock out, int k, boolean allowCSR)

### countNNZColumns

public static Future<int[]> countNNZColumns(MatrixBlock in, int k, ExecutorService pool) throws InterruptedException, ExecutionException

Throws:
  `InterruptedException`
  `ExecutionException`

### countNNZColumnsFuture

public static List<Future<int[]>> countNNZColumnsFuture(MatrixBlock in, int k, ExecutorService pool) throws InterruptedException

Throws:
  `InterruptedException`

### transposeInPlace

public static MatrixBlock transposeInPlace(MatrixBlock in, int k)

### rev

public static MatrixBlock rev(MatrixBlock in, MatrixBlock out)

### rev

public static MatrixBlock rev(MatrixBlock in, MatrixBlock out, int k)

### rev

public static void rev(IndexedMatrixValue in, long rlen, int blen, ArrayList<IndexedMatrixValue> out)

### roll

public static MatrixBlock roll(MatrixBlock in, MatrixBlock out, int shift)

### roll

public static MatrixBlock roll(MatrixBlock input, MatrixBlock output, int shift, int numThreads)

### roll

public static void roll(IndexedMatrixValue in, long rlen, int blen, int shift, ArrayList<IndexedMatrixValue> out)

### diag

public static MatrixBlock diag(MatrixBlock in, MatrixBlock out)

### sort

public static MatrixBlock sort(MatrixBlock in, MatrixBlock out, int[] by, boolean desc, boolean ixret)

### sort

public static MatrixBlock sort(MatrixBlock in, MatrixBlock out, int[] by, boolean desc, boolean ixret, int k)

Parameters:
  `in` - Input matrix to sort
  `out` - Output matrix where the sorted input is inserted to
  `by` - The Ordering parameter
  `desc` - A boolean, specifying if it should be descending order.
  `ixret` - A boolean, specifying if the return should be the sorted indexes.
  `k` - Number of parallel threads
Returns:
  The sorted out matrix.

### reshape

public static MatrixBlock reshape(MatrixBlock in, int rows, int cols, boolean rowwise)
CP reshape operation (single input, single output matrix) NOTE: In contrast to R, the rowwise parameter specifies both the read and write order, with row-wise being the default, while R uses always a column-wise read, rowwise specifying the write order and column-wise being the default.

Parameters:
  `in` - input matrix
  `rows` - number of rows
  `cols` - number of columns
  `rowwise` - if true, reshape by row
Returns:
  output matrix

### reshape

public static MatrixBlock reshape(MatrixBlock in, MatrixBlock out, int rows, int cols, boolean rowwise)
CP reshape operation (single input, single output matrix) NOTE: In contrast to R, the rowwise parameter specifies both the read and write order, with row-wise being the default, while R uses always a column-wise read, rowwise specifying the write order and column-wise being the default.

Parameters:
  `in` - input matrix
  `out` - output matrix
  `rows` - number of rows
  `cols` - number of columns
  `rowwise` - if true, reshape by row
Returns:
  output matrix

### reshape

public static MatrixBlock reshape(MatrixBlock in, MatrixBlock out, int rows, int cols, boolean rowwise, int k)
CP reshape operation (single input, single output matrix) NOTE: In contrast to R, the rowwise parameter specifies both the read and write order, with row-wise being the default, while R uses always a column-wise read, rowwise specifying the write order and column-wise being the default.

Parameters:
  `in` - input matrix
  `out` - output matrix
  `rows` - number of rows
  `cols` - number of columns
  `rowwise` - if true, reshape by row
  `k` - The parallelization degree
Returns:
  output matrix

### reshape

public static List<IndexedMatrixValue> reshape(IndexedMatrixValue in, DataCharacteristics mcIn, DataCharacteristics mcOut, boolean rowwise, boolean outputEmptyBlocks)
MR/SPARK reshape interface - for reshape we cannot view blocks independently, and hence, there are different CP and MR interfaces.

Parameters:
  `in` - indexed matrix value
  `mcIn` - input matrix characteristics
  `mcOut` - output matrix characteristics
  `rowwise` - if true, reshape by row
  `outputEmptyBlocks` - output blocks with nnz=0
Returns:
  list of indexed matrix values

### rmempty

public static MatrixBlock rmempty(MatrixBlock in, MatrixBlock ret, boolean rows, boolean emptyReturn, MatrixBlock select)
CP rmempty operation (single input, single output matrix)

Parameters:
  `in` - The input matrix
  `ret` - The output matrix
  `rows` - If we are removing based on rows, or columns.
  `emptyReturn` - Return row/column of zeros for empty input
  `select` - An optional selection vector, to remove based on rather than empty rows or columns
Returns:
  The result MatrixBlock

### rmemptyUnsafe

public static MatrixBlock rmemptyUnsafe(MatrixBlock in, MatrixBlock ret, boolean rows, boolean emptyReturn, MatrixBlock select)

### rmemptyEarlyAbort

public static MatrixBlock rmemptyEarlyAbort(MatrixBlock in, MatrixBlock ret, boolean rows, boolean emptyReturn, MatrixBlock select)
Handle the early-termination cases of removeEmpty that do not require scanning for empty rows/columns.

Parameters:
  `in` - The input matrix
  `ret` - The output matrix, reused for the empty-input case
  `rows` - If removing based on rows, or columns
  `emptyReturn` - Return a row/column of zeros for empty input
  `select` - An optional selection vector
Returns:
  The early-abort result, or `null` if no early termination applies and the caller must continue. For the select-all case the returned block is the input `in` itself (a shallow alias, not a copy).

### rmempty

public static void rmempty(IndexedMatrixValue data, IndexedMatrixValue offset, boolean rmRows, long len, long blen, ArrayList<IndexedMatrixValue> outList)
MR rmempty interface - for rmempty we cannot view blocks independently, and hence, there are different CP and MR interfaces.

Parameters:
  `data` - ?
  `offset` - ?
  `rmRows` - ?
  `len` - ?
  `blen` - block length
  `outList` - list of indexed matrix values

### rexpand

public static MatrixBlock rexpand(MatrixBlock in, MatrixBlock ret, double max, boolean rows, boolean cast, boolean ignore, int k)
CP rexpand operation (single input, single output), the classic example of this operation is one hot encoding of a column to multiple columns.

Parameters:
  `in` - Input matrix
  `ret` - Output matrix
  `max` - Number of rows/cols of the output
  `rows` - If the expansion is in rows direction
  `cast` - If the values contained should be cast to double (rounded up and down)
  `ignore` - Ignore if the input contain values below zero that technically is incorrect input.
  `k` - Degree of parallelism
Returns:
  Output matrix rexpanded

### rexpand

public static MatrixBlock rexpand(MatrixBlock in, MatrixBlock ret, int max, boolean rows, boolean cast, boolean ignore, int k)
CP rexpand operation (single input, single output), the classic example of this operation is one hot encoding of a column to multiple columns.

Parameters:
  `in` - Input matrix
  `ret` - Output matrix
  `max` - Number of rows/cols of the output
  `rows` - If the expansion is in rows direction
  `cast` - If the values contained should be cast to double (rounded up and down)
  `ignore` - Ignore if the input contain values below zero that technically is incorrect input.
  `k` - Degree of parallelism
Returns:
  Output matrix rexpanded

### fusedSeqRexpand

public static MatrixBlock fusedSeqRexpand(int seqHeight, MatrixBlock A, double w)
The DML code to activate this function:

ret = table(seq(1, nrow(A)), A, w)

Parameters:
  `seqHeight` - A sequence vector height.
  `A` - The MatrixBlock vector to encode.
  `w` - The weight matrix to multiply on output cells.
Returns:
  A new MatrixBlock with the table result.

### fusedSeqRexpand

public static MatrixBlock fusedSeqRexpand(int seqHeight, MatrixBlock A, double w, MatrixBlock ret, boolean updateClen)
The DML code to activate this function:

ret = table(seq(1, nrow(A)), A, w)

Parameters:
  `seqHeight` - A sequence vector height.
  `A` - The MatrixBlock vector to encode.
  `w` - The weight scalar to multiply on output cells.
  `ret` - The output MatrixBlock, does not have to be used, but depending on updateClen determine the output size.
  `updateClen` - Update clen, if set to true, ignore dimensions of ret, otherwise use the column dimension of ret.
Returns:
  A new MatrixBlock or ret.

### fusedSeqRexpand

public static MatrixBlock fusedSeqRexpand(int seqHeight, MatrixBlock A, double w, MatrixBlock ret, boolean updateClen, int k)
The DML code to activate this function:

ret = table(seq(1, nrow(A)), A, w)

Parameters:
  `seqHeight` - A sequence vector height.
  `A` - The MatrixBlock vector to encode.
  `w` - The weight matrix to multiply on output cells.
  `ret` - The output MatrixBlock, does not have to be used, but depending on updateClen determine the output size.
  `updateClen` - Update clen, if set to true, ignore dimensions of ret, otherwise use the column dimension of ret.
  `k` - Parallelization degree
Returns:
  A new MatrixBlock or ret.

### rexpandSingleRow

public static int rexpandSingleRow(int row, double v2, double w, int[] retIx, double[] retVals, boolean updateClen, int maxOutCol)

### checkRexpand

public static void checkRexpand(MatrixBlock in, boolean ignore)
Quick check if the input is valid for rexpand, this check does not guarantee that the input is valid for rexpand

Parameters:
  `in` - Input matrix block
  `ignore` - If zero valued cells should be ignored

### rexpand

public static void rexpand(IndexedMatrixValue data, double max, boolean rows, boolean cast, boolean ignore, long blen, ArrayList<IndexedMatrixValue> outList)
MR/Spark rexpand operation (single input, multiple outputs incl empty blocks)

Parameters:
  `data` - Input indexed matrix block
  `max` - Total nrows/cols of the output
  `rows` - If the expansion is in rows direction
  `cast` - If the values contained should be cast to double (rounded up and down)
  `ignore` - Ignore if the input contain values below zero that technically is incorrect input.
  `blen` - The block size to slice the output up into
  `outList` - The output indexedMatrixValues (a list to add all the output blocks to / modify)

### countNnzPerColumn

public static int[] countNnzPerColumn(MatrixBlock in)

### countNnzPerColumn

public static int[] countNnzPerColumn(MatrixBlock in, int rl, int ru)

### mergeNnzCounts

public static int[] mergeNnzCounts(int[] cnt, int[] cnt2)

### copyMtx

public static void copyMtx(MatrixBlock in, MatrixBlock out, int inStart, int outStart, int copyLen, boolean isAllocated, boolean copyTotalNonZeros)

### copyDenseMtx

public static void copyDenseMtx(MatrixBlock in, MatrixBlock out, int inIdx, int outIdx, int copyLen, boolean isAllocated, boolean copyTotalNonZeros)

### removeEmptyRows

public static MatrixBlock removeEmptyRows(MatrixBlock in, MatrixBlock ret, boolean emptyReturn, boolean selectNull, boolean[] flags, int rlen2)
Remove selected rows, based on the boolean array given. Note this function is internal use only, and require a boolean vector to be constructed first.

Parameters:
  `in` - Input to remove rows from
  `ret` - Output to assign the result into
  `emptyReturn` - If the output is allowed to be empty.
  `selectNull` - If the original caller did not have a selection matrix.
  `flags` - The boolean selection vector to specify which rows to keep.
  `rlen2` - The number of true values in the flags argument.
Returns:
  Another reference to the ret matrix input argument.

### transposeInPlaceDenseBrenner

public static void transposeInPlaceDenseBrenner(MatrixBlock in, int k)
Transposes a dense matrix in-place using following cycles based on Brenner's method. This method shifts cycles with a focus on less storage by using cycle leaders based on prime factorization. The used storage is in O(n+m). Quadratic matrices should be handled outside this method (using the trivial method) for a speedup. This method is based on: Algorithm 467, Brenner, https://dl.acm.org/doi/pdf/10.1145/355611.362542.

Parameters:
  `in` - The input matrix to be transposed.
  `k` - The number of threads.

### transposeInPlaceTensor

public static boolean transposeInPlaceTensor(MatrixBlock in, int[] shape, int[] perm)
Performs prime in-place tensor transposition for arbitrary permutations.

Parameters:
  `in` - Tensor stored as MatrixBlock
  `shape` - Original shape informtion of tensor
  `perm` - Permutation of tensor

### findSplitIndex

public static int findSplitIndex(int[] perm)
Finds a split index for a tensor permutation that allows reduction of the permutation to a 2D matrix transpose.

Parameters:
  `perm` - permutation of tensor axes
Returns:
  split index {i} if reducible, otherwise {-1}

### permute

public static MatrixBlock permute(MatrixBlock in, int[] inDims, int[] perm)

### permute

public static MatrixBlock permute(MatrixBlock in, int[] inDims, int[] perm, int k)