Package org.apache.sysds.hops

# Class Hop

java.lang.Object
org.apache.sysds.hops.Hop

All Implemented Interfaces:
  `ParseInfo`

Direct Known Subclasses:
  `DataOp`, `IndexingOp`, `LeftIndexingOp`, `LiteralOp`, `MultiThreadedHop`, `NaryOp`

public abstract class Hop extends Object implements ParseInfo
Hop is a High level operator, that is the first intermediate representation compiled from the definitions supplied in DML.

## Field Summary

Fields
Modifier and Type
Field
Description
`int`
`_beginColumn`
`int`
`_beginLine`
`int`
`_endColumn`
`int`
`_endLine`
`String`
`_filename`
`String`
`_text`
`static final long`
`CPThreshold`

## Constructor Summary

Constructors
Constructor
Description
`Hop(String l, Types.DataType dt, Types.ValueType vt)`

## Method Summary

Modifier and Type
Method
Description
`void`
`activatePrefetch()`
Activate prefetch of HOP.
`void`
`addAllInputs(List<Hop> list)`
`void`
`addInput(Hop h)`
`abstract boolean`
`allowsAllExecTypes()`
`void`
`checkAndSetForcedPlatform()`
`void`
`checkAndSetInvalidCPDimsAndSize()`
`void`
`clearMemEstimate()`
`abstract Object`
`clone()`
`boolean`
`colsKnown()`
`abstract boolean`
`compare(Hop that)`
`long`
`compressedSize()`
`double`
`computeBoundsInformation(Hop input)`
`static double`
`computeBoundsInformation(Hop input, LocalVariableMap vars)`
`static double`
`computeBoundsInformation(Hop input, LocalVariableMap vars, Map<Long,Double> memo)`
`void`
`computeMemEstimate(MemoTable memo)`
Computes the estimate of memory required to store the input/output of this hop in memory.
`static long`
`computeSizeInformation(Hop input)`
`long`
`computeSizeInformation(Hop input, LocalVariableMap vars)`
`long`
`computeSizeInformation(Hop input, LocalVariableMap vars, Map<Long,Long> memo)`
`void`
`constructAndSetLopsDataFlowProperties()`
`abstract Lop`
`constructLops()`
`static Lop`
`createOffsetLop(Hop hop, boolean repCols)`
`void`
`deactivatePrefetch()`
`boolean`
`dimsKnown()`
`boolean`
`dimsKnown(boolean includeNnz)`
`boolean`
`dimsKnownAny()`
`int`
`getBeginColumn()`
`int`
`getBeginLine()`
`int`
`getBlocksize()`
`long`
`getCompressedSize()`
`DataCharacteristics`
`getDataCharacteristics()`
`Types.DataType`
`getDataType()`
`long`
`getDim(int i)`
`long`
`getDim1()`
Get the number of rows in the Hop.
`long`
`getDim2()`
Get the number of columns in the Hop.
`int`
`getEndColumn()`
`int`
`getEndLine()`
`Types.ExecType`
`getExecType()`
`FEDInstruction.FederatedOutput`
`getFederatedOutput()`
`String`
`getFilename()`
`Types.ExecType`
`getForcedExecType()`
`long`
`getHopID()`
`List<Hop>`
`getInput()`
`Hop`
`getInput(int ix)`
`double`
`getInputMemEstimate()`
Get the memory estimate of inputs as the sum of input estimates in bytes.
`double`
`getInputMemEstimate(double injectedDefault)`
Get the memory estimate of inputs as the sum of input estimates in bytes.
`double`
`getInputOutputSize(Collection<String> exclVars)`
`double`
`getIntermediateMemEstimate()`
`long`
`getLength()`
`Lop`
`getLops()`
`double`
`getMemEstimate()`
NOTES: * Purpose: Whenever the output dimensions / sparsity of a hop are unknown, this hop should store its worst-case output statistics (if known) in that table.
`String`
`getName()`
`long`
`getNnz()`
`abstract String`
`getOpString()`
`double`
`getOutputMemEstimate()`
Output memory estimate in bytes.
`double`
`getOutputMemEstimate(double injectedDefault)`
Output memory estimate in bytes with negative memory estimates replaced by the injected default.
`List<Hop>`
`getParent()`
`double`
`getSparsity()`
`double`
`getSpBroadcastSize()`
`String`
`getText()`
`MatrixObject.UpdateType`
`getUpdateType()`
`Types.ValueType`
`getValueType()`
`boolean`
`hasCompressedInput()`
`boolean`
`hasFederatedOutput()`
`boolean`
`hasLocalOutput()`
`boolean`
`hasValidCPDimsAndSize()`
`boolean`
`isCompressedOutput()`
`boolean`
`isFederated()`
Checks if ExecType is federated.
`boolean`
`isFederatedDataOp()`
Checks if the hop is a DataOp with federated data.
`abstract boolean`
`isGPUEnabled()`
In memory-based optimizer mode (see OptimizerUtils.isMemoryBasedOptLevel()), the exectype is determined by checking this method as well as memory budget of this Hop.
`boolean`
`isMatrix()`
`boolean`
`isMemEstimated()`
`boolean`
`isOutputEmptyBlocks()`
`boolean`
`isRequiredDecompression()`
`boolean`
`isScalar()`
`boolean`
`isTransposeSafe()`
Defines if this operation is transpose-safe, which means that the result of op(input) is equivalent to op(t(input)).
`boolean`
`isVisited()`
`boolean`
`prefetchActivated()`
Checks if prefetch is activated for this hop.
`String`
`printErrorLocation()`
`void`
`refreshColsParameterInformation(Hop input, LocalVariableMap vars)`
`void`
`refreshColsParameterInformation(Hop input, LocalVariableMap vars, Map<Long,Long> memo)`
`void`
`refreshMemEstimates(MemoTable memo)`
Recursively computes memory estimates for all the Hops in the DAG rooted at the current hop pointed by `this`.
`void`
`refreshRowsParameterInformation(Hop input, LocalVariableMap vars)`
`void`
`refreshRowsParameterInformation(Hop input, LocalVariableMap vars, Map<Long,Long> memo)`
`abstract void`
`refreshSizeInformation()`
Update the output size information for this hop.
`boolean`
`requiresCheckpoint()`
`boolean`
`requiresCompression()`
`boolean`
`requiresLineageCaching()`
`boolean`
`requiresReblock()`
`boolean`
`requiresRecompile()`
Indicates if dynamic recompilation is required for this hop.
`void`
`resetExecType()`
`static void`
`resetRecompilationFlag(ArrayList<Hop> hops, Types.ExecType et, Recompiler.ResetType reset)`
`static void`
`resetRecompilationFlag(Hop hops, Types.ExecType et, Recompiler.ResetType reset)`
`Hop`
`resetVisitStatus()`
`static void`
`resetVisitStatus(List<Hop> hops)`
`static void`
`resetVisitStatus(List<Hop> hops, boolean force)`
`void`
`resetVisitStatusForced(HashSet<Long> memo)`
`boolean`
`rowsKnown()`
`void`
`setBeginColumn(int passed)`
`void`
`setBeginLine(int passed)`
`void`
`setBlocksize(int blen)`
`void`
`setCompressedOutput(boolean value)`
`void`
`setCompressedSize(long size)`
`void`
`setDataType(Types.DataType dt)`
`void`
`setDim(int i, long dim)`
`void`
`setDim1(long dim1)`
`void`
`setDim2(long dim2)`
`void`
`setEndColumn(int passed)`
`void`
`setEndLine(int passed)`
`void`
`setExecType(Types.ExecType execType)`
`void`
`setFederatedOutput(FEDInstruction.FederatedOutput federatedOutput)`
`void`
`setFilename(String passed)`
`void`
`setForcedExecType(Types.ExecType etype)`
`void`
`setLops(Lop lops)`
`void`
`setMemEstimate(double mem)`
Sets memory estimate in bytes
`void`
`setName(String _name)`
`void`
`setNnz(long nnz)`
`void`
`setOutputEmptyBlocks(boolean flag)`
`void`
`setParseInfo(ParseInfo parseInfo)`
Set parse information.
`void`
`setRequiresCheckpoint(boolean flag)`
`void`
`setRequiresCompression()`
`void`
`setRequiresCompression(AWTreeNode node)`
`void`
`setRequiresDeCompression()`
`void`
`setRequiresLineageCaching(boolean flag)`
`void`
`setRequiresReblock(boolean flag)`
`void`
`setRequiresRecompile()`
Marks the hop for dynamic recompilation.
`void`
`setText(String text)`
`void`
`setUpdateType(MatrixObject.UpdateType update)`
`void`
`setValueType(Types.ValueType vt)`
`void`
`setVisited()`
`void`
`setVisited(boolean flag)`
`boolean`
`someInputFederated()`
`final String`
`toString()`
`void`
`updateLopFedOut(Lop lop)`
`void`
`updateLopFedOut(Lop lop, Types.ExecType execType, FEDInstruction.FederatedOutput fedOut)`

### Methods inherited from class java.lang.Object

`equals, getClass, hashCode, notify, notifyAll, wait, wait, wait`

## Field Details

### CPThreshold

public static final long CPThreshold

See Also:
  - Constant Field Values

### _beginLine

public int _beginLine

### _beginColumn

public int _beginColumn

### _endLine

public int _endLine

### _endColumn

public int _endColumn

### _filename

public String _filename

### _text

public String _text

## Constructor Details

### Hop

public Hop(String l, Types.DataType dt, Types.ValueType vt)

## Method Details

### getHopID

public long getHopID()

### getExecType

public Types.ExecType getExecType()

### setExecType

public void setExecType(Types.ExecType execType)

### setFederatedOutput

public void setFederatedOutput(FEDInstruction.FederatedOutput federatedOutput)

### activatePrefetch

public void activatePrefetch()
Activate prefetch of HOP.

### deactivatePrefetch

public void deactivatePrefetch()

### prefetchActivated

public boolean prefetchActivated()
Checks if prefetch is activated for this hop.

Returns:
  true if prefetch is activated

### resetExecType

public void resetExecType()

### getForcedExecType

public Types.ExecType getForcedExecType()

### setForcedExecType

public void setForcedExecType(Types.ExecType etype)

### allowsAllExecTypes

public abstract boolean allowsAllExecTypes()

### isTransposeSafe

public boolean isTransposeSafe()
Defines if this operation is transpose-safe, which means that the result of op(input) is equivalent to op(t(input)). Usually, this applies to aggregate operations with fixed output dimension. Finally, this information is very useful in order to safely optimize the plan for sparse vectors, which otherwise would be (currently) always represented dense.

Returns:
  always returns false

### checkAndSetForcedPlatform

public void checkAndSetForcedPlatform()

### checkAndSetInvalidCPDimsAndSize

public void checkAndSetInvalidCPDimsAndSize()

### hasValidCPDimsAndSize

public boolean hasValidCPDimsAndSize()

### setRequiresReblock

public void setRequiresReblock(boolean flag)

### requiresReblock

public boolean requiresReblock()

### setRequiresCheckpoint

public void setRequiresCheckpoint(boolean flag)

### requiresCheckpoint

public boolean requiresCheckpoint()

### setRequiresCompression

public void setRequiresCompression()

### setRequiresCompression

public void setRequiresCompression(AWTreeNode node)

### setRequiresDeCompression

public void setRequiresDeCompression()

### isRequiredDecompression

public boolean isRequiredDecompression()

### requiresCompression

public boolean requiresCompression()

### setCompressedOutput

public void setCompressedOutput(boolean value)

### setCompressedSize

public void setCompressedSize(long size)

### getCompressedSize

public long getCompressedSize()

### isCompressedOutput

public boolean isCompressedOutput()

### hasCompressedInput

public boolean hasCompressedInput()

### compressedSize

public long compressedSize()

### setRequiresLineageCaching

public void setRequiresLineageCaching(boolean flag)

### requiresLineageCaching

public boolean requiresLineageCaching()

### updateLopFedOut

public void updateLopFedOut(Lop lop)

### updateLopFedOut

public void updateLopFedOut(Lop lop, Types.ExecType execType, FEDInstruction.FederatedOutput fedOut)

### constructAndSetLopsDataFlowProperties

public void constructAndSetLopsDataFlowProperties()

### createOffsetLop

public static Lop createOffsetLop(Hop hop, boolean repCols)

### setOutputEmptyBlocks

public void setOutputEmptyBlocks(boolean flag)

### isOutputEmptyBlocks

public boolean isOutputEmptyBlocks()

### getInputOutputSize

public double getInputOutputSize(Collection<String> exclVars)

### getMemEstimate

public double getMemEstimate()
NOTES: * Purpose: Whenever the output dimensions / sparsity of a hop are unknown, this hop should store its worst-case output statistics (if known) in that table. * Invocation: Intended to be called for ALL root nodes of one Hops DAG with the same (initially empty) memo table.

Returns:
  memory estimate in bytes

### setMemEstimate

public void setMemEstimate(double mem)
Sets memory estimate in bytes

Parameters:
  `mem` - memory estimate

### clearMemEstimate

public void clearMemEstimate()

### isMemEstimated

public boolean isMemEstimated()

### getInputMemEstimate

public double getInputMemEstimate()
Get the memory estimate of inputs as the sum of input estimates in bytes.

Returns:
  input memory estimate in bytes

### getInputMemEstimate

public double getInputMemEstimate(double injectedDefault)
Get the memory estimate of inputs as the sum of input estimates in bytes.

Parameters:
  `injectedDefault` - default memory estimate (bytes) used when the memory estimate of the input is negative
Returns:
  input memory estimate in bytes

### getOutputMemEstimate

public double getOutputMemEstimate()
Output memory estimate in bytes.

Returns:
  output memory estimate in bytes

### getOutputMemEstimate

public double getOutputMemEstimate(double injectedDefault)
Output memory estimate in bytes with negative memory estimates replaced by the injected default. The injected default represents the memory estimate per output cell, hence it is multiplied by the estimated dimensions of the output of the hop.

Parameters:
  `injectedDefault` - memory estimate to be returned in case the memory estimate defaults to a negative number
Returns:
  output memory estimate in bytes

### getIntermediateMemEstimate

public double getIntermediateMemEstimate()

### getSpBroadcastSize

public double getSpBroadcastSize()

### computeMemEstimate

public void computeMemEstimate(MemoTable memo)
Computes the estimate of memory required to store the input/output of this hop in memory. This is the default implementation (orchestration of hop-specific implementation) that should suffice for most hops. If a hop requires more control, this method should be overwritten with awareness of (1) output estimates, and (2) propagation of worst-case matrix characteristics (dimensions, sparsity). TODO remove memo table and, on constructor refresh, inference in refresh, single compute mem, maybe general computeMemEstimate, flags to indicate if estimate or not.

Parameters:
  `memo` - memory table

### refreshMemEstimates

public void refreshMemEstimates(MemoTable memo)
Recursively computes memory estimates for all the Hops in the DAG rooted at the current hop pointed by `this`.

Parameters:
  `memo` - memory table

### isFederated

public boolean isFederated()
Checks if ExecType is federated.

Returns:
  true if ExecType is federated

### someInputFederated

public boolean someInputFederated()

### isFederatedDataOp

public boolean isFederatedDataOp()
Checks if the hop is a DataOp with federated data.

Returns:
  true if hop is a federated DataOp

### getParent

public List<Hop> getParent()

### getInput

public List<Hop> getInput()

### getInput

public Hop getInput(int ix)

### addInput

public void addInput(Hop h)

### addAllInputs

public void addAllInputs(List<Hop> list)

### getBlocksize

public int getBlocksize()

### setBlocksize

public void setBlocksize(int blen)

### setNnz

public void setNnz(long nnz)

### getNnz

public long getNnz()

### getFederatedOutput

public FEDInstruction.FederatedOutput getFederatedOutput()

### hasFederatedOutput

public boolean hasFederatedOutput()

### hasLocalOutput

public boolean hasLocalOutput()

### setUpdateType

public void setUpdateType(MatrixObject.UpdateType update)

### getUpdateType

public MatrixObject.UpdateType getUpdateType()

### constructLops

public abstract Lop constructLops()

### getOpString

public abstract String getOpString()

### toString

public final String toString()

Overrides:
  `toString` in class `Object`

### isGPUEnabled

public abstract boolean isGPUEnabled()
In memory-based optimizer mode (see OptimizerUtils.isMemoryBasedOptLevel()), the exectype is determined by checking this method as well as memory budget of this Hop. Please see findExecTypeByMemEstimate for more detail. This method is necessary because not all operator are supported efficiently on GPU (for example: operations on frames and scalar as well as operations such as table).

Returns:
  true if the Hop is eligible for GPU Exectype.

### dimsKnown

public boolean dimsKnown()

### dimsKnown

public boolean dimsKnown(boolean includeNnz)

### dimsKnownAny

public boolean dimsKnownAny()

### rowsKnown

public boolean rowsKnown()

### colsKnown

public boolean colsKnown()

### resetVisitStatus

public static void resetVisitStatus(List<Hop> hops)

### resetVisitStatus

public static void resetVisitStatus(List<Hop> hops, boolean force)

### resetVisitStatus

public Hop resetVisitStatus()

### resetVisitStatusForced

public void resetVisitStatusForced(HashSet<Long> memo)

### resetRecompilationFlag

public static void resetRecompilationFlag(ArrayList<Hop> hops, Types.ExecType et, Recompiler.ResetType reset)

### resetRecompilationFlag

public static void resetRecompilationFlag(Hop hops, Types.ExecType et, Recompiler.ResetType reset)

### getDim1

public long getDim1()
Get the number of rows in the Hop.

Returns:
  a long.

### setDim1

public void setDim1(long dim1)

### getDim2

public long getDim2()
Get the number of columns in the Hop.

Returns:
  a long.

### setDim2

public void setDim2(long dim2)

### getDim

public long getDim(int i)

### setDim

public void setDim(int i, long dim)

### getLength

public long getLength()

### getSparsity

public double getSparsity()

### getDataCharacteristics

public DataCharacteristics getDataCharacteristics()

### getLops

public Lop getLops()

### setLops

public void setLops(Lop lops)

### isVisited

public boolean isVisited()

### getDataType

public Types.DataType getDataType()

### setDataType

public void setDataType(Types.DataType dt)

### isScalar

public boolean isScalar()

### isMatrix

public boolean isMatrix()

### setVisited

public void setVisited()

### setVisited

public void setVisited(boolean flag)

### setName

public void setName(String _name)

### getName

public String getName()

### getValueType

public Types.ValueType getValueType()

### setValueType

public void setValueType(Types.ValueType vt)

### requiresRecompile

public boolean requiresRecompile()
Indicates if dynamic recompilation is required for this hop.

Returns:
  true if dynamic recompilation required

### setRequiresRecompile

public void setRequiresRecompile()
Marks the hop for dynamic recompilation.

### refreshSizeInformation

public abstract void refreshSizeInformation()
Update the output size information for this hop.

### computeSizeInformation

public static long computeSizeInformation(Hop input)

### refreshRowsParameterInformation

public void refreshRowsParameterInformation(Hop input, LocalVariableMap vars)

### refreshRowsParameterInformation

public void refreshRowsParameterInformation(Hop input, LocalVariableMap vars, Map<Long,Long> memo)

### refreshColsParameterInformation

public void refreshColsParameterInformation(Hop input, LocalVariableMap vars)

### refreshColsParameterInformation

public void refreshColsParameterInformation(Hop input, LocalVariableMap vars, Map<Long,Long> memo)

### computeSizeInformation

public long computeSizeInformation(Hop input, LocalVariableMap vars)

### computeSizeInformation

public long computeSizeInformation(Hop input, LocalVariableMap vars, Map<Long,Long> memo)

### computeBoundsInformation

public double computeBoundsInformation(Hop input)

### computeBoundsInformation

public static double computeBoundsInformation(Hop input, LocalVariableMap vars)

### computeBoundsInformation

public static double computeBoundsInformation(Hop input, LocalVariableMap vars, Map<Long,Double> memo)

### clone

public abstract Object clone() throws CloneNotSupportedException

Throws:
  `CloneNotSupportedException`

### compare

public abstract boolean compare(Hop that)

### setBeginLine

public void setBeginLine(int passed)

Specified by:
  `setBeginLine` in interface `ParseInfo`

### setBeginColumn

public void setBeginColumn(int passed)

Specified by:
  `setBeginColumn` in interface `ParseInfo`

### setEndLine

public void setEndLine(int passed)

Specified by:
  `setEndLine` in interface `ParseInfo`

### setEndColumn

public void setEndColumn(int passed)

Specified by:
  `setEndColumn` in interface `ParseInfo`

### setFilename

public void setFilename(String passed)

Specified by:
  `setFilename` in interface `ParseInfo`

### setText

public void setText(String text)

Specified by:
  `setText` in interface `ParseInfo`

### getBeginLine

public int getBeginLine()

Specified by:
  `getBeginLine` in interface `ParseInfo`

### getBeginColumn

public int getBeginColumn()

Specified by:
  `getBeginColumn` in interface `ParseInfo`

### getEndLine

public int getEndLine()

Specified by:
  `getEndLine` in interface `ParseInfo`

### getEndColumn

public int getEndColumn()

Specified by:
  `getEndColumn` in interface `ParseInfo`

### getFilename

public String getFilename()

Specified by:
  `getFilename` in interface `ParseInfo`

### getText

public String getText()

Specified by:
  `getText` in interface `ParseInfo`

### printErrorLocation

public String printErrorLocation()

### setParseInfo

public void setParseInfo(ParseInfo parseInfo)
Set parse information.

Parameters:
  `parseInfo` - parse information, such as beginning line position, beginning column position, ending line position, ending column position, text, and filename