# Package org.apache.sysds.runtime.controlprogram.parfor

package org.apache.sysds.runtime.controlprogram.parfor

Related Packages
Package
Description
org.apache.sysds.runtime.controlprogram
org.apache.sysds.runtime.controlprogram.parfor.opt
org.apache.sysds.runtime.controlprogram.parfor.util
org.apache.sysds.runtime.controlprogram.caching
org.apache.sysds.runtime.controlprogram.context
org.apache.sysds.runtime.controlprogram.federated
org.apache.sysds.runtime.controlprogram.paramserv
Class
Description
CachedReuseVariables
DataPartitioner
This is the base class for all data partitioner.
DataPartitionerLocal
Partitions a given matrix into row or column partitions with a two pass-approach.
DataPartitionerRemoteSpark
MR job class for submitting parfor remote partitioning MR jobs.
DataPartitionerRemoteSparkMapper
NOTE: for the moment we only support binary block here TODO extend impl for binarycell and textcell Interface of Writable output in order to support both PairWritableBlock and PairWritableCell.
DataPartitionerRemoteSparkReducer
LocalParWorker
Instances of this class can be used to execute tasks in parallel.
LocalTaskQueue<T>
This class provides a way of dynamic task distribution to multiple workers in local multi-threaded environments.
ParForBody
Wrapper for exchanging parfor body data structures.
ParWorker
Super class for master/worker pattern implementations.
RemoteDPParForSpark
TODO heavy hitter maintenance TODO data partitioning with binarycell
RemoteDPParForSparkWorker
RemoteParForJobReturn
Wrapper for job return of ParFor REMOTE for transferring statistics and result symbol table.
RemoteParForSpark
This class serves two purposes: (1) isolating Spark imports to enable running in environments where no Spark libraries are available, and (2) to follow the same structure as the parfor remote_mr job submission.
RemoteParForSparkWorker
RemoteParForUtils
Common functionalities for parfor workers in MR jobs.
ResultMerge<T extends CacheableData<?>>
ResultMergeFrameLocalMemory
ResultMergeLocalAutomatic
ResultMergeLocalFile
ResultMergeLocalMemory
Local in-memory realization of result merge.
ResultMergeMatrix
Due to independence of all iterations, any result has the following properties:
ResultMergeRemoteSpark
ResultMergeRemoteSparkWCompare
Task
A task is a logical group of one or multiple iterations (each iteration is assigned to exactly one task).
Task.TaskType
TaskPartitioner
This is the base class for all task partitioner.
TaskPartitionerFactoring
This factoring task partitioner virtually iterates over the given FOR loop (from, to, incr), creates iterations and group them to tasks.
TaskPartitionerFactoringCmax
Factoring with maximum constraint (e.g., if LIX matrix out-of-core and we need to bound the maximum number of iterations per map task -> memory bounds)
TaskPartitionerFactoringCmin
Factoring with minimum constraint (e.g., if communication is expensive)
TaskPartitionerFactory
TaskPartitionerFixedsize
This naive task partitioner virtually iterates over the given FOR loop (from, to, incr), creates iterations and group them to tasks according to the given task size.
TaskPartitionerNaive
This static task partitioner virtually iterates over the given FOR loop (from, to, incr), creates iterations and group them to tasks according to a task size of numIterations/numWorkers.
TaskPartitionerStatic
This static task partitioner virtually iterates over the given FOR loop (from, to, incr), creates iterations and group them to tasks according to a task size of numIterations/numWorkers.