Package org.apache.sysds.utils.stats

# Class InfrastructureAnalyzer

java.lang.Object
org.apache.sysds.utils.stats.InfrastructureAnalyzer

public class InfrastructureAnalyzer extends Object
Central place for analyzing and obtaining static infrastructure properties such as memory and number of logical processors.

## Field Summary

Fields
Modifier and Type
Field
Description
`static final long`
`DEFAULT_JVM_SIZE`
`static final String`
`MR_FRAMEWORK_NAME`
`static final String`
`MR_JOBTRACKER_ADDRESS`

## Constructor Summary

Constructors
Constructor
Description
`InfrastructureAnalyzer()`

## Method Summary

Modifier and Type
Method
Description
`static long`
`extractMaxMemoryOpt(String javaOpts)`
`static long`
`getBlockSize(org.apache.hadoop.fs.FileSystem fs)`
`static int`
`getCkMaxCP()`
Gets the maximum local parallelism constraint.
`static int`
`getCkMaxMR()`
Gets the maximum remote parallelism constraint
`static double`
`getClusterUtilization(boolean mapOnly)`
Gets the fraction of running map/reduce tasks to existing map/reduce task slots.
`static long`
`getCmMax()`
Gets the maximum memory constraint [in bytes].
`static long`
`getHDFSBlockSize()`
Gets the HDFS blocksize of the used cluster in bytes.
`static long`
`getLocalMaxMemory()`
Gets the maximum memory [in bytes] of the current JVM.
`static double`
`getLocalMaxMemoryFraction()`
`static int`
`getLocalParallelism()`
Gets the number of logical processors of the current node, including hyper-threading if enabled.
`static int`
`getRemoteParallelMapTasks()`
Gets the number of remote parallel map slots.
`static int`
`getRemoteParallelNodes()`
Gets the number of cluster nodes (number of tasktrackers).
`static int`
`getRemoteParallelReduceTasks()`
Gets the total number of remote parallel reduce slots.
`static boolean`
`isLocalMode()`
`static boolean`
`isLocalMode(org.apache.hadoop.mapred.JobConf job)`
`static void`
`setLocalMaxMemory(long localMem)`
`static void`
`setLocalPar(int localPar)`
`static void`
`setMaxMemoryOpt(org.apache.hadoop.mapred.JobConf job, String key, long bytes)`
`static void`
`setRemoteParallelMapTasks(int pmap)`
`static void`
`setRemoteParallelReduceTasks(int preduce)`

### Methods inherited from class java.lang.Object

`equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

## Field Details

### DEFAULT_JVM_SIZE

public static final long DEFAULT_JVM_SIZE

See Also:
  - Constant Field Values

### MR_FRAMEWORK_NAME

public static final String MR_FRAMEWORK_NAME

See Also:
  - Constant Field Values

### MR_JOBTRACKER_ADDRESS

public static final String MR_JOBTRACKER_ADDRESS

See Also:
  - Constant Field Values

## Constructor Details

### InfrastructureAnalyzer

public InfrastructureAnalyzer()

## Method Details

### getLocalParallelism

public static int getLocalParallelism()
Gets the number of logical processors of the current node, including hyper-threading if enabled.

Returns:
  number of local processors of the current node

### getRemoteParallelNodes

public static int getRemoteParallelNodes()
Gets the number of cluster nodes (number of tasktrackers). If multiple tasktracker are started per node, each tasktracker is viewed as individual node.

Returns:
  number of cluster nodes

### getRemoteParallelMapTasks

public static int getRemoteParallelMapTasks()
Gets the number of remote parallel map slots.

Returns:
  number of remote parallel map tasks

### setRemoteParallelMapTasks

public static void setRemoteParallelMapTasks(int pmap)

### getRemoteParallelReduceTasks

public static int getRemoteParallelReduceTasks()
Gets the total number of remote parallel reduce slots.

Returns:
  number of remote parallel reduce tasks

### setRemoteParallelReduceTasks

public static void setRemoteParallelReduceTasks(int preduce)

### getLocalMaxMemory

public static long getLocalMaxMemory()
Gets the maximum memory [in bytes] of the current JVM.

Returns:
  maximum memory of the current JVM

### setLocalMaxMemory

public static void setLocalMaxMemory(long localMem)

### setLocalPar

public static void setLocalPar(int localPar)

### getLocalMaxMemoryFraction

public static double getLocalMaxMemoryFraction()

### isLocalMode

public static boolean isLocalMode()

### isLocalMode

public static boolean isLocalMode(org.apache.hadoop.mapred.JobConf job)

### getCkMaxCP

public static int getCkMaxCP()
Gets the maximum local parallelism constraint.

Returns:
  maximum local parallelism constraint

### getCkMaxMR

public static int getCkMaxMR()
Gets the maximum remote parallelism constraint

Returns:
  maximum remote parallelism constraint

### getCmMax

public static long getCmMax()
Gets the maximum memory constraint [in bytes].

Returns:
  maximum memory constraint

### getBlockSize

public static long getBlockSize(org.apache.hadoop.fs.FileSystem fs)

### getHDFSBlockSize

public static long getHDFSBlockSize()
Gets the HDFS blocksize of the used cluster in bytes.

Returns:
  HDFS block size

### extractMaxMemoryOpt

public static long extractMaxMemoryOpt(String javaOpts)

### setMaxMemoryOpt

public static void setMaxMemoryOpt(org.apache.hadoop.mapred.JobConf job, String key, long bytes)

### getClusterUtilization

public static double getClusterUtilization(boolean mapOnly) throws IOException
Gets the fraction of running map/reduce tasks to existing map/reduce task slots. NOTE: on YARN the number of slots is a spurious indicator because containers are purely scheduled based on memory.

Parameters:
  `mapOnly` - if true, only look at map tasks
Returns:
  cluster utilization (current / capacity)
Throws:
  `IOException` - if IOException occurs