Package org.apache.sysds.runtime.frame.data.columns

# Class Array<T>

java.lang.Object
org.apache.sysds.runtime.frame.data.columns.Array<T>

All Implemented Interfaces:
  `org.apache.hadoop.io.Writable`

Direct Known Subclasses:
  `ABooleanArray`, `ACompressedArray`, `CharArray`, `DoubleArray`, `FloatArray`, `HashIntegerArray`, `HashLongArray`, `IntegerArray`, `LongArray`, `OptionalArray`, `RaggedArray`, `StringArray`

public abstract class Array<T> extends Object implements org.apache.hadoop.io.Writable
Generic, resizable native arrays for the internal representation of the columns in the FrameBlock. We use this custom class hierarchy instead of Trove or other libraries in order to avoid unnecessary dependencies.

## Nested Class Summary

Nested Classes
Modifier and Type
Class
Description
`class`
`Array.ArrayIterator`

## Field Summary

Fields
Modifier and Type
Field
Description
`static int`
`ROW_PARALLELIZATION_THRESHOLD`
Parallelization threshold for parallelizing vector operations

## Method Summary

Modifier and Type
Method
Description
`final Pair<Types.ValueType,Boolean>`
`analyzeValueType()`
Analyze the column to figure out if the value type can be refined to a better type.
`abstract Pair<Types.ValueType,Boolean>`
`analyzeValueType(int maxCells)`
Analyze the column to figure out if the value type can be refined to a better type.
`abstract void`
`append(String value)`
Append a string value to the current Array, this should in general be avoided, and appending larger blocks at a time should be preferred.
`abstract Array<T>`
`append(Array<T> other)`
Append other array, if the other array is fitting in current allocated size use that allocated size, otherwise allocate new array to combine the other with this.
`abstract void`
`append(T value)`
Append a value of the same type of the Array.
`static long`
`baseMemoryCost()`
Get the base memory cost of the Arrays allocation.
`Array<?>`
`changeType(Types.ValueType t)`
Change the allocated array to a different type.
`Array<?>`
`changeType(Types.ValueType t, boolean containsNull)`
Change type taking into consideration if the target type must be able to contain Null.
`final Array<?>`
`changeType(Array<?> ret)`
Change type by moving this arrays value into the given ret array.
`final Array<?>`
`changeType(Array<?> ret, int rl, int ru)`
Put the changed value types into the given ret array inside the range specified.
`Array<?>`
`changeTypeWithNulls(Types.ValueType t)`
`final Array<?>`
`changeTypeWithNulls(Array<?> ret)`
`final Array<?>`
`changeTypeWithNulls(Array<?> ret, int l, int u)`
`abstract Array<T>`
`clone()`
Overwrite of the java internal clone function for arrays, return a clone of underlying data that is mutable, (not immutable data.) Immutable data is dependent on the individual allocated arrays
`boolean`
`containsNull()`
analyze if the array contains null values.
`boolean`
`equals(Object other)`
`abstract boolean`
`equals(Array<T> other)`
Equals operation on arrays.
`double[]`
`extractDouble(double[] ret, int rl, int ru)`
Extract the sub array into the ret array as doubles.
`abstract void`
`fill(String val)`
fill the entire array with specific value.
`abstract void`
`fill(T val)`
fill the entire array with specific value.
`final void`
`findEmpty(boolean[] select)`
Find the empty rows, it is assumed that the input is to be only modified to set variables to true.
`void`
`findEmptyInverse(boolean[] select)`
Find the filled rows, it is assumed that the input i to be only modified to set variables to true;
`abstract Object`
`get()`
Get the underlying array out of the column Group, it is the responsibility of the caller to know what type it is.
`abstract T`
`get(int index)`
Get the value at a given index.
`abstract byte[]`
`getAsByteArray()`
Return the current allocated Array as a byte[], this is used to serialize the allocated Arrays out to the PythonAPI.
`abstract double`
`getAsDouble(int i)`
Get the index's value.
`double`
`getAsNaNDouble(int i)`
Get the index's value.
`final SoftReference<Map<T,Integer>>`
`getCache()`
Get the current cached recode map.
`abstract long`
`getExactSerializedSize()`
Get the exact serialized size on disk of this array.
`abstract ArrayFactory.FrameArrayType`
`getFrameArrayType()`
Get the internal FrameArrayType, to specify the encoding of the Types, note there are more Frame Array Types than there is ValueTypes.
`long`
`getInMemorySize()`
Get in memory size, not counting reference to this object.
`T`
`getInternal(int index)`
Get the internal value at a given index.
`Array<T>.ArrayIterator`
`getIterator()`
`Pair<Integer,Integer>`
`getMinMaxLength()`
Get the minimum and maximum length of the contained values as string type.
`ABooleanArray`
`getNulls()`
`final Map<T,Integer>`
`getRecodeMap()`
Get a recode map that maps each unique value in the array, to a long ID.
`final Map<T,Integer>`
`getRecodeMap(int estimate)`
Get a recode map that maps each unique value in the array, to a long ID.
`final Map<T,Integer>`
`getRecodeMap(int estimate, ExecutorService pool, int k)`
Get a recode map that maps each unique value in the array, to a long ID.
`abstract Types.ValueType`
`getValueType()`
Get the current value type of this array.
`abstract double`
`hashDouble(int idx)`
Hash the given index of the array.
`abstract boolean`
`isEmpty()`
Get if this array is empty, aka filled with empty values.
`abstract boolean`
`isNotEmpty(int i)`
`abstract boolean`
`isShallowSerialize()`
analyze if this array can be shallow serialized.
`double[]`
`minMax()`
Get the minimum and maximum double value of this array.
`double[]`
`minMax(int l, int u)`
Get the minimum and maximum double value of a specific sub part of this array.
`abstract boolean`
`possiblyContainsNaN()`
`abstract void`
`reset(int size)`
Reset the Array and set to a different size.
`abstract Array<T>`
`select(boolean[] select, int nTrue)`
Slice out the true indices in the select input and return the sub array.
`abstract Array<T>`
`select(int[] indices)`
Slice out the specified indices and return the sub array.
`abstract void`
`set(int index, double value)`
Set index to given double value (cast to the correct type of this array)
`final void`
`set(int rl, int ru, Array<T> value)`
Set range to given arrays value
`void`
`set(int rl, int ru, Array<T> value, int rlSrc)`
Set range to given arrays value with an offset into other array
`abstract void`
`set(int index, String value)`
Set index to the given value of the string parsed.
`abstract void`
`set(int index, T value)`
Set index to the given value of same type
`final void`
`setCache(SoftReference<Map<T,Integer>> m)`
Set the cached hashmap cache of this Array allocation, to be used in transformEncode.
`abstract void`
`setFromOtherType(int rl, int ru, Array<?> value)`
Set range to given arrays value
`abstract void`
`setFromOtherTypeNz(int rl, int ru, Array<?> value)`
Set non default values in the range from the value array given
`final void`
`setFromOtherTypeNz(Array<?> value)`
Set non default values from the value array given
`void`
`setM(HashMapToInt<T> map, int si, AMapToData m, int i)`
Set the index i in the map given based on the mapping provided.
`void`
`setM(HashMapToInt<T> map, AMapToData m, int i)`
Set the index i in the map given based on the mapping provided.
`abstract void`
`setNz(int rl, int ru, Array<T> value)`
Set non default values in the range from the value array given
`final void`
`setNz(Array<T> value)`
Set non default values from the value array given
`final int`
`size()`
Get the number of elements in the array, this does not necessarily reflect the current allocated size.
`abstract Array<T>`
`slice(int rl, int ru)`
Slice out the sub range and return new array with the specified type.
`ArrayCompressionStatistics`
`statistics(int nSamples)`
Get the compression statistics of this array allocation.
`String`
`toString()`

### Methods inherited from class java.lang.Object

`getClass, hashCode, notify, notifyAll, wait, wait, wait`

### Methods inherited from interface org.apache.hadoop.io.Writable

`readFields, write`

## Field Details

### ROW_PARALLELIZATION_THRESHOLD

public static int ROW_PARALLELIZATION_THRESHOLD
Parallelization threshold for parallelizing vector operations

## Method Details

### getCache

public final SoftReference<Map<T,Integer>> getCache()
Get the current cached recode map.

Returns:
  The cached recode map

### setCache

public final void setCache(SoftReference<Map<T,Integer>> m)
Set the cached hashmap cache of this Array allocation, to be used in transformEncode.

Parameters:
  `m` - The element to cache.

### getRecodeMap

public final Map<T,Integer> getRecodeMap()
Get a recode map that maps each unique value in the array, to a long ID. Null values are ignored, and not included in the mapping. The resulting recode map in stored in a soft reference to speed up repeated calls to the same column.

Returns:
  A recode map

### getRecodeMap

public final Map<T,Integer> getRecodeMap(int estimate)
Get a recode map that maps each unique value in the array, to a long ID. Null values are ignored, and not included in the mapping. The resulting recode map in stored in a soft reference to speed up repeated calls to the same column.

Parameters:
  `estimate` - The estimated number of unique values in this Array to start the initial hashmap size at
Returns:
  A recode map

### getRecodeMap

public final Map<T,Integer> getRecodeMap(int estimate, ExecutorService pool, int k) throws InterruptedException, ExecutionException
Get a recode map that maps each unique value in the array, to a long ID. Null values are ignored, and not included in the mapping. The resulting recode map in stored in a soft reference to speed up repeated calls to the same column.

Parameters:
  `estimate` - the estimated number of unique values in this array.
  `pool` - An executor pool to be used for parallel execution (Note this method does not shutdown the pool)
  `k` - Parallelization degree allowed
Returns:
  A recode map
Throws:
  `ExecutionException` - if the parallel execution fails
  `InterruptedException` - if the parallel execution fails

### size

public final int size()
Get the number of elements in the array, this does not necessarily reflect the current allocated size.

Returns:
  the current number of elements

### get

public abstract T get(int index)
Get the value at a given index. This method returns objects that have a high overhead in allocation. Therefore it is not as efficient as using the vectorized operations specified in the object.

Parameters:
  `index` - The index to query
Returns:
  The value returned as an object

### getInternal

public T getInternal(int index)
Get the internal value at a given index. For instance HashIntegerArray would return the underlying long not a string.

Parameters:
  `index` - the index to get
Returns:
  The value to get

### get

public abstract Object get()
Get the underlying array out of the column Group, it is the responsibility of the caller to know what type it is. Also it is not guaranteed that the underlying data structure does not allocate an appropriate response to the caller. This in practice means that if called there is a possibility that the entire array is allocated again. So the method should only be used for debugging purposes not for performance.

Returns:
  The underlying array.

### getAsDouble

public abstract double getAsDouble(int i)
Get the index's value. returns 0 in case of Null.

Parameters:
  `i` - index to get value from
Returns:
  the value

### getAsNaNDouble

public double getAsNaNDouble(int i)
Get the index's value. returns Double.NaN in case of Null.

Parameters:
  `i` - index to get value from
Returns:
  the value

### set

public abstract void set(int index, T value)
Set index to the given value of same type

Parameters:
  `index` - The index to set
  `value` - The value to assign

### set

public abstract void set(int index, double value)
Set index to given double value (cast to the correct type of this array)

Parameters:
  `index` - the index to set
  `value` - the value to set it to (before casting to correct value type)

### set

public abstract void set(int index, String value)
Set index to the given value of the string parsed.

Parameters:
  `index` - The index to set
  `value` - The value to assign

### setFromOtherType

public abstract void setFromOtherType(int rl, int ru, Array<?> value)
Set range to given arrays value

Parameters:
  `rl` - row lower
  `ru` - row upper (inclusive)
  `value` - value array to take values from (other type)

### set

public final void set(int rl, int ru, Array<T> value)
Set range to given arrays value

Parameters:
  `rl` - row lower
  `ru` - row upper (inclusive)
  `value` - value array to take values from (same type) offset by rl.

### set

public void set(int rl, int ru, Array<T> value, int rlSrc)
Set range to given arrays value with an offset into other array

Parameters:
  `rl` - row lower
  `ru` - row upper (inclusive)
  `value` - value array to take values from
  `rlSrc` - the offset into the value array to take values from

### setNz

public final void setNz(Array<T> value)
Set non default values from the value array given

Parameters:
  `value` - array of same type and length

### setNz

public abstract void setNz(int rl, int ru, Array<T> value)
Set non default values in the range from the value array given

Parameters:
  `rl` - row start
  `ru` - row upper inclusive
  `value` - value array of same type

### setFromOtherTypeNz

public final void setFromOtherTypeNz(Array<?> value)
Set non default values from the value array given

Parameters:
  `value` - array of other type

### setFromOtherTypeNz

public abstract void setFromOtherTypeNz(int rl, int ru, Array<?> value)
Set non default values in the range from the value array given

Parameters:
  `rl` - row start
  `ru` - row end inclusive
  `value` - value array of different type

### append

public abstract void append(String value)
Append a string value to the current Array, this should in general be avoided, and appending larger blocks at a time should be preferred.

Parameters:
  `value` - The value to append

### append

public abstract void append(T value)
Append a value of the same type of the Array. This should in general be avoided, and appending larger blocks at a time should be preferred.

Parameters:
  `value` - The value to append

### append

public abstract Array<T> append(Array<T> other)
Append other array, if the other array is fitting in current allocated size use that allocated size, otherwise allocate new array to combine the other with this. This method should use the set range function, and should be preferred over the append single values.

Parameters:
  `other` - The other array of same type to append to this.
Returns:
  The combined arrays.

### slice

public abstract Array<T> slice(int rl, int ru)
Slice out the sub range and return new array with the specified type. If the conversion fails fallback to normal slice.

Parameters:
  `rl` - row start
  `ru` - row end (not included)
Returns:
  A new array of sub range.

### reset

public abstract void reset(int size)
Reset the Array and set to a different size. This method is used to reuse an already allocated Array, without extra allocation. It should only be done in cases where the Array is no longer in use in any FrameBlocks.

Parameters:
  `size` - The size to reallocate into.

### getAsByteArray

public abstract byte[] getAsByteArray()
Return the current allocated Array as a byte[], this is used to serialize the allocated Arrays out to the PythonAPI.

Returns:
  The array as bytes

### getValueType

public abstract Types.ValueType getValueType()
Get the current value type of this array.

Returns:
  The current value type.

### analyzeValueType

public final Pair<Types.ValueType,Boolean> analyzeValueType()
Analyze the column to figure out if the value type can be refined to a better type. The return is in two parts, first the type it can be, second if it contains nulls.

Returns:
  A better or equivalent value type to represent the column, including null information.

### analyzeValueType

public abstract Pair<Types.ValueType,Boolean> analyzeValueType(int maxCells)
Analyze the column to figure out if the value type can be refined to a better type. The return is in two parts, first the type it can be, second if it contains nulls.

Parameters:
  `maxCells` - maximum number of cells to analyze
Returns:
  A better or equivalent value type to represent the column, including null information.

### getFrameArrayType

public abstract ArrayFactory.FrameArrayType getFrameArrayType()
Get the internal FrameArrayType, to specify the encoding of the Types, note there are more Frame Array Types than there is ValueTypes.

Returns:
  The FrameArrayType

### getInMemorySize

public long getInMemorySize()
Get in memory size, not counting reference to this object.

Returns:
  the size in memory of this object.

### baseMemoryCost

public static long baseMemoryCost()
Get the base memory cost of the Arrays allocation.

Returns:
  The base memory cost

### getExactSerializedSize

public abstract long getExactSerializedSize()
Get the exact serialized size on disk of this array.

Returns:
  The exact size on disk

### getNulls

public ABooleanArray getNulls()

### containsNull

public boolean containsNull()
analyze if the array contains null values.

Returns:
  If the array contains null.

### possiblyContainsNaN

public abstract boolean possiblyContainsNaN()

### changeType

public Array<?> changeType(Types.ValueType t, boolean containsNull)
Change type taking into consideration if the target type must be able to contain Null.

Parameters:
  `t` - The target type
  `containsNull` - If the target should be able to contain null
Returns:
  The changed type array.

### changeTypeWithNulls

public Array<?> changeTypeWithNulls(Types.ValueType t)

### changeTypeWithNulls

public final Array<?> changeTypeWithNulls(Array<?> ret)

### changeTypeWithNulls

public final Array<?> changeTypeWithNulls(Array<?> ret, int l, int u)

### changeType

public Array<?> changeType(Types.ValueType t)
Change the allocated array to a different type. If the type is the same a deep copy is returned for safety.

Parameters:
  `t` - The type to change to
Returns:
  A new column array.

### changeType

public final Array<?> changeType(Array<?> ret)
Change type by moving this arrays value into the given ret array.

Parameters:
  `ret` - The Array to put this arrays values into
Returns:
  The ret array given

### changeType

public final Array<?> changeType(Array<?> ret, int rl, int ru)
Put the changed value types into the given ret array inside the range specified.

Parameters:
  `ret` - The Array to put this arrays values into
  `rl` - inclusive lower bound
  `ru` - exclusive upper bound
Returns:
  The ret array given.

### getMinMaxLength

public Pair<Integer,Integer> getMinMaxLength()
Get the minimum and maximum length of the contained values as string type.

Returns:
  A Pair of first the minimum length, second the maximum length

### fill

public abstract void fill(String val)
fill the entire array with specific value.

Parameters:
  `val` - the value to fill with.

### fill

public abstract void fill(T val)
fill the entire array with specific value.

Parameters:
  `val` - the value to fill with.

### isShallowSerialize

public abstract boolean isShallowSerialize()
analyze if this array can be shallow serialized. to allow caching without modification.

Returns:
  boolean saying true if shallow serialization is available

### isEmpty

public abstract boolean isEmpty()
Get if this array is empty, aka filled with empty values.

Returns:
  boolean saying true if empty

### select

public abstract Array<T> select(int[] indices)
Slice out the specified indices and return the sub array.

Parameters:
  `indices` - The indices to slice out
Returns:
  the sliced out indices in an array format

### select

public abstract Array<T> select(boolean[] select, int nTrue)
Slice out the true indices in the select input and return the sub array.

Parameters:
  `select` - a boolean vector specifying what to select
  `nTrue` - number of true values inside select
Returns:
  the sliced out indices in an array format

### findEmpty

public final void findEmpty(boolean[] select)
Find the empty rows, it is assumed that the input is to be only modified to set variables to true.

Parameters:
  `select` - Modify this to true in indexes that are not empty.

### isNotEmpty

public abstract boolean isNotEmpty(int i)

### findEmptyInverse

public void findEmptyInverse(boolean[] select)
Find the filled rows, it is assumed that the input i to be only modified to set variables to true;

Parameters:
  `select` - modify this to true in indexes that are empty.

### clone

public abstract Array<T> clone()
Overwrite of the java internal clone function for arrays, return a clone of underlying data that is mutable, (not immutable data.) Immutable data is dependent on the individual allocated arrays

Returns:
  A clone

### toString

public String toString()

Overrides:
  `toString` in class `Object`

### hashDouble

public abstract double hashDouble(int idx)
Hash the given index of the array. It is allowed to return NaN on null elements.

Parameters:
  `idx` - The index to hash
Returns:
  The hash value of that index.

### getIterator

public Array<T>.ArrayIterator getIterator()

### equals

public boolean equals(Object other)

Overrides:
  `equals` in class `Object`

### extractDouble

public double[] extractDouble(double[] ret, int rl, int ru)
Extract the sub array into the ret array as doubles. The ret array is filled from - rl, meaning that the ret array should be of length ru - rl.

Parameters:
  `ret` - The array to return
  `rl` - The row to start at
  `ru` - The row to end at (not inclusive.)
Returns:
  The ret array given as argument

### equals

public abstract boolean equals(Array<T> other)
Equals operation on arrays.

Parameters:
  `other` - The other array to compare to.
Returns:
  True if the arrays are equivalent.

### statistics

public ArrayCompressionStatistics statistics(int nSamples)
Get the compression statistics of this array allocation.

Parameters:
  `nSamples` - The number of sample elements suggested (not forced) to be used.
Returns:
  The compression statistics of this array.

### minMax

public double[] minMax()
Get the minimum and maximum double value of this array. Note that we ignore NaN Values.

Returns:
  The min and max in index 0 and 1 of the array.

### minMax

public double[] minMax(int l, int u)
Get the minimum and maximum double value of a specific sub part of this array. Note that we ignore NaN Values.

Parameters:
  `l` - The lower index to search from
  `u` - The upper index to end at (not inclusive)
Returns:
  The min and max in index 0 and 1 of the array in the range.

### setM

public void setM(HashMapToInt<T> map, AMapToData m, int i)
Set the index i in the map given based on the mapping provided. The map should be guaranteed to contain all unique values.

Parameters:
  `map` - A map containing all unique values of this array
  `m` - The MapToData to set the value part of the Map from
  `i` - The index to set in m

### setM

public void setM(HashMapToInt<T> map, int si, AMapToData m, int i)
Set the index i in the map given based on the mapping provided. The map should be guaranteed to contain all unique values except null. Therefore in case of null we set the provided si value.

Parameters:
  `map` - A map containing all unique values of this array
  `si` - The default value to use in m if this Array contains null at index i
  `m` - The MapToData to set the value part of the Map from
  `i` - The index to set in m