1. Attribute

1.1. TIFStatus

This section describes the TopsInference Status code.

enum TIFStatus

Status code.

Values:

enumerator TIF_SUCCESS

Success.

enumerator TIF_ENGINE_RUN_FAILED

Engine run inference failed.

enumerator TIF_PARSER_UNSUPPORTED_MODEL

Model is not supported by current parser.

enumerator TIF_OPTIMIZER_BUILD_FAILED

Optimizer build failed.

enumerator TIF_NOT_INITIALIZED

Uninitialized error.

enumerator TIF_INTERNAL_ERROR

Internal error.

enumerator TIF_ALLOCATE_FAILED

Allocate buffer failed.

enumerator TIF_INVALID_PARAM

Invalid parameters.

enumerator TIF_INVALID_VALUE

These value maybe is nullptr or zero.

enumerator TIF_MIS_MATCHED_DATA_TYPE

Mismatched data type.

enumerator TIF_MIS_MATCHED_SHAPE_OR_DIM

Mismatched shape or dimension.

enumerator TIF_MAPPING_ERROR

Mapping error.

enumerator TIF_UN_SUPPORTED_DATA_TYPE

Un-supported datatype.

enumerator TIF_UN_SUPPORTED_OPERATOR

Un-supported operator.

enumerator TIF_LICENSE_ERROR

License error.

enumerator TIF_RUNTIME_PREREQUISITE_MISSING

Miss prerequisite in runtime.

enumerator TIF_RUNTIME_FP_OVERFLOW

Has occur overflow.

enumerator TIF_VALIUE_OVERFLOW

Some value is out of limit.

enumerator TIF_NOT_FOUND

Has some keys or names not find in lists.

enumerator TIF_ALREADY_EXISTED_TENSOR

Tensor has already existed.

enumerator TIF_ALREADY_EXISTED_OP

Operater has already existed.

enumerator TIF_COMPILE_QUANTIZE_FAILED

Graph quantize failed.

enumerator TIF_COMPILE_RUN_PASS_FAILED

Run graph pass failed.

enumerator TIF_COMPILE_GRAPH_ERROR

Graph Error, eg.cycle Graph.

enumerator TIF_COMPILE_PARSE_FAILED

Unsupported model structure.

enumerator TIF_COMPILE_CONVERT_FAILED

Convert fail.

enumerator TIF_LOAD_NON_EXIST_FILE

Load File not exist.

enumerator TIF_SAVE_NON_EXIST_FILE

Save File not exist.

enumerator TIF_LOAD_NON_EXIST_PB_FILE

Load PB file not exist.

enumerator TIF_EXECUTABLE_CRC_VERIFY_FAILED

Executable CRC verify failed.

enumerator TIF_EXECUTABLE_DEVICE_VERIFY_FAILED

Executable device verify failed.

enumerator TIF_MORE_CLUSTER_APPLIED

Too many clusters are applied without auto serving.

enumerator TIF_TERMINATED

Terminate.

enumerator TIF_EXECUTABLE_WEIGHT_REFIT_FAILED

Executable weight refit failed.

enumerator TIF_FINISHED

Finish.

enumerator TIF_SHAPE_INFER_FAILED

dynamic shape mode, An error occurred during the inference process eg: op shape execute coredump.

enumerator TIF_SHAPE_INFER_INACCURATE

dynamic shape mode, The inference can be fully executed, but the output shape result calculated by the shape inference is not accurate.

1.2. Type

This section describes the TopsInference Type information.

1.2.1. BuildFlag

This section describes the TopsInference Type BuildFlag.

enum BuildFlag

The flags used in Optimizer when building.

Values:

enumerator TIF_KTYPE_DEFAULT

If the flag was set, then using the building precision inside for each layer, fp32 in general.

enumerator TIF_KTYPE_FLOAT16

Using fp16 for all layers.

enumerator TIF_KTYPE_MIX_FP16

Using f16 for each layer except that some layers which are sensitive to precision will use f32.

enumerator TIF_KTYPE_INT8_MIX_FP32

Using int8 for each layer except that some layers which are sensitive to precision will use f32.

enumerator TIF_KTYPE_INT8_MIX_FP16

Using int8 for each layer except that some layers which are sensitive to precision will use f16, unsupported.

enumerator TIF_KTYPE_UINT8_MIX_FP32

Using uint8 for each layer except that some layers which are sensitive to precision will use f32, unsupported.

enumerator TIF_KTYPE_UINT8_MIX_FP16

Using uint8 for each layer except that some layers which are sensitive to precision will use f16, unsupported.

enumerator TIF_REFIT

Refitting an engine with new weights without having to rebuild it.

1.2.2. DataType

This section describes the TopsInference DataType.

enum DataType

Data type definition for ITensor.

Values:

enumerator TIF_BOOL

bool.

enumerator TIF_INDEX

index.

enumerator TIF_INT8

int8.

enumerator TIF_INT16

int16.

enumerator TIF_INT32

int32.

enumerator TIF_INT64

int64.

enumerator TIF_UINT8

uint8.

enumerator TIF_UINT16

uint16.

enumerator TIF_UINT32

uint32.

enumerator TIF_UINT64

uint64.

enumerator TIF_FP16

fp16.

enumerator TIF_FP32

fp32.

enumerator TIF_BF16

bf16.

enumerator TIF_TUPLE

tuple.

enumerator TIF_INVALID

invalid.

1.2.3. DataDeviceType

This section describes the TopsInference DataDeviceType.

enum class DataDeviceType : int32_t

The buffer type of the tensor.

Values:

enumerator HOST

On HOST memory.

enumerator DEVICE

On GCU device memory.

1.2.4. ParserType

This section describes the TopsInference ParserType.

enum ParserType

Parsed model type definition.

Values:

enumerator TIF_ONNX

ONNX Model.

1.2.5. MemcpyKind

This section describes the TopsInference MemcpyKind.

enum MemcpyKind

Memory copy kind.

Values:

enumerator TIF_MEMCPY_HOST_TO_DEVICE

Copy data from host to GCU device.

enumerator TIF_MEMCPY_DEVICE_TO_HOST

Copy data from GCU device to host.

1.2.6. BufferType

This section describes the TopsInference BufferType.

enum BufferType

Buffer type definition used in engine inference.

Now, mixed buffer type is not supported, e.g. IN_HOST_OUT_DEVICE or IN_DEVICE_OUT_HOST.

Values:

enumerator TIF_ENGINE_RSC_IN_HOST_OUT_HOST

The input buffer and output buffer is all allocated on host.

enumerator TIF_ENGINE_RSC_IN_DEVICE_OUT_DEVICE

The input buffer and output buffer is all allocated on device.

class INoCopy

Subclassed by TopsInference::IHostMemory, TopsInferenceV2::IExecutionContext

1.2.7. CalibrationAlgoType

This section describes the TopsInference CalibrationAlgoType.

enum class CalibrationAlgoType : int32_t

calibration algorithm to use.

enum CalibrationAlgoType

Values:

enumerator INVALID

Invalid.

enumerator KL_ENTROPY

Kullback-Leibler Divergence Entropy / Relative Entropy.

enumerator MAX_MIN

Max Min.

enumerator MAX_MIN_EMA

Max Min EMA.

enumerator PERCENTILE

Percentile.

template<>
inline constexpr int32_t EnumMax<CalibrationAlgoType>() noexcept

Maximum number of elements in CalibrationAlgoType enum.

Returns

in32_t.

1.2.8. LayerType

This section describes the TopsInference LayerType.

enum LayerType

Layer type definition used in ILayer.

Values:

enumerator TIF_DECONVOLUTION

DeConvolution layer.

enumerator TIF_CONVOLUTION

Convolution layer.

enumerator TIF_UNARY

UnaryOp operation layer.

enumerator TIF_TRANSCENDENTAL

Transcendental layer.

enumerator TIF_ELEMENTWISE

Elementwise operation layer.

enumerator TIF_SELECT

Select layer.

enumerator TIF_POOLING

Pooling layer.

enumerator TIF_BATCHNORM

Batch normalization layer.

enumerator TIF_CONVERT

Convert layer for converting between different data precision.

enumerator TIF_CONCAT

Concat layer.

enumerator TIF_CONSTANT

Constant layer.

enumerator TIF_SHUFFLE

Shuffle layer.

enumerator TIF_ACTIVATION

Activation layer.

enumerator TIF_ORDER

Layer for sorting by a certain rule.

enumerator TIF_RNN

Rnn layer.

enumerator TIF_GATHER

Gather layer.

enumerator TIF_MATMUL

MatMul layer.

enumerator TIF_COMPARE

Compare layer.

enumerator TIF_CONDITION

Condition layer.

enumerator TIF_NMS

Non max suppression layer.

enumerator TIF_PAD

Padding layer.

enumerator TIF_RANDOM

Random generator layer.

enumerator TIF_REDUCE

Reduce layer.

enumerator TIF_RESHAPE

Reshape layer.

enumerator TIF_RESIZE

Resize layer.

enumerator TIF_ROIALIGN

ROI align layer.

enumerator TIF_SCATTER

Scatter layer.

enumerator TIF_SIGMOID

Sigmoid layer.

enumerator TIF_SLICE

Slice layer.

enumerator TIF_TOPK

TopK layer.

enumerator TIF_TRANSPOSE

Transpose layer.

enumerator TIF_MVN

Mean-variance normalization layer.

enumerator TIF_SOFTMAX

Softmax layer.

enumerator TIF_LOG_SOFTMAX

Log softmax layer.

enumerator TIF_SAMPLE

Sample layer.

enumerator TIF_DEQUANTIZE

Dequantize layer.

enumerator TIF_CUMSUM

Cumsum layer.

enumerator TIF_DFT

DTF(discrete Fourier transform) layer.

enumerator TIF_DET

DET layer.

enumerator TIF_EINSUM

einsum layer.

enumerator TIF_LOOP

loop layer.

enumerator TIF_MELWEIGHT

mel weight matrix layer.

enumerator TIF_OPTIONAL

optinal serise op layer.

enumerator TIF_QUANTIZE

Quantize layer.

enumerator TIF_SEQUENCE

sequence serise op layer.

enumerator TIF_STFT

STFT(Short-time Fourier Transform) layer.

enumerator TIF_TFIDF

TFIDF layer.

enumerator TIF_TRILU

TRILU(triangular) layer.

enumerator TIF_STRINGNORM

STRINGNORM layer.

enumerator TIF_EXPAND

Expand layer.

enumerator TIF_CUSTOM

Customcall layer.

enumerator TIF_NORMLIZE

Normlization layer.

enumerator TIF_MAP

map layer.

enumerator TIF_TILE

tile layer.

enumerator TIF_DROPOUT

dropout layer.

enumerator TIF_STRUCTURAL

structural layer.

1.2.9. Weights

This section describes the TopsInference Weights.

struct TopsInference::Weights

#include <TopsInferRuntime.h>

An array of weights used as a layer parameter. The weights are held by reference until the engine has been built. Therefore the data referenced by values field should be preserved until the build is complete.

Public Members

DataType type

The data type of the weights.

int64_t count

The number of weights in the array.

const void *values

The weight values.

Dims dims

The weights dims.

1.3. VersionInfo

This section describes the TopsInference version information.

struct VersionInfo

#include <TopsInferRuntime.h>

TopsInference version information.