3. Function¶

3.1. init_and_finish¶

This section describes the TopsInference init and finish functions.

3.1.1. TopsInference_init¶

This section describes the TopsInference topsInference_init funtion.

TOPS_INFERENCE_EXPORT TIFStatus topsInference_init()¶

sdk init function only call once at the beginning of the process.

Returns: TIFStatus.

3.1.2. TopsInference_finish¶

This section describes the TopsInference topsInference_finish funtion.

TOPS_INFERENCE_EXPORT TIFStatus topsInference_finish()¶

sdk finish function. only call once at the end of the process.

Returns: TIFStatus.

3.2. device¶

This section describes the TopsInference device functions.

typedef void *handler_t¶

3.2.1. Set_device¶

This section describes the TopsInference set_device funtion.

TOPS_INFERENCE_EXPORT handler_t set_device(uint32_t card_id, const uint32_t *cluster_ids, uint32_t cluster_ids_size = 1, IErrorManager *error_manager = nullptr)¶

specify the running device until release. set_device are isolated from each other under multi-process.

Under multi-thread condition, each sub-thread will exclusively utilize the claimed resource if set_device is called within the sub-thread.
if set_device() is called in main thread, not called in sub thread, the sub thread will share the cluster resources claimed by main thread.
if both main thread and sub thread claimed resource with set_device(), the resource claimed by sub thread is used within sub thread.
if some sub threads claims resource with set_device(), some does not, each sub thread individually follow the above rule 3 and 2 based on its resource claiming status.

Parameters

card_id – Specify the current card id.
cluster_ids – Specify the current clusters ids.
cluster_ids_size – Specify the current cluster num.

Returns

handler_t Specify the handler.

3.2.2. Release_device¶

This section describes the TopsInference release_device funtion.

TOPS_INFERENCE_EXPORT bool release_device(handler_t)¶

free the running device.

Parameters

handler_t – use set_device() handler_t.

Returns

bool.

Returns

true – Return true if succeed.
false – Return false if fail.

3.3. error_manager¶

This section describes the TopsInference error_manager functions.

3.3.1. Create_error_manager¶

This section describes the TopsInference create_error_manager funtion.

TOPS_INFERENCE_EXPORT IErrorManager *create_error_manager()¶: Create an instance of IErrorManager.

3.3.2. Release_error_manager¶

This section describes the TopsInference release_error_manager funtion.

TOPS_INFERENCE_EXPORT bool release_error_manager(IErrorManager *error_manager)¶

Release ErrorManager object.

Parameters

error_manager – pointer of IErrorManager.

Returns

bool.

Returns

true – Return true if succeed.
false – Return false if fail.

3.4. parser¶

This section describes the TopsInference parser functions.

3.4.1. Create_parser¶

This section describes the TopsInference create_parser funtion.

TOPS_INFERENCE_EXPORT IParser *create_parser(ParserType parse_type, IErrorManager *error_manager = nullptr)¶

Create an instance of IParser with specified parse_type.

Note

Examples:

TopsInference::IParser* parser = TopsInference::create_parser(
    TopsInference::TIF_ONNX);

Parameters: parse_type – use TIF_ONNX to parse onnx model.
Returns: IParser* pointer of IParser, or nullptr if create IParser failure.

3.4.2. Release_parser¶

This section describes the TopsInference release_parser funtion.

TOPS_INFERENCE_EXPORT bool release_parser(IParser *parser)¶

Release Parser object.

Parameters

parser – pointer of IParser.

Returns

bool.

Returns

true – Return true if succeed.
false – Return false if fail.

3.5. optimizer¶

This section describes the TopsInference optimizer functions.

3.5.1. Create_optimizer¶

This section describes the TopsInference create_optimizer funtion.

TOPS_INFERENCE_EXPORT IOptimizer *create_optimizer(IErrorManager *error_manager = nullptr)¶

Create an instance of IOptimizer.

Note

Examples:

TopsInference::IOptimizer* optimizer = TopsInference::create_optimizer();

Returns: IOptimizer* pointer of IOptimizer, or nullptr if create IOptimizer failure.

3.5.2. Release_optimizer¶

This section describes the TopsInference release_optimizer funtion.

TOPS_INFERENCE_EXPORT bool release_optimizer(IOptimizer *optimizer)¶

Release Optimizer object.

Parameters

optimizer – pointer of IOptimizer.

Returns

bool.

Returns

true – Return true if succeed.
false – Return false if fail.

3.6. engine¶

This section describes the TopsInference engine functions.

3.6.1. Create_engine¶

This section describes the TopsInference create_engine funtion.

TOPS_INFERENCE_EXPORT IEngine *create_engine(IErrorManager *error_manager = nullptr)¶

Create an instance of IEngine.

Note

Examples:

TopsInference::IEngine* engine = TopsInference::create_engine();

Returns: IEngine* pointer of IEngine, or nullptr if create IEngine failure.

3.6.2. Release_engine¶

This section describes the TopsInference release_engine funtion.

TOPS_INFERENCE_EXPORT bool release_engine(IEngine *engine)¶

Release Engine object.

Parameters

engine – pointer of IEngine.

Returns

bool.

Returns

true – Return true if succeed.
false – Return false if fail.

3.7. network¶

This section describes the TopsInference network functions.

3.7.1. Release_network¶

This section describes the TopsInference release_network funtion.

TOPS_INFERENCE_EXPORT bool release_network(INetwork *network)¶

Release Network object.

Parameters

network – pointer of INetwork.

Returns

bool.

Returns

true – Return true if succeed.
false – Return false if fail.

3.8. stream¶

This section describes the TopsInference stream functions.

3.8.1. Create_stream¶

This section describes the TopsInference create_stream funtion.

TOPS_INFERENCE_EXPORT bool create_stream(topsInferStream_t *streamCtx)¶

Create an instance of Stream, cannot create more than 1000 streams.

Parameters: streamCtx – the double pointer of stream to be created.
Returns: bool whether the stream has been created.

3.8.2. Synchronize_stream¶

This section describes the TopsInference synchronize_stream funtion.

TOPS_INFERENCE_EXPORT bool synchronize_stream(topsInferStream_t stream)¶

Stream Synchronize, Wait for any pending asynchronous action in stream.

Parameters: stream – the pointer of stream to synchronize.

3.8.3. Destroy_stream¶

This section describes the TopsInference destroy_stream funtion.

TOPS_INFERENCE_EXPORT bool destroy_stream(topsInferStream_t stream)¶

Destroy stream object.

Parameters

stream – pointer of stream.

Returns

bool.

Returns

true – Return true if succeed.
false – Return false if fail.

3.9. future¶

This section describes the TopsInference future functions.

3.9.1. Create_future¶

This section describes the TopsInference create_future funtion.

TOPS_INFERENCE_EXPORT IFuture *create_future()¶: Create an instance of IFuture.

3.9.2. Destroy_future¶

This section describes the TopsInference destroy_future funtion.

TOPS_INFERENCE_EXPORT bool destroy_future(IFuture *future)¶

Destroy Future object.

Parameters

future – pointer of IFuture.

Returns

bool.

Returns

true – Return true if succeed.
false – Return false if fail.

3.10. tensor¶

This section describes the TopsInference tensor functions.

3.10.1. Create_tensor¶

This section describes the TopsInference create_tensor funtion.

TOPS_INFERENCE_EXPORT TensorPtr_t create_tensor()¶

Create an instance of ITensor.

Returns: TensorPtr_t pointer of ITensor, or nullptr if create tensor failure.

3.10.2. Destroy_tensor¶

This section describes the TopsInference destroy_tensor funtion.

TOPS_INFERENCE_EXPORT TIFStatus destroy_tensor(TensorPtr_t itensor)¶

Destroy tensor.

See also

Status code.

Parameters: itensor – pointer of a tensor object.
Returns: TIFStatus.

3.11. refit¶

This section describes the TopsInference refit functions.

3.11.1. Create_refitter¶

This section describes the TopsInference create_refitter funtion.

TOPS_INFERENCE_EXPORT IRefitter *create_refitter(IEngine *engine, IErrorManager *error_manager = nullptr)¶

Refitter Create.

Parameters: engine – pointer of IEngine.
Returns: IRefitter* pointer of IRefitter, or nullptr if create IRefitter failure.

3.11.2. Release_refitter¶

This section describes the TopsInference release_refitter funtion.

TOPS_INFERENCE_EXPORT bool release_refitter(IRefitter *refitter)¶

Refiter Release.

Parameters

refitter – pointer of IRefitter.

Returns

bool.

Returns

true – Return true if succeed.
false – Return false if fail.

3.12. memory¶

This section describes the TopsInference memory functions.

3.12.1. Mem_alloc¶

This section describes the TopsInference mem_alloc funtion.

TOPS_INFERENCE_EXPORT bool mem_alloc(void **ptr, int64_t size)¶

memory allocate.

Parameters

ptr – double pointer of GCU memory to allocate.
size – GCU memory size.

Returns

bool.

Returns

true – Return true if succeed.
false – Return false if fail.

3.12.2. Mem_free¶

This section describes the TopsInference mem_free funtion.

TOPS_INFERENCE_EXPORT bool mem_free(void *ptr)¶

Free GCU device memory.

Parameters

ptr – The allocated buffer address.

Returns

bool.

Returns

true – Return true if succeed.
false – Return false if fail.

3.12.3. Mem_copy¶

This section describes the TopsInference mem_copy funtion.

TOPS_INFERENCE_EXPORT bool mem_copy(void *src, void *dst, int64_t size, MemcpyKind kind)¶

Copy memory from GCU device memory to Host memory, or copy memory from Host memory to GCU device memory.

See also

MemcpyKind.

Parameters

src – The source buffer address.
dst – The destination buffer address.
size – The bytes copied from src to dst.
kind – The kind of copying.

Returns

bool.

Returns

true – Return true if succeed.
false – Return false if fail.

3.12.4. Mem_copy_async¶

This section describes the TopsInference mem_copy_async funtion.

TOPS_INFERENCE_EXPORT bool mem_copy_async(void *src, void *dst, int64_t size, MemcpyKind kind, topsInferStream_t stream)¶

Copy memory from GCU device memory to Host memory async, or copy memory from Host memory to GCU device memory async.

See also

MemcpyKind.

See also

topsInferStream_t.

Parameters

src – The source buffer address.
dst – The destination buffer address.
size – The bytes copied from src to dst.
kind – The kind of copying.
stream – The stream on which the mem copy runs.

Returns

bool.

Returns

true – Return true if succeed.
false – Return false if fail.

3.13. topsinference_version¶

This section describes the TopsInference topsinference_version functions.

3.13.1. version¶

This section describes the TopsInference version funtion.

TOPS_INFERENCE_EXPORT VersionInfo version()¶

TopsInference version information.

Returns: VersionInfo, TopsInference version.