Embedded machine learning

The engineering objective

Fit model inference into a constrained software platform.

The work starts from model graph, datasets, target memory and latency budget. The team should identify the authoritative revision of each input and record any assumptions needed to proceed. This prevents a later result from being interpreted against a different specification or configuration.

What needs technical attention

A smaller model is only useful if it preserves the application's acceptance metrics.

The review centres on accuracy change, memory peaks, operator support and latency tails. Each finding should identify the input revision, observed behaviour and supporting evidence. Distinguish a confirmed defect from a configuration issue, an unsupported case or an open engineering question before deciding how to resolve it.

How the work is reviewed

The practical sequence is: profile operators; assess quantisation; integrate runtime; compare target results. Intermediate reviews allow the client to resolve decisions while changes are still manageable. If an input changes, identify the affected activities and repeat the checks that depend on it.

The expected package comprises model package, runtime integration and benchmark report. Acceptance criteria should specify what the receiving team needs to reproduce, inspect or integrate these artifacts. Include known limitations and outstanding actions in that package, rather than assuming that delivery of a file implies acceptance.

Measure AI quality on the target system

An AI work package should begin with the product decision the model supports and the cost of an incorrect result. Define the evaluation dataset, representative operating conditions and the metrics that matter to the application. Keep training and evaluation data separate. Record provenance and permitted use of datasets, weights and third-party components before integrating them into a deliverable.

For deployment, inspect the entire inference path: data acquisition, preprocessing, model execution and postprocessing. Model size and peak accelerator throughput alone do not describe application performance. Measure memory peaks, transfer costs, latency distribution and sustained power on the intended device. Quantisation or operator replacement should be evaluated for accuracy as well as speed.

Plan for inputs outside the model’s useful operating range and for unavailable dependencies. Establish whether the device must operate offline, how it reports uncertainty or errors and what fallback is permitted. Version the model together with preprocessing and runtime configuration. Updates need a comparison against the established baseline and a recovery strategy suitable for the product, rather than only a demonstration on selected examples.

An illustrative review scenario

A vision model works on stored test images but produces different results after camera integration. Compare colour order, resizing, normalisation and sensor timing before retraining. Measure the full camera-to-output delay and include difficult lighting or motion conditions from the agreed use case. This is an evaluation scenario, not an accuracy or safety guarantee.