Embedded AI
from model to edge system.

Define the edge workload

Start with the decision the device must make: a classification, detection, anomaly flag, estimate or control input. Specify the sensor, input rate, acceptable response time, accuracy criteria and operating environment. A model that performs well on a development workstation may not fit the memory, compute or power envelope of the target. The engineering plan needs a representative workload and measurable acceptance criteria before accelerator or model choices are fixed.

Partition the system between sensing, preprocessing, inference and application logic. Determine which functions run on the device and which may use a network service. Offline behaviour, communication failures and recovery paths are part of the system requirement, especially when the output affects a physical process.

Data and evaluation

Training and evaluation data should reflect deployment conditions, including lighting, noise, device variation and uncommon operating states. Keep evaluation data separate from the training process and agree which metrics matter to the application. False positives, false negatives and latency tails can be more meaningful than a single accuracy figure. Data rights and retention need to be established before collecting or transferring samples.

Document the model version, dataset revision, preprocessing parameters and evaluation method together. This makes results comparable across hardware and software changes. A small demonstration dataset can support exploration, but it does not establish production performance.

Optimise against measured hardware limits

Profile the complete inference path on the intended target. Measure preprocessing, transfers, inference and postprocessing separately to find the actual bottleneck. Quantisation and model simplification can reduce memory or latency, but their impact must be checked against the application metrics. Unsupported operators and tensor-layout conversions can erase the apparent benefit of an accelerator.

Memory allocation, buffer reuse, thermal behaviour and sustained throughput matter alongside a best-case benchmark. Record peak and steady-state behaviour, including cold start and recovery. The deliverable should explain the tested configuration so that the client can reproduce the measurements.

Integrate, release and maintain

The deployment package can include the versioned model, preprocessing code, runtime configuration, interfaces, test vectors and benchmark results. Integration tests should exercise invalid inputs, sensor failures, resource exhaustion and fallback behaviour. Model outputs need to be interpreted by application logic that respects the product safety and reliability requirements.

Agree how updates are authenticated, rolled out, monitored and rolled back. Changes to sensors, preprocessing or model weights require regression evaluation. For connected products, observability must be balanced with privacy and bandwidth constraints. Acceptance is based on the agreed device-level evidence; it is not a promise that every future input will produce a correct result.

Measure AI quality on the target system

An AI work package should begin with the product decision the model supports and the cost of an incorrect result. Define the evaluation dataset, representative operating conditions and the metrics that matter to the application. Keep training and evaluation data separate. Record provenance and permitted use of datasets, weights and third-party components before integrating them into a deliverable.

For deployment, inspect the entire inference path: data acquisition, preprocessing, model execution and postprocessing. Model size and peak accelerator throughput alone do not describe application performance. Measure memory peaks, transfer costs, latency distribution and sustained power on the intended device. Quantisation or operator replacement should be evaluated for accuracy as well as speed.

Plan for inputs outside the model’s useful operating range and for unavailable dependencies. Establish whether the device must operate offline, how it reports uncertainty or errors and what fallback is permitted. Version the model together with preprocessing and runtime configuration. Updates need a comparison against the established baseline and a recovery strategy suitable for the product, rather than only a demonstration on selected examples.