Where Inference Happens

Edge AI usually means inference happens close to where data is produced or where action is required: on the device itself or on nearby local compute.

Cloud inference sends data or a derived representation to remote infrastructure where the model runs.

A hybrid design can divide responsibilities between both.

These are architectural placements, not model types. The same model family might be deployed differently depending on hardware and system requirements.

Placement changes the trust and failure boundary

When inference is local:

• raw data may remain on the device;
• network loss may not stop inference;
• compute and memory are limited by local hardware;
• updates must reach many devices safely.

When inference is remote:

• more compute may be available;
• model updates can be centralized;
• network latency and availability enter every request;
• transmitted data may create additional privacy and security exposure.

A camera system that must react within 40 ms cannot simply ignore a 100 ms network round trip because the cloud GPU is faster. End-to-end latency includes transport, queueing, preprocessing, inference, and the return path.

Likewise, keeping inference local does not automatically make a system private or secure. Local storage, logs, physical access, and update mechanisms still matter.

Start with requirements, not fashion

Before choosing placement, document:

• maximum acceptable latency;
• offline requirements;
• model memory/compute needs;
• data sensitivity;
• power budget;
• update frequency;
• expected network quality;
• failure and fallback behavior.

Before moving on

Name one application where local inference is attractive and one where cloud inference is attractive. For each, state the constraint that drives the choice.