Latency, Memory, Privacy, and Power Budgets

Deployment decisions become easier to defend when vague preferences are replaced with budgets.

A latency budget sets the maximum time available from input to required output.

A memory budget limits how much model and runtime state the target device can hold.

A power budget matters when compute competes with battery life, heat, or other device functions.

A privacy boundary identifies what data may leave the device and under which conditions.

Model quality is only one constraint

A larger model may improve one evaluation metric while exceeding device memory or latency requirements. A cloud model may be more capable while becoming unusable during disconnection.

Suppose two models are available:

If the device must operate offline and has 256 MB available for the entire application, Model B is not a feasible local deployment simply because its test score is higher.

Engineering quality includes satisfying the system constraints.

Privacy should be described as data movement

Avoid statements like “edge is private” or “cloud is unsafe.” Ask what actually crosses boundaries.

Document:

• raw input sent or retained;
• intermediate features transmitted;
• logs retained;
• identifiers attached;
• encryption and access controls;
• retention period;
• whether the application can function without transfer.

Before moving on

Create a four-line budget for a fictional AI application: latency, memory, connectivity, and privacy. Then identify which budget is most likely to eliminate one deployment option immediately.