A cheap agent can still generate high monitoring costs. Budget the telemetry you keep and the evaluations you run as two separate workloads, then compare each platform against the same traffic, evidence requirements, and dated billing units.
Collection determines what reaches telemetry storage. Evaluator selection determines which retained interactions receive additional assessment. The trace topology is illustrative.
For an AI agent observability cost comparison, hold the workload and evidence coverage constant. Record monthly runs, spans per run, billable bytes, collection and evaluation fractions, retention, queries, and reviewer hours. Map those inputs to each provider's meter before applying rates. The worksheet below includes an assumed example you can replace with your own measurements; it makes no dollar estimate or vendor ranking.
| Input or item | Value or source | Why it matters |
|---|---|---|
| Input or itemTraffic and payload | Value or source Your execution counts and exported telemetry | Why it matters Runs, spans, and bytes describe different quantities. |
| Input or itemEvidence policy | Value or source Your incident, privacy, and evaluation requirements | Why it matters A cheaper configuration may preserve less useful evidence. |
| Input or itemRetention and queries | Value or sourceRequired lookback and measured scans | Why it matters Storage accumulates; repeated reads add work. |
| Input or itemEvaluation and people | Value or source Judge calls, token usage, and review time | Why it matters These costs sit alongside telemetry collection. |
Compare the meters before comparing the plans
Use each provider's published unit and your selected plan, region, and retention class. A trace count cannot substitute for a gigabyte measurement. The following public pricing terms were checked on 18 October 2026.
| Setup | Meter to record | Dominant Failure Mode & Prevention |
|---|---|---|
| Setup Amazon Bedrock AgentCore | Meter to record CloudWatch telemetry ingestion, storage, queries, and log masking. Built-in evaluations use input/output tokens; custom evaluations add a per-evaluation charge and separate model inference. | Dominant Failure Mode & Prevention Counting only judge calls misses telemetry and custom-model usage. Keep separate rows; built-in evaluator model usage is included. |
| SetupLangSmith | Meter to record Trace volume, retention tier, plan allowances, and the plan's usage-unit conversion. A trace is one application execution and can contain multiple steps. | Dominant Failure Mode & Prevention Treating every span as a trace or pricing every trace as short-lived. Count executions and retention classes separately, then verify the plan conversion. |
| SetupOpenObserve Cloud | Meter to record GB ingested and GB queried, plus additional non-metrics retention. The listed included retention is 30 days for non-metrics and 15 months for metrics. | Dominant Failure Mode & Prevention Treating included retention as indefinite free storage. Cloud Enterprise BYOB storage is paid to the cloud provider; include that bill separately. |
LangSmith's current pricing page lists base traces at 14 days and extended traces at 180 days. Its retention documentation dates the SaaS long-lived maximum of 180 days to 14 September 2026 and says policy changes apply to new traces. Verify existing trace cohorts and self-hosted terms separately.
For AWS, follow through to CloudWatch pricing. Record the exact telemetry category, region, log class, storage basis, query meter, and any allowances. Its examples distinguish ingestion, archived storage, and data-protection scanning; exported payload size alone does not establish the billable quantity.
Turn runs and spans into a telemetry workload
Estimate collection volume from kept runs and measured payloads, then reconcile the estimate with the platform's usage report. Measure logs, span events, and metrics separately where they use separate meters. Include retries and duplicate exports in the measurements.
For a simplified workload with whole-run collection:
Kept runs = monthly runs × collection fraction
Kept spans = kept runs × mean spans per run
Payload GB = kept spans × mean bytes per span ÷ 1,000,000,000
Assumed example: 100,000 runs/month × 25% collection × 8 spans/run × 2,000 bytes/span ÷ 1,000,000,000 = 0.4 GB/month across 25,000 kept runs and 200,000 spans. These are hypothetical inputs, not typical trace sizes or recommended sampling settings. This decimal-GB estimate excludes extra signal streams, compression, protocol overhead, and provider metering rules.
At a steady arrival rate with 30-day months and no other deletion, retained payload GB ≈ monthly payload GB × retention days ÷ 30. Thus 0.4 GB/month × 90/30 = 1.2 GB retained, compared with 0.4 GB at 30 days. This is steady inventory before compression, not a storage invoice: startup months, retention cohorts, hourly prorating, and billable storage units need separate treatment.
Longer retention keeps more arrival cohorts at once. The example assumes a steady 0.4 GB per 30-day month; it does not model compressed storage or billing increments.
Record query volume independently. If each of 60 monthly queries scans an assumed 0.4 GB, query GB = 60 × 0.4 = 24 GB/month. Measure actual scans after filters and query optimizations; do not assume the returned result size equals the queried volume.
Count evaluation calls, tokens, and reviewer effort
Multiply selected traces by evaluators per trace before estimating judge-token usage. Keep evaluation sampling distinct from collection sampling, and write down its denominator. If evaluations only use collected traces, reducing collection also reduces the pool available for assessment.
In the same hypothetical workload, evaluate 10% of the 25,000 kept runs, with two judges per selected run:
Judge calls = 25,000 × 10% × 2 = 5,000 calls/month
Assuming each call consumes 2,000 input tokens and produces 200 output tokens, input tokens = 5,000 × 2,000 = 10,000,000 and output tokens = 5,000 × 200 = 1,000,000. Replace those assumptions with the evaluator's measured context and output usage, including retries where billed.
Two judges double calls for this fixed evaluation set. Token totals depend on the context sent to each judge, rather than the agent's original model bill.
Match these counts to the evaluation product. AWS includes model usage in built-in evaluation charges; custom evaluation inference is separate. LangSmith's pricing page says Tuned Evaluators bill successful runs and exclude failed or skipped evaluations. Apply that rule only to that product, and verify other evaluators' retry and failure policies.
Reviewer time needs its own row: hours = reviews × minutes per review ÷ 60. An assumed 100 reviews × 3 minutes ÷ 60 = 5 hours/month. Track escalation and adjudication separately if your process needs them.
Copy the worksheet and replace the assumptions
Copy this table into your budget document and fill the last column from a representative measurement window. The example is complete enough to check the arithmetic; your estimate needs your own traffic and policy. For every priced row, append the provider's exact billing unit, increment, allowance, dated rate, and plan or region.
| Input | Worked example | Your measurement or decision |
|---|---|---|
| InputExecutions and span payload | Worked example 100,000 runs; 8 spans/run; 2,000 bytes/span | Your measurement or decision Run count; span distribution; measured bytes by signal |
| InputCollection rule | Worked example 25% whole-run collection; 25,000 runs kept | Your measurement or decision Selection rule, denominator, exceptions, and actual kept count |
| InputIngested payload | Worked example200,000 spans; 0.4 GB/month | Your measurement or decision Provider-reported billable volume; extra logs/events/metrics |
| InputRetention | Worked example 30 days; approximately 0.4 GB steady payload inventory | Your measurement or decision Each class/tier and duration; compressed billable storage; extended traces |
| InputQueries | Worked example 60 queries × 0.4 GB scanned = 24 GB/month | Your measurement or decision Dashboards, alerts, ad-hoc queries, exports, and actual scan volume |
| InputEvaluation selection | Worked example10% of kept runs = 2,500 selected runs | Your measurement or decision Eligible pool, policy, selected count, and repeat evaluations |
| InputJudges and inference | Worked example 2 judges/run; 5,000 calls; 10M input and 1M output tokens | Your measurement or decision Evaluator type/model; measured tokens; billed retries/failures |
| InputHuman review | Worked example100 reviews × 3 minutes = 5 hours | Your measurement or decision Review, escalation, and adjudication hours; internal labor rate |
| InputPrivacy and evidence limits | Worked example Owner-defined; no assumed safety threshold | Your measurement or decision Redaction/masking workload, access policy, required incidents and lookback |
Build the budget as telemetry charges + evaluation charges + operating effort. Telemetry can include ingestion, retention, query/scan, masking, export or egress where applicable. Evaluation can include service calls and model inference where charged separately. Operating effort includes collector maintenance, review, incident investigation, and storage administration. Add mandatory seats or plan fees, then reconcile included allowances and tiered rates rather than multiplying every raw count by a single headline price.
The agent memory cost worksheet covers the adjacent persistence budget. Keep it separate unless a specific charge genuinely belongs to both workflows, in which case allocate it once.
Set evidence requirements before reducing collection
Choose a configuration only after it can preserve the evidence your team needs. As a decision rule, if the required incident lookback exceeds a plan's included retention, budget an extension or a suitable archive before accepting that plan's subtotal. If required failure classes are absent from the collected set, a smaller bill does not meet the same requirement.
To optimize AI agent observability costs, test a change against those requirements and compare the resulting meter readings. Payload redaction can change byte volume; a narrower query can change scans; fewer judges can change evaluation calls. Reducing any one of them does not imply the whole bill falls by the same fraction. Sampling after ingestion, for example, cannot remove bytes already ingested.
Keep an owner and a reason beside each exception to sampling or deletion. High-risk actions, rare failures, and privacy-sensitive content need policies based on the application, rather than a universal percentage supplied by a cost worksheet. Verify access controls and masking behavior as well as their billable workload.
Method, sources, and next reading
Updated 18 October 2026. This worksheet compiles public billing definitions from current official pages and uses explicitly assumed arithmetic to show how to normalize workload. It compares meters and evidence coverage; it does not estimate a dollar bill, establish a safe sampling rate, or rank platforms. Rates, usage-unit mappings, contract terms, and provider metering must be checked for the configuration you select.
- Amazon Bedrock AgentCore pricing: observability and evaluation charge categories.
- Amazon CloudWatch pricing: telemetry categories, regional pricing, and storage/data-protection examples.
- LangSmith pricing and retention documentation: trace definitions, retention policy, and Tuned Evaluator billing.
- OpenObserve pricing: ingestion/query units, included retention, and the Cloud Enterprise BYOB option.
Use the editorial methodology when assessing source scope. For conference context, open the Microsoft Build agent observability record or the re:Invent OpenSearch observability record; their figures are not inputs to this worksheet. Related analysis sits under agentic AI.
Before sharing a comparison, attach the filled workload table and the dated official pricing references. Use this page's citation control to cite the method, or report a source correction if an official billing term has changed.