User GuideRemote Trace Sources

Remote Trace Sources

TraceCraft’s TUI can pull traces that already live in your cloud observability platform — no need to copy data locally. Connect directly to AWS X-Ray, GCP Cloud Trace, Azure Monitor, or DataDog and browse those traces with the same interactive interface you use for locally-stored traces.

Read-only connection

Remote backends are read-only. The TUI fetches and displays traces; it never writes, modifies, or deletes records in your platform. save() and delete() operations always raise NotImplementedError.


Supported Platforms

PlatformURL SchemeExtraAuth
AWS X-Ray (AgentCore)xray://region/servicestorage-xrayboto3 credential chain
GCP Cloud Trace (Vertex AI)cloudtrace://project/servicestorage-cloudtraceADC (google.auth.default())
Azure Monitor (AI Foundry)azuremonitor://workspace-id/servicestorage-azuremonitorDefaultAzureCredential
DataDog APMdatadog://site/servicestorage-datadogDD_API_KEY + DD_APP_KEY

Installation

Install the extras for the platform(s) you use:

# Individual platforms
pip install "tracecraft[storage-xray]"
pip install "tracecraft[storage-cloudtrace]"
pip install "tracecraft[storage-azuremonitor]"
pip install "tracecraft[storage-datadog]"
 
# All remote backends at once
pip install "tracecraft[storage-all]"
 
# Already covered by the cloud bundle
pip install "tracecraft[cloud]"

AWS X-Ray

Quick Start

# Browse all traces in us-east-1 from the last hour
tracecraft tui xray://us-east-1/
 
# Filter to a specific service
tracecraft tui xray://us-east-1/my-bedrock-agent

Authentication

Uses the standard boto3 credential chain — no credentials are stored in the config file:

  1. Environment variables: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN
  2. AWS profile: ~/.aws/credentials (selected by AWS_PROFILE)
  3. IAM instance profile / ECS task role / EKS IRSA (in EC2/ECS/EKS environments)
  4. AWS SSO
# Required IAM permissions:
# xray:GetTraceSummaries
# xray:BatchGetTraces

Config File

# .tracecraft/config.yaml
default:
  storage:
    type: xray
    xray_region: us-east-1
    xray_service_name: my-bedrock-agent   # optional
    xray_lookback_hours: 1
    xray_cache_ttl_seconds: 60

GCP Cloud Trace

Quick Start

# Set your project (or use GOOGLE_CLOUD_PROJECT env var)
export GOOGLE_CLOUD_PROJECT=my-gcp-project
 
# Browse all traces
tracecraft tui cloudtrace://my-gcp-project/
 
# Filter to a specific service
tracecraft tui cloudtrace://my-gcp-project/my-vertex-agent

Authentication

Uses Application Default Credentials (google.auth.default()) — no credentials are stored in the config file:

  1. Workload Identity (GKE, Cloud Run)
  2. gcloud auth application-default login
  3. GOOGLE_APPLICATION_CREDENTIALS env var pointing to a service account key file
# Required IAM roles:
# roles/cloudtrace.viewer

Config File

# .tracecraft/config.yaml
default:
  storage:
    type: cloudtrace
    cloudtrace_project_id: my-gcp-project   # or set GOOGLE_CLOUD_PROJECT
    cloudtrace_service_name: my-agent       # optional
    cloudtrace_lookback_hours: 1
    cloudtrace_cache_ttl_seconds: 60

Azure Monitor (Application Insights)

Quick Start

# Set your workspace ID
export AZURE_MONITOR_WORKSPACE_ID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
 
tracecraft tui azuremonitor://xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/
 
# Filter to a service (cloud_RoleName)
tracecraft tui azuremonitor://xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/my-ai-foundry-agent

Authentication

Uses DefaultAzureCredential — no credentials are stored in the config file:

  1. Managed Identity (Azure VMs, App Service, AKS)
  2. az login (Azure CLI)
  3. Environment variables: AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, AZURE_TENANT_ID
  4. VS Code Azure extension / Interactive browser
# Required Azure RBAC role:
# Log Analytics Reader
⚠️

Workspace ID vs Connection String

The workspace_id is the Log Analytics Workspace ID (a GUID), not the Application Insights connection string or instrumentation key. Find it in the Azure Portal under your Log Analytics workspace → Overview → Workspace ID.

Config File

# .tracecraft/config.yaml
default:
  storage:
    type: azuremonitor
    # Set AZURE_MONITOR_WORKSPACE_ID env var instead of hardcoding here
    azuremonitor_workspace_id: null
    azuremonitor_service_name: my-agent     # optional (cloud_RoleName)
    azuremonitor_lookback_hours: 1
    azuremonitor_cache_ttl_seconds: 60

DataDog APM

Quick Start

# Set credentials via env vars (never in config files)
export DD_API_KEY=your_api_key
export DD_APP_KEY=your_application_key
 
# US1 site (default)
tracecraft tui datadog://us1/my-service
 
# EU site
tracecraft tui datadog://eu1/my-service

Authentication

Requires two environment variables set at shell level:

VariableDescription
DD_API_KEYDataDog API key (from Organization Settings → API Keys)
DD_APP_KEYDataDog Application key (from Organization Settings → Application Keys)
🚫

Never put DataDog credentials in config files

DD_API_KEY and DD_APP_KEY must be set as environment variables. TraceCraft validates at startup that both are present and raises ValueError with a clear error message if either is missing.

Supported Sites

Site KeyAPI Host
us1 (default)api.datadoghq.com
us3api.us3.datadoghq.com
us5api.us5.datadoghq.com
eu1api.datadoghq.eu
ap1api.ap1.datadoghq.com

Config File

# .tracecraft/config.yaml
default:
  storage:
    type: datadog
    datadog_site: us1
    datadog_service: my-service   # optional
    datadog_lookback_hours: 1
    datadog_cache_ttl_seconds: 60
# Secrets: set DD_API_KEY and DD_APP_KEY as environment variables

Caching and Rate Limiting

All remote backends use an in-memory TTL cache (default: 60 seconds) to prevent excessive API calls when the TUI polls for new traces.

  • The TUI polls every ~1 second; without caching this would hammer platform APIs
  • Cache is per-store instance; it resets when the TUI process restarts
  • Press R in the TUI to force a cache refresh (calls invalidate_cache())
  • Tune *_cache_ttl_seconds in the config file to balance freshness vs. API cost

Data Mapping

What Gets Populated

Remote backends map platform-specific spans to TraceCraft’s canonical AgentRun and Step models. The accuracy of LLM-specific fields depends on whether your instrumentation wrote them:

FieldPopulated When
model_namegen_ai.request.model span attribute/annotation present
input_tokensgen_ai.usage.input_tokens present
output_tokensgen_ai.usage.output_tokens present
cost_usdNot populated by remote backends
cloud_providerAlways set ("aws", "gcp", "azure", "datadog")
cloud_trace_idAlways set to the platform’s native trace ID

Best field coverage

If your agents are instrumented with TraceCraft’s own decorators or the OpenTelemetry GenAI semantic conventions, all LLM fields will be populated when viewing traces from any platform.

Step Type Inference

When reading from remote platforms, step types are inferred from span metadata:

PlatformSignalMapped To
X-Raynamespace: "aws" or HTTP subsegmentTOOL
X-Raygen_ai.* annotationLLM
Cloud Tracespan_kind: CLIENT/PRODUCERTOOL
Cloud Tracegen_ai.system attributeLLM
Azure MonitorAppDependencies tableTOOL
Azure MonitorAppExceptions tableERROR
DataDogtype: "web", "db"TOOL
DataDogtype: "llm"LLM

Troubleshooting

Missing Optional Dependency

ImportError: boto3 is required for XRayTraceStore.
Install with: pip install tracecraft[storage-xray]

Install the required extra and retry.

Authentication Error

AWS: Run aws sts get-caller-identity to verify credentials are configured.

GCP: Run gcloud auth application-default print-access-token to verify ADC.

Azure: Run az account show to verify CLI login. For managed identity issues, check that the correct role (Log Analytics Reader) is assigned.

DataDog: Verify DD_API_KEY and DD_APP_KEY are exported in your shell, not just set in .env files.

No Traces Returned

  • Increase *_lookback_hours — traces may be older than 1 hour
  • Verify the service name matches exactly (case-sensitive)
  • Check platform-specific sampling: if only 10% of traces are sampled, you may see an empty window during low traffic

Slow TUI Response

Increase *_cache_ttl_seconds to reduce API call frequency:

storage:
  type: xray
  xray_cache_ttl_seconds: 300  # 5 minutes