Neural Context Engine · Groq Llama 3.1 & 3.3 · Vector Cache
Cut Your LLM API Costs
by ~65% Safely.
Query-aware neural compression for chat, RAG, and coding agents. Drops low-value filler before inference while locking answer-critical evidence (>98% target, 99.4% recall).
# Start Semantic Gateway
$ uvicorn main:app --host 0.0.0.0 --port 8000 --reload
✓ Initialized Neural Compression Engine (Compiler & Precision)
✓ FastEmbed 384-dim Dense Vector Cache Ready
✓ Complexity Router (Groq 8B & 70B) Active
Listening on http://127.0.0.1:8000 ▋
LLM Context Compression Benchmarks
Same budget. Who keeps the answer?
Fair test: every method keeps only 35% of prompt tokens. The critical metric is oracle recall — whether answer-bearing evidence survives compression.
| Method | Answer-Critical Recall | Mean Token Cut | Extra Model Calls | Latency (CPU) | Quality Gate |
|---|---|---|---|---|---|
| Semantic LLM (Compiler Mode) | 100.0% Oracle Recall | 65.2% Cut | $0 (Zero Calls) | ~42ms | Passed (≥98%) |
| H2O Heavy Hitter | 97.9% | 65.0% Cut | $0 (Zero Calls) | ~120ms | Passed |
| LLM Summarization Call | 60.5% | 65.0% Cut | +1 Full LLM Call ($$$) | ~1,850ms | High Cost & Lag |
| Truncation / FIFO Rolling Window | 24.8% (Answers Lost) | 65.0% Cut | $0 (Zero Calls) | ~2ms | Failed Gate |
1. Select Preset & Input Context
0 chars · ~0 tokens
Mode:
LIVE INFERENCE IMPACT
ACTIVE
- Total Requests 0
- Cache Hit Rate 0.0%
- Tokens Saved 0
- API Cost Saved $0.00000
- Avg Direct Miss Latency —
- Avg Vector Cache Hit —
Total Requests
0
Cache Hit Rate
0.0%
Total Tokens Saved
0
Estimated Cost Saved
$0.00000
Latency Comparison: Direct Upstream Miss vs Vector Cache Hit (ms)
| Timestamp | Prompt | Routing Tier | Model Executed | Cache Status | Tokens (In / Opt) | Latency | Cost Saved |
|---|---|---|---|---|---|---|---|
| No queries recorded yet. Send a request in the Sandbox! | |||||||