Eco-Guard

Eco-Guard is a control plane for serving, observing and managing LLM inference on your own infrastructure. It presents an OpenAI-compatible API, routes to whichever inference backend you run, adds content safety and cost governance in front of it, and manages the model lifecycle behind it. The whole platform deploys as a single Docker container.

The inference gateway

The API is a drop-in for the OpenAI chat completions, embeddings and models endpoints, so any OpenAI SDK client works unchanged. Behind that surface it routes to llama.cpp, Ollama, vLLM, Hugging Face TGI or any OpenAI-compatible endpoint through a common backend interface, which means backends can be hot-swapped without a restart. It supports streaming over server-sent events, model-specific chat templates for Llama 3, Mistral, ChatML, Gemma and Zephyr, OpenAI-compatible function calling, batch inference of up to 100 prompts, and fallback chains with per-step timeouts.

Content safety

A guardrails pipeline runs with per-stage actions: block, flag, sanitize or allow. It detects prompt injection and jailbreak patterns, redacts PII such as card numbers, emails, phone numbers and IP addresses, classifies self-harm, violence and hate speech, and flags prompt anomalies like spam floods and unusual token counts.

Cost and governance

Per-model pricing is pre-configured, requests can be compared across six providers before they run, and workspaces carry budget caps with alert thresholds. Token counting uses tiktoken with a heuristic fallback, every administrative action is audit-logged, and GDPR export and deletion endpoints are built in.

MLOps

A model registry moves models through Registered, Staging, Production and Archived with checksum verification and rollbacks. Deployments support direct, canary, blue-green and A/B strategies, with automated evaluation gating rollouts on pass or fail thresholds. Drift detection uses a statistical Z-score and can trigger retraining, experiments log metrics per step, and a leaderboard ranks models by latency, accuracy, token efficiency and drift. There is also a RAG pipeline for ingestion, chunking, embedding and retrieval.

Observability and platform

Prometheus metrics, OpenTelemetry tracing with an OTLP exporter, structured JSON logs with request correlation, a real-time WebSocket for live metrics and a 17-panel Grafana dashboard. On the platform side there is multi-tenancy with roles and token quotas, JWT cookies with brute-force protection and scrypt hashing, hashed API keys, Google and GitHub SSO, a Redis-backed sliding-window rate limiter and circuit breaker, CIDR IP allowlisting, and GitOps configuration with hot reload. The security posture is explicit: no analytics, no tracking, no phone-home.

Deployment

Docker Compose brings up the API with PostgreSQL 15 and Redis 7 with health checks and migrations. A Helm chart adds a Deployment, Service, Ingress, HorizontalPodAutoscaler, NetworkPolicy, PodDisruptionBudget, ServiceMonitor and PersistentVolumeClaim, and a Terraform module covers infrastructure as code.