Special Interest Groups (SIGs)

info

This page mirrors SIGS.md from the llm-d repository. Edit it there.

SIG Overview

Special Interest Groups (SIGs) are the primary organizational units for coordinating work across the llm-d project. Each SIG focuses on a specific area of the project's technology stack and is responsible for driving design, implementation, and maintenance of their respective components.

SIGs provide a mechanism for:

• Focused expertise: Bringing together contributors with specialized knowledge in specific areas
• Coordinated development: Ensuring consistent architectural decisions across related components
• Community building: Creating smaller, more manageable groups for collaboration and mentorship
• Accountability: Clear ownership and responsibility for specific project areas

SIG Structure and Governance

SIG Leadership

Each SIG has:

• SIG Leads (2-3 people): Responsible for overall SIG direction, coordination, and decision-making

SIG Responsibilities

• Drive technical design and implementation in their area
• Maintain documentation and architectural decisions
• Coordinate with other SIGs on cross-cutting concerns
• Mentor new contributors and grow the community
• Participate in project-wide planning and releases

SIG Meetings

• Regular meetings (typically weekly/bi-weekly) for technical discussions

Relationship to Project Governance

SIGs operate within the broader llm-d project governance framework defined in PROJECT.md:

• SIGs follow the project's lazy consensus decision-making process
• Major cross-SIG decisions require project maintainer approval
• All SIG work follows the project's contribution guidelines

Active Special Interest Groups

For up-to-date meeting times, see the Public Meeting Calendar.

SIG	Focus Area	Documentation
SIG Router (Formerly Inference Scheduler)	Intelligent request routing, load balancing, and traffic management	• Meeting Recordings and Docs • llm-d-router Repository
SIG Benchmarking	Performance testing, benchmarking frameworks, and optimization	• Meeting Recordings and Docs • llm-d-benchmark Repository
SIG PD-Disaggregation	Prefill/decode separation, distributed serving, and workload disaggregation	• Meeting Recordings and Docs • llm-d-routing-sidecar Repository
SIG KV-Disaggregation	KV caching, prefix caching, and distributed storage systems	• Meeting Recordings and Docs • llm-d-kv-cache Repository
SIG Installation	Kubernetes integration, deployment tooling, and platform operations	• Meeting Recordings and Docs • llm-d-modelservice Repository • llm-d-infra Repository
SIG Autoscaling	Traffic-aware autoscaling, resource management, and capacity planning	• Meeting Recordings and Docs • workload-variant-autoscaler Repository
SIG Observability	Monitoring, logging, metrics, and operational visibility	• Meeting Recordings and Docs • llm-d Observability Documentation
SIG RL	Improve SOTA performance for RL workloads	• Meeting Recordings and Docs
SIG Agentic Inference	Optimizing inference for agentic and multi-step AI workloads	• Meeting Recordings and Docs
SIG Inference Payload Processor	Pluggable request/response payload processing, intelligent model selection, and external model integration	• Meeting Recordings and Docs • llm-d-inference-payload-processor Repository
SIG Batch Inference	Asynchronous processing, request queueing, and batch gateway management	• Meeting Recordings and Docs • llm-d-async Repository • llm-d-batch-gateway Repository

SIG Detailed Descriptions

SIG Router (Formerly Inference Scheduler)

👥 Leadership: Nili Guy, Abdullah Gharaibeh, Vita Bortnikov

⭐️ North Star Design Document ↗️ (Google Docs)

Charter: Develop and maintain intelligent request routing and load balancing systems that optimize for latency, throughput, and resource utilization across distributed inference workloads.

Key Areas:

• vLLM-optimized optimized baseline algorithms
• KV-cache aware routing and load balancing
• Integration with Kubernetes Gateway API and Inference Gateway Extension
• Flow control and traffic shaping
• SLA-aware request prioritization

💬 Communication:

• Slack Channel: #sig-router
• Meeting Recordings and Docs: Public Google Drive
• GitHub Issues: github.com/llm-d/llm-d-router

SIG Benchmarking

👥 Leadership: Marcio A L Silva, Ashok Chandrasekar

⭐️ North Star Design Document ↗️ (Google Docs)

Charter: Establish comprehensive performance testing and benchmarking frameworks to ensure llm-d delivers optimal performance across diverse workloads and hardware configurations.

Key Areas:

• Benchmarking frameworks and methodologies
• Performance regression testing
• Workload simulation and synthetic data generation
• Hardware-specific optimization
• Performance analysis and profiling tools

💬 Communication:

• Slack Channel: #sig-benchmarking
• Meeting Recordings and Docs: Public Google Drive
• GitHub Issues: github.com/llm-d/llm-d-benchmark

SIG PD-Disaggregation

👥 Leadership: Robert Shaw, Tyler Michael Smith

⭐️ North Star Design Document ↗️ (Google Docs)

Charter: Design and implement prefill/decode disaggregation patterns that enable efficient separation of inference workloads across heterogeneous hardware and scaling requirements.

Key Areas:

• Prefill/decode workload separation
• Disaggregated serving architecture
• Cross-instance communication protocols
• Heterogeneous hardware optimization
• Dynamic workload balancing between Prefill and Decode instances

💬 Communication:

• Slack Channel: #sig-pd-disaggregation
• Meeting Recordings and Docs: Public Google Drive
• GitHub Issues: github.com/llm-d/llm-d-routing-sidecar

SIG KV-Disaggregation

👥 Leadership: Maroon Ayoub, Danny Harnik

⭐️ North Star Design Document ↗️ (Google Docs)

Charter: Design and implement distributed KV caching solutions that improve inference performance through intelligent cache management, prefix sharing, and disaggregated storage.

Key Areas:

• Distributed KV cache architecture
• Prefix cache hierarchies (local, remote, shared)
• Cache-aware scheduling and routing
• Storage optimization for inference workloads
• Integration with vLLM's KVConnector

💬 Communication:

• Slack Channel: #sig-kv-disaggregation
• Meeting Recordings and Docs: Public Google Drive
• GitHub Issues: github.com/llm-d/llm-d-kv-cache

SIG Installation

👥 Leadership: Brent Salisbury, Greg Pereira

⭐️ North Star Design Document ↗️ (Google Docs)

Charter: Ensure llm-d integrates seamlessly with Kubernetes and provides robust deployment, scaling, and operational capabilities for production environments.

Key Areas:

• Kubernetes-native deployment patterns
• Helm charts and operators
• Installation and configuration management
• Multi-node orchestration with LeaderWorkerSet
• Platform integration and operational best practices

💬 Communication:

• Slack Channel: #sig-installation
• Meeting Recordings and Docs: Public Google Drive
• GitHub Issues: github.com/llm-d-incubation/llm-d-modelservice | github.com/llm-d-incubation/llm-d-infra

SIG Autoscaling

👥 Leadership: Abhishek Malvankar, Lionel Villard

⭐️ North Star Design Document ↗️ (Google Docs)

Charter: Develop intelligent autoscaling solutions that automatically adjust llm-d deployments based on traffic patterns, workload characteristics, and hardware utilization.

Key Areas:

• Traffic-aware autoscaling algorithms
• Hardware-specific scaling policies
• Workload-based capacity planning
• Integration with Kubernetes HPA/VPA/KEDA
• Cost-optimized scaling strategies

💬 Communication:

• Slack Channel: #sig-autoscaling
• Meeting Recordings and Docs: Public Google Drive
• GitHub Issues: github.com/llm-d/llm-d-workload-variant-autoscaler

👥 Emeritus Leadership:

• Tamar Eilam

SIG Observability

👥 Leadership: Sally O'Malley, Roy Nissim, Benedikt Bongartz

⭐️ North Star Design Document ↗️ (Google Docs)

Charter: Provide comprehensive monitoring, logging, and observability capabilities that enable operators to understand system behavior, diagnose issues, and optimize performance.

Key Areas:

• Metrics collection and visualization
• Distributed tracing and logging
• Performance monitoring and alerting
• Operational dashboards and reporting
• Integration with monitoring ecosystems (Prometheus, Grafana, etc.)

💬 Communication:

• Slack Channel: #sig-observability
• Meeting Recordings and Docs: Public Google Drive

SIG RL

👥 Leadership: Bogdan Berce, Robert Shaw North Star Design Document TODO

Charter: Assist RL teams in achieving SOTA performance for RL workloads

Key Areas:

• TODO

💬 Communication:

• Slack Channel: #sig-rl
• Meeting Recordings and Docs: Public Google Drive

SIG Inference Payload Processor

👥 Leadership: Nir Rozenbaum, Maroon Ayoub

Charter: Develop and maintain a pluggable framework for request/response payload processing that enables intelligent model selection, seamless integration with both in-cluster and out-of-cluster models (external models), and extensible request/response transformations.

Key Areas:

• Unified serving of in-cluster and out-of-cluster models through a single inference gateway
• Intelligent model selection via pluggable Filter/Score/Picker pipeline based on runtime signals
• Custom payload processing including request/response mutations, field extraction, and header transformations
• Circuit breaking and resilience patterns for model endpoints
• API key injection and credential management for out-of-cluster model providers
• Integration with payload-level systems such as guardrails, content filtering, metering, and usage tracking

💬 Communication:

• Slack Channel: #sig-inference-payload-processor
• Meeting Recordings and Docs: Public Google Drive

SIG Batch Inference

👥 Leadership: Shimi Bandiel, Jacob Murry, Lior Aronovich

North Star Design Document TODO

Charter: Develop solutions for efficient batch inference in Kubernetes environments.

Key Areas:

• Efficient processing of batch workloads in combination with interactive workloads
• Client interfaces and APIs for batch inference job management

💬 Communication:

• Slack Channel: #sig-batch-inference
• Meeting Recordings and Docs: Public Google Drive
• GitHub Issues: github.com/llm-d/llm-d-async | github.com/llm-d/llm-d-batch-gateway

SIG Agentic Inference

👥 Leadership: Maroon Ayoub, Sean Horgan

⭐️ North Star Design Document ↗️

Charter: TODO

Key Areas:

• TODO

💬 Communication:

• Slack Channel: #sig-agentic-inference
• Meeting Recordings and Docs: Public Google Drive

Getting Involved

Joining a SIG

1. Attend a meeting: Check the project calendar for SIG meeting times
2. Join the conversation: Participate in SIG-specific channels on Slack
3. Review documentation: Read the SIG's charter and current initiatives
4. Start contributing: Look for "good first issues" labeled with the SIG's area

SIG Communication Channels

• Slack: Each SIG has dedicated channels in the llm-d Slack workspace
• Google Groups: Join llm-d-contributors for comment access to SIG documents
• GitHub: Issues and discussions are labeled by SIG area
• Calendar: All SIG meetings are on the shared project calendar

SIG Formation and Evolution

Creating a New SIG

1. Identify need: Demonstrate community interest and technical necessity
2. Draft charter: Define scope, goals, and initial leadership
3. Proposal process: Submit proposal following project contribution guidelines
4. Community review: Present at bi-weekly project standup and gather feedback
5. Approval: Obtain approval from project maintainers

SIG Lifecycle Management

• Active: Regular meetings, active development, engaged community
• Maintenance: Limited active development, focus on stability and bug fixes
• Archived: No longer active, historical reference only

SIGs may evolve, merge, or be archived based on project needs and community engagement.

Resources

• Project Calendar: llm-d Public Calendar
• Slack Workspace: https://llm-d.slack.com
• Google Groups: https://groups.google.com/g/llm-d-contributors
• Community Governance: PROJECT.md
• Contributing Guidelines: CONTRIBUTING.md

Maintenance

This document is maintained by the project maintainers and updated as SIGs evolve. For questions or suggestions about SIG structure, please reach out via:

• Bi-Weekly project standup (Every other Wednesday 12:30 PM ET)
• llm-d Slack channel
• GitHub issues in the llm-d/llm-d repository