Standard RAG systems retrieve individual document chunks based on semantic similarity, which works well for questions whose answers live in a single passage. But many important questions require understanding relationships across multiple documents: "Who are the key stakeholders in Project X and what are their roles?" or "What are the main themes discussed across all board meeting minutes this quarter?" These questions require connecting information scattered across the corpus, something that vector similarity search alone cannot accomplish.

GraphRAG addresses this limitation by building a knowledge graph from your documents and using graph-based retrieval alongside or instead of traditional vector search. Pioneered by Microsoft Research, GraphRAG has become one of the most exciting advances in retrieval-augmented generation.

The Limitations of Standard RAG

Standard RAG excels at local queries whose answers are contained in one or two chunks. "What is our return policy?" or "How do I reset my password?" are well-served by vector similarity search because the answer exists in a specific, retrievable passage.

Standard RAG struggles with global queries that require synthesizing information across many documents. "What are the main challenges facing our organization?" requires aggregating insights from dozens of documents, extracting common themes, and synthesizing them into a coherent summary. No single chunk contains the answer.

GraphRAG was designed specifically to handle global queries that require understanding the structure and themes of an entire document corpus, not just retrieving individual passages.

How GraphRAG Works

The GraphRAG pipeline involves several key stages during indexing:

Entity and Relationship Extraction

An LLM processes each document chunk to extract entities (people, organizations, concepts, events) and the relationships between them. For example, from a news article, the system might extract entities like "Microsoft," "Satya Nadella," and "Azure" along with relationships like "Nadella leads Microsoft" and "Microsoft operates Azure."

Graph Construction

Extracted entities and relationships are assembled into a knowledge graph. Entities become nodes and relationships become edges. When the same entity appears across multiple documents, its node is enriched with information from all occurrences, creating a comprehensive representation that no single document contains.

Community Detection

Graph algorithms like the Leiden algorithm identify communities of densely connected nodes within the graph. These communities represent natural clusters of related entities and concepts. A corporate knowledge graph might have communities for "Engineering Team," "Product Roadmap," "Customer Feedback," and "Financial Performance," each containing entities and relationships relevant to that theme.

Community Summaries

Each community is summarized by an LLM into a natural language description that captures the key entities, relationships, and themes within the community. These summaries serve as the primary retrieval targets for global queries, providing pre-computed answers to broad questions about the corpus.

Key Takeaway

GraphRAG transforms unstructured documents into a structured knowledge graph with pre-computed community summaries, enabling both local queries (entity-specific) and global queries (theme-level synthesis).

Query Modes

Local Search

For specific, focused queries, GraphRAG uses the knowledge graph to find relevant entities and their neighborhoods. Starting from entities mentioned in the query, the system traverses relationships to gather connected information, then retrieves the source document chunks associated with those entities. This provides more targeted context than blind vector similarity search.

Global Search

For broad, thematic queries, GraphRAG leverages the pre-computed community summaries. It maps the query to relevant communities, retrieves their summaries, and synthesizes a comprehensive answer. This enables answering questions like "What are the main topics discussed in these documents?" without needing to retrieve and process every individual chunk.

When to Use GraphRAG

GraphRAG provides the most value when:

  • Your queries are often global: Questions about themes, patterns, summaries, and cross-document insights
  • Entity relationships matter: Your domain involves complex relationships between people, organizations, products, or concepts
  • Document corpus is large: With enough documents, the graph structure reveals patterns invisible at the individual document level
  • Multi-hop reasoning is needed: Answers require connecting information across multiple intermediate entities

GraphRAG may not be worth the additional complexity for simple factual Q&A over small document sets where standard RAG performs well.

Implementation Considerations

Building a GraphRAG system requires significant compute for the indexing phase. Every chunk must be processed by an LLM for entity extraction, and community detection and summarization add additional LLM calls. For a large corpus, indexing costs can be substantial.

Microsoft's open-source GraphRAG implementation provides a reference architecture with configurable extraction prompts, community detection parameters, and query modes. Alternative implementations exist in LlamaIndex with its Property Graph Index and in LangChain with graph-based retrieval components.

Quality of Entity Extraction

The quality of the knowledge graph depends directly on the quality of entity and relationship extraction. LLMs can miss entities, create duplicate entities with different names for the same concept, or extract incorrect relationships. Entity resolution, the process of merging duplicate entities, is critical for graph quality.

A messy knowledge graph produces worse results than no graph at all. Invest in entity extraction quality and entity resolution to ensure the graph accurately represents your knowledge base.

GraphRAG vs Standard RAG

GraphRAG is not a replacement for standard RAG but a complement. The best approach combines both:

  1. Use standard RAG for specific, factual queries that can be answered from individual passages
  2. Use GraphRAG local search when the query involves specific entities and their relationships
  3. Use GraphRAG global search for broad, synthetic questions about themes and patterns

A query router can classify incoming questions and direct them to the appropriate retrieval strategy, combining the efficiency of standard RAG with the broader reasoning capabilities of GraphRAG.

Key Takeaway

GraphRAG excels at answering questions that require understanding relationships and themes across many documents. Combine it with standard RAG for a comprehensive retrieval system that handles both local and global queries effectively.

As knowledge graphs become easier to build with LLM-assisted extraction and as graph databases become more accessible, GraphRAG is positioned to become a standard component of production RAG architectures. The key is understanding when the additional complexity and cost are justified by your use case's requirements for cross-document reasoning.