SCAND Study: Only 2 of 15 Open-Source LLMs Reliably Search Company Documents Before the Internet

Tests show most private AI assistants quietly skip the knowledge base — a hidden data-leak risk for enterprises

WARSAW, POLAND, November 12, 2026 /EINPresswire.com/ — SCAND, a software development company headquartered in Warsaw, today released a study evaluating 15 open-weight large language models (LLMs) for use in private retrieval-augmented generation (RAG) systems.

The evaluation found that only 2 of the 15 models reliably retrieved information from a company knowledge base before relying on external information, highlighting a potential weakness in how enterprise AI assistants handle internal knowledge and source selection.

The study assessed the models across seven questions covering summarization, information retrieval, instruction following, and tool calling. The goal was to understand how well models perform in a practical enterprise scenario: answering employee questions using internal documents while following the system’s instructions about where to find information.

Key Findings

The results reveal a gap between a model’s ability to generate fluent answers and its reliability in following a prescribed retrieval workflow. For enterprises considering locally hosted AI assistants, the findings show why models should be evaluated on source selection and instruction adherence, not just answer quality.

14 of 15 models passed both summarization tests. Most models could condense supplied information effectively, suggesting that summarization alone is not a sufficient measure of readiness for enterprise RAG deployments.

Qwen 3.8 27B in 4-bit quantization was the strongest compact option in the evaluation. It achieved a score of 6 out of 7 while requiring approximately 16 GB of memory, making it a practical candidate for organizations exploring locally hosted AI assistants.

A larger model did not necessarily perform better. In the tested scenarios, a 120-billion-parameter model performed worse than the 27-billion-parameter Qwen model on key tasks. Parameter count alone therefore did not reliably predict suitability for a knowledge-base assistant.

The findings point to a distinction between generating a plausible answer and retrieving the correct information from an approved source. A model may summarize fluently yet fail to follow the intended retrieval workflow.

In enterprise environments, that behavior can undermine answer traceability and create a potential data-governance risk if a system is permitted to use external sources when internal information should take precedence.

Addressing these challenges requires more than selecting a capable language model. It also requires a well-designed retrieval architecture, reliable data integration, and clear source-priority controls.

Through its RAG development services, SCAND helps businesses build AI-powered solutions that retrieve relevant information from private knowledge bases, integrate with enterprise data sources, and generate context-aware responses grounded in approved internal information.

“Source selection should be controlled by the architecture, not left to the model. Even capable models can bypass retrieval when these controls are missing. Enterprise systems need explicit rules for information sources and answer grounding,” said Vadzim Tashlikovich, Head of ERP Solutions Department at SCAND.

SCAND recommends evaluating models against realistic workflows rather than relying solely on benchmark scores, parameter counts, or summarization quality. For enterprise RAG deployments, testing should include retrieval reliability, instruction adherence, tool use, memory requirements, and the ability to respect source-priority rules.

The full methodology, model comparison, and recommendations are available in the study Best Local LLM for RAG: What 15 Open-Source Models Taught Us About Knowledge-Base Assistants.

About SCAND

SCAND is a software development company headquartered in Warsaw, Poland, with more than 25 years of experience, a team of over 250 engineers, and more than 900 completed projects. The company holds ISO 9001 and ISO/IEC 27001 certifications and works with organizations on software engineering and AI-related development projects. For more information, visit scand.com.

SCAND
Scand Poland Sp. z o.o.
sales@scand.com
+1 208-900-5392
Visit us on social media:
LinkedIn
Facebook
YouTube
X
Other