Evaluating Local Language Model Performance in Legacy Codebase Discovery: Qwen 3.6 35B vs. Opus 4.7
In the rapidly evolving landscape of AI-assisted development, the capability of local language models to efficiently analyze and interpret extensive, legacy codebases has become increasingly pertinent. Recent informal testing sheds light on how two prominent models—Qwen 3.6 35B and Opus 4.7—perform in real-world code comprehension tasks, emphasizing practical applicability over formal benchmarking.
Context and Setup
The test involved analyzing a substantial, aged PHP code repository dating from 2005 to 2016, encompassing over 200,000 lines of code. The task was straightforward: generate a concise summary outlining the business context and core features of this legacy system, which notably lacked documentation or README files. To maintain fairness, both models were executed using an identical Claude Code-style interface, and the entire process was conducted on an RTX 5090 GPU.
The evaluation was not a formal benchmark but a practical exploration aligned with everyday development needs. Utilizing GPT-5.5 as an impartial A/B judge enabled a blind comparison of the outputs, supplemented by manual validation.
Key Findings
| Evaluation Aspect | Opus 4.7 | Local Qwen 3.6 35B A3B Int4 | Observation |
|———————-|————|—————————-|————–|
| Context Handling | Approx. 26k tokens | Approx. 40k tokens | Qwen managed a broader context, indicating superior capacity for extensive code analysis. |
| Processing Speed | ~1 minute 7 seconds | ~37 seconds | Qwen demonstrated significantly faster responses. |
| Summary Quality | Good, comprehensive, cautious | Focused, concrete, sharper | Qwen provided clearer, actionable insights. |
| Risk of Overclaiming | Less prone | More confident but requires verification | Opus maintained conservative output, reducing potential inaccuracies. |
Overall, while Opus 4.7 offered careful, less ambitious summaries, Qwen 3.6 not only delivered quicker results but also facilitated more precise and actionable summaries—an essential advantage in development workflows.
Implications for Development Experience
This informal assessment highlights a pivotal trend: local models like Qwen are becoming sufficiently capable of handling large, complex codebases efficiently. Performance metrics on my setup reveal processing speeds reaching approximately 115 tokens per second on Qwen 3.6 27B and up to 205 tokens/sec on the 35B variant with optimized inference. These speeds are enabling more seamless integration into day-to-day coding activities, reducing dependence on cloud-based solutions.
While Opus remains cautious by nature, its outputs, albeit slower, tend to overstate less and prioritize safety. Conversely, Qwen’s ability to surface detailed, precise information rapidly has the potential to significantly enhance developer productivity and decision-making.
Concluding Remarks
Having contributed to the legacy codebase used in this test, I can attest that Qwen’s summaries aligned accurately with the actual system. Although this was a non-scientific, real-world test centered on a single repository and prompt, the results point toward a growing importance of inference speed and development-friendly model architectures in practical AI-assisted coding.
Looking Ahead
This experience prompts reflection on whether the industry is shifting towards valuing inference efficiency and leaner model deployments over just large, cumbersome model sizes. For developers seeking a balance of speed, accuracy, and practical utility, models like Qwen are increasingly compelling.
For those interested in replicating or extending these tests, I’ve shared the current vllm preset and build configurations used for speed and performance assessments on my GitHub repository: https://github.com/gogluejf/rig-stack
Final Thoughts
As local AI models mature, they are poised to reshape development workflows by providing faster, more accurate insights directly within the IDE or local environment—ushering in a new era of efficient, autonomous code comprehension.
