The Open-Source Model Evaluation Checklist CTOs Are Missing Before They Commit to a Stack

Enterprise teams evaluating open-weight models in 2026 are making a common and costly mistake: treating benchmark leaderboards as procurement criteria. The real production risks sit elsewhere, in training data provenance, multimodal component composability, the infrastructure constraints that headline context figures conceal, and the licensing conditions that create compliance exposure at scale. The latest open-source releases expose exactly where these gaps sit, and they provide a structured basis for building an evaluation checklist that goes beyond accuracy scores.

Companion piece to our broader work on model evaluation frameworks. See Beyond Benchmarks: How CTOs Should Actually Evaluate New Model Releases Before Committing to Them for coverage of benchmark literacy, architecture trade-offs, and inference cost realities.

Training Data Transparency Is Not Optional

The first question to ask about any open-weight model is not what it scored on MMLU. It is whether you can trace what it was trained on and whether that data was collected with appropriate consent.

Most open-weight releases provide partial data cards at best. The practical consequence is that enterprise legal and compliance teams cannot assess IP exposure, and regulated industries cannot satisfy audit requirements. A model trained on scraped web data with no opt-out mechanism creates a different risk profile than one built on documented, consent-respecting pipelines.

Apertus 1.5 from Swiss AI is one of the few current releases that takes a materially different position here. The model is trained exclusively on fully open data, and the project explicitly describes methods to respect retroactive opt-out consent from data owners and to prevent memorisation of training data. That is a meaningful differentiator for any organisation operating under GDPR, sector-specific data regulations, or internal IP governance policies. It does not eliminate due diligence, but it substantially reduces the surface area of risk.

Context Window Claims Require Infrastructure Qualification

A 262K or 1M token context window is only useful if your infrastructure can actually serve it at production latency and cost. The headline figure and the operational reality are frequently not the same number.

Apertus 1.5 supports up to 262,144 tokens by default. GLM-5.2-Vision, through its NVFP4 quantised implementation, advertises a 1M token context, but the serving configuration reveals the constraint directly: 1M context requires eight B200 GPUs, while 256K context requires four. That is a hardware requirement that most enterprise teams cannot satisfy without significant capital commitment or cloud spend, and it needs to be modelled before the architecture decision is made.

The evaluation question is not "what is the maximum context?" but "what context length is achievable at acceptable cost on the hardware we can realistically provision?" Teams should benchmark attention memory scaling across their target document lengths and factor KV cache size into total cost of ownership calculations before treating long-context capability as a deployment-ready feature.

Multimodal Composability and the Frozen-Component Risk

Integrated vs. Bolted-On Architectures

The multimodal landscape has split into two distinct architectural patterns: models built with multimodal training from the ground up, and models where a vision or audio encoder has been connected to an existing text backbone via a trained projector. Both can perform well on benchmarks. They have different risk profiles in production.

GLM-5.2-Vision illustrates the second pattern directly. MoonViT from Kimi-K2.6 is connected to the frozen GLM-5.2 text backbone through a 49.5M-parameter PatchMerger projector. The text weights and vision tower are both frozen and byte-identical to their upstream releases. The only trained component is the projector mapping patch embeddings into the text model's token space. This is an efficient and transparent approach, but it means that the vision capability is entirely dependent on the projector generalising well to your specific input distribution.

What Frozen Weights Mean for Fine-Tuning

If your use case requires domain adaptation, frozen component architectures constrain your options. You cannot fine-tune the vision encoder or the text backbone independently without breaking the projector alignment. Apertus 1.5 takes a different path, training multimodal capability into the model across a 4T token multimodal mix for the 8B variant, which means the text and vision representations are jointly optimised from the start. The practical implication is that fine-tuning surface and adaptation flexibility differ significantly between these two architectural choices, and that difference matters for teams building specialised document processing, medical imaging, or industrial inspection pipelines.

Licensing Conditions and the Contact-Information Trap

Apache 2.0 is widely understood as a permissive licence, but the access conditions attached to specific releases can add obligations that are not visible in the licence identifier alone.

Apertus 1.5 is distributed under Apache 2.0 alongside an Acceptable Use Policy, and accessing the model files requires agreeing to share contact information, specifically email and username, with the repository authors. A privacy policy governs how that information is used. For most individual researchers this is a minor friction point. For enterprise procurement teams, it introduces a data-sharing obligation that needs to be reviewed against internal vendor management and data minimisation policies before deployment.

The checklist item here is not to avoid models with access conditions, but to surface those conditions explicitly during evaluation rather than discovering them at the point of deployment. Licence identifier, acceptable use policy, and data-sharing requirements are three separate documents that all need review.

Tooling Ecosystem and Serving Infrastructure Readiness

A model that requires custom serving plugins to run in any standard inference framework is not production-ready in the same sense as one with upstream support. This is a frequently overlooked dimension of open-model evaluation.

GLM-5.2-Vision requires an out-of-tree SGLang plugin because the Glm5vForConditionalGeneration architecture is not yet merged upstream. The plugin ships with the model repository, which reduces friction, but it means your serving infrastructure is now dependent on a non-standard code path that will not receive upstream maintenance automatically. Any future SGLang updates require manual compatibility checks.

The evaluation question is whether the model has first-class support in the inference frameworks your organisation already uses, or whether you are accepting a custom integration dependency. Teams should audit vLLM, SGLang, and TGI compatibility before committing to a model, and should specifically check whether support is upstream or maintained by a third party. Token efficiency tooling, speculative decoding support, and quantisation compatibility all fall into the same category: they determine whether the model fits into your existing data-to-deployment pipeline or requires you to build around it.