WeChat’s WeLM project is quietly redefining what efficient scaling looks like for large language models at a moment when everyone else is racing toward ever larger and more expensive systems. By focusing on Chinese language performance, tight integration with the WeChat ecosystem, and clever use of sparse architectures plus new decoding tricks, the team is showing that there is still room for architectural innovation rather than brute force scaling alone.
From early WeLM to a mature large model stack
WeLM did not begin as a frontier scale assistant but as a focused research model. In 2022 the WeChat AI team released WeLM as a ten billion parameter Chinese language model trained on a curated corpus that included over ten terabytes of raw text and about 260 billion tokens. On eighteen Chinese monolingual benchmarks the original WeLM significantly outperformed other pre-trained models of similar size and even matched the performance of models up to twenty-five times larger. That early work also showed strong ability on multilingual and code-switching tasks, beating many multilingual models trained on far more languages. This success highlights the importance of dedicated safety teams in ensuring robust model performance.
A 10B Chinese WeLM trained on vast curated data rivaled models 25× larger while excelling at multilingual code-switching
What looked at first like a one-off research effort has since evolved into an internal foundation model family for WeChat. Multiple reports confirm that newer generations of WeLM now power Xiaowei, the conversational assistant embedded inside WeChat, rather than Tencent’s group level Hunyuan model. The project has iterated through several versions, with the current WeLM V4 version described as the main engine behind Xiaowei in production.
This long-running trajectory matters for two reasons. First, WeLM was one of the earliest general-purpose Chinese large models released around the time ChatGPT captured global attention, but it stayed relatively low profile outside China. Second, the team has used that time to experiment with more efficient architectures tailored to WeChat’s reality of more than one billion users, where inference cost and latency are not academic details but core business constraints.
The new WeLM: sparse mixture of experts at practical scale
The latest public details describe WeLM as a highly sparse mixture of experts model designed specifically for the WeChat ecosystem rather than as a general cloud product. Instead of pushing toward several hundred billion or trillion parameters in a dense configuration, the team focuses on an eighty billion parameter mixture of experts model referred to as 80B A3B, along with a deeper one hundred thirty billion parameter variant.
In this setup, only about three billion parameters are active for each token during inference, even though the full model holds around eighty billion parameters. That is typical of mixture of experts designs, where a routing mechanism sends each token to a small subset of experts, keeping per token compute low while preserving total model capacity. For WeLM, the team also reports using techniques such as loss-free load balancing and an unnormalized sigmoid gate together with a shared expert to keep routing stable and utilization even across experts.
Training scale is another interesting signal for how the team views the scaling law. The eighty billion parameter WeLM is reported to train on fewer than fourteen trillion tokens, which is relatively modest compared with some recent English-centric frontier models that push far beyond that level. Yet WeChat’s own evaluations claim that this model outperforms other systems of similar or even larger scale on Chinese tasks, continuing the pattern first seen in the ten billion parameter generation.
To extend capacity further without redesigning everything, the team applies what they call depth upscaling. By effectively repeating the middle part of the transformer stack, they build a one hundred thirty billion parameter version that shows clear performance gains after a relatively small continuation of training at a low learning rate. This is an incremental but meaningful example of architectural scaling: deepen the network in a targeted way instead of simply widening it or throwing vastly more data and compute at the problem.
Taken together, these details amount to a deliberate stance on scaling. Parameter count is important, but the WeLM team is trying to find the sweet spot where sparsity, depth, and careful training data selection yield the best tradeoff between capability, latency, and cost for a very large but mostly domestic user base.
The most novel part of the new WeLM work is the idea of Hidden Decoding. In conventional autoregressive decoding, sequence length is visible to the user. If you want more reasoning or more detail, you typically ask the model to produce more tokens. That increases both latency and cost roughly in proportion to the number of generated tokens.
Hidden Decoding attacks that constraint from another angle. Instead of extending the user-visible sequence, it lets each token travel through a longer internal trajectory before it is emitted. The decoded text length remains fixed, but every token can traverse multiple latent steps that deepen its internal computation. In other words, the model spends more compute per token by expanding an internal hidden sequence rather than by making the output longer.
To do this, tokens are organized into several processing streams that evolve in parallel. Within each stream, token representations move through additional hidden positions that act like a latent sequence dimension that is much longer than the visible text. The extra compute budget is therefore spent on the evolution of internal states, giving the model more chances to refine its intermediate representations before deciding on the final output token. Throughput and response length as seen by the user can remain roughly unchanged, while the internal path each token takes becomes richer and more expressive.
This is conceptually similar to how many human writers think. One can draft, edit, and refine silently before saying anything out loud. Hidden Decoding formalizes a similar idea for language models by separating internal reasoning length from external response length. It also implicitly proposes a new axis in the scaling law: not just more parameters or more data, but more internal steps per output token. In practice, the WeLM-HD4-80B and WeLM-HD4-617B models show that this kind of sequence-length scaling at 100B+ MoE scale, combined with continued pretraining, can monotonically lift MMLU scores while slightly reducing bits-per-byte on Pile-style evaluations.
A longer latent sequence brings a familiar problem. Standard transformer attention cost grows roughly quadratically with sequence length. If the model simply extended the internal sequence in the naive way, the compute and memory cost would explode. This would be unacceptable for a mixture of experts model operating at more than eighty billion parameters and serving production traffic for an application with over one billion active users.
Stream Factorized Attention addresses that issue by restructuring how attention operates along the hidden sequence dimension. Most layers in the stack perform attention only within individual processing streams. Only a carefully chosen subset of layers is allowed to mix information across streams. Full sequence attention is thus confined to a small fraction of the model.
This design yields an important computational property. Because full sequence mixing is rare, the effective attention cost grows roughly in proportion to the number of streams, rather than with the square of the total latent length across all hidden steps. In practical terms, the model can extend the internal sequence quite far while keeping memory usage and inference time under control. For a large sparse model like WeLM, that means the team can add depth in the hidden dimension without breaking the latency budget needed for messaging style interactions.
The combination of Hidden Decoding and Stream Factorized Attention therefore amounts to an internal scaling law. You can increase the complexity of token-level reasoning by adding more latent steps, and you can do so in a way that keeps compute growth close to linear in the number of streams rather than quadratic in the total hidden sequence length. That is a very different picture from the straightforward scaling of dense transformers simply by parameter count and visible context length.
How this connects to the broader WeChat AI strategy
WeChat’s use of WeLM as the primary model behind Xiaowei reveals a clear strategic choice. Reports note that Xiaowei relies mainly on WeLM, with support from external models such as DeepSeek for some responses, rather than on Tencent’s group level Hunyuan model. For an application with more than one point four billion users, that suggests WeChat values tight vertical integration and fine-grained control over both performance and cost.
The sparse mixture of experts approach fits this environment well. Activating only a small portion of the model for each token keeps per request compute modest, which is crucial given the message volume in WeChat. At the same time, the total parameter pool remains large enough to capture diverse patterns in Chinese language usage across social chat, official accounts, mini programs, and business workflows that live inside the app.
The WeChat AI team is also exploring other architectures focused on efficient inference. A separate effort called WeDLM applies a diffusion-inspired framework to language modeling with the explicit goal of faster parallel generation while preserving reasoning quality. While technically distinct from WeLM, both efforts point in the same direction. The aim is not to chase headline-grabbing parameter counts but to find architectures that deliver strong reasoning under strict latency and hardware constraints.
In that sense, WeLM’s new decoding and attention strategies can be viewed as the sequence depth counterpart of what WeDLM is to parallel sampling: targeted architectural changes that bend the traditional scaling curves of transformers in favor of real-world deployment.
Opportunities and risks in this new scaling regime
The ideas behind Hidden Decoding and Stream Factorized Attention have several promising implications.
First, they invite the community to treat internal computation depth as a first-class scaling axis. Instead of always asking models to produce longer chains of thought, research can focus on enriching the hidden computation that happens between visible tokens. That could produce more stable and concise answers with less need for verbose reasoning prompts.
Second, these techniques may unlock better value for mid-sized infrastructure. By making internal sequence scaling more compute efficient, they open the door for organizations to get better reasoning from models that are not at the absolute frontier in size but are carefully designed around mixture of experts and latent depth. This matches WeChat’s need to serve huge traffic on budget, and similar patterns will matter for other consumer applications.
Third, this approach could influence how future benchmarks are designed. If models can perform more reasoning per visible token, simple metrics based on output length or few-shot prompts may no longer fully capture their capabilities. Evaluation may need to account for the possibility that two models with similar output lengths have very different internal compute footprints and reasoning paths.
There are also clear risks and open questions. A more complex internal trajectory makes the model harder to interpret and debug. Hidden sequences and multiple streams of token evolution introduce new failure modes that are not immediately visible in the output. Careful monitoring and safety evaluations will be required, especially when such models interact with financial features and high-stakes services inside super apps.
Moreover, the benefits of Hidden Decoding may depend strongly on training regime and data quality. If the internal steps are not well aligned with the training objective, additional latent computation could amplify spurious patterns instead of improving reasoning. The mixture of experts architecture introduces its own sensitivity to routing stability, expert underuse, and potential bias if some experts specialize on particular user segments.
Finally, it remains to be seen how transferable these ideas are to other languages and domains. WeLM’s core strength has always been Chinese language understanding, especially in the context of WeChat specific usage. Extending the same architecture to multilingual scenarios or to code and tool use at scale may require further adaptation.
What to watch next
For practitioners and decision makers, WeLM’s latest evolution offers several concrete lessons.
- Efficient scaling is becoming as important as raw scaling. WeChat is showing that a carefully engineered eighty to one hundred thirty billion parameter sparse model with smart decoding can compete with much larger dense systems on targeted workloads.
- Internal computation depth is emerging as a new lever. Hidden Decoding and related ideas suggest that the next wave of gains may come from structuring how models think between tokens, not only from bigger parameter counts or longer contexts.
- Ecosystem specific models will matter more. WeLM is tuned to the language, traffic patterns, and product surfaces of WeChat, and that tight coupling is likely a key part of its real-world effectiveness.
Looking forward, the most interesting question is whether the scaling law implied by WeLM’s design generalizes. If a combination of sparse mixture of experts, internal sequence scaling, and factorized attention can repeatedly deliver better capability per unit of compute, it will shape how both Chinese and global players design their next generation models. If not, these techniques may still leave a lasting mark by clarifying the limits of architecture level improvements and pushing the field toward deeper understanding of where the real bottlenecks lie.
Either way, WeLM demonstrates that even in a landscape dominated by headline parameter counts, there is still room for thoughtful engineering choices that align capability, efficiency, and product reality inside one of the world’s most heavily used applications.
Conclusion
Tencent’s decision to run its WeLM model at the heart of WeChat’s new Xiaowei assistant is a quiet but important shift in how artificial intelligence is being scaled in the real world. Instead of chasing the largest possible model on paper, Tencent is showing what it looks like when a frontier model is deeply tuned to a massive consumer ecosystem and backed by smarter use of compute and context.
Why WeLM inside WeChat matters now
WeChat is a national level super app in China, touching messaging, payments, services, entertainment and more for roughly 1.4 billion users across WeChat and Weixin. Embedding a production AI assistant into that fabric is not just another product launch, it is a test of whether large models can operate at huge scale without collapsing under latency and cost.
Reports indicate that Xiaowei is currently in limited internal testing, with a broader rollout expected around the third quarter. During these tests, most interactions are powered by Tencent’s own WeLM large language model, while some queries are routed to external models such as DeepSeek, giving Tencent a hybrid stack that balances quality, cost and control. This mix is a practical example of scaling AI not only through bigger models but also through model orchestration and ecosystem design.
A short history of scaling laws
Over the past few years, scaling laws have become one of the central ideas in large model development. Early work showed that the loss of language models tends to decrease in a predictable power law relationship as researchers increase three main inputs: model parameters, training data and compute. In simplified form, this work suggested that doubling parameters or data reduces error by a roughly fixed percentage, and that these gains remain stable over many orders of magnitude.
These empirical laws helped teams plan budgets and decide whether to spend more on bigger models, more data or more training steps, rather than tuning everything by trial and error. At the same time, researchers and practitioners have noted that these laws are still curve fits over data rather than fundamental physical rules, and their validity can break down once models saturate on certain tasks or domains. Community discussions have emphasized that treating scaling laws as absolute can be misleading, especially when they are applied without attention to mechanisms, data quality or deployment context.
Against this backdrop, Tencent’s WeLM work can be read as an attempt to push scaling laws into a more practical direction, where the focus is not only on training curves but also on product integration and test time compute.
Tencent’s ecosystem first scaling strategy
WeLM is a large language model developed by the WeChat team, designed from the outset to fit the WeChat ecosystem rather than dominate generic benchmarks. The latest generation uses a highly sparse mixture of experts architecture, which activates only part of the parameters during each inference so that compute per request stays manageable while the effective capacity remains high. On top of that, the team has added optimizations such as KV Mirror, grouped query attention and multi token prediction, all aimed at keeping single round inference cost low enough to serve a user base that can reach into the billions while meeting quality targets.
Technically, this is a significant evolution from earlier WeLM versions. The previous generation reportedly included a V3 configuration with around 258 billion total parameters and 22 billion active parameters per token, while earlier work started from autoregressive models closer to 10 billion parameters. The new WeLM line centers on an 80 billion parameter A3B sparse mixture of experts model trained on fewer than around fourteen trillion tokens, with a deeper 130 billion variant created through depth scaling and limited additional training. Performance is said to match or exceed systems of similar or even larger reported sizes, which reinforces the point that architecture and training strategy can be more important than raw headline size.
From the deployment side, Tencent is not relying on WeLM alone. Company statements and independent reporting indicate that Xiaowei uses multiple models, including WeLM for core dialogue and WeChat specific scenarios and more cost effective open source models such as DeepSeek for generic knowledge tasks. This kind of routing lets Tencent reserve its most capable internal model for interactions that truly depend on WeChat context, long term memory and fine grained product integration, while offloading simpler queries to cheaper systems.
This is an operational scaling law in action. Parameter growth is paired with careful allocation of inference compute and with explicit ecosystem adaptation, instead of a uniform push to run the biggest model for every query.
Latent computation scaling and Hidden Decoding
A separate strand of WeLM research explores how to scale the amount of computation per token without extending outputs or widening the transformer architecture. This is the idea behind Hidden Decoding and latent computation scaling.
In this approach, each token in the sequence is expanded into multiple internal streams during continued pretraining, each with its own embedding table. The model keeps the key value cache for these internal streams as context, so a single token can perform more internal computation before producing the final output representation. To make this affordable, the team introduces stream factorized attention, in which most layers attend only within streams and only a small number of layers mix information across streams, reducing attention cost from quadratic in the number of streams to a cost that is roughly linear.
Experiments described for WeLM show models such as WeLM HD4 80B and WeLM HD4 617B, where HD4 refers to four computation streams per token. At scales above one hundred billion parameters in mixture of experts form, these configurations improved on matched baselines that did not use Hidden Decoding, demonstrating sequence length scaling at frontier mixed expert scales. The key point is that instead of adding more parameters across the entire model, Tencent is folding extra compute into the sequence itself, making each token effectively think more deeply while keeping the visible output length unchanged.
This introduces another dimension to scaling laws. Compute is not only a function of parameters and training steps but can be allocated inside the sequence through latent computation, opening room to tune how much thinking each token does without making the user wait for longer responses.
From parameter races to contextual compute
Put together, WeLM’s architecture and deployment illustrate a broader transition in AI strategy. The dominant story of the past few years has often been framed as a race toward larger model sizes supported by bigger training datasets and massive training runs. In contrast, Tencent is making the WeLM family deeper and more specialized while keeping inference costs in a range that can serve a billion scale consumer application.
Inside WeChat, Xiaowei’s capabilities do not exist in isolation. The assistant is plugged into payment flows, messaging threads, mini programs and content, and it can act as an agent that calls external services on behalf of users. In that context, the main question is not how many parameters the model has but how intelligently the system allocates compute and data across different surfaces. Routing complex analytical work to DeepSeek, dedicating WeLM to WeChat specific reasoning and employing optimizations such as multi token prediction are all ways of turning abstract model capacity into practical functionality.
This blend of latent computation scaling and ecosystem integration suggests a new organizing principle. Future progress will depend less on single headline numbers and more on how well an organization can align its models with its data, user flows and product levers and on how flexibly it can dial inference compute up or down for different tasks.
What this means for technology and business
For other technology teams, the WeLM example underscores that building a successful assistant for a major platform is at least as much about systems engineering as about frontier model training. Tencent’s sparse mixture of experts and Hidden Decoding work show that it is possible to maintain high capability while controlling inference costs, which is crucial for any service that could receive billions of calls per day.
Businesses that operate large platforms can draw several lessons. First, a domain adapted model that is deeply aware of product context may outperform a generic model with more parameters but less integration. Second, a multi model architecture lets teams match different tasks to different cost and quality profiles rather than forcing a single model to handle everything. Third, investing in infrastructure for long term memory, knowledge bases and agent style orchestration can unlock more value from the same underlying models by giving them richer context to work with.
At the societal level, embedding an AI assistant into a super app used by over a billion people raises questions about information access, transparency and dependency. On one hand, users gain immediate access to generative tools in environments they already trust. On the other, there is a risk that decision making and discovery become increasingly mediated by a single platform’s assistant, with limited visibility into how models are trained, how data is used and how routing between models such as WeLM and DeepSeek is decided.
Risks, limitations and open questions
The reliance on scaling laws and latent computation raises several open issues. Empirical scaling relationships do not hold forever, and performance can plateau or even degrade when data becomes repetitive or misaligned with real user queries. Hidden Decoding adds architectural complexity that could create new failure modes or interpretability challenges, especially when each token carries multiple internal computation streams that are not directly visible.
From a governance perspective, Xiaomi style assistants running inside platforms like WeChat will need clear policies around data retention, personalization and the degree of autonomy granted to AI agents when they perform actions such as payments or bookings. Community discussions also remind us that calling these relationships laws may obscure the fact that they remain statistical regularities, not guarantees, which makes continuous evaluation and robust safeguards essential.
Finally, the dual use of proprietary and open source models raises competitive and regulatory questions. If Tencent’s approach proves highly effective, it could reinforce the centrality of super apps and large platforms in AI deployment, making it harder for smaller players to offer comparable assistants without similar ecosystems and compute resources.
Key takeaways and what to watch next
WeLM’s deployment inside WeChat’s Xiaowei assistant shows a mature approach to AI scaling that combines larger and more sophisticated models with careful control over inference compute and deep product integration. Tencent’s work on sparse mixture of experts architectures and latent computation through Hidden Decoding points to a future where progress comes from making each bit of compute smarter and more context aware, rather than simply piling on more parameters.
For practitioners, the main takeaway is that scaling laws are becoming more multidimensional. Effective scaling today means balancing training scale, inference cost, model routing and ecosystem adaptation. For observers, Xiaowei offers an early look at how AI assistants might operate when they are woven into national level platforms and backed by large internal models like WeLM plus complementary external systems.
The next milestones to watch will be Xiaowei’s public rollout, user adoption patterns and technical disclosures around how Tencent continues to evolve WeLM’s architecture and its scaling strategies. Those details will reveal whether this ecosystem first path to scaling can match or surpass the more traditional race toward ever larger standalone foundation models. reddit
