lemon👑 @lemonDefi1

Qwen3.8-27B is running locally on an RTX 5060 Ti with 16GB VRAM, connected over USB4. This test used the Q3_K_M quantization with a context window of around 65K tokens. The model fits within the GPU’s memory budget, making it possible to run a 27B-parameter model on a consumer

FHILY👑 @Oluwaphilemon1

6 hours ago

Qwen3.8-27B is running locally on an RTX 5060 Ti with 16GB VRAM, connected over USB4. This test used the Q3_K_M quantization with a context window of around 65K tokens. The model fits within the GPU’s memory budget, making it possible to run a 27B-parameter model on a consumer

3 4 13 3K 5

Qwen3.8-27B is running locally on an RTX 5060 Ti with 16GB VRAM, connected over USB4. This test used the Q3_K_M quantization with a context window of around 65K tokens. The model fits within the GPU’s memory budget, making it possible to run a 27B-parameter model on a consumer GPU. A few hardware observations stood out: GPU temperature: Around 63°C. The exhaust air felt only lukewarm, suggesting the cooling setup was handling the workload without producing particularly hot airflow. GPU utilization: Consistently above 90%. The GPU remained heavily occupied throughout the run. USB4 connection: Despite using USB4 rather than a direct PCIe connection, there was no obvious sign of a severe hardware bottleneck in this test. However, utilization and temperature alone cannot establish whether USB4 affects throughput. That would require comparing generation speeds against a direct PCIe connection under the same conditions. There was one problem with the run. I forgot to set a turn limit. Instead of stopping after a defined number of agent interactions, Hermes kept going until the context was exhausted. With a context window around 65K tokens, that can mean a lot of accumulated conversation, tool outputs, and reasoning before the session finally stops. If you’re running Hermes with a similar setup, set a turn limit before starting: hermes --max-turns 30 The intended limit here is 30 turns. Check your installed Hermes version’s CLI help if the option differs. The interesting part of this test is how accessible local inference has become. A 27B model, a 16GB consumer GPU, and a USB4 connection are enough to run this setup locally. Just remember that fitting the model is only part of the job. Quantization, context length, GPU memory pressure, and agent turn limits all affect how practical the experience will be. x.com/UtaAoya/status…

Qwen Flash generated a game in about 60 minutes using a single M4 Mac mini with 16GB of unified memory paired with an RTX 3090. The reported generation speed was around 22 tokens per second.

FHILY👑 @Oluwaphilemon1

6 hours ago

Qwen Flash generated a game in about 60 minutes using a single M4 Mac mini with 16GB of unified memory paired with an RTX 3090. The reported generation speed was around 22 tokens per second. And this was a one-shot attempt. One prompt, one run, no extended cycle of manual

2 3 4 2K 2

Qwen Flash generated a game in about 60 minutes using a single M4 Mac mini with 16GB of unified memory paired with an RTX 3090. The reported generation speed was around 22 tokens per second. And this was a one-shot attempt. One prompt, one run, no extended cycle of manual fixes. There’s still room to improve the setup. I think we could potentially cut the generation time significantly, possibly doubling the effective speed by the end of the week. That remains a target, not a measured result, but it will be interesting to see how far the setup can go with further optimization. The hardware combination is worth paying attention to. The M4 Mac mini provides the compact host system, while the RTX 3090 brings 24GB of dedicated GPU memory to the setup. Depending on how inference is distributed between the two machines, this can offer more flexibility than relying on the Mac mini alone. One important distinction: 22 tokens per second is the reported generation speed, not necessarily the speed of the entire game-building process. Planning, tool calls, code execution, and any validation or repair work can all affect the final completion time. Still, finishing a game generation run in roughly an hour with a one-shot workflow is a useful starting point. If you bought a Mac mini during the Clawdbot craze, your purchase might not have aged quite as badly as you thought. x.com/0xSero/status/…

Qwen3.8-27B Uncensored is generating 3D models in Blender on a Ryzen AI Max+ 395 system, and the token generation speed is better than expected. What’s interesting here is the workload.

FHILY👑 @Oluwaphilemon1

6 hours ago

Qwen3.8-27B Uncensored is generating 3D models in Blender on a Ryzen AI Max+ 395 system, and the token generation speed is better than expected. What’s interesting here is the workload. Generating a text response is one thing. Creating a 3D model through Blender involves

0 2 17 2K 12

Qwen3.8-27B Uncensored is generating 3D models in Blender on a Ryzen AI Max+ 395 system, and the token generation speed is better than expected. What’s interesting here is the workload. Generating a text response is one thing. Creating a 3D model through Blender involves translating an idea into geometry, writing scripts, configuring objects, and potentially iterating on the result when something doesn’t work as intended. A model can produce code quickly and still struggle to create a usable scene. The real test is whether it can turn a prompt into a working Blender project with the right geometry, materials, lighting, and composition. The Ryzen AI Max+ 395 makes this experiment particularly interesting because it’s an integrated platform with a large unified memory pool. That gives local models more room to run than many conventional systems with limited GPU memory, although the memory available to the model still depends on the operating system, runtime, and other applications. Using an uncensored fine-tune adds another dimension to the experiment. It changes the model’s refusal behavior, but that doesn’t automatically make it better at 3D modeling. Code quality, spatial reasoning, and the ability to recover from Blender errors still determine how useful it is. The next things I’d want to measure are actual tokens per second, time to produce the finished scene, and how much manual correction the model needs. Local AI-assisted 3D modeling is an interesting workload to push on consumer hardware, especially when the model can generate Blender scripts directly. The speed is encouraging. Now the question is how reliably it can turn that speed into finished 3D assets. x.com/posi_posi8/sta…

Qwen3.8-Flash-Next can reportedly generate at 100 to 140 tokens per second on an RTX 5090 using Strata. Even an RTX 5070 Ti can reach around 50 to 70 tokens per second. That is already interesting for local inference. But what if you want to run an uncensored version of the model without hunting down a special fine-tune or converting model weights yourself? I initially assumed there were two options: modify Qwen3.8-Flash-Next to reduce its refusal behavior, or find an uncensored version on Hugging Face and convert it to work with Strata. Both approaches introduce extra work. Model modification requires experimentation, while converting an existing fine-tune can involve compatibility issues and additional setup. It turns out Strata already has an experimental option for censorship removal built into its setup process. How to enable it On Windows, open a terminal in your Strata folder and run: START-HERE.bat --setup --experimental-speed-projection on On Linux: ./setup.sh --setup --experimental-speed-projection on According to the reported procedure, you don’t need to download the model again. Rerun the installer with the experimental option enabled. Linux compatibility notes The Windows procedure reportedly works directly. On Ubuntu 22.04 and 24.04, the Linux setup script can automatically build the required components before launching the application. Other distributions may not pass the automatic build process. The tester used a Docker container to build and run the setup on Ubuntu 26.04. What happened to the refusal rate? The RTX 5090 test reported these results: • Rejection rate: 73% to 0% • False rejection rate: 0% to 0% • GSM8K accuracy: 95% to 95% The tester also reported that the model stopped refusing prompts in their RTX 5070 Ti setup. The GSM8K result is particularly worth noting. In this test, reducing refusals did not reduce the reported score on that maths benchmark. There is an important distinction, though. A zero-refusal result means the model stopped rejecting the tested prompts. It does not establish that every answer is correct, that performance is unchanged across all tasks, or that the modification has no other behavioral effects. Likewise, GSM8K measures mathematical problem-solving, not general reasoning, coding ability, or factual reliability. The practical advantage is that you may not need to locate, download, and convert a separate uncensored checkpoint just to experiment with a less restrictive version of Qwen3.8-Flash-Next. Strata’s built-in option gives you another route to test, using the model you already have installed. For anyone experimenting with local models on consumer GPUs, that is a useful feature to know about.

FHILY👑 @Oluwaphilemon1

7 hours ago

Qwen3.8-27B reportedly went from 82 to 381 tokens/sec on a single RTX 3090 with 24GB VRAM. No GPU upgrade. The gains came from optimizing the inference stack. Performance progression: Baseline: 82 tok/s Optimized MTP: 114 tok/s DFlash2 + context lookup: 138 tok/s Optimized

2 3 21 3K 23

Bonsai 2 9B has gone from 144.9 to 913.1 tokens per second on Apple Silicon in five days of open optimization work. That is a reported 442.3% increase in decode speed. The NVIDIA track is seeing a similar jump. On the DGX Spark, CUDA optimizations have delivered a reported 401.1% improvement. Both platforms are now generating at several times their starting speed, and the lead has shifted between Mac and NVIDIA throughout the week. At the latest update, Apple Silicon is ahead. These results are coming from an open optimization challenge organized by PrismML and DarkbloomAI on Yukon Research, using mlx .fast. Five days into the challenge, the leaderboard has recorded: • 13 solvers • 113 promoted runs • Optimization agents powered by Opus 5.5, GPT-6 Astra, and DeepSeek-V4 The competition is not just about finding one clever kernel or changing a single inference parameter. Participants are experimenting with ways to make the model execute more efficiently on different hardware, with each promoted improvement available for others to examine and build upon. The contributor behind the update also reported that their agents reached the top position twice overnight. Those runs currently sit at third and fifth place, while their Saturday submission produced a 41.9% single-run improvement on the Mac track, the largest individual jump reported there so far. Bonsai 2 9B is an open-weight model with a reported model size of around 2GB in this configuration. Getting more than five times the original decode throughput without changing the basic premise of running the model locally makes a substantial difference to the experience. Faster decoding can make interactive generation more responsive and reduce the time spent waiting for output. There is an important distinction between decode speed and overall workload performance, however. These numbers describe token generation, not necessarily prompt processing, end-to-end task completion, or benchmark accuracy. The exact hardware, inference configuration, and measurement methodology also matter when comparing results. The most interesting part is the optimization process itself. People are using their own machines and AI coding agents to improve open inference software, and the resulting work is being shared rather than kept behind a private API. Every new best run gives the next contributor a stronger starting point. Five days in, both the Mac and NVIDIA tracks have made substantial progress. The challenge is still underway, so these figures represent the current leaderboard rather than a final result.

FHILY👑 @Oluwaphilemon1

7 hours ago

Qwen3.8-27B reportedly went from 82 to 381 tokens/sec on a single RTX 3090 with 24GB VRAM. No GPU upgrade. The gains came from optimizing the inference stack. Performance progression: Baseline: 82 tok/s Optimized MTP: 114 tok/s DFlash2 + context lookup: 138 tok/s Optimized

2 3 21 3K 23

Qwen3.8-27B is challenging the assumption that you need a massive model to get strong reasoning and coding performance. The interesting part is its position on the size-versus-intelligence chart. A 27B-parameter model is appearing near the Pareto frontier, the boundary where improving one metric usually requires sacrificing another. Models below that frontier offer a worse trade-off. They may be larger without being more capable, or smaller without delivering comparable performance. A model on the frontier gives you a more competitive balance between capability and resource requirements. Why does Qwen3.8-27B stand out? At 27 billion parameters, it occupies a different hardware category from models with hundreds of billions of parameters. With appropriate quantization, it can be practical to run locally on consumer GPUs with around 16GB to 24GB of VRAM, depending on the quantization level, context length, and inference engine. Some configurations may require additional system RAM or GPU offloading. That changes the economics of experimenting with capable models. You can run coding agents, test multi-step reasoning, generate applications, and build local workflows without automatically relying on a hosted frontier model for every task. There are trade-offs, of course. Quantization can affect quality, long contexts increase memory requirements, and a model’s benchmark position doesn’t guarantee equal performance across every real-world task. The chart also needs context: which benchmarks define intelligence, how model size is measured, and whether all models were evaluated under comparable conditions can change the apparent frontier. Still, Qwen3.8-27B is an interesting example of why parameter count alone is becoming a less useful way to judge an open-weight model. The question is no longer simply how large a model can be. It’s how much capability you can get from the hardware you already own.

FHILY👑 @Oluwaphilemon1

7 hours ago

Qwen3.8-27B reportedly went from 82 to 381 tokens/sec on a single RTX 3090 with 24GB VRAM. No GPU upgrade. The gains came from optimizing the inference stack. Performance progression: Baseline: 82 tok/s Optimized MTP: 114 tok/s DFlash2 + context lookup: 138 tok/s Optimized

2 3 21 3K 23

Qwen3.8-27B reportedly went from 82 to 381 tokens/sec on a single RTX 3090 with 24GB VRAM. No GPU upgrade. The gains came from optimizing the inference stack. Performance progression:

FHILY👑 @Oluwaphilemon1

7 hours ago

Qwen3.8-27B reportedly went from 82 to 381 tokens/sec on a single RTX 3090 with 24GB VRAM. No GPU upgrade. The gains came from optimizing the inference stack. Performance progression: Baseline: 82 tok/s Optimized MTP: 114 tok/s DFlash2 + context lookup: 138 tok/s Optimized

2 3 21 3K 23

Qwen3.8-27B reportedly went from 82 to 381 tokens/sec on a single RTX 3090 with 24GB VRAM. No GPU upgrade. The gains came from optimizing the inference stack. Performance progression: Baseline: 82 tok/s Optimized MTP: 114 tok/s DFlash2 + context lookup: 138 tok/s Optimized lookup verification: 381 tok/s That’s roughly 4.6× the original throughput, but there’s an important catch: the 381 tok/s result came from reproducing text already present in the input, not ordinary chat. What made the difference? 1. DFlash2 speculative decoding Instead of generating every token sequentially, the system proposes multiple tokens and lets the main model verify them together. 2. Context lookup When the expected output overlaps with the prompt, the system can reuse matching text as draft tokens. The developer reports verifying up to 16 tokens per block, with around 15 accepted on the target workload. For ordinary conversation, reported generation speed is closer to 133 tok/s. 3. Prefix caching Repeated questions about the same 25K-token document can reuse previously processed prompt data. Reported time to first token fell from 22.4 seconds to 0.56 seconds, around 40× faster. Where this matters These optimizations are particularly useful for document processing, code transformations, retrieval-augmented generation, and repeated queries against long prompts. The results demonstrate how much software optimization can improve local inference without changing the GPU. Just remember that context-heavy workloads can achieve very different speeds from open-ended generation. x.com/TeksEdge/statu…

Qwen3.8-27B can reportedly reach around 70 tokens per second on a single NVIDIA DGX Spark, and the full recipe is now available for others to reproduce. The release goes beyond sharing a speed number. It includes the configuration, deployment files, benchmark results, and optimization details needed to understand how the performance was achieved. Here’s what’s included: 1. The complete SGLang launch configuration The repository provides the launch script for the SGLang inference engine, with explanations for each configuration parameter. That matters because inference performance depends on more than the model itself. Memory allocation, batching, scheduling, attention kernels, and other runtime settings can all influence throughput. Having the actual configuration makes it easier to reproduce the result and understand which settings contribute to performance. 2. A Dockerfile for the inference engine The project includes the Dockerfile used to build the engine image. This gives users a defined environment for setting up the runtime instead of having to reconstruct the software stack from scratch. 3. A 4-bit speculative decoding drafter The setup includes a 4-bit drafter for speculative decoding. The idea is to use a smaller, faster model to propose candidate tokens while the main model verifies them. When the proposals are accepted, the system can generate multiple tokens with less sequential work than ordinary decoding. The actual speedup depends on how often the drafter’s proposals are accepted and how much verification costs. 4. Six benchmarks and the underlying CSV files The release includes six benchmarks, with CSV files containing the measurements behind the reported results. That makes it possible to inspect individual numbers rather than relying entirely on a headline throughput figure. It also gives other developers a starting point for comparing configurations and investigating performance differences. 5. Kernel-tuning cache optimization One particularly interesting detail is the kernel-tuning cache. The author reports that this optimization produces a 16% improvement by addressing variability in the initial kernel-selection process, described as the boot lottery. In practical terms, the tuning process can influence which kernels the runtime selects. Capturing and reusing the relevant tuning results can make performance more consistent instead of relying on whichever configuration happens to be selected during startup. The exact gain will depend on the hardware, software versions, and baseline used for comparison. You can delegate the setup to a coding agent The README also includes a prompt designed for coding agents. Instead of manually working through every installation step, you can point Claude Code or Codex at the repository on your DGX Spark and have the agent download the required files, build the environment, launch the model, and verify the setup. You’ll still want to inspect the commands and results, but this lowers the barrier to reproducing a fairly involved inference configuration. Why this release is useful A throughput claim is much more valuable when other people can reproduce it, inspect the configuration, and determine which optimizations actually matter. With the scripts, Dockerfile, drafter setup, benchmark data, and agent-assisted installation instructions, this project provides a practical starting point for running Qwen3.8-27B on the DGX Spark. The reported target is around 70 tokens per second on a single machine. The repository gives the community a way to test that claim, understand the trade-offs, and build on the work.

FHILY👑 @Oluwaphilemon1

7 hours ago

Qwen3.8-27B reportedly went from 82 to 381 tokens/sec on a single RTX 3090 with 24GB VRAM. No GPU upgrade. The gains came from optimizing the inference stack. Performance progression: Baseline: 82 tok/s Optimized MTP: 114 tok/s DFlash2 + context lookup: 138 tok/s Optimized

2 3 21 3K 23

A 74B-parameter MoE model running on an RTX 3060 with just 12GB VRAM and 16GB system RAM. The reported results for Nemotron-3.5-Bolt-74B-A9B are impressive for such a constrained setup:

FHILY👑 @Oluwaphilemon1

7 hours ago

A 74B-parameter MoE model running on an RTX 3060 with just 12GB VRAM and 16GB system RAM. The reported results for Nemotron-3.5-Bolt-74B-A9B are impressive for such a constrained setup: • Context window: 262K tokens • Decode speed: ~30 tokens/sec • Prefill speed: ~800

2 4 12 2K 11

A 74B-parameter MoE model running on an RTX 3060 with just 12GB VRAM and 16GB system RAM. The reported results for Nemotron-3.5-Bolt-74B-A9B are impressive for such a constrained setup: • Context window: 262K tokens • Decode speed: ~30 tokens/sec • Prefill speed: ~800 tokens/sec • GPU: RTX 3060, 12GB VRAM • System RAM: 16GB The setup uses llama.cpp with Unsloth’s 1-bit UD-IQ1_S GGUF quantization. Why the 1-bit quant is interesting The developer reports that Unsloth’s UD-IQ1_S file outperformed NVIDIA’s official 2-bit version in this setup. According to the report, it was 3.5GB smaller, faster, and delivered the same perplexity. Perplexity measures how well a language model predicts text. Matching perplexity is encouraging, but it doesn’t guarantee identical coding, reasoning, or instruction-following performance. Independent task benchmarks will be important. The practical advantage is memory. Compressing the weights more aggressively makes a much larger model accessible to hardware that normally couldn’t accommodate it. The configuration matters The setup uses --n-cpu-moe 57 to place 57 MoE layers’ experts on the CPU side, while -ngl 99 requests GPU offloading for as many layers as the runtime allows. The key flag is --load-mode none. The developer reports that this prevents CPU-resident experts from being evicted from RAM, helping avoid the memory-management behavior that can disrupt inference. The server also uses a 262,144-token context, Flash Attention, Q8 KV-cache types, and a single parallel sequence. One important distinction: supporting a 262K context doesn’t mean the entire context is inexpensive to run. The KV cache and other runtime allocations still consume memory, and actual performance depends on the workload and available system resources. Full llama.cpp configuration llama-server \ -m NVIDIA-Nemotron-3.5-Bolt-74B-A9B-UD-IQ1_S.gguf \ --load-mode none \ -ngl 99 \ --n-cpu-moe 57 \ -c 262144 \ -fa on \ --jinja \ -np 1 \ --cache-type-k q8_0 \ --cache-type-v q8_0 \ -b 2048 \ -ub 1024 \ --temp 1.0 \ --top-p 0.95 \ --top-k 50 \ --min-p 0.0 \ --repeat-penalty 1.0 The most interesting part is that the model’s total parameter count doesn’t tell the whole story. MoE architectures activate only a subset of their experts for each token, while quantization and CPU/GPU placement determine whether the full model can fit into a constrained system. At roughly 30 tokens/sec, this setup suggests that running a large MoE model locally may be possible without a high-end GPU, provided the quantization and memory configuration are carefully chosen. The developer says benchmark results are coming next. Those tests will help establish how well this configuration performs beyond raw generation speed. x.com/DogukanUrker/s…

Qwen3.8-27B and Ornith 1.5 35B have been put through a local benchmark on a single NVIDIA DGX Spark. The results are remarkably close. Qwen3.8-27B: 89.4 Ornith 1.5 35B: 88.7 Qwen takes the overall lead by just 0.7 points. But looking at the individual categories reveals a more interesting comparison. Neither model dominates every workload. Where Ornith 1.5 35B leads Coding: 85.8 Long-context tasks: 70.6 Agentic performance: 95.2 Tool use: 82.9 These results make Ornith worth considering for workflows involving code generation, extended context, and agents that need to interact with tools. Its 95.2 agentic score is particularly notable, although the underlying tasks and evaluation methodology determine how well that result translates to real-world agent work. Where Qwen3.8-27B leads Safety: 97.3 Robustness: 100 Planning: 95.5 Qwen’s results point to strengths in the evaluated safety and robustness tasks, alongside planning performance. Its perfect robustness score is worth examining more closely. A score of 100 means it achieved the maximum on that particular evaluation, not that it will never fail under unfamiliar conditions. Why this comparison matters The models have different parameter counts, with Qwen at 27B and Ornith at 35B. Yet their overall scores differ by less than one point. That is a useful reminder that parameter count alone doesn’t determine practical capability. Training, model architecture, fine-tuning, inference configuration, and the evaluation itself all influence the outcome. It also gives DGX Spark owners a more concrete basis for choosing a model according to their workload rather than relying on general reputation or headline benchmarks. If coding and tool-heavy agent workflows are your priority, Ornith’s category scores deserve attention. If planning, robustness, and the tested safety tasks matter more, Qwen’s results are relevant. The overall scores are close enough that the category breakdown is more informative than the headline ranking. Another important detail: the results are presented as verifiable and repeatable. The accompanying benchmark repository gives other users a way to inspect the methodology and reproduce the comparison rather than simply accepting the numbers at face value. For a fair comparison, it is still important to check the benchmark definitions, model versions, inference settings, and scoring procedure. Those details determine what the 0.7-point difference actually tells us.

FHILY👑 @Oluwaphilemon1

7 hours ago

A 74B-parameter MoE model running on an RTX 3060 with just 12GB VRAM and 16GB system RAM. The reported results for Nemotron-3.5-Bolt-74B-A9B are impressive for such a constrained setup: • Context window: 262K tokens • Decode speed: ~30 tokens/sec • Prefill speed: ~800

2 4 12 2K 11

Qwen3.8-27B Uncensored (Cyber/Agentic) is now available in GGUF, targeting developers who want to run cyber-focused AI agents locally. The release combines a 27B-parameter model, agent-oriented tuning, a reported 262K context window, and quantization calibrated using real usage patterns. Three details make it worth examining. 1. A quantized model targeting around 15GB of VRAM Running a 27B model locally can be demanding, particularly when you want enough memory left for long conversations and tool interactions. GGUF quantization reduces the memory required to store model weights by representing them at lower precision. This makes local deployment more accessible on consumer hardware, although the actual requirements depend on the quantization level and inference configuration. The reported 15GB VRAM figure is a target, not a guarantee for every setup. KV-cache allocation, runtime buffers, and context length can increase memory consumption substantially. 2. A 262K context window for extended agent sessions A large context window can help an agent work with lengthy technical documents, retain more conversation history, and examine larger portions of a codebase without repeatedly discarding earlier information. For cyber-focused workflows, that could be useful when interpreting terminal output, tracing errors across files, or following a multi-step investigation. However, a supported 262K context does not mean the entire window will fit into 15GB of VRAM at full speed. Memory requirements and throughput depend on the runtime, KV-cache precision, and other configuration choices. 3. Quantization calibrated around real usage This is arguably the most interesting technical detail. Quantization is not simply about making a model smaller. Compressing weights can also affect how well the model follows instructions, produces structured outputs, and performs particular tasks. Calibration helps the quantization process identify which model behaviors and internal activations need to be preserved. For an agentic model, that matters because its job extends beyond generating readable text. It may need to interpret command output, construct arguments, call tools, and follow structured instructions across multiple steps. Using real-traffic calibration data may help preserve these behaviors under compression. The actual benefit, however, depends on the quality and diversity of the calibration data and how the resulting model is evaluated. What does uncensored mean here? The release is described as zero-refusal, indicating that its tuning is intended to eliminate or substantially reduce refusals on prompts the model would otherwise reject. That is a change in response behavior, not proof of improved intelligence or accuracy. A model that answers more prompts can still misunderstand a request, generate an incorrect command, or produce unreliable code. When connected to a terminal or other tools, its outputs should be validated before execution. Why local developers should pay attention This release brings together several features relevant to local agent development: A 27B-parameter model in a quantized GGUF format. A reported context window of up to 262K tokens. Cyber-focused and agent-oriented tuning. Quantization reportedly calibrated using real usage patterns. A claimed VRAM target of approximately 15GB. The real test is whether these features work together under practical conditions. How much memory does the model consume at 128K or 262K context? Does quantization preserve reliable tool calls? How does it perform on actual coding and terminal tasks compared with the original model? Those measurements will tell us much more than the model’s size or refusal rate alone. For anyone experimenting with local AI agents, Qwen3.8-27B Cyber/Agentic is another model worth putting through a proper workload-specific evaluation.

FHILY👑 @Oluwaphilemon1

7 hours ago

A 74B-parameter MoE model running on an RTX 3060 with just 12GB VRAM and 16GB system RAM. The reported results for Nemotron-3.5-Bolt-74B-A9B are impressive for such a constrained setup: • Context window: 262K tokens • Decode speed: ~30 tokens/sec • Prefill speed: ~800

2 4 12 2K 11

Qwen3.8-27B still appears to outperform Meta’s newly released 30B dense model in this comparison. Meta’s release gives local AI enthusiasts another option, particularly those working with budget-conscious GPUs. A model in this parameter range can be attractive for developers who want to run capable AI locally without relying entirely on hosted APIs. But similar parameter counts don’t guarantee similar capabilities. The architectural difference matters. A dense model uses its full set of model layers for each token, while a mixture-of-experts (MoE) model activates selected experts for each token. MoE models can reduce computation per token, although their full weight footprint can still be substantial. That distinction affects inference speed, memory requirements, and how practical each model is on consumer hardware. I also added Qwen3.8-27B to the original comparison because it was missing from the initial table. With Qwen included, Meta’s new 30B model still appears to trail it in the results being compared. For local AI users, the next questions are straightforward: Which model delivers stronger coding and reasoning performance? How much VRAM does each quantized version require? Which one generates tokens faster on the same GPU? How well does each handle long contexts and agentic workflows? The exact model versions, quantization levels, hardware, and benchmark methodology matter before drawing firm conclusions. For now, Qwen3.8-27B remains a useful reference point for evaluating Meta’s new 30B dense model, especially if your goal is to get the most practical performance out of consumer hardware.

FHILY👑 @Oluwaphilemon1

6 hours ago

A 125B-parameter model running on a single RTX 4090 with 24GB VRAM, generating around 21 tokens/sec at a 250K context window. That’s what a developer reports achieving with Qwen3.8-Flash-Next using an experimental llama.cpp build. No MTP. No DFlash2. No KV-cache quantization.

0 2 4 2K 5

Qwen3.8-Flash-Next GSQ-RCO IQ3_S tested on Strata v0.2.0-beta, and my first results show a prefill improvement at shorter context lengths. I tested the beta, released on October 19, on my home GPU server and compared it against Strata v0.1.45, my everyday version. Hardware and configuration: RTX 5070 12GB + RTX 5060 Ti 16GB 28GB total VRAM across two GPUs 64GB system RAM, with 56GB allocated to WSL2 Expert resident RAM budget: 33 GiB I used the same startup settings and inputs for both versions. Prefill performance At approximately 14K input tokens: v0.1.45: 417–440 tok/s v0.2.0-beta: 486 tok/s That’s roughly a 10–17% improvement over the previous version’s measured range. At approximately 33K input tokens: v0.1.45: 512 tok/s v0.2.0-beta: 508 tok/s At this longer context length, performance was essentially unchanged. Decode speed was also roughly the same across both versions. The initial results suggest Strata v0.2.0-beta may improve prompt processing at shorter context lengths on this dual-GPU setup, without a noticeable change in generation speed. However, I only ran one or two measurements on v0.1.45, so these numbers are preliminary. More repeated tests across different context lengths are needed to establish whether the improvement is consistent. For now, the shorter-prompt results look encouraging, but I wouldn’t call this a general performance upgrade just yet.

FHILY👑 @Oluwaphilemon1

6 hours ago

A 125B-parameter model running on a single RTX 4090 with 24GB VRAM, generating around 21 tokens/sec at a 250K context window. That’s what a developer reports achieving with Qwen3.8-Flash-Next using an experimental llama.cpp build. No MTP. No DFlash2. No KV-cache quantization.

0 2 4 2K 5

Qwen3.8-27B reportedly shrank from 55.6GB to just 5.9GB using PrismML’s Bonsai 2, while retaining 98.2% of its aggregate performance across 20 benchmarks. That’s an approximately 89.4% reduction in model size, with the reported benchmark retention improving on the roughly 95% achieved by the previous Bonsai 27B version. Here’s the comparison: Original Qwen3.8-27B: 55.6GB Bonsai 2 compressed version: 5.9GB Aggregate benchmark performance retained: 98.2% Previous Bonsai 27B version: approximately 95% retention If independent testing confirms these figures, this could make a large language model considerably easier to deploy on hardware with limited memory. How does Bonsai 2 achieve this? The technology behind it is ternary quantization. Traditional model weights use numerical representations with many possible values. Ternary quantization reduces the available weight values to three, typically -1, 0 and +1, alongside scaling factors that help preserve information about the original weights. This substantially reduces the number of bits needed to represent the model. The challenge is preserving what those weights have learned. Aggressive quantization can damage a model’s ability to reason, write code, follow instructions and handle complex tasks. A compressed model might still produce convincing answers while quietly becoming less reliable on demanding workloads. Bonsai 2 reportedly maintains a high proportion of the original model’s benchmark performance despite the drastic reduction in size. PrismML also reports improvements over its previous Bonsai version in areas such as coding agents, multimodal reasoning and long-horizon tool use. Agentic coding is an especially interesting test. A coding agent doesn’t simply answer a question and stop. It may need to inspect an unfamiliar repository, trace dependencies across multiple files, modify code, execute tests, diagnose failures and revise its implementation. Each step depends on the model maintaining enough context and reasoning ability to make sensible decisions throughout the workflow. Preserving those capabilities in a heavily compressed model could make local AI agents more practical on consumer hardware, where memory is often the main constraint. There is one important caveat: a 5.9GB model file does not mean you only need 5.9GB of memory to run it. Inference also requires memory for the KV cache, temporary buffers, activations and other runtime operations. Longer context windows can increase those requirements considerably. The 98.2% figure also represents aggregate performance across 20 benchmarks. It doesn’t establish that the compressed model matches the original on every individual task, or that it will retain the same performance across every inference engine and hardware configuration. Independent testing is still needed to establish its real-world strengths and weaknesses. Still, the reported improvement from approximately 95% benchmark retention in the previous Bonsai version to 98.2% is notable. The real challenge in model compression is not merely making weights smaller. It’s making them smaller while preserving the capabilities people actually need. If Bonsai 2 consistently achieves that balance, running a 27B-class model locally could become much more accessible without requiring a large GPU memory budget.

FHILY👑 @Oluwaphilemon1

6 hours ago

A 125B-parameter model running on a single RTX 4090 with 24GB VRAM, generating around 21 tokens/sec at a 250K context window. That’s what a developer reports achieving with Qwen3.8-Flash-Next using an experimental llama.cpp build. No MTP. No DFlash2. No KV-cache quantization.

0 2 4 2K 5

Qwen3.8-27B reportedly improved from 51.7% to 86.52% on Terminal-Bench 2.1 using a recursive self-improvement method developed by Tencent Hunyuan. That is a gain of 34.82 percentage points, achieved through targeted post-training rather than increasing the model’s parameter count. The method is called Terminal-RSI. Its goal is to improve how reliably an AI agent completes real tasks through a terminal, including situations where it must execute multiple commands, handle errors, and verify its results. The interesting part is how it uses the model’s own successful work to improve future performance. How Terminal-RSI works The training process alternates between two stages, repeating the cycle twice. 1. Recursive Self-Rewrite (RSR) The model takes verified successful execution trajectories and rewrites them into training examples for supervised fine-tuning (SFT). Instead of treating a successful run as a one-off achievement, the system turns it into data that can teach the model to reproduce that behavior. 2. Policy Refinement (RPR) Reinforcement learning then refines the model’s behavior. The process inherits critic information from the previous stage to guide optimization. The model is trained not just to produce plausible commands, but to improve its policy based on feedback about its actions and outcomes. The cycle then repeats: RSR-1: Generate training examples from verified successes. RPR-1: Refine the model through reinforcement learning. RSR-2: Use the improved model to generate another round of training data. RPR-2: Refine the policy again using the accumulated improvements. Each stage builds on the preceding one, allowing the model’s successful behavior to become a stronger part of its learned capabilities. The reported benchmark results Tencent Hunyuan reports substantial improvements across several terminal-agent evaluations: Terminal-Bench 2.1: 51.70% → 86.52% Terminal-Bench 4.0: 0% → 19.84% Terminal-Bench Hard: 79% Long-Horizon Terminal-Bench process reward: 0.21 → 0.37 Strict long-horizon completion: 0/46 → 5/46 SWR100: Reported success rate doubled The paper also reports an average improvement of roughly 30 percentage points across multiple benchmarks. These measurements capture different aspects of performance, so they should not be compared as if they were the same metric. The long-horizon results deserve particular attention. Although the model improved from zero successful runs to five out of 46 under the strict completion criterion, that still leaves most attempts unsuccessful. That illustrates the gap between improving an agent’s behavior and making it consistently reliable on extended tasks. Why this approach matters for AI agents A terminal agent needs more than command-generation skills. It may have to inspect a codebase, locate the relevant files, modify configurations, install dependencies, execute tests, interpret error messages, recover from failures, and confirm that the original objective has been achieved. A model can perform several of these steps correctly and still fail the overall task. Terminal-RSI attempts to address that problem through several mechanisms: Verified trajectory rewriting: Successful execution histories become structured training examples. Critic weight inheritance: Critic information carries forward into later refinement stages, helping guide optimization. TB-GAE credit assignment: Generalized advantage estimation helps distribute learning signals across sequences of actions. Success-Prioritized Reward: The reward strategy emphasizes successful completion rather than treating partial progress as equivalent to finishing the task. Together, these techniques aim to improve the relationship between an agent’s actions and its final outcome. What this says about smaller models Qwen3.8-27B demonstrates why parameter count alone is not enough to understand agent capability.

FHILY👑 @Oluwaphilemon1

6 hours ago

Qwen3.8-27B is running locally on an RTX 5060 Ti with 16GB VRAM, connected over USB4. This test used the Q3_K_M quantization with a context window of around 65K tokens. The model fits within the GPU’s memory budget, making it possible to run a 27B-parameter model on a consumer

3 4 13 3K 5