"It is impossible to design a system so perfect that no one needs to be good." - TS Eliot The quote attributed to T.S. Eliot highlights the philosophical and ethical understanding that no matter how well-designed a system is, it cannot entirely compensate for the moral character and intentions of the individuals interacting with it. In the context of AI, this quote suggests that while we can strive to create advanced and robust AI systems with many safeguards, these systems cannot fully preempt or prevent all potential misuse by individuals who are determined to behave unethically. AI systems rely on human users and developers to use them responsibly and ethically. This is why the role of human oversight, ethical guidelines, and continuous monitoring is crucial in the development and deployment of AI technologies. It's a reminder that technology alone cannot solve all problems related to human behavior and that ethical use is a shared responsibility.

Finally stress tested the 4× DGX Spark rack properly. This wasn’t really about chasing benchmark numbers. I wanted to know if the rack itself was doing its job when all four Sparks were working hard at the same time mainly airflow, thermals, power behavior and whether one unit was getting punished by its position in the rack. Methodology: • Same production Docker image on all 4 Sparks • Same 8192×8192 BF16 GEMM workload on each node • 30 second isolated run per Spark first • Then a synchronized 60 second run with all 4 Sparks loaded at the same time • GPUs capped at ~2400 MHz • GPU telemetry sampled throughout the runs: SM clock, power, temperature, utilization and P-state • Results compared against the 4 node median rather than treating one node as the reference The isolated phase was meant to establish whether any Spark was inherently slower or hotter on its own. The simultaneous phase was the real rack test: same workload, same time, all four nodes producing heat and drawing power together. That’s the phase that can expose airflow imbalance, a bad rack position, thermal interaction between units or a node losing clocks under full system load. Single node tests were extremely close, with only ~0.5% spread between the four units and temperatures between 69–72°C. With all four running together: • 92.51 TFLOP/s cluster median • only ~1.3% spread between fastest and slowest • 76–78°C max GPU temperatures • 87–89 W average GPU power per node • ~96% GPU utilization • 100% P0 residency under load • peak clocks around 2.37–2.39 GHz The important part: there was no meaningful performance drop when the whole rack was loaded, no thermal outlier, and no Spark behaving differently from the others. Only 2°C separated all four nodes under full load. I also checked the MikroTik CRS812 in the same rack: • CPU: 38°C • switch: 44°C • PHY: 37°C • SFP: 68°C • board sensors: 30 / 35 / 40°C • all 4 chassis fans OK, running ~6630–7230 RPM • PSU1: OK, 49.8 W, 27°C • PSU input: 234.2 V • PSU2: no input / intentional in my current setup So the result is really about the whole rack, not just four GPU numbers. The Sparks stay very closely matched under simultaneous load, airflow looks balanced, the CRS812 is thermally comfortable, and there’s no sign that any device is being disadvantaged by its position in the rack. That’s what I wanted to validate.

Finally got DeepSeek-V4.1-Flash-1017 running properly across my 4× NVIDIA DGX Spark cluster. Repo: github.com/knapcio/DeepSeek-… Model: deepseek-ai/DeepSeek-V4.1-Flash-1017 revision: 1bae0e1088993b9b82de08f59de5be5a34e42170 My setup: • 4× DGX Spark / GB10 • TP4 / EP1 • SGLang canary • 2× 200GbE RoCE rails through MikroTik CRS812 • RoCEnante • B12X-next routed MoE • DSpark speculative decoding, block size 5 • packed local Engram shards on every node • FP8 KV cache • 1,048,576 context • 16 concurrent requests Production settings: • CHUNKED_PREFILL_SIZE=8192 • MEM_FRACTION_STATIC=0.78 • HEAD_MEM_FRACTION_STATIC=0.78 • DSV41_CACHE_GIB=4 • NCCL_IB_TC=104 • NCCL_PROTO=^LL128 • NCCL_MAX_NCHANNELS=8 Also patched the older RoCEnante path so it actually honors NCCL_IB_TC=104 instead of silently using TC0 Runtime validation: • RoCEnante active on both rails • B12X next active at EP1 • shared expert K pad 576→640 accepted by b12x • RoCE compact gather active • all 4 containers healthy • qeval: 71/75 sparkDash results: Structured C1: 157.0 tok/s - TTFT 229 ms Prose C1: 87.7 tok/s - TTFT 221 ms Code C1: 142.4 tok/s - TTFT 235 ms Code C1 @ 64K context: 126.7 tok/s - TTFT 411 ms Prefill: 64K: 5,953.7 tok/s 128K: 6,026.2 tok/s 256K: 5,137.9 tok/s This is the first DeepSeek V4.1 Flash setup on my Sparks where the whole stack - model, RoCE, NCCL, MoE, Engram and speculative decoding -feels properly tuned as one system rather than a collection of individual optimizations. Very impressive work in the @majewskizby repo.

Spent some time tuning the 4× NVIDIA DGX Spark fabric today, and this is the result I wanted to see. Setup: 4× DGX Spark / NVIDIA GB10 MikroTik CRS812 dual-RoCE fabric NCCL 2.30.7 + CUDA 13.0 nccl-tests 2.21.1 (afd59ab) 1 GPU per node 4 MPI ranks AllReduce 32 KiB → 256 MiB 20 warmup iterations 100 measured iterations per size Final 256 MiB result: 15.27 GB/s algorithm bandwidth 22.90 GB/s NCCL bus bandwidth 22.89 GB/s in-place 0 validation errors That is roughly 183 Gbit/s bus-bandwidth equivalent on the 200 GbE RoCE path. The more interesting part was not the peak number, though. It was understanding why the default NCCL tuning was leaving performance on the table. With NCCL fully automatic, the mid-size region was surprisingly weak. Around 1–8 MiB the performance could collapse badly even though large transfers eventually reached ~22 GB/s. Using NCCL TUNING logs showed what was happening. The automatic tuner was progressively increasing the number of channels: 128 KiB → 8 channels 256 KiB → 16 channels 512 KiB → 32 channels 1 MiB and above → 64 channels On this particular 4-node DGX Spark topology, 64 channels were simply too aggressive for the medium-size AllReduce path. The best reproducible configuration turned out to be: NCCL_PROTO='^LL128' and NCCL_MAX_NCHANNELS=8 Important detail: ^LL128 means exclude LL128, not force it. With that tuning, the curve became much healthier: 512 KiB → 6.87 GB/s busbw 1 MiB → 12.17 GB/s 2 MiB → 17.07 GB/s 4 MiB → 19.95 GB/s 8 MiB → 20.54 GB/s 16 MiB → 21.66 GB/s 32 MiB → 22.27 GB/s 64 MiB → 22.57 GB/s 128 MiB → 22.74 GB/s 256 MiB → 22.90 GB/s Full sweep average: 13.8526 GB/s bus bandwidth And most importantly: # Out of bounds values : 0 OK I also tested the newer NCCL_MAX_CTAS=8 control. It worked, but on this exact NCCL 2.30.7 / GB10 setup it was not as fast as the older NCCL_MAX_NCHANNELS=8 setting, particularly in the 512 KiB–2 MiB region. So for now I am keeping the configuration that the hardware actually benchmarks best with, rather than changing it just because a newer control exists. One more useful lesson from this: A healthy 200 GbE fabric is not only about reaching a good peak number. You need to look at the entire bandwidth curve. A system can hit ~22–23 GB/s at 256 MiB and still have a serious tuning problem at 1–8 MiB. NCCL’s algorithm/protocol/channel decisions matter a lot. For this 4× DGX Spark cluster, the current reproducible baseline is: 22.90 GB/s NCCL bus bandwidth ~183 Gbit/s equivalent 0 correctness errors Now that is a fabric result I’m happy to keep as the baseline.

One of the more frustrating issues I hit while building my 4 × NVIDIA DGX Spark cluster had nothing to do with the GPUs. It was the 10GbE management network. The Sparks use a Realtek RTL8127 NIC, and under sustained traffic the stock r8127 11.014.00-NAPI driver was showing very unhealthy RX behaviour. This wasn't just an iperf number looking bad. On Spark1 I was seeing tests like: ~36,000 UDP packets lost rx_mac_error increasing by roughly the same amount rx_mac_missed also moving meanwhile the MikroTik CRS812 showed no corresponding FCS/drop problem That was an important clue: the switch and cable path looked clean. The packets were effectively being lost at the host RX side. Disabling EEE helped somewhat loss dropped from roughly 36K to ~22K packets in one test but clearly did not solve the underlying problem. Then came an interesting workaround. I tested: net.ipv4.tcp_sack=0 With TCP SACK enabled, long TCP tests could fall into ugly 5–10 second periods with throughput effectively dropping to zero and the congestion window collapsing. With SACK disabled, Spark1 completed a 120 second test at about 8.11 Gbit/s without those hard stalls. A later 300 second validation also held around 8.11 Gbit/s. So tcp_sack=0 was genuinely useful. But it was NOT the fix. The 300 second test still produced ~26,929 retransmissions. SACK-off was basically preventing the TCP stack from reacting catastrophically to the packet reordering/loss pattern coming from the NIC path. I eventually deployed that workaround across all four Sparks because stability mattered more than theoretical protocol purity: net.ipv4.tcp_sack = 0 But I kept looking for the actual cause. The real improvement came from moving Sparks to the newer custom Realtek r8127 11.016.00-NAPI-RSS driver and fixing the receive side configuration: RSS enabled 8 RX / 8 TX queues corrected RX CPU affinity RXMITI01 tuning GRO enabled DASH disabled persistent warm reboot handling That changed the character of the problem completely. The catastrophic RX missed/error growth stopped, UDP became dramatically cleaner, and TCP became stable in the 8+ Gbit/s range. Some representative results: single-stream TCP: ~7.2–8.2 Gbit/s 4 streams: ~8.5 Gbit/s 8 streams: ~8.9 Gbit/s Spark → Mac: ~9.05 Gbit/s Spark ↔ Spark: ~9.4 Gbit/s In one particularly clean UDP run I saw roughly 5.38 Gbit/s with only 48 packets lost out of ~4.64 million around 0.001%. That is a very different system from the one dropping tens of thousands of packets at the RX MAC. Later I also performed the full NVIDIA/DGX platform update across all four Sparks. That is now part of the cluster baseline, but I keep the two issues conceptually separate: the NVIDIA platform update and the RTL8127 driver/RSS work are not the same fix. The biggest lesson from this debugging session: tcp_sack=0 was a good mitigation, not a cure. It removed the nasty TCP stalls and made the machines usable while debugging, but the retransmissions were telling us the underlying problem was still there. The actual fix had to happen lower in the stack at the NIC driver, queueing, RSS and interrupt affinity level. And this is exactly why looking only at “iperf says 8 Gbit/s” can be misleading. Always look at: retransmissions NIC hardware counters RX missed/error counters queue distribution IRQ affinity switch counters packet loss Throughput tells you how fast the system went. The counters tell you whether it got there cleanly. 4 x Sparks DGX OS 7.6.0 kernel 7.0.0-1019-nvidia NVIDIA driver 580.178.04 EC 3.13.7 SoC / UEFI 2.158.15 USB-PD 0.5.25 tcp_sack 0

After I finally found some time, I sat down and moved my cluster into a proper rack. 4× NVIDIA DGX Spark 1× MikroTik CRS812 2× 400G → 2×200G DAC breakouts Dedicated 10GbE management 2 shelves, rack PDU, cable management all packed into a compact 8U rack on wheels. The 400G breakouts from the CRS812 feed all four Sparks at 200G, while the separate management network keeps administration clean. A lot of compute in a very small footprint. 🤖⚡

Reduce your attack surface

Things are getting worse every day.