Introduction

My perfectly stable homelab server gradually descended into chaos — random crashes, segfaults, system hangs — and how months of troubleshooting, community support, and hardware swaps eventually led to one conclusion: the CPU itself was defective from the factory.

If you're experiencing unexplained instability on an Intel 13th or 14th Gen system, this story might save you months of frustration.

Quick Symptom Checklist

If you're seeing multiple of these symptoms, you may be affected:

Random segfaults in unrelated applications
System instability appearing after months of stability
Crashes or hangs under light load or while idle
VMs freezing while still marked as "running"
Issues persisting despite PSU, RAM, or OS checks or changes

The Setup

In November 2022, I built a homelab server with the following components:

Component	Model
CPU	Intel Core i7-13700K (Raptor Lake)
Motherboard	ASUS PRIME Z690-P D4 (Intel Z690, ATX, DDR4)
RAM	Patriot Viper Steel DDR4 3600 MHz C18, XMP 2.0
Cooler	Cooler Master Hyper 212 Black Edition
PSU	Corsair RM-850e
OS	Proxmox VE

The system ran Proxmox VE 7, later upgraded to v8, hosting a mix of Windows, Ubuntu, and Debian VMs, plus a couple of LXC containers. For over a year, everything was rock-solid. NUT (UPS monitoring), LAN bonding, automated backups — all worked flawlessly.

The Trouble Begins: A Timeline

August–September 2024: First Signs of Instability

After upgrading VirtIO drivers from 0.1.248 to 0.1.262, I started seeing x86/split lock detection errors. The entire Proxmox host would crash, taking down all VMs and containers.

Even though I didn’t fully trust that theory, I pointed a finger at the VirtIO upgrade — it was the only change I had made.

What the community said:

"Don't think VirtIO would make the whole host crash... this seems more like a hardware issue with storage or memory"
"Split lock is an indication of your VM doing some weird stuff... generally related to very buggy software/OS or faulty hardware — bad memory, bad CPU, bad power supply"
Suggestions: Run memtest, stress-test the CPU, check thermals, check fans

I disabled split lock detection as a workaround. The crashes continued.

September 2024: Testing Everything

Following community advice, I:

Ran PassMark memory tests — no errors
Updated the motherboard BIOS - I was one version before the latest
Ran stress-ng on the CPU — no errors
Replaced the PSU with a spare one I had.

October 2024: Stability... or So I Thought

The issue appeared to disappear.

On 17 October 2024, I reported back to the Proxmox forum that the system seemed stable. I waited a full month before posting to be sure. I assumed—incorrectly—that the component I had changed was the culprit.

The stability was temporary—the degradation was progressive.

December 2024 – January 2025: New Symptoms Emerge

New problems surfaced:

A Windows Server 2022 VM would hang during backup restarts
QEMU guest agent fs-freeze/fs-thaw commands would time out
VMs appeared "running" but were unresponsive
Console showed improper shutdown states

I tried:

Switching backup modes
Changing machine types (q35 → i440fx)
Updating QEMU

Nothing resolved the issue.

February 2025: The Penny Drops

While continuing to seek help on Reddit, someone pointed me to the Intel 13th/14th Gen instability issue. The symptoms matched perfectly:

Random crashes under varied workloads
Segfaults in unrelated binaries
System hangs after running for days
Progressive worsening over time

It was suggested that I run the Intel Processor Diagnostic Tool (IPDT) (https://www.intel.com/content/www/us/en/support/articles/000005567/processors.html).

This tool is Windows-only. Intel should have created a utility that is OS-agnostic making it something you could run from a bootable USB drive.

Having to jump through hoops to perform a test for the company's product is counterproductive.

Stress tests like mprime / Prime95 didn’t expose the issue.

March 2025: Intel RMA Process

Armed with the information, I hopped over to the Intel Community (https://community.intel.com/) and asked about the process.

At https://www.intel.com/content/www/us/en/support/articles/000057098/processors.html I found all the information to start the process.

You need to provide:

Processor number
ATPO
FPO

If you no longer have the box, you’ll need to remove the CPU and clean off the thermal paste to read the markings.

When the Intel support agent asked me for the following additional information (from Intel's email):

Motherboard: Please include the model and any relevant details.
Power Supply Unit (PSU): Model, wattage, and manufacturer.
Dedicated Graphics Card: Model and manufacturer, if applicable.
Thermal Solution Details (Air Cooler or a Liquid Cooler):
Operating System Details:
Have you tested this processor on a different motherboard, if yes, on which motherboard was it tested?
Have you tested your current motherboard with a different compatible processor, if yes, which processor have you used?

The process was excellent.

I live in Malta, an island at the edge of Europe, where cross-border logistics usually take time.

Key dates:

16 March 2025: Case opened / CPU information provided / collection process initiated
18 March 2025: DHL collected the defective CPU
20 March 2025: Replacement CPU delivered by DHL

Thermal Paste

Make sure you have thermal paste on hand. If you need to read information off the CPU (and using the machine while your case is being processed) and when you mount the new CPU on your motherboard.

Replacement CPU

The replacement CPU arrived in Intel-branded packaging.

Post-Replacement: Problems Resolved

After installing the replacement i7-13700K, all previously experienced problems disappeared. Two months after replacing the CPU, the system returned to the rock-solid stability it had enjoyed during its first year of operation.

The Bigger Picture: Intel's Raptor Lake Defect

My experience was not isolated. It was part of one of the largest CPU reliability crises in recent computing history. Earlier in my troubleshooting process, I had seen references to CPU-related problems, but:

CPU tests passed
in over 40 years I had never encountered a time-delayed CPU defect

What Went Wrong

Intel's 13th Gen ("Raptor Lake") and 14th Gen ("Raptor Lake Refresh") desktop processors suffered from a fundamental defect that caused progressive, irreversible degradation. The issue affected high-performance SKUs — primarily the Core i5, i7, and i9 K/KF/KS variants with the 8P+16E core configuration.

Root Cause: Vmin Shift Instability

On 25 September 2024, Intel employee Thomas Hannaford posted the official root cause analysis on the Intel Community forums (https://community.intel.com/t5/Blogs/Tech-Innovation/Client/Intel-Core-13th-and-14th-Gen-Desktop-Instability-Root-Cause/post/1633239)

Intel localized the problem to a clock tree circuit within the IA core that is particularly vulnerable to reliability aging under elevated voltage and temperature. These conditions lead to a duty cycle shift of the clocks, causing system instability.

Intel identified four operating scenarios that lead to Vmin shift in affected processors:

1. Motherboard Power Delivery Exceeding Intel Guidance

Mitigation: Intel Default Settings recommendations for 13th/14th Gen desktop processors. It is common practice for motherboards to exceed these settings and they have done so for years. It is normally set by default.

2. eTVB Microcode Algorithm Issue

The Enhanced Thermal Velocity Boost algorithm was allowing i9 desktop processors to operate at higher performance states even at high temperatures.

Mitigation: Microcode 0x125 (June 2024)

3. SVID Algorithm Requesting High Voltages

The microcode's Serial Voltage Identification algorithm was requesting high voltages at a frequency and duration that caused Vmin shift.

Mitigation: Microcode 0x129 (August 2024)

4. Elevated Core Voltages During Idle/Light Activity

Microcode and BIOS code were requesting elevated core voltages especially during periods of idle and/or light activity — exactly the condition a homelab server experiences most of the time.

Mitigation: Microcode 0x12B, which encompasses 0x125 and 0x129, and addresses elevated voltage requests during idle and/or light activity periods

Intel confirmed that mobile processors and future product families (Lunar Lake, Arrow Lake) are unaffected by the Vmin Shift Instability issue.

Manufacturing Oxidation (Early Units)

For some early 13th Gen processors (manufactured in late 2022 — exactly when I purchased mine), there was an additional manufacturing defect involving oxidation. Intel confirmed this was identified internally in late 2022 and addressed in production by early 2024, but on-shelf inventory with the defect may have persisted into early 2024.

The Critical Point: Damage Is Irreversible

Once a processor has been exposed to excessive voltage for long enough, the damage to the clock tree circuit is permanent and cannot be repaired by any software update. Microcode patches can only prevent further damage on CPUs that haven't yet degraded — they cannot restore already-damaged processors.

This is why my system's stability gradually worsened over time, and why replacing the PSU only appeared to help temporarily.

Why My Homelab Was the Perfect Victim

Looking at Intel's four identified scenarios, my Proxmox homelab hit the worst-case combination:

Scenario 4 (idle/light activity): A homelab server spends most of its time in light-load or idle states — exactly when the faulty microcode was requesting the highest inappropriate voltages
Scenario 3 (SVID high voltage requests): The constant power-state transitions of a virtualization host (VMs starting, stopping, idling) triggered frequent SVID voltage requests
Early manufacturing (oxidation): Purchased November 2022, squarely in the window for the oxidation manufacturing defect
Always-on operation: Running 24/7 meant maximum cumulative exposure to the damaging conditions

The May 2025 microcode update (0x12F) specifically addressed "systems continuously running for multiple days with low-activity and lightly-threaded workloads" — a near-perfect description of a homelab server.

Intel's Response: A Timeline

Date	Action
Late 2023 – Early 2024	Community reports of instability begin accumulating
April 2024	Intel recommends motherboard manufacturers use "Intel Default Settings"
July 2024	Intel officially acknowledges elevated voltage as the cause
June 2024	Microcode 0x125 released — fixes eTVB algorithm issue
August 2024	Microcode 0x129 released — addresses high voltage requests
August 2024	Intel announces 2-year warranty extension (3 years → 5 years) for affected SKUs
September 2024	Intel identifies root cause (clock tree circuit / Vmin Shift)
September 2024	Microcode 0x12B released — "final" fix encompassing 0x125 + 0x129 + idle voltage control
October 2024	Intel confirms the voltage issue was the sole root cause
November 2024	Class action lawsuit filed in San Jose, California
May 2025	Microcode 0x12F released — addresses edge cases in systems running continuously for multiple days with light workloads

Warranty Extension

Intel extended the warranty from 3 years to 5 years for all affected boxed 13th/14th Gen desktop processors (https://community.intel.com/t5/Mobile-and-Desktop-Processors/Additional-Warranty-Updates-on-Intel-Core-13th-14th-Gen-Desktop/m-p/1620853).

Key points from the announcement:

The extension applies to new and previously purchased processors
Coverage applies to all customers globally
The warranty eligibility period starts on the original purchase date and does not reset if Intel provides a replacement
Intel committed to supporting all customers experiencing instability symptoms through the exchange process

Affected Processors

The instability primarily affects desktop processors with the 8P+16E Raptor Lake silicon:

13th Gen (Raptor Lake):

Core i9-13900K/KF/KS
Core i7-13700K/KF
Core i5-13600K/KF

14th Gen (Raptor Lake Refresh):

Core i9-14900K/KF/KS
Core i7-14700K/KF
Core i5-14600K/KF

Lower-power variants (non-K, mobile) were less commonly affected but not entirely immune.

Lessons Learned

For Users Experiencing Instability

Don't assume it's software. If your system was stable for months and gradually becomes unstable, hardware degradation is a real possibility — especially with Intel 13th/14th Gen CPUs.
The symptoms are deceptive. The crashes manifest as memory errors, storage corruption, split lock violations, segfaults in random binaries — anything that looks like "something else" is broken. This is because the CPU is making computation errors.
Run the Intel Processor Diagnostic Tool / equivalent. Download it from Intel's support site. If your CPU fails, you have clear evidence for an RMA. Remember that a Pass is not a sign that your CPU is not impacted.
Don't waste money replacing other components first. I had a perfectly good PSU to replace but didn't have spare RAM, storage and a motherboard lying about. While testing RAM and storage is reasonable, be aware that a degrading CPU can make other components appear faulty.
Check your warranty status. Intel extended the warranty to 5 years for affected 13th/14th Gen desktop processors. If you purchased after October 2022, you likely have coverage through at least 2027.
Update your BIOS/microcode. If your CPU hasn't yet degraded, microcode 0x12B (or newer) can prevent the excessive voltage that causes damage. Check your motherboard manufacturer's website for the latest BIOS. A replacement is ultimately the best solution if you are eligible.

Conclusion

What started as random crashes turned out to be a widespread hardware issue.

The community played a critical role in identifying the root cause.

People who dedicate time to maintaining forums and helping others are a key part of the ecosystem—thank you.

If you're reading this because your 13th or 14th Gen Intel system is acting up take action. The fix exists, the warranty coverage is there, and Intel's replacement process works. Don't spend months chasing ghosts like I did. The clock is running out and these CPUs will eventually no longer be covered for replacement.

Search This Blog

TT(O)M

When Your CPU Dies: My Journey with a Defective Intel Core i7-13700K