Following Amazon CEO Andy Jassy’s announcement of AWS’s monumental $50 billion investment agreement with OpenAI, Amazon extended an invitation for a private tour of the chip development lab central to this partnership, covering most of the expenses.
The Trainium chip, developed at this facility, is being closely monitored by industry specialists due to its potential to reduce AI inference costs and possibly challenge Nvidia’s dominant market position.
Intrigued, I accepted the invitation.
My guides for the day included Kristopher King, the lab’s director (shown below right), Mark Carroll, the director of engineering (below left), and Doron Aronson, the team’s PR representative who organized the visit (featured with me later in this piece).
AWS has served as Anthropic’s primary cloud platform since the AI lab’s inception, a partnership robust enough to endure Anthropic’s subsequent collaboration with Microsoft as a cloud provider, alongside Amazon’s expanding alliance with OpenAI.
Through the OpenAI agreement, AWS becomes the sole provider for OpenAI’s new AI agent builder, Frontier, which could significantly boost OpenAI’s operations if agents gain the expected traction in Silicon Valley. However, the exact terms of this exclusivity are uncertain, as The Financial Times recently reported this week that Microsoft might consider OpenAI’s deal with Amazon a breach of its own existing agreement, which grants Redmond access to all of OpenAI’s models and technology.
Why is AWS so attractive to OpenAI? Under this agreement, the cloud behemoth has committed to providing OpenAI with 2 gigawatts of Trainium computing power. This represents a substantial pledge, especially since both Anthropic and Amazon’s proprietary Bedrock service are already utilizing Trainium chips faster than Amazon can manufacture them.
The company stated that 1.4 million Trainium chips across all three generations are currently in use, with Anthropic’s Claude leveraging over 1 million of the deployed Trainium2 chips.
Notably, while Trainium was initially designed for more efficient and cost-effective model training (a primary focus a few years back), it is now also optimized and employed for inference. Inference, which involves executing an AI model to produce outputs, represents the most significant performance limitation in the industry today.
For instance: Trainium2 manages most of the inference workload on Amazon’s Bedrock service, which facilitates the creation of AI applications for Amazon’s numerous enterprise clients and enables these applications to utilize various models.
King commented, “Our customer base is just expanding as fast as we can supply capacity.” He further suggested that “Bedrock has the potential to become as substantial as EC2,” referencing AWS’s enormous compute cloud offering.
Trainium vs. Nvidia
In addition to providing an alternative to Nvidia’s frequently backordered and difficult-to-obtain GPUs, Amazon asserts that its new chips, operating on its specialized Trn3 UltraServers, reduce running costs by up to 50% for equivalent performance compared to traditional cloud servers.
This AWS team, alongside the December release of Trainium3, also developed innovative Neuron switches, a combination Carroll describes as revolutionary.
Carroll stated, “What that gives us is a tremendous advantage.” The switches enable each Trainium3 chip to communicate with every other chip in a mesh setup, thereby lowering latency. “That’s why Trainium3 is setting new benchmarks,” especially regarding “price per power,” he added.
These enhancements accumulate significantly when processing trillions of tokens daily.
Indeed, Amazon’s chip development team received acclaim from Apple in 2024. In an unusual display of transparency for the typically private company, Apple’s AI director openly discussed their use of another chip from the team — Graviton, a low-power, ARM-based server CPU and the team’s initial breakthrough chip. Apple also commended Inferentia — a chip specifically engineered for inference — and acknowledged Trainium, which was then a recent introduction.
These chips exemplify Amazon’s characteristic strategy: identify popular products, then develop a competitive in-house alternative focused on pricing.
Historically, the main impediment for chips has been the cost of migration. Applications designed for Nvidia’s chips require extensive re-engineering to function with other chips, a labor-intensive process that deters developers from making a change.
However, the AWS chip team proudly informed me that Trainium now supports PyTorch, a widely used open-source framework for developing AI models. This compatibility extends to many models available on Hugging Face, a comprehensive library for shared open-source models among developers.
Carroll explained that the transition “essentially involves a one-line alteration, followed by recompilation, and then execution on Trainium.” This indicates Amazon’s efforts to incrementally erode Nvidia’s market leadership.
This month, AWS also revealed a collaboration with Cerebras Systems, integrating their inference chip onto servers alongside Trainium, which Amazon touts will deliver exceptionally powerful, low-latency AI performance.
Amazon’s aspirations extend beyond just the chips; the company also engineers the servers that house them. In addition to networking components, this team has developed “Nitro,” a hardware-software solution offering virtualization technology (enabling multiple software instances to operate independently on a single server); advanced liquid cooling systems; and the server sleds (shown below) that accommodate this equipment.
These innovations collectively aim to optimize cost-efficiency and performance.
Working 24/7 on the “bring-up”
Amazon’s bespoke chip design division originated with the cloud leader’s acquisition of Israeli chip designer Annapurna Labs in January 2015 for approximately $350 million, granting this team over a decade of experience in developing chips for AWS. The unit proudly maintains its Annapurna heritage and name — its logo is prominently featured throughout the office.
The chip lab is situated in a gleaming, chrome-windowed building within Austin’s high-end “The Domain” district, a pedestrian-friendly zone bustling with retail and dining, occasionally referred to as Austin’s Silicon Valley.
The offices feature a typical tech corporate ambiance, complete with cubicle desks, collaboration areas, and conference rooms. However, the actual lab, offering panoramic city views, is discreetly located at the rear of an upper floor within the building.
The lab, filled with shelving and roughly the size of two substantial conference rooms, is a loud industrial environment due to equipment fans. It presents an appearance combining elements of a high school workshop and a high-budget film set for an advanced laboratory, though the engineers wear jeans instead of traditional white lab coats.
It’s important to note that chip manufacturing does not occur here, eliminating the need for white hazmat suits. The Trainium3, a cutting-edge 3-nanometer chip, is fabricated by TSMC, widely considered the leader in 3-nanometer production, while other chips are manufactured by Marvell.
Nevertheless, this is the location where the critical “bring-up” process unfolds.
King clarified, “A silicon bring-up involves receiving the chip for the first time, resembling a large overnight gathering where everyone stays, like a lock-in.” Following 18 months of development, the chip is powered on for the first time to confirm its intended functionality. The team even documented part of the Trainium3 bring-up and shared it on YouTube.
A brief heads-up: the process is rarely without its challenges.
Initially, the Trainium3 prototype chip utilized air-cooling, similar to earlier iterations. The present chip, however, incorporates liquid-cooling, a significant engineering achievement that provides energy efficiency benefits.
During the bring-up phase, inaccuracies in the dimensions for attaching the chip to the air-cooling heat sink prevented its activation.
Undeterred, the team “promptly acquired a grinder and began removing the metal,” according to King. To avoid disturbing the festive bring-up pizza party, they discreetly performed the grinding in a conference room.
King emphasized that “the essence of silicon bring-up is staying awake all night and resolving issues.”
The lab also features a welding station, where Isaac Guevara, a hardware lab engineer and expert welder, showcased the intricate process of welding minute integrated circuit components using a microscope. This task is so incredibly challenging that senior leader Carroll candidly admitted his inability to perform it, eliciting laughter from Guevara and the other engineers present.
Additionally, the lab is equipped with both bespoke and off-the-shelf tools for testing and diagnosing chip-related problems. Below, signal engineer Arvind Srinivasan illustrates the lab’s method for testing each minute component on a chip:
Sleds are the star of the lab
However, the most prominent feature of the lab is a dedicated display row exhibiting every iteration of the “sleds” developed by the team.
Sleds are structural trays designed to hold Trainium AI chips, Graviton CPU chips, and their associated boards and components. When assembled onto a rack with the networking hardware, also custom-engineered by this team, they form the core systems integral to the success of Anthropic Claude.
Presented below is the sled unveiled during the AWS re:invent conference in December:
Proven by Anthropic and OpenAI
During the tour, I anticipated my guides would highlight the OpenAI agreement, but they did not.
This hesitancy might stem from the previously mentioned potential legal ambiguities surrounding the deal. However, my impression was that these frontline engineers (presently developing Trainium4, the next iteration) have not yet had significant opportunities to collaborate with OpenAI, with their daily efforts primarily centered on fulfilling Anthropic’s and Amazon’s requirements.
At present, the largest allocation of Trainium2 chips is dedicated to Project Rainier — one of the globe’s most extensive AI compute clusters — which became operational in late 2025 with 500,000 chips and is utilized by Anthropic.
Nonetheless, a wall monitor in the main office featured a quotation detailing OpenAI’s intended use of Trainium, subtly conveying a sense of pride.
Beyond this lab, the team also operates its own private data center for quality assurance and testing. Located a short drive away, this facility does not process customer workloads and is therefore situated within a co-location center rather than an AWS data center.
Security measures are stringent, with strict protocols governing entry to the building and access to Amazon’s designated areas inside.
The data center’s cooling system generates such intense noise that earplugs are compulsory, and the atmosphere is heavy with the pungent odor of warmed metal, making it an unpleasant environment for casual visits.
Within this data center, countless servers are packed with sleds incorporating all of Amazon’s latest custom chips: Graviton CPU, liquid-cooled Trainium3, and Amazon Nitro, all actively processing. The engineers explained that the liquid cooling operates in a closed loop, allowing for reuse and contributing to a reduced environmental footprint.
Pictured below is a current Trn3 UltraServer: it features multiple sleds at both the top and bottom, with Neuron switches positioned in the center. Hardware development engineer David Martinez-Darrow is shown here conducting maintenance on a sled:
Although the team has consistently received considerable attention, the level of scrutiny has markedly intensified recently.
Amazon CEO Andy Jassy closely monitors this lab, publicly championing its products with paternal pride. In December, he said Trainium was already a multi-billion dollar enterprise for AWS and highlighted it as one of the AWS technologies he finds most thrilling. He also specifically mentioned the chip during the announcement of the OpenAI deal.
The team also experiences significant pressure. Engineers dedicate three to four weeks of continuous, round-the-clock work surrounding each bring-up event to resolve any problems, ensuring chips can be mass-produced and deployed in data centers.
Carroll stated, “It is crucial that we swiftly demonstrate its operational capability.” He added, “Thus far, our performance has been exceptionally strong.”
*Disclosure: Amazon financed air travel and a single night’s accommodation at a nearby hotel. Adhering to its Leadership Principle of Frugality, this involved a middle seat at the rear of the aircraft and a simple room. TechCrunch covered additional related travel expenses, such as Uber rides and luggage charges. (Yes, I checked a bag for an overnight journey; I have high standards.)