What an AI factory is, and how to plan one

An AI factory is a production system. It takes power, data and accelerated compute, and it returns intelligence an organisation can use: answers, rankings, drafts, detections, or the next step in a workflow. The phrase is useful because it keeps the conversation on throughput, quality and operating cost, which is where an infrastructure decision actually lives.

NVIDIA’s glossary describes an AI factory as specialised computing infrastructure that manages the AI life cycle, from data ingestion through training, fine-tuning and high-volume inference, with intelligence as the product. The same account, on NVIDIA’s AI factories (opens in a new tab) pages, talks about five layers that have to work together: energy, chips, infrastructure, models and applications. Those layers are a sound planning checklist even if you never buy a particular vendor’s stack. They force the conversation out of a single purchase order.

This briefing is a planning sequence for CIOs, CTOs, and infrastructure and data leaders. It is general information. It is not a sizing tool, and it does not assume a vendor, a site or a budget.

Start with the product, not the rack

A traditional data centre is built to run many kinds of workload with a shared idea of availability. An AI factory is narrower. It is optimised for a class of work: training a model, adapting one, or serving inference at a latency and a cost you can defend. The planning question is what the factory must produce in a week of ordinary operation, and what “good” means for that output.

Write that down before anyone draws a rack elevation.

• What is the workload mix: training, fine-tuning, batch inference, interactive inference, or some of each?
• What latency, throughput and concurrency does the business process actually need?
• What does a wrong or slow answer cost, in money, safety or trust?
• Which data is allowed into the system, and where must processing stay?
• Who consumes the output: a person, another system, or both?

If those answers are vague, the architecture will be vague, and the power and cooling request will be a guess. A guess at this density is an expensive way to learn the building’s limits. The facilities side of that problem is covered in Power, cooling and the building in front of you.

The five layers, as decisions

Treat each layer as a decision with an owner.

Energy. How many kilowatts the room can deliver, with what redundancy, and how the heat leaves the building. This layer bounds every other choice. A reference design that ignores it is a brochure.

Chips and systems. The shape of the accelerated compute: GPU class, memory, the scale of the NVLink or equivalent domain, and whether the first deployment is one rack or a larger pod. The point is fit to the workload, including the memory footprint of the models you intend to run. A larger domain is not a better domain if the job is a fleet of smaller models with strict latency.

Infrastructure. Fabric, storage and the software platform around the compute. Training and large-scale inference punish a slow data path and a congested network. Storage has to serve the checkpointing and the data pipeline, not only the home directories. The fabric decision (InfiniBand or Ethernet, and at what speed) follows the communication pattern of the job. The scheduling layer, whether Slurm, Kubernetes or a managed platform, follows the team who will run it at 2am.

Models. Which models you train, adapt or serve, and how they are versioned, evaluated and promoted. This is where data quality, licensing and evaluation sets belong. Infrastructure that cannot name the models it is for will be sized on a slogan.

Applications. The business processes that call the factory. An application that cannot say which system of record it reads, and which it is allowed to change, is not ready for a production agent. That path is integration work, described in From an agent pilot to a system you can run.

A plan that assigns each layer an owner, a constraint and a date is already more useful than a parts list.

Decisions that are costly to reverse

Some choices can be changed in a maintenance window. Others are poured into the building or into the identity model.

• Power topology and the heat-rejection path. Liquid cooling, air, or a mix, is a facility project. It is not a line item you add after the racks are on the dock.
• The network fabric and the physical pathways into the room. Undersized pathways and a fabric that does not match the job show up as idle GPUs.
• The storage performance profile. A design that hits capacity and misses throughput will fail the workload you bought it for.
• Data residency and the trust boundary. Where weights, prompts, logs and customer data sit is a governance decision. Moving it later means moving the controls with it.
• The operating team. A platform nobody is staffed to patch will decay, however current it was on day one.

Write the irreversible items first. They are the brief. The reversible items can stay as options.

On-premises, cloud, and both

The location of the factory is a constraint problem.

On-premises, or in a facility you contract, is the right conversation when data residency, latency to a system of record, or a long-running training estate dominates, and when you can actually power and cool the density. Cloud is the right conversation when the load is bursty, when you want to test a model class before you commit floor space, or when the operational team is already built around a cloud platform. Many enterprises will run both: experiment and burst in cloud, steady and sensitive workloads on infrastructure they control.

The weak version of this decision is a slogan about sovereignty or about speed. The strong version is a table: workload, data class, latency, duration, skills on the team, and the kilowatts the building can offer. Dynacore’s AI Factory practice starts from that table.

What a first briefing should produce

A useful first pass is short. It should leave four artefacts, even if each is only a few pages.

1. A workload brief. The product of the factory, the mix of jobs, and the measures of “good”.
2. A constraint note. Residency, identity, the systems in scope, dates, and the budget band if you know it.
3. A site and platform view. What the building and the current platforms can support, and what they cannot.
4. A sequenced plan. What to decide now, what to pilot, and what must wait for power, data or a governance answer.

That pack is enough to choose a reference architecture. It is also enough to stop. Stopping, when the building or the data is not ready, is a successful outcome of the briefing.

How to use this with a board

Boards and executive teams do not need a parts catalogue. They need the product, the constraint, the spend logic and the operating owner.

Frame the request as a production system with a measured output. Separate capital for the facility from capital for the compute, because they move on different clocks and often through different budgets. Name the person who will accept the system at commissioning, and the person who will own it in month six. If those are the same person, say so. If they are not, the handover is part of the plan.

Published performance claims from vendors are an input to a shortlist. They are not a forecast of your job on your data. Ask for the configuration the number was measured on, and put your own acceptance tests in the contract: power, thermals, fabric health, storage throughput, and a failure case you care about. That is commissioning. It is how you discover whether the factory you bought is the factory that is running.

A note on hype

The factory metaphor is easy to inflate. A single server with a GPU is not a factory. A slide that says “1,000 GPUs” is not a design. The test is mundane: can you describe the output, the energy path, the data path, the operating team and the acceptance tests on a few pages? If you can, you have the start of a plan. If you cannot, the next meeting should produce those pages, not a purchase order.

Dynacore designs and delivers this class of system for Australian enterprises and government, from the brief through commissioning. If you want that conversation, book a briefing.