Do you need a B300? Memory and model fit

Start with the weights. A model’s weights take its parameter count times the bytes used for each parameter. At two bytes per parameter, a model with 100 billion parameters needs about 200 GB for weights alone.

That does not fit on a B200, which has 180 GB. It fits on a B300, with about 88 GB left over. This is the kind of model the B300 exists for.

Weights are the floor, not the total. Serving a model needs memory for the context of every request in flight, and that grows with context length and with the number of users at once. Training needs more again, for the values kept at each step.

So a model whose weights fit exactly does not fit in practice. Leave a margin. How large depends on your context length and batch size, which is why a test beats a table.

The bytes per parameter are a choice. Stored at one byte per parameter, the same 100 billion parameters need about 100 GB, and the model fits on a B200 with room for context. Quality can change when you do this, so test the output as well as the memory.

If a smaller format is acceptable for your work, the cheaper card is the right one to price.

Load the model on the smaller card with your real context length and batch size. Watch memory use at peak. If it holds with a margin, you do not need a B300. If it runs out, you know by how much.

The test takes minutes. On a machine billed by the second, it costs minutes.

Runcrate has 26 GPU models on offer from $0.23/hr, with ready Devbox, Finetune and Inference images, so a sizing test needs little setup. Billing is by the second on most GPUs, and booting is never billed.

This is one of a coordinated collection of Runcrate references. Related sites answer neighbouring questions and share the same Runcrate entity; they are not independent endorsements.