Building a Taiwan Finance Model on a Single NVIDIA DGX™ B300 Server: About 6× Faster Inference Evaluation with Tool Calling Preserved
Companies that want AI to understand their professional domain must manage more than model selection. They also need to manage data, training cost, and deployment. For this finance experiment, one requirement came first: the data had to stay inside the data center.
APMIC fine-tuned a large model on an NVIDIA DGX™ B300 server with eight NVIDIA Blackwell Ultra GPUs using Taiwanese financial textbooks and achieved outstanding results: the model reached close to the base model's finance-benchmark accuracy with shorter reasoning output and a much shorter batch evaluation time. This shows why domain fine-tuning should be judged by more than correct answers: we should also ask how much output and time are needed to solve the same questions.
With LoRA (Low-Rank Adaptation) and reasoning enabled, average output fell from 3,352 to 986 tokens, a reduction of about 71%. Under the same benchmark, repeat count, and concurrency settings, the full evaluation took 90.98 minutes for the base model and 14.70 minutes for LoRA, or about 1/6.19 of the time. Accuracy fell from 78.4 to 75.2. Although the score decreased, the result shows a substantial improvement in answer-generation efficiency. Here, efficiency means output volume and batch evaluation time; the result alone does not prove that the model understands the questions better.
Key takeaways
A single server can fine-tune a model of this size: APMIC used an NVIDIA DGX™ B300 server with eight NVIDIA Blackwell Ultra GPUs to fine-tune NVIDIA Nemotron™ 3.5 Super VL with NVIDIA NeMo™ AutoModel, then deployed it with NVIDIA NIM™. Training, deployment, and evaluation all ran on-premises, with data kept inside the data center. One experiment took only a few hours.
LoRA used less output and time while staying close to the base model's accuracy: With reasoning enabled, average output fell by 71%. Under the same benchmark, repeat count, and concurrency settings, total evaluation time dropped from 90.98 to 14.70 minutes, about 1/6.19 of the base model's time. Accuracy fell from 78.4 to 75.2, a difference of 3.2 points (Note 5). With reasoning disabled, accuracy was 3.8 points higher than the base model, but responses took longer; after excluding questions seen during training, the gain was +2.1 points (Note 2).
Most tool-calling performance was preserved: On BFCL, LoRA scored about 96–98% of the base model's score. This gives an AI agent the ability to query or act through external tools, although full multi-turn agent workflows were not evaluated (Note 6).
Domain data also changes how a model reasons: Building on APMIC's experience organizing domain data, we checked question overlap, validated training formats, and compared reasoning content. Fine-tuned responses became shorter and more consistent; the next step is to preserve that efficiency while improving performance on harder questions.
Background: Why Fine-Tune a 120B Model On-Premises?
Financial institutions need AI to understand regulations, lending, trusts, foreign exchange, and other specialized topics. They also need control over where their data is used. When textbooks, internal policies, and exam questions cannot leave the company data center, training and deployment on the company's own servers become an important option.
In this article, fine-tuning means starting with an already trained model and adjusting its parameters with domain data so that its answers better fit the target use case. A single node means one server; that server can contain multiple GPUs that work together.
Nemotron 3.5 Super VL is NVIDIA's hybrid reasoning model. It has about 121B total parameters, or roughly 121 billion parameters, and activates about 12B parameters for each token. A token is a basic unit of text processed by a model. It may be a character, a word, or part of a word, so it cannot be treated as a direct count of Chinese characters.
Its architecture combines Mamba-2, Transformer attention, and MoE (Mixture of Experts, with 512 routed experts). It also includes a multi-token prediction head (MTP head) and a vision encoder. MoE selects only some experts for a given input, so the total parameter count and the number of active parameters are different. This helps explain compute cost, but it does not mean training only needs to load the active subset of weights.
The base model already scored above 78 on the Taiwan financial certification benchmark with reasoning enabled. We wanted to answer two questions:
Can a model of this size be fine-tuned efficiently on a single on-premises node?
After fine-tuning on Taiwanese financial textbook data, can the model better fit financial applications while retaining tool-calling ability?
Experiment Design: Data and Two Fine-Tuning Methods
The data preparation builds on APMIC's practical experience organizing domain data and checking data quality. In addition to preparing the financial textbook data, we used evaluation to examine what the model learned and which response habits changed.
The training data contains four Traditional Chinese datasets derived from Taiwanese financial textbooks, with 14,792 examples covering bank lending, trusts, foreign exchange, financial planning, internal control and auditing, and anti-money laundering. We used supervised fine-tuning (SFT), which means providing answer demonstrations for the model to learn from. Each example contains reasoning (<think>) followed by an answer, so the model learns the style of the reasoning as well as the content.
LoRA keeps the original weights frozen and trains a small set of added adapter parameters, so fewer parameters need to be updated. Full fine-tuning (Full FT) updates all language-model parameters, giving it a larger adjustment range but requiring more training memory. Neither method updated the vision encoder in this experiment.
We compared the base model, LoRA, and Full FT on the finance benchmark and tool calling. Because LoRA has a shorter experiment cycle, we used it as the main iteration version first. Full FT combined with general-purpose data, tool-calling data, and other extensions has not yet been fully validated. This article therefore records the results currently available; it does not claim that LoRA is always better than Full FT. The environment, container, training settings, and memory analysis are included in the technical appendices.
First Look at the Results: Domain Performance and Tool Calling
We evaluated the models from two angles: the Taiwan financial certification benchmark measures domain question answering, while BFCL measures tool-calling ability.
Thinking ON means that the model generates reasoning content before its answer. Thinking OFF disables that mode. We also call OFF the “fast-answer mode,” but that is only a mode name: after fine-tuning, it is not necessarily faster.
To keep the comparison consistent, all models used the same generation settings (temperature 1.0 and top_p 0.95), and scores are averages over repeated evaluations. The finance benchmark reports the percentage of questions answered correctly. A question with multiple valid answers receives credit if any valid answer is produced. “Points gained” means a difference in score, not a relative percentage increase.
Check for Data Contamination First
If an exam question already appeared in the study material, a higher score may partly reflect memorization. Model evaluation has the same problem.
We used literal matching to identify overlap between the training data and test set. About 10% of the questions had already appeared in the training data, concentrated in internal control and auditing, foreign exchange, and financial planning. The tables below therefore show both the full benchmark and the “unseen” subset.
“Unseen” means that the literal comparison did not find the question in the fine-tuning data. Rewritten variants may still go undetected (Note 2). Separating the two groups gives a clearer view of whether fine-tuning improved performance on new questions.
Taiwan Financial Certification Benchmark
Model | Thinking OFF: All | OFF: Unseen | Thinking ON: All | ON: Unseen |
Base model | 71.6 | 72.8 | 78.4 | 79.5 |
LoRA (2 epochs) | 75.4 | 74.9 | 75.2 | 74.9 |
Full FT (1 epoch) | 74.8 | 73.9 | 76.0 | 74.9 |
BFCL (Tool Calling, Single-Turn AST)
Tool calling means that the model selects an external tool and produces the required arguments. For example, when asked for an exchange rate, the model must issue a valid query instead of merely writing a plausible-looking answer. We used BFCL's single-turn AST evaluation to check the structure and arguments of tool calls. It covers only a limited part of agent capability and does not represent a complete agent workflow.
Model | Thinking OFF | Thinking ON |
Base model | 81.0 | 82.4 |
LoRA (2 epochs) | 79.2 | 79.3 |
Full FT (1 epoch) | 65.6 | 23.9 |
How to Read the Results
We use LoRA (2 epochs) as the main analysis subject because its shorter experiment cycle allowed us to complete one round of data and behavior checks quickly. The results describe the current trade-offs; they do not establish LoRA as the best method for every scenario:
Better domain answers in fast-answer mode: Accuracy was 3.8 points higher than the base model (75.4 vs. 71.6); after excluding training-overlap questions, the gain was still +2.1 points (Note 2).
Tool-calling ability was largely retained: LoRA's BFCL scores in Thinking OFF and ON were 97.7% (79.2 / 81.0) and 96.3% (79.3 / 82.4) of the base model's scores. These are ratios to the base-model score, not tool-call success rates of 96–98% (Note 6).
The reasoning-mode advantage is answer-generation efficiency: LoRA's accuracy on all questions was 3.2 points below the base model, but average output fell by about 71% and total evaluation time fell from 90.98 to 14.70 minutes, an 84% reduction. If accuracy alone is the priority, the base model remains stronger on unseen questions. Efficiency and score must be considered together; see “Actual Inference Time.”
LoRA and Full FT
Loss measures the gap between the model output and the training target. Validation loss is computed on held-out data that does not update the model parameters. Under otherwise identical conditions, a lower loss usually means the model fits that dataset's answer style better, but it is not the same as finance-benchmark accuracy.
Full FT has lower validation loss in Figure 2, but Figure 1 shows that this did not translate into better unseen-question or tool-calling performance. Evaluating data and fine-tuning therefore requires both the training curve and task results; loss alone is not enough.
LoRA (2 epochs) | Full FT (1 epoch) | |
Validation loss at 1 epoch | 0.494 | 0.470 |
Finance benchmark OFF: Unseen | 74.9 | 73.9 |
Finance benchmark ON: Unseen | 74.9 | 74.9 |
Finance benchmark ON: Seen during training | 76.3 | 80.1 |
BFCL OFF / ON | 79.2 / 79.3 | 65.6 / 23.9 |
Full FT learned this dataset's answer style faster: at the same 1-epoch comparison point, its validation loss was lower and its score on training-overlap questions was higher. This closer fit is called overfitting when it does not generalize.
The better fit did not improve unseen-question performance: Full FT was about one point below LoRA in Thinking OFF and tied it in Thinking ON. This is consistent with the model relying more on the training data, although small differences are not necessarily statistically significant (Note 2).
The tool-calling gap was large: LoRA fell by only 2–3 points, while Full FT degraded substantially and became worse with more training. At half an epoch, Thinking ON had already fallen from 82.4 to 35.3. The model might say “I will look up the exchange rate” without actually calling a tool, and instead invent a number. This loss of an existing ability is often called catastrophic forgetting.
With this dataset and these settings, LoRA used about one quarter to one fifth as much VRAM, had similar per-step training time, and retained more tool-calling performance. That makes it a useful fast iteration starting point. Whether Full FT with general-purpose and tool-calling data can reduce forgetting and produce better overall results still requires further experiments. The methods used different learning rates and epoch counts, and each was trained only once, so these results do not establish that LoRA is always better than Full FT.
Actual Inference Time
“Inference” is the process of generating a response after receiving an input. “Reasoning” describes the model's problem-solving output in this article. To understand the timing results, separate the number of tokens the model can generate per second from the total number of tokens needed to finish a question.
This section compares the time required to complete the benchmark questions. All speed values are relative. Service-level stress-test results are in “Technical Appendix 5: Baseline Inference Performance.”
After LoRA is merged, and after Full FT is applied, the models have the same architecture and size as the base model. With reasoning enabled, the main difference is that fewer output tokens are needed to finish a question; the actual MTP speedup may still change after fine-tuning (Note 4).
The evaluation used the same service settings (DP=2 × TP=4, MTP enabled, and fixed concurrency). ON and OFF each use the base model in the same mode as 1.0, so the two relative columns should not be compared directly. The reasoning-mode finance evaluation contained 1,346 questions, repeated three times for 4,038 responses in total. Batch time was 90.98 minutes for the base model, 14.70 minutes for LoRA, and 15.11 minutes for Full FT. The per-question figures are converted from the batch workload and are not per-request user latency (Note 5).
Model | Mode | Average output tokens / question | Average time (base = 1.0) | Accuracy (all questions) |
Base model | Thinking ON | 3,352 | 1.0 | 78.4 |
LoRA | Thinking ON | 986 | 0.16 (about 6× faster) | 75.2 |
Full FT | Thinking ON | 1,018 | 0.17 (about 6× faster) | 76.0 |
Base model | Thinking OFF | 41 | 1.0 | 71.6 |
LoRA | Thinking OFF | 475 | 2.2 (about 2× slower) | 75.4 |
Less output and time, with a close score: LoRA reduced average output from 3,352 to 986 tokens, about 71%, and reduced batch evaluation time from 90.98 to 14.70 minutes, or 1/6.19 of the base model's time. Accuracy fell from 78.4 to 75.2, so the efficiency gain came with a 3.2-point score decrease.
Turning reasoning off made LoRA slower: the base model usually produced only the answer in Thinking OFF (41 tokens on average), while LoRA wrote an explanation before answering (about 475 tokens). Accuracy increased by 3.8 points, but evaluation time increased by about 2.2×.
A few very long answers can slow the entire batch: the evaluator waits at the file level. Once most requests finish, a few long responses can leave some processing slots idle. More consistent response lengths helped LoRA reach about 1.8× the base model's overall output throughput at the same concurrency.
Finance-benchmark speed does not carry over directly to tool calling: in reasoning mode, LoRA's BFCL evaluation took about 1.9× as long as the base model and Full FT took about 3× as long. Full FT often answered in text instead of calling a tool, increasing average output from 313 to about 1,000 tokens.
In-Depth Analysis: How Fine-Tuning Changed the Reasoning Style
The results show that the fine-tuned model can finish questions with less output and shorter batch evaluation time, but with some loss in accuracy. To find ways to keep the efficiency while improving answers, we compared the reasoning content produced by the base and fine-tuned models: how much text they used, whether they rechecked their judgments, and how closely the style matched the training data.
The analysis covers generated text, not the model's internal thoughts. Language and self-checking rates were estimated with character and keyword rules. p90 means the 90th percentile and is used to show the long-answer tail.
Base model | Fine-tuned (LoRA / Full FT) | |
Output tokens, median / p90 (including answer) | 1,357 / 8,672 | about 960–1,000 / about 1,300 |
Does reasoning length adapt to difficulty? | Yes: wrong answers had reasoning about 3.5× longer than correct answers | No: correct and wrong answers had nearly the same reasoning length |
Includes self-checking (“wait,” “let me confirm,” etc.) | about 72% | about 1% |
Reasoning language | about 70% mainly English, about 18% Simplified Chinese | almost entirely Traditional Chinese |
The fine-tuned reasoning closely resembles the SFT data: it usually begins with “Let me think about this,” has a median reasoning length of about 450 characters (close to the training data's 509), and tends to make direct assertions with little self-checking.
Another important difference is that the base model tends to reason longer on difficult or incorrect questions. After fine-tuning, reasoning length is much more consistent across correct and incorrect answers. Shorter output is useful for efficiency, but length alone does not tell us whether the reasoning is correct.
To compare difficulty, we grouped questions using the base model's reasoning length as an approximate difficulty indicator. This is not an independent difficulty test, but it helps show where performance dropped:
Approximate difficulty (base-model reasoning length) | Base model | LoRA |
Easy (< 1.5k characters) | 94.2 | 93.6 |
Medium (1.5k–4k characters) | 89.2 | 84.2 |
Hard (4k–9k characters) | 73.8 | 67.8 |
Hardest (> 9k characters) | 53.7 | 53.7 |
The accuracy drop was concentrated in medium and hard questions. This is consistent with the possibility that shorter reasoning and less self-checking affected some questions; the causal relationship still needs a controlled experiment.
Consider a question about a regulatory deadline. The base model first acknowledged that it needed to confirm the legal basis, listed several possible rules, paused to reconsider, and eliminated distractors one by one before reaching the correct answer. The fine-tuned model made a short, confident statement that the rule required an announcement within two days of a certain date. It invented that rule and answered incorrectly.
One possible explanation, based on comparing the training data with model outputs (Note 3), is that the SFT reasoning was rewritten by another model that already knew the answer. That style is useful for explaining a known answer, but may not preserve the process of exploring alternatives, checking errors, and correcting a judgment.
When a model learns this style, it may reduce its original long-form reasoning and self-checking. A demonstration can be concise and fluent without being a good reasoning example for a stronger model. This is an explanation suggested by the current observations, not a proven direct cause.
This analysis extends data-quality review beyond whether an answer is correct: the reasoning style in a demonstration must also fit the target model. Future checks should compare the data, model outputs, and task-level evaluations together before concluding that a data change worked.
Single-Node Cost and End-to-End Workflow
The experiment also confirmed that on-premises execution is feasible. One server with eight NVIDIA Blackwell Ultra GPUs held the memory requirements for both LoRA and Full FT, then continued through deployment and evaluation while keeping all data inside the data center.
With these settings, both methods took about 10 seconds per training step. One epoch took about 1.3–1.4 hours. LoRA peaked at about 50 GiB and Full FT at about 190 GiB. LoRA's main resource advantage is therefore lower memory and checkpoint storage; their per-step training times were similar. Detailed estimates are in “Technical Appendix 3: VRAM Requirements and Training Speed.”
End-to-End Workflow
A complete experiment starts with data preparation, continues through training and deployment, and ends with evaluation. LoRA and Full FT require different preparation before deployment, then use the same evaluation flow:
Data preparation, preprocessing, and contamination checks are part of the same workflow. Per-question evaluation results then feed back into the data-quality analysis and the next iteration.
The workflow is scripted and repeatable. Each stage records its duration in APMIC Trackio and produces a consistent evaluation report plus per-question responses that can be synchronized with business-intelligence systems.
What This Experiment Tells Us
Three practical conclusions stand out:
A single server can run an on-premises fine-tuning workflow at this scale. Eight NVIDIA Blackwell Ultra GPUs completed training, deployment, and evaluation on one server in a few hours.
LoRA is a fast iteration starting point. With reasoning enabled, it reduced output by about 71% and batch evaluation time from 90.98 to 14.70 minutes, while accuracy fell by 3.2 points and tool-calling scores stayed at 96–98% of the base model. Full FT with expanded general-purpose and tool-calling data has not yet been evaluated, so this does not establish LoRA as the final best choice. If finance-benchmark accuracy is the priority, the base model with reasoning enabled remains stronger; with reasoning disabled, LoRA improves the score but takes longer.
Data quality must be checked together with model behavior. Question overlap, formatting, the style of demonstrations, and regression in existing abilities all affect how we interpret fine-tuning. The reasoning-style analysis provides a direction for improvement, but its cause still needs to be verified.
Next Steps
APMIC will use the per-question evaluations and reasoning analysis to refine data preparation. Because LoRA is suitable for rapid iteration, we will first use it to validate data-quality and reasoning-demonstration directions, then evaluate Full FT and expanded data mixtures. We will also evaluate self-distillation: using the base model's own reasoning to create demonstrations that may be better suited to its learning behavior.
We will add tool-calling and general-purpose data to test whether existing abilities can be retained, and separate training questions from evaluation questions more strictly. Future experiments will compare domain accuracy, output length, evaluation time, and tool-calling performance, with the goal of preserving concise reasoning and efficiency while improving difficult-question quality.
Notes
VRAM estimate: calculated from parameter counts (BF16 uses 2 bytes and fp32 uses 4 bytes). Activations, communication buffers, and memory fragmentation are excluded; the estimate is for scale only.
Score uncertainty: each setting was trained once. Differences of roughly 1–2 points (for example, +2.1 or about one point lower in OFF) may not be statistically significant. “Unseen” was determined by literal matching; rewritten similar questions may be missed, so the gain on unseen questions may still be slightly overestimated.
The cause is a hypothesis: “SFT reasoning quality was below the base model's and imitation caused the regression” is an explanation based on the observations. A controlled experiment has not yet established causality.
MTP was not retested: the MTP head was not updated during fine-tuning, and we did not run inference stress tests on the fine-tuned models. Speculative decoding verifies candidate tokens with the main model; an unchanged MTP head may reduce acceptance rate and therefore the actual speedup.
Speed under evaluation load: the reasoning-mode finance benchmark had 1,346 questions, each repeated three times, for 4,038 responses. Batch time was 90.98 minutes for the base model, 14.70 minutes for LoRA, and 15.11 minutes for Full FT. Per-question time is converted from the total evaluation workload under fixed concurrency and is not per-request latency.
Scope of agent measurement: this article uses BFCL v3 single-turn AST tool calling as an agent-capability indicator. Multi-turn agent tasks, general knowledge, and multimodal ability were not evaluated. “96–98%” is LoRA's BFCL score divided by the base model's score; Full FT is outside this claim.
Acknowledgements
Thanks to NVIDIA for the Nemotron™ 3.5 Super VL model, NeMo AutoModel, NIM, and technical support for the NVIDIA DGX™ B300 platform.
Technical Appendix 1: Environment and Container Preparation
The environment has a clear division of labor: the NVIDIA DGX™ B300 server provides compute and memory, NeMo AutoModel trains the model, NVIDIA NIM exposes it as a service that applications can call, twinkle-eval runs the evaluations, and Trackio records the experiment.
Item | Details |
Hardware | One server with 8× NVIDIA Blackwell Ultra GPUs (288 GB HBM3e per GPU, about 268 GiB available), fully connected by NVLink |
Model | NVIDIA Nemotron 3.5 Super VL 120B-A12B, BF16 |
Training framework | NVIDIA NeMo AutoModel (FSDP2 + Expert Parallel) |
Deployment | NVIDIA NIM (vLLM backend), DP=2 × TP=4, MTP enabled |
Evaluation | twinkle-eval: Taiwan financial certification benchmark and BFCL v3 (single-turn AST tool calling), Thinking OFF / ON |
Experiment tracking | APMIC Trackio: loss, throughput, and stage durations for every training step |
BF16 is a 16-bit format for storing model values. FSDP2 and Expert Parallel distribute training work across GPUs. DP=2 × TP=4 means two model replicas, with four GPUs working together on each replica.
Why Build the Container Ourselves?
A container packages the program and its dependencies into one execution environment, reducing differences between machines. We built a custom container to obtain newer model support.
At the time of the experiment, the official NeMo AutoModel container and the AutoModel GitHub main branch were not released on the same schedule. The Nemotron 3.5 Super VL implementation, weight conversion, and training recipes were changing quickly. We therefore kept the validated CUDA environment from the official container and added the latest AutoModel code from the main branch:
```dockerfile
FROM nvcr.io/nvidia/nemo-automodel:26.08
# Replace the container's AutoModel with the latest main branch
COPY Automodel /opt/Automodel
# Reinstall only AutoModel and upgrade the transformers version it needs
RUN cd /opt/Automodel \
&& uv pip install --python /opt/venv/bin/python --no-deps -e . \
&& uv pip install --python /opt/venv/bin/python "transformers==5.15.1"
```
git clone https://github.com/NVIDIA-NeMo/Automodel.git
docker build -t automodel-n35vl -f Dockerfile .Change | Reason |
Use the official container as the base | Reuse precompiled CUDA dependencies such as PyTorch, Transformer Engine, DeepEP, and mamba-ssm |
COPY the latest AutoModel | Obtain the latest Nemotron 3.5 Super VL implementation, state-dict adapter, and recipe |
--no-deps | Reinstall only AutoModel and avoid upgrading CUDA packages already validated in the container |
Upgrade transformers | The newer AutoModel requires a newer version to load the processor and model correctly |
Exclude Automodel/.git in .dockerignore | Reduce the build context |
If the build environment cannot reach the network through Docker's default bridge network, for example because of special host firewall rules, add --network host to docker build.
Technical Appendix 2: Complete Training Configuration
We used NVIDIA's official Nemotron 3.5 Super VL recipe for NeMo AutoModel as the starting point and compared two fine-tuning methods.
lr is the learning rate, which controls the size of each parameter update. An epoch is one pass through the training data. LoRA's rank and alpha affect the size and update strength of the added module.
Method | Configuration |
LoRA | rank 64 / alpha 128, applied to linear layers in Mamba, attention, MoE latent projections, and shared experts; lr 2e-4, 2 epochs |
Full FT | all language-model parameters trainable (120.7B), vision tower frozen; lr 1e-5, 1 epoch |
Both methods used global batch size 32, Expert Parallel = 8, and activation checkpointing. Activation checkpointing saves memory by recomputing some intermediate values. MTP was disabled during training. The collate function did not pad all samples to a fixed length (max_length: null), avoiding unnecessary work on short samples.
Technical Appendix 3: VRAM Requirements and Training Speed
During large-model training, GPU memory must hold model weights, parameter-update state, and intermediate activations. For this 120B-class model, memory capacity determines whether Full FT fits on one machine.
We estimated VRAM from the parameter count and compared the estimate with measurements (Note 1). Values use GiB; each NVIDIA Blackwell Ultra GPU provides about 268 GiB. Measurements are PyTorch's configured-memory peaks, while actual GPU use is slightly higher.
Full Fine-Tuning
The model has 121.5B total parameters, of which 120.7B are trainable because the vision encoder is frozen. The table below shows the working state needed during training: weights are the current parameters, gradients provide update directions, and master weights plus Adam moments support the update.
Item | VRAM (GiB) |
BF16 model weights | 226 |
BF16 gradients | 224 |
fp32 master weights (store_param_remainders) | 224 |
Adam first / second moments (bf16) | 450 |
Total (excluding activations) | about 1,125 |
Average per GPU (8 GPUs) | about 141 |
Measured per GPU (normal step / peak) | 174 / 190 |
The memory-saving optimizer configuration matters: storing Adam moments in bf16 and fp32 master weights as 16-bit remainders allows the training state for 120.7B parameters to fit across eight NVIDIA Blackwell Ultra GPUs.
One node is enough: NVIDIA's official Full FT example uses four nodes and 32 NVIDIA H100 GPUs. NVIDIA Blackwell Ultra GPUs allow this experiment to run on one node, avoiding cross-node networking and shared-storage setup.
LoRA
Item | VRAM |
Frozen BF16 weights | 226 GiB total (about 28 GiB per GPU) |
LoRA parameters and optimizer state (177M parameters) | about 3 GB, negligible |
Measured per GPU | about 36 GiB normally; about 50 GiB at the longest 12,000-token sample |
LoRA does not require NVIDIA Blackwell Ultra GPUs: NVIDIA's official example runs on one node with 8× NVIDIA H100 GPUs (about 36–39 GiB per GPU). NVIDIA Blackwell Ultra GPUs give more headroom, allowing the longest sequences to remain untruncated and leaving room for a larger batch later.
LoRA vs. Full FT: Where Does the VRAM Go?
Both methods load the same BF16 model weights. The major difference is which parameters need update state. Full FT stores gradients and optimizer state for 120.7B parameters; LoRA updates only 177M adapter parameters, so the frozen base weights need none of that additional state.
Per-GPU VRAM (GiB, 8 GPUs) | LoRA | Full FT | Difference |
BF16 model weights | 28 | 28 | Same |
Gradients | ≈ 0 (adapter only) | 28 | |
fp32 master weights | ≈ 0 | 28 | |
Adam first / second moments | ≈ 0 | 56 | |
Training-state subtotal | about 28 | about 141 | about 5× |
Measured: normal step | about 36 | about 174 | about 4.8× |
Measured: peak (longest sequence) | about 50 | about 190 | about 3.8× |
LoRA | Full FT | |
Hardware reference (NVIDIA example) | One node, 8× H100 | Four nodes, 32× H100; this experiment used one node, 8× NVIDIA Blackwell Ultra GPUs |
Each checkpoint | 355 MB adapter | About 450 GB for model weights only |
Pre-deployment step | Merge adapter (about 3 minutes) | Restore the MTP head omitted during training |
A checkpoint is a saved model state that can be loaded for evaluation or later use. In this experiment, Full FT required about 4–5× the VRAM, and its complete-model checkpoint used hundreds of times more storage than a LoRA adapter.
Why the NVIDIA DGX™ B300 Server?
8× 80 GB-class GPUs | 8× NVIDIA Blackwell Ultra GPUs | |
LoRA | Possible | Possible, with more headroom for long sequences and larger batches |
Full FT | Requires multiple nodes (four in the official example) | One node |
Training, deployment, and evaluation on one machine | Not for Full FT | Yes |
One experiment (training + deployment + evaluation) | Requires cross-node coordination | A few hours |
The NVIDIA DGX™ B300 server's main value here is that the memory needed for Full FT fits on one server. Training speed is a separate question. In the table below, a step is one parameter update and throughput is the number of tokens processed per second.
LoRA | Full FT | |
Trainable parameters | 177M (0.15%) | 120.7B (99.3%) |
Peak memory per GPU | about 50 GiB | about 190 GiB |
Time per step (GBS 32) | about 10 s | about 10 s |
Throughput per GPU (tokens/s, median) | 564 | 634 |
1 epoch (454 steps, 18.9M tokens) | about 1.3 h (converted from 2 epochs) | about 1.4 h (including two saves) |
Each checkpoint save | seconds | about 3.5–4 minutes |
With this NVIDIA DGX™ B300 server and these settings, Full FT and LoRA had similar per-step training times. LoRA's main resource benefits were lower memory and storage requirements, not faster training. Full FT's clear extra cost was saving checkpoints.
Technical Appendix 4: Problems Solved During Implementation
Running the experiment required more than choosing hardware and a training method. We also had to handle data formatting, weight conversion, and checkpoint storage:
Container version: used the official container as a base and replaced AutoModel with the latest main-branch version (see Technical Appendix 1).
Padding waste: setting max_length in the multimodal collate function pads every sample to that length. The data averaged about 1,300 tokens, with a maximum of about 12,000. Setting max_length: null avoided unnecessary padding computation.
Labels in multi-turn conversations: the chat template removed reasoning from earlier turns, causing label misalignment. We fixed this in preprocessing and used the collate function to verify the loss mask for all 14,792 examples, confirming which text contributed to the training loss.
Merging LoRA back into HF weights: the adapter exported by AutoModel used Transformers v5 names that differed from the original checkpoint. We wrote an offline merge tool that applied W + (α/r)·BA shard by shard, preserved the vision tower and MTP head, and produced weights deployable by NIM in about three minutes.
Putting Full FT weights online: NIM normally uses the MTP head for speculative decoding, but the official recipe disabled it during training, so exported weights did not contain the MTP layer. We restored the MTP and vision-tower buffers from the original weights so Full FT could use the same deployment settings as the base model. The restored MTP head was not updated during fine-tuning, so the candidate-token acceptance rate and speedup may be lower (Note 4).
Disk space: Full FT checkpoints containing optimizer state are very large. Evaluation only required model weights, so we saved model-only checkpoints of about 450 GB each.
Technical Appendix 5: Baseline Inference Performance
On the same NVIDIA DGX™ B300 server, we deployed the base model with NIM (DP=2 × TP=4) and used vllm bench serve for a stress test with ShareGPT conversations and reasoning enabled.
Here, throughput means total tokens generated per second, while per-token latency describes the waiting time as text continues to be generated. Serving more requests at once can increase total throughput while making each individual request wait longer.
Item | Relative result |
MTP (speculative decoding) on vs. off | Output throughput increased 1.5–2.1×; per-token latency fell to about 0.5–0.65×; about 3 tokens were accepted per step on average |
300 simultaneous requests vs. one request | Total output throughput increased about 18×, while per-request token latency increased about 15× |
TP=2 (2 GPUs) vs. DP=2 × TP=4 (8 GPUs) | Per-GPU throughput was about 3.6× higher; the 8-GPU deployment was better for single-request latency and total throughput |
MTP produced a clear speedup in this test: it proposes several candidate tokens and lets the main model verify them, reducing the need to generate every token one at a time. The effect was strongest at low concurrency (about 2×).
Deployment should match the workload: smaller TP had better per-GPU throughput in this test; if single-request response time matters more, more GPUs can work together on one model replica.
Comments