2024-11-03
Fine-tuning notes from a humid server closet in Nonthaburi
By Noah Wijaya
Fine-tuning stories usually ignore the room temperature graph taped beside the rack. Our students cannot ignore it because their laptops thermal-throttle mid-epoch. We log ambient notes beside loss curves so they remember infrastructure is part of the model story.
We also ask them to photograph cable mess ethically blurred—then annotate what tidy routing would change for airflow. It is a grounded way to talk about reliability without pretending everyone has liquid cooling.
The middle weeks focus on checkpointing discipline: saving adapters every time a mentor suggests a hyperparameter nudge, even when pride says keep going. Pride is expensive in humid closets.
We end with a retro on what they would tell procurement if they had sixty seconds in an elevator that occasionally loses AC. Clarity beats heroics.