Meta Proposes Cost-Reduction Model for LLM Training

Meta's FAIR research team has pointed out the limitations of predicting the scale of large language model (LLM) training and proposed a correction model called 'Skaling'. The '10 times less compute' mentioned in the paper refers to a reduction in experimental costs rather than the overall training costs. The researchers stated that the existing assumption that model size and training data operate independently on loss values leads to errors in certain ranges. The Skaling law aims to reduce prediction errors by reflecting the combined effects of model size N and training tokens D. The researchers found that Skaling outperformed Chinchilla in 76% of the tested configurations, with a median error improvement of 2.2 times. The Mean Absolute Percentage Error (MAPE) decreased by 1.5 to 3 times depending on the experimental conditions. The paper explains that using an 'L-shaped' profiling grid targeting low-cost intervals can achieve similar prediction accuracy with about 10 times less compute than uniformly scanning the entire grid. This does not mean that the overall training costs are reduced to one-tenth. Meta stated that after training the 8B model on up to 15 trillion tokens, which exceeds the optimal point of Chinchilla, performance continued to improve. This paper is an archive preprint focusing on the efficiency of AI training and the debate over scaling law corrections.