Platypus: Quick, Cheap, and Powerful LLM
In recent years, model parameters have exploded to a huge number of parameters (540 B with PaLM). The question that has been asked is whether this number of parameters is necessary.
According to OpenAI, as models grow, there is an increase in performance. In addition, there is the appearance of emergent properties (properties that cannot be observed except at a certain scale).
This view has been challenged by the fact that actually more data, and thus scaling is limited by the number of tokens needed to train a model optimally. Moreover, even these emergent properties may not even exist.
Second, these proprietary models cannot be analyzed or used freely by the scientific community. Therefore, first with BLOOM and then with META’s LLaMA, the community has moved toward using open-source models. LLaMA also showed that an increased focus on data allows smaller models to compete with larger models.
0 Comments