Small Language Models
We are developing small language models with around 100 million parameters for various applications. Our smaller Large Language Models (LLMs) offer a unique advantage: They can be deployed on affordable cloud computing resources or managed in-house, giving developers and organizations more control over their machine learning operations.
In terms of technical specifications, our models can run on standard hardware, requiring just 1 virtual CPU and 1 gigabyte of RAM. This modest infrastructure enables our models to generate 20-30 tokens per second, making them well-suited for a wide range of applications.
The challenges which needs to be addressed to build small language models are:
- Generating or having right distribution of training data suitable for custom tasks (such as summarization).
- Necessary english understanding data and instructions
- Basic knowledge about world and instructions
One of the key benefits of our smaller LLMs is their reduced environmental impact. Unlike larger models, they don't require powerful Graphics Processing Units (GPUs), which significantly reduces their carbon footprint.
Despite their smaller size, our models maintain the same high level of performance and quality, ensuring that results are accurate and reliable.
This is achieved by combining relevant, artificially generated training data tailored to meet the specific needs of the application on top of:
- Knowledge of the world
- English Skills
- Basic Math