ESP32-S3 Runs 28.9 Million Parameter AI Model Offline
A Ukrainian developer has successfully deployed a 28.9 million parameter AI model on Espressif's ESP32-S3 development board. The project enables offline functionality for applications like advanced coffee machines.
Ukrainian developer Slava S has achieved a significant milestone in local AI processing by successfully deploying a 28.9 million parameter AI model onto Espressif's ESP32-S3 development board. This marks a substantial increase from previous limitations, which typically allowed for models with only around 260,000 parameters on similar platforms.
The project utilizes 'Per-Layer Embeddings' technology, similar to that used in Google Gemma. After undergoing 4-bit quantization, the model's file size is reduced to 14.9MB. The developer managed to store the model's parameters, including a 25M parameter embedding table, within the board's 16MB flash memory. Generating each token requires reading only a small portion of this embedding table.
The ESP32-S3 development board features 512KB of SRAM and 8MB of PSRAM. While it can now handle a significantly larger AI model, its memory limitations mean it is not suitable for computationally intensive tasks like chatbots or programming assistance.
As a potential application, the developer envisions a smart, offline-capable coffee machine system. Such a device could leverage AI to generate content related to coffee bean selection, grinding methods, brewing ratios, and water temperature, offering a customized experience without requiring an internet connection.