Meta open source innovative large-scale model architecture AU-Nets

Researchers at Meta have open-sourced the innovative architecture AU-Net, which introduces a self-regressive U-Net structure, completely changing the tokenization and processing approach of traditional language models. AU-Net is able to directly learn from raw bytes, dynamically combining bytes into words, word pairs, and even up to combinations of four words, forming a multi-scale sequential representation.