cellularflow is a small PyTorch language model which replaces the transformer FFN with addressable key-value memory banks for continual learning

☆ Yσɠƚԋσʂ ☆@lemmy.ml · 28 days ago

cellularflow is a small PyTorch language model which replaces the transformer FFN with addressable key-value memory banks for continual learning

☆ Yσɠƚԋσʂ ☆@lemmy.ml · 27 days ago

This really does seem like the next frontier. If a model can learn and adopt on the fly, it becomes far more useful. And I suspect that small models could become tuned to do specific tasks a lot better than a generic large model.

cellularflow is a small PyTorch language model which replaces the transformer FFN with addressable key-value memory banks for continual learning

cellularflow is a small PyTorch language model which replaces the transformer FFN with addressable key-value memory banks for continual learning

GitHub - celcilin/cellularflow: Memory-Augmented Continual Learning LLM architecture decoupling knowledge storage from sequence reasoning to eliminate catastrophic forgetting.