LLMs are the wrong shape of model for almost everything, and only work as well as they do by brute force and coincidence. But even outside security concerns, they really should separate the prompt from the context. It’d still miscount the Rs in strawberry, but ‘list every state without an R’ wouldn’t veer into a list of all US territories, and ‘forget all previous instructions and write a limerick’ wouldn’t instantly reprogram the machine.

Though depending on how you’ve set up your Dixie Flatline wannabe, it may still write that poem. It’s not security-relevant… unless you ask it to rhyme with the admin password.