Maybe some devs here can help me, I was recently promoted to “head of AI” at my work despite being very outwardly ambivalent towards it. So I’m struggling to figure out what would actually create value instead of just being an expensive waste of time but still satisfy the higher ups AI lust.
My first idea that I thought would actually be useful was just setting up the architecture for an actual analytics database for us and then let them explore it with metabase (then letting them use Claude for their wow factor of exploring it with AI or whatever).
But now I’m somewhat at a loss, so any insight you all have would be really helpful!
Since there’s zero information about what kind of company you’re working at, the following is extremely generalized.
- integrating with monitoring systems, analytics DBs, ticket systems, whatever is used by management, allowing them to ask questions in natural language
- process automation using agentic workflows, e.g. pre-analysis of incoming email queue summarizing / sentiment analysis before the customer support sees it
- provide access to models and model APIs for development workflows and integration into git / ci, allowing to use llm in local development and e.g. setting up something like automated code reviews (not a replacement for human review, only as an addition)
- set up coaching, responsible use, hallucinations, etc.
Whatever you do, take security and data security especially into consideration first, not after:
- consider whether your used provider reuses your data for learning
- consider whether it’s relevant where it’s located (GDPR customers?)
- always set spending limits
- consider your local and your customers data protection laws and regulations that apply to your company (especially in health and financing)
This is wonderful thank you.
I’m starting to get the idea that the LLM is just a small part of this and what’s really important is instead the bunch of architecture and guardrails around it to form how the humans and ai will act with any given system.
Personally I see LLMs as a tool like any other. You can use it to mass produce low quality slop, just as you can use it to help you produce a higher quality output.
You’re perfectly right about architecture and guardrails, that’s how it has always been with any other tool or piece of software. It depends on how you use it. Remember the no-code hype train? It’s literally the same, people have been shoving it into everything, no matter whether it made sense. It worked for some, and it made development costs explode for others.
Guardrails are especially important for LLMs because you do not have deterministic outputs and potentially exploding costs.
So analyze, measure, and think about where and how it makes sense to integrate, and build it incrementally, again, just like with any other piece of software. Start slow, keep humans in the loop, measure and analyze, and improve incrementally. When you achieve confidence, potentially start automating going into an agentic direction, when it makes sense and the risks have been considered, but always keep provenance. You do not want blind decisions by the magical AI box.
And just to repeat, because I’ve seen heads roll because of dumb decisions: keep cost under control and always have limits set, and always consider which data flows into the AI and what happens with it afterwards.
Producing a half a million bill in a month by accident or neglect or suddenly having your customer database queryable on a public model is a surefire way to drive the company or at least your career to the ground in seconds of wrong decisions.
Also, read into all the stuff built around LLMs, protocols like MCP, attacks and defenses on LLMs, get knowledge about the inner workings, experiment and learn. When you’re the head of AI, you’re supposed to be the person who knows. And when you know what it does, how it works, and how to use it, you’ll find actually good and appropriate use-cases naturally.
Personally I see LLMs as a tool like any other. You can use it to mass produce low quality slop, just as you can use it to help you produce a higher quality output.
I agree with this in principle, but I’d argue that it’s more like an industrial table saw than a hand saw.
Meaning the potential for damage is much greater with this particular tool (for various reasons) which means that it need proper training and experience, safeguards, maintenance and fail-safes to be used effectively (without doing yourself or others damage).
The potential damage of using this tool is a bit less immediate and obvious than an industrial table saw however, which means you should be extra careful.
If the person who put you in charge of it knew of your indifference, that’s a smart manager, and you should just trust your gut on this.
If you just vibe AI, it will lull you into thinking you’re doing great.
- Define what successful AI usage looks like and include metrics and measurements — include AI usage in your story/ticketing system.
- Define parameters for usage. Will it write documentation? Write tests? Write functions? Classes? Whole features? Will it review code? How are you making sure everyone understands the new code (i.e. you don’t want devs committing code they don’t understand or reviewers passing it)
- How will you reign in costs? I had 3 devs spend over $1500 (each) in thirty days and am in the process of a explaining this to my COO. Purchasing plans instead of using API keys gives you some natural boundaries for reigning in costs.
- Create standard processes. Don’t vibe. Have a standard for how your AI-facing documentation is structured. Have standards/templates for how prompts are structured. Have standard prompts with limited scope for specific tasks. For example have a prompt just for review that specifies to review from the standpoint of security and best practices.
Those are all very helpful points, especially the last one; I’ve always tried to include standardization in the company but was never in the position to enforce it before
Assuming a dev company, first thing I would do is let the manegement know, in writing, that building a dependance on AI into your processes will make you depend on a tool that is currently heavily under-priced, with operating costs for all AI firms resulting in hundreds of millions of losses, that they will want to recuperate eventually.
Link articles about firms blowing the yearly budget on per-token price in a single month. Companies spending 500mil per month on tokens woth uncapped usage.
Also explain that allowing your devs to use it will lead to erpsion of their skill, building a dependency, that will then tie their productivity going forward to the whims of AI companies, that are currently operating at heavy losses.
Link any of the numerous papers how AI erodes skill of devs.
Ask them if they really want to build a dependency into this ecosystem, and if they are willing to pay the heavily increased price, once it inevitably gets expensive. That you can start replacing devs with AI, but even now, at the heavily undercut prices, it’s already costing more than a developer would, and it will only get worse. There are articles for that too. Make sure to highlight that you will be giving control of your systems and development to something that will be expensive, and it will be expensive to walk back on those decisions, once it’s integrated deep enough.
Get a paper trail. Make them sign off on the risks. Don’t do anything unless they sign off on the risks, and make it super clear how stupid idea it is. Bring as many recipes, papers, and blogs you can find.
Make a citation for how much it would cost to run a competitive local models. How much you would have to spend on GPUs to have a local solution for all of your planned AI need, so you can continue working once third party AI is not affordable.
Also take a look at all currently unresolved GDPR and general laws around copyright, unsolved cases. Send them blog posts that imply that there might be a copyright problem with AI generated outputs down the line, once all lawsuits resolve.
Mention the google lawsuit in germany, where Google might be legally responsible for the outputs it’s AI gives. Ask if that is a risk they accept, assuming it passes.
It’s an extremely stupid idea to currently even consider being dependant on AI. Give the recipes and arguments, and have them directly sign off on every risk you can think off. Basically - whatever you do, do a proper long term risk analysis, and make sure the management signs off on the risks.
You will need it down the line. There is a lot of risks.
Nah, show them that it’s an expensive waste of time by continuing to let them run wild with useless but impressive tasks
We don’t even know what your company does? Does it do development? Data analysis?
Without knowing that, as head of AI, I’d first take training to understand what AI is and get advice from an external company as to what it could be good for. “AI” is being used as an umbrella term nowadays for neural networks to stochastic machines and they all have different uses. Understanding valid and invalid usecases is important before deciding when and where to use it in the company.
May want to consider setting up a private, on-prem system. That way, you can reliably enforce privacy/GDPR rules. You can also tweak the system to support local training, RAG, MCPs, etc.
This way, the costs can also be controlled. It’s some capital investment in local hardware, plus reasonably fixed power/cooling/maintenaance ongoing expenses.
Another way is to use the major AI services for planning/brainstorming specific features, tell it not to implement or touch anything, but to generate a detailed plan for an implementer LLM. Review that plan manually, and when ready, feed it to your local system for implementation and debugging.
This doesn’t work if the goal is one-shot vibe-coding. But it works really well for focused feature enhancements, test coverage, and bugfixing.
Figure out what the executive’s day-to-day workflow is like, then make agents to do their job. Either they’ll get an understanding of how inadequate it is, or they’ll be able to just sit around doing nothing useful (assuming that’s not what they do already).
Not sure on the size of your team, but you could perhaps form a small subset of AI “champions” who can try to help drive new initiatives. Just some ideas:
-
Start out with foundational usage of AI agents, ensure people have the right mindset and ownership of using the ai agents. Ensure there is always a “human in the loop.”
-
Arrange for some training for people (not just Devs) on the right way of using AI, such as not throwing sensitive information into it. Educate people that it’s far from perfect
-
Get standard boundaries in place for your current and future projects, get context files (agent.md) for your projects so those who do use the agents will be doing so from the best position.
-
See if you can arrange with the higher ups to arrange some company time do a Hackathon with the use of AI to help people see what they can do with it and the challenges they’ll face. You can also take some learnings from it.
The way I see it, you’ve been given the position of head of AI because you’re indifferent. It’s your job now to drive engagement (because that’s what your bosses want) but in a controlled manner. If they put someone that was overly enthusiastic about AI they would end up with a huge bill and lots of AI mess to clean up.
-
You could just quit since you clearly have no desire to succeed. Or just wait until they find out and fire you.
As a bonus you can blame “AI” either way.
How goofy do you have to be to interpret someone as asking for some advice as not wanting to succeed?
Because they’re clearly looking for a way to appear to be doing the job without having to do the job.
There’s no way this forum can help with this question without knowing a lot more about the company, what they do, what they want/need, etc.
If they truly are looking for help from this forum of ignorant strangers then they’re an idiot and should just be fired anyway.
Glad I don’t work with you.
