https://futurism.com/the-byte/government-ai-worse-summarizing
The upshot: these AI summaries were so bad that the assessors agreed that using them could require more work down the line, because of the amount of fact-checking they require. If that’s the case, then the purported upsides of using the technology — cost-cutting and time-saving — are seriously called into question.
I still think in development environments, limited LLM systems can be used in tandem with other systems like linters and OG snippets to help maintain style and simplify boilerplate.
I use Co-Pilot at work because I do development on the side and need something to help me bash out simple scripts really fast that use our apis. The codebase we have is big enough now (50,000 ish lines and hundreds of files) so it tends to pick up primarily on the context of the codebase. It does still fallback to the general context pretty often though and that’s a fucking pain.
Having the benefits of an LLM trained on your own code and examples without the drawbacks of it occasionally just injecting random bullshit from its training data would be great.