The Hidden Costs of Using AI

Complexity compounds
The biggest danger of using LLMs in software development is not immediate catastrophic failure, but the gradual accumulation of complexity and hidden defects. LLMs increase code volume and complexity. Because LLMs can generate features and refactors very quickly, organizations may produce far more code than before. As complexity grows, the system becomes harder to understand, modify, and verify. A mature system can absorb some additional complexity without immediately breaking. Existing safeguards may continue to catch most errors for months or years. This creates the illusion that the new development approach is safe and highly productive. LLMs may provide a short-term productivity boost while simultaneously making software systems more complex, fragile, and difficult to evolve. Organizations will not notice the increased fragility until much later, when several hidden weaknesses interact and produce a major failure.

Ironies of Automation
Automation tends to de-skill programmers. When humans do not practice a skill—either physical or mental—their ability to execute that skill degrades. We fail to maintain long-term knowledge, but by disengaging from day-to-day work we also lose the short-term contextual understanding of "what's going on right now." Software engineers feel less able to write code after relying on code-generation models, a designer feels less creative after offloading work to ML, and studies suggesting doctors using AI-assisted polyp detection become worse at spotting adenomas themselves. Automation bias can also cause clinicians to be influenced by incorrect AI recommendations.
While automation itself is not new, previous technologies such as power looms, calculators, and CNC machines automated comparatively narrow tasks. LLMs, by contrast, are expected to automate a much broader range of cognitive work. As a result, society will need to apply the classic lessons of automation—deskilling, automation bias, monitoring fatigue, and takeover hazards—to many new professions that have not previously faced these issues.

https://aphyr.com/posts/411-the-future-of-everything-is-lies-i-guess