Thanks for this initial foray — I've thought about this a little, expect to think about and work with/around it a lot more in the coming years.
What I notice with your puddle visual/metaphor vis-a-vis the MVC data/logic flow diagram: The ML/puddle obfuscates/muddies the logic, makes it inaccessible. The problem with this is that the principles of Design — and of humanism and any/ever other principles we want to bring to bear on AI/ML functioning — must be applied to this puddle in a nuanced way if we are not to forfeit Design's role in this emerging computational era.
The key question is How. It occurs to me that it took a couple decades of computing before the notion/framework of MVC became possible to conceive — and perhaps our current era of explainability struggles with AI will similarly lead to clear frameworks that we have not yet imagined.