Cleverness is not wisdom

The sharpest objection to deploying a system you cannot explain comes from Aristotle, and it turns on a distinction most people never draw. In Nicomachean Ethics VI.12 (1144a23–29) he separates practical wisdom, phronesis, from something he calls cleverness, deinotes. Cleverness is the ability to hit whatever target you have set. If the target is good the cleverness is praiseworthy; if the target is bad it is "mere smartness." Practical wisdom includes cleverness but is not the same thing, because the wise person also sees what the target should be and can say why.

An opaque optimiser is cleverness in its purest form. It hits the mark. What it cannot do, by construction, is give an account of the mark, notice when the mark is wrong, or tell a case its training covered from one it did not. On Aristotle's view that is the gap between a tool and an agent, and the gap does not close because the tool's hit rate is high.

So the objection is not that explanation would be nice to have. It is that the deployers have handed the judgement about the target to something with no faculty for judging targets, and kept only the easy part of the decision. A long record of hitting the mark says nothing about whether anyone is still looking at it. The utilitarian reply has to say why that does not matter, and "suffering went down" restates the claim without defending it.