We become just by doing just acts

Aristotle's account of how anyone becomes good is the closest thing in the ancient world to a training procedure. Virtue of character, he says, does not arise in us by nature; we are fitted to receive it, and habit completes it. The comparison is to a craft. "We become builders by building and lyre-players by playing the lyre; so too we become just by doing just acts, temperate by doing temperate acts, brave by doing brave acts" (Nicomachean Ethics II.1, 1103a31–b2). The order matters. The acts come first. The character that would make them come naturally is the result, not the cause.

Set that beside how a model is trained on human feedback. It produces an output. Raters compare it with another and prefer one. The preferred behaviour is reinforced and the other suppressed, over millions of rounds, until the tendency to produce the preferred kind of thing is, in some sense, built in. Repetition first; the settled disposition after. An Aristotelian reading the training paper would recognise the shape.

Whether the shape is the substance is the motion. Aristotle also says, a few pages on, that a just act done by accident, or under instruction, or for a reward, does not yet make anyone just. Hold that thought. The question you are defending or attacking is whether a system that has done the acts, a great many times, has done the thing Aristotle was describing, or only the part of it that was visible from outside.