Black-box judicial decision-making

generative AI, apparent reasoning and the accountability crisis

Abstract

This article critically examines the legal limits of using generative artificial intelligence in the decision-making stage of the Brazilian digital judicial process. It addresses the extent to which systems capable of producing legal language may, under the pretext of efficiency, undermine judicial reasoning, legal certainty, and accountability for errors, biases, or hallucinations incorporated into judicial decisions. The research adopts a qualitative approach, a deductive method, and bibliographic and documentary procedures, analyzing legal scholarship on judicial decision-making, due process, algorithmic opacity, and accountability, as well as national and foreign regulatory instruments, especially CNJ Resolution No. 615/2025, Brazilian Bill No. 2,338/2023, and the European AI Act. It argues that generative AI may assist organizational, research, and drafting activities, but cannot occupy the place of human judgment responsible for interpreting the law, assessing evidence, and constructing the reasons for the decision. The article concludes that AI-assisted judicial decisions will only be legitimate if subjected to functional transparency, effective human oversight, traceability, auditability, contestability, algorithmic impact assessment, and a clear allocation of responsibilities. Without such safeguards, technology no longer serves jurisdiction and instead produces a merely apparent form of reasoning: formally sophisticated, legally opaque, uncontrollable, and incompatible with due process of law.