Understanding the limitations of systems based on deep learning and their impacts

Systems based on deep learning require large amounts of high-quality data to achieve good performance. In addition, the decisions developers make during system development also influence quality. Although these systems can achieve high performance and learn impressive tasks, they can still draw incorrect conclusions or be biased.

Furthermore, systems based on deep learning are inherently complex and are often referred to as a "black box" because they produce an output for a given input without revealing exactly how it was processed. In our simulation, we used a very simple artificial neural network with a basic architecture, a few inputs, and a few hidden layers. In practice, however, applications often look very different: they use deep neural networks trained on massive datasets and have significantly more complex architectures—with many inputs and numerous hidden layers and weights (also called parameters). As a result, it is more difficult to understand how these systems function and why they produce specific outputs.

Alexa Steinbrück / https://betterimagesofai.org / https://creativecommons.org/licenses/by/4.0/

Based on this, the following problems may arise:

Lack of trust in outputs from AI systems

Due to the potential limitations of these AI systems, users still lack trust in using them to support decision-making processes. Much of this mistrust stems from the fact that users do not understand how these systems were developed and how they function, which data were used for training and evaluation, and how these systems arrived at such conclusions, making it difficult to assess whether the results can be trusted or not.

Excessive reliance on outputs from AI systems

On the other hand, users may blindly trust the outputs of AI systems because they do not understand how these systems work and where their limitations lie. The uncritical acceptance of AI system outputs can amplify errors or misinformation and lead to harmful consequences. This can be particularly problematic in highly sensitive areas such as criminal justice (e.g., recidivism prediction) or policing (e.g., facial recognition).

Both issues can raise ethical concerns. If users do not trust AI outputs, they may miss opportunities to benefit from AI-supported decision-making. Conversely, if users rely too heavily on AI outputs and those outputs are inaccurate or biased, this can result in serious consequences that directly affect people's lives.