advertisement
Jay Ramnundlall: Why Human Accountability Must Remain At The Centre Of AI Governance
Artificial intelligence is moving beyond generating reports, answering questions and recommending decisions. Increasingly, organisations are deploying AI agents capable of executing instructions, approving transactions and interacting directly with corporate systems.
For business leaders, this transition presents a difficult question: when an AI system acts independently and something goes wrong, who takes responsibility?
It is a question explored by Jay Ramnundlall, CEO, Alpvest Institute of FutureTech (AIFT), in his recently published book, The Human Advantage: Judgement, Responsibility and Leadership in the Age of Artificial Intelligence.
advertisement
Ramnundlall argues that the growing capabilities of AI should not translate into the surrender of human judgement. While organisations can delegate intelligence and routine decision-making to machines, responsibility and accountability must remain with people.
His central argument is captured in a simple principle: when AI acts, leadership must answer.
In an interview with CIO Africa, Ramnundlall discusses the changing relationship between human authority and machine autonomy, the responsibilities of corporate boards, the risks of excessive delegation and the skills leaders will need as AI assumes more organisational functions.
advertisement
Your new book explores responsible AI adoption. What is its central argument?
Ramnundlall: The book is about responsible use of AI, placing judgement, responsibility and leadership firmly under human control.
There are different views from world leaders, religious leaders and technology companies about how artificial intelligence should be regulated and what policies should govern its development.
advertisement
The book seeks to raise the important questions and provide direction on how these issues should be considered.
It is not prescriptive, except in one fundamental respect: it places the control and growth of AI firmly in human hands.
What changed when AI moved from simply advising people to making decisions and taking action on their behalf?
Ramnundlall: That is the concerning part. When AI moves from generating content and making recommendations to taking decisions and executing actions, the amount of authority we delegate to it becomes critical.
Traditionally, AI might generate a message or recommend approving a transaction. A human being would then intervene before anything happened.
Now, machines can execute those decisions before a person intervenes. Errors can therefore spread quickly.
Leaders must determine the system’s authority, its limits and, more importantly, its escalation requirements.
We can delegate routine, lower-level decisions because doing so is economical and efficient. However, where greater judgement is required, human intervention must remain possible.
People must have the authority to stop a machine from making an inappropriate or incorrect decision, slow it down or even shut it down completely.
We can delegate intelligence, but the responsibility for what has been delegated must remain in human hands.
Where should accountability for AI decisions ultimately rest: with the CIO, CEO, board or organisation?
Ramnundlall: The board is ultimately accountable for governance. Before approving autonomous AI systems, it has a responsibility to define their parameters at the design stage.
That includes establishing the purpose of the system, the evidence it should support, the expected benefits and what it can do without further approval.
The board must also consider deployment and operational accountability. What could go wrong? Who could be harmed? What happens when the system makes an incorrect decision?
There are three key levels of responsibility: CIOs, CEOs and boards.
Each must map the authority delegated to AI, establish where it is being used and identify who owns each task or application.
They must also test accountability through realistic scenarios, particularly failure scenarios.
Can people detect, stop, explain or remedy an AI-driven action? If the system goes out of control, who presses the kill switch, and what happens afterwards?
Finally, organisations must invest in the capabilities needed to manage these systems.
It is easy to say the board is accountable, but can an average board manage a complex AI system? That is a question organisations must answer.
Should all decisions remain under human control, or can some be entirely delegated to AI?
Ramnundlall: AI can make decisions up to a point. It makes economic sense for routine, structured decisions to be automated.
For example, McKinsey recently published research on banks in Europe involving what it called an investigator agent and a judge agent. The judge agent is given greater responsibility to assess the investigator agent’s findings.
This demonstrates that decision-making can be delegated quite deeply while retaining human checks and balances.
One of the concepts I introduce in the book is the governance gap. This is the gap between effective control and AI capability.
I also describe it through the accelerator and brake model.
The accelerator represents computing power, investment, adoption, automation and increasing autonomy. It also includes AI-assisted research and the ability of models to develop, test and train other models.
The brake represents evaluation, security, law, institutional competence, independent assurance, oversight and intervention.
The wider the gap between these two forces, the greater the risk of a catastrophic event.
We must use both the accelerator and the brake to maintain meaningful control and realise AI’s benefits without allowing systems to go out of control.
What does meaningful human oversight look like beyond having someone approve an AI-generated recommendation?
Ramnundlall: Meaningful human oversight is critical because organisations sometimes place someone in a process simply to rubber-stamp decisions.
Consider a situation where a large volume of applications is being processed. If the reviewer simply approves everything, that is not oversight. It is an abandonment of responsibility.
The reviewer must understand the decision and have the capacity to challenge its underlying assumptions.
They also need sufficient information, time and, more importantly, authority to reject an AI-generated recommendation.
They should be able to pause a process and escalate concerns promptly.
Warning signs include excessive workloads and reviewers who cannot explain what they are approving.
A human signature alone does not demonstrate effective oversight. People must have the real authority to act independently and meaningfully.
What are the biggest challenges facing boards and executives overseeing increasingly complex AI systems?
Ramnundlall: The challenge is building the capabilities that prepare people, particularly those in leadership positions, for AI.
In my book, I identify three forms of competence.
The first is independent competence. This means understanding a subject and being able to reason without complete dependence on AI. It preserves the human element of deeper knowledge and wisdom.
The second is augmented competence. This involves using AI effectively while checking its contribution.
Using AI without verifying its work amounts to abandoning one’s responsibilities. Verification becomes particularly important as organisations apply AI to increasingly complex tasks.
The third is supervisory competence. This involves understanding the limitations of AI, challenging its recommendations and knowing when human intervention is necessary.
The objective is to ensure that people understand their work and take responsibility for it rather than surrendering that responsibility to an AI model.
As machines become more capable, leaders must become better at asking questions and exercising judgement.
Wisdom is particularly important. Human values, relationships, feelings and a sense of belonging are dimensions that cannot simply be reduced to what an AI model can define.
Leaders must also develop the capacity to challenge both technology and authority, particularly when governance structures or economic considerations stand in the way of appropriate decisions.
My simple message is that we may delegate to intelligence, but we should never surrender judgement, responsibility or accountability.
What risks emerge when organisations give AI agents access to corporate systems, financial platforms, customer records and other sensitive resources without adequate human supervision?
Ramnundlall: The risks are inherent in the system’s design and the guardrails established around it.
Whenever an AI system is conceptualised, there must be an upfront assessment of potential risks. Organisations must identify what happens when particular scenarios occur and establish whether the system should respond automatically or escalate the matter for human intervention.
The greater the authority delegated to a model, the greater the risk of losing control if the necessary safeguards are absent.
Organisations must assess risks associated with the technology, cloud infrastructure and consumer protection before deployment.
They must then establish appropriate guardrails.
We might also see greater migration towards private cloud environments because of concerns about AI systems accessing private networks, databases and other sensitive resources.
That is another area organisations need to consider seriously.
AI systems are increasingly operating at speeds beyond human supervision. Will people eventually lack the capacity to control them?
Ramnundlall: There is no doubt that the capabilities of these machines can spiral out of control.
With new models being used to train future models, humans may simply be unable to control every development.
That makes it critical to preserve the ability to apply the brakes when designing these systems.
I recently experienced something similar while driving a hired car. As I travelled downhill, I felt that I was losing control. What do you do in that situation? You slow down and press the brakes.
That is precisely the principle organisations should apply to AI.
We should welcome the benefits of speed, productivity, technology adoption and profitability. However, the resulting AI dividend should also be invested in building safeguards.
The moment we realise that systems are accelerating faster than our ability to control them, we must start applying the brakes, or even slam them if necessary.
What leadership and workforce capabilities will become essential as AI assumes responsibilities traditionally held by employees and managers?
Ramnundlall: Traditionally, people developed skills progressively through education and employment.
A school leaver would enter university, complete a degree and then join the workplace, where they would acquire basic skills through practical experience. They would progress into middle management before eventually becoming senior managers.
AI is changing that progression.
Elementary tasks that traditionally helped employees develop professional competence can now be automated or completed by machines.
This creates a risk that organisations will skip the lower steps of the career ladder.
We may enjoy the immediate efficiency gains, but five to 10 years from now, organisations could find themselves without enough middle-level professionals to progress into senior positions.
If AI takes over the tasks through which people traditionally gained experience, we risk denying future leaders the opportunities to develop.
Business leaders must take this seriously. They should invest in training people to develop the skills required to work with future technologies and use AI responsibly.
Education faces similar challenges and opportunities. Traditional assessments are becoming less reliable because AI can produce work that previously demonstrated a student’s knowledge and competence.
We may therefore need to place greater emphasis on oral assessments, practical demonstrations and other ways of validating what students actually understand.
Ultimately, the AI dividend should not simply mean exploiting the additional profits generated by automation.
Organisations must also invest in people and embed human accountability throughout their processes.