Human oversight for high-risk AI (EU AI Act)

I was watching this video about how to keep agents controlled and not let them do things they shouldn't. It's quite a technical talk, but if you're interested, it's worth a watch.

One thing during the talk got my attention. In one of the slides, I saw this:

I know that Europe is doing a big push in getting things controlled and some companies don't like it (I'm talking about you, Apple), but it's interesting to see that these projects are taking these kinds of things into consideration. In this case, I'm talking about Gitmesh in case you're interested in checking it (again, super technical and probably something you don't need to worry about).

Going back to the EU AI Act (Art. 14), here are some interesting points.

1. High-risk AI systems shall be designed and developed in such a way, including with appropriate human-machine interface tools, that they can be effectively overseen by natural persons during the period in which they are in use.

Looks obvious but not quite. All AI companies, one way or another, are telling us to prompt the agent and let it work for us. Having tools that let us control the system and stop it from doing nasty stuff is super important.

2. Human oversight shall aim to prevent or minimise the risks to health, safety or fundamental rights that may emerge when a high-risk AI system is used in accordance with its intended purpose or under conditions of reasonably foreseeable misuse, in particular where such risks persist despite the application of other requirements set out in this Section.

This is a given, especially in an age where AI is being used in war and other nasty things. Notice that the article states "risks to health", whatever that means and however this can be interpreted. We don't associate "health" with AI, since we use it daily for automating stuff, but it's such a huge concern that it's the second point in the act.

5. For high-risk AI systems ... no action or decision is taken by the deployer on the basis of the identification resulting from the system unless that identification has been separately verified and confirmed by at least two natural persons with the necessary competence, training and authority. The requirement for a separate verification by at least two natural persons shall not apply to high-risk AI systems used for the purposes of law enforcement, migration, border control or asylum, where Union or national law considers the application of this requirement to be disproportionate.

I like the idea that two people who are trained need to verify and authorize. It looks like the movies where we have two people with keys turning them at the same time for something important to happen, but AI is quickly becoming something so dangerous and so propagated that it's important to think about the implications.

Final Thoughts

I liked the video a lot, especially for the safety it brings, but I especially liked the concept that it's not only about limiting AI from doing wrong things (and there's a tool for that), but it's also based on rules and guidelines put in place by Europe. AI is global and Europe is playing, unfortunately, a small role, but I'm happy that someone is thinking of how to keep things safe when other nations are "pushing forward and breaking things".

Photo by Sushobhan Badhai on Unsplash