An Information-Theoretic Predictive Model for the Accuracy of AI Agents Adapted from Psychometrics

• Published in: Artificial General Intelligence pp. 225 - 236
• Main Authors: Chmait, Nader, Dowe, David L., Li, Yuan-Fang, Green, David G.
• Format: Book Chapter
• Language: English
• Published: Cham Springer International Publishing
• Series: Lecture Notes in Computer Science
• ISBN: 9783319637020; 3319637029
• ISSN: 0302-9743; 1611-3349
• DOI: 10.1007/978-3-319-63703-7_21

We propose a new model to quantitatively estimate the accuracy of artificial agents over cognitive tasks of approximable complexities. The model is derived by introducing notions from algorithmic information theory into a well-known (psychometric) measurement paradigm called Item Response Theory (IRT). A lower bound on accuracy can be guaranteed with respect to task complexity and the breadth of its solution space using our model. This in turn permits formulating the relationship between agent selection cost, task difficulty and accuracy as optimisation problems. Further results indicate some of the settings over which a group of cooperative agents can be more or less accurate than individual agents or other groups.