UsefulResources

This page is supposed to gather useful resources regarding tasks for evaluating in-development AGI systems (ultimately aimed at human-level, virtually or physically embodied general intelligence).

Links may be placed here (preferably with careful annotation regarding what is at the end of each link and what its importance is), or information may be pasted here when appropriate.

The initial goal here is to form a list of actual tasks that have been used to evaluate developing human minds, which are sensibly portable to the virtual world and/or (preferably but not necessarily "and") robotics context. So if you point to a resource describing tasks used for humans, inserting some comments on how they can be ported to the virtual world and/or robotics context would be valuable.

Once a list of tasks has been made here, we can set about categorizing them, organizing them, and so forth. I guess Step 1 is "just" to brainstorm about what kinds of tasks make most sense though.

Piaget can be one guide, but there is a lot else out there....

Arrabales Roadmap to Advanced Machine Consciousness

These two papers (variants of the same paper) by Raoul Arrabales

http://agi-roadmap.org/Image:Arrabales_ICCI09_v30_CR.pdf

http://agi-roadmap.org/Image:Arrabales_JCS2009_v34_refs.pdf

are highly relevant and could be used as an intuitive guide to constructing an AGI roadmap (though the completeness of Arrabales' categorization is not obvious, it seems a good start.)

(From an AGI Roadmap perspective, we can ignore the issue of "what is consciousness" and just look at Arrabeles ideas as related to cognition.)

Statistical Learning

Speech Segmentation: Even young infants can learn statistical contingencies in continuous streams of speech, thus enabling them to distinguish words from nonwords. (cf. Dick Aslin)

Word Learning: Analogously, infants and adults can learn a surprising number of novel word-object mappings (i.e., nouns) from a set of individually ambiguous trials, solely on the basis of consistent co-occurrence of "correct" words and objects. (cf. Chen Yu)

Recently, a number of other phenomena have been investigated in the statistical learning framework, including category learning, causality, and other forms of visual learning.

The statistical structures used in these tasks are all quite simple, so an AGI that can do them in principle need not be impressive. However, the embodied tasks (involving actual recognition of aural and visual stimuli) are still relatively hard problems in AI. I do think that these basic tasks (which are not entirely standardized, unfortunately) are good initial targets, but certainly not sufficient for intelligence.

Emotional Intelligence

An interesting alternative measure is Emotional Intelligence. [1]
Many of the IQ tests rely heavily on language [2].
Also, a lot of the tests and articles are copyrighted and only accessible for a fee. It makes it harder to evaluate them for AGI purposes.

Animal Intelligence

I am interested in what kinds of non-verbal tests can be done before an AGI has reached the vebal stage. There are all sorts of tests that are being done with very young children and animals.

Early tests:

• Recognition of self
  

While the AGI is asleep, put a colored dot on the AGI where it can't be seen except with a mirror. Wake up the AGI in a room with a mirror. See if the AGI responds to the mirror as if the reflection is another AGI/person or recognizes that the reflection matches itself. For example, it might try to remove the dot. [3]

• Maintenance of self
  

The AGI is damaged by:

Removing part of its code

Removing part of its knowledge base

Altering its embodiment (eg. removing a wheel or a limb or a sensor)

To pass this type of test the AGI should firstly be able to detect that some change has occurred to itself and secondly take compensatory actions which minimise the effects of that change upon subsequent performance.

• Spatial memory
  

Can the AGI remember where the bathroom/kitchen/bedroom is when it needs it?

Can the AGI find its way out of a maze?

Can it recognize and cope with environmental changes over time? (The room is the same; only the furnishings have changed.)

• Problem solving

An adult raven can solve this in one attempt:

Tie a piece of meat to the end of a string which hangs from a branch. The string is too long to be pulled up by the beak in one pull. The raven looks at it awhile, then pulls the string up and steps on the string, repeating until the meat is within reach.

• Learning by observing others

One chimpanzee sees another use a stick to pull termites out of a nest and imitates the behaviour.

• Building an maintaining group identity

For intelligent context based learning within a social group which may include humans and other AGIs:

Identify key points within the self model which correspond to phase changes in cognition

Use the self model to create status models for conspecifics

When in communication with other systems actively seek to evaluate and update conspecific status models

Use predictions based upon status models for disambiguation and pattern completion

Within a simulation environment where the cognitive state of all entities is known the the efficiency/accuracy of status models can be measured.

• Ability to predict future outcomes

One raven sees another raven raid the food cache of a third raven and then takes steps to be sure the second raven can't see where it hides its own food. [4]