Agents placed in the economic society we just described will have to learn, via trial and error, what actions give them the highest expected reward and under which circumstances. In this section we will present techniques that these agents might use to maximize their rewards.
An important question we wish to answer is: when do agents benefit from having deeper (i.e. more complex) models of other agents? It should be intuitive that, ignoring computational costs, the agents with more complete models of others will always do better. This seems to be usually true, however, there are instances when it is significantly better to have deeper models, and instances when the difference is barely noticeable. These instances are defined in part by the set of other agents present and their capabilities and preferences. In order to precisely determine what these instances are, and in the hopes of providing a more general framework for studying the effects of increased agent-modeling capabilities within our economic model, we defined a set of techniques that our agents can use for learning and using models.
We divide the agents into classes that correspond to their modeling capabilities. The hierarchy we present is inspired by RMM [6], but is function-based rather than matrix-based, and includes learning. We start with agents with no models (also referred to as 0-level agents), who must base their actions purely on their inputs and the rewards they receive. They are not aware that there are other agents out there. Agents with 1-level models are aware that there are other agents out there but have no idea what the ``interior'' of these agents looks like. That is, in RMM terminology, they are incapable of ascribing intentions to others. They must make their predictions simply based on the previous actions of the other agents, by building sub-intentional models of others. Agents with 2-level models have intentional models of others (i.e. have models of their beliefs and inference processes) and believe that others keep sub-intentional (i.e. 1-level) models of others. We can similarly keep defining agents of three, four, five-level models, but so far we have concentrated only on the first three levels. In the following sections, we talk about each one of these in more detail and give details about their implementation. Our current theory only considers agents that are either buyers or sellers, but not both.