Research in Virtual Worlds FAQ

Figure 1: CSN staff avatars

This page is intended as a brief primer on various aspects of performing research using virtual worlds, aimed towards economists familiar with experimental research methods. If you have any questions that aren't covered here, please get in touch!

Why do research using virtual worlds?

A key goal of experimental design is to create procedures with high external validity - that is, we should be able to make inferences from the behaviors observed under the simplified models that we implement in our experiments to the real-world situations that inspired our models. This is the parallelism condition elucidated in Smith (1982) as one of the sufficient criteria for economic experiments. In recent decades, economists have become increasingly-focused on running field experiments in order to test the external validity of laboratory results as well help develop new theories that can be explored in the lab.

The research I've been involved in at CSN has aimed to create environments that help bridge the gulf between the experimental laboratory and the field. The virtual world tools I've worked with can be used to create environments that are closer to the field in many important ways. Virtual worlds such as the one we use at CSN (OpenSim) can be used to easily implement many field-like features, including but not limited to:
  • Avatar-intermediated interactions that reduce social distance between subjects while retaining anonymity
  • Graphical contextual cues in the environment that invoke field-like heuristic reasoning
  • Real-time interactions
  • Spatial dynamics
Furthermore, these features can be leveraged in ways that avoid several common (although not ubiquitous) drawbacks of field experiments:
  • Full experimenter control is maintained - as experimenters, we can observe everything that subjects in our environment are doing, which allows us to generate rich behavioral records of subject behavior as well as preserve the integrity of the design.
  • We can implement our designs at an equal cost to traditional laboratory experiments.
  • We can readily share our experimental code and procedures in order to readily execute our experiments at new locations or to facilitate attempts at replication.
It is not the goal of my research to replace traditional laboratory experiments or field experiments, but to help develop new tools that we can use to explore the many research questions to which virtual world experiments may be well-suited to answer. In many ways, the initial steps that we've taken at CSN into this area have been quite conservative, and one could imagine implementing more-ambitious designs involving experiments that last for days or weeks rather than hours and utilize subjects from all over the world all interacting in the virtual spaces that we design.

What tools do you use for virtual world research? How hard are they to use? How expensive are virtual world experiments to run?

Figure 2: OpenSim build mode screenshot, using the Firestorm viewer.

The software platform that CSN uses for our virtual world experiments is known as OpenSim, a screenshot of which can be seen above. OpenSim is an open-source virtual world platform that can be downloaded and installed for free. Running OpenSim sets up a server that can be connected to by client programs known as "viewers." All viewers are free to use and many are open-source, including the preferred viewer of CSN - Firestorm Viewer. Since initializing and maintaining a persistent OpenSim server can be a daunting task, there are many commercial firms that can take care of these steps for a reasonable fee. One popular host, Kitely, offers hosting services starting at $15/month.

Once a server has been established, you will be able to generate a number of regions on that server. Regions are the basic unit of terrain on OpenSim (by default they are 256 by 256 virtual meters in size), and region data can be moved from one OpenSim server to another, which is a key feature that enables the replicability of OpenSim experiments. All of the experiments we've run at CSN have taken place within single regions, which means that we can readily export all the code for our experiments and transfer it to others.

Actually programming implementing an experiment within the virtual world can seem like a daunting task at first, but with experience becomes extremely manageable. By logging into an OpenSim region on an avatar using a viewer, you will be able to use your viewer's building tools to construct the objects in your environment as well as script them. The scripting language used by OpenSim is known as OpenSim Scripting Language ("OSSL") and utilizes a C-like syntax. Figure 2 depicts an avatar editing the default "Hello, World!" script within an object. In many ways this language is limited and lacks many of the useful features of common languages such as Python or Ruby, but these drawbacks can be managed. As is the case with most projects involving code, there is a learning curve to implementing experiments using OSSL but at CSN we've established a fairly general framework for implementing these experiments that allows us to recycle a lot of our code between different projects, allowing us to develop software just as quickly as we could using other platforms.

In terms of subject training and payments, the costs of running virtual world experiments is comparable to the cost of running a traditional laboratory experiment - largely because the experiments we've done so far have used normal laboratory procedures. We recruit subjects and pay a showup fee, and then pay them based on their decisionmaking in sessions lasting 1-3 hours. In general, subjects are able to quickly understand how to control the avatars and interact with our environments, and depending on the complexity of the environment will take between 15 and 45 minutes.

Are funding agencies interested in virtual world research? Does it produce publishable papers?

We're not the first researchers to use OpenSim or its close cousin Second Life as an experimental platform. A non-exhaustive list of recently-published research using these virtual worlds includes:
Furthermore, a broader literature exists that explores other virtual environments and how richer spatio-temporal environments may generate more field-like patterns of behavior. In many ways, this literature has just begun to explore the potential for virtual world usage to address outstanding problems in various domains of social science inquiry.

In terms of funding, grant-giving authorities have been very receptive to the proposals I've submitted to fund my research. The experiment that lead to the creation of my job market paper was funded by IFREE's Small Grants Program. This research was extended into a successful proposal for an NSF Dissertation Improvement Grant entitled Endogenous Institutions in Virtual Worlds.

What are the primary opportunities and directions that future virtual world research may explore?

Many of the prior studies that use virtual environments can be seen as taking tentative first steps into establishing the scientific merit of virtual world research, either by demonstrating that virtual world environments produce more field-like behaviors in particular environments or by demonstrating that the native users of different virtual worlds can be used as subjects to replicate prior laboratory-derived results. These experiments, however, have only begun to embrace the potential that virtual worlds may have for allowing researchers to explore new designs that would be comparatively difficult to implement in a simplified laboratory environment or the field. There are several key features of these virtual environments that can be profitably leveraged by researchers:

  • The context-rich spatio-temporal environments that virtual worlds can cheaply instantiate can be used to implement designs that want to investigate various sorts of framing effects. Virtual world environments can also be used in conjunction with different neuroscientific methods (EEG or fMRI technologies) to investigate the neurocorrelates of decisionmaking in these environments.
  • Many virtual worlds (including Second Life and OpenSim) have native populations that could be used in online experimental environments. These experiments could be implemented either as virtual field experiments within already-existing regions used by these populations, or subjects could be recruited from within the virtual world to come and interact with the virtual environment set up by experimenters. Behavioral research using online subject pools are subject to their own methodological concerns, but the growing literature that uses Amazon's Mechanical Turk (AMT) service to cheaply and quickly generate large datasets shows that best practices can be developed to overcome these problems. However, AMT is not optimized for behavioral research that requires subjects to interact for extended periods of time, and as such the technical burden of implementing complex designs on AMT may be substantially-higher than a virtual world design.
  • The persistence of many online virtual worlds means that experiments can be easily run over long time periods with a large number of participants who would intermittently interact with the experimental environment. If the subjects used belong to high-usage native populations, this could result in these experiments have very low attrition rates. Longer experiments with larger group populations could enable researchers to explore topics such as the endogenous formation of institutions in more-naturalistic contexts. Additionally, virtual world environments may be well-suited for the implementation of macroeconomic experiments that need large and complex environments.
I'm interested in exploring many of these potential applications of virtual world research in the future!