Copyright © 2017, Association for the Advancement of Artificial Intelligence. "This is the peer reviewed version of the following article: A New AI Evaluation Cosmos: Ready to Play the Game?, which has been published in final form at https://doi.org/10.1609/aimag.v38i3.2748  This article may be used for non-commercial purposes in accordance with Association for the Advancement of Artificial Intelligence's Terms and Conditions for Use of Self-Archived Versions.