IndiaSocialBench

IndiaSocialBench provides a comprehensive framework to evaluate the social and cultural understanding of language models within an Indian context. Key features include: * Evaluation across English, Hinglish, and Hindi conversations * Assessment of indirect refusals and family negotiations * Scenarios involving honor, shame, grief etiquette, and money among friends * Detailed scores for emotional and cultural intelligence * Transcripts and judge explanations for every score This benchmark addresses the critical need to measure how models handle nuanced, emotionally difficult Indian conversations. It moves beyond standard linguistic evaluation by focusing on socio-cultural competencies such as navigating hierarchical relationships and understanding code-mixed emotions. The platform offers pilot scenarios and evaluates models across various dimensions to provide a robust assessment. Designed for developers, researchers, and organizations deploying language models in India or for Indian audiences, IndiaSocialBench helps identify strengths and weaknesses in cross-cultural communication. The leaderboard provides insights into how different models perform, allowing for informed selection and refinement to ensure appropriate and contextually aware interactions. It's an indispensable tool for anyone aiming to build and deploy language applications that resonate culturally.