In light of the proposed smart urban development projects and AI-based applications, this paper examines various use cases of AI being developed for cities in India, in order to identify key challenges and risks.