New York City (NYC)'s AI-powered chatbot is creating buzz after it generated wrong answers, including a piece of advice for businesses to break the law.
The AI chatbot was designed to help small business owners and was launched last October to act as a "one-stop shop."
NYC AI Chatbot Faces Criticism for Bizzare Advice
The chatbot has a knack for dispensing advice that misstates local policies and sometimes even encourages companies to violate the law. The AI model was designed to offer generated text responses that will help guide business owners about the city's bureaucratic maze.
Upon its launch, it came with a disclaimer stating that it may "occasionally produce incorrect, harmful, or biased" information. However, it has been consistent in providing these types of information to its customers which caused concerns with experts.
"They're rolling out software that is unproven without oversight," said Julia Stoyanovich, a computer science professor and director of the Center for Responsible AI at New York University. She also claimed that the city has no intention to be responsible for it.
NYC Mayor Defends Faulty AI Chatbot
Despite a series of issues being reported, the city decided to retain the tool on its official government website. Moreover, mayor Eric Adams only acknowledged the lapses of the chatbot and it was "wrong in some areas."
The chatbot in question had falsely suggested that it is legal to fire an employee who complains about sexual harassment, doesn't disclose pregnancy, or refuses to cut dreadlocks. In addition, it also provided some contradictory advice about the signature waste initiatives of NYC.
Microsoft, which powers the bot through its Azure AI services, assured that they are already working with city employees to improve the service. The company also promised that they would fix it to ensure that the answers will be grounded to the city's official documentation.
