New York City's AI Chatbot is Telling Businesses to Break the Law: Report

New York City's chatbot assistant for business owners is reportedly generating misinformation and advising users on actions that could possibly violate the law, according to local watchdog The City.

According to reports, the AI chatbot is providing incomplete or "dangerously inaccurate" information on housing policies, worker rights, and advises for entrepreneurs.

The chatbot is also spotted to be not knowledgeable with recent laws the state has passed, including recent tipping and transaction laws.

The AI supposedly "access trusted information from more than 2,000 NYC Business web pages" for all answers it gives.

A spokesperson from the NYC Office of Technology and Innovation told The City that it will "continue to focus on upgrading this tool" to fix such issues.

The city, however, maintained that the chatbot "has already provided thousands of people with timely, accurate answers" and that it is open about its risks in using the AI tool.

New York City first launched the chatbot last October using Microsoft's Azure AI, which in turn was powered by OpenAI's technology.

The report was made in coordination with nonprofit tech news publication The Markup.

NYC Leans More on AI Technology for City Operations

The business assistant is not the only chatbot New York City has employed as it started leaning more into AI amid the technology's increasing trend.

The city has earlier started using the technology for its robot police assistant to protect commuters in subways.

Much recently, city announced that will soon enable more AI into city subways to detect guns and other dangerous metal weapons.

Despite the ramp-up on the technology, New York has provided few safety measures to prevent the AI from negatively impacting its recipients.

AI Still Prone to 'Hallucinations'

The issues on New York City's business AI assistant are not an isolated case of chatbots providing inaccurate and even sometimes dangerous information to its inquirers.

Several AI experts have warned AI firms to improve safety measures against "AI hallucinations," where the AI is conjuring answers out of nowhere, amid increasing accessibility of their technology.

Experts are worried that such issues can further cause political information ahead of the election period as people turn to AI as a source of information.