What Uber's breach notice says about giving data to AI vendors

SC Media

Source published: Site published:

Original article

COMMENTARY: Uber is telling about 150,000 drivers that pictures of their driver's licenses, along with their names, home addresses, and dates of birth, were stolen last month.

The company's notices to state attorneys general explain how this happened. The details are worth looking at carefully. On October 7, someone used login details belonging to one of Uber's service providers to enter a cloud storage system that held data Uber had shared with that provider. Uber's own cloud security team had set up and managed this storage system. Security alerts were turned on, but none went off because the login details were valid. Uber has since confirmed that the provider was Fireworks AI, a platform used to run and fine-tune open-source AI models.

I should mention that my company works in AI infrastructure, which is the same general market as Fireworks. So I have an interest in this topic, and I don't want to unfairly criticize anyone.

Fireworks published a notice within a few days and has updated it since then. Uber followed the usual security advice: it kept the data in a storage system that it controlled. But the incident still happened. That makes it a useful example to learn from.

Three things stand out to me.

First, the security monitoring was based on checking logins. A system that only checks whether someone used the correct key cannot spot a thief who has the correct key. To detect problems when a vendor has access to part of your system, security checks must also look at behavior. They should check how much data was read, where it was read from, and at what time. They should compare this with what the vendor normally does. A system that fine-tunes an AI model reads data in a regular, recognizable way. Someone copying an entire storage folder behaves differently.

Second, when a vendor has your login details, your security also depends on the vendor's security. The notices don't explain what kind of login details were involved or how they were stolen, so I won't guess. But the general lesson is clear. Keys that work for months and can be used from anywhere are only as safe as the least secure computer holding them. Short-lived login details that only work for a particular job and network connection make stolen keys much less useful. Setting this up takes more work, which is why these special access arrangements are often missing.

Third, there is a special problem with AI: the data was in its original form. Uber hasn't explained why the license pictures were being used. One possible reason for using an AI platform is to train or test a model that can read documents. For this kind of work, the vendor's systems need the actual pictures. This is different from older types of outsourcing. A payment processor can work with tokens, and a customer management system provider can work with information that has been hidden or covered up. But an AI model that learns to read driver's licenses needs the licenses themselves. As a result, AI projects often put the most sensitive version of the data in a place where access controls are weakest.

I don't believe there is one solution that fixes everything. I would be careful about trusting anyone who claims there is. But some steps can help. Only send the amount of data needed to do the job. If 5,000 pictures are enough, don't send more. Set an end date for access to the storage folder. Delete the data when the job is finished, and check that it really has been deleted. Ask whether the vendor needs access all the time or only while a job is running. If the platform allows it, keep the data encrypted and only release the key to the specific job that needs to read it, rather than to anyone who can use the vendor's login details. Before the project begins, decide who should be contacted if the vendor reports a security problem on a Friday night.

One more detail deserves attention. At first, Fireworks said it had found no evidence that customer data, models, or datasets had been accessed. A month later, it said that some had been accessed. I don't blame the company for this because early investigations into security incidents are usually incomplete. But customers should take this as a reason to assume from the first day that more data may be affected. They should not wait for the vendor's investigation to tell them what was in their own storage folder. Uber stopped sharing data with the provider on the day it learned about the incident.