Seattle Seahawks fans win no matter who their team is playing on the field. Their secret weapon? Amazon’s artificial intelligence (AI).
Last year, the Seahawks’ Lumen Field became the first NFL stadium to be installed. Amazon’s Just Walk Out Technology. This allows fans to walk into the concession stand, pick up what they want, and quickly return to their seats without having to wait in line or stop at the register.
This feat combines computer vision, object recognition, advanced sensors, deep machine learning models, and generative AI, a type of artificial intelligence that has recently captured the public’s imagination.
Amazon researchers have discovered a way to make these technologies work together seamlessly behind the scenes. Now they can almost instantly see which customers had a tuna sandwich and who had a sandwich with chicken salad, charge them correctly, and walk away. their items.
“Even if you don’t know the technology, it feels like magic,” says Gerard Medioni, Amazon’s vice president and renowned scientist. “But creating that magic—identifying who took what—is harder than you might think.”
Medioni led the team of research scientists who designed the technology, which began eight years ago. Just Walk Out technology is currently available in over 70 Amazon-owned stores and over 85 third-party stores including sports stadiums, airports, grocery stores, convenience stores, and college campuses in the US, UK, and Australia is. .
“People don’t like to wait in line when it’s time to board a plane or when there’s a game to watch,” Medioni said. “We built Just Walk Out technology for shoppers who want to walk into a store, find what they need, and walk out without having to wait for the person in front of them to count their change. ”
At Lumen Field, Amazon’s Just Walk Out technology was a big winner. After installing Just Walk Out technology in September 2022, District Market, a convenience store offering beer, soda, game-day snacks and food from stadium neighbors, compared to its previous store in the same location. This led to more than doubling sales and increasing the number of fans. Satisfaction survey scores hit record highs. Lumen Field has since added Just Walk Out technology to the concession areas of eight of his stadiums, the most of any sports venue in the world.
“One of our narrowest concourses dramatically increases throughput, reduces crowding and allows us to operate at full capacity,” said Zach Hensley, vice president of operations for the Seattle Seahawks and general manager of Lumen Field. We have also seen a more stable flow in the City.” “Just Walk Out is a seamless technology that allows fans to get food and drinks and return to their seats as quickly as possible during timeouts.”
Here we take a closer look at the various technologies that make Just Walk Out technology possible.
Computer vision and machine learning enable Just Walk Out technology to protect customer privacy
To enter stores like District Market at Lumen Field, use your credit card at the entrance gate or scan your palm over the card. amazon one device. Amazon One is Amazon’s palm-based identity and payment system. Link your credit card to the palm of your hand using AI and machine learning.
Once inside the store, there are several specialized cameras installed on the ceiling that monitor the entire store area. Jon Jenkins, Just Walk Out’s vice president of technology, says that by combining computer vision and machine learning, the system “knows who has what and charges correctly when you’re out and about.”
When you pick up a can of soda, Just Walk Out technology adds it to your virtual cart. If you decide to put it back on the shelf, the technology will automatically remove it.
“Our technology can distinguish shoppers from each other without collecting or using biometric information,” Jenkins added. “Just Walk Out technology detects when a shopper’s hand touches an item on the shelf. When that happens, machine learning algorithms place the correct item in the virtual cart without any special knowledge of the person. It will definitely be added.”
Just Walk Out technology operates independently of Amazon One and does not use or collect shopper biometric information. Customer trust and privacy are paramount to the experience. Just Walk Out technology simply connects customers to a payment method. When a shopper enters a store, the technology assigns a temporary numeric code that serves as the shopper’s unique digital signature for their purchase. The system saves the code the entire time the shopper is in the store. The code disappears when you exit and you get a new code when you come back.
“Every image contains many pixels that represent a person,” says Medioni. “For example, when a shopper takes off their jacket, their signature updates as they move through the store. Just Walk Out technology does not collect biometrics. It’s just a matter of where you are and where your hands are relative to the products in the store.”
This technology is so accurate that it can also track groups of shoppers. Multiple people can enter the store using one credit card. The system tracks each individual but associates groups with the same payment method, helping families shop. When the group leaves, the system recognizes which shoppers used the same card and generates one receipt for the bulk transaction.
“One day, a tour bus came in and there were 90 people all paying with one credit card,” Medioni added. “We can track group purchases even if people leave the store separately.”
Fusion is a shelf sensor and sensor that realizes accurate receipts.
Just Walk Out technology uses object recognition to accurately identify sandwiches, bags of potato chips, and other items. Medioni says the system is trained to recognize what it’s looking at, and “if you accidentally return an item to the wrong place, the system will adjust accordingly.” The technology also alerts store staff to return misplaced items to their correct locations.
For small, hard-to-see items like chewing gum or lipstick, weight sensors on the shelf detect when a customer picks up something.
“Very small items present an interesting challenge for computer vision systems because there is a limit to how far the primary sensor, the camera, can see,” Jenkins says. “Adding sensors allows us to do some really cool things in terms of solving problems that are very difficult to solve with cameras alone.”
By fusing input from cameras and weight sensors known as “load cell sensors” on the shelves, the system helps shoppers understand which items they’ve picked up.
“The key with Just Walk Out technology is that we always inspect each store individually to make sure we have the right set of technology in place,” Jenkins says.
Build robust systems with generative AI and synthetic data
Before generative AI became a buzzword, Amazon’s research team was helping develop Just Walk Out technology by using generative AI to create photo-realistic sets of synthetic data. This is information about situations that are difficult to find in real life and difficult to teach to a computer.
“It’s a lot like self-driving cars, where the basic cases are easy, but there’s a long tail of complex cases,” Medioni said. “If the system has never experienced an event like this, it might not work well. We wanted lots of examples, so we generated video clips of synthetic shoppers performing similar activities, and We trained the system to make the right decisions.”
Medioni and his team used a type of generative AI called generative adversarial networks (GANs) to create synthetic data to train Amazon’s Just Walk Out technology.
“With GANs, you have complete control over every movement a virtual shopper makes,” he says.
The team uses a dataset from millions of synthetic AI-generated images and video clips to model realistic and sometimes unusual shopping scenarios, such as changes in store format, lighting conditions, and even shopper crowds. I imitated it. This training allowed us to expand the original idea of the Just Walk Out technology into a system that can recognize millions of actions without fail.
“When a customer exits, it’s important to know exactly what they purchased,” says Medioni.
Just Walk Out technology, based on computer vision, is already available worldwide, with new locations opening every month. But Jenkins says Amazon has “just scratched the surface” in terms of where his Just Walk Out technology will work.
“We started with some of the most obvious use cases that people are in a hurry, but we believe that in the long term it can be applied almost anywhere, and there are cases outside of retail as well.” he says. “The underlying technology is inherently transferable.”
Click here for details Just walk out technology and Stores using Just Walk Out technology It’s near you.
