Transforming Product Discovery: Eclipse Engineer Hackathon Highlights

On September 9th, 2024, the Eclipse Group team came together for an exciting project during our annual Engineer Hackathon: building an innovative product recommendation tool designed to transform how users discover products online. The project aims to simplify and enhance the product search process by leveraging a dynamic, image-based interface, departing from traditional text-based search engines.

A new approach to product recommendations

The inspiration behind this tool is a Tinder-like swipe interface, where users swipe left or right to dislike or like products. Unlike conventional recommendation systems, this one dynamically adjusts in real time based on user interactions, tailoring the products displayed next to their preferences. The goal was to make the experience more intuitive and engaging, allowing users to quickly discover products that match their style and interests.By focusing on product images rather than titles or descriptions, the tool makes recommendations based on visual cues. This is particularly useful for subjective elements like style, where words often fall short of accurately capturing user preferences.

Tackling a complex problem: defining user preferences

One of the key challenges in eCommerce is that product preferences—especially visual and stylistic ones—are often hard for users to define explicitly. For example, someone’s idea of "minimalist" style may differ drastically from another person’s. Traditional filters struggle to capture this level of nuance.

‍

Our solution bypasses the need for rigid filters by adapting recommendations based on users' interactions with product images. As users swipe left or right, the system adjusts to their preferences in real time, learning from each choice to present more relevant products.

Technology behind the tool

To power this innovative system, we used OpenAI’s CLIP model to generate image embeddings—numerical representations of product images that make it possible to compare and recommend similar products. This allows the tool to analyze and suggest products based on what the user visually interacts with, not just based on product names or descriptions.

As our dataset grew from 60 to over 700 product images during the hackathon, we needed a way to efficiently manage these embeddings. We used Chroma, a powerful vector database, to store and retrieve image embeddings quickly, ensuring the system remained responsive and able to generate real-time recommendations as the number of products increased.

To compare products, we applied cosine similarity, a technique that measures how closely products match each other based on their embeddings. With each swipe, the tool refines its understanding of the user’s preferences, becoming increasingly personalized.

As our dataset grew from 60 to over 700 product images during the hackathon, we needed a way to efficiently manage these embeddings. We used Chroma, a powerful vector database, to store and retrieve image embeddings quickly, ensuring the system remained responsive and able to generate real-time recommendations as the number of products increased.

Image consistency

One of the major challenges we faced was ensuring that the product images used for recommendations were consistent and focused solely on the products. Since our algorithm analyzes the entire image, irrelevant elements like backgrounds or models could skew the similarity scores. For example, a shoe shown with a model’s leg could affect the algorithm’s judgment of similarity, as it would account for both the shoe and surrounding elements.To resolve this, we explored techniques for masking irrelevant parts of the image, a concept inspired by a Pinecone webinar. This approach focuses the algorithm on the product itself, filtering out distracting elements like backgrounds or models to ensure that the recommendations are more accurate.

Refining recommendations

Initially, the system’s recommendations were simple: the next product shown was the one most visually similar to the last liked product. However, this approach lacked depth in understanding the user’s evolving preferences over time.

To enhance the system, we explored methods to better refine recommendations. One option was to create a combined representation of the user’s taste by summing or averaging the embeddings of liked products while subtracting those of disliked products. This allowed the tool to offer more personalized results based on a broader understanding of user preferences.

Another approach was to organize the recommendations in a tree-like structure, where the system could offer similar products after each dislike before moving to something entirely different. This structure helps users find what they’re looking for more efficiently by fine-tuning suggestions as they interact with the tool.We also considered using a weighted average of embeddings, giving more recent likes greater influence as the user’s preferences became clearer over time.

Real-world application & next steps

This product recommendation tool has immense potential in the eCommerce space. It could be integrated into existing platforms to provide dynamic, personalized product suggestions, or even serve as a standalone tool that aggregates products from multiple online marketplaces.By focusing on image-based recommendations rather than text-based filters, this approach offers a more fluid and intuitive shopping experience. It’s especially useful for platforms with large product inventories, where users might otherwise miss relevant items or feel overwhelmed by long lists of filters.

While we made significant progress during the hackathon, challenges remain. Testing the tool with a small dataset limits our ability to fully evaluate its effectiveness, and certain product categories may not be well-represented within the image embedding space. However, we are excited about the possibilities and look forward to expanding and refining the system.Ultimately, we believe this image-based recommendation system opens up new possibilities for product discovery, offering a more personalized and enjoyable shopping experience. By relying on visual cues rather than text, we hope to revolutionize the way users find products online, making the process faster, easier, and more engaging.

Conclusion

The Eclipse Engineer Hackathon was an inspiring event that brought innovative ideas to life. The product recommendation tool we developed offers a fresh approach to online shopping by emphasizing image-based personalization. With ongoing improvements, this tool has the potential to enhance eCommerce experiences and transform how users explore products.

‍