Sam Bean, Zain Hasan, and John Trengrove on You.com and Spark

Weaviate Podcast #32.

Thank you so much for watching the Weaviate podcast! We are super excited to host Sam Bean from You.com! As well as welcome Zain Hasan and John Trengrove to the Weaviate podcast for the first time! Sam begins by describing You.com, and then we dive into the Weaviate Spark Connector that Sam played a massive role in creating. I thought this was such a masterclass in the Spark big data technology; John, Sam, and Zain are all data engineering pros and I've never learned more about a new technology from a podcast than this one. I really hope you enjoy listening to it, please let us know any questions or ideas you have. Also, please see Zain's blog post on "The Details Behind the Sphere Dataset in Weaviate" - https://weaviate.io/blog/2022/12/deta.... This provides great detail on exactly how to use the Spark connector in Weaviate! In this case for a billion-scale dataset upload!!!

Chapters
0:00 Thanks for Watching!
0:18 Welcome Zain and John
0:28 Welcome Sam Bean, You.com!
1:48 Search Interface and Search Apps / Widgets
3:40 Searching through Specific Websites
4:00 Origin Story of You.com
6:53 How did you come across Weaviate?
8:33 Text, Image, Audio Search
10:28 What do you use Spark for?
14:20 Datasets used with Weaviate
16:14 Creating a Spark Connector to Weaviate
21:05 Adding Streaming support
22:50 Vectorizing Data at You.com
27:15 More on ONNX + Spark
29:52 Performance Questions, Spark + Weaviate
34:35 Parquet for HuggingFace Dataset Files
34:54 What is Parquet? Spark Pushdown Filters Explained
39:04 Similar to HDF5?
39:45 Spark for Extracting Ranking Features
43:25 Hybrid Search
46:48 Collecting Search Relevance Data
51:07 Thank you for watching! Thanks Sam, Zain, and John!