Whispered Secrets: Building An Open-Source Tool To Live Transcribe & Summarize Conversations (emf2024)

Are you secretly a spy and/or passionate about open-source? Maybe you don't trust a cloud-hosted service with your highly classified information, or perhaps you like to build things for yourself. In this light-hearted talk, you will learn how to make a real-time on-device GenAI-powered application that can live transcribe and summarize conversations without internet access, using open-source components. Our journey begins with an introduction to open-source LLMs and the latest trends in running GenAI tools on your own hardware. We will build up our application step-by-step, first creating a live streaming voice-to-text transcription pipeline, then an LLM-based conversation summarization layer, presented within a Streamlit frontend, with conversation summaries sent to a lightweight Django API backend for storage. Here's why this matters: 1️⃣ Cloud-hosted SaaS tools cannot store highly sensitive information. 2️⃣ Good open-source alternatives exist for most GenAI tasks; the more people who use them, the more they will thrive. 3️⃣ Commercial tools will solve for common use cases, but developers can build personalised tools that are highly specialised for their own bespoke needs. This talk is tailored for Python enthusiasts and requires no ML expertise. By seeing a practical demo come together piece by piece, attendees will gain a deeper understanding of how to build their own complex Generative AI applications and be pushed to imagine what they could make for themselves using on-device computation in real-world scenarios. Expect plenty of Python code and some fun live demos, with GitHub code provided for attendees to try it at home. Licensed to the public under https://creativecommons.org/licenses/by-sa/4.0/ about this event: https://www.emfcamp.org/schedule/2024/183-whispered-secrets