Regular Programming

About things you built long ago that start doing weird things

Andreas tells the story of a old system which suddenly exhibited a new and frightening bug. Lars shares similar experiences of things going wrong in new and novel ways.When things do go wrong, it is so nice to have supervision trees or other things which allow you to hear about problems, not to mention recover from them.Also covered are some stories about TCP, networks, and timeouts. And a realization that testing the frameworks upon which you build could have saved some bacon, had it just been done a long time ago.LinksDjangoModel-view-controllerDrupalUnicode collationSupervision treesOxide and friends - episode 27TCP_NODELAYQUIC and HTTP/3UDPNyqvist-Shannon sampling theoremHexagonal designQuotesGaming convention management systemWhen I say view, I mean controllerView is a better wordIf I ignore it, it will go awayDestructive favouritesAlternative class hierarchiesFailed in new and novel waysBoth a mistake, and interestingAaah, circumflex!TCP the good parts

About Data Pipelines

Lars dove into data pipelines, and emerged bearing arrows and wishing for a lot fewer copies.What is there to think about regarding data pipelines, what is interesting about them?Which tools are out there, and why might you want to use them?Why all this talk about making fewer copies of data?What does Lars' current ideal pipeline look like, and where does Elixir fit in?LinksMatt TopolApache ArrowLarge language modelsVector searchBigQuerysedAWKjqReplacing Hadoop with bash - "Command-line Tools can be 235x Faster than your Hadoop Cluster"HadoopMapReduceUnix pipesDirected acyclic graphtee - to "materialize inbetween states"Apache BeamApache SparkApache FlinkApache PulsarAirbyte - shoves data between systems using connectorsCronjobFivetran - Airbyte competitorApache AirflowETL - Extract, transform, loadDesigning data-intensive applicationsStream processingEphemeralityData lakeData warehouseThe people's front of JudeaDBT - SQL-SQL batch-work-thingySQL with Jinja templatesSnowflake - data warehouse thingScalaBroadwayOban - "robust job processing for Elixir"Dashbitpandas - Python data libraryAPLArrow flightGRPCDataFusion - query execution enginePolars - "DataFrames in Rust"Explorer - built on top of PolarsVoltron dataThe Composable CodexPyarrow - Arrow bindings for PythonQuotesI've been reading a lot about data pipelinesWhat's so special about data pipelines?There's a lot of special toolingThere's a lot of bad, bad toolingLess than optimal toolingConverging on something biggerlkHe got me eventuallyAll of your steps in one bucketWhat tools do you associate with data?I inherited a data pipelineBashReduceIterate on the L and the TThe modern data stackAnd then you demand more workNo unnecessary copiesBarely a copyReconnecting with my Python roots

About Fun With GenServers

GenServers are fun! Andreas gives all the context. Things were learned, knowledge was aquired. You can do so much with GenServers, but make sure you have a good reason.If you don't watch out, this is where concurrency goes to die.Dynamic supervisors, and their children, are thoroughly considered.Also delved into is the mess other ecosystems make of doing things at the same time, waiting, and so on.The strange worlds of C and other unusual languages are considered.Finally, an interesting bug.LinksAlan TuringTuring machineGenServerCowboyPlugUmbrellaETS - Erlang Term StorageØredevThe actor modelVirding's first rule of programmingRegistryDynamicSupervisorThe Goth library - Google auth library for ElixirThe GIL - the global interpreter lockFriday afternoon deployPromisesEsbuildUiua - "A stack-based array programming language"Prefix treePackmatic library, by Evadne Wu - streaming zip archivesQuotesWhere the system grows horizontallyThe kind of thing that starts happening when you hire developersIt was missing a hatI have become nothing, the simplifier of thingsWhere all the concurrency goes to dieA whole dance party of sad, dark peopleThe children of the dynamic supervisorHomes can be nodesHundreds of interested partiesTurns life into promisesPoking some C programmers

About What Every Web App Needs But Your Developer Does Not Want You To Know About

Every web app starts out fine, the tabula rasa of an unwritten BODY. But sooner or later you need users. And a million other things which live in trees.Also: email.And that layer between the controller and the database where things like fine-grained access control goes.I'd like to have an admin, please.Eventually, web apps grows up. And while a larger framework with solutions and conventions for all those grown-up features may not necessarily be fun, it can certainly be useful.LinksAPM - Application Performance ManagementDjangoTeams should be an MVP feature!Bullet Train - a "Ruby on Rails SaaS framework"FlaskExpressSinatraScottyPhoenixAuth0OktaPostfixPostmarkDjango AnymailSwooshModel-view-templateACL:s - access-control listsEctoMultitenancyZack Daniel on Beam RadioZack's Elixirconf talkAsh frameworkPlugDSL - domain-specific languageBigquerygRPCHIPPAPostgrestFunction based viewsDjango RESTLaravelTitlesCheck in on your applicationDo you want details?The view is the controllerBecause namesI'd like to have an admin, pleaseThe admin is kind of roughAll the data is introspectableEndgame applicationNot another user management systemA very special can of worms

About Code Nerds

The software development industry is very much built for code nerds. It shouldn’t be.Many of us know many people who are really into coding. Not every working developer can, or even should, be though. Doesn't that create kind of a weird gap between professionals who live and breathe code both on and off work, and those who have a more balanced life?Being passionate about your job shouldn't be an expectation or requirement for anyone or anything.Is there too little space for learning - are we assumed to know too much, and assumed to spend our own time figuring out things we don't?Your path into coding is not, can not, and should not be the only path possible.LinksThe Python 2 to 3 transitionRobert A. Heinlein in 999 Words: What Every Human Should KnowGhost in the shellHarvest moon4x - Explore, expand, exploit, exterminateTDD - test-driven developmentBDD - behavior-driven developmentCharity Majors 2017 blog post about career paths for developers. (Bonus: 2019 follow-up about engineering managers)Late-stage capitalismQuotesI think that's perfectly healthySurrounded by themDelving into softwareSurrounded by nerdsMuch more reasonable answersWhere the nerd doesn't go so deepComputers are troublesomeWhy should you be passionate about your job?Squeeze the passion juiceToo passionate to defend themselvesExperience or scar tissue?Many developers have livesPopping out for the big pictureDoing good work takes all kinds