April 27, 2021
Data pipelines are everywhere: just about every dashboard you have created or consulted, any report you have published or read and any application that has used data to make a decision (e.g. set a price or serve a recommendation) will have at least one data pipeline that processes input data and delivers the output in the format required for the dashboard, report or application. Analysts have been building data pipelines for years (often in Excel), without talking about them very much. Recently, data pipelines have become a much larger topic, as companies have started to use data in many more places, to drive much more decision making and production applications. This has led to people thinking about “data pipelines” much more formally, as infrastructure that needs to be managed, monitored, maintained and evolved as the needs of a business evolve. In this talk, Yali Sassoon, cofounder and CPO at Snowplow, will walk through the key considerations data folks should think about as they start to formally build out their companies capabilities to design, deploy and manage an increasingly complicated ecosystem of production data pipelines, with a particular focus on those that process web and mobile data.