SnailTrail finds graph patterns and checks invariants on real-time execution traces of stream processors to monitor and analyze distributed dataflows while they're online.
3DF and its Clojure adapter clj-3DF provide a high-performance, reactive, declarative query layer on a distributed, data-parallel stream processor backed by differential computations.
LoView exposes major data leaks in popular dating applications. It can be used for precise user location trilateration, to generate behavioral heatmaps, and much more.
kondens explores tools for thought that augment human ability to systematically communicate and solve complex problems.
We rigorously generalize critical path analysis (CPA) to long-running and streaming computations and present SnailTrail, a system built on Timely Dataflow, which applies our analysis to a range of popular distributed dataflow engines. Our technique uses the novel metric of critical participation, computed on time-based snapshots of execution traces, that provides immediate insights into specific parts of the computation. This allows SnailTrail to work online in real-time, rather than requiring complete offline traces as with traditional CPA. It is thus applicable to scenarios like model training in machine learning, and sensor stream processing.
SnailTrail assumes only a highly general model of dataflow computation (which we define) and we show it can be applied to systems as diverse as Spark, Flink, TensorFlow, and Timely Dataflow itself. We further show with examples from all four of these systems that SnailTrail is fast and scalable, and that critical participation can deliver performance analysis and insights not available using prior techniques.
Eine zentrale Herausforderung von IoT-Anwendungen ist die Auswertung von hochdynamischen Data Streams in Real-Time. Vor dem Hintergrund klassischer Data-Pipelines stellen wir eine Dataflow-Architektur vor, mit der Data Streams korrekt, effizient und schnell verarbeitet werden können. Unsere Architektur erlaubt es, komplexe, aufeinander aufbauende high-level Queries über heterogene Datenquellen zu stellen, die mit dem Eintreffen neuer Daten inkrementell aktualisiert werden und den Anfragesteller reaktiv über neue Ergebnisse informieren. Ereignisse können dabei bis auf die Nanosekunde genau aggregiert werden.
The functional approach to state management in the frontend was first enabled by React, pioneered by the likes of om.next, Redux, and Elm, and has ushered in a golden era in web development. It is captured by the two signatures
view :: DB -> HTMLand
mutate :: DB -> Tx -> DB. What might happen when we start working with more than one world, i.e. when we replace the notion of mutate with that of
solve :: DB -> Tx -> [DB]?
Using Clojure and Rust, we explore these ideas in the context of 3DF, a stream processing system based on differential dataflows.
3DF is a stream processing system which feeds off of Datomic’s transaction log and provides clients with the ability to register arbitrary Datalog queries, for which they will then continuously receive any changes to the result set. It does this efficiently by compiling Datalog queries to differential dataflows.
Using 3DF on top of Datomic provides a powerful, reactive interface to Datomic, making it an even more attractive choice for the real-time web. It also opens up Datomic to non-JVM runtimes and processes without a peer cache, without sacrificing performance. Finally, it hints at the possibility of significantly speeding up functional UI frameworks like D3 and React, because it allows these systems to skip their own change detection.
This talk will explore the “why?“, “how?“, and “what now?“ of working with a reactive database.
Imagine turning the way that web applications interact with the database on its head: instead of polling for changes, clients register their information interests and then continuously receive updates as new data enters the system. Can we do this while maintaining the power and flexibility of Datomic?
This talk introduces 3DF, a stream processing system based on differential dataflows, which aims to do just that. Feeding off of Datomic’s transaction log, 3DF provides a reactive interface to your favourite database, making it an even more attractive choice for the next generation of web applications.