Unlocking Business Potential with Acorn Agent​

Imagine a financial institution, such as a bank, which has extensive data on their users and their transactions. How can this data be used to drive additional business? Here are some examples that unlock the value of customer data through LLM Agents powered by Acorn Agent.

Step 1: Developing a Financial Assistant ChatBot​

A Financial Assistant ChatBot that can retrieve user transactions and credit card history when responding to user questions and extract valuable insights about user spending. Having access to the user data, the Financial Assistant can perform tasks like:

• Transaction Analysis: Which transactions accounted for most of my spending this month?
• Expense Breakdown: Show me a breakdown of my expenses per category for this month.
• Trend Identification: Highlight categories where my expenses increased from last month.

By displaying information in natural language, tables and visual charts, this chatbot enables users to retrieve the information they need. This improves customer satisfaction and reduces customer support costs. And as users interact with the chatbot for insights and analytics, they use their bank accounts more, fostering loyalty and increasing user engagement.

Step 2: Elevating to a Financial Advisor​

Taking it a step further, the next layer of value comes from creating a comprehensive Financial Advisor. While the Financial Assistant focuses on past behavior analysis, the Financial Advisor delivers forward-looking recommendations for future actions and potential savings. For instance, it can answer:

• Reward Tracking: How many rewards have I earned with this credit card so far?
• Product Review: Based on my transactions over the past six months, which credit cards would have given me the most rewards?
• Product Recommendation: What financial products or services would I benefit from and why?

By accessing user transactions and credit card data through the institution’s API, the Financial Advisor can calculate total rewards and suggest better alternatives. It presents this information in natural language, tables, or even charts, making it easy for users to understand and act upon.

How Acorn Agent works​

The Acorn Agent Framework makes it simple to implement such LLM applications by safely integrating with the bank's existing data APIs. Acorn Agent creates a layer of APIs on top of the bank's internal data, which can take the form of REST or GraphQL APIs. These APIs provide all the necessary data for an agent to respond to user queries. The API endpoints are described in an LLM tool definition, which is a JSON file that outlines the required data and its type. Fortunately, most APIs already come with reference documentation, making it possible to automate their transformation into a JSON Schema.

With the Acorn Agent Framework, an LLM Agent application is built. Developers can choose from a wide range of LLMs available through the platform, including OpenAI models and open-source models on AWS Bedrock. Once the model is selected, the API tool definitions are loaded into the application, followed by the creation of a user-friendly interface – the access point for end-users.

The user interacts with the Agent as they would with a personal assistant. When the Agent receives a user query, the LLM behind it determines which API tools to call to gather the necessary data. The Acorn Agent framework handles the LLM tool calls, retrieves the API responses, and returns them to the LLM. This enables the Agent to use the additional information to provide a more accurate response to the user's query. By facilitating this workflow, we can deliver real value through user-centric applications.

The above diagram shows the typical data flow for an Application built with Acorn Agent:

1. User inputs a question about their data (“Show me my most expensive transaction from last month”) into the Acorn Agent - enabled User Interface
2. The Acorn Agent - enabled Server Application sends this question to the LLM. The LLM decides that it can not answer this question based on its training data, so it formulates a Function Call (function transactions with arguments {fromDate: 2024-05-09T00:00:00, toDate: 2024-06-09T00:00:00})
3. The Acorn Agent - enabled Server Application recognizes this LLM function call and executes this Request against the API by injecting a secure context that authenticates the user. The important safety feature defends against injection attacks. The API Response is retrieved.
4. The Acorn Agent - enabled Server Application sends the initial user question and the API response to the LLM, which is then enabled to answer the initial question.
5. The LLM sends the Final Response to the Server, which, in turn surfaces it to the user on the User Interface

Acorn Agent creates a secure “sandbox” around the LLM and guarantees that the LLM can only access user-authorized data. It guarantees safe and secure access to data and gives the LLM access to the most up-to-date information in real-time, which reduces hallucinations and results in the most accurate responses for users.

Other potential Business Applications of Acorn Agent​

Acorn Agent can be the backbone of a multitude of applications across various industries. Let's delve into a few scenarios illustrating how businesses can harness this technology to unlock additional value.

Enhanced Customer Support​

Imagine a telecommunications company that leverages Acorn Agent to revolutionize their customer support operations. Instead of relying on traditional scripted responses, the company's customer service agents could interact with a powerful, AI-driven chat interface that accesses customer records, identifies issues, and suggests resolutions in real time.

For instance, a customer might inquire, "Why was my bill higher this month?" With Acorn Agent, the AI can quickly access the billing API, analyze the statement, and provide a detailed explanation, such as additional data usage or recent plan changes. This not only speeds up resolution processes, enhances customer satisfaction by providing precise and immediate responses and reduces the cost of customer support.

Streamlined Internal Operations​

In industries such as logistics and supply chain management, the ability to quickly access and update records can be a game-changer. Using Acorn Agent, employees can chat with their internal databases to track shipments, update inventory levels, and manage deliveries.

For example, an operations manager might ask, "What’s the current status of order #12345?" The AI, via the Acorn Agent interface, can query the order management system and provide immediate updates with relevant details, such as location, expected delivery time, and any issues encountered. This capability significantly reduces manual record-keeping time and allows staff to concentrate on high-value tasks that drive business growth.

Dynamic HR Management​

Human Resource departments can leverage Acorn Agent to streamline numerous processes, from recruitment to employee management. An HR manager might ask, "Who are the top-performing employees this quarter?" The AI would retrieve performance data from the company's HR management system and generate a detailed report with actionable insights.

When integrated with scheduling APIs, employees could interact with the AI to manage requests like leave, check holiday entitlements, or learn about upcoming training sessions. Queries such as "Do I have any remaining vacation days?" or "When is the next training session on workplace safety?" can be instantly resolved, simplifying HR processes, making HR managers more efficient and boosting employee engagement.

Retail and Customer Experience​

Retail businesses can utilize Acorn Agent to enhance customer interactions and improve the shopping experience. For instance, a customer might ask, "Do you have this product in stock in size M?" By connecting to the inventory management API, the AI provides real-time availability, suggests alternatives if the item is out of stock, and guides the user through the purchase process seamlessly.

Order tracking in retail also stands to benefit. Customers can inquire, "What’s the status of my order #987654?" and the AI can fetch real-time tracking information from the logistics API, offering immediate updates on order status, expected delivery times, and any potential delays, increasing customer satisfaction and engagement.

Driving Business Growth with Acorn Agent​

The Acorn Agent Framework’s powerful capability to transform data into actionable insights opens avenues for driving new business growth by:

• Enhancing user engagement and satisfaction with interactive and insightful tools.
• Providing data-driven recommendations for cross-selling and upselling products.
• Personalizing user experiences, ultimately leading to higher customer satisfaction and loyalty.

Conclusion​

In conclusion, the Acorn Agent Framework represents a transformative leap in how businesses can utilize their data. By integrating LLMs with APIs, companies can unlock unprecedented value, drive user engagement, and unveil new business opportunities. Explore the power of Acorn Agent and set your business on a path to innovation and growth.

Background​

Large-Language Models are neural networks that process input text and incrementally generate intelligent responses. Advances in the size, topology, and training of LLMs increased their performance as conversational interfaces to near-human levels. To date, LLMs have been confined to chat and search applications, where they are trained on extensive corpora and augmented with query-specific information through methods such as RAG, FLARE, or text search.

The recent addition of "tooling" as a primary feature has broadened the applicability of LLMs. "Tooling" is a set of function definitions provided to the LLM either in-prompt or through training. The LLM can invoke these functions to retrieve information on demand (e.g., looking up the current weather) or trigger an external action (e.g., placing an order). This enables LLMs to interact with external systems and information outside the immediate user context. The LLM can invoke APIs, run database queries, execute computations, trigger UI updates, and more.

Tooling makes it possible to build conversational interfaces with LLMs for user facing applications that enable the user to interact naturally with the application through text or voice. However, integrating LLMs with custom tooling poses several challenges:

• Safety: LLMs are non-deterministic and prone to hallucinations, requiring careful validation and comprehensive observability to build safe and predictable applications.
• Security: LLMs are susceptible to injection attacks, necessitating safeguards to build secure applications that do not leak user information or secrets.
• Performance: LLMs are expensive to invoke and highly sensitive to prompt variation, requiring efficient function management to reduce context tokens and improve performance.
• Efficiency: The implementations of tools in chatbots or agents have to be written from scratch by the developers, needing to create the function definitions for the LLM, but also the function execution against an API or a Database.

Acorn Agent Framework​

Acorn Agent is a framework for building LLM-powered applications through tooling. It provides the necessary infrastructure to instrument LLMs with custom tools, addressing challenges related to safety, security, performance, and efficiency:

• Safety: The framework validates function calls and supports auto-retry of invalid function invocations. It enables quick iteration of function definitions to improve accuracy and performance.
• Security: The framework sandboxes function calls through a defined "context," which includes sensitive function arguments (e.g., userid, sessionid, tokens) that are handled by the framework and not exposed to the LLM call stack.
• Performance: The framework gives developers full control over the context window construction to optimize cost and performance.
• Efficiency: The framework provides abstraction layers for managing tooling across many popular LLMs through a standard interface which reduces boilerplate code and custom instrumentation logic. while giving developers full control over context window constructionAt its core, Acorn Agent is a repository of tools that instruments and executes tools for LLMs using a JSON configuration format for semantic annotations, execution logic, and security context.

Additionally, Acorn Agent facilitates seamless integration of APIs, databases, libraries, and UI frontends as tooling for LLMs. Acorn Agent supports three types of tools:

• API: Invokes an external system through an API to retrieve information or trigger an action. The framework supports GraphQL, REST, and JDBC, giving the LLM access to internal microservices, external web services, databases, search engines, ERP systems, and more.
• Local: Invokes a local function to execute a library method or computation. This enables the LLM to execute mathematical or other computations where accuracy and determinism are important.
• Client: Forwards the tool call to the client or frontend to trigger a UI update, implemented as a function callback in JavaScript. This allows the LLM to customize the presentation of information to the user.

The Acorn Agent framework is open-source under the Apache 2.0 license. You can view the full source code, download it, and contribute to the project on Github.

Example LLM Application​

To illustrate how the Acorn Agent framework works, consider building an LLM application providing a natural language interface for an online garden store. This application uses the same GraphQL API that the web application uses to place and retrieve customer orders. Additionally, it implements functions for unit conversions and React components on the frontend to display order status. With Acorn Agent, we can register all of these as tools within the framework, configure an LLM (such as OpenAI’s GPT, Llama3 on AWS Bedrock, or Google Gemini), and set up our system prompt for a friendly shopping assistant. When a user issues a request, the Acorn Agent framework manages the tools for the LLM.

For example, suppose the user requests: "Order me fertilizer for 3 plots of lawn: 10x20 ft, 50x15 ft, and 30x35 ft. The same fertilizer I ordered last time." Acorn Agent injects the relevant tool definitions into the context and hands it to the LLM. The LLM processes the request and calls tools as follows:

• Look up the last orders for the fertilizer product ID and weight. Acorn Agent augments secure information from the user session, invokes the GraphQL API to retrieve the user’s last orders, and returns the information to the LLM. The LLM identifies which order was for fertilizer as well as the associated product id and weight.
• Convert the given measurements to total square footage and compute the number of bags needed for the fertilizer product based on the retrieved weight. Acorn Agent executes that tool by invoking a local function that implements the math and conversion. It then returns the number of bags to the LLM.
• Place the order for the computed number of fertilizer bags and retrieved product ID. Acorn Agent invokes the GraphQL API to place the order within the secure sandbox of the user session. The order details are returned to the LLM.
• Update the UI with the order information. Acorn Agent forwards that client function call with additional context to the UI to update the React component.

Acorn Agent handles all tool validations, sandboxing, invocations, and result propagations, allowing developers to focus on building tools and optimizing the context window.

Using Acorn Agent Framework​

To experiment with Acorn Agent as a natural language interface for an existing API or database, you need only write tools and a configuration file—no coding required. Check out the examples for public APIs to get started. To build an LLM microservice or web application, you can include the Acorn Agent framework as a dependency in your Java, Scala, or Kotlin project. See the documentation for all the details. As an example, refer to the Spring Boot application using Acorn Agent to implement a ChatBot. If you have any questions, feedback, or would like to contribute, please join us on Slack. Currently, the Acorn Agent framework is limited to the JVM. We aim to support other programming environments soon.

Conclusion​

Building natural language interfaces for user facing applications requires instrumenting LLMs with custom tooling. The open-source Acorn Agent framework provides the infrastructure to manage custom tooling for LLMs and ensures that application is safe, secure, and efficient.

It’s been an incredible ten months waking up and not having to be anywhere other than school dropoff.  However, all good things must turn into even better things!

I’m happy to announce that I’ve joined my old friends from DataStax, Matthias Broecheler, and Daniel Henneberger, as co-founder of DataSQRL.

And what a time to join! Our self-funded venture is already profitable, with a growing customer base.

What does DataSQRL do?​

Your “Operating System for Data” builds data products in minutes.  Gone are months wasted attempting to integrate multiple vendor technologies for a data pipeline.

DataSQRL abstracts and automates the building of data pipelines through our SQL interface, which we’ve affectionately dubbed SQRL.  Yes, pronounced “Squirrel,” just like the fuzzy little guys running around your yard.

Have you struggled with Apache Kafka?  Flink?  Streaming?  Data in motion? Integrating your Artificial Intelligence into your systems?  Anything “Real-Time?” Struggle no more, and get started here!

We provide an open-source tool that can help you along with some professional services.

Now that that is out of the way, onto more interesting things…

What did I do over these last ten months? You’re wondering…​

What didn’t I do?  I…

• Spent a ton of time watching my autistic son make incredible progress
• Helped produce a shark documentary, Unmasking Monsters Below, with my friend, Kevin Lapsley
• Traveled to more than 30 different places, both nationally and internationally (pictures below)
• Read 41 books
• Wrote 200 pages in my journal
• Started writing a new book
• Started writing a new screenplay
• Continued working on my photography skills
• Watched SO MUCH college football (We are!)
• Cooked my wife breakfast (almost) every day
• Made a ton of new recipes
• Worked out 4-6 days per week
• Mentored several young professionals to find their way
• Caught up with some old friends
• Served as an advisor to ten AI-based startups
• And attended a few concerts…

More on all of this as the months roll on…

What’s next for me and DataSQRL?​

If you’re attending Data Days in Austin, TX, next week, check out Matthias’ presentation.

Otherwise, make sure you follow us here on LinkedIn.

I promise that some of my upcoming posts will cover what I did over the last ten months.  You know me… However, here are some highlights in pictures…

A 3x Silicon Valley Unicorn veteran with IPO experience, award-winning screenwriter, and 2006 Time Magazine “Person of the Year,” Canz is often mistaken for Joe Rogan while walking down the street.  He can be found on LinkedIn, IMDB, and helping people how to pronounce “Conshohocken.”

#autism #journaling #imdb #joerogan #movies #metallica #concerts

Picture Highlights​

Innescron, County Sligo, Ireland

Mystic, CT

Innescron, County Sligo, Ireland

Often, we have the context to infer what kind of jacket a user is looking for based on their interactions on the site. For example, if a user has men’s running shoes in their shopping cart, they are likely looking for men’s sports jackets when they search for “jacket”.

At least to a human that seems pretty obvious. Yet, Amazon will return a somewhat random assortment of jackets in this scenario as shown in the screenshot below.

To humans the semantic association between “running shoes” and “sport jackets” is natural, but for machines making such associations has been a challenge. With recent advances in large-language models (LLMs) computers can now compute semantic similarities with high accuracy.

We are going to use LLMs to compute the semantic context of past user interactions via vector embeddings, aggregate them into a semantic profile, and then use the semantic profile to order search results by their semantic similarity to a user’s profile.

In other words, we are going to rank search results by their semantic similarity to the things a user has been browsing. That gives us the context we are missing when the user enters a search query.

In this article, you will learn how to build a personalized shopping search with semantic vector embeddings step-by-step. You can apply the techniques in this article to any kind of search where a user can browse and search a collection of items: event search, knowledge bases, content search, etc.

Step 1: Setup​

First, we need to find some realistic product data we can use for our personalized search. We’ll use a recent web scrape of the Flipkart product catalog from Kaggle. Flipkart is India’s largest e-commerce site. We removed incomplete and duplicate records from the dataset.

Second, we need a vector embedding model. A vector embedding model maps a piece of text to a high-dimensional vector that preserves the semantic meaning of the text in the sense that two pieces of text that are similar map to vectors that are close to each other in space.

The embedding model we are using for this tutorial is a quantized version of the all-MiniLM-L6-v2 pre-trained model. This is an off-the-shelf model for sentence embedding trained on a large corpus. The model is small and fast while delivering good performance. “Quantized” means that the model has been transformed to run efficiently on CPUs.

Download this zip archive for the data and vector model you need to follow the steps in this tutorial. It also contains all the code we’ll be writing.

Step 2: API & Architecture​

We are going to implement the personalized search as a microservice that exposes a GraphQL API. The API has one query endpoint for the personalized search and one mutation endpoint to record the products a user has browsed. The GraphQL schema for our API is shown below:

type Query {
 ProductSearch(query: String!, userid: String!): [Products!]
}


type Products {
 id: String!
 title: String!
 description: String!
 brand: String!
 average_rating: Float!
 sub_category: String!
}


type Mutation {
 ProductView(view: ProductViewInput!): ProductViewed
}


input ProductViewInput {
 productid: String!
 userid: String!
}


type ProductViewed {
 _source_time: String!
 userid: String!
}

We use PostgreSQL to implement the search and store the semantic user profiles. PostgreSQL is a relational database that supports efficient full-text search and vector data types.

We use Apache Kafka to capture users' product views so that we can process them asynchronously to compute the aggregated semantic user profiles. Kafka is a stream platform to efficiently store and process data streams.

We use Apache Flink to ingest the product data, compute vector embeddings, and compute the semantic user profiles. Flink is a stream processor that reliably processes data from streams with exactly-once guarantees.

We use these 3 data systems for a reliable, flexible, and scalable architecture. PostgreSQL manages data at rest, Kafka manages data in motion, and Flink processes the data. The architecture diagram shows how the data moves through those systems.

1. The ProductView mutation endpoint gets invoked when a user views a product on the website.
2. The event gets recorded in Kafka
3. Flink processes the ProductView events and aggregates them into semantic user profiles (more on that below).
4. Flink also ingests product data from the external product catalog, so we have up-to-date product information to search.
5. Flink writes the semantic user profiles and product information to PostgreSQL.
6. The ProductSearch query endpoint gets invoked when a user searches, which is translated to an SQL query against PostgreSQL, and the results are returned to the user.

We could have implemented the entire search microservice with just PostgreSQL and a server. That’s a simpler architecture but has several downsides:

1. We have to implement a lot of data processing and failure handling logic in the server to make it work reliably under load.
2. Computing vector embedding and aggregating them synchronously can lead to long latencies.
3. A spike in traffic to our shopping site would result in a high server load and lots of inserts to PostgreSQL which would degrade search performance.

Using an event-driven architecture as shown above allows us to process the product views and product updates asynchronously, which results in a more robust, reliable, and performant system.

Step 3: Implementation​

The big downside of our event-driven architecture is that it’s a lot of work to implement all of those components and make them work efficiently together. Luckily, there is a tool that makes this a lot easier: DataSQRL.

DataSQRL is an open-source compiler for event-driven microservices. With DataSQRL we only have to implement the processing logic of our search microservice and DataSQRL compiles the entire event-driven architecture for us.

Here is how that works. We implement our personalized search engine in a file called aisearch.sqrl using SQL. DataSQRL uses an SQL dialect that adds some syntax and features for stream processing. Let’s look at each of the four parts of the script in more detail.

-- ### PART 1: Imports
IMPORT externaldata.Products; -- Import product information from the catalog
IMPORT searchapi.ProductView; -- Ingest ProductView events from the API
-- Import some functions we are going to need for data processing
IMPORT vector.*;
IMPORT string.concat;
IMPORT text.textsearch;


-- ### PART 2: Process product data
-- Compute a semantic vector embedding for each product
Products.semantics := onnxEmbed(concat(title, '\n', description), '/build/embedding/model_quantized.onnx');
-- Convert the stream of product updates to a state table with the most recent product information
Products := DISTINCT Products ON id ORDER BY _ingest_time DESC;


-- ### PART 3: Compute semantic profiles from product views
-- Join in the semantic vector embedding for the product a user viewed
ProductViewVector := SELECT v.userid, p.semantics, v._source_time
FROM ProductView v TEMPORAL JOIN Products p ON v.productid = p.id;
-- Aggregate all those vector embeddings into a semantic profile for each user
UserProfile := SELECT userid, center(semantics) as semanticContext FROM ProductViewVector GROUP BY userid;


-- ### PART 4: Personalized Search
-- Retrieve the products that match the query and order them by semantic similarity to the user's profile
ProductSearch(@query: String, @userid: String) := SELECT p.* FROM
(SELECT * FROM Products WHERE textsearch(@query, title) > 0) p
LEFT JOIN (SELECT * FROM UserProfile WHERE userid = @userid) u
ORDER BY coalesce(cosineSimilarity(p.semantics, u.semanticContext),0.0) DESC;

Let’s go over the 4 parts of the implementation:

1. We import the product data from an external data system. For this tutorial, the product data is imported from a file (included in the archive), but DataSQRL supports multiple data sources. We also import ProductView events from the API that are created when a user invokes the corresponding mutation endpoint. Lastly, we import a few functions that we’ll use in our script.
2. We preprocess the Products data by computing a vector embedding for the title and description of each product using the LLM. Since we are importing a stream of product data, because products may change over time, we deduplicate the stream to get the most recent version for each product.
3. We join the ProductView events against the Products state table to look up the vector embedding for the product a user viewed. We use a temporal join so that we look up the associated vector at the time of the event. Then we aggregate all the ProductViewVectors by userid to create the semantic UserProfile as the centroid of all product vectors the user has visited.
4. To answer a search query, we retrieve the matching products and the vector that represents the semantic profile of a user. We order the results by their semantic similarity to the semantic user profile by computing the cosine similarity between the respective vectors.

Step 4: Running Our Personalized Search​

That’s all we have to implement. Let’s run our personalized search and see how it works in practice.

First, we compile the SQRL script to a microservice with the command:

docker run --rm -v $PWD:/build datasqrl/cmd compile aisearch.sqrl searchapi.graphqls --mnt $PWD

Make sure you run the command in the folder where you unzipped the archive containing the data.

Second, we run the compiled microservice with docker:

(cd build/deploy; docker compose up)

You can access the API in your browser through GraphiQL, which is an IDE for GraphQL APIs, by opening http://localhost:8888/graphiql/.

Copy-paste the following GraphQL query into the left-hand side and hit the run button to execute the search query for “jackets”. You’ll get a list of jackets that are offered on Flipkart.

{
    ProductSearch(query: "jacket", userid: "myuserid") {
    title
    description
    brand
  }
}

The search results are not personalized because we don’t have any context for the user “myuserid” yet. Let’s add some context by having our user visit a product page for running shoes. We do that by running the following mutation query with a payload:

mutation AddView($view: ProductViewInput!) {
  ProductView(view: $view) {
    _source_time
  }
}

Mutations require a payload that specifies which product a user visited. Enter the following JSON payload under "Query Variables" in the bottom left of GraphiQL before running the query:

{
  "view": 
  {
    "productid": "f40b93aa-c3b2-5cf4-97c0-92151279464f",
    "userid": "myuserid"
  }
}

Now, if you run the same search query for “jackets” again with the “myuserid” (click on the history tab to go back to the first query), you should see that the results are tailored to the semantic context of our user. Or try searching for "shorts" for the same user, and you'll get athletic and running shorts.

Step 5: Next Steps​

And that’s how you build a personalized search with vector embeddings. Our implementation is only a few lines of code, because DataSQRL generated all the data plumbing for our microservice.

We hope you can use this tutorial as a starting point to implement your own personalized search with LLMs. With DataSQRL, you can add additional sources of data, implement custom scoring functions, and fine-tune the search to your particular use case. For extra credit, you could change the aggregation function to prioritize recent product views or fine-tune the LLM for sentence embedding to our dataset. We’ll discuss those improvements in a future tutorial.

Check out the DataSQRL documentation for more information and star DataSQRL on Github to bookmark the open-source project. Good luck building your AI search and reach out to the community if you have any questions.

We will implement the whole microservice in 50 lines of code thanks to the DataSQRL compiler, which eliminates all the data plumbing so we can focus on building.

Watch the video to see the microservice in action or read below for step-by-step building instructions and details.

What We Will Build​

We are going to build a recommendation engine and semantic search that uses AI to provide personalized results for users based on user interactions.

Let’s break that down: Our input data is a stream of conference events, namely the talks with title, abstract, speakers, time, and so forth. We consume this data from an external data source.

In addition, our microservice has endpoints to capture which talks a user has liked and what interests a user has expressed. We use those user interactions to create a semantic user profile for personalized recommendations and personalized search results.

We create the semantic user profile through vector embeddings, an AI technique for mapping text to numbers in a way that preserves the content of the text for comparison. It’s a great tool for representing the meaning of text in a computable way. It's like mapping addresses (i.e. street, city, zip, country) onto geo-coordinates. It’s hard to compare two addresses, but easy to compute the distance between two geo-coordinates. Vector embeddings do the same thing for natural language text.

Those semantic profiles are then used to serve recommendations and personalized search results.

To summarize, our microservice will expose the following API (expressed in GraphQL schema):

type Mutation {
    Likes(liked: LikedInput!): LikeAdded
    AddInterest(interest: AddInterest!): InterestAdded
}

type Query {
    Events(limit: Int!, offset: Int = 0): [Events!]!
    EventsLiked(userid: String!): [EventsLiked!]!
    RecommendedEvents(userid: String!): [RecommendedEvents!]
    PersonalizedEventSearch(query: String!, userid: String!): [PersonalizedEventSearch!]
}

The API has two mutations (for REST folks, think of those as POST): one captures which events a user has liked, and another captures a user’s interests.

We have four query endpoints (those are like GET): two simple ones that return all the events and the events a user has liked and two AI-powered ones that return recommended events for a user and personalized search results for a user’s search query.

You can see the full GraphQL API with the mutation and return types here.

Architecture​

We will implement our conference recommendation service as an event-driven microservice for robust, real-time processing at scale. The architecture of the microservice is shown below and uses Kafka for event storage, Flink for stream processing, Postgres for querying, and GraphQL for the API.

The data travels counter-clockwise from the top:

1. When a user interaction is captured through the mutation endpoint,
2. the input data is written to Kafka as an event,
3. which gets picked up by Flink, processed, embedded as a vector, and aggregated into a semantic user profile,
4. which is stored in Postgres.
5. When a user requests data through the query endpoint, the data is fetched from Postgres.
6. In addition, the conference events are ingested by Flink from an external data source and similarly processed and stored in the database for querying.

Each component serves a distinct purpose in this event-driven architecture: the GraphQL server acts as the interface to the outside world, Kafka manages events in motion, Flink processes the event stream, and Postgres stores the processed data for retrieval on request.

Implementation​

Now, we could implement this event-driven microservice by implementing each of the components: implement the GraphQL server, set up the Kafka topics and event schemas, implement a Flink job for data processing, and design a database schema plus SQL queries. We would have to write a ton of data plumbing code: moving data between systems, mapping schemas, designing data models, and stitching it all together. There is a reason event-driven microservices are built by teams of specialists.

There is a better way: We are going to use the open-source DataSQRL compiler to generate all of that data plumbing code for us. That means we can implement our entire microservice in just 50 lines of SQL code as follows (click here to see the entire SQL script):

Imports​

IMPORT conference.Events;  --import external data
IMPORT recAPI.Likes;        --import data from API mutation
IMPORT recAPI.AddInterest;  --import data from API mutation

IMPORT string.*;
IMPORT text.*;
IMPORT vector.*;
IMPORT time.parseTimestamp;

At the beginning, we import the source tables that we are processing in this script. DataSQRL uses packages to represent data sources for modularity and ease of reuse. It’s like importing an external library but for data. Our API is treated as a data source which allows us to import the mutation input data as a table.

We are also importing functions for string processing, vector embedding, etc. DataSQRL uses the same packaging structure to organize functions.

Now, onto the actual logic of our script.

Processing Event Data​

Events.id := coalesce(CAST(regexExtract(url, '(\d*)$') AS INT),0);
Events.full_text := concat(title,'\n',abstract);
Events.startTime := concat(trim(regexExtract(date, '^[^-]*')),' ',trim(regexExtract(time, '\d\d?:\d\d\s(AM|PM)')));
Events.startTimestamp := parseTimestamp(concat(startTime,' PDT'), 'MMMM d, yyyy h:mm a z')
Events.embedding := onnxEmbed(full_text, '/build/embedding/model_quantized.onnx');

Events := DISTINCT Events ON id ORDER BY last_updated DESC;

First, we are adding additional columns to the Events table and then deduplicating the data stream so we have the most recent version of each event.

We are adding columns mostly to clean up our ingested events data. When you are dealing with external data, cleanup is often necessary. In this case, we need to do some work to extract the event timestamp and id.

We are also adding the embedding column to compute a vector embedding for the title and abstract of each talk. We are using the ONNX AI runtime to execute the embedding model. The embedding model we are using here is a quantized version of the all-MiniLM-L6-v2 pre-trained model. This is a model for sentence embedding trained on a large corpus that is small and fast while delivering good performance. “Quantized” means that the model has been transformed to run efficiently on CPUs for those of us who aren’t hoarding GPUs right now.

Processing User Interactions​

AddInterest.embedding := onnxEmbed(text, '/build/embedding/model_quantized.onnx');

LikeVector := SELECT l.userid, e.embedding, l._source_time
              FROM Likes l TEMPORAL JOIN Events e ON l.eventId = e.id WHERE l.liked;

UserInterestVectors := 
    (SELECT userid, embedding, _source_time FROM LikeVector)
        UNION ALL
    (SELECT userid, embedding, _source_time FROM AddInterest)

UserInterests := SELECT userid, center(embedding) as interestVector 
                 FROM UserInterestVectors GROUP BY userid;

Next, we are processing the user interactions. We are adding an embedding vector for the user interests captured in the AddInterst table. We are joining the user Likes with the deduplicated events table using a temporal join to get the embedding vector for the liked event. Both of those capture the semantic interests of a user.

Then, we combine those data streams in the UserInterstVectors table and aggregate them by computing the centroid for all those vectors for each user. That summary of user interest vectors gives us the semantic profile of each user.

User Analytics​

UserLikes := DISTINCT Likes ON userid, eventId ORDER BY _source_time DESC;

EventLikeCount := SELECT eventid, count(*) as num, avg(eventid) as test
                  FROM UserLikes l WHERE l.liked GROUP BY eventid;

Events.likeCount := JOIN EventLikeCount l ON @.id = l.eventid;

EventsLiked(@userid: String) := 
    SELECT e.* FROM UserLikes l JOIN Events e ON l.eventId = e.id
    WHERE l.userid = @userid
    ORDER BY e.startTimestamp ASC;

To show you that DataSQRL also supports good old-fashioned data analytics, we are adding some user likes analytics. We deduplicate the stream of Likes (in case a user liked and then unliked an event) and aggregate them by event into the EventLikeCount table.

We add a relationship between Events and EventLikeCount so that the like count can be accessed from the event through the API. DataSQRL adds relationships to SQL, so you can structure your data for API access and to simplify join expressions.

We add a table function that returns all the events a user has liked which maps onto the query endpoint in the GraphQL API of the same name.

Personalized Recommendation​

RecommendedEvents(@userid: String) :=
    SELECT e.*, cosineSimilarity(i.interestVector, e.embedding) as score
    FROM UserInterests i JOIN Events e
    WHERE i.userid = @userid ORDER BY score DESC;

To serve users personalized recommendations, we compute the similarity between the event embedding and the aggregated semantic user profile of the UserInterests table using cosine similarity between the vectors.

Personalized Search​

PersonalizedEventSearch(@query: String, @userid: String) :=
    SELECT e.*, coalesce(cosineSimilarity(i.interestVector, e.embedding),0.0) as score
    FROM Events e LEFT JOIN UserInterests i ON i.userid = @userid
    WHERE textsearch(@query, title, abstract) > 0
    ORDER BY score DESC;

For personalized search, we retrieve those events where the title or abstract matches the search query and then rank the results based on how similar the event is to the aggregated user interests.

Conclusion​

And that’s it. A complete event-driven microservice with vector embedding, personalized search, and user interaction analytics in 50 lines of SQL code.

And DataSQRL handles all the rest: mapping mutations onto Kafka topics and events, ingesting those events into Flink and mapping schemas, designing the physical data models in Kafka, Flink, and the database, mapping API calls onto database queries, optimizing index structures, and moving the data efficiently between all those components. That’s a whole lot of soul-sucking work we did not have to do.

If you want to learn more about DataSQRL, visit datasqrl.com, take a look at the quickstart tutorial, or join the community on Slack to ask questions and share your thoughts and feedback.

Run Microservice​

Want to run the recommendation microservice yourself? It’s easy. Follow these steps:

1. In your command line, create an empty folder and go into the folder:

mkdir current23; cd current23

2. Download the SQRL script, GraphQL schema, vector embedding model and event data source by clicking here, moving the zip file into folder you just created and unpacking it. You should see 3 directories (conference, conferencedata, and embedding) as well as a sqrl and graphqls file. The SQRL script and GraphQL schema are the ones we walked through above.

3. Compile the SQRL script and GraphQL schema into an event-driven microservice by running:

docker run --rm -v $PWD:/build datasqrl/cmd compile conference-recommendation.sqrl recAPI.graphqls --mnt $PWD

4. Stand up the entire microservice in docker by running:

(cd build/deploy; docker compose up)

Note, that the microservice does not contain DataSQRL. DataSQRL is only the compiler and generates the docker-compose script for orchestrating the microservice. The microservice itself only consists of Kafka, Flink, Postgres, and GraphQL server. If you want to take a look at the deployment artifacts that DataSQRL compiled for each component, take a peek inside the build/deploy folder.

Once the microservice is up and running (it takes a little while for all the components to boot up and initialize), the GraphQL API is accessible. You can access the API directly or open http://localhost:8888/graphiql/ to try out queries in your browser.

For example, run this query to get a list of events.

{
    Events(limit:20) {
        id
        title
        abstract
        time
        location
    }
}

Add a user interest by running the following mutation:

mutation AddInterest($interest: AddInterest!) {
  AddInterest(interest: $interest) {
  _source_time
}}

And add the following query payload under "Query Variables":

{
  "interest": {
    "userid": "1234",
    "text": "I want to learn more about Apache Flink and how to use it for real-time stream processing."
  }
}

Then look at the recommendations for the user 1234:

{
  RecommendedEvents(userid: "1234") {
    id
    title
    abstract
  }
}

You can like an event with this mutation:

mutation AddLike($liked: LikedInput!) {
  Likes(liked: $liked) {
  _source_time
}}

and this payload:

{
  "liked": {
    "userid": "1234",
    "eventId": 1136822,
    "liked": true
  }
}

And then see how that impacts the personalized search results with this query:

{
  PersonalizedEventSearch(query: "kafka", userid: "1234") {
    id
    title
    abstract
  }
}

As you can see, our search results are strongly biased in the direction of Apache Flink since that's what we liked a Flink talk and expressed an interest in Flink.

Enjoy playing with the API and finding the talks you want to attend at the conference.

To shut the microservice down, hit CTRL-C and then run (cd build/deploy; docker compose down -v) to remove the volumes.

In this article, we’ll delve into the tradeoff between preprocessing and querying, guiding you to make the right decision. We’ll also introduce tools to simplify this process. Plus, you’ll learn how building streaming applications is related to fine cuisine. It’ll be a fun journey through the land of stream processing and database querying. Let’s go!

Recap: Anatomy of a Streaming Application​

If you're in the process of building an event-driven microservice or streaming application, let's recap what that entails. An event-driven microservice consumes data from one or multiple data streams, processes the data, writes the results to a data store, and exposes the final data through an API for external users to access.

The figure below visualizes the high-level architecture of a streaming application and its components: data streams (e.g. Kafka), stream processor (e.g. Flink), database (e.g. Postgres), and API server (e.g. GraphQL server).

An actual event-driven microservice might have a more intricate architecture, but it will always include these four elements: a system for managing data streams, an engine for processing streaming data, a place to store the results, and a server to expose the service endpoint.

This means an event-driven architecture has two stages: the preprocess stage, which processes data as it streams in, and the query stage which processes user requests against the API. Each stage handles data, but they differ in what triggers the processing: incoming data triggers the preprocess stage, while user requests trigger the query stage. The preprocess stage handles data before the user needs it, and the query stage handles data when the user explicitly requests it.

Understanding these two stages is vital for the successful implementation of event-driven microservices. Unlike most web services with only a query stage or data pipelines with only a preprocess stage, event-driven microservices require a combination of both stages.

This leads to the question: Where should data transformations be processed? In the preprocessing stage or the query stage? And what’s the difference, anyways? That’s what we will be investigating in this article.

The Balancing Act: Preprocess vs. Query​

Picture yourself as the chef of a renowned Italian restaurant famous for its pasta dishes. You must decide what food to prepare during the day and what to cook when the order comes in. Cooking everything from scratch upon receiving an order would lead to long waiting times and require more kitchen staff. However, preparing all dishes ahead of time would result in overwork, significant food waste, and stale food. The solution lies somewhere in between. So you marinate the chicken and harvest the basil in the morning but cook the pasta on order.

Building streaming applications is a lot like being a chef. You must determine what data to preprocess for immediate availability and what data to compute fresh upon user request. If all your data processing happens in the database or server at request time, you'll face high request latencies and need to store a significant amount of data, increasing your costs. However, preprocessing everything might also be too expensive, result in unnecessary computations, and yield stale results.

Therefore, the optimal solution for data processing in streaming applications, like in a kitchen, often involves a balance between preprocessing and querying

Striking a Balance: Preprocess or Query?​

How do you find the right balance to determine whether a particular data transformation should be preprocessed or queried?

The answer lies in the following 4 requirements of your streaming application:

• Latency: What is the maximum request latency for the API endpoint?
• Cost: What’s the operational budget for your application?
• Freshness: How quickly should incoming data points be reflected in request responses?
• Consistency: What API results need to be consistent with each other?

Let’s look at each of these factors in detail to learn how they influence the decision on where the data should be processed.

Latency​

Quantifying the latency requirements of your microservice API is typically straightforward. For customer-facing microservices, it’s common to expect a p99.9 latency of a few hundred milliseconds, meaning 99.9% of all user requests complete within this time frame. If unsure, your product manager probably has an opinion on the target request latency 😉.

The maximum request latency sets a limit on the amount of data processing you can perform in the query stage. More data processing in the query stage results in longer request latency. This is largely influenced by the number of records you need to process and the complexity of the data processing required.

For instance, if a running deep-learning inference model at request time consumes 700 of your 1000 ms p99.9 latency, you must do most of the data processing (i.e., feature computation) in the preprocessing stage.

Likewise, pulling a million records from the database for aggregation is likely going to eat up your entire request latency budget. In particular, watch out for data transformations on data with highly skewed distributions. This occurs when the average number of records processed per request is small, but occasionally you have to process a lot of records which impacts your p99.9 latency.

The general rule of thumb is: The lower the request latency, the more preprocessing you want to do.

Cost​

The cost of running your microservice is another factor you have to keep within a certain budget. Let's explore how to optimize the cost of our streaming application, considering both the query and preprocessing stages.

Query Cost​

A big factor in our database cost is how much data we need to store. Naturally, that determines the storage cost for the database but it also increases the cost of all database operations. The more data a database must manage, the more computing resources it must utilize to retrieve data, maintain index structures, manage buffers, etc. Therefore, the simplest way to decrease our database expense is by reducing the volume of data we store in it. One effective method to achieve this is by aggregating data in the preprocessing stage.

For example, imagine we are building an IoT microservice that gathers temperature readings from a million sensors that operate at 10Hz (i.e. they take 10 readings every second). This amounts to a quarter gigabyte per second or over 20 TB per day. If we stored all this data in a database, we’d face a pretty hefty bill. However, if we only require min, max, avg, and median readings for each minute, we can aggregate the data during preprocessing and significantly reduce the volume of data written to the database (approximately 500 times less), thereby reducing our database expenses.

The other factor that drives our query stage cost is the amount of data processing. Many applications are read-heavy, meaning the same processed data is accessed multiple times in different requests. By preprocessing that data and storing the result in the database, we can reduce our cost because we only have to run the computation once. This approach is somewhat akin to caching but with the assurance that the results are instantly updated as new data is received by the preprocessing stage.

Preprocessing Cost​

Now, let’s look at the preprocessing stage. Similar to the query stage, the cost is driven primarily by the amount of computation and the volume of data that needs to be stored. Stream engines are highly efficient at executing time-bound operations, such as time-window aggregations and temporal joins, because those limit the amount of state - and therefore storage - that the stream engine has to maintain over time.

On the other hand, data transformations that aren’t bound in time can accumulate state indefinitely which is costly, hurts performance, and can crash the entire application due to memory exhaustion. A common culprit for high preprocessing cost is the inner join. Computing inner joins on changing streams requires that the stream engine hold the entire dataset for both sides of the stream in memory AND issue updates for all previously joined records whenever a record on either side of the join changes. That’s why joins are typically much cheaper to execute in the database at request time.

The general rule of thumb is: Reduce the cost of the query stage by preprocessing data but be mindful of costly streaming operations like joins.

Data Freshness​

Much like chicken cooked the previous day, some data preprocessing leads to stale results that aren’t acceptable to the user of your API.

For example, let’s go back to our IoT microservice that collects temperature data. The sensors are producing 10 readings every second that we are aggregating into 1 second time-windows. In other words, we are effectively downsampling our readings from 10 to 1 per second. For applications where users expect to see the most up-to-date temperature that may not be acceptable.

The impact of data freshness on your application is harder to quantify than latency or cost, because it depends a lot on your customer’s expectations and what they are doing with the data you produce. But like the kitchen, you get the freshest results when you prepare everything from scratch when the request comes in.

The general rule of thumb is: The fresher you want your aggregations to be, the more you have to compute at query time.

Consistency​

Data transformations often utilize a single record of incoming data in multiple ways. Take for example a banking application that presents a list of transactions, categorized by type, and also offers a monthly spending analysis, broken down by type. If a user modifies the type of a transaction, the spending analysis should automatically update. This is what we mean by data consistency: any changes in the input data should be reflected in all the results computed from that data.

When we preprocess data, such changes are often not immediately reflected or we cannot guarantee that they are reflected in all preprocessed results at the same time. That can lead to inconsistencies in the data returned by the API. A user may update the transaction type but the spending analysis won’t update until a moment later when the preprocessing stage catches up and adjusts the aggregates.

Like data freshness, the impact of data inconsistency on your application can be harder to quantify. It typically needs to be evaluated in the context of the entire application, not just the event-driven microservice. For instance, if the front-end application that allows the user to modify the transaction type automatically updates the affected aggregates, the temporary inconsistency of data in the API response may not be problematic.

Maintaining data consistency is easier when data computations are executed as queries against the database because we compute the results from the original records.

The general rule of thumb is: The more consistent you want computed results to be across changes in input data, the more computation has to be done in the query stage.

Practical Advice for Balancing Preprocessing and Querying​

Choosing between preprocessing and querying isn't a straightforward decision. Rather, it's like a tug-of-war between your application's various requirements that will dictate whether a particular data transformation should be preprocessed or executed at query time. Low latency and cost-effectiveness tend to favor preprocessing, while data freshness and consistency often lean towards querying.

What makes this decision so challenging is that preprocessing data and querying data are implemented very differently. Preprocessing data with tools like Kafka and Flink requires a different approach, language, and implementation compared to executing the exact same data transformation with Postgres and an API server at query time. It's common to have different team members handling preprocessing and query stage implementations. Therefore, moving a single transformation from the query to the preprocess stage (or vice versa) is expensive because it requires reimplementing the transformation, coordination within the team, and refactoring the interface between the components of the microservice.

Because it is so expensive to change what stage we execute our data transformations in, we try to get it right during the planning phase. But we often won’t know what the right balance is until we try it. And, application requirements change over time, of course. That makes event-driven microservice implementations time-consuming and costly.

But what if we could implement all of our data transformations in one script and have a compiler translate them to preprocess stage or query stage implementations based on application requirements? That would eliminate the problem because we can easily change where data transformations get executed. We could try out different allocations and iterate quickly as requirements change.

Meet DataSQRL: The Compiler for Event-Driven Microservices​

That’s exactly what DataSQRL does. With DataSQRL, you can write your streaming application's data transformations in SQL. The compiler then translates your SQL into preprocess and query implementations. Specifically, DataSQRL translates SQL into a combination of FlinkSQL and the datastream API for preprocessing, and PostgreSQL statements for querying.

Let's see how this works using our IoT microservice example:

IMPORT datasqrl.example.sensors.SensorReading; -- Import sensor data
IMPORT time.endOfSecond;  -- Import time function
/* Aggregate sensor readings to second */
SecReading := SELECT sensorid, endOfSecond(time) as timeSec,
                     avg(temperature) as temp FROM SensorReading
              GROUP BY sensorid, timeSec;
/* Get max temperature in last minute */
SensorMaxTemp := SELECT sensorid, max(temp) as maxTemp
                 FROM SecReading
                 WHERE timeSec >= now() - INTERVAL 1 MINUTE
                 GROUP BY sensorid;

First, we import the sensor reading data stream from Kafka. We define a new table SecReading that aggregates the readings per second. Finally, we create SensorMaxTemp, a table that calculates the maximum temperature for each sensor over the last minute.

DataSQRL's intelligent optimizer automatically determines where to execute the data transformations defined in your SQL script, based on your microservice's API. For our example, we expose the following GraphQL API that allows users to query the SecReading and SensorMaxTemp tables by sensorid:

type Query {
    SecReading(sensorid: Int!): [SecReading!]
    SensorMaxTemp(sensorid: Int): [SensorMaxTemp!]
}

type SecReading {
    sensorid: Int!
    timeSec: String!
    temp: Float!
}

type SensorMaxTemp {
    sensorid: Int!
    maxTemp: Float!
}

Based on the API definition, DataSQRL's optimizer finds the best allocation of data transformations to stages by minimizing the request latency and cost. In our example, this means that the SecReading table is preprocessed as a tumbling time-window, and SensorMaxTemp is preprocessed as a sliding time-window. Both tables are aggregated in Flink during preprocessing, with the results written to Postgres for querying on request.

But what if data freshness of the SensorMaxTemp table is more important for our application than low latency? No problem, simply tell the optimizer to execute this data transformation in the database:

/*+ EXEC(database) */
SensorMaxTemp := SELECT sensorid, max(temp) as maxTemp
                 FROM SecReading
                 WHERE timeSec >= now() - INTERVAL 1 MINUTE
                 GROUP BY sensorid;

With a simple hint above the table definition, you can direct the optimizer where to process the table, and it compiles the microservice accordingly. What would have required a significant refactor of your entire microservice is now a simple code change.

DataSQRL also handles all the data plumbing needed to integrate your event-driven microservice components. There's no need to connect Flink to Kafka, define the Flink table schema, define the Postgres table schema, or map API requests to prepared SQL statements. The DataSQRL compiler does it all for you. You get the same microservice architecture and technologies but with significantly less work.

Now, that takes the edge off making the right decision on where to compute your data transformations and turns the balancing act into a straightforward process: You can let the DataSQRL optimizer handle it for you, experiment with different allocations, and iterate quickly as your application evolves. DataSQRL makes the implementation of your event-driven microservices and streaming applications faster, easier, and more cost-effective.

Take a look at the DataSQRL tutorial to learn more about DataSQRL and join us on Slack if you need any help. If you're looking to empower your team to build event-driven microservices like the pros, we're here to help you hit the ground running.

This article introduces the temporal join, compares it to the traditional inner join, explains when to use it, and why it is a secret superpower.

Table of Contents:

• The Join: A Quick Review
• The Temporal Join: Linking Stream and State
• Temporal Join vs Inner Join
• Why Temporal Joins are Fast and Efficient
• Temporal Joins Made Easy to Use
• Summary

The Join: A Quick Review​

Let's take a quick detour down memory lane and revisit the good ol' join operation. That trusty sidekick in your SQL utility belt helps you link data from two or more tables based on a related column between them.

Suppose we are operating a factory with a number of machines that roast and package coffee. We place sensors on each machine to monitor the temperature and detect overheating.

We keep track of the sensors and machines in two database tables.

The Sensor table contains the serial number and machine id that the sensor is placed on.

The Machine table contains the name of each machine.

To identify all the sensors on the machine “Iron Roaster” we use the following SQL query which joins the Sensor and Machine tables:

SELECT s.id, s.serialNo FROM Sensor s 
    JOIN Machine m ON s.machineid = m.id 
    WHERE m.name = “Iron Roaster”

Why are joins as important as your morning coffee? Without it, your data tables are like islands, isolated and lonely. Joins bring them together, creating meaningful relationships between data, and enriching data records with context to see the bigger picture.

By default, databases execute joins as inner joins which means only matching records are included in the join.

So, now that we've refreshed our memory about the classic join, let's dive into the exciting world of temporal joins in stream processing systems like Apache Flink.

The Temporal Join: Linking Stream and State​

Picture this: you're a time traveler. You have the power to access any point in time, past or future, at your will. Now, imagine that your data could do the same. Enter the Temporal Join, the DeLorean of data operations, capable of taking your data on a time-traveling adventure.

A Temporal Join is like a regular join but with a twist. It allows you to join a stream of data (the time traveler) with a versioned table (the timeline) based on the time attribute of the data stream. This means that for each record in the stream, the join will find the most recent record in the versioned table that is less than or equal to the stream record's time.

The versioned table is a normal state table where we keep track of data changes over time. That is, we keep older versions of each record around to allow the stream to match the correct version in time. Like time travel, temporal joins can make your head spin a bit. Let’s look at an example to break it down.

Temporal Join vs Inner Join​

Back to our coffee roasting factory, we collect the temperature readings from each sensor in a data stream.

And we want to know the maximum temperature recorded for each machine.

Easy enough, let’s join the temperature data stream with the Sensors table and aggregate by machine id:

SELECT s.machineid, MAX(r.temperature) AS maxTemp 
FROM SensorReading r INNER JOIN Sensor s 
    ON r.sensorid = s.id GROUP BY s.machineid

But here is a problem: What if we moved a sensor from one machine to another during the day? With an inner join, all of the sensor’s readings would be linked to the machine it was last placed on. So, if sensor 1 records a high temperature of 105 degrees in the morning and we move the sensor to the “Iron Roaster” machine in the afternoon, then we might see the 105 degrees falsely show up as the maximum temperature for the Iron Roaster. See how time played a trick on our join?

And this happens whenever we join a data stream with a state table that changes over time, like our sensors that get moved around the factory. What to do? Let’s call the temporal join to our rescue:

SELECT s.machineid, MAX(r.temperature) AS maxTemp 
FROM SensorReading r TEMPORAL JOIN Sensor s 
    ON r.sensorid = s.id GROUP BY s.machineid

Pretty much the same query, just a different join type. Just a heads-up: the syntax for temporal joins in Flink SQL is more complex - we'll get to that later.

As a temporal join, we are joining each sensor reading with the version of the sensor record at the time of the data stream. In other words, the join not only matches the sensor reading with the sensor record based on the id but also based on the timestamp of the reading to ensure it matches the right version of the sensor record. Pretty neat, right?

Whenever you join a data stream with a state that changes over time, you want to use the temporal join to make sure your data is lined up correctly in time. Temporal joins are a powerful feature of stream processing engines that would be difficult to implement in a database.

Why Temporal Joins are Fast and Efficient​

Not only do temporal joins solve the time-alignment problem when joining data streams with changing state, modern stream processors like Apache Flink are also incredibly efficient at executing temporal joins. A powerful feature with great performance? Sounds too good to be true. Let’s peek behind the stream processing curtain to find out why.

In stream processing, joins are maintained as the underlying data changes over time. That requires the stream engine to hold all the data it needs to update join records when either side of the join changes. This makes inner joins pretty expensive on data streams.

Consider our max-temperature query with the inner join: When we join a temperature reading with the corresponding sensor record, and that record changes, the engine has to update the result join record. To do so, it has to store all the sensor readings to determine which join results are affected by a change in a sensor record. This can lead to a lot of updates and hence a lot of downstream computation. It can also cause system failure when there are a lot of temperature readings in our data stream because the stream engine has to store all of them.

Temporal joins, on the other hand, can be executed much more efficiently. The stream engine only needs to store the versions of the sensor table that are within the time bounds of the sensor reading data stream. And it only has to briefly store (if at all) the sensor reading records to ensure they are joined with the most up-to-date sensor records. Moreover, temporal joins don’t require sending out a massive amount of updated join records when sensors change placement since the join is fixed in time.

Temporal Joins Made Easy to Use​

Because temporal joins are so powerful, we made them easy to use in DataSQRL. DataSQRL is a compiler for Apache Flink that builds integrated data pipelines for your event-driven or streaming applications. DataSQRL supports the simplified temporal join syntax shown in the queries above. In addition, DataSQRL defaults to a temporal join whenever you join a state and a stream table on the state table’s primary key. In that way, DataSQRL helps you pick the right join for your data and makes it easy for developers new to stream processing.

Apache Flink supports temporal joins in Flink SQL using the following syntax:

SELECT s.machineid, MAX(r.temperature) AS maxTemp 
FROM SensorReading AS r  JOIN Sensor 
    FOR SYSTEM_TIME AS OF SensorReading.timestamp AS s 
    ON r.sensorid = s.id GROUP BY s.machineid

You need to be careful that the join column for the state table is the primary key of that table and that you set the timestamp for the SensorReading table. DataSQRL does that for you automatically based on watermark.

Time to Wrap Up This Temporal Journey​

We've reached the end of our time-traveling adventure through the universe of temporal joins. We've seen how they're like the DeLorean of data operations, zipping us back and forth through time to make sure our data matches up just right. We've also compared them to the good ol' inner join.

Temporal joins help us avoid the pitfalls of time-alignment problems when joining data streams with changing state. They're also super efficient, making them a great choice for high-volume, real-time data processing.

And that’s why the temporal join is stream processing's secret superpower.

DataSQRL makes using temporal joins a breeze. With its simplified syntax and smart defaults, it's like having a personal tour guide leading you through the sometimes bewildering landscape of stream processing. Take a look at our IoT tutorial to see a complete example of temporal joins in action or take a look at our other tutorials for a step-by-step guide to stream processing including temporal joins. And we are here to help if you have any questions.

Happy data time-traveling, folks!

In this blog post, we’ll discuss what GraphQL is and why it is a good fit for Flink applications. Like peanut butter and jelly, Flink and GraphQL don’t seem related but the combination is surprisingly good.

Table of Contents:

• How To Access Flink Results?
• What is GraphQL?
• Benefit #1: Flexible Access for Data APIs
• Benefit #2: Realtime Data Updates with GraphQL Subscriptions
• Benefit #3: Simplify Event Ingestion with GraphQL Mutations
• Downsides of using GraphQL in Flink Applications
• How to Build GraphQL APIs with Flink

How To Access Flink Results?​

Quick background before we dive into the details. Apache Flink is a scalable stream processor that can ingest data from multiple sources, integrate, transform, and analyze the data, and produce results in real time. Apache Flink is the brain of your data processing operations.

But Apache Flink cannot make the processed results accessible to users of your application. Flink has an API, but that API is only for administering and monitoring Flink jobs. It doesn’t give outside users access to the result data. In other words, Flink is a brain without a mouth to communicate results externally.

To make results accessible, you have to write them somewhere and expose them through an interface. But how? We have built a number of Flink applications and in most cases, the answer was: write the results to a database or Kafka and expose them through an API. Over the years, our default choice for the API has become GraphQL. Here’s why.

What is GraphQL?​

First, let's talk about GraphQL, the data query and manipulation language that's been shaking up the API world.

GraphQL is a robust alternative to REST for building APIs. It provides a complete and understandable description of your data in the API and gives clients the power to ask for exactly what they need. This means no more over-fetching of data or making multiple requests, which can be an issue with REST.

In contrast to REST, which uses a separate URL for each resource, GraphQL operates over a single endpoint using HTTP. This simplifies the process of making API calls, as you don't have to construct multiple URLs. Furthermore, GraphQL uses a type system to describe data, which can make your API calls more predictable and easier to debug.

Like REST, GraphQL is mature and well-established with lots of tooling and client libraries for pretty much all programming languages under the sun.

Okay, so GraphQL allows you to build APIs that are easy to use. But what makes it such a good match for Apache Flink applications? GraphQL has three key features that unlock the power of Flink’s data processing: flexible queries, subscriptions, and mutations. Let’s look at those in more detail.

1. Flexible Access for Data APIs​

Imagine you're at an all-you-can-eat buffet, but instead of food, it's data. You can pick and choose exactly what you want and how much of it you need. That's what GraphQL does for you when it comes to exposing data APIs. Developers can query the processed streaming data however they need it, without being constrained by predefined endpoints.

Other API standards like REST or gRPC are like ordering dishes from a menu. You can only access the endpoints that are defined in the API. That’s fine for many applications, but for data APIs, there are many possible combinations of data records users want to query. So, you either define a lot of endpoints or shift the burden onto the user to stitch their desired result sets together. In the end, it’s more work and a lot of food gets wasted.

But wait, there's more! GraphQL not only allows you to select the data you need, but it also simplifies the process of combining data from multiple sources. This means you can easily merge data from various microservices and databases, creating a seamless and unified experience for developers.

In a nutshell, GraphQL empowers developers by providing a flexible, powerful, and efficient way to access the data they need.

2. Realtime Data Updates with GraphQL Subscriptions​

Remember those walkie-talkies you used to play with as a kid? You'd press the button, and your voice would be magically transmitted to your friend's walkie-talkie in real-time. Well, GraphQL subscriptions are kind of like that, but for data.

Subscriptions allow Flink developers to push real-time data updates to consumers of the API, ensuring that everyone is always up-to-date with the latest information. It's like having a walkie-talkie channel dedicated to data updates, so you never miss a beat.

But that's not all! GraphQL subscriptions also enable developers to filter and control the data they receive. This means that each consumer can specify exactly what data they want to be updated on, reducing the amount of unnecessary information being transmitted.

In essence, GraphQL subscriptions provide a powerful and efficient way to keep everyone in the loop with real-time data updates. It's like having a personal news ticker, tailored specifically to your needs.

3. Simplify Event Ingestion with GraphQL Mutations​

Mutations make it simple to ingest events into your system, streamlining the process and keeping everything running smoothly. GraphQL mutations support complex event payloads that are defined as input types directly in the API specification, making it easy for users of your API to submit data.

GraphQL was developed with mobile applications in mind, where the connection between phone and server can be spotty. GraphQL mutations follow an event-centric model to separate the state of the mobile device from the state on the server and ensure continued operations when the connection between the two gets interrupted. That’s a perfect match for data pipelines and event-driven microservices which follow the same model for decoupling.

And unlike REST, you don’t have to worry about HTTP methods and state management which add a level of complexity you don’t need for event-driven applications.

In summary, GraphQL mutations structure data ingestion and updates as events - just like Flink.

Downsides of using GraphQL in Flink Applications​

Despite the many advantages of using GraphQL with Apache Flink, it's important to note that there are some potential downsides to this approach. Like any technology, GraphQL is not a silver bullet and there are certain challenges that developers may encounter when implementing it in their Flink applications.

Firstly, the very flexibility that makes GraphQL so appealing can also present challenges. The ability to request exactly the data you need means that your backend has to be prepared to handle a potentially wide variety of query structures. This can be difficult to implement, and it requires careful consideration of how your database is structured and indexed. We’ve seen a single missing index bring down an entire application because the database became overloaded. Not a fun day.

Secondly, while GraphQL's subscription model is a powerful tool for delivering real-time updates, it can also be difficult to implement with low latency. Especially in high-volume data environments like those typically handled by Flink. Ensuring that updates are delivered to clients as quickly as possible, without overwhelming the server or the network, can be a complex task that requires careful planning and optimization.

However, these challenges are not insurmountable. With the right tooling and a thoughtful approach to implementation, it's possible to take full advantage of the power and flexibility that GraphQL offers without optimizing the implementation by hand. DataSQRL is a tool that optimizes your Flink jobs as well as generates the database schema and index structures for your GraphQL API, compiling an entire data pipeline and saving you a ton of work. More on DataSQRL below.

How to Build GraphQL APIs with Flink​

To sum up, the combination of Apache Flink and GraphQL provides a potent solution for data-driven applications. Like peanut butter and jelly, these two distinct technologies complement each other and create a powerful synergy. Flink processes the data, and GraphQL exposes it in a flexible, efficient, and real-time manner. GraphQL’s ability to provide flexible access to data APIs, push real-time data updates through subscriptions, and simplify event ingestion with mutations, makes it an ideal interface for Flink's data processing capabilities. Together, they enable developers to build robust, efficient, and user-friendly data applications. So, whether you're building a real-time analytics platform, a complex event-driven application, or a data-intensive microservice, consider the Flink-GraphQL combo.

By now you are probably hungry to learn more. It may just be all the food analogies, but if you want to learn how to build GraphQL APIs with Flink then read on.

You can build Flink+GraphQL data pipelines by hand, but like many things in software, it gets a lot easier with the appropriate tooling. We recommend using DataSQRL for building GraphQL APIs on top of Flink. DataSQRL compiles the SQL that defines your data processing in Flink and your GraphQL API into a complete data pipeline that integrates these components efficiently.

Of course, we are biased. But, who knows, DataSQRL might save you a ton of work, time, and frustration.

Check out the Quickstart tutorial for a quick overview of how to consume and ingest data, process it with Flink, and expose the results through a GraphQL API. Looking for a step-by-step guide to building your own GraphQL APIs with Flink? Check out the quickstart tutorial for a detailed explanation of all the steps.

And if you run into any issues or have questions, don’t hesitate to reach out.

Happy coding, and may the match of GraphQL and Apache Flink turn your application into a delicious experience for your users!

DataSQRL allows you to focus on your application logic without getting bogged down in the details of how to execute your data transformations efficiently across multiple technologies.

The Challenge of Building Applications with Flink​

We have built several applications in Flink: recommendation engines, data mesh endpoints, monitoring dashboards, Customer 360 APIs, smart IoT apps, and more. Across those use cases, Flink proved to be versatile and powerful in its ability to instantly analyze and aggregate data from multiple sources. But we found it quite difficult and time-consuming to build applications with Flink.

To start, you need to learn Flink: the table and datastream API, watermarking, windowing, and all the other stream processing concepts. Flink alone gets our heads spinning. And Flink is just one component of the application.

To build a complete data pipeline, you need Kafka to hold your streaming data and a database like Postgres to query the processed data. On top, you need an API layer that captures input data and provides access to the processed data. Your team must learn, implement, and integrate multiple complex technologies. It takes a village to build a Flink app.

Introducing DataSQRL: A Solution for Streamlined Flink Development​

That’s why we built DataSQRL. DataSQRL compiles the SQL that defines your data processing into an integrated data pipeline that orchestrates Flink, Kafka, Postgres, and API - saving us a ton of time and headache in the process. Why not let the computer do all the hard work?

Let me show you how DataSQRL works by building an IoT monitoring service.

First, we implement the data processing for our monitoring service in SQL.

IMPORT datasqrl.example.sensors.SensorReading; -- Import sensor data
IMPORT time.endOfSecond;  -- Import time function
/* Aggregate sensor readings to second */
SecReading := SELECT sensorid, endOfSecond(time) as timeSec,
                     avg(temperature) as temp FROM SensorReading
              GROUP BY sensorid, timeSec;
/* Get max temperature in last minute */
SensorMaxTemp := SELECT sensorid, max(temp) as maxTemp
                 FROM SecReading
                 WHERE timeSec >= now() - INTERVAL 1 MINUTE
                 GROUP BY sensorid;

This script imports the metrics stream which are temperature readings collected by sensors. DataSQRL treats external data sources like software dependencies that you import, allowing the compiler to handle configuration management, data mapping, and schema evolution for us. We also import a time function we’ll use for aggregation.

Then we define two tables: The first table smoothes out the readings by taking the average temperature each second. The SensorMaxTemp table computes the maximum temperature for each sensor over the last minute.

Next, we are going to define the API for our monitoring service. That’s how users of our service query the data. The API is defined by a GraphQL schema. It specifies the query endpoints and result types.

type Query {
    SecReading(sensorid: Int!): [SecReading!]
    SensorMaxTemp(sensorid: Int): [SensorMaxTemp!]
}


type SecReading {
    sensorid: Int!
    timeSec: String!
    temp: Float!
}


type SensorMaxTemp {
    sensorid: Int!
    maxTemp: Float!
}

Note, how the tables map to query endpoints and types. That’s how it all fits together.

And don’t worry, you don’t have to write this schema - DataSQRL can generate it for you from the SQL script.

DataSQRL compiles the script and GraphQL schema into an integrated data pipeline with the following command:

docker run --rm -v $PWD:/build datasqrl/cmd compile metrics.sqrl metricsapi.graphqls

It also generates a docker-compose template to stand up the entire service.

(cd build/deploy; docker compose up)

We can now interact with the API and try it out by opening http://localhost:8888/graphiql/.

DataSQRL Does the Work for You​

Pretty simple, right? And the best part is that DataSQRL compiles deployment artifacts for each component that you can inspect and deploy anywhere. There is no magic or black box. For example, DataSQRL compiles a Flink jar you can execute on an existing Flink cluster or Flink managed service.

Most importantly, consider all the work we didn’t have to do. No data source configuration, watermark setting, Kafka integration, database schema definition, index structure selection, API implementation, and so on. DataSQRL compiles all that for you.

DataSQRL also supports inserts to ingest events, subscriptions to push data updates in real-time to the client, and exporting data to Kafka topics or downstream systems. Take a look at the Quickstart tutorial which shows you how to do that - it only takes a few minutes.

DataSQRL is an open-source project. If you like the idea of DataSQRL, have questions, or need help building your streaming application, don’t hesitate to reach out.

To sum up, DataSQRL is a tool for simplifying the development of Apache Flink applications by automating the integration of various technologies and allowing developers to focus on their application logic. It makes Flink accessible to lazy developers like us.

Have fun building applications with Flink!

SQRL is a high-level data programming language that compiles into executables for all your data systems, so you can implement your data logic in one place. SQRL adds support for data streams and relationships to SQL while maintaining its familiar syntax and semantics.

Why Do We Need SQRL?​

The data layer of a data-driven application comprises multiple components: There’s the good ol’ database for data storage and queries, a server for handling incoming data and translating API requests into database queries, a queue/log for asynchronous data processing, and a stream processor for pre-processing and writing new data to the database. Consequently, your data processing code becomes fragmented across various systems, technologies, and languages.

For example, consider a project I was once working on. We were building a data product integrating customer data from various silos into a data API for a mobile app. The objective was to provide customers with an integrated view of their service and billing history, support requests, profile information, etc. This is a typical "Customer 360°" challenge many large organizations face when customer data is spread across numerous operational systems.

The data layer of that service consisted of a bunch of scripts ingesting customer data from CDC (change-data-capture) streams, a SQL database for data persistence, and a Java-based API server.

The data logic of this application was trivial: some translation of billing codes and aggregation of service items, but mostly it was straight-forward mapping of data. Yet, it took the team multiple months to build a prototype because of all the code fragmentation and glue code we had to write to stitch the components together. Integration testing was a big pain. And a simple sprint ticket to add a customer field took a week to implement and test.

The worst part was all the miscommunication. See, each component of the data layer has a different mental model. For the data ingestion and pre-processing, the developers thought in terms of events and streams. For the database modeling and querying, the developers thought in terms of rows and tables. And for the API implementation, the developers thought in terms of objects and classes.

It felt like we were building the Tower of Babel. Everybody was speaking a different language and we couldn’t understand each other. But with a twist: We thought we understood each other until it was time to integrate the components and we discovered a mismatch in how we represented the data. That’s a type of punishment not even a jealous God will dish out.

To save ourselves from this tedious work and mental gymnastics, we built SQRL as a high-level data programming language for implementing the data logic of your application in one place.

Introducing SQRL​

SQRL enhances SQL. If you’ve used SQL before, we hope that you find it easy to pick up SQRL. And if not, there is always ChatGPT 😜.

Let’s take a look at a SQRL script implementing a Customer 360° API that integrates and aggregates customer and order data:

IMPORT datasqrl.seedshop.Orders;
IMPORT datasqrl.seedshop.Customers;
IMPORT time.*;

/* Clean orders data and compute subtotals */
Orders.items.discount0 := coalesce(discount,0.0);
Orders.items.total    := quantity * unit_price - discount0;
Orders.totals         := SELECT sum(total) as price,
                         sum(discount0) as saving FROM @.items;
/* Deduplicate customer CDC stream */
Customers := DISTINCT Customers ON id ORDER BY timestamp DESC;
/* Create relationship between Customers and Orders */
Customers.purchases := JOIN Orders ON Orders.customerid = @.id;
/* Aggregate customer spending by state */
Customers.spending := SELECT state, sum(t.price) AS spend,
                             sum(t.saving) AS saved
                      FROM @.purchases.totals t GROUP BY state;

This script imports customer data and order streams. It processes data in multiple steps, culminating in an aggregated spending analysis by state.

And that’s all you have to implement to get a functioning customer 360° API. DataSQRL compiles this script into executables for all your data systems and handles data mapping and schema synchronization between them.

SQRL Features Overview​

Simple Syntax​

The first thing you notice is the syntactic sugar that SQRL adds to SQL.

It allows you to define the data sources that you are importing into your script so that a package manager can handle data access configuration and schema management.

It uses the := assignment operator to define new tables and allows incremental column definitions.

The goal is to make SQRL feel a little more like a development language where you build your data logic as a sequence of small, incremental steps instead of writing one massive query.

Nested Data​

Nested data, like JSON documents, is ubiquitous in data-driven applications. It’s how we exchange data. It’s how we expose data in APIs.

SQRL provides native support for nested data by representing it as child tables, accessed through the familiar "." dot notation.

In the example, we sum up the price and saving for all items in an order:

Orders.totals := SELECT sum(total) as price, sum(discount0) as saving FROM @.items;

There are a couple of things happening here:

• We define a new nested table in Orders called totals that contains the aggregates
• The FROM clause @.items selects the items from each order. The special table handle @ refers to the parent table in the local context, i.e. Orders in this example.

Being able to write queries within a nested context makes it possible to process tree-structured data within SQL.

For example, when we define the totals column for each item in an order, we can refer to the other columns of items within the local context:

Orders.items.total := quantity * unit_price - discount0;

Nested data support simplifies data consumption from external sources and result data mapping to API calls, eliminating a significant amount of mapping and data transformation code.

Relationships​

SQRL adds relationships to SQL. You can define relationship columns on tables that relate to rows in other tables using the familiar JOIN syntax.

Customers.purchases := JOIN Orders ON Orders.customerid = @.id;

Making relationships explicit in SQL simplifies joins and adds structure to the data that is exposed in the API without separate mapping logic.

For example, the FROM clause of the spending analysis query uses the relationship expression @.purchases.totals to select from the nested totals table of the purchase orders for each customer. It eliminates a double-join and makes the query easier to read.

Support for relationships and nested data makes it convenient to handle inter-related data and bridges the gap between the relational data model and the tree or object-relationship structure we use in our APIs and applications.

Stream Processing​

SQRL introduces support for stream tables to ingest external data streams and react to data changes. Data streams are an important part of data-driven applications. It’s how we consume data from other systems or applications and communicate changes in data to subscribers.

Unlike normal SQL tables where records can change over time, a stream table has immutable records that are fixed in time. As we saw with the orders stream in our example, SQRL makes it easy to process stream data in steps.

SQRL has operators to convert between stream tables and state tables. Our customer 360° script uses the DISTINCT operator to convert a CDC stream into a state table. The STREAM operator creates a change stream from a state table, so you can react to changes in state.

In addition, SQRL overloads the JOIN operator to support time-consistent joins between state and stream tables. For example, consider the join between the Customers and Orders tables in the spending analysis query. We want to join the Orders stream with the state of the Customers table at the time of a particular order, so that we aggregate by the state that the customer lived in when the order was placed. If we had used an INNER JOIN, the state would update every time the customer moved and the query would aggregate all orders under the state the customer currently lives in.

Making stream tables a first-class citizen in SQL allows us to process stream data, react to changes in data, and bridge the mental model between the set semantics of the relational world and the event orientation of streams.

Take a look at the documentation for a more detailed rundown of all the features SQRL adds to SQL.

Help Us Design SQRL​

To take SQRL for a spin and learn how to build data-driven applications, we recommend you start with the Quickstart tutorial. If you have questions, we are happy to answer them on Slack.

SQRL is still young, and we would love to hear your feedback on the language to shape its future.

You need a queue like Kafka to hold your events, a stream processor like Flink to process data, a database like Postgres to store and query the result data, and an API layer to tie it all together.

And that’s just the price of admission. To get a functioning data layer, you need to make sure that all these components talk to each other and that data flows smoothly between them. Schema synchronization, data model tuning, index selection, query batching … all that fun stuff.

The point is, you need to do a ton of data plumbing if you want to build a data-driven application. All that data plumbing code is time-consuming to develop, hard to maintain, and expensive to operate.

We need to make building with data easier. That’s why we are sending out this call to action to uplevel our database game. Join us in figuring out how to simplify the data layer.

We have an idea to get us started: Meet DataSQRL.

Introducing DataSQRL​

DataSQRL is a build tool that compiles your application’s data layer from a high-level data development language, dubbed SQRL.

Our goal is to create a new abstraction layer above the low-level languages often used in data layers, allowing a compiler to handle the tedious tasks of data plumbing, infrastructure assembly, and configuration management.

Much like how you use high-level languages such as Javascript, Python, or Java instead of Assembly for software development, we believe a similar approach should be used for data.

SQRL is designed to be a developer-friendly version of SQL, maintaining familiar syntax while adding features necessary for building data-driven applications, like support for nested data and data streams.

Check out this simple SQRL script to build a recommendation engine from clickstream data.

IMPORT datasqrl.example.clickstream.Click;  -- Import data
/* Find next page visits within 10 minutes */
VisitAfter := SELECT b.url AS beforeURL, a.url AS afterURL,
                     a.timestamp AS timestamp
        FROM Click b JOIN Click a ON b.userid=a.userid AND
                b.timestamp <= a.timestamp AND
                b.timestamp >= a.timestamp - INTERVAL 10 MINUTE;
/* Recommend pages that are visited shortly after */
Recommendation := SELECT beforeURL AS url, afterURL AS rec,
                         count(1) AS frequency FROM VisitAfter
        GROUP BY url, rec ORDER BY url ASC, frequency DESC;

This little SQRL script imports clickstream data, identifies pairs of URLs visited within a 10-minute interval, and compiles these pairs into a set of recommendations, ordered by the frequency of co-visits.

DataSQRL then takes this script and compiles it into an integrated data layer, complete with all necessary data plumbing pre-installed. It configures access to the clickstream. It generates an executable for the stream processor that ingests, validates, joins, and aggregates the clickstream data. It creates the data model and writes the aggregated data to the database. It synchronizes timestamps and schemas between all the components. And it compiles a server executable that queries the database and exposes the computed recommendations through a GraphQL API.

The bottom line: These 9 lines of SQRL code can replace hundreds of lines of complex data plumbing code and save hours of infrastructure setup.

We believe that all this low-level data plumbing work should be done by a compiler since it is tedious, time-consuming, and error-prone. Let’s uplevel our data game, so we can focus on what we are trying to build with data and less on the how.

Join Us on this Journey​

We have the ambitious goal of designing a higher level of abstraction for data to enable millions of developers to build data-driven applications.

We just released the first version of DataSQRL, and we recognize that we are at the beginning of a long, long road. We need your help. If you are a data nerd, like building with data, or wish it was easier, please join us on this journey. DataSQRL is an open-source project, and all development activity is transparent.

Here are some ideas for how you can contribute:

• Share your thoughts: Do you have ideas on how we can improve the SQRL language or the DataSQRL compiler? Jump into our slack community and let us know!
• Test the waters: Do you like playing with new technologies? Try out DataSQRL and let us know if you find any bugs or missing features.
• Spread the word: Think DataSQRL has potential? Share this blog post and star DataSQRL on Github. Your support can help us reach more like-minded individuals.
• Code with us: Do you enjoy contributing to open-source projects? Dive into the code with us and pick up a ticket.

Let’s uplevel our database game. With your help, we can make building with data fun and productive.

More Information​

You probably have a ton of questions now. How do I import my own data? How do I customize the API? How do I deploy SQRL scripts to production? How do I import functions from my favorite programming language?

Those are all great questions. Check out datasqrl.com for answers, join the community to ask us, or wait for a future blog post where we dive into all of those topics.

This is our first “official” release of DataSQRL after many months of testing and bug-fixing.
Check out the release notes on GitHub for a rundown of all the features.

Our Vision​

Every time we wanted to build a new use case for our application and expose a data API, we found ourselves getting distracted. Distracted by all the orchestration, the technology choices, all the micro-decisions, and the 'plumbing' that goes into the modern data layer. So we up-leveled the abstraction and kept it simple. We designed DataSQRL to handle those nitty-gritty details, so you could stay focused on what truly mattered - building cool things.

Simplicity Through SQL​

We've kept DataSQRL true to SQL, so it feels familiar and easy to use. We enhanced and modernized the language while maintaining the simplicity of SQL queries. No more wrestling with subqueries, window functions, or repetitive joins - just straightforward SQL.

Flexible APIs​

One size doesn't fit all when it comes to APIs. We made DataSQRL non-opinionated, giving you the freedom to use your preferred GraphQL schema and customize your query patterns with SQRL scripts.

Our Road Ahead​

We're seeking your feedback to help shape the future of DataSQRL. Our current architecture supports a range of platforms, and we're working on making it more extensible and useful. Your input is invaluable as we continue to refine and expand DataSQRL's capabilities.

Conclusion​

The only danger now is that your boss might think he can start coding again. Join us as we explore the story behind DataSQRL, its impact on the world of data processing, and the exciting possibilities it holds for the future.

Every developer, whether you build applications or backend services, encounters two distinct types of data problems: transactional and reactive. The need to store and retrieve application state is a quintessential example of a transactional data problem. Conversely, when you're processing events or consuming data from external sources, you're confronted with a reactive data problem.

Knowing which problem you're up against is crucial to selecting the right tools from your developer's kit. It’s important to determine what type of data problem you are dealing with to choose the right tools and approaches for implementing a solution. After all, using a hammer for a screw job can leave you with more than a few cracks to mend.

In this post, we'll guide you on how to differentiate between transactional and reactive data problems and pick the right tools and strategies to solve each of them.

Table of Contents:

• The Pitfall of Misinterpreting Reactive Problems as Transactional
• What are Transactional Data Problems?
• Solving Transactional Data Problems
• What are Reactive Data Problems?
• Solving Reactive Data Problems
• Conclusion

The Pitfall of Misinterpreting Reactive Problems as Transactional​

Let's kick things off with an anecdote from my career. An episode where I mistakenly treated a reactive data problem as a transactional one, resulting in a full-blown application meltdown. Definitely not a shining moment of my career.

Flashback to 2011, I was the backend developer for a sprouting startup named "Arrived". Our vision was to connect people in the real world by using their phone’s location data. That was the time when smartphones started to support geo-fencing and folks thought Foursquare was going to become the next Facebook. Fun times.

We built an iPhone app that allowed users to establish geo-fences and automatically check-in, alerting their connections upon entry. For a brief overview of the app, check out this brutally hilarious review from our TechCrunch Disrupt final presentation. While it was soul-crushing at the time, it's quite a fun read in retrospect.

I implemented the backend of the application as a Java web service, complete with a REST API for user creation, user connection management, and alert dispatch. The API primarily dealt with storing and retrieving user states, connections, geo-fences, and more. These are typical transactional data problems: how to maintain your application state in a durable, reliable, and consistent way. To tackle this, I used MySQL as the database and an object-relational mapping library to translate my Java objects to database rows.

All was sailing smoothly until we decided to implement a social feature to enhance the onboarding experience and boost the app's "virality". This feature uploaded a user’s phone contacts to check if any of their contacts were already using Arrived, suggesting them as potential connections.

That looked like another transactional data problem to me. Or rather, I was oblivious to the existence of other types of data problems and defaulted to it being transactional.

Thus, I embarked on a path that would eventually lead to disaster. The "upload contacts" API call I set up did the following:

• stored all contact entries in the database,
• ran a for-loop to match any entry hashes already in the database,
• added a "potential connection" record to another table in case of a match.

To my credit, the feature worked as intended. I even had a passing test case. But once we launched the feature in production, our database froze.

As it turned out, some users had an expansive social circle with over a thousand contacts. Running a transaction that writes thousands of records and fires off as many read queries on your primary operational database, which also services all your API requests, is a recipe for disaster. Needless to say, the database was not a fan of this idea and promptly crashed.

But my mistake was not a coding error. The code worked fine. The mistake was failing to realize that the “contacts matching feature” was a reactive data problem, not a transactional one. We were ingesting data from an external source - a user’s contact list - and reacting to it by comparing matches against our existing user base.

In the upcoming sections, we'll delve deeper into the differences between transactional and reactive data problems and how to solve them. We'll also revisit my reactive data problem and explore how a more informed approach could have saved me from a full-blown, hair-on-fire database crisis.

We will discuss how I could have solved my reactive data problem better and avoided a hair-on-fire database resurrection after we explore transactional and reactive data problems in more detail and how to distinguish between them.

As for "Arrived", we learned that our most active users were over-vigilant parents monitoring their children, which was not our target audience. Consequently, we had to close shop in less than two years.

What are Transactional Data Problems?​

Transactional data problems arise when you need to store and retrieve data concurrently while maintaining consistency. Here, "concurrently" refers to the simultaneous reading and writing of data by multiple threads or users. The trick is to ensure that data remains consistent throughout this flurry of updates.

There are two forms of inconsistencies we need to avoid. The first relates to upholding application constraints. For instance, if a username is required to be unique, we cannot allow two user records with identical usernames. This would be inconsistent with our application's unique username constraint. We may have several such constraints, like "account balances can't be negative," or "each product id in the orders table must correspond to an existing row in the product table."

The second inconsistency type relates to multiple updates triggered by a single request. We want to dodge situations where only some updates are stored. It's an all-or-nothing game - we either want all updates to be stored or none at all. For example, a request to transfer $100 from account A to account B requires updating both account balances. If only account A's balance is updated while account B's update fails, we've got a magical disappearing act of money.

Ensuring data consistency while managing concurrent user updates can be quite a challenge. You might encounter scenarios where two users try to register with the same username simultaneously or two users attempt to withdraw from the same account, causing the balance to plummet below zero. Situations like these are why data storage and retrieval often turn into a "problem" for developers.

Transactional data problems typically surface when storing state for applications that multiple users can access concurrently, or when building CRUD APIs.

Solving Transactional Data Problems​

The panacea for transactional data problems? Databases. Developers harness the power of databases to efficiently handle the concurrency and consistency issues associated with transactional data problems.

However, databases aren't one-size-fits-all. They differ in the types of consistency and concurrency guarantees they offer. If you're using any of the popular relational databases (like Postgres, MySQL, SqlServer, Oracle, and Aurora), breathe easy. They're likely equipped with all the support you need. For other databases, it's worth checking what exactly they support to avoid surprises down the line.

Alongside the choice of database, you'll also want to equip yourself with a tool that simplifies interactions with the database from your programming language. Wrestling with databases directly can be cumbersome, requiring the use of drivers, query string writing, and data mapping. If you're working in an object-oriented programming language, an object-relational mapping layer (or ORM for short) can be your best friend, translating seamlessly between your application and the database. If not, seek out an SDK or database abstraction layer that's compatible with your chosen database.

What are Reactive Data Problems?​

You have a Reactive data problem when your data source is outside your application or service's control, and you're required to respond to the data quickly. Let’s break this down.

Unconstrained Data​

When the data originates from an external source or isn't subject to any application constraints, your application does not control the data source. External data sources could include other systems like logs, message queues, files, external databases, or applications. Here, the data pre-exists independently of your control. For instance, a user's contact list is an external data source.

Moreover, data could be uncontrolled even within your application, provided it's free of any constraints. This includes events that occur organically within your application, such as a user clicking a button or visiting a webpage. These events aren't within your direct control - they just happen.

This is a stark contrast to transactional data problems, where the key challenge lies in maintaining data consistency amidst concurrent updates.

The Need for Speed​

Another characteristic of reactive data problems is the necessity for swift data processing and result generation. This quick reaction is twofold: it must occur shortly after receiving the data, and it involves computational processing of that data.

Take the contacts matching feature in Arrived, for instance. The goal was to encourage users to establish connections during the signup process. Consequently, we had to compute the matches within a few seconds - before the user completed the signup and exited the app.

How swift does this reaction need to be to qualify as "quick"? It varies according to your use case. Customer-facing use cases typically demand reactions within seconds to minutes, tops. For use cases like fraud detection, system automation, or financial transactions, you may need to respond within milliseconds. If the results can wait for hours or even days, it wouldn't qualify as quick.

The "reaction" element primarily involves generating a response to incoming data, which could either serve back to the user or trigger an action. This could mean processing a user's shopping cart to suggest other products they might like, analyzing system metrics to detect potential overload, feeding user activity into a machine learning model for a personalized journey, or evaluating if a transaction request is fraudulent. In each instance, we take a piece of data, evaluate it within the application context, and produce a useful response.

In essence, reactive data problems call for quick, efficient responses to one or multiple data sources. The challenges arise from the need to carry out data-intensive computations rapidly, efficiently, and robustly.

Reactive data problems commonly crop up in use cases such as:

• Personalization or recommendation engines
• User experience features
• Metrics or time-series analysis
• Machine learning features
• Fraud detection
• Cybersecurity and intrusion detection

How do you solve Reactive Data Problems?​

Let's circle back to the reactive data problem of the contacts matching feature. My initial solution involved splitting the transaction into several parts, moving some computation to a background thread, fine-tuning the database schema, and writing a hefty amount of SQL. This strategy was time-consuming, fragile, and a nightmare to maintain. A colleague shrugged it off with “I’ve no idea what’s happening here, but I guess it works…”

The problem? When all you have is a hammer, everything looks like a nail. So, I tried hammering that screw into the wall. As expected, it was neither pretty nor productive.

To solve reactive data problems more effectively, we need to reimagine our data layer architecture. Let’s give our database some company by introducing additional components that make it easier to process data reactively:

• Queue: Introducing a persisted message queue (or log) into our architecture can ease asynchronous data processing. This robust, scalable tool allows you to write incoming data to the queue and process it when resources are available, significantly simplifying multi-step data processing.
• Stream Processor: This is a framework dedicated to managing consecutive data processing steps, from data ingestion (reading data off the queue or from external systems) to data transformation, to writing the results to the database. This framework handles all task scheduling and execution, allowing you to focus on the actual processing logic.
• Server: This component accepts incoming data, writes it to the queue, and serves the processed data from the database. Acting as the entry and exit point for the reactive data layer, the server brings everything together. You can integrate this functionality into an existing API server implementation or create a standalone server to isolate the reactive data use case into its own backend service.

To improve the contact matching feature, we finally adopted a reactive data architecture as illustrated in the diagram above. Here's how the data flowed:

1. The server received the submitted contacts and wrote the entire data blob to a persisted messaging queue.
2. Three asynchronous tasks ran in the processing framework:
   1. Splitter: This task read an entire contacts list from the queue, divided it into chunks of a maximum of 50 contacts, and wrote the resulting chunks back to the queue under a different topic.
   2. Storer: This task read contact chunks from the queue and wrote the contact entries as individual records to the database.
   3. Matcher: This task read contact chunks, matched the contact entries against the user table, and wrote the found matches to the database.
3. The database stored contact entries and contact matches in a separate logical database, isolated from the main operational database serving the CRUD API of the Arrived app.
4. The server responded to "recommended contacts" requests during the signup process by running a query against the database that combined all pre-computed matches for the user with matches from checking the user's phone number against previously stored contact entries.

This solution was not only efficient and robust but also easier to maintain. Above all, it allowed us to concentrate on enhancing the feature instead of hacking around the database.

So, here's the key takeaway: instead of grappling with reactive data problems using a database alone, build a reactive data layer. It will save you considerable time and frustration.

If building a custom data layer seems intimidating, consider checking out DataSQRL. It's a tool that constructs reactive data layers for you. We've been developing DataSQRL to assist developers in resolving reactive data problems quickly and efficiently. We would love to hear your feedback!

Conclusion​

Transactional data problems arise when your application requires concurrent storage and retrieval of consistent data. On the other hand, reactive data problems occur when your application needs to quickly respond to data from external sources or events. Recognizing the distinction between these two types of data issues is crucial to implementing the most effective solution.

For transactional data problems, a data layer comprising a database and an Object-Relational Mapping (ORM) tool is often the best solution. On the contrary, reactive data problems are more efficiently addressed with a data layer that includes a queue, stream processor, database, and server. Understanding these distinctions and applying the appropriate solutions can significantly improve the efficiency and robustness of your data layer.

We are excited to launch DataSQRL with the mission to help developers and organizations build with data.

Collectively, we have spent decades building or helping others build data products. We have seen many struggles, failures, and piles of money being thrown out the window and figured that there must be a better way. We started DataSQRL to find it.

We believe that the technologies used to build data products are too complex and that the engineering processes used to build them are broken. Here is how we plan to fix these issues.

We developed DataSQRL which compiles a developer-friendly version of SQL into a fully integrated and optimized data pipeline and API server. It takes care of all the laborious plumbing, data massaging, and stitching together of technologies that makes building data products so harrowing. Check out this short tutorial to see how it works - it only takes a few minutes to build an end-to-end data product.

In addition, we are refining a value-focused process for implementing data products that we have developed over the years while working with development teams and organizations. The basic idea is to apply the same software engineering principles that have proven to be successful to implementing data products. That means you don't need a dedicated team of specialists to implement data products and can keep your customers and stakeholders in the feedback loop. Click here to learn more about our process.

That's our starting point for enabling developers to build successful data products quickly and efficiently. We think we got some good ideas, but have been building data technologies long enough to realize that there is a fine line between innovation and wishful thinking.
We hope that you will join the DataSQRL community to share your experience, insights, and opinions to help set us straight.

If you are trying to enable your organization to turn data into valuable data products, consider working with us and get in touch.

We are excited to be on this journey and hope you will join us. Let's build with data together.