Entity event log in Datomic

TL;DR

Create an event log from Datomic transactions that's related to an entity. Look at transaction metadata, diffs and the entity from different angles.

Intro

I am pretty new to Datomic and writing a small Clojure app. In my domain models I have a model called event-log. This model is used to keep track of changes (mutations) in the system. It have the following attributes: author, datetime, entity, id, type. author is the user that made the change, entity is a reference to the entity that's changed, datetime is a timestamp of when the change occurred, id is a unique identifier of the event log entity and type is a categorization of the change. If a change was successful an event-log entity would be created. This implies that for each change, a new event-log entity will be created.

After watching this talk I started to wonder if the event-log model was really needed in this app (it have only one database). Features of Datomic maybe could replace this model and the app could benefit from less code and one less domain model. In the end it looks like I can. This document will describe my finds.

In this document I use the term transaction metadata. This is data that is passed along with transactions but doesn't belong to a system/business model. This data is queried from Datomic and will together with some Datomic transaction data replace my event-log model completely.

While learning about this I created this Emacs Org mode document. With help from org-babel, CIDER and Clojure CLI I have a document where is can do a style of literate programming. Video about literate devops in Emacs. I am on a learning path. If I have got something wrong please give me feedback via Twitter. Would love that!

Thanks to

Lucas Cavalcanti & Edward Wible - Exploring four hidden superpowers of Datomic Gave me idea of removing the "event-log" model from my domain models and embrace Datomic.
August Lilleaas post, Datomic: Querying for the history of an entity (v2) Gave me a lot of inspiration and some of my code is heavily inspired from his post.

Dependencies

Clojure Deps handles dependencies. The deps.edn looks like this:

  {:deps {org.clojure/clojure {:mvn/version "1.10.1"}
          com.datomic/datomic-free {:mvn/version "0.9.5697"}}}

In memory database.

Connect to a random, in memory database. If you would like to get a new fresh database, eval (connect!).

  (require '[datomic.api :as d])

  (defonce state (atom {:datomic-connection nil}))

  (defn establish-connection
    "Establish connection to random in memory database."
    []
    (let [uri (str "datomic:mem://" (gensym))]
      (d/create-database uri)
      (d/connect uri)))

  (defn conn
    "Get the database connection"
    []
    (:datomic-connection @state))

  (defn connect!
    "Start a connection to a new random in memory database."
    []
    (let [db-conn (establish-connection)]
      (swap! state assoc :datomic-connection db-conn)))

   (connect!)

Schema

Here is the schema. We will work with a person model. person-schema describes that model. The audit-schema describes transaction metadata that will be passed along with all transactions. This is the data that we are interested in when building the event log. :audit/user holds a reference to the user that made the transaction. :audit/type is the categorization of the transaction. It's also used as a unique value to make sure that we get correct data from our queries.

  (def person-schema
    [{:db/ident       :person/id
      :db/cardinality :db.cardinality/one
      :db/unique      :db.unique/identity
      :db/valueType   :db.type/uuid}

     {:db/ident       :person/first-name
      :db/cardinality :db.cardinality/one
      :db/valueType   :db.type/string}

     {:db/ident       :person/last-name
      :db/cardinality :db.cardinality/one
      :db/valueType   :db.type/string}

     {:db/ident       :person/birth-year
      :db/cardinality :db.cardinality/one
      :db/valueType   :db.type/long}])

  (def audit-schema
    [{:db/ident       :audit/user
      :db/cardinality :db.cardinality/one
      :db/valueType   :db.type/uuid}

     {:db/ident       :audit/type
      :db/cardinality :db.cardinality/one
      :db/valueType   :db.type/keyword}])

  ;; conj all schemas into a single vector
  (def schema-set
    (->> (conj
          person-schema
          audit-schema)
         (flatten)
         (set)
         (vec)))

  ;; transact schema
  @(d/transact (conn) schema-set)

Transactions

Create a person with the first name Anna and then make some changes to that entity. Anna will be our main target from now on. We also have a person called Erik. Erik is just here to make sure that we get the correct results. If we find data related to Erik, something is wrong.

Each transaction have a unique :audit/type. This is to make sure that we query the correct data later on.

  ;; uuid to the user that makes the transactions
  (def user-id #uuid "19c14367-a230-4a8f-9cc3-b6bb72a23dcb")

  (def person
    {:person/id #uuid "431a2745-07b7-4850-80d5-075c78b306d1"
     :person/first-name "Anna"
     :person/last-name  "Anderss"})

  @(d/transact
    (conn)
    [person
     {:audit/user user-id
      :audit/type :create}])

  @(d/transact
    (conn)
    [{:person/id #uuid "431a2745-07b7-4850-80d5-075c78b306d1" :person/last-name "Andersson"}
     {:audit/user user-id
      :audit/type :update1}])

  @(d/transact
    (conn)
    [{:person/id #uuid "431a2745-07b7-4850-80d5-075c78b306d1" :person/birth-year 1986}
     {:audit/user user-id
      :audit/type :update2}])

  @(d/transact
    (conn)
    [{:person/id #uuid "431a2745-07b7-4850-80d5-075c78b306d1" :person/birth-year 1987}
     {:audit/user user-id
      :audit/type :update3}])

  @(d/transact
    (conn)
    [#:person{:id #uuid "e193ad2b-3095-49ff-a6d8-47147db702fa"
              :first-name "Erik"
              :last-name "Eriksson"}
     {:audit/user user-id
      :audit/type :error}])

Quick look at Anna

Query and pull Anna to see that all the transactions where successful.

  (d/q '[:find (pull ?e [*]) .
         :in $ ?id
         :where [?e :person/id ?id]]
       (d/db (conn)) #uuid "431a2745-07b7-4850-80d5-075c78b306d1")

This is the result and Anna looks correct.

{:db/id 17592186045418,
 :person/first-name "Anna",
 :person/last-name "Andersson",
 :person/birth-year 1987,
 :person/id #uuid "431a2745-07b7-4850-80d5-075c78b306d1"}

Transaction metadata

Lets look at what metadata we can get from the transactions we did on Anna.

The history database gives access to all of the datoms in the database. Not just recent ones. This gives us the ability to ask for all of the transactions related to a entity.

It's not possible to pull from the history database. This example uses two databases to enable pulls.

  (->> (d/q '{:find  [(pull ?audit [:audit/user :audit/type])
                      (pull ?tx    [[:db/txInstant :as :audit/datetime]])]
              :in    [$history-db $ ?person-id]
              :where [[$history-db ?history-e :person/id  ?person-id]
                      [$history-db ?history-e ?attr       _          ?tx]
                      [$           ?audit     :audit/type _          ?tx]]}
            (d/history (d/db (conn)))
            (d/db (conn))
            #uuid "431a2745-07b7-4850-80d5-075c78b306d1") ;; Annas :person/id
       (map (fn [[a b]] (merge a b)))
       (sort-by :audit/datetime))

This is the results.

(#:audit{:user #uuid "19c14367-a230-4a8f-9cc3-b6bb72a23dcb",
         :type :create,
         :datetime #inst "2020-05-18T19:49:02.305-00:00"}
 #:audit{:user #uuid "19c14367-a230-4a8f-9cc3-b6bb72a23dcb",
         :type :update1,
         :datetime #inst "2020-05-18T19:49:08.926-00:00"}
 #:audit{:user #uuid "19c14367-a230-4a8f-9cc3-b6bb72a23dcb",
         :type :update2,
         :datetime #inst "2020-05-18T19:49:14.794-00:00"}
 #:audit{:user #uuid "19c14367-a230-4a8f-9cc3-b6bb72a23dcb",
         :type :update3,
         :datetime #inst "2020-05-18T19:49:26.063-00:00"})

We get a collection with all the changes made to Anna. For each change/transaction we get the author, timestamp and our categorization.

Metadata together with entity attribute

This is more or less the same as above. But here we retrieve the first name on the most recent version of Anna. In the next section we use this value to create an event log message.

  (d/q '{:find  [(pull ?audit [:audit/user :audit/type])
                 (pull ?tx    [[:db/txInstant :as :audit/datetime]])
                 ?person-name]
         :in    [$history-db $ ?person-id]
         :where [[$history-db ?history-e  :person/id         ?person-id]
                 [$history-db ?history-e  _                  _            ?tx]
                 [$           ?audit      :audit/type        _            ?tx]
                 [$           ?e          :person/id         ?person-id]
                 [$           ?e          :person/first-name ?person-name]]}
       (d/history (d/db (conn)))
       (d/db (conn))
       #uuid "431a2745-07b7-4850-80d5-075c78b306d1")

The result is truncated.

  [[#:audit{:user #uuid "19c14367-a230-4a8f-9cc3-b6bb72a23dcb",
            :type :update1}
    #:audit{:datetime #inst "2020-05-17T19:07:39.735-00:00"}
    "Anna"]
   ,,,]

Event log

Now lets assemble it and create a collection with event log strings and a timestamp.

  (defmulti event-log-msg :audit/type)
  (defmethod event-log-msg :create [{:audit/keys [user] :keys [entity]}]
    (str "User " user " created " entity))
  (defmethod event-log-msg :update1 [{:audit/keys [user] :keys [entity]}]
    (str "User " user " updated " entity))
  (defmethod event-log-msg :default [{:audit/keys [user] :keys [entity]}]
    (str "User " user " updated " entity))

  (->> (d/q '{:find  [(pull ?audit [:audit/user :audit/type])
                      (pull ?tx    [[:db/txInstant :as :audit/datetime]])
                      ?person-name]
              :in    [$history-db $ ?person-id]
              :where [[$history-db ?history-e  :person/id         ?person-id]
                      [$history-db ?history-e  _                  _            ?tx]
                      [$           ?audit      :audit/type        _            ?tx]
                      [$           ?e          :person/id         ?person-id]
                      [$           ?e          :person/first-name ?person-name]]}
            (d/history (d/db (conn)))
            (d/db (conn))
            #uuid "431a2745-07b7-4850-80d5-075c78b306d1")
       (map (fn [[audit tx person-name]]
              (-> (merge audit tx)
                  (assoc :entity person-name))))
       (map (juxt :audit/datetime event-log-msg))
       (sort-by first))

Looks pretty good and gives a good overview of the transactions that is related to Anna.

([#inst "2020-05-17T19:07:37.992-00:00"
  "User 19c14367-a230-4a8f-9cc3-b6bb72a23dcb created Anna"]
 [#inst "2020-05-17T19:07:39.735-00:00"
  "User 19c14367-a230-4a8f-9cc3-b6bb72a23dcb updated Anna"]
 [#inst "2020-05-17T19:07:43.242-00:00"
  "User 19c14367-a230-4a8f-9cc3-b6bb72a23dcb updated Anna"]
 [#inst "2020-05-17T19:07:48.210-00:00"
  "User 19c14367-a230-4a8f-9cc3-b6bb72a23dcb updated Anna"])

By providing author and categorization to each transaction it is possible to trace the changes made to an entity and create an event log. The id and entity reference that existed in the event-log model is not needed. The transaction data replaces that need. This makes it possible for me to remove my event-log model and some of the code that's associated to it.

Entity before and after each transaction

This is an elaboration on the event log and I am not sure I have use for in my current app. But August Lilleaas post shows how to look at the entity before and after each transaction. This is really interesting and could be really useful when needed.

d/tx->t returns the Datomic time the transaction was made. This is not the same as wall clock time. Instead it's and incremental number starting from 1000.

d/entity pulls an entity from a database and returns a dynamic/lazy map. To make it eager we use into on it.

d/as-of returns a database for a specific time.

  (let [db (d/db (conn))]
    (->> (d/q '{:find  [?e ?tx ?audit]
                :in    [$ ?person-id]
                :where [[?e     :person/id  ?person-id]
                        [?e     ?attr       ?val        ?tx ?added]
                        [?audit :audit/type _           ?tx]]}
              (d/history db)
              #uuid "431a2745-07b7-4850-80d5-075c78b306d1") ;; uuid of Anna
         (map (fn [[eid tx-eid audit-eid]]
                {:timestamp (:db/txInstant (d/entity db tx-eid))
                 :audit     (into {} (d/entity db audit-eid))
                 :before    (into {} (d/entity (d/as-of db (dec (d/tx->t tx-eid))) eid))
                 :after     (into {} (d/entity (d/as-of db tx-eid) eid))}))
         (sort-by :timestamp)))

The results.

  ({:timestamp #inst "2020-05-18T19:49:02.305-00:00",
    :audit
    #:audit{:user #uuid "19c14367-a230-4a8f-9cc3-b6bb72a23dcb",
            :type :create},
    :before {},
    :after
    #:person{:first-name "Anna",
             :last-name "Anderss",
             :id #uuid "431a2745-07b7-4850-80d5-075c78b306d1"}}
   {:timestamp #inst "2020-05-18T19:49:08.926-00:00",
    :audit
    #:audit{:user #uuid "19c14367-a230-4a8f-9cc3-b6bb72a23dcb",
            :type :update1},
    :before
    #:person{:first-name "Anna",
             :last-name "Anderss",
             :id #uuid "431a2745-07b7-4850-80d5-075c78b306d1"},
    :after
    #:person{:first-name "Anna",
             :last-name "Andersson",
             :id #uuid "431a2745-07b7-4850-80d5-075c78b306d1"}}
   {:timestamp #inst "2020-05-18T19:49:14.794-00:00",
    :audit
    #:audit{:user #uuid "19c14367-a230-4a8f-9cc3-b6bb72a23dcb",
            :type :update2},
    :before
    #:person{:first-name "Anna",
             :last-name "Andersson",
             :id #uuid "431a2745-07b7-4850-80d5-075c78b306d1"},
    :after
    #:person{:first-name "Anna",
             :last-name "Andersson",
             :birth-year 1986,
             :id #uuid "431a2745-07b7-4850-80d5-075c78b306d1"}}
   {:timestamp #inst "2020-05-18T19:49:26.063-00:00",
    :audit
    #:audit{:user #uuid "19c14367-a230-4a8f-9cc3-b6bb72a23dcb",
            :type :update3},
    :before
    #:person{:first-name "Anna",
             :last-name "Andersson",
             :birth-year 1986,
             :id #uuid "431a2745-07b7-4850-80d5-075c78b306d1"},
    :after             ;; This is the most recent version of Anna
    #:person{:first-name "Anna",
             :last-name "Andersson",
             :birth-year 1987,
             :id #uuid "431a2745-07b7-4850-80d5-075c78b306d1"}})

Changes in each transaction

Lets continue on the elaboration. This code is also heavily inspired of August Lilleaas post.

Query and pull each transaction that is related to Anna. Return the changes that have been made in each transaction together with metadata.

Also something that's not useful for my application but something that could be really handy when needed.

d/ident return the attribute of the entity. Keywords like: :person/first-name, :person/last-name and so on.

added returns true if something was added and false if something was retracted.

  (let [db (d/db (conn))]
    (->> (d/q '{:find  [?tx ?attr ?val ?added ?audit]
                :in    [$ ?person-id]
                :where [[?e     :person/id  ?person-id]
                        [?e     ?attr       ?val        ?tx ?added]
                        [?audit :audit/type _           ?tx]]}
              (d/history db)
              #uuid "431a2745-07b7-4850-80d5-075c78b306d1") ;; uuid of the person we audit
         (group-by first)
         (map (fn [[tx transactions]]
                {:timestamp (:db/txInstant (d/entity db tx))
                 :audit     (into {} (d/entity db (->> transactions (map last) first)))
                 :changes   (->> transactions
                                 (map (fn [[_ attr val added]]
                                        [(d/ident db attr) val added]))
                                 (sort-by last))}))
         (sort-by :timestamp)))