Subject: m

From: EuAndreh <[email-masked]>
Date: Thu, 3 Sep 2026 09:19:35 -0300

---
 src/comments.mbox | 3464 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 3464 insertions(+)
 create mode 100644 src/comments.mbox

(limited to 'src/comments.mbox')

diff --git a/src/comments.mbox b/src/comments.mbox
new file mode 100644
index 0000000..edf49e0
--- /dev/null
+++ b/src/comments.mbox
@@ -0,0 +1,3464 @@
+From [email-masked] Mon Aug 31 16:44:13 2020
+Received: from mail-qv1-f42.google.com (mail-qv1-f42.google.com [209.85.219.42])
+	by mail-b.sr.ht (Postfix) with ESMTPS id 1E84AFF0FB
+	for <~euandreh/[email-masked]>; Mon, 31 Aug 2020 16:44:13 +0000 (UTC)
+Received: by mail-qv1-f42.google.com with SMTP id m14so1468478qvt.1
+        for <~euandreh/[email-masked]>; Mon, 31 Aug 2020 09:44:13 -0700 (PDT)
+X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
+        d=1e100.net; s=20161025;
+        h=x-gm-message-state:mime-version:from:date:message-id:subject:to;
+        bh=1N/5MWE9d7/I+SxiuWn81KliArG9XSFF8duwpxyY0oA=;
+        b=GiBnVI+Pf7DSl9ycPdfwPRjfdrlqlQQdgGKKyGsiIXPB1mLh6sBlzhoEQASxa/9hJm
+         rfhT8wRfQyBwscG2FpECYK09N/0ks7a71Pi3vy9ePKvSHJl0FYhywyAWzZ9Z7Xfp0/SF
+         oohs7Vzovs/sA9D9m2oKPcyfrj6lHWqf5qHJdfatP4LfVHCdRg1uwj3CmtCJWzmIFzuM
+         aIkUXLrK5YyBbfVl2L6EvEU1L7EDqOaTrXJt/+hWOev813kGYdYWsxMbYmirBAmO8sal
+         rgaF9vR/Jp8C+bxV/LSONsVOi4CA8azpLd1SYYpmS5yM4H5ISjaIOZfrZycx/Zd5Rle8
+         tn6Q==
+X-Gm-Message-State: AOAM530nGnF5g0fpYzDDTNbp5T67sHNZCqeZcVlHkD7CLIqa78AzYkW0
+	44VZGvmhPwGnRAVlt7ZPbGJQAlmieuw=
+X-Google-Smtp-Source: ABdhPJwgZnnyjMFpfNBttjOyxb0b4VZgNwyJNN7HkBIHn457LHtZ8CSNy1RJMts49AmqIUIw2YT9Yg==
+X-Received: by 2002:a0c:b255:: with SMTP id k21mr1811354qve.134.1598892252561;
+        Mon, 31 Aug 2020 09:44:12 -0700 (PDT)
+Received: from mail-qt1-f176.google.com (mail-qt1-f176.google.com. [209.85.160.176])
+        by smtp.gmail.com with ESMTPSA id n33sm10800778qtd.43.2020.08.31.09.44.12
+        for <~euandreh/[email-masked]>
+        (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
+        Mon, 31 Aug 2020 09:44:12 -0700 (PDT)
+Received: by mail-qt1-f176.google.com with SMTP id z2so5166925qtv.12
+        for <~euandreh/[email-masked]>; Mon, 31 Aug 2020 09:44:12 -0700 (PDT)
+X-Received: by 2002:ac8:73d0:: with SMTP id v16mr2161160qtp.384.1598892251883;
+ Mon, 31 Aug 2020 09:44:11 -0700 (PDT)
+MIME-Version: 1.0
+From: =?UTF-8?Q?Rory_O=E2=80=99Kane?= <[email-masked]>
+Date: Mon, 31 Aug 2020 12:44:01 -0400
+X-Gmail-Original-Message-ID: <[email-masked]>
+Message-ID: <[email-masked]>
+Subject: Re: The database I wish I had
+To: ~euandreh/[email-masked]
+Content-Type: text/plain; charset="UTF-8"
+
+In this discussion of your post on Lobsters, there are some comments
+that may aid you in your quest:
+https://lobste.rs/s/m9vkg4/database_i_wish_i_had
+
+From [email-masked] Mon Aug 31 23:00:25 2020
+Received: from box.euandre.org (box.euandre.org [46.101.160.115])
+	by mail-b.sr.ht (Postfix) with ESMTPS id 95690FF10A
+	for <~euandreh/[email-masked]>; Mon, 31 Aug 2020 23:00:24 +0000 (UTC)
+Authentication-Results: mail-b.sr.ht;
+	dkim=pass (2048-bit key) header.d=euandre.org header.i=@euandre.org header.b=UPSgUUXq
+Received: from authenticated-user (box.euandre.org [46.101.160.115])
+	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
+	(No client certificate requested)
+	by box.euandre.org (Postfix) with ESMTPSA id 70152FC81E;
+	Mon, 31 Aug 2020 20:00:20 -0300 (-03)
+DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=euandre.org; s=mail;
+	t=1598914822; bh=f4y8R0BR6plzUtLJhSLdKJQrnDt5TiYAaxpeGwaCRBQ=;
+	h=From:To:Cc:Subject:In-Reply-To:References:Date:From;
+	b=UPSgUUXqESOz5UhGXtlYTFrg0Vy42raDUdQSF1jRvRJpFdVQ7MWErphZhOvLjMrSB
+	 UxSspR/+WTYJDMf61hcqMfCqFYqGLpFH6Q6I6V4M7MABddX6mKgoyL16fBJl2V/FcH
+	 DVJBcrI+FeDrr3eoLTbXR5ZZzddSTSzfS2w93/QLad95xyd443VBEtUXEvl+G6pLC1
+	 Bzv7oeG07V2OZF+jIsfW116e0tyUsT1JTY377yiWTIxoqP27+PIyP2SZjPyBsS/Xzb
+	 0MqK6PhL3IbwQ8xvv/h2SklKD9PYXtrjU5hupBXJSb/puN7fZHoaSv/qM69mfNuMxI
+	 yTfUF1RUxJy5A==
+From: EuAndreh <[email-masked]>
+To: Rory =?utf-8?Q?O=E2=80=99Kane?= <[email-masked]>,
+ ~euandreh/[email-masked]
+Cc: 
+Subject: Re: The database I wish I had
+In-Reply-To: <[email-masked]>
+References: <[email-masked]>
+Date: Mon, 31 Aug 2020 19:55:51 -0300
+Message-ID: <[email-masked]>
+MIME-Version: 1.0
+Content-Type: text/plain; charset=utf-8
+Content-Transfer-Encoding: quoted-printable
+
+Rory O=E2=80=99Kane <[email-masked]> writes:
+
+> In this discussion of your post on Lobsters, there are some comments
+> that may aid you in your quest:
+> https://lobste.rs/s/m9vkg4/database_i_wish_i_had
+
+I don't have an account to discuss things there, and I don't have an
+invite, either.
+
+If you could invite me there I would gladly engage on the discussion =E2=98=
+=BA=EF=B8=8F.
+
+From [email-masked] Mon Aug 31 23:43:09 2020
+Received: from mail-qk1-f180.google.com (mail-qk1-f180.google.com [209.85.222.180])
+	by mail-b.sr.ht (Postfix) with ESMTPS id 43D60FF107
+	for <~euandreh/[email-masked]>; Mon, 31 Aug 2020 23:43:09 +0000 (UTC)
+Received: by mail-qk1-f180.google.com with SMTP id w186so7938858qkd.1
+        for <~euandreh/[email-masked]>; Mon, 31 Aug 2020 16:43:09 -0700 (PDT)
+X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
+        d=1e100.net; s=20161025;
+        h=x-gm-message-state:mime-version:references:in-reply-to:from:date
+         :message-id:subject:to:cc:content-transfer-encoding;
+        bh=gaaBELP8x2iHF+2fExDi20fg3n/w86cYg/IGy9Qn3xc=;
+        b=DRAjD5Bo+V1b4ugqTEX8uvyep/1C0WyUO+myK/uHn6v95bQRDYNS1+whm98sXMPXrb
+         qcafP0m0kbUfsTjEboLvrqZBZa3Ztv+8GFW6XmRjrtfJuZXx7RL0wXj16daqa2ik/dEe
+         RA0WM9UrrczXtTBjwnrG7ozNOGy1SKDKHO/c4f7aylfYGP+wVxyDl6nAaMQk+wHGkJcY
+         I37uXMATBkcsNPO+R8/lsyiCLkgk3wED+XJpYKZekgRA8p2I4S7wOP1SqmIpIyQlNllq
+         cYtPJkv8wi5MNtwR8t0k+DU6QZwYk8H1U+3YhWfwuVl+S+zHN1DC3Tw5ZrlxcIlpsP5v
+         0fQA==
+X-Gm-Message-State: AOAM530oi09/wtTwog1b67r3kLqfg4eevHc/vZXa69bokmICUCS/wXgK
+	ZU0PXrAbUDFg2UNYNvv/oKkU5//k/mo=
+X-Google-Smtp-Source: ABdhPJyzZixNEwElMWvVTgTATohR8FEqbOW0p/68jnNsN8u+gCY6xUFeVZhDVtPlvGcDFDsynVK1gQ==
+X-Received: by 2002:a05:620a:2224:: with SMTP id n4mr4016163qkh.18.1598917388635;
+        Mon, 31 Aug 2020 16:43:08 -0700 (PDT)
+Received: from mail-qk1-f178.google.com (mail-qk1-f178.google.com. [209.85.222.178])
+        by smtp.gmail.com with ESMTPSA id u15sm12587433qtj.3.2020.08.31.16.43.08
+        for <~euandreh/[email-masked]>
+        (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
+        Mon, 31 Aug 2020 16:43:08 -0700 (PDT)
+Received: by mail-qk1-f178.google.com with SMTP id p25so7942117qkp.2
+        for <~euandreh/[email-masked]>; Mon, 31 Aug 2020 16:43:08 -0700 (PDT)
+X-Received: by 2002:a37:b6c6:: with SMTP id g189mr4006715qkf.491.1598917387885;
+ Mon, 31 Aug 2020 16:43:07 -0700 (PDT)
+MIME-Version: 1.0
+References: <[email-masked]>
+ <[email-masked]>
+In-Reply-To: <[email-masked]>
+From: =?UTF-8?Q?Rory_O=E2=80=99Kane?= <[email-masked]>
+Date: Mon, 31 Aug 2020 19:42:57 -0400
+X-Gmail-Original-Message-ID: <CAD_ag1Obqre+uRN+fQgX1-O7WWkT=[email-masked]>
+Message-ID: <CAD_ag1Obqre+uRN+fQgX1-O7WWkT=[email-masked]>
+Subject: Re: The database I wish I had
+To: EuAndreh <[email-masked]>
+Cc: ~euandreh/[email-masked]
+Content-Type: text/plain; charset="UTF-8"
+Content-Transfer-Encoding: quoted-printable
+
+EuAndreh <[email-masked]> wrote:
+
+> I don't have an account to discuss things there, and I don't have an
+> invite, either.
+>
+> If you could invite me there I would gladly engage on the discussion =EF=
+=B8=8F.
+
+I checked out your website and based on your writing, I can probably
+trust you with an invite. Invite sent=E2=80=94enjoy.
+
+From [email-masked] Tue Sep  1 01:00:13 2020
+Received: from box.euandre.org (box.euandre.org [46.101.160.115])
+	by mail-b.sr.ht (Postfix) with ESMTPS id B6E38FF11C
+	for <~euandreh/[email-masked]>; Tue,  1 Sep 2020 01:00:12 +0000 (UTC)
+Authentication-Results: mail-b.sr.ht;
+	dkim=pass (2048-bit key) header.d=euandre.org header.i=@euandre.org header.b=P61dJynV
+Received: from authenticated-user (box.euandre.org [46.101.160.115])
+	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
+	(No client certificate requested)
+	by box.euandre.org (Postfix) with ESMTPSA id 167EDFC820;
+	Mon, 31 Aug 2020 22:00:09 -0300 (-03)
+DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=euandre.org; s=mail;
+	t=1598922011; bh=VOay/ul6frlnGv4X4iPDMI75WLW6Uop6PCjAV5nwAac=;
+	h=From:To:Cc:Cc:Subject:In-Reply-To:References:Date:From;
+	b=P61dJynVkkTLwtYleZjejOi5dPaIMpaJR1omX34om5nTFtkTU/7ckXbqlU/AlUxfb
+	 IC516Xu9/pkk1uW45SdeoU8+9nYnK9/KK7VCiifC5yQeMLZwlzNupanCYeycFes500
+	 D5Ygal4+Kz6prDX2cKrF4O97/nBKQKG6QQDzRG61Q1v+Tg2OMoJ45XENtQ7ONYzt9H
+	 RkbSKiNA4ipJDzLx1vd9VX+cKSC9VljZn9dFxIBA6KQ+nmbMNXtNmirc10sREJ2g3u
+	 zYefnUA3vRF1csdbIkTjfbiWnD2w2RN3Zj/oMwhNiAjHix4FyIg89WXwzv/CHmP9gr
+	 3+SrXBIC6XxzQ==
+From: EuAndreh <[email-masked]>
+To: Rory =?utf-8?Q?O=E2=80=99Kane?= <[email-masked]>
+Cc: ~euandreh/[email-masked]
+Cc: 
+Subject: Re: The database I wish I had
+In-Reply-To: <CAD_ag1Obqre+uRN+fQgX1-O7WWkT=[email-masked]>
+References: <[email-masked]>
+ <[email-masked]>
+ <CAD_ag1Obqre+uRN+fQgX1-O7WWkT=[email-masked]>
+Date: Mon, 31 Aug 2020 21:42:38 -0300
+Message-ID: <[email-masked]>
+MIME-Version: 1.0
+Content-Type: text/plain; charset=utf-8
+Content-Transfer-Encoding: quoted-printable
+
+Rory O=E2=80=99Kane <[email-masked]> writes:
+
+> I checked out your website and based on your writing, I can probably
+> trust you with an invite. Invite sent=E2=80=94enjoy.
+
+Thanks for the invitation, I hope I can add to the discussion and the
+community.=20
+
+From [email-masked] Tue Sep  1 03:22:08 2020
+Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28])
+	by mail-b.sr.ht (Postfix) with ESMTPS id 85748FF119
+	for <~euandreh/[email-masked]>; Tue,  1 Sep 2020 03:22:08 +0000 (UTC)
+Authentication-Results: mail-b.sr.ht;
+	dkim=pass (1024-bit key) header.d=josephg.com header.i=@josephg.com header.b=XpX/7nfb;
+	dkim=fail reason="key not found in DNS" (0-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=MSacOUNm
+Received: from compute1.internal (compute1.nyi.internal [10.202.2.41])
+	by mailout.nyi.internal (Postfix) with ESMTP id 4403D5C0190
+	for <~euandreh/[email-masked]>; Mon, 31 Aug 2020 23:22:08 -0400 (EDT)
+Received: from imap35 ([10.202.2.85])
+  by compute1.internal (MEProxy); Mon, 31 Aug 2020 23:22:08 -0400
+DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=josephg.com; h=
+	mime-version:message-id:date:from:to:subject:content-type; s=
+	mesmtp; bh=D0CRtslwgxf19V66w3zMnP44hZ3wW3VGv7uaTI0Tszs=; b=XpX/7
+	nfbLe6I0o40Nfy5xKR+AcUanUComIWaSJMa5Gz5V2bCbIFdj6j2sSHEJWMJ/OvnG
+	K4hgJmu5ZXcYOmhZBRIrGpjackRdPfipduHOMYuWsT/tdMecLsxBU79WhNR71xfi
+	gCQK/U2ChiHuu2JvN2lhP9UZHk7EoaAkBGVtwQ=
+DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
+	messagingengine.com; h=content-type:date:from:message-id
+	:mime-version:subject:to:x-me-proxy:x-me-proxy:x-me-sender
+	:x-me-sender:x-sasl-enc; s=fm3; bh=D0CRtslwgxf19V66w3zMnP44hZ3wW
+	3VGv7uaTI0Tszs=; b=MSacOUNm/cU+7m2AkVNHn56i7XS7rZXVQlRheFQ10MXFQ
+	gY7AEoJYghtK1hl+FZhe4yTOOUJfj7xGQsnp7ENjd0yHSZPIqh4w1WDWMeWgvIwg
+	p3v7xHQc/ce/1zKCV7NNftJh9ietQ63JTujrUTGC0Hszv5+D+7mkLO+OGtMwAmVR
+	auNg0sFFa6BdTfaQLRWAX/BqQkiYcJ9oFW+Lxw1wiN6D/K005aRgHbbOwCAHGi9v
+	6qkg/cVFzzqR52ydek4Oq0AtGwRBVMGK7llRBDRe83VDOcjzRNZScU/Z1hlEhDWF
+	fYm0B4nWev8MbeQUup3DjryE9U0WrMv/1MmI+G25A==
+X-ME-Sender: <xms:X75NX0NkXIq-arNl9tbSsnRDUhjMNpg-2ozeoZqLY6_2_W3DklLffQ>
+X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduiedrudefiedgjedtucetufdoteggodetrfdotf
+    fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen
+    uceurghilhhouhhtmecufedttdenucenucfjughrpefofgggkfffhffvufgtsehttdertd
+    erredtnecuhfhrohhmpedfufgvphhhucfivghnthhlvgdfuceomhgvsehjohhsvghphhhg
+    rdgtohhmqeenucggtffrrghtthgvrhhnpeekgffhuedttedvjeelkeffkeeuueelkeeite
+    dttefgleejledtueetgfeijefhueenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgr
+    mhepmhgrihhlfhhrohhmpehmvgesjhhoshgvphhhghdrtghomh
+X-ME-Proxy: <xmx:X75NX6_-JLu-Eneod1EKsHyKkOEIRYFaAZQH4uBldS7GxRe_0WzDdw>
+    <xmx:X75NX7QCRm4jcdxXVZf_-SgheExet0j7JbTijuDCSZfm7EpG8GoXxg>
+    <xmx:X75NX8tKELWH7ryIzS_tU-Gi7kQXm8uw2XHfCRdbB4HJ08Y4s4Nheg>
+    <xmx:YL5NX4HEO3VR6t7Yz1-vKIi99YuUZV2t1wRHH3Tqv8f1bMEvEjAbAw>
+Received: by mailuser.nyi.internal (Postfix, from userid 501)
+	id C51F714C00E5; Mon, 31 Aug 2020 23:22:07 -0400 (EDT)
+X-Mailer: MessagingEngine.com Webmail Interface
+User-Agent: Cyrus-JMAP/3.3.0-232-g4bdb081-fm-20200825.002-g4bdb081a
+Mime-Version: 1.0
+Message-Id: <[email-masked]>
+Date: Tue, 01 Sep 2020 03:21:47 +0000
+From: "Seph Gentle" <[email-masked]>
+To: ~euandreh/[email-masked]
+Subject: Making a database
+Content-Type: text/plain
+
+I've been thinking along similar lines, and am talking to some folks about building something on top of CRDTs. Modern CRDTs like automerge and y.js are just getting to the point where they're fast and small enough to work fine. I can see something really valuable that looks kind of like git, except where you can store arbitrary data. And live-sync peers - so if I make changes you see those changes live on your computer too - without (necessarily) needing a centralized server.
+
+Love to chat if you & others are keen - I/we are still in the process of figuring out what this thing will look like. But, I feel like the database piece is really core here too, and I like a lot of your thinking in this space.
+
+-Seph
+
+From [email-masked] Tue Sep  1 09:26:09 2020
+Received: from mout-p-102.mailbox.org (mout-p-102.mailbox.org [80.241.56.152])
+	by mail-b.sr.ht (Postfix) with ESMTPS id CEBA2FF127
+	for <~euandreh/[email-masked]>; Tue,  1 Sep 2020 09:26:08 +0000 (UTC)
+Received: from smtp2.mailbox.org (smtp2.mailbox.org [IPv6:2001:67c:2050:105:465:1:2:0])
+	(using TLSv1.2 with cipher ECDHE-RSA-CHACHA20-POLY1305 (256/256 bits))
+	(No client certificate requested)
+	by mout-p-102.mailbox.org (Postfix) with ESMTPS id 4BghWz4k2gzKmTx
+	for <~euandreh/[email-masked]>; Tue,  1 Sep 2020 11:26:07 +0200 (CEST)
+X-Virus-Scanned: amavisd-new at heinlein-support.de
+Received: from smtp2.mailbox.org ([80.241.60.241])
+	by spamfilter04.heinlein-hosting.de (spamfilter04.heinlein-hosting.de [80.241.56.122]) (amavisd-new, port 10030)
+	with ESMTP id lwr5j4QAvkHQ for <~euandreh/[email-masked]>;
+	Tue,  1 Sep 2020 11:26:02 +0200 (CEST)
+To: ~euandreh/[email-masked]
+Subject: Re: The database I wish I had
+From: =?UTF-8?Q?Jonas_Sch=c3=bcrmann?= <[email-masked]>
+Message-ID: <[email-masked]>
+Date: Tue, 1 Sep 2020 11:26:02 +0200
+MIME-Version: 1.0
+Content-Type: text/plain; charset=utf-8; format=flowed
+Content-Language: en-US
+X-MBO-SPAM-Probability: 
+X-Rspamd-Score: -1.02 / 15.00 / 15.00
+X-Rspamd-Queue-Id: 6015B17AB
+X-Rspamd-UID: af255d
+Content-Transfer-Encoding: quoted-printable
+
+Hello EuAndreh,
+
+I read your blog post with excitement because I'm currently on a very
+similar quest. My goal is to bring back application data into the hands
+of the users (and onto their devices) to replace centralized,
+cloud-based data stores. I wrote down what I wanted in this blog post
+[1].
+
+Like you, I didn't find anything that seemed to work like I wanted, so I
+decided to start exploring the problem myself. While you go for the
+Git-like approach of three-way-merges, I am trying to model conflict
+situations with completed categories.
+
+I can't say that I've come very far yet. I realized that properly
+modelling conflicts with completed categories is a hard problem and
+while I had some progress already, there isn't anything that I would
+consider ready to be shared just yet (if you're curious you can look at
+the source [2]).
+
+But through discussions on lobste.rs [3] and further research I built a
+list of references, prior work and concepts in this space [4]. From your
+blog post I even got a few more references that I just added :) Perhaps
+this list is also useful for you.
+
+I wish you the best of luck with your endeavor and I hope to see further
+posts about your progress on the blog :) When I have more time I will
+also write more about what I've found and what's next.
+
+Cordially, Jonas Sch=C3=BCrmann
+
+[1]=20
+https://jonas-schuermann.name/projects/dvcs-for-structured-data/blog/2020=
+-03-22-manifesto.html
+[2]=20
+https://gitlab.com/MazeChaZer/dvcs-for-structured-data/-/blob/master/sour=
+ce-code/EventSourcing.hs
+[3] https://lobste.rs/s/qsehiu/i_want_decentralized_version_control_for
+[4]=20
+https://jonas-schuermann.name/projects/dvcs-for-structured-data/notes/ref=
+erences.html
+
+From [email-masked] Tue Sep  1 10:00:15 2020
+Received: from box.euandre.org (box.euandre.org [46.101.160.115])
+	by mail-b.sr.ht (Postfix) with ESMTPS id 6FC96FF127
+	for <~euandreh/[email-masked]>; Tue,  1 Sep 2020 10:00:14 +0000 (UTC)
+Authentication-Results: mail-b.sr.ht;
+	dkim=pass (2048-bit key) header.d=euandre.org header.i=@euandre.org header.b=Fm0cCCjb
+Received: from authenticated-user (box.euandre.org [46.101.160.115])
+	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
+	(No client certificate requested)
+	by box.euandre.org (Postfix) with ESMTPSA id 82F61FCB45;
+	Tue,  1 Sep 2020 07:00:11 -0300 (-03)
+DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=euandre.org; s=mail;
+	t=1598954412; bh=d3MsFaC9js1laWdI/WHjnTEpvQ9IMPR4/17eUYzh5Z8=;
+	h=From:To:Cc:Subject:In-Reply-To:References:Date:From;
+	b=Fm0cCCjbaKJpKhoedrPl0XWIYJujunIk9LXBI7OUyABb7hgxRqEkLmW93LCA60QAE
+	 +aZMzweJYdcNtmRBMeQmlkFYwnZZq8L9OBlsqi+BOILAEA5jv8Ly8RMQ6ZV6kogRpY
+	 rV0RwyBEvskQ0vao5a/IWLuCSSalXShntwRIy44liQYZqxjFwJTQEXPDeb+gvmYaz1
+	 9aEvg5edL/OEGfo0VZpRk//f47V6a8o/3/uEaLpJAtEfCtQY1YYk5BdQGafCp5LrVR
+	 fdp/6ju5YnM3yjM52MxTUvEjFpvK+FL/FSQDJ2DhdtbrH/EHHmNpImLP9WL1Bw9mNY
+	 N2ar7lD/+9A6g==
+From: EuAndreh <[email-masked]>
+To: Seph Gentle <[email-masked]>, ~euandreh/[email-masked]
+Cc: 
+Subject: Re: Making a database
+In-Reply-To: <[email-masked]>
+References: <[email-masked]>
+Date: Tue, 01 Sep 2020 06:39:19 -0300
+Message-ID: <[email-masked]>
+MIME-Version: 1.0
+Content-Type: text/plain
+
+"Seph Gentle" <[email-masked]> writes:
+
+> I've been thinking along similar lines, and am talking to some folks
+> about building something on top of CRDTs. Modern CRDTs like automerge
+> and y.js are just getting to the point where they're fast and small
+> enough to work fine. I can see something really valuable that looks
+> kind of like git, except where you can store arbitrary data. And
+> live-sync peers - so if I make changes you see those changes live on
+> your computer too - without (necessarily) needing a centralized
+> server.
+>
+> Love to chat if you & others are keen - I/we are still in the process
+> of figuring out what this thing will look like. But, I feel like the
+> database piece is really core here too, and I like a lot of your
+> thinking in this space.
+
+Thanks for the encouragement :)
+
+I've looked at CRDTs, but I haven't done a deep dive yet, because my
+first impression was that they generally choose to limit the types of
+data structures and operations that you can perform on them in favour of
+being conflict-free.
+
+That's why I didn't mention it on my article directly, but I admit to
+having it as pre-requisite to making an informed decision on the
+subject.
+
+WDYT?
+
+From [email-masked] Tue Sep  1 11:00:18 2020
+Received: from box.euandre.org (box.euandre.org [46.101.160.115])
+	by mail-b.sr.ht (Postfix) with ESMTPS id 7E055FF122
+	for <~euandreh/[email-masked]>; Tue,  1 Sep 2020 11:00:17 +0000 (UTC)
+Authentication-Results: mail-b.sr.ht;
+	dkim=pass (2048-bit key) header.d=euandre.org header.i=@euandre.org header.b=aexrWU2E
+Received: from authenticated-user (box.euandre.org [46.101.160.115])
+	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
+	(No client certificate requested)
+	by box.euandre.org (Postfix) with ESMTPSA id A3E67FD6B9;
+	Tue,  1 Sep 2020 08:00:14 -0300 (-03)
+DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=euandre.org; s=mail;
+	t=1598958016; bh=RLLLMwMzFeXBwtfwXnEsHK/UTMXFBGSC5qgHaz/ZmOE=;
+	h=From:To:Cc:Subject:In-Reply-To:References:Date:From;
+	b=aexrWU2Ee5zBZdVCWHCoLentiEWPHDdm7cDXSVNCrsIO+wkZPDXWbT+YwGbX7pldB
+	 rs//2AD5Bz0VOPByW33jD53/lmCCBeZ8eBjSLpfN2/w3sXHKPXh348R6/YLLRMSuwj
+	 9bCKuxK3X8qJ2SMS3fg8OGIzHgBq8rL59U0pFgCiV6WR5Z92mG80P8iogtSI8LzU/g
+	 yBhu4sv4gJ3IlLhTiwX4HgAQkSoOCu6rdm5AbDdkwPwteUyy+7F2KlHc6eD7UTktU8
+	 ic7l2oVEyQrUYddwtZmUfJJCG9tffy5RDAyBO/TaLjVJpesdFEY5nRIG/Xm+1geH1j
+	 fv++ZMqsLTgxQ==
+From: EuAndreh <[email-masked]>
+To: Jonas =?utf-8?Q?Sch=C3=BCrmann?= <[email-masked]>,
+ ~euandreh/[email-masked]
+Cc: 
+Subject: Re: The database I wish I had
+In-Reply-To: <[email-masked]>
+References: <[email-masked]>
+Date: Tue, 01 Sep 2020 07:26:47 -0300
+Message-ID: <[email-masked]>
+MIME-Version: 1.0
+Content-Type: text/plain; charset=utf-8
+Content-Transfer-Encoding: quoted-printable
+
+Jonas Sch=C3=BCrmann <[email-masked]> writes:
+
+> Hello EuAndreh,
+
+Hi there!
+
+> I read your blog post with excitement because I'm currently on a very
+> similar quest. My goal is to bring back application data into the hands
+> of the users (and onto their devices) to replace centralized,
+> cloud-based data stores. I wrote down what I wanted in this blog post
+> [1].
+>
+> Like you, I didn't find anything that seemed to work like I wanted, so I
+> decided to start exploring the problem myself. While you go for the
+> Git-like approach of three-way-merges, I am trying to model conflict
+> situations with completed categories.
+
+Hmmm, I haven't heard this term before. Are these [0] the complete
+categories that you mean?
+
+[0]: https://en.wikipedia.org/wiki/Complete_category
+=20
+> I can't say that I've come very far yet. I realized that properly
+> modelling conflicts with completed categories is a hard problem and
+> while I had some progress already, there isn't anything that I would
+> consider ready to be shared just yet (if you're curious you can look at
+> the source [2]).
+>
+> But through discussions on lobste.rs [3] and further research I built a
+> list of references, prior work and concepts in this space [4]. From your
+> blog post I even got a few more references that I just added :) Perhaps
+> this list is also useful for you.
+
+Thanks for all the links, I'll try to use them to enrich my
+understanding on the subject.
+
+From [email-masked] Tue Sep  1 11:25:21 2020
+Received: from wout2-smtp.messagingengine.com (wout2-smtp.messagingengine.com [64.147.123.25])
+	by mail-b.sr.ht (Postfix) with ESMTPS id 50AC3FF122
+	for <~euandreh/[email-masked]>; Tue,  1 Sep 2020 11:25:21 +0000 (UTC)
+Authentication-Results: mail-b.sr.ht;
+	dkim=pass (1024-bit key) header.d=josephg.com header.i=@josephg.com header.b=O0rYtvTA;
+	dkim=fail reason="key not found in DNS" (0-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=f1mPF/1x
+Received: from compute1.internal (compute1.nyi.internal [10.202.2.41])
+	by mailout.west.internal (Postfix) with ESMTP id E4956E77;
+	Tue,  1 Sep 2020 07:25:19 -0400 (EDT)
+Received: from imap35 ([10.202.2.85])
+  by compute1.internal (MEProxy); Tue, 01 Sep 2020 07:25:20 -0400
+DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=josephg.com; h=
+	mime-version:message-id:in-reply-to:references:date:from:to
+	:subject:content-type:content-transfer-encoding; s=mesmtp; bh=BO
+	dAgwWsH67HEzbojnfogot17D0WKCAzyItvgTLmJCw=; b=O0rYtvTAoINKZTJObn
+	vUnmONdChl7CPOEklbKnqUQ+Sk8qaAoCe0YyqwTLq0YZywk49u56Zwff5QaSJfLm
+	SYA9bd2RUEagih/gPTR4BHJarqfWSHQeUsHsvMmVtj4w8jrJt261gcmh1Git8Jkt
+	lJSUIZ9tpOecI/nFsnhxCleA0=
+DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
+	messagingengine.com; h=content-transfer-encoding:content-type
+	:date:from:in-reply-to:message-id:mime-version:references
+	:subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender
+	:x-sasl-enc; s=fm3; bh=BOdAgwWsH67HEzbojnfogot17D0WKCAzyItvgTLmJ
+	Cw=; b=f1mPF/1xX+9SrbduF4hyrnDZlojUnotl/ZuV6F+yuzbQlOHFTtLgMSJ5u
+	ghKQ7iTGJdFGqxzM87y1JJLCWb6BZjbRw2TjhDK+z24JeMwOVOpZgDsZKZ4vWlHI
+	x6C8Ca/TnlF8Glsyvepoczx5SP6kbYCOe/JrBGp3gTIfTGohNHuxCsxxHvW3gn35
+	hQLIa42DMt3y1p0q2rl4ltdrCYpCzQVvfbhKssX3AmCkwndYDhbP6l4jLnWH6jeM
+	oBw77aR3keR94fhUnSCsnlX5ZSljRkIecR7Z9sGsR8KeB5j72A/ND3rncJyOOQ/e
+	+DXRIAj+KVgqvbY2AimyrweAA9mQg==
+X-ME-Sender: <xms:nS9OX8TbEfFCRuUyS6F4hwgBx8v03SzB2Ggzuy63tMjyFVg-UrVkNQ>
+    <xme:nS9OX5zGsoiG5l8lthnVpFgOEb1gqC7oeFR2NwTbn4uDQFjSZ-ZvMZeo0aSeRF8GV
+    exRk5tGsHuiPRtDjw>
+X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduiedrudefjedggedtucetufdoteggodetrfdotf
+    fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen
+    uceurghilhhouhhtmecufedttdenucenucfjughrpefofgggkfgjfhffhffvufgtgfesth
+    hqredtreerjeenucfhrhhomhepfdfuvghphhcuifgvnhhtlhgvfdcuoehmvgesjhhoshgv
+    phhhghdrtghomheqnecuggftrfgrthhtvghrnhepkeegtdfhgfdvheetveetleejgfdvhf
+    egtdegfedvhfegveffgfdthfehgedukeegnecuffhomhgrihhnpehslhgrtghkrdgtohhm
+    necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepmhgvse
+    hjohhsvghphhhgrdgtohhm
+X-ME-Proxy: <xmx:nS9OX50QvymIMNNSVjvYuwnELZacWmVFdJvm4gYZ7wZ28Ysp8XeqaA>
+    <xmx:nS9OXwCN5irAMSLyp8Oejw_bDz4dedc6-GQj4ThImfuyCt5P3RUUhA>
+    <xmx:nS9OX1jNG59H8yiSSKrEdc830ToS-wnjIO8Dqrr2hhrFB4opADJCoQ>
+    <xmx:ny9OX6tPG_S_xI2Z3IHEmzknqb1-PlAmuX_wv6n9qglqXIcgp0MqSQ>
+Received: by mailuser.nyi.internal (Postfix, from userid 501)
+	id 72BAB14C034C; Tue,  1 Sep 2020 07:25:17 -0400 (EDT)
+X-Mailer: MessagingEngine.com Webmail Interface
+User-Agent: Cyrus-JMAP/3.3.0-248-gcd102cb-fm-20200901.001-gcd102cb9
+Mime-Version: 1.0
+Message-Id: <[email-masked]>
+In-Reply-To: <[email-masked]>
+References: <[email-masked]>
+ <[email-masked]>
+Date: Tue, 01 Sep 2020 21:24:55 +1000
+From: "Seph Gentle" <[email-masked]>
+To: EuAndreh <[email-masked]>, ~euandreh/[email-masked]
+Subject: Re: Making a database
+Content-Type: text/plain;charset=utf-8
+Content-Transfer-Encoding: quoted-printable
+
+On Tue, Sep 1, 2020, at 7:39 PM, EuAndreh wrote:
+> "Seph Gentle" <[email-masked]> writes:
+>=20
+> > I've been thinking along similar lines, and am talking to some folks=
+
+> > about building something on top of CRDTs. Modern CRDTs like automerg=
+e
+> > and y.js are just getting to the point where they're fast and small
+> > enough to work fine. I can see something really valuable that looks
+> > kind of like git, except where you can store arbitrary data. And
+> > live-sync peers - so if I make changes you see those changes live on=
+
+> > your computer too - without (necessarily) needing a centralized
+> > server.
+> >
+> > Love to chat if you & others are keen - I/we are still in the proces=
+s
+> > of figuring out what this thing will look like. But, I feel like the=
+
+> > database piece is really core here too, and I like a lot of your
+> > thinking in this space.
+>=20
+> Thanks for the encouragement :)
+>=20
+> I've looked at CRDTs, but I haven't done a deep dive yet, because my
+> first impression was that they generally choose to limit the types of
+> data structures and operations that you can perform on them in favour =
+of
+> being conflict-free.
+>=20
+> That's why I didn't mention it on my article directly, but I admit to
+> having it as pre-requisite to making an informed decision on the
+> subject.
+>=20
+> WDYT?
+
+Well, they=E2=80=99re conflict-free in the sense that (unlike Postgres/f=
+oundationdb/etc) the database doesn=E2=80=99t reject conflicting concurr=
+ent updates. Which is a property you need if you=E2=80=99re going to com=
+mit local writes while you=E2=80=99re offline. And if I=E2=80=99m readin=
+g your blog post right it sounds like that=E2=80=99s what you=E2=80=99re=
+ going for.
+
+You kinda gave a good description of them in your post already. They wor=
+k by requiring all peers to resolve concurrent edits to an object the sa=
+me way. That could be last-writer-wins (based on time stamps like you sa=
+id in your article). Or it could store all conflicting versions together=
+ to be resolved by the next reader (Eg riak). Or it could do something m=
+ore clever (like merge changes using automerge or y.js or the like). Git=
+ is sort of a CRDT - it=E2=80=99s just it=E2=80=99s 3 way merge algorith=
+m is a bit of a dogs breakfast as far as these things go.
+
+Anyway, happy to chat about this stuff in more detail if you wanna zoom =
+or something. Or hop into the automerge slack (with pvh from ink&switch,=
+ and Martin Kleppmann, and others) and we can talk about this stuff - ht=
+tps://join.slack.com/t/automerge/shared_invite/zt-e4p3760n-kKh7r3KRH1Yww=
+NfiZM8ktw
+
+As I said - I=E2=80=99m looking to build something like this too and it=E2=
+=80=99d be good to compare notes on our approaches!
+
+From [email-masked] Tue Sep  1 15:46:34 2020
+Received: from a27-232.smtp-out.us-west-2.amazonses.com (a27-232.smtp-out.us-west-2.amazonses.com [54.240.27.232])
+	by mail-b.sr.ht (Postfix) with ESMTPS id 02A70FF0E3
+	for <~euandreh/[email-masked]>; Tue,  1 Sep 2020 15:46:33 +0000 (UTC)
+Authentication-Results: mail-b.sr.ht;
+	dkim=pass (1024-bit key) header.d=twinql.com header.i=@twinql.com header.b=cNOZcfp4;
+	dkim=pass (1024-bit key) header.d=amazonses.com header.i=@amazonses.com header.b=Hg/JwA02
+DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple;
+	s=qvs2k2sd2ivtrflyu2nswmravuqizjml; d=twinql.com; t=1598975192;
+	h=Subject:From:To:Date:Mime-Version:Content-Type:Content-Transfer-Encoding:References:Message-Id;
+	bh=h5tYwgdiFvKfB58ME/qKUDlZfMWWDjBD1us0jqb+obo=;
+	b=cNOZcfp4gLnUosw6qaEpaunrxtpsBSNsQgvh/C3StK1VPirNXCoHKVx5ZtOO/szG
+	PdOufKL9NM0bQu6OiYFBxUdp2fuZ2FGwONd2nzGBOGcMN4V2aSOuMAGcuClGe2Kf+SM
+	xfSfiY2iyxEUtlFoNU9qfvTtJ5Wutm+/EkIRuJ+g=
+DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple;
+	s=hsbnp7p3ensaochzwyq5wwmceodymuwv; d=amazonses.com; t=1598975192;
+	h=Subject:From:To:Date:Mime-Version:Content-Type:Content-Transfer-Encoding:References:Message-Id:Feedback-ID;
+	bh=h5tYwgdiFvKfB58ME/qKUDlZfMWWDjBD1us0jqb+obo=;
+	b=Hg/JwA02/i8iV50zfzagoO6qkIy3fWh3xaS3Ea/Mbew1Ve1B70ilQdJIrthnSRCF
+	S50Nn0YzMQlyzXXd8rCHSfwTcqDUYV14hdN+GPbGPF+O+6w/X7KG6PxWnYTE/5w/cE9
+	5HAQ9xp3FdpTqa2RvVsLtEJVweoEtHmNR5319VRs=
+Subject: Re: The database I wish I had
+From: =?UTF-8?Q?Richard_Newman?= <[email-masked]>
+To: =?UTF-8?Q?=7Eeuandreh/public-inbox=40lists=2Esr=2Eht?=
+  <~euandreh/[email-masked]>
+Date: Tue, 1 Sep 2020 15:46:31 +0000
+Mime-Version: 1.0
+Content-Type: text/plain; charset=UTF-8
+Content-Transfer-Encoding: quoted-printable
+References: <[email-masked]>
+X-Priority: 3 (Normal)
+X-Mailer: Amazon WorkMail
+Thread-Index: AQHWgHcPJ4udfXYrRaOwhoO7EcsDJQ==
+Thread-Topic: The database I wish I had
+X-Original-Mailer: Apple Mail (2.3445.104.14)
+X-Wm-Sent-Timestamp: 1598975191
+Message-ID: <[email-masked]>
+X-SES-Outgoing: 2020.09.01-54.240.27.232
+Feedback-ID: 1.us-west-2.An468LAV0jCjQDrDLvlZjeAthld7qrhZr+vow8irkvU=:AmazonSES
+
+Hello! Original author of Mentat here.=0D=0A=0D=0AThe use of SQLite for M=
+entat was intended to be an accelerant for development: by leveraging SQL=
+ite we could ignore large parts of the tricky problem of getting bits on =
+disk across platforms, and also get things like transactionality and full=
+-text search for free.=0D=0A=0D=0AWe used the same leverage trick when we=
+ built the first proof of concept using Clojure and ClojureScript =E2=80=94=
+ we could run it in a JVM locally, compare to Datomic, and test in Node, =
+and entirely avoid having to build parsers and lexers. This allowed us to=
+ focus on the problem of representing a datom/log/index data model in SQL=
+ite.=0D=0A=0D=0AThe downside of using SQLite was a large amount of time w=
+riting code to compile queries =E2=80=94 which were written in much the s=
+ame language as Datomic=E2=80=99s =E2=80=94 into SQL for execution. That =
+Datomic compatibility also pushed us towards building an EDN parser in Ru=
+st, and we found some under-specified parts of Datomic, too.=0D=0A=0D=0AP=
+ortability to the web itself was not an initial goal, but it was somethin=
+g we thought about. With adequate resourcing, and the benefit of hindsigh=
+t, we would have instead built a chunk-based backing store abstraction, u=
+sing something like rkv/LMDB for embedded use and indexeddb on the web. W=
+e did not feel like we had the engineering capacity to take on building a=
+ truly portable, battle-tested storage layer like SQLite=E2=80=99s *and* =
+a bits-on-disk query executor =E2=80=94 I=E2=80=99ve done it in the past,=
+ and it takes a lot of time and is hard to get right.=0D=0A=0D=0AMentat a=
+ttempted to address a number of things that you didn=E2=80=99t cover in y=
+our brief post. A robust definition of identity, end-to-end encryption, a=
+nd evolution of schema (even simple schema like cardinality and uniquenes=
+s constraints on attributes, which are the foundation of identity) are th=
+ree such. Schema evolution becomes tremendously important when you have m=
+ultiple teams using the same storage, or a long-lived product =E2=80=94 t=
+he more useful the database is, the more likely you are to need this feat=
+ure!=0D=0A=0D=0AThose three things cascade into the problem of synchroniz=
+ation, which was my focus for a number of years.=0D=0A=0D=0ASynchronizati=
+on is a hard problem. Doing it without server assistance is even harder, =
+but is necessary for a portable and secure product. Synchronization of ge=
+neric data =E2=80=94 that is, data where the synchronization tool itself =
+doesn=E2=80=99t have its own model of the world =E2=80=94 is extremely tr=
+icky: everything from identifiers through to basic constraints needs to b=
+e aware of the possibility of change, and naturally it=E2=80=99s an itera=
+tive and distributed situation. Most of the approaches I found rely on a =
+server and are for systems that omit schema, and are typically incomplete=
+: e.g., detecting conflicts on individual attributes, leading to nonsensi=
+cal results, and offload to the developer the problem of separately handl=
+ing stable identifiers.=0D=0A=0D=0AI still have a persistent itch to star=
+t with a clean slate and solve this problem. No large company has the rig=
+ht incentives to solve the much harder problems faced by a truly secure s=
+tore, most hobbyists lack the understanding of the importance of managing=
+ change and concurrent collaboration over time, and the current trend is =
+towards using =E2=80=98dumb=E2=80=99 object/key-value stores or specialis=
+t systems like time-series databases, so I think it=E2=80=99s unlikely th=
+at anybody else is going to tackle all three in the same solution. Here=E2=
+=80=99s hoping!
+
+From [email-masked] Tue Sep  1 18:00:18 2020
+Received: from box.euandre.org (box.euandre.org [46.101.160.115])
+	by mail-b.sr.ht (Postfix) with ESMTPS id CD25BFF0C9
+	for <~euandreh/[email-masked]>; Tue,  1 Sep 2020 18:00:17 +0000 (UTC)
+Authentication-Results: mail-b.sr.ht;
+	dkim=pass (2048-bit key) header.d=euandre.org header.i=@euandre.org header.b=q+qyYbsV
+Received: from authenticated-user (box.euandre.org [46.101.160.115])
+	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
+	(No client certificate requested)
+	by box.euandre.org (Postfix) with ESMTPSA id F050CFCB45;
+	Tue,  1 Sep 2020 15:00:14 -0300 (-03)
+DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=euandre.org; s=mail;
+	t=1598983216; bh=HeMLnOZIeRgOrLdGepra14BxoBWpvE7fXfdvB9MFFjw=;
+	h=From:To:Cc:Subject:In-Reply-To:References:Date:From;
+	b=q+qyYbsVLZrz3+swol7Y+Sc59kNUZkM1tSaXTcgLVIOp3CSJ+96XlP7i6oKZdeubH
+	 fZIdw2fmFCMQotaXCnLJKBUNauX5Jrvt+4t2TrujIie5i5o/VSp08DxtxyWAmUWqdt
+	 nPZrAand3styW0vvpWFYCZ7RjJKs4Ujek2G4H713X/eDOTooKalFNIXZ+A7t/xzq+9
+	 mYojhphqHBDFQeqNKs9500YqetgTg4hsYfqRmGHzOs5ADw1dkQegQhjduodWvj2SpI
+	 YeUu4OoXYDYUR9o+6VkXV+KtC4zDnoXyvgdSfHaLrahnkfiCBgg1Y6Kmcxy8iKnXjP
+	 q5baz6m+cXC9Q==
+From: EuAndreh <[email-masked]>
+To: Richard Newman <[email-masked]>,
+ "~euandreh/[email-masked]" <~euandreh/[email-masked]>
+Cc: 
+Subject: Re: The database I wish I had
+In-Reply-To: <[email-masked]>
+References: <[email-masked]>
+ <[email-masked]>
+Date: Tue, 01 Sep 2020 14:50:07 -0300
+Message-ID: <[email-masked]>
+MIME-Version: 1.0
+Content-Type: text/plain; charset=utf-8
+Content-Transfer-Encoding: quoted-printable
+
+Richard Newman <[email-masked]> writes:
+
+> Hello! Original author of Mentat here.
+
+Hi there! You wouldn't be suprised if I told I recognize you by name,
+would you?
+
+Great job on Mentat, and please write more installments on "Thinking
+about Syncing" if possible =E2=98=BA=EF=B8=8F
+
+> The use of SQLite for Mentat was intended to be an accelerant for
+> development: by leveraging SQLite we could ignore large parts of the
+> tricky problem of getting bits on disk across platforms, and also get
+> things like transactionality and full-text search for free.
+
+For your goal of supporting other teams working on Firefox, SQLite is a
+perfectly reasonable choice.
+
+> We used the same leverage trick when we built the first proof of
+> concept using Clojure and ClojureScript =E2=80=94 we could run it in a JVM
+> locally, compare to Datomic, and test in Node, and entirely avoid
+> having to build parsers and lexers. This allowed us to focus on the
+> problem of representing a datom/log/index data model in SQLite.=20
+>
+> The downside of using SQLite was a large amount of time writing code
+> to compile queries =E2=80=94 which were written in much the same language=
+ as
+> Datomic=E2=80=99s =E2=80=94 into SQL for execution. That Datomic compatib=
+ility also
+> pushed us towards building an EDN parser in Rust, and we found some
+> under-specified parts of Datomic, too.=20
+
+Hmm, I'd be interested in learning more about those under-specified
+pitfalls to try to avoid them if possible.=20
+
+> Portability to the web itself was not an initial goal, but it was
+> something we thought about. With adequate resourcing, and the benefit
+> of hindsight, we would have instead built a chunk-based backing store
+> abstraction, using something like rkv/LMDB for embedded use and
+> indexeddb on the web. We did not feel like we had the engineering
+> capacity to take on building a truly portable, battle-tested storage
+> layer like SQLite=E2=80=99s *and* a bits-on-disk query executor =E2=80=94=
+ I=E2=80=99ve done it
+> in the past, and it takes a lot of time and is hard to get right.=20
+
+I don't expect to match SQLite's robustness for quite some time, and
+having plugable storage APIs makes me consider having a SQLite storage
+API to have as a correctness reference.
+
+What do you mean by "chunk-based backing store abstraction"? Could you
+elaborate further?
+
+> Mentat attempted to address a number of things that you didn=E2=80=99t co=
+ver
+> in your brief post. A robust definition of identity, end-to-end
+> encryption, and evolution of schema (even simple schema like
+> cardinality and uniqueness constraints on attributes, which are the
+> foundation of identity) are three such. Schema evolution becomes
+> tremendously important when you have multiple teams using the same
+> storage, or a long-lived product =E2=80=94 the more useful the database i=
+s,
+> the more likely you are to need this feature!=20
+
+What do you mean by "identity"? As in ":db.unique/value" and
+":db.unique/identity"? Couldn't you just use UUIDs for
+":db.unique/identity" values?
+
+For schema evolution wouldn't the "Grow your schema, and never break
+it"[0] approach be enough?
+
+On encryption, my first approach would be to see if git-remote-gcrypt[1]
+functionality can be translated, and have the local database be ignorant
+of all encryption whatsoever. However, I haven't explored it enough to
+be able to declare it to be possible, so I would welcome a more mature
+view on the subject.
+
+[0]: https://blog.datomic.com/2017/01/the-ten-rules-of-schema-growth.html=20
+[1]: https://spwhitton.name/tech/code/git-remote-gcrypt/
+
+> Those three things cascade into the problem of synchronization, which
+> was my focus for a number of years.=20
+>
+> Synchronization is a hard problem. Doing it without server assistance
+> is even harder, but is necessary for a portable and secure product.
+> Synchronization of generic data =E2=80=94 that is, data where the
+> synchronization tool itself doesn=E2=80=99t have its own model of the wor=
+ld =E2=80=94
+> is extremely tricky: everything from identifiers through to basic
+> constraints needs to be aware of the possibility of change, and
+> naturally it=E2=80=99s an iterative and distributed situation. Most of the
+> approaches I found rely on a server and are for systems that omit
+> schema, and are typically incomplete: e.g., detecting conflicts on
+> individual attributes, leading to nonsensical results, and offload to
+> the developer the problem of separately handling stable identifiers.=20
+
+If sound and robust, what's the downside of detecting conflicts on
+individual attributes?
+
+If an Datomic-like entity with 10 attributes has 4 of them change by
+different database instances, but only 1 change is on the same
+attribute, would this type of detection offer more granularity instead?
+
+Wouldn't the 5-tuple be the equivalent of a single line of code on git,
+and aggregates are built on top of it?
+
+From [email-masked] Tue Sep  1 18:00:22 2020
+Received: from box.euandre.org (box.euandre.org [46.101.160.115])
+	by mail-b.sr.ht (Postfix) with ESMTPS id 097B1FF0C9
+	for <~euandreh/[email-masked]>; Tue,  1 Sep 2020 18:00:22 +0000 (UTC)
+Authentication-Results: mail-b.sr.ht;
+	dkim=pass (2048-bit key) header.d=euandre.org header.i=@euandre.org header.b=bsr8vYSW
+Received: from authenticated-user (box.euandre.org [46.101.160.115])
+	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
+	(No client certificate requested)
+	by box.euandre.org (Postfix) with ESMTPSA id C1FCCFD6BB;
+	Tue,  1 Sep 2020 15:00:19 -0300 (-03)
+DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=euandre.org; s=mail;
+	t=1598983221; bh=VxPcwqhwGX+tAOR9QBGKCYzKImUIi+BqaSfmrBhsR6g=;
+	h=From:To:Cc:Subject:In-Reply-To:References:Date:From;
+	b=bsr8vYSWGIarl0IY64Ew+GLFGsQhZe971tKRr72HG9/UJru3FMjjA0VXK5LmyRGAh
+	 ZjamXQpzM0biS0NNhY6mhfrvHXGKaGr19E6I5NDokNZFc/T/hXD1qMzY4fgEkDmayl
+	 fteqI7T//DT06yseOXOkfrKzNCDWKmSmlsOo9PZgSsAicwEsqxb/UqehXN6ON9vpG4
+	 VCXsOW8y8/aQGR7/WL2M9rkTd2cf6p/csDiUTkGf8YMFu8Qz/u5MgzY/14LCShXzfL
+	 S+iz36YnA/kIu1RjrOzR22rPPcQRfZE1MyS9B+h9QeHHWR7A1SnsfE8WnlhBCCRxU7
+	 ZcKgKfiihKPxQ==
+From: EuAndreh <[email-masked]>
+To: Seph Gentle <[email-masked]>, ~euandreh/[email-masked]
+Cc: 
+Subject: Re: Making a database
+In-Reply-To: <[email-masked]>
+References: <[email-masked]>
+ <[email-masked]>
+ <[email-masked]>
+Date: Tue, 01 Sep 2020 14:56:23 -0300
+Message-ID: <[email-masked]>
+MIME-Version: 1.0
+Content-Type: text/plain; charset=utf-8
+Content-Transfer-Encoding: quoted-printable
+
+> You kinda gave a good description of them in your post already. They
+> work by requiring all peers to resolve concurrent edits to an object
+> the same way. That could be last-writer-wins (based on time stamps
+> like you said in your article). Or it could store all conflicting
+> versions together to be resolved by the next reader (Eg riak). Or it
+> could do something more clever (like merge changes using automerge or
+> y.js or the like). Git is sort of a CRDT - it=E2=80=99s just it=E2=80=99s=
+ 3 way merge
+> algorithm is a bit of a dogs breakfast as far as these things go.=20
+
+My ignorace is really on the applicability of it. The "store all
+conflicting versions together or to be resolved by the next reader"
+sounds like what I'm thinking about, so maybe I'm trying to re-invent
+CRDTs after all.
+
+I've just bumped the priority of this topic to better understand those
+nuances.=20
+
+From [email-masked] Tue Sep  1 19:30:22 2020
+Received: from a27-132.smtp-out.us-west-2.amazonses.com (a27-132.smtp-out.us-west-2.amazonses.com [54.240.27.132])
+	by mail-b.sr.ht (Postfix) with ESMTPS id 67FA4FF130
+	for <~euandreh/[email-masked]>; Tue,  1 Sep 2020 19:30:21 +0000 (UTC)
+Authentication-Results: mail-b.sr.ht;
+	dkim=pass (1024-bit key) header.d=twinql.com header.i=@twinql.com header.b=HXFRpUpY;
+	dkim=pass (1024-bit key) header.d=amazonses.com header.i=@amazonses.com header.b=Fknn4Oqd
+DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple;
+	s=qvs2k2sd2ivtrflyu2nswmravuqizjml; d=twinql.com; t=1598988619;
+	h=Subject:From:To:Cc:Date:Mime-Version:Content-Type:Content-Transfer-Encoding:In-Reply-To:References:Message-Id;
+	bh=Y4wqQqAQDJ1Sa/E5OhrtvJ+Yk3Uij2laWmG/aD8wLfw=;
+	b=HXFRpUpY+Dt9MJW6Z60km/fTpPfIYak8NKLDvv6UeroaBfvWClO0loOtSYksmT88
+	dWeTwau14008g4YQwDwEY8msngXXDx/9W05ArClEsNXDygcmOYea3eHhqVMcvS/dtuA
+	QYV4fjMoIj9QsGiHAP/QKjampkgK6nHx+Qfhlti4=
+DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple;
+	s=hsbnp7p3ensaochzwyq5wwmceodymuwv; d=amazonses.com; t=1598988619;
+	h=Subject:From:To:Cc:Date:Mime-Version:Content-Type:Content-Transfer-Encoding:In-Reply-To:References:Message-Id:Feedback-ID;
+	bh=Y4wqQqAQDJ1Sa/E5OhrtvJ+Yk3Uij2laWmG/aD8wLfw=;
+	b=Fknn4Oqdb29M5jY4pJV5V9gX2yK3n+WkgnRWmjQl46DdAzcaL4dHoZvIbk8cQxSt
+	v6z1Rdu/S1IvHVSIHLYloebUmPBBfmLzEksz9sh5yCGO4IJhfdIEk0CPgPvNNkXC7as
+	DpyhbHSQzUMFT1EBYwmgHNK/DQMSwe+2Jy7SpfPg=
+Subject: Re: The database I wish I had
+From: =?UTF-8?Q?Richard_Newman?= <[email-masked]>
+To: =?UTF-8?Q?EuAndreh?= <[email-masked]>
+Cc: =?UTF-8?Q?=7Eeuandreh/public-inbox=40lists=2Esr=2Eht?=
+  <~euandreh/[email-masked]>
+Date: Tue, 1 Sep 2020 19:30:19 +0000
+Mime-Version: 1.0
+Content-Type: text/plain; charset=UTF-8
+Content-Transfer-Encoding: quoted-printable
+In-Reply-To: <[email-masked]>
+References: <[email-masked]> 
+ <[email-masked]> 
+ <[email-masked]> 
+ <[email-masked]>
+X-Priority: 3 (Normal)
+X-Mailer: Amazon WorkMail
+Thread-Index: AQHWgHcPJ4udfXYrRaOwhoO7EcsDJQAErSAlAAfQ7+g=
+Thread-Topic: The database I wish I had
+X-Original-Mailer: Apple Mail (2.3445.104.14)
+X-Wm-Sent-Timestamp: 1598988618
+Message-ID: <[email-masked]>
+X-SES-Outgoing: 2020.09.01-54.240.27.132
+Feedback-ID: 1.us-west-2.An468LAV0jCjQDrDLvlZjeAthld7qrhZr+vow8irkvU=:AmazonSES
+
+> Hi there! You wouldn't be suprised if I told I recognize you by name,=0D=
+=0A> would you=3F=0D=0A>=20=0D=0A> Great job on Mentat, and please write =
+more installments on "Thinking=0D=0A> about Syncing" if possible =E2=98=BA=
+=EF=B8=8F=0D=0A=0D=0AOne day I do hope to return to them! Thank you for t=
+he kind words.=0D=0A=0D=0A=0D=0A> Hmm, I'd be interested in learning more=
+ about those under-specified=0D=0A> pitfalls to try to avoid them if poss=
+ible.=20=0D=0A=0D=0AMostly the usual stuff you get when a thing is define=
+d by its implementation:=0D=0A=0D=0Ahttps://github.com/mozilla/mentat/pul=
+l/665=0D=0Ahttps://github.com/mozilla/mentat/issues/412=0D=0A=0D=0AThere =
+are probably more if you poke around the issue tracker.=0D=0A=0D=0A=0D=0A=
+> What do you mean by "chunk-based backing store abstraction"=3F Could yo=
+u=0D=0A> elaborate further=3F=0D=0A=0D=0AI'm familiar with the way Allegr=
+oGraph does storage, and fairly familiar with how Datomic does it.=0D=0A=0D=
+=0AIn short, a datom store like this consists of the log (all datoms, wit=
+h retractions, ordered by ID), and some current state (which Mentat calle=
+d `datoms`), which consists of all current asserted datoms.=0D=0A=0D=0ATh=
+e current state is typically stored in a number of redundant permutations=
+ to support querying. In AllegroGraph (subject, predicate, object, graph,=
+ id), it would maintain multiple indices: `spogi`, `posgi`, etc. Each of =
+these store the same datoms in different orders.=0D=0A=0D=0AIf we have th=
+e data=0D=0A=0D=0A```=0D=0A  :bob foaf:knows :alice=0D=0A  :alice foaf:na=
+me "Alice"=0D=0A  :bob foaf:name "Bob"=0D=0A  :bob foaf:age 26=0D=0A  :al=
+ice foaf:age 30=0D=0A```=0D=0A=0D=0Ait might appear in the `spogi` index =
+as:=0D=0A=0D=0A```=0D=0A  :alice foaf:age 30=0D=0A  :alice foaf:name "Ali=
+ce"=0D=0A  :bob foaf:age 26=0D=0A  :bob foaf:knows :alice=0D=0A  :bob foa=
+f:name "Bob"=0D=0A```=0D=0A=0D=0Aand in the `posgi` index as:=0D=0A=0D=0A=
+```=0D=0A  foaf:age 26 :bob=0D=0A  foaf:age 30 :alice=20=0D=0A  foaf:name=
+ "Alice" :alice=20=0D=0A  foaf:name "Bob" :bob=20=0D=0A  foaf:knows :alic=
+e :bob=20=0D=0A```=0D=0A=0D=0ANaturally these would be using efficient (i=
+nterned) sortable representations of each term, with rows of fixed size, =
+memory mapped on disk.=0D=0A=0D=0AThis allows you to implement query patt=
+ern execution as an index walk.=0D=0A=0D=0A`SELECT =3Fs WHERE =3Fs foaf:n=
+ame 'Alice'`=0D=0A=0D=0Afor example turns into a walk on `posgi`: you fin=
+d the spot that corresponds to `foaf:name`, find the next place which is =
+`Alice`, and then read out the `s`.=0D=0A=0D=0AWhen you insert data, you =
+typically don't want to immediately regenerate all of these indices. Inst=
+ead you accrue new *chunks*, which are just all of the datoms asserted in=
+ some window of time, or within a particular transaction. The query engin=
+e walks all of the relevant chunks and merges the results together. One c=
+hunk might have Bob's name, and one might have Alice's.=0D=0A=0D=0ATypica=
+lly _metaindices_ are used to help the query engine navigate which chunks=
+ to use.=0D=0A=0D=0AAllegroGraph periodically merges chunks together: a f=
+ully merged store is faster to query. Datomic goes the other way: it buil=
+ds new merged chunks and keeps the old ones around, because those old chu=
+nks represent the state of the database at a point in time! This is how y=
+ou efficiently query past states in Datomic: it trades space in your Cass=
+andra cluster for the ability to grab the right chunks for any query.=0D=0A=
+=0D=0AYou can easily imagine that any system that allows you to put a nam=
+e on some region of data can be used to store these chunks, and their met=
+aindices, and the log itself. The only requirement is that the chunk stor=
+e be durable, and ideally replicated and relatively fast to read from.=0D=
+=0A=0D=0A=0D=0A> What do you mean by "identity"=3F As in ":db.unique/valu=
+e" and=0D=0A> ":db.unique/identity"=3F Couldn't you just use UUIDs for=0D=
+=0A> ":db.unique/identity" values=3F=0D=0A=0D=0AIf on Computer A I visit =
+https://aclu.org, and on Computer B I visit https://aclu.org, my two comp=
+uters will each give the same history item a different internal identifie=
+r =E2=80=94 1234 and 4567, say.=0D=0A=0D=0AWhen I sync them, I need the d=
+ata store to know that the visits I made to each page _were visits to the=
+ same thing_.=0D=0A=0D=0AThere is more value to it than that (e.g., Datom=
+ic's "lookup refs"), but it's important to recognize that identity is _do=
+main specific_, and it is often _compound_ and even contextual or conditi=
+onal.=0D=0A=0D=0AA large part of synchronization, and a large part of dat=
+a modeling in general, consists of figuring out exactly what it is you're=
+ trying to record data about, and how to identify it. It's tremendously i=
+mportant in NoSQL systems because there is no foreign key constraint to h=
+elp you, and changing an identifier later is difficult. It's just as impo=
+rtant in syncing systems because these identifiers are the only thing you=
+ have to help you merge data =E2=80=94 to zip two graphs together.=0D=0A=0D=
+=0A=0D=0A> For schema evolution wouldn't the "Grow your schema, and never=
+ break=0D=0A> it"[0] approach be enough=3F=0D=0A=0D=0AIt definitely helps=
+; this is how teams generally try to work with DynamoDB, for example, tho=
+ugh there the schema is implicit rather than explict.=0D=0A=0D=0AHowever,=
+ you inevitably have data migrations to worry about=E2=80=A6 not to menti=
+on that syncing makes this a distributed system, and so you cannot easily=
+ coordinate schema changes in more than one place, and they will race wit=
+h new data!=0D=0A=0D=0AThere's also the argument that coordinating change=
+ in consumers and engineers is sometimes harder than evolving a vocabular=
+y!=0D=0A=0D=0AIt's a complicated topic.=0D=0A=0D=0A=0D=0A> On encryption,=
+ my first approach would be to see if git-remote-gcrypt[1]=0D=0A> functio=
+nality can be translated, and have the local database be ignorant=0D=0A> =
+of all encryption whatsoever. However, I haven't explored it enough to=0D=
+=0A> be able to declare it to be possible, so I would welcome a more matu=
+re=0D=0A> view on the subject.=0D=0A=0D=0AI have been relatively happy wi=
+th SQLCipher, and I think the idea of block-level encryption is a good pl=
+ace to put the abstraction. The thing you lose with any of these mechanis=
+ms is, by definition, the ability for a central entity to help you with c=
+onflict resolution. The best you can get is ordering of opaque chunks and=
+ collision detection.=0D=0A=0D=0A=0D=0A> If sound and robust, what's the =
+downside of detecting conflicts on=0D=0A> individual attributes=3F=0D=0A=0D=
+=0AIt relies on extensive reification to achieve the property that we wan=
+t: that each semantically conflicting write results in a conflicting attr=
+ibute change.=0D=0A=0D=0AAs a contrived example example: notes have both =
+a title and a body.=0D=0A=0D=0AA user has two computers.=0D=0A=0D=0AOn Co=
+mputer A, the user decides to change their first note to something else, =
+updating both the title and body.=0D=0A=0D=0AOn Computer B, they do the s=
+ame thing to a different title and body.=0D=0A=0D=0AWith attribute-based =
+conflict resolution it is possible to end up with a note that has a combi=
+nation of A's title and B's body =E2=80=94 a combination that never exist=
+ed on either computer.=0D=0A=0D=0AThis gets even worse when you treat del=
+etion specially, when new assertions don't conflict but do alter domain s=
+emantics, or when some of the conflicts affect the way the rest of the da=
+ta is interpreted (e.g., when you have an identity property or a lookup r=
+ef).=0D=0A=0D=0AThe only data modeling solution to this case is to introd=
+uce an immutable 'note node', and have the _change_ be to have Note One p=
+oint to a different note node =E2=80=94 one attribute change!=0D=0A=0D=0A=
+In a JSON-ish system like Pouch you'd swap an entire JS object within the=
+ same attribute, rather than having two different attributes.=0D=0A=0D=0A=
+Developers =E2=80=94 at least, non-Clojure developers =E2=80=94 rarely th=
+ink in these terms, and now everywhere else you need to make really caref=
+ul decisions about whether you point to the 'note' or the 'node'. Getting=
+ it wrong means dangling pointers.=0D=0A=0D=0AUsing per-attribute automat=
+ic merging without having an expert ontologist model your domain is a rec=
+ipe for weird edge-case conflict resolution bugs that result in data soup=
+ or data loss.=0D=0A=20=0D=0A=0D=0A> If an Datomic-like entity with 10 at=
+tributes has 4 of them change by=0D=0A> different database instances, but=
+ only 1 change is on the same=0D=0A> attribute, would this type of detect=
+ion offer more granularity instead=3F=0D=0A=0D=0AMy thinking is mostly: y=
+ou need some kind of schema mechanism to be able to recognize which prope=
+rties can be reconciled safely, and you also cannot examine each tuple in=
+dependently in many cases. It's complicated. Something like Dropbox's dat=
+astore conflict resolution (if I remember right) is fine much of the time=
+, but wrong some of the time.=0D=0A=0D=0A=0D=0A> Wouldn't the 5-tuple be =
+the equivalent of a single line of code on git,=0D=0A> and aggregates are=
+ built on top of it=3F=0D=0A=0D=0AEven source code merging isn't this sim=
+ple, no=3F A conceptual operation like "add this line underneath this oth=
+er line" is the thing you're trying to represent, but that line has _cont=
+ext_ =E2=80=94 it might have moved, the file might have been renamed, or =
+someone else already put a line there.=0D=0A=0D=0AYou've probably seen th=
+is with b0rked merges in software version control, where the result of a =
+merge won't compile because of duplicate identifiers thanks to hunks movi=
+ng around=E2=80=A6=0D=0A=0D=0ABut trying to apply text-based merging to s=
+emantic data only works if there are no constraints in your data, or if y=
+ou have fast-forward merges =E2=80=94 merging can violate uniqueness prop=
+erties in all kinds of ways.=20=0D=0A
+
+From [email-masked] Tue Sep  1 22:13:15 2020
+Received: from wout4-smtp.messagingengine.com (wout4-smtp.messagingengine.com [64.147.123.20])
+	by mail-b.sr.ht (Postfix) with ESMTPS id 53A0EFF124
+	for <~euandreh/[email-masked]>; Tue,  1 Sep 2020 22:13:14 +0000 (UTC)
+Authentication-Results: mail-b.sr.ht;
+	dkim=pass (1024-bit key) header.d=josephg.com header.i=@josephg.com header.b=2Kj6nHib;
+	dkim=fail reason="key not found in DNS" (0-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=eH2pUNJ6
+Received: from compute1.internal (compute1.nyi.internal [10.202.2.41])
+	by mailout.west.internal (Postfix) with ESMTP id CB8F377E;
+	Tue,  1 Sep 2020 18:13:12 -0400 (EDT)
+Received: from imap35 ([10.202.2.85])
+  by compute1.internal (MEProxy); Tue, 01 Sep 2020 18:13:12 -0400
+DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=josephg.com; h=
+	mime-version:message-id:in-reply-to:references:date:from:to
+	:subject:content-type:content-transfer-encoding; s=mesmtp; bh=Q9
+	PUfqkL153ICDLSRfraWMnWRXOclTHibxQiOyAV3lA=; b=2Kj6nHibSzEldHCTUN
+	lgQhb0piuopatnU3wV94ozxZ95sUW7zyCnSE/s9gqFgTgYg0o2ZeqMffp92mRZis
+	frDqWjWNhxv4WN78vO8dcwA8U/a+Uwp4cfxUyjuyY/zw7NHHM8LORDQCfembOUFj
+	O6a5SW/fQRgU9XJIemyvxSDlM=
+DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
+	messagingengine.com; h=content-transfer-encoding:content-type
+	:date:from:in-reply-to:message-id:mime-version:references
+	:subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender
+	:x-sasl-enc; s=fm3; bh=Q9PUfqkL153ICDLSRfraWMnWRXOclTHibxQiOyAV3
+	lA=; b=eH2pUNJ6n7/Ng/S9WTH+X0QePTCyzr5SjXdLgG3eN4a4yktWOtaMr44AU
+	nW7SE782cX3xP2TEyvVSJ58tnx6r+6M5Jzauja+b2OJiTtnrJyWfIZ6iK/URp/y2
+	v++7NoAfWmyXOfUQAiLbK92FCRrRkR8nGx0EksjMInRiDUozhHcg1mRYRipvULUx
+	UsaQ0gtdGNDm76c6VwZU7YwBxssaeVQNP8ROlvm/y/KC3gpLYILLdNJPcI/r1846
+	FrhxPQLgAQIe+4UV6NYtjNIRbFNPjdIWHZvWZ4LdMRPQ4yFdbpD6aHhKa+wLBPyB
+	sIgGBCGwzMhfTDSRZQw8DrWpcx21w==
+X-ME-Sender: <xms:dsdOX2zlGD9BWCFDAMf7fpxK8tc8KYm5XDsDnDguZM2Sy8xflOH9kA>
+    <xme:dsdOXyTTPqu0-Pa3UYN2l_7sXVE04FX8AClSMYgSkAoeLeTspezWpQUlQYsja54Jc
+    wrAfUl58HrBhQRp6w>
+X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduiedrudefkedgtdeiucetufdoteggodetrfdotf
+    fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen
+    uceurghilhhouhhtmecufedttdenucenucfjughrpefofgggkfgjfhffhffvufgtgfesth
+    hqredtreerjeenucfhrhhomhepfdfuvghphhcuifgvnhhtlhgvfdcuoehmvgesjhhoshgv
+    phhhghdrtghomheqnecuggftrfgrthhtvghrnhepteeugfeufeeludekieduiefggfehhf
+    eiffelfeetteejheehveelteffhefffeeunecuffhomhgrihhnpehklhgvphhpmhgrnhhn
+    rdgtohhmnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomh
+    epmhgvsehjohhsvghphhhgrdgtohhm
+X-ME-Proxy: <xmx:dsdOX4WWAjDIjt91xpvOW8wGgg-eiGlTvG2yYFGPVBe2qPkuU169eg>
+    <xmx:dsdOX8gA1SFXeEhDY-io8js7YHP8z7rKmyRtM81cBA0MFgLBRHhtow>
+    <xmx:dsdOX4DJCCv4jHxex2u1y3w6cvLGKhAfSjHmtdhvaRTHlJcfSu4J5w>
+    <xmx:eMdOX7M12Nj2n8pxrtLuGPyovCEq1wtzixBdJWACKU6e8qYcMiQKKg>
+Received: by mailuser.nyi.internal (Postfix, from userid 501)
+	id 762F414C00B3; Tue,  1 Sep 2020 18:13:10 -0400 (EDT)
+X-Mailer: MessagingEngine.com Webmail Interface
+User-Agent: Cyrus-JMAP/3.3.0-248-gcd102cb-fm-20200901.001-gcd102cb9
+Mime-Version: 1.0
+Message-Id: <[email-masked]>
+In-Reply-To: <[email-masked]>
+References: <[email-masked]>
+ <[email-masked]>
+ <[email-masked]>
+ <[email-masked]>
+Date: Wed, 02 Sep 2020 08:12:45 +1000
+From: "Seph Gentle" <[email-masked]>
+To: EuAndreh <[email-masked]>, ~euandreh/[email-masked]
+Subject: Re: Making a database
+Content-Type: text/plain;charset=utf-8
+Content-Transfer-Encoding: quoted-printable
+
+On Wed, Sep 2, 2020, at 3:56 AM, EuAndreh wrote:
+> > You kinda gave a good description of them in your post already. They=
+
+> > work by requiring all peers to resolve concurrent edits to an object=
+
+> > the same way. That could be last-writer-wins (based on time stamps
+> > like you said in your article). Or it could store all conflicting
+> > versions together to be resolved by the next reader (Eg riak). Or it=
+
+> > could do something more clever (like merge changes using automerge o=
+r
+> > y.js or the like). Git is sort of a CRDT - it=E2=80=99s just it=E2=80=
+=99s 3 way merge
+> > algorithm is a bit of a dogs breakfast as far as these things go.=20=
+
+>=20
+> My ignorace is really on the applicability of it. The "store all
+> conflicting versions together or to be resolved by the next reader"
+> sounds like what I'm thinking about, so maybe I'm trying to re-invent
+> CRDTs after all.
+>=20
+> I've just bumped the priority of this topic to better understand those=
+
+> nuances.=20
+
+This is worth watching if you haven=E2=80=99t seen it. Martin is at the =
+forefront of the field - but this gives you a sense of where the researc=
+h tech is at:
+
+https://martin.kleppmann.com/2020/07/06/crdt-hard-parts-hydra.html
+
+From [email-masked] Wed Sep  2 00:00:16 2020
+Received: from box.euandre.org (box.euandre.org [46.101.160.115])
+	by mail-b.sr.ht (Postfix) with ESMTPS id E53C0FF14B
+	for <~euandreh/[email-masked]>; Wed,  2 Sep 2020 00:00:15 +0000 (UTC)
+Authentication-Results: mail-b.sr.ht;
+	dkim=pass (2048-bit key) header.d=euandre.org header.i=@euandre.org header.b=Ts/gQOWS
+Received: from authenticated-user (box.euandre.org [46.101.160.115])
+	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
+	(No client certificate requested)
+	by box.euandre.org (Postfix) with ESMTPSA id E5F6DFC052;
+	Tue,  1 Sep 2020 21:00:11 -0300 (-03)
+DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=euandre.org; s=mail;
+	t=1599004813; bh=GS02iBK+iDwoGCfVqsjbqMenu+Wpn9OO6GpSXGHcpsY=;
+	h=From:To:Cc:Subject:In-Reply-To:References:Date:From;
+	b=Ts/gQOWSiM5Vq/pHxaNuXKM4mXoy3VW9rD5mv/uLMGHYQcslItj1TNcvxINLzU/yj
+	 ZaBmqtENnSd9AXap4PHxsFz48JlDI5oYcn98HE9nLsAZrD7/9GkhJMNjRDsf0ODAl9
+	 W7do6QS8LRAZo0RhTguAzgoTJXf/IxuRqykG9Nm8imNuLDN4T04EbGvRw4ibN4UhlT
+	 DED1G5O4Ul96BT6mY/im9tBQPGGNURLLcNcUkqG4fpddJrg58/yH9JOPMhnNhatEiV
+	 39+m3xyL2rM8HSBfKun87TXjE6CD4/kQ0U6OpSB9ej/4zdNJ7Ei9K5NSuF78KRvkWJ
+	 qHKRHqP/84rQw==
+From: EuAndreh <[email-masked]>
+To: Seph Gentle <[email-masked]>, ~euandreh/[email-masked]
+Cc: 
+Subject: Re: Making a database
+In-Reply-To: <[email-masked]>
+References: <[email-masked]>
+ <[email-masked]>
+ <[email-masked]>
+ <[email-masked]>
+ <[email-masked]>
+Date: Tue, 01 Sep 2020 20:18:08 -0300
+Message-ID: <[email-masked]>
+MIME-Version: 1.0
+Content-Type: text/plain; charset=utf-8
+Content-Transfer-Encoding: quoted-printable
+
+"Seph Gentle" <[email-masked]> writes:
+
+> This is worth watching if you haven=E2=80=99t seen it. Martin is at the f=
+orefront of the field - but this gives you a sense of where the research te=
+ch is at:
+>
+> https://martin.kleppmann.com/2020/07/06/crdt-hard-parts-hydra.html
+
+Coincidentally, that's the  third item on my list =F0=9F=98=AC
+
+From [email-masked] Wed Sep  2 01:35:17 2020
+Received: from mx3.rhyason.org (mx3.rhyason.org [104.245.39.21])
+	by mail-b.sr.ht (Postfix) with ESMTPS id 6C175FF0E7
+	for <~euandreh/[email-masked]>; Wed,  2 Sep 2020 01:35:17 +0000 (UTC)
+Authentication-Results: mail-b.sr.ht;
+	dkim=pass (1024-bit key) header.d=rhyason.org header.i=@rhyason.org header.b=weEVa7Zv
+Received: from rhyason.org (d173-183-32-226.bchsia.telus.net [173.183.32.226])
+	by mx3.rhyason.org (8.15.2/8.15.2) with ESMTP id 0821ZI7G073010
+	for <~euandreh/[email-masked]>; Tue, 1 Sep 2020 21:35:19 -0400 (EDT)
+	(envelope-from [email-masked])
+Received: from rhyason.org (localhost [127.0.0.1]);
+	by rhyason.org (OpenSMTPD) with ESMTP id 4aa4e076;
+	for <~euandreh/[email-masked]>;
+	Tue, 1 Sep 2020 19:35:08 -0600 (MDT)
+DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=rhyason.org; h=from
+	:content-type:content-transfer-encoding:mime-version:subject
+	:message-id:date:to; s=selector1; bh=61RPMANfo/dmUC5BfkkO8NNOMrg
+	=; b=weEVa7ZvWqlxTjSrAxMCoiPy3GBkyNONviKR6lPNlGM3/eV5+0g3KhjKIKs
+	g5+xXXB1G69y1xvNyo5hpOYY4Gk51OeGCQ/QxtE3tD2IgTxYjWlsB5SuNTrWI6Kd
+	Onz8i5Fdan7ZigJ1ambirCGyI/IWPb8Q8gG7+i76Wb2vj3W8=
+DomainKey-Signature: a=rsa-sha1; c=nofws; d=rhyason.org; h=from
+	:content-type:content-transfer-encoding:mime-version:subject
+	:message-id:date:to; q=dns; s=selector1; b=mpt0BK56C+LRHqEw8Bx8M
+	y7nJWXZqD5kxPw4J7wh3mh8VcMYiBuA/2BpbrT7wYH9IxpMGFjlSP/lIZIHUPFTt
+	cpbgqiL6CrW1zvP4CJ+6ygSQkfnsEGekcmEgLGa/E49lcKxA/y7Ep1G7maJJIZxN
+	3OqgdtRLK1MiajxwW/MvBA=
+Received: from [10.0.88.117] (10.0.88.117 [10.0.88.117]);
+	by rhyason.org (OpenSMTPD) with ESMTPSA id d7674b26;
+	TLS version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO;
+	for <~euandreh/[email-masked]>;
+	Tue, 1 Sep 2020 19:35:08 -0600 (MDT)
+From: Jeff Rhyason <[email-masked]>
+Content-Type: text/plain;
+	charset=us-ascii
+Content-Transfer-Encoding: quoted-printable
+Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.1\))
+Subject: building MSTs->CRDTs-> Re: The database I wish I had
+Message-Id: <[email-masked]>
+Date: Tue, 1 Sep 2020 18:35:08 -0700
+To: ~euandreh/[email-masked]
+X-Mailer: Apple Mail (2.3608.120.23.2.1)
+
+It's amazing to see this discussion. Let's keep it going!
+
+My friend and I started Indelible[1] (now defunct) for a lot of the same =
+reasons you want to build a datastore -- there aren't enough options for =
+a datastore that's immutable, relational, highly-vaailable, and =
+private.. We're still super-passionate about the ideas, love Datomic but =
+want something a bit different, decided not to commercialize it and =
+start more simply with just the data structure. We don't want a huge =
+thing, just a library, to keep coordination down; the complexity doesn't =
+spread out of the client.
+
+Where we are at now is we have a great versioned immutable B-Tree-like =
+structure that is the foundation for the verioned and diffable database =
+we want.  FWIW The Go implementation of the tree is at =
+https://github.com/jrhy/mast .  We have plans to do a CRDT based on it, =
+possibly add an SQL layer, but frankly, having a strongly-consistent, =
+verioned + diffable tree, is pretty good for now. We'd love to =
+collaborate!
+
+
+
+[1] Indelible API Reference =
+http://htmlpreview.github.io/?https://github.com/indeliblesystems/indelibl=
+e/blob/master/doc/000-API-ref.html
+
+
+From [email-masked] Wed Sep  2 08:00:14 2020
+Received: from box.euandre.org (box.euandre.org [46.101.160.115])
+	by mail-b.sr.ht (Postfix) with ESMTPS id E2229FF148
+	for <~euandreh/[email-masked]>; Wed,  2 Sep 2020 08:00:13 +0000 (UTC)
+Authentication-Results: mail-b.sr.ht;
+	dkim=pass (2048-bit key) header.d=euandre.org header.i=@euandre.org header.b=Gv++l+b4
+Received: from authenticated-user (box.euandre.org [46.101.160.115])
+	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
+	(No client certificate requested)
+	by box.euandre.org (Postfix) with ESMTPSA id BBE37FC052;
+	Wed,  2 Sep 2020 05:00:10 -0300 (-03)
+DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=euandre.org; s=mail;
+	t=1599033612; bh=L0rF3hNESOLacI6plS3F0G5McLpB7MuPNAKgItTuBzg=;
+	h=From:To:Cc:Cc:Subject:In-Reply-To:References:Date:From;
+	b=Gv++l+b4mbVug6D7ywconYDZ/EzLKxn27zY5F1FhWm2A97mClGgf20SVzjqrxZNOE
+	 LHVxQX1qwB7I/MzwWDf0ACHFQsOu0Y2VgQyffVE0BHInz/TPbXlv9Qbfe1L9yRFvD7
+	 Z+3BoX5//m/1Na1ZKeuP1KT5neo8dF75wyyUhG0Zh/hFEDa1ILdR8wrnovtouqmwxA
+	 JoMH2EzjLJ8ad2pox2VU0BskunOUvIX/CRXsrnXDCxybseFECE7h2jghegme7MlLae
+	 /kvzmmg6Ow1z/lzvftKWGJRPb47MvQYSFIbnLb2kkkhTb6evJ5eXPO2nPY6/vJ8t6c
+	 jhgzwfN/DzLeA==
+From: EuAndreh <[email-masked]>
+To: Richard Newman <[email-masked]>
+Cc: "~euandreh/[email-masked]" <~euandreh/[email-masked]>
+Cc: 
+Subject: Re: The database I wish I had
+In-Reply-To: <[email-masked]>
+References: <[email-masked]>
+ <[email-masked]>
+ <[email-masked]>
+ <[email-masked]>
+ <[email-masked]>
+Date: Wed, 02 Sep 2020 04:30:43 -0300
+Message-ID: <[email-masked]>
+MIME-Version: 1.0
+Content-Type: text/plain; charset=utf-8
+Content-Transfer-Encoding: quoted-printable
+
+This is some rich discussion, thanks for engaging!
+
+Richard Newman <[email-masked]> writes:
+
+> Mostly the usual stuff you get when a thing is defined by its
+> implementation:
+>
+> https://github.com/mozilla/mentat/pull/665
+> https://github.com/mozilla/mentat/issues/412
+>
+> There are probably more if you poke around the issue tracker.
+
+Thanks for the references.
+
+> I'm familiar with the way AllegroGraph does storage, and fairly
+> familiar with how Datomic does it.
+
+I've used Common Lisp a lot before, and I heard a lot about
+AllegroGraph, but I've never worked with it. Would you recommed it as a
+place to take inspirations from?
+
+> In short, a datom store like this consists of the log (all datoms,
+> with retractions, ordered by ID), and some current state (which Mentat
+> called `datoms`), which consists of all current asserted datoms.
+>
+> The current state is typically stored in a number of redundant
+> permutations to support querying. In AllegroGraph (subject, predicate,
+> object, graph, id), it would maintain multiple indices: `spogi`,
+> `posgi`, etc. Each of these store the same datoms in different orders.
+>
+> If we have the data
+>
+> ```
+>   :bob foaf:knows :alice
+>   :alice foaf:name "Alice"
+>   :bob foaf:name "Bob"
+>   :bob foaf:age 26
+>   :alice foaf:age 30
+> ```
+>
+> it might appear in the `spogi` index as:
+>
+> ```
+>   :alice foaf:age 30
+>   :alice foaf:name "Alice"
+>   :bob foaf:age 26
+>   :bob foaf:knows :alice
+>   :bob foaf:name "Bob"
+> ```
+>
+> and in the `posgi` index as:
+>
+> ```
+>   foaf:age 26 :bob
+>   foaf:age 30 :alice
+>   foaf:name "Alice" :alice
+>   foaf:name "Bob" :bob
+>   foaf:knows :alice :bob
+> ```
+
+These look just like the EAVT and AVET indexes in Datomic.
+
+> Naturally these would be using efficient (interned) sortable
+> representations of each term, with rows of fixed size, memory mapped
+> on disk.
+>
+> This allows you to implement query pattern execution as an index walk.
+>
+> `SELECT ?s WHERE ?s foaf:name 'Alice'`
+>
+> for example turns into a walk on `posgi`: you find the spot that
+> corresponds to `foaf:name`, find the next place which is `Alice`, and
+> then read out the `s`.
+>
+> When you insert data, you typically don't want to immediately
+> regenerate all of these indices. Instead you accrue new *chunks*,
+> which are just all of the datoms asserted in some window of time, or
+> within a particular transaction. The query engine walks all of the
+> relevant chunks and merges the results together. One chunk might have
+> Bob's name, and one might have Alice's.
+>
+> Typically _metaindices_ are used to help the query engine navigate
+> which chunks to use.
+>
+> AllegroGraph periodically merges chunks together: a fully merged store
+> is faster to query. Datomic goes the other way: it builds new merged
+> chunks and keeps the old ones around, because those old chunks
+> represent the state of the database at a point in time! This is how
+> you efficiently query past states in Datomic: it trades space in your
+> Cassandra cluster for the ability to grab the right chunks for any
+> query.
+>
+> You can easily imagine that any system that allows you to put a name
+> on some region of data can be used to store these chunks, and their
+> metaindices, and the log itself. The only requirement is that the
+> chunk store be durable, and ideally replicated and relatively fast to
+> read from.
+
+The most intriguing part of this for me is trying to mimic Datomic's
+implementation of those merged chunks, specifically how Rich Hickey
+describes it as an equivalent of persistent data structures used for
+storage, with sharing of previous versions, etc. I believe he calls it
+"durable persistent" datastructure, like making things on disk to be
+a bit similar on how they are on memory.
+
+In fact, that's exactly where I'm focused at right now: deepening my
+grasp on persistent data structures to enable me to navigate this
+subject with more fluidity, and try to implement such durable persistent
+data structures.
+
+The merging of the AllegroGraph that you describe sounds like the flat
+files merge of indexes that RH describes, and contrasts with his
+proposal. I'll try to find this quote, and re-read the paper he mentions
+on the talk (IIRC it's the BigTable one).
+
+I was considering having all indexes be derived from the raw data
+transported between instances, but it might get computationally
+expensive, and making those indexes replicable sounds like a good idea.
+
+> If on Computer A I visit https://aclu.org, and on Computer B I visit
+> https://aclu.org, my two computers will each give the same history
+> item a different internal identifier =E2=80=94 1234 and 4567, say.
+>
+> When I sync them, I need the data store to know that the visits I made
+> to each page _were visits to the same thing_.
+
+Isn't that a domain-specifc problem of identity?
+
+I would solve this by allowing those two entries to cooexist without any
+conflict, and do aggregation queries to answers those questions,
+something like (pseudo-SQL):
+
+SELECT DISTINCT(url) FROM urls;
+
+...to show the all the visited URLs, and:
+
+SELECT SUM(1) from URLS GROUP BY url
+
+...to show the visit count for each URL.
+
+WDYT?
+
+> It definitely helps; this is how teams generally try to work with
+> DynamoDB, for example, though there the schema is implicit rather than
+> explict.
+>
+> However, you inevitably have data migrations to worry about=E2=80=A6 not =
+to
+> mention that syncing makes this a distributed system, and so you
+> cannot easily coordinate schema changes in more than one place, and
+> they will race with new data!
+>
+> There's also the argument that coordinating change in consumers and
+> engineers is sometimes harder than evolving a vocabulary!
+>
+> It's a complicated topic.
+
+On migrations:
+
+I used to share your view on needing a migration scape hatch, but I
+don't anymore. On the company I work those kind of database migrations
+never really occur. People either create a new attribute, copy data from
+a database to another, process it using ETL, compute things on the fly
+with functions, and many other strategies. We've already had to comply
+with excisions in production, but never on schema, only on data itself.
+
+So I don't think that's an actual issue, and I would insist that the
+"Grow your schema" approach actually does work at scale, on the long
+run, for multiple scenarios.=20
+
+> I have been relatively happy with SQLCipher, and I think the idea of
+> block-level encryption is a good place to put the abstraction. The
+> thing you lose with any of these mechanisms is, by definition, the
+> ability for a central entity to help you with conflict resolution. The
+> best you can get is ordering of opaque chunks and collision detection.
+
+I believe that's not what I meant by comparing it to git-remote-gcrypt,
+but maybe I misunderstood your point.
+
+Here's a code snippet to try to highlight what I was referring to:
+
+--8<---------------cut here---------------start------------->8---
+$ pushd $(mktemp -d)
+/tmp/tmp.iapFd5iBCr ~/
+$ mkdir repo-1
+$ cd repo-1/
+$ git init
+$ echo 'a' > file
+$ git add file
+$ git commit -m commit-a
+[master (commit racine) c4112e4] commit-a
+ 1 file changed, 1 insertion(+)
+ create mode 100644 file
+$ git init --bare ../remote/
+D=C3=A9p=C3=B4t Git vide initialis=C3=A9 dans /tmp/tmp.iapFd5iBCr/remote/
+$ git remote add origin gcrypt::../remote
+$ git push origin
+gcrypt: Repository not found: ../remote
+gcrypt: Setting up new repository
+gcrypt: Remote ID is :id:KjDStP/7m18wJagDO0/o
+=C3=89num=C3=A9ration des objets: 3, fait.
+D=C3=A9compte des objets: 100% (3/3), fait.
+Total 3 (delta 0), r=C3=A9utilis=C3=A9s 0 (delta 0), r=C3=A9utilis=C3=A9s d=
+u pack 0
+gcrypt: Encrypting to: --throw-keyids --default-recipient-self
+gcrypt: Requesting manifest signature
+To gcrypt::../remote
+ * [new branch]      master -> master
+$ cd ..
+$ git clone gcrypt::remote repo-2
+Clonage dans 'repo-2'...
+gcrypt: Decrypting manifest
+gpg: selecting card failed: No such device
+gpg: Signature faite le mer. 02 sept. 2020 03:43:12 -03
+gpg:                avec la clef RSA 5BDAE9B8B2F6C6BCBB0D6CE581F90EC3CD3560=
+60
+gpg: Bonne signature de =C2=AB=C2=A0EuAndreh <[email-masked]>=C2=A0=C2=BB [=
+ultime]
+gcrypt: Remote ID is :id:KjDStP/7m18wJagDO0/o
+R=C3=A9ception d'objets: 100% (3/3), fait.
+$ cd repo-2/
+$ echo 'b' > file
+$ git add file
+$ git commit -m commit-b
+[master a5f5f0b] commit-b
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+$ git remote set-url origin gcrypt::../remote
+$ git push origin
+gcrypt: Decrypting manifest
+gpg: selecting card failed: No such device
+gpg: Signature faite le mer. 02 sept. 2020 03:43:12 -03
+gpg:                avec la clef RSA 5BDAE9B8B2F6C6BCBB0D6CE581F90EC3CD3560=
+60
+gpg: Bonne signature de =C2=AB=C2=A0EuAndreh <[email-masked]>=C2=A0=C2=BB [=
+ultime]
+=C3=89num=C3=A9ration des objets: 3, fait.
+D=C3=A9compte des objets: 100% (3/3), fait.
+Total 3 (delta 0), r=C3=A9utilis=C3=A9s 0 (delta 0), r=C3=A9utilis=C3=A9s d=
+u pack 0
+gcrypt: Encrypting to: --throw-keyids --default-recipient-self
+gcrypt: Requesting manifest signature
+To gcrypt::../remote
+   c4112e4..a5f5f0b  master -> master
+$ cd ../repo-1/
+$ echo 'c' > file
+$ git add file
+$ git commit -m commit-c
+[master 3d71137] commit-c
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+--8<---------------cut here---------------end--------------->8---
+
+At this point, repo-1 has changes that conflict with repo-2, and remote
+doesn't know about it, but we can resolve it locally:
+
+--8<---------------cut here---------------start------------->8---
+$ git pull origin master
+gcrypt: Decrypting manifest
+gpg: selecting card failed: No such device
+gpg: Signature faite le mer. 02 sept. 2020 03:44:34 -03
+gpg:                avec la clef RSA 5BDAE9B8B2F6C6BCBB0D6CE581F90EC3CD3560=
+60
+gpg: Bonne signature de =C2=AB=C2=A0EuAndreh <[email-masked]>=C2=A0=C2=BB [=
+ultime]
+Depuis gcrypt::../remote
+ * branch            master     -> FETCH_HEAD
+Fusion automatique de file
+CONFLIT (contenu) : Conflit de fusion dans file
+La fusion automatique a =C3=A9chou=C3=A9 ; r=C3=A9glez les conflits et vali=
+dez le r=C3=A9sultat.
+$ cat file
+<<<<<<< HEAD
+c
+=3D=3D=3D=3D=3D=3D=3D
+b
+>>>>>>> a5f5f0b6c780a7ab3f7e28f373b06fa9a10c443f
+--8<---------------cut here---------------end--------------->8---
+
+This is just the general behaviour that I'm aiming towards, but that is
+really just a long shot, I still need to reasearch more on it.
+
+At a glance I can't really tell if that is what you mean when citing
+SQLCipher.=20
+
+>> If sound and robust, what's the downside of detecting conflicts on
+>> individual attributes?
+>
+> It relies on extensive reification to achieve the property that we
+> want: that each semantically conflicting write results in a
+> conflicting attribute change.=20
+>
+> As a contrived example example: notes have both a title and a body.=20
+>
+> A user has two computers.
+>
+> On Computer A, the user decides to change their first note to
+> something else, updating both the title and body.=20
+>
+> On Computer B, they do the same thing to a different title and body.=20
+>
+> With attribute-based conflict resolution it is possible to end up with
+> a note that has a combination of A's title and B's body =E2=80=94 a
+> combination that never existed on either computer.=20
+>
+> This gets even worse when you treat deletion specially, when new
+> assertions don't conflict but do alter domain semantics, or when some
+> of the conflicts affect the way the rest of the data is interpreted
+> (e.g., when you have an identity property or a lookup ref).=20
+>=20
+> The only data modeling solution to this case is to introduce an
+> immutable 'note node', and have the _change_ be to have Note One point
+> to a different note node =E2=80=94 one attribute change!=20
+>=20
+> In a JSON-ish system like Pouch you'd swap an entire JS object within
+> the same attribute, rather than having two different attributes.=20
+>
+> Developers =E2=80=94 at least, non-Clojure developers =E2=80=94 rarely th=
+ink in these
+> terms, and now everywhere else you need to make really careful
+> decisions about whether you point to the 'note' or the 'node'. Getting
+> it wrong means dangling pointers.=20
+>
+> Using per-attribute automatic merging without having an expert
+> ontologist model your domain is a recipe for weird edge-case conflict
+> resolution bugs that result in data soup or data loss.=20
+
+TBH I'm not aiming for automatic conflict resolution.
+
+Why not use the granularity of attributes to detect conflict, but not to
+resolve them?
+
+Let me add to the scenario you presented: notes have a title, a body and
+an author.
+
+A user has two computers.
+
+On Computer A, the user decides to change their first note to
+something else, updating both the title and body.=20
+
+On Computer B, they do the same thing to a different title and body.=20
+
+On both cases, the author stays the same.
+
+When the DB instance on Computer A gets data from Computer B, the query:
+
+SELECT * FROM notes WHERE id =3D 1234;
+
+would return both, and some metadata saying that it has a conflict. Now
+the programmer needs to decide how to deal with these: use timestamps,
+pick a winning device, sort by revision ID and pick the first, show them
+on a UI to the user to resolve, etc.
+
+Either way, the DB knows that the note has the title and body attributes
+in conflict, but the author is fine. The aggregate entity itself isn't
+in conflict, but some of it's attributes.
+
+The situation that you describe where one would end up with a note that
+is a combination of A's title and B's body would only happen if the DB
+tried to solve things "automagically".
+
+However, I agree on the costly constant reification being required, and
+for long disconnected histories to be merged together this might be
+expensive.=20
+
+>> If an Datomic-like entity with 10 attributes has 4 of them change by
+>> different database instances, but only 1 change is on the same
+>> attribute, would this type of detection offer more granularity instead?
+>
+> My thinking is mostly: you need some kind of schema mechanism to be
+> able to recognize which properties can be reconciled safely, and you
+> also cannot examine each tuple independently in many cases. It's
+> complicated. Something like Dropbox's datastore conflict resolution
+> (if I remember right) is fine much of the time, but wrong some of the
+> time.=20
+
+Couldn't you leverage the schema for that? If the said conflicting
+attribute has:
+
+- cardinality one and both DBs changed it, we have a conflict;
+- cardinality many and both changed the same attribute, we have a
+  conflict;
+- cardinality may and Computer A changed an entry and Computer B changed
+  another one, we don't have a conflict.
+
+and so on.
+
+I wouldn't build it with an exaustive search, but just leverage the
+schema to make those decisions.
+
+I don't know Dropbox's datastore, is this[0] the one you're refferring
+to?=20
+
+[0]: https://dropbox.tech/developers/how-the-datastore-api-handles-conflict=
+s-part-1-basics-of-offline-conflict-handling
+
+> Even source code merging isn't this simple, no? A conceptual operation
+> like "add this line underneath this other line" is the thing you're
+> trying to represent, but that line has _context_ =E2=80=94 it might have
+> moved, the file might have been renamed, or someone else already put a
+> line there.=20
+>
+> You've probably seen this with b0rked merges in software version
+> control, where the result of a merge won't compile because of
+> duplicate identifiers thanks to hunks moving around=E2=80=A6=20
+>
+> But trying to apply text-based merging to semantic data only works if
+> there are no constraints in your data, or if you have fast-forward
+> merges =E2=80=94 merging can violate uniqueness properties in all kinds of
+> ways.=20
+
+Yep, that was an oversimplification.
+
+<hr />
+
+Again, thanks for engaging, this discussion feels like a golden nugget.
+
+From [email-masked] Wed Sep  2 12:48:41 2020
+Received: from mout-p-201.mailbox.org (mout-p-201.mailbox.org [80.241.56.171])
+	by mail-b.sr.ht (Postfix) with ESMTPS id E843EFF14B
+	for <~euandreh/[email-masked]>; Wed,  2 Sep 2020 12:48:40 +0000 (UTC)
+Received: from smtp1.mailbox.org (smtp1.mailbox.org [80.241.60.240])
+	(using TLSv1.2 with cipher ECDHE-RSA-CHACHA20-POLY1305 (256/256 bits))
+	(No client certificate requested)
+	by mout-p-201.mailbox.org (Postfix) with ESMTPS id 4BhNzC5xW2zQlX2;
+	Wed,  2 Sep 2020 14:48:39 +0200 (CEST)
+X-Virus-Scanned: amavisd-new at heinlein-support.de
+Received: from smtp1.mailbox.org ([80.241.60.240])
+	by spamfilter03.heinlein-hosting.de (spamfilter03.heinlein-hosting.de [80.241.56.117]) (amavisd-new, port 10030)
+	with ESMTP id 3fpZQfKdp0ux; Wed,  2 Sep 2020 14:48:36 +0200 (CEST)
+Subject: Re: The database I wish I had
+To: EuAndreh <[email-masked]>, ~euandreh/[email-masked]
+References: <[email-masked]>
+ <[email-masked]>
+From: =?UTF-8?Q?Jonas_Sch=c3=bcrmann?= <[email-masked]>
+Message-ID: <[email-masked]>
+Date: Wed, 2 Sep 2020 14:48:35 +0200
+MIME-Version: 1.0
+In-Reply-To: <[email-masked]>
+Content-Type: text/plain; charset=utf-8; format=flowed
+Content-Language: en-US
+Content-Transfer-Encoding: 7bit
+X-MBO-SPAM-Probability: 
+X-Rspamd-Score: -0.85 / 15.00 / 15.00
+X-Rspamd-Queue-Id: 6F42037F
+X-Rspamd-UID: dfaa19
+
+> Hmmm, I haven't heard this term before. Are these [0] the complete 
+> categories that you mean?
+> 
+> [0]: https://en.wikipedia.org/wiki/Complete_category
+
+Yes, exactly. The idea is pretty abstract and while I know a little bit
+about category theory I'm by no means an expert on the subject. This
+blog post [1] is a very down-to-earth, practical explanation how the
+concept of (co)completed categories can be applied to conflict
+representation in version control systems.
+
+[1] https://jneem.github.io/merging/
+
+From [email-masked] Wed Sep  2 16:14:41 2020
+Received: from a58-46.smtp-out.us-west-2.amazonses.com (a58-46.smtp-out.us-west-2.amazonses.com [54.240.58.46])
+	by mail-b.sr.ht (Postfix) with ESMTPS id AF4DCFF12F
+	for <~euandreh/[email-masked]>; Wed,  2 Sep 2020 16:14:40 +0000 (UTC)
+Authentication-Results: mail-b.sr.ht;
+	dkim=pass (1024-bit key) header.d=twinql.com header.i=@twinql.com header.b=grgVRwHH;
+	dkim=pass (1024-bit key) header.d=amazonses.com header.i=@amazonses.com header.b=SGue23U5
+DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple;
+	s=qvs2k2sd2ivtrflyu2nswmravuqizjml; d=twinql.com; t=1599063278;
+	h=Subject:From:To:Cc:Date:Mime-Version:Content-Type:Content-Transfer-Encoding:In-Reply-To:References:Message-Id;
+	bh=UZ3b7SzbsIcKi0Si4ZISwM274bq+ODQFNXqhUN8KRVo=;
+	b=grgVRwHHOiZAo7asJgQxtXN5anM8LDtht/oVf7XVjh3IJNFPui4FSOPJK+Eh4Xdf
+	MniQ9lndbRWU+3IR3fr3G6NKqYNdfpJb74/j/LvjnnrwZqxQQJFt99pQmwAKADGJAyG
+	ZIwzp6kjG9pz3Uk1Bh+kRa2mHBIMmgfenRoqjiAI=
+DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple;
+	s=hsbnp7p3ensaochzwyq5wwmceodymuwv; d=amazonses.com; t=1599063278;
+	h=Subject:From:To:Cc:Date:Mime-Version:Content-Type:Content-Transfer-Encoding:In-Reply-To:References:Message-Id:Feedback-ID;
+	bh=UZ3b7SzbsIcKi0Si4ZISwM274bq+ODQFNXqhUN8KRVo=;
+	b=SGue23U5gqkhyLMKyKwkBGrqzxh9Ds7ua97mgDIB1BcWDmf58EDy2KhBrlkOtn0y
+	IP8gxNfgYZDYdJFvPt8GmlkDg8nQNYW29iWNcf8ppMHCCEchNB85KrXAOtyBXYiR2oa
+	nKriOwTP4/GeXifOl7+T8eIEkFuZI6DVL5Gas5SA=
+Subject: Re: The database I wish I had
+From: =?UTF-8?Q?Richard_Newman?= <[email-masked]>
+To: =?UTF-8?Q?EuAndreh?= <[email-masked]>
+Cc: =?UTF-8?Q?=7Eeuandreh/public-inbox=40lists=2Esr=2Eht?=
+  <~euandreh/[email-masked]>
+Date: Wed, 2 Sep 2020 16:14:38 +0000
+Mime-Version: 1.0
+Content-Type: text/plain; charset=UTF-8
+Content-Transfer-Encoding: quoted-printable
+In-Reply-To: <[email-masked]>
+References: <[email-masked]> 
+ <[email-masked]> 
+ <[email-masked]> 
+ <[email-masked]> 
+ <[email-masked]> 
+ <[email-masked]> 
+ <[email-masked]>
+X-Priority: 3 (Normal)
+X-Mailer: Amazon WorkMail
+Thread-Index: AQHWgHcPJ4udfXYrRaOwhoO7EcsDJQAErSAlAAfQ7+gAIgKd3wAzRhy+
+Thread-Topic: The database I wish I had
+X-Original-Mailer: Apple Mail (2.3445.104.14)
+X-Wm-Sent-Timestamp: 1599063277
+Message-ID: <[email-masked]>
+X-SES-Outgoing: 2020.09.02-54.240.58.46
+Feedback-ID: 1.us-west-2.An468LAV0jCjQDrDLvlZjeAthld7qrhZr+vow8irkvU=:AmazonSES
+
+> I've used Common Lisp a lot before, and I heard a lot about=0D=0A> Alle=
+groGraph, but I've never worked with it. Would you recommed it as a=0D=0A=
+> place to take inspirations from=3F=0D=0A=0D=0AHelping to build it was a=
+ very formative experience for me. twinql was the name of my first SPARQL=
+ query engine, which became the genesis of AllegroGraph's, and it stuck a=
+s a domain name!=0D=0A=0D=0ASome of AllegroGraph's performance is the res=
+ult of decades of experience (Steve Haflich comes to mind) and having som=
+e of the same exceptional developers work on it as worked on the compiler=
+, so its internals are excellent inspiration. The UPI (how EAV are repres=
+ented) is a good idea executed well. AllegroGraph is also a good example =
+of how important tooling is for exploring this kind of semi-structured da=
+ta.=0D=0A=0D=0A=0D=0A> These look just like the EAVT and AVET indexes in =
+Datomic.=0D=0A=0D=0AYes, exactly! It's a very natural way to model a fixe=
+d-width tuple store.=0D=0A=0D=0A=0D=0A> I was considering having all inde=
+xes be derived from the raw data=0D=0A> transported between instances, bu=
+t it might get computationally=0D=0A> expensive, and making those indexes=
+ replicable sounds like a good idea.=0D=0A=0D=0AIt depends where you're r=
+eplicating. In Datomic's case each of the query nodes gets a replicated c=
+opy of the index chunks it needs (and you can thus control cache warmth b=
+y routing queries to different nodes), because those nodes are meant to b=
+e fungible, and reads from S3 (or wherever) are cheap, so there's no sens=
+e doing the work twice.=0D=0A=0D=0AIf you were to stand up a hot spare, y=
+ou would probably just stream the log and have the far end create its own=
+ indices. That's particularly true if the other store needs different ind=
+ices, or is materializing views.=0D=0A=0D=0ASidenote: it surprises me tha=
+t Datomic Cloud stores its log in DynamoDB. It's hard to imagine a good p=
+rimary key scheme that wouldn't at some point exhaust write capacity, and=
+ DDB makes different availability and durability tradeoffs than S3. I'm c=
+urious how that decision was made.=0D=0A=0D=0AObligatory disclaimer: I wo=
+rk at AWS, but the above is not a statement on behalf of AWS!=0D=0A=0D=0A=
+=0D=0A>> When I sync them, I need the data store to know that the visits =
+I made=0D=0A>> to each page _were visits to the same thing_.=0D=0A>=20=0D=
+=0A> Isn't that a domain-specifc problem of identity=3F=0D=0A=0D=0AYes, a=
+bsolutely. I take the position that the goal of a database is to help me =
+efficiently, accurately, and ergonomically represent my domain. If it doe=
+sn't, every developer has to solve that problem at the application level,=
+ and that limits interoperability, correctness, performance, and expressi=
+veness.=0D=0A=0D=0AThe ability for this data layer to be able to help you=
+ with problems like syncing also relies on its ability to leverage some i=
+nformation about your domain =E2=80=94 compare to when you check in an im=
+age file or a tarball to Git, and it can't help you detect or resolve con=
+flicts at anything more granular than the containing file, even if that f=
+ile has its own internal structure.=0D=0A=0D=0A=0D=0A> I would solve this=
+ by allowing those two entries to cooexist without any=0D=0A> conflict, a=
+nd do aggregation queries to answers those questions,=0D=0A> something li=
+ke (pseudo-SQL):=0D=0A>=20=0D=0A> SELECT DISTINCT(url) FROM urls;=0D=0A>=20=
+=0D=0A> ...to show the all the visited URLs, and:=0D=0A>=20=0D=0A> SELECT=
+ SUM(1) from URLS GROUP BY url=0D=0A>=20=0D=0A> ...to show the visit coun=
+t for each URL.=0D=0A>=20=0D=0A> WDYT=3F=0D=0A=0D=0AThe advantage is inde=
+ed that you have removed the ability for data to conflict.=0D=0A=0D=0AThe=
+ disadvantage is that you have pushed a domain identity and merging into =
+application code, where:=0D=0A=0D=0A* Every query needs to implement this=
+ kind of unification in multiple places, which is more expensive and horr=
+endously difficult to get right, not to mention very hard for developers =
+to read.=0D=0A=0D=0A* Application developers can get it wrong (e.g., dele=
+ting from History needs to retract every entry with a given URL), and you=
+ make domain operations more complex as a result. Sometimes this adds val=
+ue =E2=80=94 e.g., this kind of accumulative storage gives you magic prop=
+erties like lifetime undo! =E2=80=94 but I suspect that putting that in t=
+he log is better.=0D=0A=0D=0A* You lose some conveniences like lookup ref=
+s, because you've removed the concept of a 1:1 lookup altogether.=0D=0A=0D=
+=0A* You lose the trivial ability to detect and model _changes_, which I =
+find is natural for developers: this approach essentially makes the log p=
+art of the datom state. You can still do retract/assert locally, for a pa=
+rticular entity, but if you want to e.g., get the latest title for a page=
+, you need to reduce over the log with some ordering.=0D=0A=0D=0A=0D=0AI =
+agree with your position in principle: the idea of figuring out the curre=
+nt state dynamically by rolling up history from different machines is ver=
+y event-sourcey and is theoretically pleasant. Every device would simply =
+append new non-conflicting _observations_ to the log. I do advocate for t=
+his in some circumstances.=0D=0A=0D=0AHowever, in practice we usually hav=
+e some stable idea of what we want current state to be (particularly of s=
+chema itself!), and some domain concepts around identity and cardinality =
+that we want to enforce and use to build concepts like change, conflict, =
+and merging, and we need that maintained state to be fast and obvious bec=
+ause we use it when making new assertions.=0D=0A=0D=0AIn essence, the pro=
+blem I was trying to solve with syncing in Mentat was: how do I allow dev=
+elopers to talk about their schema in such a way that they could get sync=
+ing for nearly free=3F That a sync engine could, in most cases, automatic=
+ally figure out which entities were the same, decide on a stable ordering=
+ of operations, and rewrite history to produced the expected merged state=
+, such that in the above example all the visits stitch together, we know =
+the latest title, and the log has all the historical titles=3F=0D=0A=0D=0A=
+Using observation-structured storage to remove conflict is definitely a v=
+aluable tool in the toolbox for doing that: the more you can structure yo=
+ur data as new observations about stably identified things, the easier ti=
+me you have with syncing. I believe, but have not proved, that a system w=
+ith only identity properties and cardinality-many properties, perhaps wit=
+h some additional restrictions on retraction and uniqueness, will never h=
+ave conflicts when merging.=0D=0A=0D=0ABut I also think that the best dev=
+eloper ergonomics are actually slightly _more_ expressive than even Datom=
+ic or Mentat =E2=80=94 developers told me that they wanted cardinality-N,=
+ and to be able to express types or domain restrictions on properties, an=
+d nesting, and so on =E2=80=94 and delivering those ergonomics means taki=
+ng care of turning the log into a simple, concrete current state for them=
+ to query.=0D=0A=0D=0APhrased differently: in the above, your developers =
+want some way to write=0D=0A=0D=0A  SELECT url, title FROM pages=0D=0A=0D=
+=0Aand=0D=0A=0D=0A  SET title FOR page WITH url 'foo.com' TO 'whatever'=0D=
+=0A=0D=0AWhether you do that by figuring out at sync time the current tit=
+le for pages with a given URL, or whether you give them some affordance t=
+o dynamically roll that up from the log, gets to be the same thing if you=
+ think of Mentat's 'datoms' table and schema as a way for developers to d=
+escribe how Mentat should roll up current state from the log!=0D=0A=0D=0A=
+=0D=0A> I used to share your view on needing a migration scape hatch, but=
+ I=0D=0A> don't anymore. On the company I work those kind of database mig=
+rations=0D=0A> never really occur. People either create a new attribute, =
+copy data from=0D=0A> a database to another, process it using ETL, comput=
+e things on the fly=0D=0A> with functions, and many other strategies. We'=
+ve already had to comply=0D=0A> with excisions in production, but never o=
+n schema, only on data itself.=0D=0A=0D=0AI actually view what you descri=
+be as migration: the point at which you're defining a new attribute and s=
+toring things twice, or doing a batch job to move things over, is exactly=
+ a data migration, if perhaps smeared over space and time!=0D=0A=0D=0AThe=
+se still have all the problems that traditional schema migrations have: d=
+ealing with data that doesn't conform, and dealing with clients that don'=
+t know about the new format.=0D=0A=0D=0A=0D=0A> So I don't think that's a=
+n actual issue, and I would insist that the=0D=0A> "Grow your schema" app=
+roach actually does work at scale, on the long=0D=0A> run, for multiple s=
+cenarios.=20=0D=0A=0D=0AI agree. I also think that one of the ways it doe=
+s so is by pushing the work of tracking versions and vocabulary into appl=
+ication code, either formally or informally (e.g., checking for the exist=
+ence of a property, coordinating the upgrade of client code, etc.) =E2=80=
+=94 the clients need to know how to work with the schema as it grows!=0D=0A=
+=0D=0AThis was the motivation for me spending time on vocabulary manageme=
+nt, which we did end up building:=0D=0A=0D=0Ahttps://github.com/mozilla/m=
+entat/wiki/Proposal:-application-schema-coordination-and-versioning=0D=0A=
+=0D=0A=0D=0A> At a glance I can't really tell if that is what you mean wh=
+en citing=0D=0A> SQLCipher.=20=0D=0A=0D=0AAh, we were talking about two d=
+ifferent kinds of encryption!=0D=0A=0D=0AWe were using SQLCipher for loca=
+l block-level encryption.=0D=0A=0D=0AAt Mozilla we would additionally use=
+ Firefox Account encryption to end-to-end encrypt data as it passed throu=
+gh Sync. The two are complementary. Sync acts exactly as you demonstrate =
+with git-remote-gcrypt: two local devices can access cleartext and are re=
+sponsible for resolving conflicts, and the service itself sees only ciphe=
+rtext and does not involve itself in how conflicts are resolved.=0D=0A=0D=
+=0AMy point in bringing this up is that many systems have a central entit=
+y that can help in resolving conflicts; if the central entity can only ev=
+er see ciphertext, it's limited to whatever ordering or collision propert=
+ies you enable via the envelope. For that Git system I imagine it tracks =
+refs, and the Git object store is opaque.=0D=0A=0D=0A=0D=0A> TBH I'm not =
+aiming for automatic conflict resolution.=0D=0A>=20=0D=0A> Why not use th=
+e granularity of attributes to detect conflict, but not to=0D=0A> resolve=
+ them=3F=0D=0A=0D=0AIn short:=0D=0A=0D=0A* End user software is no longer=
+ built (as iSync was) to present data conflicts to end users for resoluti=
+on. The goal is conflict resolution.=0D=0A* You cannot leave a store in a=
+n inconsistent state until conflicts have been addressed, because you sti=
+ll need to handle local writes. You cannot disconnect a device until conf=
+licts have been addressed; users expect continuity of service. Whenever y=
+ou are able, you have to do _something_.=0D=0A* _Someone_ has to write th=
+e code to resolve conflicts. Eventually you're going to have some dev com=
+e up with the bright idea of annotating "data objects" with some kind of =
+conflict resolution attributes, and then have an automated "conflict reso=
+lver"=E2=80=A6 at which point you've Greenspun yourself a sync engine and=
+ schema language. This works much better when it's part of the storage an=
+d sync later, and can participate in transaction boundaries and annotate =
+writes.=0D=0A=0D=0AThis is the doc I wrote to outline the motivations for=
+ Mentat at Mozilla:=0D=0A=0D=0Ahttps://mozilla.github.io/firefox-browser-=
+architecture/text/0008-sync-and-storage-review-packet.html=0D=0A=0D=0AThe=
+ Lay Summary in particular is worth a read.=0D=0A=0D=0AWhen we built Fire=
+fox for iOS we managed to put storage and syncing inside the same compone=
+nt, so finally we were in a position to track changes during writes, and =
+could get it right. Mentat was the next logical step: how do we let other=
+ teams use a storage system that will still do a good job of syncing thei=
+r data without them having to get elbow deep in how a sync engine works, =
+and without them having to put all the syncing gunk =E2=80=94 timestamps/=
+versions, UUIDs, etc. =E2=80=94 in their data model=3F=0D=0A=0D=0AWhat do=
+es Mentat need to know, schema-wise, to be able to do that kind of declar=
+ative conflict resolution=3F We didn't finish answering that question, bu=
+t I think the intended outcome is still valid.=0D=0A=0D=0A=0D=0A> Either =
+way, the DB knows that the note has the title and body attributes=0D=0A> =
+in conflict, but the author is fine. The aggregate entity itself isn't=0D=
+=0A> in conflict, but some of it's attributes.=0D=0A=0D=0AThis is what Co=
+uch/Pouch do, I think. Well, it actually resolves the conflict and leaves=
+ a conflict marker=E2=80=A6 no idea whether anyone actually writes code t=
+o check that!=0D=0A=0D=0A=0D=0A> Couldn't you leverage the schema for tha=
+t=3F If the said conflicting=0D=0A> attribute has:=0D=0A>=20=0D=0A> - car=
+dinality one and both DBs changed it, we have a conflict;=0D=0A> - cardin=
+ality many and both changed the same attribute, we have a=0D=0A>  conflic=
+t;=0D=0A> - cardinality may and Computer A changed an entry and Computer =
+B changed=0D=0A>  another one, we don't have a conflict.=0D=0A>=20=0D=0A>=
+ and so on.=0D=0A=0D=0AYes, that's exactly the approach we were taking. T=
+he important difference is that you also need to zip together identities =
+during this process =E2=80=94 for example, two assertions of a cardinalit=
+y-one attribute are not a conflict if the two entities at the other end t=
+urn out to be the same. Couch etc. don't have that concept.=0D=0A=0D=0A(I=
+n an open-world system like OWL-DL, this cardinality conflict actually be=
+comes implication of equality!)=0D=0A=0D=0A> I don't know Dropbox's datas=
+tore, is this[0] the one you're refferring=0D=0A> to=3F=20=0D=0A=0D=0AYep=
+! It's a really great bit of thinking.=0D=0A=0D=0A
+
+From [email-masked] Thu Sep  3 01:00:21 2020
+Received: from box.euandre.org (box.euandre.org [46.101.160.115])
+	by mail-b.sr.ht (Postfix) with ESMTPS id 71E69FF0FA
+	for <~euandreh/[email-masked]>; Thu,  3 Sep 2020 01:00:20 +0000 (UTC)
+Authentication-Results: mail-b.sr.ht;
+	dkim=pass (2048-bit key) header.d=euandre.org header.i=@euandre.org header.b=rMNL9qSC
+Received: from authenticated-user (box.euandre.org [46.101.160.115])
+	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
+	(No client certificate requested)
+	by box.euandre.org (Postfix) with ESMTPSA id E3DD6102122;
+	Wed,  2 Sep 2020 22:00:16 -0300 (-03)
+DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=euandre.org; s=mail;
+	t=1599094818; bh=VQ0pHLJALciNNh1+kpFy1DVLAgL5kn8a36k3GAV6u7E=;
+	h=From:To:Cc:Subject:In-Reply-To:References:Date:From;
+	b=rMNL9qSC0p+9DH1RAD5wEPqn9/v63VAiwAiMKQuTvccTrH5EnO2qlB7mKU594brWM
+	 P+5qCkBJymi/N2hK+BvoROdWT24A6NPyG0rORpg00rmuJsrpyn6HkpbHzOcx3CwYdH
+	 WUVzXoZQndMFNa4nN4P/KE4Mh/jA/4TvVO/4QvfrJqpnpfKdPJeIgFEdrCEQL+3ZIP
+	 b/DjqLRRSkV3fljHnB2MhsCMe/od5BD+wDEza/x+HvOwTxFHvalbWT1QVj3saKxiTG
+	 VbD8xe43tls9agJsSh/z8MErjmNI4Ngd0ygfIu8QbbLNswkpQe3RMru5lYi2o49N55
+	 1bnUqfFRkHs/w==
+From: EuAndreh <[email-masked]>
+To: Jonas =?utf-8?Q?Sch=C3=BCrmann?= <[email-masked]>,
+ ~euandreh/[email-masked]
+Cc: 
+Subject: Re: The database I wish I had
+In-Reply-To: <[email-masked]>
+References: <[email-masked]>
+ <[email-masked]>
+ <[email-masked]>
+Date: Wed, 02 Sep 2020 21:34:51 -0300
+Message-ID: <[email-masked]>
+MIME-Version: 1.0
+Content-Type: text/plain; charset=utf-8
+Content-Transfer-Encoding: quoted-printable
+
+Jonas Sch=C3=BCrmann <[email-masked]> writes:
+
+> Yes, exactly. The idea is pretty abstract and while I know a little bit
+> about category theory I'm by no means an expert on the subject. This
+> blog post [1] is a very down-to-earth, practical explanation how the
+> concept of (co)completed categories can be applied to conflict
+> representation in version control systems.
+>
+> [1] https://jneem.github.io/merging/
+
+This sounds interesting, thanks for the link.
+
+From [email-masked] Thu Sep  3 01:07:15 2020
+Received: from box.euandre.org (box.euandre.org [46.101.160.115])
+	by mail-b.sr.ht (Postfix) with ESMTPS id 20958FF0FA
+	for <~euandreh/[email-masked]>; Thu,  3 Sep 2020 01:07:15 +0000 (UTC)
+Authentication-Results: mail-b.sr.ht;
+	dkim=pass (2048-bit key) header.d=euandre.org header.i=@euandre.org header.b=Y4O+t3yg
+Received: from authenticated-user (box.euandre.org [46.101.160.115])
+	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
+	(No client certificate requested)
+	by box.euandre.org (Postfix) with ESMTPSA id B209DFC81E;
+	Wed,  2 Sep 2020 22:07:12 -0300 (-03)
+DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=euandre.org; s=mail;
+	t=1599095234; bh=Y2H9I+SgM0rzOycDHDHlfDDvDGVB79YoDxvEzTudos4=;
+	h=From:To:Cc:Subject:In-Reply-To:References:Date:From;
+	b=Y4O+t3yg4rBzXtK6CL570EiO+7TXdlHZ1OAjSDRUjr71MbaIt+HonABTUommz90+U
+	 +FkpruCMrLNQIyuveS07REukjQVp/zKHrH41FLFDaRUoBIg1OuxvIp4I+Ug1M+HQAC
+	 tS1tPA0oqbqmrI7l2ByFEP58VyYhRNpYAbrpSMJUI4gt/Busfq5TFLQfnuTZSaGbNy
+	 9AsJYnD04/B+VRQHcrST55I8sVyaHkxQM0xck0J7iEnJesbTxGU6adg89UL7fTd4Jb
+	 LP4PognYY5bvYrsRP1eTQVkm8Iefevp0X+fgNdQsvBkee8WLAykSq4jkMpPaBEyMl3
+	 WIq9JcFaLACew==
+From: EuAndreh <[email-masked]>
+To: Jeff Rhyason <[email-masked]>, ~euandreh/[email-masked]
+Cc: 
+Subject: Re: building MSTs->CRDTs-> Re: The database I wish I had
+In-Reply-To: <[email-masked]>
+References: <[email-masked]>
+Date: Wed, 02 Sep 2020 22:06:59 -0300
+Message-ID: <[email-masked]>
+MIME-Version: 1.0
+Content-Type: text/plain
+
+Jeff Rhyason <[email-masked]> writes:
+
+> It's amazing to see this discussion. Let's keep it going!
+>
+> My friend and I started Indelible[1] (now defunct) for a lot of the
+> same reasons you want to build a datastore -- there aren't enough
+> options for a datastore that's immutable, relational,
+> highly-vaailable, and private.. We're still super-passionate about the
+> ideas, love Datomic but want something a bit different, decided not to
+> commercialize it and start more simply with just the data structure.
+> We don't want a huge thing, just a library, to keep coordination down;
+> the complexity doesn't spread out of the client.
+>
+> Where we are at now is we have a great versioned immutable B-Tree-like
+> structure that is the foundation for the verioned and diffable
+> database we want.  FWIW The Go implementation of the tree is at
+> https://github.com/jrhy/mast .  We have plans to do a CRDT based on
+> it, possibly add an SQL layer, but frankly, having a
+> strongly-consistent, verioned + diffable tree, is pretty good for now.
+> We'd love to collaborate!
+
+This looks interesting! I'll take a deeper look later.
+
+I'll be particularly interested in seeing the implementation of
+persistent B-Trees that are durable.
+
+From [email-masked] Thu Sep  3 23:39:44 2020
+Received: from box.euandre.org (box.euandre.org [46.101.160.115])
+	by mail-b.sr.ht (Postfix) with ESMTPS id B8167FF1D6
+	for <~euandreh/[email-masked]>; Thu,  3 Sep 2020 23:39:43 +0000 (UTC)
+Authentication-Results: mail-b.sr.ht;
+	dkim=pass (2048-bit key) header.d=euandre.org header.i=@euandre.org header.b=oQR0LJk6
+Received: from authenticated-user (box.euandre.org [46.101.160.115])
+	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
+	(No client certificate requested)
+	by box.euandre.org (Postfix) with ESMTPSA id 2F2B2FCB45;
+	Thu,  3 Sep 2020 20:39:39 -0300 (-03)
+DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=euandre.org; s=mail;
+	t=1599176381; bh=0OTcSYr5H5tf2Nj44ZROMSLLvq65wMQATwFWuwPcGcA=;
+	h=From:To:Cc:Cc:Subject:In-Reply-To:References:Date:From;
+	b=oQR0LJk66tmm7FUnmKKEUg3GVM6KyB0ybBX7bvke8hAkcTr/UOADsGEa64ZOgq47F
+	 5+6d43VfXXhxXQie9IypcAHdqe5eYTXL0Yh6ymknjM9GIPHEhGRgzbK+jeTe6EK2KH
+	 y2bC4HqyQ0Xn9Ijj+m+OEQCACknw1vsIvVH3qQveSft3mRmIKIibpDL5uxMaGbNRHK
+	 V0ts3LFy07Knz4QI5fsPRqBwsKNQV9t+F5eL5ITa6OSC3JY+QclEbcYheBW80V42IT
+	 8+hb6HzscpFlxvBtdi1fG7PbnUtWAp/7j6sK8AtVYRCK5OybOAy0VpJhGGbFCZdale
+	 s3u7XVTPdzmeQ==
+From: EuAndreh <[email-masked]>
+To: Richard Newman <[email-masked]>
+Cc: "~euandreh/[email-masked]" <~euandreh/[email-masked]>
+Cc: 
+Subject: Re: The database I wish I had
+In-Reply-To: <[email-masked]>
+References: <[email-masked]>
+ <[email-masked]>
+ <[email-masked]>
+ <[email-masked]>
+ <[email-masked]>
+ <[email-masked]>
+ <[email-masked]>
+ <[email-masked]>
+Date: Thu, 03 Sep 2020 20:39:23 -0300
+Message-ID: <[email-masked]>
+MIME-Version: 1.0
+Content-Type: text/plain; charset=utf-8
+Content-Transfer-Encoding: quoted-printable
+
+I started quoting many parts of your email, and things started to get
+confusing. Each answer that I gave was referring to each other, and I
+almost made a loop.
+
+So I've rewritten everything as a single piece of prose. I added some
+snippets of your text to better contextualize what I was talking about,
+but I tried to address what you said directly. Here it goes.
+
+# On migrations
+
+I still don't think that I've insisted enough on *strict* schema growth.
+
+The wiki page you pointed to reinforces that: it talks a lot about how
+to coordinate attributes that changed their names, change in vocabulary,
+shared ownership of changes in schema and even schema versioning!
+
+All of them are the opposite of what I want, and they're "schema
+breakage", not "schema growth".
+
+What I mean by that is the same thing that Rich Hickey proposes as a
+what growth and breakage means in his talk "Spec-ulation"[0].
+
+All of those forms of schema change are instances of breakage, which
+should have no place here. They're all sources of undesired
+incompatibility. I'm not sure if Datomic allows you to change the type
+of an attribute, and if it does this can't be copied, since we don't
+have a transactor to coordinate such change.
+
+If you want to rename an attribute, don't! Create a new one! Maybe you
+could retract schema and assert back the same schema, but you shouldn't
+overwrite it with an incompatible name, or attribute type. This is a
+trap that you shoudln't put yourself in, and maybe even this could be
+enforced.
+
+This model of an ever-growing vocabulary is how HTML works. At some
+point people realised that using <b> to mark something as bold was not
+such a good idea, since it coupled semantics with presentation. So a new
+tag was created, <strong>, which only represents the semantic part of
+<b>. And <b> still continues to exist, and that's fine.
+
+The same can be said about JavaScript. I've written a fair share of it
+in the past 2 years, and I don't think I ever used the 'var' keyword,
+only 'let' and 'const'. And yet, every JavaScript runtime still
+recognizes it, and even transpiler emit it as their output, and that's
+also fine.
+
+Maybe a linter or the browser tooling would warn you against the usage
+of those old features, but they continue to work. If an equivalent of a
+"vocabulary change" happened, web pages would be brittle by design more
+brittle, and old pages would stop working as time passes. That isn't a
+principle I would like to adopt.
+
+When more than one database instance can add data, the programmer needs
+to think of it as an open world, where new things are accrued, but old
+things don't disapper. Changing the vocabulary mid-air only makes this
+open world brittle so "migrations" can be performed.
+
+I see no reason for having this type of migration you describe, and
+insist on *strict* schema growth, it being a saner, more robust and
+completly fine alternative.
+
+[0]: https://www.youtube.com/watch?v=3DoyLBGkS5ICk
+
+# On conflicts and ergonomics
+
+You mentioned ergonomics thrice on your message, and I believe that your
+insistence of it is a sign of the coupling of two different layers of
+the database, and makes ease of use drive the fundamentals. I also think
+that your statement on conflict resolution being a requirement is not a
+good idea. Let me expand on that.
+
+I'd argue that conflict resolution is inherit complexity [1] of creating
+a decentralized client-side application that can sync data.
+
+That means that the responsability to solve conflicts is not the
+database's, but the application's responsability, and making decisions
+of conflict resolution on the database limits its application, and I
+wouldn't consider it to be general purpose.
+
+When two offline database instances change the same field, there's no
+escaping from the bottom reality, which is: there is a conflict, and it
+needs to be resolved somewhere. There's no layer of ergonomics that can
+change this fact, and picking a strategy for conflict resolution is
+undoubtly domain-specific.
+
+You mentioned that I proposed a solution on URL history tracking that
+fowards all of the responsability to the programmer, but I was only
+giving a simple example, and that same problem could be tackled in many
+different ways, and the person creating the application has a decision
+to make on it.
+
+Let's expand on the URL history on the browser example from earlier, and
+explore how to handle identity.
+
+The application developer now has to decide what they want to consider a
+URL being "the same" as another, and that's what I think you meant when
+you talked about pushing domain identity to the application.
+
+What does the programmer wants the identity of a URL to be? Is it a
+UUID attributed to it when asserting the entitiy on the database? If so,
+you would need to do things like the SELECT DISTINCT and SELECT SUM I
+proposed, inevitably. The programmer may otherwise try to avoid this by
+always starting from an existing database, so the history could be
+common whenever a new database instance starts. One would just not
+simply 'git init', they would 'git clone' in order to work with shared
+identities that are created in a distributed way.
+
+Is the identity of a URL its string? If so, the URL would have to be
+treated as an URI, and that string would be effectively global, and
+updates would have to be handled with lots of care.
+
+The database *can't* make those decision. What it *can* do is provide
+tools that allow the programmer to work with this inherent complexity.
+And I see no way around it. No amout of ergonomics can solve this, and
+removing the ability to have 1:1 lookups is the nature of decentralized
+client-side applications, it's not me trying to forward a problem that
+the database should solve to the programmer.=20
+
+A simple comparison is: whenever we talk about code, we use SHAs to
+pinpoint which piece of code we mean. What does a SHA say about the
+existence of other branches that disagree? Nothing! You can't ever
+assume that the commit you're on is the latest version of a file, or
+even is the only version of a file. While you were not looking, any
+number of new versions could have been created, which means that when
+you fetch, there isn't a single README. There are *at least* two of
+them: README from master, and README from origin/master. They may be the
+same, and they may not be the same, but saying just "the README file" is
+an incomplete depiction of it: you either say "my README file" or talk
+about both of them, which do coexist.
+
+There's no denial that this model is much more complex than the usual
+1:1 lookup we're used to, but trying to hide this in the database
+implementation somehow would be a bad idea. The programmer brought this
+to themself when they choose to build a decentralized client-side
+application.=20
+
+However, with a solid foundation, there are still lots of room for work
+on ergonomics, like:=20
+
+1. diffing algorithms for text like you mention would totally
+make sense to be in a "contrib/" directory, so that people could pick
+different diffing algorithms for different string fields.
+
+2. whenever a 'dev comes up with the bright idea of annotating "data
+objects" with some kind of conflict resolution', I could point them to a
+"IConflictResolutionStrategy" interface for they to implement. Their
+implementation could even end-up under said "contrib/" directory. Even
+better if "IConflictResolutionStrategy" could be composed! A programmer
+would then pick different strategies for different entities in their
+domain, like in a buffet.
+
+3. offer some low-level building blocks that can be applied to different
+domains, like tombstones, max(a.last-modified, b.last-modified), etc.
+
+...and so on.
+
+The point is: the database is not in a position to make this type of
+decision. Only the programmer is, and they must be aware of it.
+
+You said that:
+
+> the goal of a database is to help me efficiently, accurately, and
+> ergonomically represent my domain=20
+
+For me, the crucial part is the accurately comes first, and ergonomics
+comes second. No accuracy should be sacrificed in name of ergonomics,
+since the inherent complexity is already too big to be dealt with.
+
+The syncing part is not free.
+
+[1]: As defined in http://curtclifton.net/papers/MoseleyMarks06a.pdf
+
+# On history rewriting
+
+You talked about how Mentat was trying to reconcile changes by rewriting
+the history in a stable and consistent order.
+
+I'd argue that instead of a linear log where things are rebased, a DAG
+log where things are merged is a more flexible approach. Let me show you
+where I'm coming from first.
+
+You said that:
+
+> Phrased differently: in the above, your developers want some way to
+> write
+>
+>   SELECT url, title FROM pages
+>
+> and
+>
+>   SET title FOR page WITH url 'foo.com' TO 'whatever'
+
+This may still be true, but with a caveat: you'd be asserting facts to
+the local "timeline", in a somewhat similar way that the Mentat wiki
+describes[2] (I feel funny giving you a link to Mentat's wiki).=20
+
+What I'd do differently is to don't try so hard to make everyone
+converge on the same thing always, and represent timelines in a more
+flexible way.
+
+Instead of having two types of timelines, =CE=B1 and =CE=B2, there would be=
+ a
+multitude of timelines, one for each instance of the database. Let's say
+we have 3 of them for the purpose of explanation.
+
+Instead of adding an extra 'timeline_id' identifier, I'd add a
+':db/txParent' to the transaction entity, with cardinality many, and the
+database instance would always know it's latest local parent. This is
+effectivelly like a branch, but not a shared one, and transations with
+more than one ':db/txParent' represent merges. Those would only happen
+when explicitly solving conflicts, and any given database instances
+could know about a multitude of timelines.
+
+This way, how can one know the current state of the database?=20
+
+>From the first root datom, the database would rollup every datom taking
+the timelines and merges in consideration, and it would be the sum of
+data from all timelines. When a conflict exists, the local timeline is
+always picked so the database can still serve writes, and the conflict
+would be presented when serving reads.=20
+
+The obvious problem of always favouring the local timeline is that each
+timeline would show different "primary" answers, but that's sounds
+somewhat reasonable to me to enable the database to keep working while
+offline, and while the conflict isn't resolved.
+
+This also means that "sync" are two distinct things: exchanging timeline
+data, and reconciling that data locally. The only part that's online is
+the "exchange", and instances resolve conflict locally. This way, the
+database is always available, even when offline, and even when conflict
+is present.
+
+But how would queries over history behave? No clue yet =F0=9F=98=AC.
+
+I tried not to sound too hand wavy, and I've only prototyped a small
+part of this in the past, so there are probably many more problems with
+this approach. I don't claim this is a sound solution yet, but it does
+handle some of the problems that Mentat faced in a different way. I'd be
+interested to see you poking holes on it.=20
+
+When combined with strict schema growth, new data always comes after
+it's schema declaration, so the application doesn't have to coordinate
+around distributed schema evolution.
+
+[2]: https://github.com/mozilla/mentat/wiki/Multiple-timeline-support
+
+# On CouchDB/PouchDB's entity conflicts
+
+CouchDB/PouchDB's most fundamental value is the document. This means
+that a single document write is atomic (not true for clustered CouchDB),
+but you end up rewritting the full document ([3], on time 3:35). If you
+represent an entity as a document, the whole entity is in conflict, not
+the attribute that changed.
+
+So it doesn't know the difference between two changes on unrelated
+attributes and full rewrites of the full document: everything is a
+conflict in CouchDB/PouchDB's view.
+
+On another talk ([4] on time 5:30, also by Joan Touzet), she mentions
+the desire to add partial updates to CouchDB, but I don't know if it was
+pursued.
+
+Checking for conflicts on revisions is considered good practice ([1], on
+time 10:21) on the community.
+
+[3]: https://www.youtube.com/watch?v=3DBKQ9kXKoHS8
+[4]: https://www.youtube.com/watch?v=3D0gAZHIYBh-g
+
+# On zipping identities together
+
+=F0=9F=91=8D
+
+# Misc
+
+TIL: I didn't know you worked on AllegroGraph.
+
+P.S.: I haven't reviewed this text too much, and it might have gotten a
+bit confusing. I'm happy to clarify any points, and I'll consider
+writing a more formal version of this braindump later. I chose to send
+this like this over taking more time and letting the discussion cool
+down.=20
+
+From [email-masked] Fri Sep  4 00:00:13 2020
+Received: from box.euandre.org (box.euandre.org [46.101.160.115])
+	by mail-b.sr.ht (Postfix) with ESMTPS id 7A675FF1B9
+	for <~euandreh/[email-masked]>; Fri,  4 Sep 2020 00:00:12 +0000 (UTC)
+Authentication-Results: mail-b.sr.ht;
+	dkim=pass (2048-bit key) header.d=euandre.org header.i=@euandre.org header.b=teyCS1z1
+Received: from authenticated-user (box.euandre.org [46.101.160.115])
+	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
+	(No client certificate requested)
+	by box.euandre.org (Postfix) with ESMTPSA id 3507CFCB45;
+	Thu,  3 Sep 2020 21:00:09 -0300 (-03)
+DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=euandre.org; s=mail;
+	t=1599177611; bh=mYcY4i3Y9/GUAx3/nY2Lu91JMvyDWaWOxTADQnXB+MM=;
+	h=From:To:Cc:Cc:Subject:In-Reply-To:References:Date:From;
+	b=teyCS1z1iKrOm7S9ivB92sQQ5uq6Kb9TRtkrIvyFYqPWIP9zFVq4sYuuU5NQZwDSi
+	 GHHeO9otqioa+dWLVcXYITaxTvgD3uAaiI/Xa9Nzj3swN1DWuzGwJ7Ovqf8vQDBSQI
+	 0VROFULtzElG2iNTv3tyWy6C7Hu12whmbfteUtq6aTzonf9pFIP6Wio6wGakmS9px9
+	 xwKDfuZgrdN8tUWhjAM+Joz9bwRgRg08+hJOf9Jd9JKShTgZGosTIOYDl8HnfQwR7F
+	 fZbYza/n4vBwLdkpDlOpX9wVdVprXmvM3cD+zoPQ/ukRomUmjQ6KFETBOksZ4t7pY1
+	 E/KZ/wEEBxRJA==
+From: EuAndreh <[email-masked]>
+To: Richard Newman <[email-masked]>
+Cc: "~euandreh/[email-masked]" <~euandreh/[email-masked]>
+Cc: 
+Subject: Re: The database I wish I had
+In-Reply-To: <[email-masked]>
+References: <[email-masked]>
+ <[email-masked]>
+ <[email-masked]>
+ <[email-masked]>
+ <[email-masked]>
+ <[email-masked]>
+ <[email-masked]>
+ <[email-masked]>
+ <[email-masked]>
+Date: Thu, 03 Sep 2020 20:57:28 -0300
+Message-ID: <[email-masked]>
+MIME-Version: 1.0
+Content-Type: text/plain
+
+I better get a POC so I can stop talking in "what ifs".
+
+I was more focused on the storage implementation side of it, but I'm
+considering not implementing a storage backend for this and just using
+SQLite instead (more detailed in [0]).
+
+Building durable persistent data structures on top of SQLite backed
+storage is still a bit distant, but the model I presented could benefit
+from a POC.
+
+I'll try starting that instead of giving so much focus on storage, and
+share it later.
+
+[0]: https://euandre.org/2020/08/31/the-database-i-wish-i-had.html#fn:posix-sqlite
+
+From [email-masked] Fri Sep  4 02:12:34 2020
+Received: from box.euandre.org (box.euandre.org [46.101.160.115])
+	by mail-b.sr.ht (Postfix) with ESMTPS id DE901FF196
+	for <~euandreh/[email-masked]>; Fri,  4 Sep 2020 02:12:33 +0000 (UTC)
+Authentication-Results: mail-b.sr.ht;
+	dkim=pass (2048-bit key) header.d=euandre.org header.i=@euandre.org header.b=aOTLZVo3
+Received: from authenticated-user (box.euandre.org [46.101.160.115])
+	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
+	(No client certificate requested)
+	by box.euandre.org (Postfix) with ESMTPSA id 13742FCB45;
+	Thu,  3 Sep 2020 23:12:30 -0300 (-03)
+DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=euandre.org; s=mail;
+	t=1599185552; bh=0eOLRgKRFwQtp0D1PSgenzUsGttO7assNfrY8hnPqz8=;
+	h=From:To:Cc:Cc:Subject:In-Reply-To:References:Date:From;
+	b=aOTLZVo3JPMEsqZSTB2G6Y+T0rDJDcBXrDuU5+R0SjJN9lX+c01/ir+CO/+Av22VH
+	 IhKMuNJU1nVjzOf4Qs0CnmJO1Agzeuq+pf41gU0qIx0kZVSaUZP8PgXzpG1DUvfQ5T
+	 tEKNEHhRVFapOZ9HYeYzbMcUdO4z4haaEkuxerFOwwvIudLPkmCCeJH7xFouMvkMSN
+	 vxyKaJK7RdWzBkiXZutwKzS6BEbvbalzdMf7IcLXAlmYsUdV3bMPH0Bvg5HsxvLY1I
+	 YQNTiPp8yRiojv69hKqGKByUvhPveXVtbZNt3k8y4cwU5UAaAXnTQrNwSFhy3pxB+9
+	 fwaRNRHiXi1mQ==
+From: EuAndreh <[email-masked]>
+To: Richard Newman <[email-masked]>
+Cc: "~euandreh/[email-masked]" <~euandreh/[email-masked]>
+Cc: 
+Subject: Re: The database I wish I had
+In-Reply-To: <[email-masked]>
+References: <[email-masked]>
+ <[email-masked]>
+ <[email-masked]>
+ <[email-masked]>
+ <[email-masked]>
+ <[email-masked]>
+ <[email-masked]>
+ <[email-masked]>
+ <[email-masked]> <[email-masked]>
+Date: Thu, 03 Sep 2020 23:02:55 -0300
+Message-ID: <[email-masked]>
+MIME-Version: 1.0
+Content-Type: text/plain
+
+Here's what I was referring to when talking about "durable persistent
+data structures": https://www.youtube.com/watch?v=Cym4TZwTCNU
+
+From [email-masked] Sat Sep  5 09:07:44 2020
+Received: from box.euandre.org (box.euandre.org [46.101.160.115])
+	by mail-b.sr.ht (Postfix) with ESMTPS id 6A917FF1F5
+	for <~euandreh/[email-masked]>; Sat,  5 Sep 2020 09:07:43 +0000 (UTC)
+Authentication-Results: mail-b.sr.ht;
+	dkim=pass (2048-bit key) header.d=euandre.org header.i=@euandre.org header.b=grYKK93t
+Received: from authenticated-user (box.euandre.org [46.101.160.115])
+	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
+	(No client certificate requested)
+	by box.euandre.org (Postfix) with ESMTPSA id 92EB0FC83F;
+	Sat,  5 Sep 2020 06:07:40 -0300 (-03)
+DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=euandre.org; s=mail;
+	t=1599296862; bh=RjoAEI4vfrQ1VCfrySR+wMnwePv+AnUVT1ZbWcm8O1A=;
+	h=From:To:Cc:Cc:Subject:In-Reply-To:References:Date:From;
+	b=grYKK93td5WpquSSrkfBXBzUeJ25jSXwwjOE/x5c8rfR1DVAT1sDu7NfGVP+80pBW
+	 oG7IM1+Kpjz7PL15Bg01iDOXTLXMD/4GeikxqSUUG2I9Pj0NEO4eNzl6hNTfNDBuMA
+	 E7cz3qVVonEDHIYGlXvavHAWxWU6MXoSdDvEn0K2zNPU3qmo5PJ0vTKYNQTPkInY1N
+	 DRSUHv6DgfTUUYyrXdcw0rIPhQdNvobloBT29Q7LOtvXcBtiMhWlLMG30ifV+ETmhU
+	 h6AIw5myD387Nf7GRL6TfmtM1zWotf92jSop38NRnUCOJQ8iBpUrTt74qaSsuhcDbx
+	 p4C1GSTXxTj2g==
+From: EuAndreh <[email-masked]>
+To: Richard Newman <[email-masked]>
+Cc: "~euandreh/[email-masked]" <~euandreh/[email-masked]>
+Cc: 
+Subject: Re: The database I wish I had
+In-Reply-To: <[email-masked]>
+References: <[email-masked]>
+ <[email-masked]>
+ <[email-masked]>
+ <[email-masked]>
+ <[email-masked]>
+ <[email-masked]>
+ <[email-masked]>
+ <[email-masked]>
+ <[email-masked]> <[email-masked]>
+ <[email-masked]>
+Date: Sat, 05 Sep 2020 06:07:31 -0300
+Message-ID: <[email-masked]>
+MIME-Version: 1.0
+Content-Type: text/plain; charset=utf-8
+Content-Transfer-Encoding: quoted-printable
+
+> The obvious problem of always favouring the local timeline is that each
+> timeline would show different "primary" answers, but that's sounds
+> somewhat reasonable to me to enable the database to keep working while
+> offline, and while the conflict isn't resolved.
+
+Also a possible ergonomics layer that can be built on top of some lower lev=
+el
+primitive:
+
+Suppose we want to add a profile page to an app, but the user will be
+able to edit their profile while offline.
+
+Such profile page contains 3 parts: name, bio and links. Here's how it
+could look like:
+
+{:profile/name  "Foo Bar"
+ :profile/links #{{:link/title    "Personal webpage"
+                   :link/position 0
+                   :link/address  #uri "https://example.com"}
+                  {:link/title    "Mastodon profile"
+                   :link/position 1
+                   :link/address  #uri "https://example.org"}
+                  {:link/title    "Photo album"
+                   :link/position 2
+                   :link/address  #uri "https://example.net"}}
+ :profile/bio   "My lengthy bio (...)"}
+
+For each of those attributes, one could want to handle conflicts in a diffe=
+rent
+way:
+
+1. For the :profile/name, you could want to be fully conservative, and never
+   resolve any conflict whatsoever. If the user changes their name to "Foo =
+Bax"
+   in one device and to "Foo Baz" in another, you may choose to not solve t=
+his
+   conflict on the application. Instead you could just pick one of them to =
+show
+   to the user on their profile page, say "Foo Baz", but add an exclamation=
+ mark
+   icon =E2=9D=97=EF=B8=8F beside it. When the user touches the icon, the a=
+pp asks: "do you want
+   to keep 'Foo Baz' or would you like to change it to 'Foo Bax'?". This co=
+uld
+   be done multiple times for multiple concurrent conflicts, and when user
+   confirms their choice, that's a merge that resolves the conflict for all
+   instances when propagated.
+
+   The upside is that the app doesn't try to be smart about reconciling the
+   :profile/name, showing things like "Foo Baxz" or "Foo BazFooBax". The
+   downside is that if the app doesn't handle this explicitly, they'll be
+   perpetually inconsistent.
+
+2. For the :profile/links, specifically the :link/position attribute, you m=
+ight
+   try an intermediate approach. Say you want to handle reordering less
+   conservatively. Say the user changes the position of the "Photo album" l=
+ink
+   to 1 in a device, and to 0 on another device, and that is their actual
+   desired final position for this link. You could solve it by doing
+   something trivial, like picking the transaction UUID with the lowest
+   alphanumerical value. This gives you consistent choices across instances=
+, and
+   a potential nuisance for the user: if the winning final state is with "P=
+hoto
+   album" a 0, nothing needs to be done, and all instances agree with this.=
+ If
+   it isn't, then the user should reorder again, and all instances would st=
+ill agree.
+
+   The upside of this is that this conflict resolution strategy allows you =
+to
+   never stay conflicted for this particular attribute, and you could have =
+1:1
+   lookups back. The downside is that this assumes that the transactions th=
+at
+   are reordering links contain only :link/position attribute asserts.
+
+3. For the :profile/bio, you could be extremely optimistic, and try to comb=
+ine
+   all edits. This could be done by modeling it using commutative edits, si=
+milar
+   to how in [0], and let everything always converge. Maybe even adopt the
+   binary enconding of edits proposed in [1], and encode all of the edit hi=
+story
+   in the database itself. They talk a lot about the metadata overhead of t=
+he
+   edit history, and recording every edit as they come in would also add a
+   datom-related overhead, too. It might be more advantegeous to accumulate=
+ the
+   edit history, and when the app closes or a relevent event occurs, all ed=
+its
+   are combined into a single binary, and that would be a single datom entr=
+y.
+   This would remove the datom-related overhead while still persisting the =
+full
+   edit history.
+
+   The upside of this is that the text could always converge, you would get
+   1:1 lookups back and you wouldn't have to deal with diffs on a huge stri=
+ng,
+   neither would the user. The downside is that you'll need a layer on top =
+of the
+   transactions to actually derive the text, and querying would get trickie=
+r.
+   The database would give you the ordered list of all binary enconding thi=
+ngs,
+   and a function would turn that into text.
+
+   Another downside is that this is very much an open research problem, and
+   probably has a lot of maturity to be gained. It would be very powerful if
+   this could be built on top of some database primitive, without the datab=
+ase
+   having to explicitly support it.
+
+All of that to handle the conflicts of *each attribute*, and they're all
+the programmer's responsability to choose.
+
+If the database was versatile enough, an ideal world the programmer could w=
+rite:
+
+;; schema definition
+[{:db/ident :profile/name
+  :db/valueType :db.type/string
+  :db/cardinality :db.cardinality/one
+  :db/conflict-strategy :db.conflict-strategy/none}
+ {:db/ident :link/position
+  :db/valueType :db.type/long
+  :db/cardinality :db.cardinality/one
+  :db/conflict-strategy :db.conflict-strategy/simple-winner}
+ {:db/ident :profile/bio
+  :db/valueType :db.type/string
+  :db/cardinality :db.cardinality/one
+  :db/conflict-strategy :db.conflict-strategy/automerge}
+ ...]
+
+And be done with it.
+
+If those :db/conflict-strategy fields could be open and extensible, the dat=
+abase
+could learn new algorithms to deal with conflict without changing it's
+internals, and people could teach how translate the :profile/bio from
+:db.type/string the actual :db.type/bytes that would be actually stored.
+
+Also, I think this is what I was trying to say with "coupling two differente
+layers of the database". However, I haven't tested any of this yet.
+
+[0]: https://doi.org/10.1016/j.jpdc.2010.12.006
+[1]: https://github.com/automerge/automerge-perf/tree/master/columnar
+
+From [email-masked] Sat Sep  5 20:04:34 2020
+Received: from a58-61.smtp-out.us-west-2.amazonses.com (a58-61.smtp-out.us-west-2.amazonses.com [54.240.58.61])
+	by mail-b.sr.ht (Postfix) with ESMTPS id D36DAFF13B
+	for <~euandreh/[email-masked]>; Sat,  5 Sep 2020 20:04:32 +0000 (UTC)
+Authentication-Results: mail-b.sr.ht;
+	dkim=pass (1024-bit key) header.d=twinql.com header.i=@twinql.com header.b=qiZnhSkC;
+	dkim=pass (1024-bit key) header.d=amazonses.com header.i=@amazonses.com header.b=MQAUAk7Q
+DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple;
+	s=qvs2k2sd2ivtrflyu2nswmravuqizjml; d=twinql.com; t=1599336271;
+	h=Subject:From:To:Cc:Date:Mime-Version:Content-Type:Content-Transfer-Encoding:In-Reply-To:References:Message-Id;
+	bh=lPgy0nUilFxkCQGCuQxBgZW5Udn+JVGxl6bDms54Sk4=;
+	b=qiZnhSkCMRBNxcfwIf1Ecic54ZZmlMpioyl5vuE02tb5tFBnHk+bAFJcpIwJInMS
+	EdJiWj0xSnX7x4dBpD6JqwFHvQfx3wvs+tW6yxyAvjIQcFYb4qoC8ufiI33atGIOW/R
+	rrhRvIaDsmSyeT53ab+mytd/6WCxBnufMhHh5rrE=
+DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple;
+	s=hsbnp7p3ensaochzwyq5wwmceodymuwv; d=amazonses.com; t=1599336271;
+	h=Subject:From:To:Cc:Date:Mime-Version:Content-Type:Content-Transfer-Encoding:In-Reply-To:References:Message-Id:Feedback-ID;
+	bh=lPgy0nUilFxkCQGCuQxBgZW5Udn+JVGxl6bDms54Sk4=;
+	b=MQAUAk7QsnWFnb7gUs7do0tFKJ/ZEjJ+XsIFxth+m525oY5TIo0jyYTBcFxAeQU3
+	ruo+SqNRgJBJVtzqIHoghOuG8UqNq50MkyFLgv+agjPe95SKqQPIgnHeraWBXXccyjN
+	C42JTZ/q9uSxe/Kxy2m7iHmbpGF3+O3tP1PMHXLk=
+Subject: Re: The database I wish I had
+From: =?UTF-8?Q?Richard_Newman?= <[email-masked]>
+To: =?UTF-8?Q?EuAndreh?= <[email-masked]>
+Cc: =?UTF-8?Q?=7Eeuandreh/public-inbox=40lists=2Esr=2Eht?=
+  <~euandreh/[email-masked]>
+Date: Sat, 5 Sep 2020 20:04:31 +0000
+Mime-Version: 1.0
+Content-Type: text/plain; charset=UTF-8
+Content-Transfer-Encoding: quoted-printable
+In-Reply-To: <[email-masked]>
+References: <[email-masked]> 
+ <[email-masked]> 
+ <[email-masked]> 
+ <[email-masked]> 
+ <[email-masked]> 
+ <[email-masked]> 
+ <[email-masked]> 
+ <[email-masked]> 
+ <[email-masked]> 
+ <[email-masked]>
+X-Priority: 3 (Normal)
+X-Mailer: Amazon WorkMail
+Thread-Index: AQHWgHcPJ4udfXYrRaOwhoO7EcsDJQAErSAlAAfQ7+gAIgKd3wAzRhy+AHUc9SwA0i0RQQ==
+Thread-Topic: The database I wish I had
+X-Original-Mailer: Apple Mail (2.3445.104.14)
+X-Wm-Sent-Timestamp: 1599336269
+Message-ID: <[email-masked]>
+X-SES-Outgoing: 2020.09.05-54.240.58.61
+Feedback-ID: 1.us-west-2.An468LAV0jCjQDrDLvlZjeAthld7qrhZr+vow8irkvU=:AmazonSES
+
+Ah, thank you for the conversation refactor!=0D=0A=0D=0A> The wiki page y=
+ou pointed to reinforces that: it talks a lot about how=0D=0A> to coordin=
+ate attributes that changed their names, change in vocabulary,=0D=0A> sha=
+red ownership of changes in schema and even schema versioning!=0D=0A>=20=0D=
+=0A> All of them are the opposite of what I want, and they're "schema=0D=0A=
+> breakage", not "schema growth".=0D=0A=0D=0AMy two observations are thes=
+e:=0D=0A=0D=0AOn the one hand, front-end developers tend to think in term=
+s of change =E2=80=94 "this thing's title was X and is now Y" =E2=80=94 a=
+nd states =E2=80=94 "give me a list of things and their current titles so=
+ I can search them or show them in a table".=0D=0A =20=0D=0AThat view of =
+change extends to schema/data, because the schema stretches most of the w=
+ay towards the front-end itself, sometimes all the way to a "view model" =
+in the MVVM sense.=0D=0A =20=0D=0AAnd it involves migration, because no f=
+ront-end developer (or back-end developer, for that matter) wants to have=
+ to write queries that accommodate all of the different ways a particular=
+ concept might have been represented in the last ten years. Doing so is m=
+entally taxing, is very difficult to test, and results in wildly variable=
+ performance. They want to fix their data once, or define how to roll it =
+up into a single state once, and make sure the old representation never g=
+ets used again. And these representations do change, often significantly;=
+ making sure they don't requires tremendous up-front deep thought (for ex=
+ample, my modeling attempt for embedded videos), and results in something=
+ that is often normalized to the point of non-viability (already a concer=
+n with tuple stores).=0D=0A=0D=0AOn the other hand, storage/syncing devel=
+opers learn to love immutable logs, because it makes a ton of sense. Sync=
+able storage systems should be based on an immutable timeline of appended=
+ observations of the world.=0D=0A=0D=0A=0D=0AThere is obvious conflict be=
+tween these two. Resolving that conflict involves some complexity, and I =
+felt it better to address that complexity once, inside the storage system=
+=2E=0D=0A=0D=0AI didn't feel that it was worthwhile to build a storage sy=
+stem that required front-end developers to adjust their mental model so f=
+ar from what they're trying to achieve on a daily basis, because they hav=
+e a choice in which tools they use.=0D=0A=0D=0AFrom my perspective, the p=
+rimary goal has to be to give them a system they're willing to use (a cle=
+ar, current-focused schema and the ability to represent change), with the=
+ additional goal of a useful log and better formalisms underneath to supp=
+ort syncing. Putting it the other way 'round would simply lead to them us=
+ing SQLite or JSON files, omitting syncing, building bad ad hoc sync solu=
+tions, and otherwise continuing in the way that Mozilla had been doing th=
+ings.=0D=0A=0D=0AMentat, like Datomic, is a stateful layer on top of log-=
+structured data, with the current schema =E2=80=94 like the current state=
+ of the world! =E2=80=94 being materialized from the log.=0D=0A=0D=0A=0D=0A=
+> When more than one database instance can add data, the programmer needs=
+=0D=0A> to think of it as an open world, where new things are accrued, bu=
+t old=0D=0A> things don't disapper. Changing the vocabulary mid-air only =
+makes this=0D=0A> open world brittle so "migrations" can be performed.=0D=
+=0A=0D=0AI am a fan of open-world systems =E2=80=94 I spent 7+ years work=
+ing on semantic web stuff! =E2=80=94 but I'm not aware of any client appl=
+ications with any degree of adoption that use the open-world model. Havin=
+g multiple components share state doesn't require an open world per se.=0D=
+=0A=0D=0ABeing able to evolve the representation of existing data, just l=
+ike being able to 'change' the state of the world, is regarded as a *feat=
+ure* by developers, and they typically would rather sacrifice interoperab=
+ility than take on the burden of dealing with a pile of data that only ge=
+ts more crufty over time. In the 'schemaless' world (e.g., DynamoDB), dev=
+elopers take a mix of painful approaches (ranging from adding version tag=
+s that drive switches in application code, to doing manual migrations bet=
+ween tables) to clean up old data when the previous representation cannot=
+ stretch far enough.=0D=0A=0D=0AI tried with Mentat to strike a balance h=
+ere: to design a system that would allow multiple teams to interact, defi=
+ning their vocabularies in an ever-growing way, with specific transformat=
+ion steps as needed, and being able to detect and handle conflicting voca=
+bulary usage within that closed world. You can absolutely use Mentat as a=
+ system with no vocabulary modification ever, complicating your queries t=
+o accommodate, but it would not have met the needs of its audience:=0D=0A=
+=0D=0AD: "So, I realized that having a page -> visit timestamp property d=
+oesn't actually do what we want; we need to be able to mark each visit wi=
+th which device visited the page. How do I change the schema and the exis=
+ting data=3F"=0D=0AR: "You can't; make a new property. Every time you wan=
+t to retrieve the visits for the page, everywhere in your app, your queri=
+es need to look at both properties."=0D=0AD: "Can I write some function t=
+hat seamlessly transforms the old data to the new format and works with s=
+yncing=3F"=0D=0AR: "I wrote a doc about that, but we decided that purity =
+was better. You can write code to manually retract the old stuff, but oth=
+er syncing devices can still mess you up. Sorry."=0D=0A=0D=0AResolving co=
+nflicts and supporting sync was a core part of Mentat's charter. The last=
+ thing I wanted was for developers to think the system was hard to use, o=
+r to build a second storage layer on top to maintain a snapshot of the cu=
+rrent data in an app-centric format.=0D=0A=0D=0A=0D=0A> You mentioned erg=
+onomics thrice on your message, and I believe that your=0D=0A> insistence=
+ of it is a sign of the coupling of two different layers of=0D=0A> the da=
+tabase, and makes ease of use drive the fundamentals.=0D=0A=0D=0AYes, exa=
+ctly. The goal of Mentat wasn't to build a pure log-structured store: it =
+was to build a store with a developer interface that wasn't too alien to =
+an engineer who might store data in JSON or SQLite, but with internals th=
+at made sharing data, syncing data, and evolving data easier and safer.=0D=
+=0A=0D=0A=0D=0A> I also think=0D=0A> that your statement on conflict reso=
+lution being a requirement is not a=0D=0A> good idea. Let me expand on th=
+at.=0D=0A>=20=0D=0A> I'd argue that conflict resolution is inherit comple=
+xity [1] of creating=0D=0A> a decentralized client-side application that =
+can sync data.=0D=0A>=20=0D=0A> That means that the responsability to sol=
+ve conflicts is not the=0D=0A> database's, but the application's responsa=
+bility, and making decisions=0D=0A> of conflict resolution on the databas=
+e limits its application, and I=0D=0A> wouldn't consider it to be general=
+ purpose.=0D=0A=0D=0AAh, here's an interesting point.=0D=0A=0D=0AThere ar=
+e two positions here: am I vending a database, or am I vending a storage =
+solution for a client app=3F=0D=0A=0D=0ABuilding Firefox, Firefox for And=
+roid, and Firefox for iOS convinced me that implementing a synchronizatio=
+n solution that is not tightly coupled with the storage layer =E2=80=94 t=
+hat is, one that is just another consumer of the same storage interface t=
+hat the rest of the app uses =E2=80=94 is difficult and error-prone. Sync=
+ needs to see writes as they happen, and synchronization has particular i=
+mplications for how data is stored.=0D=0A=0D=0AI also found that offloadi=
+ng the difficulties in syncing =E2=80=94 conflict resolution, tracking ch=
+ange, merging identities, etc. =E2=80=94 to application developers leads =
+to worse outcomes.=0D=0A=0D=0AMy feeling is that it should be possible fo=
+r the application to educate the storage layer about how to handle most o=
+f the conflicts it encounters, and to model its data in a way that makes =
+conflicts meaningful and minimal, without sacrificing the kinds of data r=
+epresentations with which developers feel comfortable.=0D=0A=0D=0A=0D=0A>=
+ When two offline database instances change the same field, there's no=0D=
+=0A> escaping from the bottom reality, which is: there is a conflict, and=
+ it=0D=0A> needs to be resolved somewhere. There's no layer of ergonomics=
+ that can=0D=0A> change this fact, and picking a strategy for conflict re=
+solution is=0D=0A> undoubtly domain-specific.=0D=0A=0D=0AThe approach I w=
+as taking for syncing in Mentat involved rewriting (or permanently associ=
+ating) identifiers based on uniqueness constraints. That is, as a develop=
+er I could transact a schema fragment:=0D=0A=0D=0A=0D=0A  {:history/url :=
+db/unique :db.unique/value}=0D=0A=0D=0A=0D=0Aand thereafter say things li=
+ke=0D=0A=0D=0A=0D=0A  {:history/url "https://foo.com/"=0D=0A   :history/t=
+itle "Today's title"}=0D=0A=0D=0A=0D=0AIf the log already contains this, =
+with a local ID:=0D=0A=0D=0A=0D=0A  [1234 :history/url "https://foo.com/"=
+]=0D=0A=0D=0A=0D=0Athe new transact would add to the log something like:=0D=
+=0A=0D=0A=0D=0A  [1234 :history/title "Today's title"]=0D=0A=0D=0A=0D=0AA=
+nother device can record similar data:=0D=0A=0D=0A=0D=0A  {:history/url "=
+https://foo.com/"=0D=0A   :history/title "A different title"}=0D=0A=0D=0A=
+=0D=0Awhich would transact=0D=0A=0D=0A=0D=0A  [4567 :history/title "A dif=
+ferent title"]=0D=0A=0D=0A=0D=0AA query on either platform can omit the I=
+D, using a lookup ref instead:=0D=0A=0D=0A=0D=0A  [[:history/url "https:/=
+/foo.com/"] :history/title =3Ftitle]=0D=0A=0D=0A=0D=0AIn both cases we ge=
+t a 1:1 association between a URL and a 'page'. The use of lookup refs an=
+d the schema definition allows the developer to use the [attribute, URL] =
+pair as an external identifier in this context, while using an efficient =
+identifier internally.=0D=0A=0D=0AWhen the two transaction logs are merge=
+d, it follows from the schema definition that B(4567) =3D=3D=3D A(1234), =
+and that identity can be used in the store to zip the two graphs together=
+ in a way that is seamless to the developer: the 1:1 association is prese=
+rved. In practice we found that holding on to those numeric IDs, which ca=
+n change during syncing, in application code was rarely or never necessar=
+y.=0D=0A=0D=0AThis example is what I mean by 'teaching' the database abou=
+t domain definitions of identity. In a relational database, the schema is=
+ largely used to define structure and constraints. The way it feels to wr=
+ite a schema in Datomic/Mentat is much closer to 'explaining' how various=
+ kinds of new assertions and sync behaviors will interact with the existi=
+ng data =E2=80=94 the schema constraints are so often used to drive looku=
+p refs and updates, and of course there's no tabular structure.=0D=0A=0D=0A=
+This can get a merge all the way down to "this page's title changed from =
+X to Y at time T on one device and from X to Z on another at time Y; whic=
+h do you want=3F" which is _also_ something that you can teach the databa=
+se to automatically resolve =E2=80=94 annotate the schema to say that a c=
+onflict takes the time-latest value.=0D=0A=0D=0A=0D=0A> They may be the=0D=
+=0A> same, and they may not be the same, but saying just "the README file=
+" is=0D=0A> an incomplete depiction of it: you either say "my README file=
+" or talk=0D=0A> about both of them, which do coexist.=0D=0A=0D=0AI actua=
+lly used this as an example when explaining Mentat =E2=80=94 timelines ar=
+e branches, but the `datoms` table (the data you actually work with) is a=
+ Git "working tree". Developers (and end users) need a working tree and a=
+ single state of the world.=0D=0A=0D=0A=0D=0A> The point is: the database=
+ is not in a position to make this type of=0D=0A> decision. Only the prog=
+rammer is, and they must be aware of it.=0D=0A=0D=0AWhich kinds of progra=
+mmers are your audience=3F=0D=0A=0D=0AIn my experience, most developers a=
+re (a) not aware of the difficulties here, (b) do not have the time to di=
+g in to the details, and (c) would rather take reasonable defaults (e.g.,=
+ latest wins) or even data duplication or corruption than sacrifice simpl=
+icity to get it right. That is: if you presented them with the option of =
+a distributed sync system where they had to handle multiple timelines, us=
+e pluggable conflict-resolution modules, and write complicated queries to=
+ deal with that complexity, they are instead going to use JSON files.=0D=0A=
+=0D=0AIt would not be inaccurate to say that Mentat is/was _already_ way =
+too much of a shift for most developers; a tuple store with a log is expe=
+nsive and confusing in ways that a key-value store is not.=0D=0A=0D=0A=0D=
+=0A> You said that:=0D=0A>=20=0D=0A>> the goal of a database is to help m=
+e efficiently, accurately, and=0D=0A>> ergonomically represent my domain=20=
+=0D=0A>=20=0D=0A> For me, the crucial part is the accurately comes first,=
+ and ergonomics=0D=0A> comes second. No accuracy should be sacrificed in =
+name of ergonomics,=0D=0A> since the inherent complexity is already too b=
+ig to be dealt with.=0D=0A=0D=0ATo broaden your point a little bit, my po=
+sition on that has shifted over the years, after working on systems that =
+prioritized correctness over adoption or user experience (e.g., Firefox S=
+ync's perfect crypto, which caused _so many users_ to lose data because t=
+hey weren't prepared to manage long-lived secure keys, and couldn't under=
+stand the setup flow=E2=80=A6).=0D=0A=0D=0AA system that is perfect but t=
+hat people =E2=80=94 developers or end users =E2=80=94 don't use it is _w=
+orse_ than a system that makes compromises and gets some adoption. As Guy=
+ Steele put it about Java:=20=0D=0A=0D=0A  "We were not out to win over t=
+he Lisp programmers; we were after the C++ programmers. We managed to dra=
+g a lot of them about half-way to Lisp."=0D=0A=0D=0AI think a brief tange=
+nt into ontology would be interesting. I know your point here is about ac=
+curately modeling change, but a similar set of positions exists when mode=
+ling information itself.=0D=0A=0D=0AThe 'correct' way to model the world =
+is as a time-series sequence of independent observations, with everything=
+ else being materialized from it through a carefully designed ontological=
+ lens, such that one doesn't have to accidentally make false statements i=
+n order to say something true.=0D=0A=0D=0AAt one end is the kind of ontol=
+ogy I drew on a whiteboard once, nearly twenty years ago, in which the co=
+ncept of a music "album" is broken down into issuances, releases, track o=
+rders, renditions, recordings, performers, names, identities, collectives=
+, etc. =E2=80=94 it's incredibly complex how one recording of a group of =
+people in a room might be recorded multiple times, remastered over the ye=
+ars, edited for various releases, and appear on multiple formats in diffe=
+rent versions, and how different versions of songs might have different b=
+ands or people playing different parts.=0D=0A=0D=0AMusic nerds really car=
+e about this, though perhaps not down to the level of detail to which I m=
+odeled it.=0D=0A=0D=0AAt the other end is how this kind of metadata is re=
+ally represented in the real world.=0D=0A=0D=0ATitle: "Landslide"=0D=0AAr=
+tist: "Fleetwood Mac"=0D=0APlay Count: 15=0D=0ATrack Number: 2=0D=0A=0D=0A=
+The theoretically pure way to model all this stuff makes it possible to s=
+ay "this recording of Landslide is of a performance by a band named Fleet=
+wood Mac =E2=80=94 the particular instance of Fleetwood Mac that included=
+ Stevie Nicks, not the one that included Bob Welch". Let's not even get i=
+nto the fact that there are two albums recorded by Fleetwood Mac titled "=
+Fleetwood Mac" =E2=80=94 (title, artist) isn't compound-unique!=0D=0A=0D=0A=
+But no user wants to say "Alexa, play Fleetwood Mac" and have her reply "=
+do you mean the band 'Fleetwood Mac' that recorded songs released by Warn=
+er Bros, or the band 'Fleetwood Mac' that recorded songs released by Blue=
+ Horizon"=3F They will exchange a certain amount of precision to cast a w=
+ider and more usable net.=0D=0A=0D=0APut bluntly: apps have trouble getti=
+ng the darn play counts on two devices to add up to the same number, let =
+alone this degree of precision. I would rather make it _possible_ for the=
+m to model data at a reasonable degree of accuracy, with no more difficul=
+ty than they would have writing a CREATE TABLE statement, and help them g=
+et their play counts right when they sync two devices, rather than force =
+them to take an extreme position on accuracy and leave them to figure out=
+ how to list how many times the user played a song.=0D=0A=0D=0A=0D=0A> Th=
+is way, how can one know the current state of the database=3F=20=0D=0A>=20=
+=0D=0A> From the first root datom, the database would rollup every datom =
+taking=0D=0A> the timelines and merges in consideration, and it would be =
+the sum of=0D=0A> data from all timelines. When a conflict exists, the lo=
+cal timeline is=0D=0A> always picked so the database can still serve writ=
+es, and the conflict=0D=0A> would be presented when serving reads.=20=0D=0A=
+>=20=0D=0A> The obvious problem of always favouring the local timeline is=
+ that each=0D=0A> timeline would show different "primary" answers, but th=
+at's sounds=0D=0A> somewhat reasonable to me to enable the database to ke=
+ep working while=0D=0A> offline, and while the conflict isn't resolved.=0D=
+=0A=0D=0AMy view is that end users expect all of their devices to present=
+ the same canonical view of their data, and manual conflict resolution sh=
+ould be rare/unnecessary/modeled as an explicit change or an explicit sta=
+te (a write like any other).=0D=0A=0D=0AFurther, developers expect querie=
+s to be fast and writes and queries to behave the same when devices are s=
+ynced, regardless of which device performs the write or runs the query.=0D=
+=0A=0D=0AThat makes it difficult to either diverge between devices for lo=
+ng periods, or to expect routine manual involvement in conflicts.=0D=0A=0D=
+=0AIt seems to me that the longer these distinct timelines exist, the mor=
+e conflicts will breed, as writes reflect different local states, and so =
+the difficulty of presenting a coherent view increases over time.=0D=0A=0D=
+=0AI expected probably one place to do manual conflict resolution in Fire=
+fox: password management. Even there it might be better to turn every con=
+flict into an explicit "previous passwords" entry, or to expose the log a=
+s a dynamic list of previous passwords =E2=80=94 that data is valuable fo=
+r detecting website breaches. That is, it's better to surface the alterna=
+tive data as a normal change and reach agreement with other devices, rath=
+er than preserving some kind of pending conflict.=0D=0A=0D=0ABy modeling =
+rebases or merges as a coherent set of retractions of conflicting data, i=
+t's still possible to implement log-oriented features like querying *ever=
+y* title a page has ever had, but eagerly reach a shared timeline. ("Show=
+ me that news article I remember reading".)=0D=0A=0D=0A=0D=0A> This also =
+means that "sync" are two distinct things: exchanging timeline=0D=0A> dat=
+a, and reconciling that data locally.=0D=0A=0D=0AAgreed.=0D=0A=0D=0A=0D=0A=
+> The only part that's online is=0D=0A> the "exchange", and instances res=
+olve conflict locally. This way, the=0D=0A> database is always available,=
+ even when offline, and even when conflict=0D=0A> is present.=0D=0A=0D=0A=
+The difference IMO is that the goal of conflict resolution is that every =
+device converges on the same canonical timeline, and does so as quickly a=
+s possible.=0D=0A=0D=0A=0D=0A> But how would queries over history behave=3F=
+ No clue yet =F0=9F=98=AC.=0D=0A>=20=0D=0A> I tried not to sound too hand=
+ wavy, and I've only prototyped a small=0D=0A> part of this in the past, =
+so there are probably many more problems with=0D=0A> this approach. I don=
+'t claim this is a sound solution yet, but it does=0D=0A> handle some of =
+the problems that Mentat faced in a different way. I'd be=0D=0A> interest=
+ed to see you poking holes on it.=0D=0A=0D=0AIt's a really difficult prob=
+lem! My suspicion is that querying history is either broadly a union ("th=
+ese pages all had this title at some point") or very much an expert featu=
+re, analogous to browsing your own 'undo' tree. But note that the union a=
+pproach can benefit from identifier rewriting to represent derived equali=
+ty at a particular point in time, even in the presence of conflicts.=0D=0A=
+=0D=0A=0D=0A> When combined with strict schema growth, new data always co=
+mes after=0D=0A> it's schema declaration, so the application doesn't have=
+ to coordinate=0D=0A> around distributed schema evolution.=0D=0A=0D=0AYes=
+, that's broadly the position I took with Mentat: by definition, data on =
+each timeline comes after its schema declaration, so we have an ordered m=
+echanism for merging and also for migrating data as the schema evolves.=0D=
+=0A=0D=0A=0D=0A> TIL: I didn't know you worked on AllegroGraph.=0D=0A=0D=0A=
+Seems like every few years I end up back on data modeling and graph stora=
+ge :D=0D=0A=0D=0A=0D=0A> If the database was versatile enough, an ideal w=
+orld the programmer could write:=0D=0A>=20=0D=0A> ;; schema definition=0D=
+=0A> [{:db/ident :profile/name=0D=0A>  :db/valueType :db.type/string=0D=0A=
+>  :db/cardinality :db.cardinality/one=0D=0A>  :db/conflict-strategy :db.=
+conflict-strategy/none}=0D=0A=0D=0AYes, that's pretty much where we were =
+headed with Mentat. My key observations were that (a) you need really goo=
+d tree-zipping, and frequent syncs, to minimize conflicts; (b) tree-zippi=
+ng and merges practically require that internal identifiers be rewritten =
+when identity is determined; (c) it is undesirable to stop the world, and=
+ so when a conflict is detected you end up with continuing divergence whi=
+ch will be even harder to merge later; (d) because this is a single graph=
+, a conflict _must_ be resolved in some way before you can merge two head=
+s; (e) lossy merges (squash then rebase or squash then merge) can help.=0D=
+=0A=0D=0A
+
+From [email-masked] Sat Sep  5 22:51:10 2020
+Received: from box.euandre.org (box.euandre.org [46.101.160.115])
+	by mail-b.sr.ht (Postfix) with ESMTPS id D9930FF12D
+	for <~euandreh/[email-masked]>; Sat,  5 Sep 2020 22:51:09 +0000 (UTC)
+Authentication-Results: mail-b.sr.ht;
+	dkim=pass (2048-bit key) header.d=euandre.org header.i=@euandre.org header.b=w2kZ+s6p
+Received: from authenticated-user (box.euandre.org [46.101.160.115])
+	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
+	(No client certificate requested)
+	by box.euandre.org (Postfix) with ESMTPSA id 2130DFC822;
+	Sat,  5 Sep 2020 19:51:06 -0300 (-03)
+DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=euandre.org; s=mail;
+	t=1599346268; bh=PWBfaRn3jOle7Z3Lc1INXcX50+v5RZ5HNwANJeQm8Js=;
+	h=From:To:Cc:Cc:Subject:In-Reply-To:References:Date:From;
+	b=w2kZ+s6pUTUZJSwLwi3NmcFMqy3a8mBj7PuNX5oJD9O3R4MZM4Td6AZVsjHQQLRVG
+	 IRQHyT9t8jngQXu0mkRtU2F4IZoKEPMQAVrRBOBq+f4+J1OR5zynUcT7UKE+hmWDtr
+	 lJqDHIbUEzNoS8yLpWXZqib6kwHFqF8B/TLtbl4scllrpf+Q0eq7ZbDpjYHfotNt7C
+	 xZ8bFSDeeSP6Ec93nh82bOpXqAEhaTA/vf7qh7T1qWKdjmFvDV3WldUtxjmMcCDvgS
+	 PYZ5Laxlrvoq8li4XF55rArXMOb8gftJMy3QY0rPRmyvpFLKFFB423i+NnIctnu93o
+	 ewcl54UnP6Iaw==
+From: EuAndreh <[email-masked]>
+To: Richard Newman <[email-masked]>
+Cc: "~euandreh/[email-masked]" <~euandreh/[email-masked]>
+Cc: 
+Subject: Re: The database I wish I had
+In-Reply-To: <[email-masked]>
+References: <[email-masked]>
+ <[email-masked]>
+ <[email-masked]>
+ <[email-masked]>
+ <[email-masked]>
+ <[email-masked]>
+ <[email-masked]>
+ <[email-masked]>
+ <[email-masked]>
+ <[email-masked]>
+ <[email-masked]>
+Date: Sat, 05 Sep 2020 19:50:56 -0300
+Message-ID: <[email-masked]>
+MIME-Version: 1.0
+Content-Type: text/plain; charset=utf-8
+Content-Transfer-Encoding: quoted-printable
+
+I see what to mean by being accessible, and it does make sense.
+
+If done properly this might be the reality, where nobody chooses to
+handle conflicts manually because they'd rather have a consistent view
+by picking one of the possible algorithms that are available for them to
+pick for each attribute. I'll experiment with this a bit, and if=20
+feasible, the "perpetual inconsistency" might be an internal view, and
+something available for those who choose to customize their conflict
+resolution strategy. This would be ideal, and perfectly aligned with
+what you said on "the application educating the storage layer about how
+to handle most of the conflicts it encouters". The unique URL you showed
+is a tempting challenge to reach, and the domain identity being
+teachable sounds indeed really powerful.
+
+This also means that with those defaults the desired 1:1 lookups would
+be restored. I don't think that doing manual resolution for each
+attribute is desired, and I wouldn't impose this on the programmer. In
+fact, I'd want to use it myself most of the time.=20
+
+That would also mean an intersection between the two conflicting forces
+you mentioned, where both sides would get what they wanted: a developer
+could use it to quickly prototype something by picking the suggested
+defaults and not finding many surprises, while allowing another
+developer to fine tune their strategy when desired.
+
+It's a database, with a robust storage solution inside it, but primarily
+a database.
+
+If not possible, I tend to err on the side of less ease. Not because I
+want to be a purist, but because I want to do this for myself =F0=9F=98=AC.=
+ The
+title of the article was actually honest on this, and I've actually had
+this exact though a few times: "I'm struggling with this database, but
+there must be more people like me with the same struggles. I bet someone
+has already solved, or will solve soon enough". I'm the target audience,
+and if I do something that solves my problem, than I would consider it a
+success :). It may not be widespread but I'd feel better knowing that
+I've built something that works over something that works most of the
+time. Again, I don't want to be a purist, I just want to create a
+database that I'd want for myself, and whoever sees themselves on the
+same position could benefit. If ease can be added on top of it with some
+choices made for you, great! From your comment, I think you'd agree with
+that last part.=20=20=20=20=20
+
+I'm not trying to convert anyone on this, though. I'd be happy to share
+the code and learning, and welcome those who are interested and
+attracted to this, but broad adoption and world domination are not
+goals.
+
+The applications that I tried to write using the existing solutions were
+just good enough to get me started, but I always felt abandoned when I
+wanted to evolve it somehow, or add some specific functionality without
+losing data, or some similar trade-off. Most of those times I'd rather
+deal with that complexity than choose an easier path.
+
+Today whenever I see a client-side database that syncs, I go to the
+documentation and search for "conflict". I'll either get a page with
+some in-depth explanation, or there will be a list of choices they made
+for me with no way out of it, like last write win, and permanent
+deletion tombstones. Sure those are valuable, but I'd rather be able to
+choose them myself. I already know by now that I will need some tweaked
+conflict resolution someday, and they won't be able to provide it for
+me without an ad-hoc approach.=20
+
+If someone would value something more familiar and more
+fragile/opinionated over something very different and more sound, then
+maybe they're correct to pick SQLite or some JSON files, and having data
+loss in some edge cases would be just fine in the problem they're
+solving. I'd encourage them to stick to SQLite, even, its a great piece
+of software.=20
+
+Also, there's something more profound on this, I believe:=20
+
+Having a solid foundation could potentially help people reason about the
+problem with good tools, instead of looking like a solid foundation that
+will crack if you step on the wrong place. Instead of helping, it could
+be hurting the application, and the superficial ease that many solutions
+offer sometimes feel more like a trap for me to figure out myself later.
+To me it's like someone saying: "have this relational fully ACID
+performant database!", and only later I realize that it isn't actually
+ACID by default, and if all change all the knobs to make it ACID, it
+becomes unusuably slow (or maybe even those knobs aren't there at all).=20
+
+I'd argue that such approach may look weirder and more foreign to any
+developer, but it would actually be "easier". This reminds me of
+"Simple Made Easy" [0]
+
+[0]: https://www.infoq.com/presentations/Simple-Made-Easy/
+
+Back on schema growth: one doesn't have to make a perfect plan of the
+data models so no mental tax gets to them later.
+
+The solution you mentioned sounds just fine: write a function that
+transforms the old data and feed it to the new attribute, and retract
+everything else. From that point on, you've effectively migrated the
+data, and you'll only need to consider the historical schema when doing
+historical queries. Nothing broke on the process, and people might just
+forget about how it used to be in the past. No mental tax, unless you
+want to dig history.
+
+The existence of the new attribute serves the same purpose that the
+version tag does.
+
+You still have an immutable log, and you've evolved your schema to
+better represent the domain. I don't see how those conflict here. The
+biggest objection would probably be the "wasted storage" of the old
+data.=20
+
+But changing schema types and versioning them is something else. It
+might even be possible to implement, doing rollups of the schema just
+like what's done with the rest of the data like you mentioned, but I
+just don't like this idea overall, I think it has brittleness written
+all over it. Maybe I'm just being grumpy.=20=20=20
+
+About your album ontology: I like it! Coincidence or not, I did 5 years
+of music major before dropping out of college, so I'd consider myself a
+"music nerd"! But I agree that one needs the database to help them model
+the domain in a useful way, and I wouldn't force everybody to have a
+particular view of it. Ultimately the database should be a useful tool
+to solve a problem.=20
+
+From [email-masked] Tue Oct  6 07:53:02 2020
+Received: from mx.h4ck.space (mx.h4ck.space [159.69.146.50])
+	by mail-b.sr.ht (Postfix) with ESMTPS id 57F5AFF0FF
+	for <~euandreh/[email-masked]>; Tue,  6 Oct 2020 07:53:02 +0000 (UTC)
+Authentication-Results: mail-b.sr.ht;
+	dkim=pass (1024-bit key) header.d=notmuch.email header.i=@notmuch.email header.b=iFvGMOSq
+Date: Tue, 6 Oct 2020 09:52:58 +0200
+DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=notmuch.email;
+	s=mail; t=1601970781;
+	bh=RfiC8xpAHKtbuatmn10k3tqQKuSGYPCep+XBVi1qDe0=;
+	h=Date:From:To:Subject;
+	b=iFvGMOSqx3JpTHGUue5rMzrKPzZiiAFpjJBgA/1P8lgnk/7gBNl51mU5M7m42XXS2
+	 cGLwhRQAF3y/x9IoyNGAAUy/hRg+u1VGeRPoV+wFWlitZNgH6dWMuos8XNxLv7Z+dr
+	 X1MbaF5gSIQaRehQyg7oyDhd9TO2WcKj6jRCia60=
+From: [email-masked]
+To: ~euandreh/[email-masked]
+Subject: crate2nix
+Message-ID: <20201006075258.52tz5u5a36fpbfwy@wrt>
+MIME-Version: 1.0
+Content-Type: text/plain; charset=utf-8
+Content-Disposition: inline
+
+Hi,
+
+Just saw your crate2nix code and it looks very nice and clean!
+
+Just a heads-up, since you might not aware of all the alternatives out
+there, there is a version from Eelco that basically does the same as
+yours but also handles git dependencies. I did blog about that briefly
+last month [1].
+
+I am curious if you were aware of that and/or naersk [2] as both of them
+also come with the promise of no generated files and no *2nix commands
+to execute.
+
+
+Regards,
+
+Andi
+
+
+
+[1]: https://andreas.rammhold.de/posts/nix-rust/#import-cargohttpsgithubcomedolstraimport-cargo
+[2]: https://github.com/nmattia/naersk
+
+From [email-masked] Tue Oct  6 09:09:49 2020
+Received: from box.euandre.org (box.euandre.org [46.101.160.115])
+	by mail-b.sr.ht (Postfix) with ESMTPS id 9ED40FF107
+	for <~euandreh/[email-masked]>; Tue,  6 Oct 2020 09:09:48 +0000 (UTC)
+Authentication-Results: mail-b.sr.ht;
+	dkim=pass (2048-bit key) header.d=euandre.org header.i=@euandre.org header.b=sXxvO0HC
+Received: from authenticated-user (box.euandre.org [46.101.160.115])
+	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
+	(No client certificate requested)
+	by box.euandre.org (Postfix) with ESMTPSA id DF0A2FC991;
+	Tue,  6 Oct 2020 06:09:45 -0300 (-03)
+DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=euandre.org; s=mail;
+	t=1601975387; bh=R37p/WxSUhw6s3+NsQXWU1P6/NnfwRrO7khrmNnifPs=;
+	h=From:To:Cc:Subject:In-Reply-To:References:Date:From;
+	b=sXxvO0HCnYXM8ovt5uOk5bZ/3ZTzOR9HVWXgURy/MicF6fwxTJJfOMCC4+b/ZanpX
+	 hcQc+ZSsiOgevD6oR8MZE630bHDpXNs/FskZFM0gRAwG68dfJ/lpj2ogo5GQ9fEbKy
+	 oH0MD7Yd2suVGevV9oULcadlt5g8Y75Yuazzwx0HOoF12A4BQWtRrBsBV/Qp7yNoZm
+	 xjNIf9MwbD2MsEWlF/K9JeDEdsHazcpQh2RiVd1/G497B9F4yEbE5HmkOP5aGP5jUv
+	 yudSn0cBCWQ7S/FXeHXUyhZeebiQnFWkpJeIfUSEhGSOoim1GE/gMi1U7rA1qsWaoy
+	 jVVWomlh//r5w==
+From: EuAndreh <[email-masked]>
+To: [email-masked], ~euandreh/[email-masked]
+Cc: 
+Subject: Re: crate2nix
+In-Reply-To: <20201006075258.52tz5u5a36fpbfwy@wrt>
+References: <20201006075258.52tz5u5a36fpbfwy@wrt>
+Date: Tue, 06 Oct 2020 06:09:27 -0300
+Message-ID: <[email-masked]>
+MIME-Version: 1.0
+Content-Type: text/plain; charset=utf-8
+Content-Transfer-Encoding: quoted-printable
+
[email-masked] writes:
+
+> Hi,
+
+Hi there!
+
+> Just saw your crate2nix code and it looks very nice and clean!
+
+=E2=98=BA=EF=B8=8F
+
+> Just a heads-up, since you might not aware of all the alternatives out
+> there, there is a version from Eelco that basically does the same as
+> yours but also handles git dependencies. I did blog about that briefly
+> last month [1].
+
+I wasn't aware of it. At a first glance, it looks remarkably similar!
+Maybe I'm on to something =F0=9F=98=AC. I'll try it later, thanks for the l=
+inks.
+
+> I am curious if you were aware of that and/or naersk [2] as both of them
+> also come with the promise of no generated files and no *2nix commands
+> to execute.
+
+I did try naersk. It was the best one I found, in fact. It does use
+Cargo under the hood to drive things, but it doesn't expose the
+environment it builds. With it you can parameterize the Cargo commands,
+but not run it by yourself.
+
+In fact, there's a snippet that I copied directly from naersk[0]. I
+didn't mention it on the README, only on the commit log.
+
+[0]: https://git.euandreh.xyz/cargo2nix/commit/?id=3Dc75e7e2d3ee7f7e074ea9f=
+8e5b959d6395338acc
+
+From [email-masked] Tue Dec 22 16:29:11 2020
+Authentication-Results: mail-b.sr.ht; dkim=none 
+Received: from se15-1.privateemail.com (se15-1.privateemail.com [198.54.127.72])
+	by mail-b.sr.ht (Postfix) with ESMTPS id B579CFF0D7
+	for <~euandreh/[email-masked]>; Tue, 22 Dec 2020 16:29:11 +0000 (UTC)
+Received: from new-01.privateemail.com ([68.65.122.22])
+	by se15.registrar-servers.com with esmtpsa (TLSv1.2:AES128-GCM-SHA256:128)
+	(Exim 4.92)
+	(envelope-from <[email-masked]>)
+	id 1krkX5-0004vZ-Jo
+	for ~euandreh/[email-masked]; Tue, 22 Dec 2020 08:29:10 -0800
+Received: from MTA-08-1.privateemail.com (unknown [10.20.147.18])
+	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
+	(No client certificate requested)
+	by NEW-01.privateemail.com (Postfix) with ESMTPS id B5E4C60041
+	for <~euandreh/[email-masked]>; Tue, 22 Dec 2020 16:29:05 +0000 (UTC)
+Received: from MTA-08.privateemail.com (localhost [127.0.0.1])
+	by MTA-08.privateemail.com (Postfix) with ESMTP id 91A866006F
+	for <~euandreh/[email-masked]>; Tue, 22 Dec 2020 11:29:05 -0500 (EST)
+Received: from DESKTOP1997JCK (unknown [10.20.151.208])
+	by MTA-08.privateemail.com (Postfix) with ESMTPA id F339F6004A
+	for <~euandreh/[email-masked]>; Tue, 22 Dec 2020 16:29:04 +0000 (UTC)
+From: <[email-masked]>
+To: <~euandreh/[email-masked]>
+Subject: Progress on 'The database I wish I had'
+Date: Tue, 22 Dec 2020 17:29:03 +0100
+Message-ID: <[email-masked]>
+MIME-Version: 1.0
+Content-Type: text/plain;
+	charset="us-ascii"
+Content-Transfer-Encoding: 7bit
+X-Mailer: Microsoft Outlook 16.0
+Thread-Index: AdbYfzkizJpIDb+oQYeLozSx36pHnw==
+Content-Language: no
+X-Virus-Scanned: ClamAV using ClamSMTP
+X-Originating-IP: 68.65.122.22
+X-SpamExperts-Domain: o1.privateemail.com
+X-SpamExperts-Username: out-01
+Authentication-Results: registrar-servers.com; auth=pass (plain) smtp.auth=[email-masked]
+X-SpamExperts-Outgoing-Class: unsure
+X-SpamExperts-Outgoing-Evidence: Combined (0.57)
+X-Recommended-Action: accept
+X-Filter-ID: Pt3MvcO5N4iKaDQ5O6lkdGlMVN6RH8bjRMzItlySaT8/DiuTODf6wrobfALkT2HqPUtbdvnXkggZ
+ 3YnVId/Y5jcf0yeVQAvfjHznO7+bT5wWykU6VVuoVZYG6cNLn3c4E9P7Wdo8qPSPEXiwPI9MYyaH
+ o/9wB5YLuaEkARV33qkqPXaGse7C78HuChWU138c4pWx7WxWFZpsjT/483ewgsz4HwYGTNBY7kFu
+ mOeXVIneVHayp+oaTkTmqro7//B+gpZK9MmGPsPG1IFhoAOccOwmVIoqJOmt2f1p/FeBBYf0It0W
+ sNcsjnTm+4g485SS8OTf6I2qJgqFlok7aJfKCMNBaM6YjzsuHtjxewJtlZX1S4orbC27wIbqb1J8
+ wyDQ43DG7JDHGkUFlLvaUAJ9EsZg/9mPb0OnPglkYW72F/Odw+XGlIW1bb6iLQaqIs5BLfTttFI5
+ MCNL/izpcNORuAUvossjam0/HVDFzCeLVAjI+ht+2XwDC3Hj+WjRz7duYp7tG8ix8mThYIoihNfI
+ Pg3O+76Hxe6qkGjOqN7KXlq8vamDuW8qjcPFOC9xtigG+MEYMo0NDeyXVO3tNSgVgckOnz0mswVX
+ ZkpFVe5ZgGA0MLjqgL337Aba0izaNK+kAlchHs+Xxee6P8LI+rvlGOeqKR5Od1/Dbcw8eAyTq1JN
+ 7i5WVjtE146jFUwt8Q10AQGCef4uNE610hEkYogGmsvnLjiRqfPZZiHIniXn+ZdiHE9X9GlwwRvN
+ 2KIObvd055RG8/vjrz5/4Qa1aRJkvZZIZN1+KRx7WOWIKK5eEM2Qqv1jJWtJXZNrnBE/zutvXj2E
+ w+ZVKNlbNM488ODolw/r6z7OkCVgacEoYdDR+UFRXxKF5tPxTxfD0dMN+t5ZjqKOEsXEdN1Fhcn9
+ wF7BNjheMPxghFsjIwINNqkbXC/wv1jB3a9oGPa4BHVKDIPK97/T4LRRVYxF+VXiiOfHJN40eTXl
+ WiUAYdLmsJdAoPKCpWwKtkkGG+bEnfOEkWTNI3SjTCvjMfNBc9ze9o81pXKSQ+GI7QB7PH97h6/L
+ 6Wa2uSQw0nxsyGH4ASfmmhbNFFwa8AyQYqjO7qYtiXb+9Q==
+X-Report-Abuse-To: [email-masked]
+
+Hi EuAndreh.
+
+I am curious if you found any existing solution for the proposed database? I
+have been looking at irmin (which was mentioned in all forums) a lot lately,
+largely as substitute to libgit2, but recently also as a database.
+
+It looks like the conversation have tapered off, or moved elsewhere? In any
+case it was interesting stumbling across your post, seeing as I have been
+having very similar ideas.
+
+Regards,
+Martin
+
+
+
+From [email-masked] Tue Dec 22 21:37:15 2020
+Authentication-Results: mail-b.sr.ht; dkim=fail header.d=euandre.org header.i=@euandre.org
+Received: from box.euandre.org (box.euandre.org [46.101.160.115])
+	by mail-b.sr.ht (Postfix) with ESMTPS id 40985FF0DB
+	for <~euandreh/[email-masked]>; Tue, 22 Dec 2020 21:37:15 +0000 (UTC)
+Received: from authenticated-user (box.euandre.org [46.101.160.115])
+	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
+	(No client certificate requested)
+	by box.euandre.org (Postfix) with ESMTPSA id 4B3B9FC4BF;
+	Tue, 22 Dec 2020 18:37:11 -0300 (-03)
+DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=euandre.org; s=mail;
+	t=1608673033; bh=9xQtiA8BJBLUgJEdjCq1SNS1YmrPaYDMrBvQ2oYoGpY=;
+	h=From:To:Cc:Subject:In-Reply-To:References:Date:From;
+	b=P++6DaNXwhc5G/perUnlDlWN9zHzoyzmASEDPcZ0nqbNUpKJy9sV5OmC7etQTyPMO
+	 bAAwkUw3Wl/tn9CaLxzYJo+IClWYQEVJDmAMwlcCIJa3Ol+fSQsceBxK1a5za5EA8W
+	 SzBrXwvpbYG9ISVwBUY+fNkqwAoRP2lQ8Sy3/pvrA0Kq894eXg08tbvlTJh3H6rh3Z
+	 mla6VfP5qftYM6oWWhGSs6EJkGtmMDA+1UkAn+ASnSROejuxcPm1Cp2qmT2VoEPapP
+	 kmHV3bjCM9v4nN/1bPy1rBvhwhx1O9UfNWvUYGWnhb24Tu0zK68FA20ZXlQc1bRnGA
+	 yBrW5x0bzEZxg==
+From: EuAndreh <[email-masked]>
+To: [email-masked], ~euandreh/[email-masked]
+Cc: 
+Subject: Re: Progress on 'The database I wish I had'
+In-Reply-To: <[email-masked]>
+References: <[email-masked]>
+Date: Tue, 22 Dec 2020 18:36:56 -0300
+Message-ID: <[email-masked]>
+MIME-Version: 1.0
+Content-Type: text/plain
+
+<[email-masked]> writes:
+
+> Hi EuAndreh.
+
+Hi there!
+
+> I am curious if you found any existing solution for the proposed database? I
+> have been looking at irmin (which was mentioned in all forums) a lot lately,
+> largely as substitute to libgit2, but recently also as a database.
+
+I have written it down as a place to steal ideas from, which I plan to
+do in the future as I mature more on the field, but irmin isn't a
+solution, being written in OCaml.
+
+> It looks like the conversation have tapered off, or moved elsewhere? In any
+> case it was interesting stumbling across your post, seeing as I have been
+> having very similar ideas.
+
+I'm working on it, slowly but surely.
+
+The previous "comment" link used to point to my email address first, and
+the this public-inbox second. Because of that I had many conversations
+done in private messages, where I imagine that people just got the first
+link. I responded those in private, but many had similar enough
+comments, and I wrote an article on that [0]. FYI: I recently added a
+"articles by category" Atom feed, if you prefer a filtered list [1] of
+articles.
+
+On the ParsecC project mentioned on the article:
+I'm already much more comfortable with creating a C API, but I got to a
+point that I needed a proper error handling solution. I stumbled upon an
+existing implementation of a parser combinator in C [2].
+
+The point I'm at right now is to read and understand that source code.
+In fact, it is 1 of 4 open tabs on my browser right now, before writing
+this message.
+
+If I find its error handling good enough, I'll replicate it in ParsecC,
+and then in libedn.
+
+I've also started reading a good book on the subject [3]. I can't speak
+yet to all of it, but I like what I've seen so far.
+
+I happy to discuss more about it, or answer questions you may have. And
+also, I'm happy to accept contributions!
+
+[0]: https://euandre.org/2020/11/12/durable-persistent-trees-and-parser-combinators-building-a-database.html
+[1]: https://euandre.org/articles-by-category.html#mediator
+[2]: https://github.com/wbhart/Cesium3/tree/combinators
+[3]: https://www.immutablearchitecture.com/
+
+
-- 
cgit v1.3