path: root/src/comments.mbox

From rory@roryokane.com Mon Aug 31 16:44:13 2020
Received: from mail-qv1-f42.google.com (mail-qv1-f42.google.com [209.85.219.42])
	by mail-b.sr.ht (Postfix) with ESMTPS id 1E84AFF0FB
	for <~euandreh/public-inbox@lists.sr.ht>; Mon, 31 Aug 2020 16:44:13 +0000 (UTC)
Received: by mail-qv1-f42.google.com with SMTP id m14so1468478qvt.1
        for <~euandreh/public-inbox@lists.sr.ht>; Mon, 31 Aug 2020 09:44:13 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-gm-message-state:mime-version:from:date:message-id:subject:to;
        bh=1N/5MWE9d7/I+SxiuWn81KliArG9XSFF8duwpxyY0oA=;
        b=GiBnVI+Pf7DSl9ycPdfwPRjfdrlqlQQdgGKKyGsiIXPB1mLh6sBlzhoEQASxa/9hJm
         rfhT8wRfQyBwscG2FpECYK09N/0ks7a71Pi3vy9ePKvSHJl0FYhywyAWzZ9Z7Xfp0/SF
         oohs7Vzovs/sA9D9m2oKPcyfrj6lHWqf5qHJdfatP4LfVHCdRg1uwj3CmtCJWzmIFzuM
         aIkUXLrK5YyBbfVl2L6EvEU1L7EDqOaTrXJt/+hWOev813kGYdYWsxMbYmirBAmO8sal
         rgaF9vR/Jp8C+bxV/LSONsVOi4CA8azpLd1SYYpmS5yM4H5ISjaIOZfrZycx/Zd5Rle8
         tn6Q==
X-Gm-Message-State: AOAM530nGnF5g0fpYzDDTNbp5T67sHNZCqeZcVlHkD7CLIqa78AzYkW0
	44VZGvmhPwGnRAVlt7ZPbGJQAlmieuw=
X-Google-Smtp-Source: ABdhPJwgZnnyjMFpfNBttjOyxb0b4VZgNwyJNN7HkBIHn457LHtZ8CSNy1RJMts49AmqIUIw2YT9Yg==
X-Received: by 2002:a0c:b255:: with SMTP id k21mr1811354qve.134.1598892252561;
        Mon, 31 Aug 2020 09:44:12 -0700 (PDT)
Received: from mail-qt1-f176.google.com (mail-qt1-f176.google.com. [209.85.160.176])
        by smtp.gmail.com with ESMTPSA id n33sm10800778qtd.43.2020.08.31.09.44.12
        for <~euandreh/public-inbox@lists.sr.ht>
        (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
        Mon, 31 Aug 2020 09:44:12 -0700 (PDT)
Received: by mail-qt1-f176.google.com with SMTP id z2so5166925qtv.12
        for <~euandreh/public-inbox@lists.sr.ht>; Mon, 31 Aug 2020 09:44:12 -0700 (PDT)
X-Received: by 2002:ac8:73d0:: with SMTP id v16mr2161160qtp.384.1598892251883;
 Mon, 31 Aug 2020 09:44:11 -0700 (PDT)
MIME-Version: 1.0
From: =?UTF-8?Q?Rory_O=E2=80=99Kane?= <rory@roryokane.com>
Date: Mon, 31 Aug 2020 12:44:01 -0400
X-Gmail-Original-Message-ID: <CAD_ag1NQoRdA3CyDPFyXBs-KcuU36dKqtZ8HCk-bKMeKRCe_UA@mail.gmail.com>
Message-ID: <CAD_ag1NQoRdA3CyDPFyXBs-KcuU36dKqtZ8HCk-bKMeKRCe_UA@mail.gmail.com>
Subject: Re: The database I wish I had
To: ~euandreh/public-inbox@lists.sr.ht
Content-Type: text/plain; charset="UTF-8"

In this discussion of your post on Lobsters, there are some comments
that may aid you in your quest:
https://lobste.rs/s/m9vkg4/database_i_wish_i_had

From eu@euandre.org Mon Aug 31 23:00:25 2020
Received: from box.euandre.org (box.euandre.org [46.101.160.115])
	by mail-b.sr.ht (Postfix) with ESMTPS id 95690FF10A
	for <~euandreh/public-inbox@lists.sr.ht>; Mon, 31 Aug 2020 23:00:24 +0000 (UTC)
Authentication-Results: mail-b.sr.ht;
	dkim=pass (2048-bit key) header.d=euandre.org header.i=@euandre.org header.b=UPSgUUXq
Received: from authenticated-user (box.euandre.org [46.101.160.115])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	(No client certificate requested)
	by box.euandre.org (Postfix) with ESMTPSA id 70152FC81E;
	Mon, 31 Aug 2020 20:00:20 -0300 (-03)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=euandre.org; s=mail;
	t=1598914822; bh=f4y8R0BR6plzUtLJhSLdKJQrnDt5TiYAaxpeGwaCRBQ=;
	h=From:To:Cc:Subject:In-Reply-To:References:Date:From;
	b=UPSgUUXqESOz5UhGXtlYTFrg0Vy42raDUdQSF1jRvRJpFdVQ7MWErphZhOvLjMrSB
	 UxSspR/+WTYJDMf61hcqMfCqFYqGLpFH6Q6I6V4M7MABddX6mKgoyL16fBJl2V/FcH
	 DVJBcrI+FeDrr3eoLTbXR5ZZzddSTSzfS2w93/QLad95xyd443VBEtUXEvl+G6pLC1
	 Bzv7oeG07V2OZF+jIsfW116e0tyUsT1JTY377yiWTIxoqP27+PIyP2SZjPyBsS/Xzb
	 0MqK6PhL3IbwQ8xvv/h2SklKD9PYXtrjU5hupBXJSb/puN7fZHoaSv/qM69mfNuMxI
	 yTfUF1RUxJy5A==
From: EuAndreh <eu@euandre.org>
To: Rory =?utf-8?Q?O=E2=80=99Kane?= <rory@roryokane.com>,
 ~euandreh/public-inbox@lists.sr.ht
Cc: 
Subject: Re: The database I wish I had
In-Reply-To: <CAD_ag1NQoRdA3CyDPFyXBs-KcuU36dKqtZ8HCk-bKMeKRCe_UA@mail.gmail.com>
References: <CAD_ag1NQoRdA3CyDPFyXBs-KcuU36dKqtZ8HCk-bKMeKRCe_UA@mail.gmail.com>
Date: Mon, 31 Aug 2020 19:55:51 -0300
Message-ID: <87r1rmzaq0.fsf@euandre.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Rory O=E2=80=99Kane <rory@roryokane.com> writes:

> In this discussion of your post on Lobsters, there are some comments
> that may aid you in your quest:
> https://lobste.rs/s/m9vkg4/database_i_wish_i_had

I don't have an account to discuss things there, and I don't have an
invite, either.

If you could invite me there I would gladly engage on the discussion =E2=98=
=BA=EF=B8=8F.

From rory@roryokane.com Mon Aug 31 23:43:09 2020
Received: from mail-qk1-f180.google.com (mail-qk1-f180.google.com [209.85.222.180])
	by mail-b.sr.ht (Postfix) with ESMTPS id 43D60FF107
	for <~euandreh/public-inbox@lists.sr.ht>; Mon, 31 Aug 2020 23:43:09 +0000 (UTC)
Received: by mail-qk1-f180.google.com with SMTP id w186so7938858qkd.1
        for <~euandreh/public-inbox@lists.sr.ht>; Mon, 31 Aug 2020 16:43:09 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-gm-message-state:mime-version:references:in-reply-to:from:date
         :message-id:subject:to:cc:content-transfer-encoding;
        bh=gaaBELP8x2iHF+2fExDi20fg3n/w86cYg/IGy9Qn3xc=;
        b=DRAjD5Bo+V1b4ugqTEX8uvyep/1C0WyUO+myK/uHn6v95bQRDYNS1+whm98sXMPXrb
         qcafP0m0kbUfsTjEboLvrqZBZa3Ztv+8GFW6XmRjrtfJuZXx7RL0wXj16daqa2ik/dEe
         RA0WM9UrrczXtTBjwnrG7ozNOGy1SKDKHO/c4f7aylfYGP+wVxyDl6nAaMQk+wHGkJcY
         I37uXMATBkcsNPO+R8/lsyiCLkgk3wED+XJpYKZekgRA8p2I4S7wOP1SqmIpIyQlNllq
         cYtPJkv8wi5MNtwR8t0k+DU6QZwYk8H1U+3YhWfwuVl+S+zHN1DC3Tw5ZrlxcIlpsP5v
         0fQA==
X-Gm-Message-State: AOAM530oi09/wtTwog1b67r3kLqfg4eevHc/vZXa69bokmICUCS/wXgK
	ZU0PXrAbUDFg2UNYNvv/oKkU5//k/mo=
X-Google-Smtp-Source: ABdhPJyzZixNEwElMWvVTgTATohR8FEqbOW0p/68jnNsN8u+gCY6xUFeVZhDVtPlvGcDFDsynVK1gQ==
X-Received: by 2002:a05:620a:2224:: with SMTP id n4mr4016163qkh.18.1598917388635;
        Mon, 31 Aug 2020 16:43:08 -0700 (PDT)
Received: from mail-qk1-f178.google.com (mail-qk1-f178.google.com. [209.85.222.178])
        by smtp.gmail.com with ESMTPSA id u15sm12587433qtj.3.2020.08.31.16.43.08
        for <~euandreh/public-inbox@lists.sr.ht>
        (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
        Mon, 31 Aug 2020 16:43:08 -0700 (PDT)
Received: by mail-qk1-f178.google.com with SMTP id p25so7942117qkp.2
        for <~euandreh/public-inbox@lists.sr.ht>; Mon, 31 Aug 2020 16:43:08 -0700 (PDT)
X-Received: by 2002:a37:b6c6:: with SMTP id g189mr4006715qkf.491.1598917387885;
 Mon, 31 Aug 2020 16:43:07 -0700 (PDT)
MIME-Version: 1.0
References: <CAD_ag1NQoRdA3CyDPFyXBs-KcuU36dKqtZ8HCk-bKMeKRCe_UA@mail.gmail.com>
 <87r1rmzaq0.fsf@euandre.org>
In-Reply-To: <87r1rmzaq0.fsf@euandre.org>
From: =?UTF-8?Q?Rory_O=E2=80=99Kane?= <rory@roryokane.com>
Date: Mon, 31 Aug 2020 19:42:57 -0400
X-Gmail-Original-Message-ID: <CAD_ag1Obqre+uRN+fQgX1-O7WWkT=2dTo4-HxabLXMBE9od6DQ@mail.gmail.com>
Message-ID: <CAD_ag1Obqre+uRN+fQgX1-O7WWkT=2dTo4-HxabLXMBE9od6DQ@mail.gmail.com>
Subject: Re: The database I wish I had
To: EuAndreh <eu@euandre.org>
Cc: ~euandreh/public-inbox@lists.sr.ht
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

EuAndreh <eu@euandre.org> wrote:

> I don't have an account to discuss things there, and I don't have an
> invite, either.
>
> If you could invite me there I would gladly engage on the discussion =EF=
=B8=8F.

I checked out your website and based on your writing, I can probably
trust you with an invite. Invite sent=E2=80=94enjoy.

From eu@euandre.org Tue Sep  1 01:00:13 2020
Received: from box.euandre.org (box.euandre.org [46.101.160.115])
	by mail-b.sr.ht (Postfix) with ESMTPS id B6E38FF11C
	for <~euandreh/public-inbox@lists.sr.ht>; Tue,  1 Sep 2020 01:00:12 +0000 (UTC)
Authentication-Results: mail-b.sr.ht;
	dkim=pass (2048-bit key) header.d=euandre.org header.i=@euandre.org header.b=P61dJynV
Received: from authenticated-user (box.euandre.org [46.101.160.115])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	(No client certificate requested)
	by box.euandre.org (Postfix) with ESMTPSA id 167EDFC820;
	Mon, 31 Aug 2020 22:00:09 -0300 (-03)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=euandre.org; s=mail;
	t=1598922011; bh=VOay/ul6frlnGv4X4iPDMI75WLW6Uop6PCjAV5nwAac=;
	h=From:To:Cc:Cc:Subject:In-Reply-To:References:Date:From;
	b=P61dJynVkkTLwtYleZjejOi5dPaIMpaJR1omX34om5nTFtkTU/7ckXbqlU/AlUxfb
	 IC516Xu9/pkk1uW45SdeoU8+9nYnK9/KK7VCiifC5yQeMLZwlzNupanCYeycFes500
	 D5Ygal4+Kz6prDX2cKrF4O97/nBKQKG6QQDzRG61Q1v+Tg2OMoJ45XENtQ7ONYzt9H
	 RkbSKiNA4ipJDzLx1vd9VX+cKSC9VljZn9dFxIBA6KQ+nmbMNXtNmirc10sREJ2g3u
	 zYefnUA3vRF1csdbIkTjfbiWnD2w2RN3Zj/oMwhNiAjHix4FyIg89WXwzv/CHmP9gr
	 3+SrXBIC6XxzQ==
From: EuAndreh <eu@euandre.org>
To: Rory =?utf-8?Q?O=E2=80=99Kane?= <rory@roryokane.com>
Cc: ~euandreh/public-inbox@lists.sr.ht
Cc: 
Subject: Re: The database I wish I had
In-Reply-To: <CAD_ag1Obqre+uRN+fQgX1-O7WWkT=2dTo4-HxabLXMBE9od6DQ@mail.gmail.com>
References: <CAD_ag1NQoRdA3CyDPFyXBs-KcuU36dKqtZ8HCk-bKMeKRCe_UA@mail.gmail.com>
 <87r1rmzaq0.fsf@euandre.org>
 <CAD_ag1Obqre+uRN+fQgX1-O7WWkT=2dTo4-HxabLXMBE9od6DQ@mail.gmail.com>
Date: Mon, 31 Aug 2020 21:42:38 -0300
Message-ID: <87mu2az5s1.fsf@euandre.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Rory O=E2=80=99Kane <rory@roryokane.com> writes:

> I checked out your website and based on your writing, I can probably
> trust you with an invite. Invite sent=E2=80=94enjoy.

Thanks for the invitation, I hope I can add to the discussion and the
community.=20

From me@josephg.com Tue Sep  1 03:22:08 2020
Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28])
	by mail-b.sr.ht (Postfix) with ESMTPS id 85748FF119
	for <~euandreh/public-inbox@lists.sr.ht>; Tue,  1 Sep 2020 03:22:08 +0000 (UTC)
Authentication-Results: mail-b.sr.ht;
	dkim=pass (1024-bit key) header.d=josephg.com header.i=@josephg.com header.b=XpX/7nfb;
	dkim=fail reason="key not found in DNS" (0-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=MSacOUNm
Received: from compute1.internal (compute1.nyi.internal [10.202.2.41])
	by mailout.nyi.internal (Postfix) with ESMTP id 4403D5C0190
	for <~euandreh/public-inbox@lists.sr.ht>; Mon, 31 Aug 2020 23:22:08 -0400 (EDT)
Received: from imap35 ([10.202.2.85])
  by compute1.internal (MEProxy); Mon, 31 Aug 2020 23:22:08 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=josephg.com; h=
	mime-version:message-id:date:from:to:subject:content-type; s=
	mesmtp; bh=D0CRtslwgxf19V66w3zMnP44hZ3wW3VGv7uaTI0Tszs=; b=XpX/7
	nfbLe6I0o40Nfy5xKR+AcUanUComIWaSJMa5Gz5V2bCbIFdj6j2sSHEJWMJ/OvnG
	K4hgJmu5ZXcYOmhZBRIrGpjackRdPfipduHOMYuWsT/tdMecLsxBU79WhNR71xfi
	gCQK/U2ChiHuu2JvN2lhP9UZHk7EoaAkBGVtwQ=
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
	messagingengine.com; h=content-type:date:from:message-id
	:mime-version:subject:to:x-me-proxy:x-me-proxy:x-me-sender
	:x-me-sender:x-sasl-enc; s=fm3; bh=D0CRtslwgxf19V66w3zMnP44hZ3wW
	3VGv7uaTI0Tszs=; b=MSacOUNm/cU+7m2AkVNHn56i7XS7rZXVQlRheFQ10MXFQ
	gY7AEoJYghtK1hl+FZhe4yTOOUJfj7xGQsnp7ENjd0yHSZPIqh4w1WDWMeWgvIwg
	p3v7xHQc/ce/1zKCV7NNftJh9ietQ63JTujrUTGC0Hszv5+D+7mkLO+OGtMwAmVR
	auNg0sFFa6BdTfaQLRWAX/BqQkiYcJ9oFW+Lxw1wiN6D/K005aRgHbbOwCAHGi9v
	6qkg/cVFzzqR52ydek4Oq0AtGwRBVMGK7llRBDRe83VDOcjzRNZScU/Z1hlEhDWF
	fYm0B4nWev8MbeQUup3DjryE9U0WrMv/1MmI+G25A==
X-ME-Sender: <xms:X75NX0NkXIq-arNl9tbSsnRDUhjMNpg-2ozeoZqLY6_2_W3DklLffQ>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduiedrudefiedgjedtucetufdoteggodetrfdotf
    fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen
    uceurghilhhouhhtmecufedttdenucenucfjughrpefofgggkfffhffvufgtsehttdertd
    erredtnecuhfhrohhmpedfufgvphhhucfivghnthhlvgdfuceomhgvsehjohhsvghphhhg
    rdgtohhmqeenucggtffrrghtthgvrhhnpeekgffhuedttedvjeelkeffkeeuueelkeeite
    dttefgleejledtueetgfeijefhueenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgr
    mhepmhgrihhlfhhrohhmpehmvgesjhhoshgvphhhghdrtghomh
X-ME-Proxy: <xmx:X75NX6_-JLu-Eneod1EKsHyKkOEIRYFaAZQH4uBldS7GxRe_0WzDdw>
    <xmx:X75NX7QCRm4jcdxXVZf_-SgheExet0j7JbTijuDCSZfm7EpG8GoXxg>
    <xmx:X75NX8tKELWH7ryIzS_tU-Gi7kQXm8uw2XHfCRdbB4HJ08Y4s4Nheg>
    <xmx:YL5NX4HEO3VR6t7Yz1-vKIi99YuUZV2t1wRHH3Tqv8f1bMEvEjAbAw>
Received: by mailuser.nyi.internal (Postfix, from userid 501)
	id C51F714C00E5; Mon, 31 Aug 2020 23:22:07 -0400 (EDT)
X-Mailer: MessagingEngine.com Webmail Interface
User-Agent: Cyrus-JMAP/3.3.0-232-g4bdb081-fm-20200825.002-g4bdb081a
Mime-Version: 1.0
Message-Id: <d78064d2-5441-4275-96d9-463a08dfd7a9@www.fastmail.com>
Date: Tue, 01 Sep 2020 03:21:47 +0000
From: "Seph Gentle" <me@josephg.com>
To: ~euandreh/public-inbox@lists.sr.ht
Subject: Making a database
Content-Type: text/plain

I've been thinking along similar lines, and am talking to some folks about building something on top of CRDTs. Modern CRDTs like automerge and y.js are just getting to the point where they're fast and small enough to work fine. I can see something really valuable that looks kind of like git, except where you can store arbitrary data. And live-sync peers - so if I make changes you see those changes live on your computer too - without (necessarily) needing a centralized server.

Love to chat if you & others are keen - I/we are still in the process of figuring out what this thing will look like. But, I feel like the database piece is really core here too, and I like a lot of your thinking in this space.

-Seph

From me@jonas-schuermann.name Tue Sep  1 09:26:09 2020
Received: from mout-p-102.mailbox.org (mout-p-102.mailbox.org [80.241.56.152])
	by mail-b.sr.ht (Postfix) with ESMTPS id CEBA2FF127
	for <~euandreh/public-inbox@lists.sr.ht>; Tue,  1 Sep 2020 09:26:08 +0000 (UTC)
Received: from smtp2.mailbox.org (smtp2.mailbox.org [IPv6:2001:67c:2050:105:465:1:2:0])
	(using TLSv1.2 with cipher ECDHE-RSA-CHACHA20-POLY1305 (256/256 bits))
	(No client certificate requested)
	by mout-p-102.mailbox.org (Postfix) with ESMTPS id 4BghWz4k2gzKmTx
	for <~euandreh/public-inbox@lists.sr.ht>; Tue,  1 Sep 2020 11:26:07 +0200 (CEST)
X-Virus-Scanned: amavisd-new at heinlein-support.de
Received: from smtp2.mailbox.org ([80.241.60.241])
	by spamfilter04.heinlein-hosting.de (spamfilter04.heinlein-hosting.de [80.241.56.122]) (amavisd-new, port 10030)
	with ESMTP id lwr5j4QAvkHQ for <~euandreh/public-inbox@lists.sr.ht>;
	Tue,  1 Sep 2020 11:26:02 +0200 (CEST)
To: ~euandreh/public-inbox@lists.sr.ht
Subject: Re: The database I wish I had
From: =?UTF-8?Q?Jonas_Sch=c3=bcrmann?= <me@jonas-schuermann.name>
Message-ID: <e0b3b042-5eda-bcde-59a5-a5b61fecd2e5@jonas-schuermann.name>
Date: Tue, 1 Sep 2020 11:26:02 +0200
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
X-MBO-SPAM-Probability: 
X-Rspamd-Score: -1.02 / 15.00 / 15.00
X-Rspamd-Queue-Id: 6015B17AB
X-Rspamd-UID: af255d
Content-Transfer-Encoding: quoted-printable

Hello EuAndreh,

I read your blog post with excitement because I'm currently on a very
similar quest. My goal is to bring back application data into the hands
of the users (and onto their devices) to replace centralized,
cloud-based data stores. I wrote down what I wanted in this blog post
[1].

Like you, I didn't find anything that seemed to work like I wanted, so I
decided to start exploring the problem myself. While you go for the
Git-like approach of three-way-merges, I am trying to model conflict
situations with completed categories.

I can't say that I've come very far yet. I realized that properly
modelling conflicts with completed categories is a hard problem and
while I had some progress already, there isn't anything that I would
consider ready to be shared just yet (if you're curious you can look at
the source [2]).

But through discussions on lobste.rs [3] and further research I built a
list of references, prior work and concepts in this space [4]. From your
blog post I even got a few more references that I just added :) Perhaps
this list is also useful for you.

I wish you the best of luck with your endeavor and I hope to see further
posts about your progress on the blog :) When I have more time I will
also write more about what I've found and what's next.

Cordially, Jonas Sch=C3=BCrmann

[1]=20
https://jonas-schuermann.name/projects/dvcs-for-structured-data/blog/2020=
-03-22-manifesto.html
[2]=20
https://gitlab.com/MazeChaZer/dvcs-for-structured-data/-/blob/master/sour=
ce-code/EventSourcing.hs
[3] https://lobste.rs/s/qsehiu/i_want_decentralized_version_control_for
[4]=20
https://jonas-schuermann.name/projects/dvcs-for-structured-data/notes/ref=
erences.html

From eu@euandre.org Tue Sep  1 10:00:15 2020
Received: from box.euandre.org (box.euandre.org [46.101.160.115])
	by mail-b.sr.ht (Postfix) with ESMTPS id 6FC96FF127
	for <~euandreh/public-inbox@lists.sr.ht>; Tue,  1 Sep 2020 10:00:14 +0000 (UTC)
Authentication-Results: mail-b.sr.ht;
	dkim=pass (2048-bit key) header.d=euandre.org header.i=@euandre.org header.b=Fm0cCCjb
Received: from authenticated-user (box.euandre.org [46.101.160.115])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	(No client certificate requested)
	by box.euandre.org (Postfix) with ESMTPSA id 82F61FCB45;
	Tue,  1 Sep 2020 07:00:11 -0300 (-03)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=euandre.org; s=mail;
	t=1598954412; bh=d3MsFaC9js1laWdI/WHjnTEpvQ9IMPR4/17eUYzh5Z8=;
	h=From:To:Cc:Subject:In-Reply-To:References:Date:From;
	b=Fm0cCCjbaKJpKhoedrPl0XWIYJujunIk9LXBI7OUyABb7hgxRqEkLmW93LCA60QAE
	 +aZMzweJYdcNtmRBMeQmlkFYwnZZq8L9OBlsqi+BOILAEA5jv8Ly8RMQ6ZV6kogRpY
	 rV0RwyBEvskQ0vao5a/IWLuCSSalXShntwRIy44liQYZqxjFwJTQEXPDeb+gvmYaz1
	 9aEvg5edL/OEGfo0VZpRk//f47V6a8o/3/uEaLpJAtEfCtQY1YYk5BdQGafCp5LrVR
	 fdp/6ju5YnM3yjM52MxTUvEjFpvK+FL/FSQDJ2DhdtbrH/EHHmNpImLP9WL1Bw9mNY
	 N2ar7lD/+9A6g==
From: EuAndreh <eu@euandre.org>
To: Seph Gentle <me@josephg.com>, ~euandreh/public-inbox@lists.sr.ht
Cc: 
Subject: Re: Making a database
In-Reply-To: <d78064d2-5441-4275-96d9-463a08dfd7a9@www.fastmail.com>
References: <d78064d2-5441-4275-96d9-463a08dfd7a9@www.fastmail.com>
Date: Tue, 01 Sep 2020 06:39:19 -0300
Message-ID: <87ft81zvi0.fsf@euandre.org>
MIME-Version: 1.0
Content-Type: text/plain

"Seph Gentle" <me@josephg.com> writes:

> I've been thinking along similar lines, and am talking to some folks
> about building something on top of CRDTs. Modern CRDTs like automerge
> and y.js are just getting to the point where they're fast and small
> enough to work fine. I can see something really valuable that looks
> kind of like git, except where you can store arbitrary data. And
> live-sync peers - so if I make changes you see those changes live on
> your computer too - without (necessarily) needing a centralized
> server.
>
> Love to chat if you & others are keen - I/we are still in the process
> of figuring out what this thing will look like. But, I feel like the
> database piece is really core here too, and I like a lot of your
> thinking in this space.

Thanks for the encouragement :)

I've looked at CRDTs, but I haven't done a deep dive yet, because my
first impression was that they generally choose to limit the types of
data structures and operations that you can perform on them in favour of
being conflict-free.

That's why I didn't mention it on my article directly, but I admit to
having it as pre-requisite to making an informed decision on the
subject.

WDYT?

From eu@euandre.org Tue Sep  1 11:00:18 2020
Received: from box.euandre.org (box.euandre.org [46.101.160.115])
	by mail-b.sr.ht (Postfix) with ESMTPS id 7E055FF122
	for <~euandreh/public-inbox@lists.sr.ht>; Tue,  1 Sep 2020 11:00:17 +0000 (UTC)
Authentication-Results: mail-b.sr.ht;
	dkim=pass (2048-bit key) header.d=euandre.org header.i=@euandre.org header.b=aexrWU2E
Received: from authenticated-user (box.euandre.org [46.101.160.115])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	(No client certificate requested)
	by box.euandre.org (Postfix) with ESMTPSA id A3E67FD6B9;
	Tue,  1 Sep 2020 08:00:14 -0300 (-03)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=euandre.org; s=mail;
	t=1598958016; bh=RLLLMwMzFeXBwtfwXnEsHK/UTMXFBGSC5qgHaz/ZmOE=;
	h=From:To:Cc:Subject:In-Reply-To:References:Date:From;
	b=aexrWU2Ee5zBZdVCWHCoLentiEWPHDdm7cDXSVNCrsIO+wkZPDXWbT+YwGbX7pldB
	 rs//2AD5Bz0VOPByW33jD53/lmCCBeZ8eBjSLpfN2/w3sXHKPXh348R6/YLLRMSuwj
	 9bCKuxK3X8qJ2SMS3fg8OGIzHgBq8rL59U0pFgCiV6WR5Z92mG80P8iogtSI8LzU/g
	 yBhu4sv4gJ3IlLhTiwX4HgAQkSoOCu6rdm5AbDdkwPwteUyy+7F2KlHc6eD7UTktU8
	 ic7l2oVEyQrUYddwtZmUfJJCG9tffy5RDAyBO/TaLjVJpesdFEY5nRIG/Xm+1geH1j
	 fv++ZMqsLTgxQ==
From: EuAndreh <eu@euandre.org>
To: Jonas =?utf-8?Q?Sch=C3=BCrmann?= <me@jonas-schuermann.name>,
 ~euandreh/public-inbox@lists.sr.ht
Cc: 
Subject: Re: The database I wish I had
In-Reply-To: <e0b3b042-5eda-bcde-59a5-a5b61fecd2e5@jonas-schuermann.name>
References: <e0b3b042-5eda-bcde-59a5-a5b61fecd2e5@jonas-schuermann.name>
Date: Tue, 01 Sep 2020 07:26:47 -0300
Message-ID: <87y2ltyeqg.fsf@euandre.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Jonas Sch=C3=BCrmann <me@jonas-schuermann.name> writes:

> Hello EuAndreh,

Hi there!

> I read your blog post with excitement because I'm currently on a very
> similar quest. My goal is to bring back application data into the hands
> of the users (and onto their devices) to replace centralized,
> cloud-based data stores. I wrote down what I wanted in this blog post
> [1].
>
> Like you, I didn't find anything that seemed to work like I wanted, so I
> decided to start exploring the problem myself. While you go for the
> Git-like approach of three-way-merges, I am trying to model conflict
> situations with completed categories.

Hmmm, I haven't heard this term before. Are these [0] the complete
categories that you mean?

[0]: https://en.wikipedia.org/wiki/Complete_category
=20
> I can't say that I've come very far yet. I realized that properly
> modelling conflicts with completed categories is a hard problem and
> while I had some progress already, there isn't anything that I would
> consider ready to be shared just yet (if you're curious you can look at
> the source [2]).
>
> But through discussions on lobste.rs [3] and further research I built a
> list of references, prior work and concepts in this space [4]. From your
> blog post I even got a few more references that I just added :) Perhaps
> this list is also useful for you.

Thanks for all the links, I'll try to use them to enrich my
understanding on the subject.

From me@josephg.com Tue Sep  1 11:25:21 2020
Received: from wout2-smtp.messagingengine.com (wout2-smtp.messagingengine.com [64.147.123.25])
	by mail-b.sr.ht (Postfix) with ESMTPS id 50AC3FF122
	for <~euandreh/public-inbox@lists.sr.ht>; Tue,  1 Sep 2020 11:25:21 +0000 (UTC)
Authentication-Results: mail-b.sr.ht;
	dkim=pass (1024-bit key) header.d=josephg.com header.i=@josephg.com header.b=O0rYtvTA;
	dkim=fail reason="key not found in DNS" (0-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=f1mPF/1x
Received: from compute1.internal (compute1.nyi.internal [10.202.2.41])
	by mailout.west.internal (Postfix) with ESMTP id E4956E77;
	Tue,  1 Sep 2020 07:25:19 -0400 (EDT)
Received: from imap35 ([10.202.2.85])
  by compute1.internal (MEProxy); Tue, 01 Sep 2020 07:25:20 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=josephg.com; h=
	mime-version:message-id:in-reply-to:references:date:from:to
	:subject:content-type:content-transfer-encoding; s=mesmtp; bh=BO
	dAgwWsH67HEzbojnfogot17D0WKCAzyItvgTLmJCw=; b=O0rYtvTAoINKZTJObn
	vUnmONdChl7CPOEklbKnqUQ+Sk8qaAoCe0YyqwTLq0YZywk49u56Zwff5QaSJfLm
	SYA9bd2RUEagih/gPTR4BHJarqfWSHQeUsHsvMmVtj4w8jrJt261gcmh1Git8Jkt
	lJSUIZ9tpOecI/nFsnhxCleA0=
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
	messagingengine.com; h=content-transfer-encoding:content-type
	:date:from:in-reply-to:message-id:mime-version:references
	:subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender
	:x-sasl-enc; s=fm3; bh=BOdAgwWsH67HEzbojnfogot17D0WKCAzyItvgTLmJ
	Cw=; b=f1mPF/1xX+9SrbduF4hyrnDZlojUnotl/ZuV6F+yuzbQlOHFTtLgMSJ5u
	ghKQ7iTGJdFGqxzM87y1JJLCWb6BZjbRw2TjhDK+z24JeMwOVOpZgDsZKZ4vWlHI
	x6C8Ca/TnlF8Glsyvepoczx5SP6kbYCOe/JrBGp3gTIfTGohNHuxCsxxHvW3gn35
	hQLIa42DMt3y1p0q2rl4ltdrCYpCzQVvfbhKssX3AmCkwndYDhbP6l4jLnWH6jeM
	oBw77aR3keR94fhUnSCsnlX5ZSljRkIecR7Z9sGsR8KeB5j72A/ND3rncJyOOQ/e
	+DXRIAj+KVgqvbY2AimyrweAA9mQg==
X-ME-Sender: <xms:nS9OX8TbEfFCRuUyS6F4hwgBx8v03SzB2Ggzuy63tMjyFVg-UrVkNQ>
    <xme:nS9OX5zGsoiG5l8lthnVpFgOEb1gqC7oeFR2NwTbn4uDQFjSZ-ZvMZeo0aSeRF8GV
    exRk5tGsHuiPRtDjw>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduiedrudefjedggedtucetufdoteggodetrfdotf
    fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen
    uceurghilhhouhhtmecufedttdenucenucfjughrpefofgggkfgjfhffhffvufgtgfesth
    hqredtreerjeenucfhrhhomhepfdfuvghphhcuifgvnhhtlhgvfdcuoehmvgesjhhoshgv
    phhhghdrtghomheqnecuggftrfgrthhtvghrnhepkeegtdfhgfdvheetveetleejgfdvhf
    egtdegfedvhfegveffgfdthfehgedukeegnecuffhomhgrihhnpehslhgrtghkrdgtohhm
    necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepmhgvse
    hjohhsvghphhhgrdgtohhm
X-ME-Proxy: <xmx:nS9OX50QvymIMNNSVjvYuwnELZacWmVFdJvm4gYZ7wZ28Ysp8XeqaA>
    <xmx:nS9OXwCN5irAMSLyp8Oejw_bDz4dedc6-GQj4ThImfuyCt5P3RUUhA>
    <xmx:nS9OX1jNG59H8yiSSKrEdc830ToS-wnjIO8Dqrr2hhrFB4opADJCoQ>
    <xmx:ny9OX6tPG_S_xI2Z3IHEmzknqb1-PlAmuX_wv6n9qglqXIcgp0MqSQ>
Received: by mailuser.nyi.internal (Postfix, from userid 501)
	id 72BAB14C034C; Tue,  1 Sep 2020 07:25:17 -0400 (EDT)
X-Mailer: MessagingEngine.com Webmail Interface
User-Agent: Cyrus-JMAP/3.3.0-248-gcd102cb-fm-20200901.001-gcd102cb9
Mime-Version: 1.0
Message-Id: <9e513706-7ebc-4a40-8e5e-a5f94d84bc77@www.fastmail.com>
In-Reply-To: <87ft81zvi0.fsf@euandre.org>
References: <d78064d2-5441-4275-96d9-463a08dfd7a9@www.fastmail.com>
 <87ft81zvi0.fsf@euandre.org>
Date: Tue, 01 Sep 2020 21:24:55 +1000
From: "Seph Gentle" <me@josephg.com>
To: EuAndreh <eu@euandre.org>, ~euandreh/public-inbox@lists.sr.ht
Subject: Re: Making a database
Content-Type: text/plain;charset=utf-8
Content-Transfer-Encoding: quoted-printable

On Tue, Sep 1, 2020, at 7:39 PM, EuAndreh wrote:
> "Seph Gentle" <me@josephg.com> writes:
>=20
> > I've been thinking along similar lines, and am talking to some folks=

> > about building something on top of CRDTs. Modern CRDTs like automerg=
e
> > and y.js are just getting to the point where they're fast and small
> > enough to work fine. I can see something really valuable that looks
> > kind of like git, except where you can store arbitrary data. And
> > live-sync peers - so if I make changes you see those changes live on=

> > your computer too - without (necessarily) needing a centralized
> > server.
> >
> > Love to chat if you & others are keen - I/we are still in the proces=
s
> > of figuring out what this thing will look like. But, I feel like the=

> > database piece is really core here too, and I like a lot of your
> > thinking in this space.
>=20
> Thanks for the encouragement :)
>=20
> I've looked at CRDTs, but I haven't done a deep dive yet, because my
> first impression was that they generally choose to limit the types of
> data structures and operations that you can perform on them in favour =
of
> being conflict-free.
>=20
> That's why I didn't mention it on my article directly, but I admit to
> having it as pre-requisite to making an informed decision on the
> subject.
>=20
> WDYT?

Well, they=E2=80=99re conflict-free in the sense that (unlike Postgres/f=
oundationdb/etc) the database doesn=E2=80=99t reject conflicting concurr=
ent updates. Which is a property you need if you=E2=80=99re going to com=
mit local writes while you=E2=80=99re offline. And if I=E2=80=99m readin=
g your blog post right it sounds like that=E2=80=99s what you=E2=80=99re=
 going for.

You kinda gave a good description of them in your post already. They wor=
k by requiring all peers to resolve concurrent edits to an object the sa=
me way. That could be last-writer-wins (based on time stamps like you sa=
id in your article). Or it could store all conflicting versions together=
 to be resolved by the next reader (Eg riak). Or it could do something m=
ore clever (like merge changes using automerge or y.js or the like). Git=
 is sort of a CRDT - it=E2=80=99s just it=E2=80=99s 3 way merge algorith=
m is a bit of a dogs breakfast as far as these things go.

Anyway, happy to chat about this stuff in more detail if you wanna zoom =
or something. Or hop into the automerge slack (with pvh from ink&switch,=
 and Martin Kleppmann, and others) and we can talk about this stuff - ht=
tps://join.slack.com/t/automerge/shared_invite/zt-e4p3760n-kKh7r3KRH1Yww=
NfiZM8ktw

As I said - I=E2=80=99m looking to build something like this too and it=E2=
=80=99d be good to compare notes on our approaches!

From rnewman@twinql.com Tue Sep  1 15:46:34 2020
Received: from a27-232.smtp-out.us-west-2.amazonses.com (a27-232.smtp-out.us-west-2.amazonses.com [54.240.27.232])
	by mail-b.sr.ht (Postfix) with ESMTPS id 02A70FF0E3
	for <~euandreh/public-inbox@lists.sr.ht>; Tue,  1 Sep 2020 15:46:33 +0000 (UTC)
Authentication-Results: mail-b.sr.ht;
	dkim=pass (1024-bit key) header.d=twinql.com header.i=@twinql.com header.b=cNOZcfp4;
	dkim=pass (1024-bit key) header.d=amazonses.com header.i=@amazonses.com header.b=Hg/JwA02
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple;
	s=qvs2k2sd2ivtrflyu2nswmravuqizjml; d=twinql.com; t=1598975192;
	h=Subject:From:To:Date:Mime-Version:Content-Type:Content-Transfer-Encoding:References:Message-Id;
	bh=h5tYwgdiFvKfB58ME/qKUDlZfMWWDjBD1us0jqb+obo=;
	b=cNOZcfp4gLnUosw6qaEpaunrxtpsBSNsQgvh/C3StK1VPirNXCoHKVx5ZtOO/szG
	PdOufKL9NM0bQu6OiYFBxUdp2fuZ2FGwONd2nzGBOGcMN4V2aSOuMAGcuClGe2Kf+SM
	xfSfiY2iyxEUtlFoNU9qfvTtJ5Wutm+/EkIRuJ+g=
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple;
	s=hsbnp7p3ensaochzwyq5wwmceodymuwv; d=amazonses.com; t=1598975192;
	h=Subject:From:To:Date:Mime-Version:Content-Type:Content-Transfer-Encoding:References:Message-Id:Feedback-ID;
	bh=h5tYwgdiFvKfB58ME/qKUDlZfMWWDjBD1us0jqb+obo=;
	b=Hg/JwA02/i8iV50zfzagoO6qkIy3fWh3xaS3Ea/Mbew1Ve1B70ilQdJIrthnSRCF
	S50Nn0YzMQlyzXXd8rCHSfwTcqDUYV14hdN+GPbGPF+O+6w/X7KG6PxWnYTE/5w/cE9
	5HAQ9xp3FdpTqa2RvVsLtEJVweoEtHmNR5319VRs=
Subject: Re: The database I wish I had
From: =?UTF-8?Q?Richard_Newman?= <rnewman@twinql.com>
To: =?UTF-8?Q?=7Eeuandreh/public-inbox=40lists=2Esr=2Eht?=
  <~euandreh/public-inbox@lists.sr.ht>
Date: Tue, 1 Sep 2020 15:46:31 +0000
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
References: <7454DA45-C5A4-4A54-8D02-1F3BD37726E9@twinql.com>
X-Priority: 3 (Normal)
X-Mailer: Amazon WorkMail
Thread-Index: AQHWgHcPJ4udfXYrRaOwhoO7EcsDJQ==
Thread-Topic: The database I wish I had
X-Original-Mailer: Apple Mail (2.3445.104.14)
X-Wm-Sent-Timestamp: 1598975191
Message-ID: <010101744a592b75-1dce9281-f0b8-4226-9d50-fd2c7901fa72-000000@us-west-2.amazonses.com>
X-SES-Outgoing: 2020.09.01-54.240.27.232
Feedback-ID: 1.us-west-2.An468LAV0jCjQDrDLvlZjeAthld7qrhZr+vow8irkvU=:AmazonSES

Hello! Original author of Mentat here.=0D=0A=0D=0AThe use of SQLite for M=
entat was intended to be an accelerant for development: by leveraging SQL=
ite we could ignore large parts of the tricky problem of getting bits on =
disk across platforms, and also get things like transactionality and full=
-text search for free.=0D=0A=0D=0AWe used the same leverage trick when we=
 built the first proof of concept using Clojure and ClojureScript =E2=80=94=
 we could run it in a JVM locally, compare to Datomic, and test in Node, =
and entirely avoid having to build parsers and lexers. This allowed us to=
 focus on the problem of representing a datom/log/index data model in SQL=
ite.=0D=0A=0D=0AThe downside of using SQLite was a large amount of time w=
riting code to compile queries =E2=80=94 which were written in much the s=
ame language as Datomic=E2=80=99s =E2=80=94 into SQL for execution. That =
Datomic compatibility also pushed us towards building an EDN parser in Ru=
st, and we found some under-specified parts of Datomic, too.=0D=0A=0D=0AP=
ortability to the web itself was not an initial goal, but it was somethin=
g we thought about. With adequate resourcing, and the benefit of hindsigh=
t, we would have instead built a chunk-based backing store abstraction, u=
sing something like rkv/LMDB for embedded use and indexeddb on the web. W=
e did not feel like we had the engineering capacity to take on building a=
 truly portable, battle-tested storage layer like SQLite=E2=80=99s *and* =
a bits-on-disk query executor =E2=80=94 I=E2=80=99ve done it in the past,=
 and it takes a lot of time and is hard to get right.=0D=0A=0D=0AMentat a=
ttempted to address a number of things that you didn=E2=80=99t cover in y=
our brief post. A robust definition of identity, end-to-end encryption, a=
nd evolution of schema (even simple schema like cardinality and uniquenes=
s constraints on attributes, which are the foundation of identity) are th=
ree such. Schema evolution becomes tremendously important when you have m=
ultiple teams using the same storage, or a long-lived product =E2=80=94 t=
he more useful the database is, the more likely you are to need this feat=
ure!=0D=0A=0D=0AThose three things cascade into the problem of synchroniz=
ation, which was my focus for a number of years.=0D=0A=0D=0ASynchronizati=
on is a hard problem. Doing it without server assistance is even harder, =
but is necessary for a portable and secure product. Synchronization of ge=
neric data =E2=80=94 that is, data where the synchronization tool itself =
doesn=E2=80=99t have its own model of the world =E2=80=94 is extremely tr=
icky: everything from identifiers through to basic constraints needs to b=
e aware of the possibility of change, and naturally it=E2=80=99s an itera=
tive and distributed situation. Most of the approaches I found rely on a =
server and are for systems that omit schema, and are typically incomplete=
: e.g., detecting conflicts on individual attributes, leading to nonsensi=
cal results, and offload to the developer the problem of separately handl=
ing stable identifiers.=0D=0A=0D=0AI still have a persistent itch to star=
t with a clean slate and solve this problem. No large company has the rig=
ht incentives to solve the much harder problems faced by a truly secure s=
tore, most hobbyists lack the understanding of the importance of managing=
 change and concurrent collaboration over time, and the current trend is =
towards using =E2=80=98dumb=E2=80=99 object/key-value stores or specialis=
t systems like time-series databases, so I think it=E2=80=99s unlikely th=
at anybody else is going to tackle all three in the same solution. Here=E2=
=80=99s hoping!

From eu@euandre.org Tue Sep  1 18:00:18 2020
Received: from box.euandre.org (box.euandre.org [46.101.160.115])
	by mail-b.sr.ht (Postfix) with ESMTPS id CD25BFF0C9
	for <~euandreh/public-inbox@lists.sr.ht>; Tue,  1 Sep 2020 18:00:17 +0000 (UTC)
Authentication-Results: mail-b.sr.ht;
	dkim=pass (2048-bit key) header.d=euandre.org header.i=@euandre.org header.b=q+qyYbsV
Received: from authenticated-user (box.euandre.org [46.101.160.115])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	(No client certificate requested)
	by box.euandre.org (Postfix) with ESMTPSA id F050CFCB45;
	Tue,  1 Sep 2020 15:00:14 -0300 (-03)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=euandre.org; s=mail;
	t=1598983216; bh=HeMLnOZIeRgOrLdGepra14BxoBWpvE7fXfdvB9MFFjw=;
	h=From:To:Cc:Subject:In-Reply-To:References:Date:From;
	b=q+qyYbsVLZrz3+swol7Y+Sc59kNUZkM1tSaXTcgLVIOp3CSJ+96XlP7i6oKZdeubH
	 fZIdw2fmFCMQotaXCnLJKBUNauX5Jrvt+4t2TrujIie5i5o/VSp08DxtxyWAmUWqdt
	 nPZrAand3styW0vvpWFYCZ7RjJKs4Ujek2G4H713X/eDOTooKalFNIXZ+A7t/xzq+9
	 mYojhphqHBDFQeqNKs9500YqetgTg4hsYfqRmGHzOs5ADw1dkQegQhjduodWvj2SpI
	 YeUu4OoXYDYUR9o+6VkXV+KtC4zDnoXyvgdSfHaLrahnkfiCBgg1Y6Kmcxy8iKnXjP
	 q5baz6m+cXC9Q==
From: EuAndreh <eu@euandre.org>
To: Richard Newman <rnewman@twinql.com>,
 "~euandreh/public-inbox@lists.sr.ht" <~euandreh/public-inbox@lists.sr.ht>
Cc: 
Subject: Re: The database I wish I had
In-Reply-To: <010101744a592b75-1dce9281-f0b8-4226-9d50-fd2c7901fa72-000000@us-west-2.amazonses.com>
References: <7454DA45-C5A4-4A54-8D02-1F3BD37726E9@twinql.com>
 <010101744a592b75-1dce9281-f0b8-4226-9d50-fd2c7901fa72-000000@us-west-2.amazonses.com>
Date: Tue, 01 Sep 2020 14:50:07 -0300
Message-ID: <87sgc1xu7k.fsf@euandre.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Richard Newman <rnewman@twinql.com> writes:

> Hello! Original author of Mentat here.

Hi there! You wouldn't be suprised if I told I recognize you by name,
would you?

Great job on Mentat, and please write more installments on "Thinking
about Syncing" if possible =E2=98=BA=EF=B8=8F

> The use of SQLite for Mentat was intended to be an accelerant for
> development: by leveraging SQLite we could ignore large parts of the
> tricky problem of getting bits on disk across platforms, and also get
> things like transactionality and full-text search for free.

For your goal of supporting other teams working on Firefox, SQLite is a
perfectly reasonable choice.

> We used the same leverage trick when we built the first proof of
> concept using Clojure and ClojureScript =E2=80=94 we could run it in a JVM
> locally, compare to Datomic, and test in Node, and entirely avoid
> having to build parsers and lexers. This allowed us to focus on the
> problem of representing a datom/log/index data model in SQLite.=20
>
> The downside of using SQLite was a large amount of time writing code
> to compile queries =E2=80=94 which were written in much the same language=
 as
> Datomic=E2=80=99s =E2=80=94 into SQL for execution. That Datomic compatib=
ility also
> pushed us towards building an EDN parser in Rust, and we found some
> under-specified parts of Datomic, too.=20

Hmm, I'd be interested in learning more about those under-specified
pitfalls to try to avoid them if possible.=20

> Portability to the web itself was not an initial goal, but it was
> something we thought about. With adequate resourcing, and the benefit
> of hindsight, we would have instead built a chunk-based backing store
> abstraction, using something like rkv/LMDB for embedded use and
> indexeddb on the web. We did not feel like we had the engineering
> capacity to take on building a truly portable, battle-tested storage
> layer like SQLite=E2=80=99s *and* a bits-on-disk query executor =E2=80=94=
 I=E2=80=99ve done it
> in the past, and it takes a lot of time and is hard to get right.=20

I don't expect to match SQLite's robustness for quite some time, and
having plugable storage APIs makes me consider having a SQLite storage
API to have as a correctness reference.

What do you mean by "chunk-based backing store abstraction"? Could you
elaborate further?

> Mentat attempted to address a number of things that you didn=E2=80=99t co=
ver
> in your brief post. A robust definition of identity, end-to-end
> encryption, and evolution of schema (even simple schema like
> cardinality and uniqueness constraints on attributes, which are the
> foundation of identity) are three such. Schema evolution becomes
> tremendously important when you have multiple teams using the same
> storage, or a long-lived product =E2=80=94 the more useful the database i=
s,
> the more likely you are to need this feature!=20

What do you mean by "identity"? As in ":db.unique/value" and
":db.unique/identity"? Couldn't you just use UUIDs for
":db.unique/identity" values?

For schema evolution wouldn't the "Grow your schema, and never break
it"[0] approach be enough?

On encryption, my first approach would be to see if git-remote-gcrypt[1]
functionality can be translated, and have the local database be ignorant
of all encryption whatsoever. However, I haven't explored it enough to
be able to declare it to be possible, so I would welcome a more mature
view on the subject.

[0]: https://blog.datomic.com/2017/01/the-ten-rules-of-schema-growth.html=20
[1]: https://spwhitton.name/tech/code/git-remote-gcrypt/

> Those three things cascade into the problem of synchronization, which
> was my focus for a number of years.=20
>
> Synchronization is a hard problem. Doing it without server assistance
> is even harder, but is necessary for a portable and secure product.
> Synchronization of generic data =E2=80=94 that is, data where the
> synchronization tool itself doesn=E2=80=99t have its own model of the wor=
ld =E2=80=94
> is extremely tricky: everything from identifiers through to basic
> constraints needs to be aware of the possibility of change, and
> naturally it=E2=80=99s an iterative and distributed situation. Most of the
> approaches I found rely on a server and are for systems that omit
> schema, and are typically incomplete: e.g., detecting conflicts on
> individual attributes, leading to nonsensical results, and offload to
> the developer the problem of separately handling stable identifiers.=20

If sound and robust, what's the downside of detecting conflicts on
individual attributes?

If an Datomic-like entity with 10 attributes has 4 of them change by
different database instances, but only 1 change is on the same
attribute, would this type of detection offer more granularity instead?

Wouldn't the 5-tuple be the equivalent of a single line of code on git,
and aggregates are built on top of it?

From eu@euandre.org Tue Sep  1 18:00:22 2020
Received: from box.euandre.org (box.euandre.org [46.101.160.115])
	by mail-b.sr.ht (Postfix) with ESMTPS id 097B1FF0C9
	for <~euandreh/public-inbox@lists.sr.ht>; Tue,  1 Sep 2020 18:00:22 +0000 (UTC)
Authentication-Results: mail-b.sr.ht;
	dkim=pass (2048-bit key) header.d=euandre.org header.i=@euandre.org header.b=bsr8vYSW
Received: from authenticated-user (box.euandre.org [46.101.160.115])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	(No client certificate requested)
	by box.euandre.org (Postfix) with ESMTPSA id C1FCCFD6BB;
	Tue,  1 Sep 2020 15:00:19 -0300 (-03)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=euandre.org; s=mail;
	t=1598983221; bh=VxPcwqhwGX+tAOR9QBGKCYzKImUIi+BqaSfmrBhsR6g=;
	h=From:To:Cc:Subject:In-Reply-To:References:Date:From;
	b=bsr8vYSWGIarl0IY64Ew+GLFGsQhZe971tKRr72HG9/UJru3FMjjA0VXK5LmyRGAh
	 ZjamXQpzM0biS0NNhY6mhfrvHXGKaGr19E6I5NDokNZFc/T/hXD1qMzY4fgEkDmayl
	 fteqI7T//DT06yseOXOkfrKzNCDWKmSmlsOo9PZgSsAicwEsqxb/UqehXN6ON9vpG4
	 VCXsOW8y8/aQGR7/WL2M9rkTd2cf6p/csDiUTkGf8YMFu8Qz/u5MgzY/14LCShXzfL
	 S+iz36YnA/kIu1RjrOzR22rPPcQRfZE1MyS9B+h9QeHHWR7A1SnsfE8WnlhBCCRxU7
	 ZcKgKfiihKPxQ==
From: EuAndreh <eu@euandre.org>
To: Seph Gentle <me@josephg.com>, ~euandreh/public-inbox@lists.sr.ht
Cc: 
Subject: Re: Making a database
In-Reply-To: <9e513706-7ebc-4a40-8e5e-a5f94d84bc77@www.fastmail.com>
References: <d78064d2-5441-4275-96d9-463a08dfd7a9@www.fastmail.com>
 <87ft81zvi0.fsf@euandre.org>
 <9e513706-7ebc-4a40-8e5e-a5f94d84bc77@www.fastmail.com>
Date: Tue, 01 Sep 2020 14:56:23 -0300
Message-ID: <87pn75xtx4.fsf@euandre.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

> You kinda gave a good description of them in your post already. They
> work by requiring all peers to resolve concurrent edits to an object
> the same way. That could be last-writer-wins (based on time stamps
> like you said in your article). Or it could store all conflicting
> versions together to be resolved by the next reader (Eg riak). Or it
> could do something more clever (like merge changes using automerge or
> y.js or the like). Git is sort of a CRDT - it=E2=80=99s just it=E2=80=99s=
 3 way merge
> algorithm is a bit of a dogs breakfast as far as these things go.=20

My ignorace is really on the applicability of it. The "store all
conflicting versions together or to be resolved by the next reader"
sounds like what I'm thinking about, so maybe I'm trying to re-invent
CRDTs after all.

I've just bumped the priority of this topic to better understand those
nuances.=20

From rnewman@twinql.com Tue Sep  1 19:30:22 2020
Received: from a27-132.smtp-out.us-west-2.amazonses.com (a27-132.smtp-out.us-west-2.amazonses.com [54.240.27.132])
	by mail-b.sr.ht (Postfix) with ESMTPS id 67FA4FF130
	for <~euandreh/public-inbox@lists.sr.ht>; Tue,  1 Sep 2020 19:30:21 +0000 (UTC)
Authentication-Results: mail-b.sr.ht;
	dkim=pass (1024-bit key) header.d=twinql.com header.i=@twinql.com header.b=HXFRpUpY;
	dkim=pass (1024-bit key) header.d=amazonses.com header.i=@amazonses.com header.b=Fknn4Oqd
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple;
	s=qvs2k2sd2ivtrflyu2nswmravuqizjml; d=twinql.com; t=1598988619;
	h=Subject:From:To:Cc:Date:Mime-Version:Content-Type:Content-Transfer-Encoding:In-Reply-To:References:Message-Id;
	bh=Y4wqQqAQDJ1Sa/E5OhrtvJ+Yk3Uij2laWmG/aD8wLfw=;
	b=HXFRpUpY+Dt9MJW6Z60km/fTpPfIYak8NKLDvv6UeroaBfvWClO0loOtSYksmT88
	dWeTwau14008g4YQwDwEY8msngXXDx/9W05ArClEsNXDygcmOYea3eHhqVMcvS/dtuA
	QYV4fjMoIj9QsGiHAP/QKjampkgK6nHx+Qfhlti4=
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple;
	s=hsbnp7p3ensaochzwyq5wwmceodymuwv; d=amazonses.com; t=1598988619;
	h=Subject:From:To:Cc:Date:Mime-Version:Content-Type:Content-Transfer-Encoding:In-Reply-To:References:Message-Id:Feedback-ID;
	bh=Y4wqQqAQDJ1Sa/E5OhrtvJ+Yk3Uij2laWmG/aD8wLfw=;
	b=Fknn4Oqdb29M5jY4pJV5V9gX2yK3n+WkgnRWmjQl46DdAzcaL4dHoZvIbk8cQxSt
	v6z1Rdu/S1IvHVSIHLYloebUmPBBfmLzEksz9sh5yCGO4IJhfdIEk0CPgPvNNkXC7as
	DpyhbHSQzUMFT1EBYwmgHNK/DQMSwe+2Jy7SpfPg=
Subject: Re: The database I wish I had
From: =?UTF-8?Q?Richard_Newman?= <rnewman@twinql.com>
To: =?UTF-8?Q?EuAndreh?= <eu@euandre.org>
Cc: =?UTF-8?Q?=7Eeuandreh/public-inbox=40lists=2Esr=2Eht?=
  <~euandreh/public-inbox@lists.sr.ht>
Date: Tue, 1 Sep 2020 19:30:19 +0000
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
In-Reply-To: <87sgc1xu7k.fsf@euandre.org>
References: <7454DA45-C5A4-4A54-8D02-1F3BD37726E9@twinql.com> 
 <010101744a592b75-1dce9281-f0b8-4226-9d50-fd2c7901fa72-000000@us-west-2.amazonses.com> 
 <87sgc1xu7k.fsf@euandre.org> 
 <9652AAE0-10DF-4D13-ABE8-B30667BE27DD@twinql.com>
X-Priority: 3 (Normal)
X-Mailer: Amazon WorkMail
Thread-Index: AQHWgHcPJ4udfXYrRaOwhoO7EcsDJQAErSAlAAfQ7+g=
Thread-Topic: The database I wish I had
X-Original-Mailer: Apple Mail (2.3445.104.14)
X-Wm-Sent-Timestamp: 1598988618
Message-ID: <010101744b260d62-1bb508ab-647f-4877-bdf9-f988667d56e1-000000@us-west-2.amazonses.com>
X-SES-Outgoing: 2020.09.01-54.240.27.132
Feedback-ID: 1.us-west-2.An468LAV0jCjQDrDLvlZjeAthld7qrhZr+vow8irkvU=:AmazonSES

> Hi there! You wouldn't be suprised if I told I recognize you by name,=0D=
=0A> would you=3F=0D=0A>=20=0D=0A> Great job on Mentat, and please write =
more installments on "Thinking=0D=0A> about Syncing" if possible =E2=98=BA=
=EF=B8=8F=0D=0A=0D=0AOne day I do hope to return to them! Thank you for t=
he kind words.=0D=0A=0D=0A=0D=0A> Hmm, I'd be interested in learning more=
 about those under-specified=0D=0A> pitfalls to try to avoid them if poss=
ible.=20=0D=0A=0D=0AMostly the usual stuff you get when a thing is define=
d by its implementation:=0D=0A=0D=0Ahttps://github.com/mozilla/mentat/pul=
l/665=0D=0Ahttps://github.com/mozilla/mentat/issues/412=0D=0A=0D=0AThere =
are probably more if you poke around the issue tracker.=0D=0A=0D=0A=0D=0A=
> What do you mean by "chunk-based backing store abstraction"=3F Could yo=
u=0D=0A> elaborate further=3F=0D=0A=0D=0AI'm familiar with the way Allegr=
oGraph does storage, and fairly familiar with how Datomic does it.=0D=0A=0D=
=0AIn short, a datom store like this consists of the log (all datoms, wit=
h retractions, ordered by ID), and some current state (which Mentat calle=
d `datoms`), which consists of all current asserted datoms.=0D=0A=0D=0ATh=
e current state is typically stored in a number of redundant permutations=
 to support querying. In AllegroGraph (subject, predicate, object, graph,=
 id), it would maintain multiple indices: `spogi`, `posgi`, etc. Each of =
these store the same datoms in different orders.=0D=0A=0D=0AIf we have th=
e data=0D=0A=0D=0A```=0D=0A  :bob foaf:knows :alice=0D=0A  :alice foaf:na=
me "Alice"=0D=0A  :bob foaf:name "Bob"=0D=0A  :bob foaf:age 26=0D=0A  :al=
ice foaf:age 30=0D=0A```=0D=0A=0D=0Ait might appear in the `spogi` index =
as:=0D=0A=0D=0A```=0D=0A  :alice foaf:age 30=0D=0A  :alice foaf:name "Ali=
ce"=0D=0A  :bob foaf:age 26=0D=0A  :bob foaf:knows :alice=0D=0A  :bob foa=
f:name "Bob"=0D=0A```=0D=0A=0D=0Aand in the `posgi` index as:=0D=0A=0D=0A=
```=0D=0A  foaf:age 26 :bob=0D=0A  foaf:age 30 :alice=20=0D=0A  foaf:name=
 "Alice" :alice=20=0D=0A  foaf:name "Bob" :bob=20=0D=0A  foaf:knows :alic=
e :bob=20=0D=0A```=0D=0A=0D=0ANaturally these would be using efficient (i=
nterned) sortable representations of each term, with rows of fixed size, =
memory mapped on disk.=0D=0A=0D=0AThis allows you to implement query patt=
ern execution as an index walk.=0D=0A=0D=0A`SELECT =3Fs WHERE =3Fs foaf:n=
ame 'Alice'`=0D=0A=0D=0Afor example turns into a walk on `posgi`: you fin=
d the spot that corresponds to `foaf:name`, find the next place which is =
`Alice`, and then read out the `s`.=0D=0A=0D=0AWhen you insert data, you =
typically don't want to immediately regenerate all of these indices. Inst=
ead you accrue new *chunks*, which are just all of the datoms asserted in=
 some window of time, or within a particular transaction. The query engin=
e walks all of the relevant chunks and merges the results together. One c=
hunk might have Bob's name, and one might have Alice's.=0D=0A=0D=0ATypica=
lly _metaindices_ are used to help the query engine navigate which chunks=
 to use.=0D=0A=0D=0AAllegroGraph periodically merges chunks together: a f=
ully merged store is faster to query. Datomic goes the other way: it buil=
ds new merged chunks and keeps the old ones around, because those old chu=
nks represent the state of the database at a point in time! This is how y=
ou efficiently query past states in Datomic: it trades space in your Cass=
andra cluster for the ability to grab the right chunks for any query.=0D=0A=
=0D=0AYou can easily imagine that any system that allows you to put a nam=
e on some region of data can be used to store these chunks, and their met=
aindices, and the log itself. The only requirement is that the chunk stor=
e be durable, and ideally replicated and relatively fast to read from.=0D=
=0A=0D=0A=0D=0A> What do you mean by "identity"=3F As in ":db.unique/valu=
e" and=0D=0A> ":db.unique/identity"=3F Couldn't you just use UUIDs for=0D=
=0A> ":db.unique/identity" values=3F=0D=0A=0D=0AIf on Computer A I visit =
https://aclu.org, and on Computer B I visit https://aclu.org, my two comp=
uters will each give the same history item a different internal identifie=
r =E2=80=94 1234 and 4567, say.=0D=0A=0D=0AWhen I sync them, I need the d=
ata store to know that the visits I made to each page _were visits to the=
 same thing_.=0D=0A=0D=0AThere is more value to it than that (e.g., Datom=
ic's "lookup refs"), but it's important to recognize that identity is _do=
main specific_, and it is often _compound_ and even contextual or conditi=
onal.=0D=0A=0D=0AA large part of synchronization, and a large part of dat=
a modeling in general, consists of figuring out exactly what it is you're=
 trying to record data about, and how to identify it. It's tremendously i=
mportant in NoSQL systems because there is no foreign key constraint to h=
elp you, and changing an identifier later is difficult. It's just as impo=
rtant in syncing systems because these identifiers are the only thing you=
 have to help you merge data =E2=80=94 to zip two graphs together.=0D=0A=0D=
=0A=0D=0A> For schema evolution wouldn't the "Grow your schema, and never=
 break=0D=0A> it"[0] approach be enough=3F=0D=0A=0D=0AIt definitely helps=
; this is how teams generally try to work with DynamoDB, for example, tho=
ugh there the schema is implicit rather than explict.=0D=0A=0D=0AHowever,=
 you inevitably have data migrations to worry about=E2=80=A6 not to menti=
on that syncing makes this a distributed system, and so you cannot easily=
 coordinate schema changes in more than one place, and they will race wit=
h new data!=0D=0A=0D=0AThere's also the argument that coordinating change=
 in consumers and engineers is sometimes harder than evolving a vocabular=
y!=0D=0A=0D=0AIt's a complicated topic.=0D=0A=0D=0A=0D=0A> On encryption,=
 my first approach would be to see if git-remote-gcrypt[1]=0D=0A> functio=
nality can be translated, and have the local database be ignorant=0D=0A> =
of all encryption whatsoever. However, I haven't explored it enough to=0D=
=0A> be able to declare it to be possible, so I would welcome a more matu=
re=0D=0A> view on the subject.=0D=0A=0D=0AI have been relatively happy wi=
th SQLCipher, and I think the idea of block-level encryption is a good pl=
ace to put the abstraction. The thing you lose with any of these mechanis=
ms is, by definition, the ability for a central entity to help you with c=
onflict resolution. The best you can get is ordering of opaque chunks and=
 collision detection.=0D=0A=0D=0A=0D=0A> If sound and robust, what's the =
downside of detecting conflicts on=0D=0A> individual attributes=3F=0D=0A=0D=
=0AIt relies on extensive reification to achieve the property that we wan=
t: that each semantically conflicting write results in a conflicting attr=
ibute change.=0D=0A=0D=0AAs a contrived example example: notes have both =
a title and a body.=0D=0A=0D=0AA user has two computers.=0D=0A=0D=0AOn Co=
mputer A, the user decides to change their first note to something else, =
updating both the title and body.=0D=0A=0D=0AOn Computer B, they do the s=
ame thing to a different title and body.=0D=0A=0D=0AWith attribute-based =
conflict resolution it is possible to end up with a note that has a combi=
nation of A's title and B's body =E2=80=94 a combination that never exist=
ed on either computer.=0D=0A=0D=0AThis gets even worse when you treat del=
etion specially, when new assertions don't conflict but do alter domain s=
emantics, or when some of the conflicts affect the way the rest of the da=
ta is interpreted (e.g., when you have an identity property or a lookup r=
ef).=0D=0A=0D=0AThe only data modeling solution to this case is to introd=
uce an immutable 'note node', and have the _change_ be to have Note One p=
oint to a different note node =E2=80=94 one attribute change!=0D=0A=0D=0A=
In a JSON-ish system like Pouch you'd swap an entire JS object within the=
 same attribute, rather than having two different attributes.=0D=0A=0D=0A=
Developers =E2=80=94 at least, non-Clojure developers =E2=80=94 rarely th=
ink in these terms, and now everywhere else you need to make really caref=
ul decisions about whether you point to the 'note' or the 'node'. Getting=
 it wrong means dangling pointers.=0D=0A=0D=0AUsing per-attribute automat=
ic merging without having an expert ontologist model your domain is a rec=
ipe for weird edge-case conflict resolution bugs that result in data soup=
 or data loss.=0D=0A=20=0D=0A=0D=0A> If an Datomic-like entity with 10 at=
tributes has 4 of them change by=0D=0A> different database instances, but=
 only 1 change is on the same=0D=0A> attribute, would this type of detect=
ion offer more granularity instead=3F=0D=0A=0D=0AMy thinking is mostly: y=
ou need some kind of schema mechanism to be able to recognize which prope=
rties can be reconciled safely, and you also cannot examine each tuple in=
dependently in many cases. It's complicated. Something like Dropbox's dat=
astore conflict resolution (if I remember right) is fine much of the time=
, but wrong some of the time.=0D=0A=0D=0A=0D=0A> Wouldn't the 5-tuple be =
the equivalent of a single line of code on git,=0D=0A> and aggregates are=
 built on top of it=3F=0D=0A=0D=0AEven source code merging isn't this sim=
ple, no=3F A conceptual operation like "add this line underneath this oth=
er line" is the thing you're trying to represent, but that line has _cont=
ext_ =E2=80=94 it might have moved, the file might have been renamed, or =
someone else already put a line there.=0D=0A=0D=0AYou've probably seen th=
is with b0rked merges in software version control, where the result of a =
merge won't compile because of duplicate identifiers thanks to hunks movi=
ng around=E2=80=A6=0D=0A=0D=0ABut trying to apply text-based merging to s=
emantic data only works if there are no constraints in your data, or if y=
ou have fast-forward merges =E2=80=94 merging can violate uniqueness prop=
erties in all kinds of ways.=20=0D=0A

From me@josephg.com Tue Sep  1 22:13:15 2020
Received: from wout4-smtp.messagingengine.com (wout4-smtp.messagingengine.com [64.147.123.20])
	by mail-b.sr.ht (Postfix) with ESMTPS id 53A0EFF124
	for <~euandreh/public-inbox@lists.sr.ht>; Tue,  1 Sep 2020 22:13:14 +0000 (UTC)
Authentication-Results: mail-b.sr.ht;
	dkim=pass (1024-bit key) header.d=josephg.com header.i=@josephg.com header.b=2Kj6nHib;
	dkim=fail reason="key not found in DNS" (0-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=eH2pUNJ6
Received: from compute1.internal (compute1.nyi.internal [10.202.2.41])
	by mailout.west.internal (Postfix) with ESMTP id CB8F377E;
	Tue,  1 Sep 2020 18:13:12 -0400 (EDT)
Received: from imap35 ([10.202.2.85])
  by compute1.internal (MEProxy); Tue, 01 Sep 2020 18:13:12 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=josephg.com; h=
	mime-version:message-id:in-reply-to:references:date:from:to
	:subject:content-type:content-transfer-encoding; s=mesmtp; bh=Q9
	PUfqkL153ICDLSRfraWMnWRXOclTHibxQiOyAV3lA=; b=2Kj6nHibSzEldHCTUN
	lgQhb0piuopatnU3wV94ozxZ95sUW7zyCnSE/s9gqFgTgYg0o2ZeqMffp92mRZis
	frDqWjWNhxv4WN78vO8dcwA8U/a+Uwp4cfxUyjuyY/zw7NHHM8LORDQCfembOUFj
	O6a5SW/fQRgU9XJIemyvxSDlM=
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
	messagingengine.com; h=content-transfer-encoding:content-type
	:date:from:in-reply-to:message-id:mime-version:references
	:subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender
	:x-sasl-enc; s=fm3; bh=Q9PUfqkL153ICDLSRfraWMnWRXOclTHibxQiOyAV3
	lA=; b=eH2pUNJ6n7/Ng/S9WTH+X0QePTCyzr5SjXdLgG3eN4a4yktWOtaMr44AU
	nW7SE782cX3xP2TEyvVSJ58tnx6r+6M5Jzauja+b2OJiTtnrJyWfIZ6iK/URp/y2
	v++7NoAfWmyXOfUQAiLbK92FCRrRkR8nGx0EksjMInRiDUozhHcg1mRYRipvULUx
	UsaQ0gtdGNDm76c6VwZU7YwBxssaeVQNP8ROlvm/y/KC3gpLYILLdNJPcI/r1846
	FrhxPQLgAQIe+4UV6NYtjNIRbFNPjdIWHZvWZ4LdMRPQ4yFdbpD6aHhKa+wLBPyB
	sIgGBCGwzMhfTDSRZQw8DrWpcx21w==
X-ME-Sender: <xms:dsdOX2zlGD9BWCFDAMf7fpxK8tc8KYm5XDsDnDguZM2Sy8xflOH9kA>
    <xme:dsdOXyTTPqu0-Pa3UYN2l_7sXVE04FX8AClSMYgSkAoeLeTspezWpQUlQYsja54Jc
    wrAfUl58HrBhQRp6w>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduiedrudefkedgtdeiucetufdoteggodetrfdotf
    fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen
    uceurghilhhouhhtmecufedttdenucenucfjughrpefofgggkfgjfhffhffvufgtgfesth
    hqredtreerjeenucfhrhhomhepfdfuvghphhcuifgvnhhtlhgvfdcuoehmvgesjhhoshgv
    phhhghdrtghomheqnecuggftrfgrthhtvghrnhepteeugfeufeeludekieduiefggfehhf
    eiffelfeetteejheehveelteffhefffeeunecuffhomhgrihhnpehklhgvphhpmhgrnhhn
    rdgtohhmnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomh
    epmhgvsehjohhsvghphhhgrdgtohhm
X-ME-Proxy: <xmx:dsdOX4WWAjDIjt91xpvOW8wGgg-eiGlTvG2yYFGPVBe2qPkuU169eg>
    <xmx:dsdOX8gA1SFXeEhDY-io8js7YHP8z7rKmyRtM81cBA0MFgLBRHhtow>
    <xmx:dsdOX4DJCCv4jHxex2u1y3w6cvLGKhAfSjHmtdhvaRTHlJcfSu4J5w>
    <xmx:eMdOX7M12Nj2n8pxrtLuGPyovCEq1wtzixBdJWACKU6e8qYcMiQKKg>
Received: by mailuser.nyi.internal (Postfix, from userid 501)
	id 762F414C00B3; Tue,  1 Sep 2020 18:13:10 -0400 (EDT)
X-Mailer: MessagingEngine.com Webmail Interface
User-Agent: Cyrus-JMAP/3.3.0-248-gcd102cb-fm-20200901.001-gcd102cb9
Mime-Version: 1.0
Message-Id: <bab08df6-be8e-47bd-a254-7b86c41a9b47@www.fastmail.com>
In-Reply-To: <87pn75xtx4.fsf@euandre.org>
References: <d78064d2-5441-4275-96d9-463a08dfd7a9@www.fastmail.com>
 <87ft81zvi0.fsf@euandre.org>
 <9e513706-7ebc-4a40-8e5e-a5f94d84bc77@www.fastmail.com>
 <87pn75xtx4.fsf@euandre.org>
Date: Wed, 02 Sep 2020 08:12:45 +1000
From: "Seph Gentle" <me@josephg.com>
To: EuAndreh <eu@euandre.org>, ~euandreh/public-inbox@lists.sr.ht
Subject: Re: Making a database
Content-Type: text/plain;charset=utf-8
Content-Transfer-Encoding: quoted-printable

On Wed, Sep 2, 2020, at 3:56 AM, EuAndreh wrote:
> > You kinda gave a good description of them in your post already. They=

> > work by requiring all peers to resolve concurrent edits to an object=

> > the same way. That could be last-writer-wins (based on time stamps
> > like you said in your article). Or it could store all conflicting
> > versions together to be resolved by the next reader (Eg riak). Or it=

> > could do something more clever (like merge changes using automerge o=
r
> > y.js or the like). Git is sort of a CRDT - it=E2=80=99s just it=E2=80=
=99s 3 way merge
> > algorithm is a bit of a dogs breakfast as far as these things go.=20=

>=20
> My ignorace is really on the applicability of it. The "store all
> conflicting versions together or to be resolved by the next reader"
> sounds like what I'm thinking about, so maybe I'm trying to re-invent
> CRDTs after all.
>=20
> I've just bumped the priority of this topic to better understand those=

> nuances.=20

This is worth watching if you haven=E2=80=99t seen it. Martin is at the =
forefront of the field - but this gives you a sense of where the researc=
h tech is at:

https://martin.kleppmann.com/2020/07/06/crdt-hard-parts-hydra.html

From eu@euandre.org Wed Sep  2 00:00:16 2020
Received: from box.euandre.org (box.euandre.org [46.101.160.115])
	by mail-b.sr.ht (Postfix) with ESMTPS id E53C0FF14B
	for <~euandreh/public-inbox@lists.sr.ht>; Wed,  2 Sep 2020 00:00:15 +0000 (UTC)
Authentication-Results: mail-b.sr.ht;
	dkim=pass (2048-bit key) header.d=euandre.org header.i=@euandre.org header.b=Ts/gQOWS
Received: from authenticated-user (box.euandre.org [46.101.160.115])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	(No client certificate requested)
	by box.euandre.org (Postfix) with ESMTPSA id E5F6DFC052;
	Tue,  1 Sep 2020 21:00:11 -0300 (-03)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=euandre.org; s=mail;
	t=1599004813; bh=GS02iBK+iDwoGCfVqsjbqMenu+Wpn9OO6GpSXGHcpsY=;
	h=From:To:Cc:Subject:In-Reply-To:References:Date:From;
	b=Ts/gQOWSiM5Vq/pHxaNuXKM4mXoy3VW9rD5mv/uLMGHYQcslItj1TNcvxINLzU/yj
	 ZaBmqtENnSd9AXap4PHxsFz48JlDI5oYcn98HE9nLsAZrD7/9GkhJMNjRDsf0ODAl9
	 W7do6QS8LRAZo0RhTguAzgoTJXf/IxuRqykG9Nm8imNuLDN4T04EbGvRw4ibN4UhlT
	 DED1G5O4Ul96BT6mY/im9tBQPGGNURLLcNcUkqG4fpddJrg58/yH9JOPMhnNhatEiV
	 39+m3xyL2rM8HSBfKun87TXjE6CD4/kQ0U6OpSB9ej/4zdNJ7Ei9K5NSuF78KRvkWJ
	 qHKRHqP/84rQw==
From: EuAndreh <eu@euandre.org>
To: Seph Gentle <me@josephg.com>, ~euandreh/public-inbox@lists.sr.ht
Cc: 
Subject: Re: Making a database
In-Reply-To: <bab08df6-be8e-47bd-a254-7b86c41a9b47@www.fastmail.com>
References: <d78064d2-5441-4275-96d9-463a08dfd7a9@www.fastmail.com>
 <87ft81zvi0.fsf@euandre.org>
 <9e513706-7ebc-4a40-8e5e-a5f94d84bc77@www.fastmail.com>
 <87pn75xtx4.fsf@euandre.org>
 <bab08df6-be8e-47bd-a254-7b86c41a9b47@www.fastmail.com>
Date: Tue, 01 Sep 2020 20:18:08 -0300
Message-ID: <87eenlxf0v.fsf@euandre.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

"Seph Gentle" <me@josephg.com> writes:

> This is worth watching if you haven=E2=80=99t seen it. Martin is at the f=
orefront of the field - but this gives you a sense of where the research te=
ch is at:
>
> https://martin.kleppmann.com/2020/07/06/crdt-hard-parts-hydra.html

Coincidentally, that's the  third item on my list =F0=9F=98=AC

From jeff@rhyason.org Wed Sep  2 01:35:17 2020
Received: from mx3.rhyason.org (mx3.rhyason.org [104.245.39.21])
	by mail-b.sr.ht (Postfix) with ESMTPS id 6C175FF0E7
	for <~euandreh/public-inbox@lists.sr.ht>; Wed,  2 Sep 2020 01:35:17 +0000 (UTC)
Authentication-Results: mail-b.sr.ht;
	dkim=pass (1024-bit key) header.d=rhyason.org header.i=@rhyason.org header.b=weEVa7Zv
Received: from rhyason.org (d173-183-32-226.bchsia.telus.net [173.183.32.226])
	by mx3.rhyason.org (8.15.2/8.15.2) with ESMTP id 0821ZI7G073010
	for <~euandreh/public-inbox@lists.sr.ht>; Tue, 1 Sep 2020 21:35:19 -0400 (EDT)
	(envelope-from jeff@rhyason.org)
Received: from rhyason.org (localhost [127.0.0.1]);
	by rhyason.org (OpenSMTPD) with ESMTP id 4aa4e076;
	for <~euandreh/public-inbox@lists.sr.ht>;
	Tue, 1 Sep 2020 19:35:08 -0600 (MDT)
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=rhyason.org; h=from
	:content-type:content-transfer-encoding:mime-version:subject
	:message-id:date:to; s=selector1; bh=61RPMANfo/dmUC5BfkkO8NNOMrg
	=; b=weEVa7ZvWqlxTjSrAxMCoiPy3GBkyNONviKR6lPNlGM3/eV5+0g3KhjKIKs
	g5+xXXB1G69y1xvNyo5hpOYY4Gk51OeGCQ/QxtE3tD2IgTxYjWlsB5SuNTrWI6Kd
	Onz8i5Fdan7ZigJ1ambirCGyI/IWPb8Q8gG7+i76Wb2vj3W8=
DomainKey-Signature: a=rsa-sha1; c=nofws; d=rhyason.org; h=from
	:content-type:content-transfer-encoding:mime-version:subject
	:message-id:date:to; q=dns; s=selector1; b=mpt0BK56C+LRHqEw8Bx8M
	y7nJWXZqD5kxPw4J7wh3mh8VcMYiBuA/2BpbrT7wYH9IxpMGFjlSP/lIZIHUPFTt
	cpbgqiL6CrW1zvP4CJ+6ygSQkfnsEGekcmEgLGa/E49lcKxA/y7Ep1G7maJJIZxN
	3OqgdtRLK1MiajxwW/MvBA=
Received: from [10.0.88.117] (10.0.88.117 [10.0.88.117]);
	by rhyason.org (OpenSMTPD) with ESMTPSA id d7674b26;
	TLS version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO;
	for <~euandreh/public-inbox@lists.sr.ht>;
	Tue, 1 Sep 2020 19:35:08 -0600 (MDT)
From: Jeff Rhyason <jeff@rhyason.org>
Content-Type: text/plain;
	charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.1\))
Subject: building MSTs->CRDTs-> Re: The database I wish I had
Message-Id: <99C92E06-E0BC-4DB0-BF63-20FD44DC4062@rhyason.org>
Date: Tue, 1 Sep 2020 18:35:08 -0700
To: ~euandreh/public-inbox@lists.sr.ht
X-Mailer: Apple Mail (2.3608.120.23.2.1)

It's amazing to see this discussion. Let's keep it going!

My friend and I started Indelible[1] (now defunct) for a lot of the same =
reasons you want to build a datastore -- there aren't enough options for =
a datastore that's immutable, relational, highly-vaailable, and =
private.. We're still super-passionate about the ideas, love Datomic but =
want something a bit different, decided not to commercialize it and =
start more simply with just the data structure. We don't want a huge =
thing, just a library, to keep coordination down; the complexity doesn't =
spread out of the client.

Where we are at now is we have a great versioned immutable B-Tree-like =
structure that is the foundation for the verioned and diffable database =
we want.  FWIW The Go implementation of the tree is at =
https://github.com/jrhy/mast .  We have plans to do a CRDT based on it, =
possibly add an SQL layer, but frankly, having a strongly-consistent, =
verioned + diffable tree, is pretty good for now. We'd love to =
collaborate!



[1] Indelible API Reference =
http://htmlpreview.github.io/?https://github.com/indeliblesystems/indelibl=
e/blob/master/doc/000-API-ref.html


From eu@euandre.org Wed Sep  2 08:00:14 2020
Received: from box.euandre.org (box.euandre.org [46.101.160.115])
	by mail-b.sr.ht (Postfix) with ESMTPS id E2229FF148
	for <~euandreh/public-inbox@lists.sr.ht>; Wed,  2 Sep 2020 08:00:13 +0000 (UTC)
Authentication-Results: mail-b.sr.ht;
	dkim=pass (2048-bit key) header.d=euandre.org header.i=@euandre.org header.b=Gv++l+b4
Received: from authenticated-user (box.euandre.org [46.101.160.115])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	(No client certificate requested)
	by box.euandre.org (Postfix) with ESMTPSA id BBE37FC052;
	Wed,  2 Sep 2020 05:00:10 -0300 (-03)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=euandre.org; s=mail;
	t=1599033612; bh=L0rF3hNESOLacI6plS3F0G5McLpB7MuPNAKgItTuBzg=;
	h=From:To:Cc:Cc:Subject:In-Reply-To:References:Date:From;
	b=Gv++l+b4mbVug6D7ywconYDZ/EzLKxn27zY5F1FhWm2A97mClGgf20SVzjqrxZNOE
	 LHVxQX1qwB7I/MzwWDf0ACHFQsOu0Y2VgQyffVE0BHInz/TPbXlv9Qbfe1L9yRFvD7
	 Z+3BoX5//m/1Na1ZKeuP1KT5neo8dF75wyyUhG0Zh/hFEDa1ILdR8wrnovtouqmwxA
	 JoMH2EzjLJ8ad2pox2VU0BskunOUvIX/CRXsrnXDCxybseFECE7h2jghegme7MlLae
	 /kvzmmg6Ow1z/lzvftKWGJRPb47MvQYSFIbnLb2kkkhTb6evJ5eXPO2nPY6/vJ8t6c
	 jhgzwfN/DzLeA==
From: EuAndreh <eu@euandre.org>
To: Richard Newman <rnewman@twinql.com>
Cc: "~euandreh/public-inbox@lists.sr.ht" <~euandreh/public-inbox@lists.sr.ht>
Cc: 
Subject: Re: The database I wish I had
In-Reply-To: <010101744b260d62-1bb508ab-647f-4877-bdf9-f988667d56e1-000000@us-west-2.amazonses.com>
References: <7454DA45-C5A4-4A54-8D02-1F3BD37726E9@twinql.com>
 <010101744a592b75-1dce9281-f0b8-4226-9d50-fd2c7901fa72-000000@us-west-2.amazonses.com>
 <87sgc1xu7k.fsf@euandre.org>
 <9652AAE0-10DF-4D13-ABE8-B30667BE27DD@twinql.com>
 <010101744b260d62-1bb508ab-647f-4877-bdf9-f988667d56e1-000000@us-west-2.amazonses.com>
Date: Wed, 02 Sep 2020 04:30:43 -0300
Message-ID: <878sdsy6sc.fsf@euandre.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

This is some rich discussion, thanks for engaging!

Richard Newman <rnewman@twinql.com> writes:

> Mostly the usual stuff you get when a thing is defined by its
> implementation:
>
> https://github.com/mozilla/mentat/pull/665
> https://github.com/mozilla/mentat/issues/412
>
> There are probably more if you poke around the issue tracker.

Thanks for the references.

> I'm familiar with the way AllegroGraph does storage, and fairly
> familiar with how Datomic does it.

I've used Common Lisp a lot before, and I heard a lot about
AllegroGraph, but I've never worked with it. Would you recommed it as a
place to take inspirations from?

> In short, a datom store like this consists of the log (all datoms,
> with retractions, ordered by ID), and some current state (which Mentat
> called `datoms`), which consists of all current asserted datoms.
>
> The current state is typically stored in a number of redundant
> permutations to support querying. In AllegroGraph (subject, predicate,
> object, graph, id), it would maintain multiple indices: `spogi`,
> `posgi`, etc. Each of these store the same datoms in different orders.
>
> If we have the data
>
> ```
>   :bob foaf:knows :alice
>   :alice foaf:name "Alice"
>   :bob foaf:name "Bob"
>   :bob foaf:age 26
>   :alice foaf:age 30
> ```
>
> it might appear in the `spogi` index as:
>
> ```
>   :alice foaf:age 30
>   :alice foaf:name "Alice"
>   :bob foaf:age 26
>   :bob foaf:knows :alice
>   :bob foaf:name "Bob"
> ```
>
> and in the `posgi` index as:
>
> ```
>   foaf:age 26 :bob
>   foaf:age 30 :alice
>   foaf:name "Alice" :alice
>   foaf:name "Bob" :bob
>   foaf:knows :alice :bob
> ```

These look just like the EAVT and AVET indexes in Datomic.

> Naturally these would be using efficient (interned) sortable
> representations of each term, with rows of fixed size, memory mapped
> on disk.
>
> This allows you to implement query pattern execution as an index walk.
>
> `SELECT ?s WHERE ?s foaf:name 'Alice'`
>
> for example turns into a walk on `posgi`: you find the spot that
> corresponds to `foaf:name`, find the next place which is `Alice`, and
> then read out the `s`.
>
> When you insert data, you typically don't want to immediately
> regenerate all of these indices. Instead you accrue new *chunks*,
> which are just all of the datoms asserted in some window of time, or
> within a particular transaction. The query engine walks all of the
> relevant chunks and merges the results together. One chunk might have
> Bob's name, and one might have Alice's.
>
> Typically _metaindices_ are used to help the query engine navigate
> which chunks to use.
>
> AllegroGraph periodically merges chunks together: a fully merged store
> is faster to query. Datomic goes the other way: it builds new merged
> chunks and keeps the old ones around, because those old chunks
> represent the state of the database at a point in time! This is how
> you efficiently query past states in Datomic: it trades space in your
> Cassandra cluster for the ability to grab the right chunks for any
> query.
>
> You can easily imagine that any system that allows you to put a name
> on some region of data can be used to store these chunks, and their
> metaindices, and the log itself. The only requirement is that the
> chunk store be durable, and ideally replicated and relatively fast to
> read from.

The most intriguing part of this for me is trying to mimic Datomic's
implementation of those merged chunks, specifically how Rich Hickey
describes it as an equivalent of persistent data structures used for
storage, with sharing of previous versions, etc. I believe he calls it
"durable persistent" datastructure, like making things on disk to be
a bit similar on how they are on memory.

In fact, that's exactly where I'm focused at right now: deepening my
grasp on persistent data structures to enable me to navigate this
subject with more fluidity, and try to implement such durable persistent
data structures.

The merging of the AllegroGraph that you describe sounds like the flat
files merge of indexes that RH describes, and contrasts with his
proposal. I'll try to find this quote, and re-read the paper he mentions
on the talk (IIRC it's the BigTable one).

I was considering having all indexes be derived from the raw data
transported between instances, but it might get computationally
expensive, and making those indexes replicable sounds like a good idea.

> If on Computer A I visit https://aclu.org, and on Computer B I visit
> https://aclu.org, my two computers will each give the same history
> item a different internal identifier =E2=80=94 1234 and 4567, say.
>
> When I sync them, I need the data store to know that the visits I made
> to each page _were visits to the same thing_.

Isn't that a domain-specifc problem of identity?

I would solve this by allowing those two entries to cooexist without any
conflict, and do aggregation queries to answers those questions,
something like (pseudo-SQL):

SELECT DISTINCT(url) FROM urls;

...to show the all the visited URLs, and:

SELECT SUM(1) from URLS GROUP BY url

...to show the visit count for each URL.

WDYT?

> It definitely helps; this is how teams generally try to work with
> DynamoDB, for example, though there the schema is implicit rather than
> explict.
>
> However, you inevitably have data migrations to worry about=E2=80=A6 not =
to
> mention that syncing makes this a distributed system, and so you
> cannot easily coordinate schema changes in more than one place, and
> they will race with new data!
>
> There's also the argument that coordinating change in consumers and
> engineers is sometimes harder than evolving a vocabulary!
>
> It's a complicated topic.

On migrations:

I used to share your view on needing a migration scape hatch, but I
don't anymore. On the company I work those kind of database migrations
never really occur. People either create a new attribute, copy data from
a database to another, process it using ETL, compute things on the fly
with functions, and many other strategies. We've already had to comply
with excisions in production, but never on schema, only on data itself.

So I don't think that's an actual issue, and I would insist that the
"Grow your schema" approach actually does work at scale, on the long
run, for multiple scenarios.=20

> I have been relatively happy with SQLCipher, and I think the idea of
> block-level encryption is a good place to put the abstraction. The
> thing you lose with any of these mechanisms is, by definition, the
> ability for a central entity to help you with conflict resolution. The
> best you can get is ordering of opaque chunks and collision detection.

I believe that's not what I meant by comparing it to git-remote-gcrypt,
but maybe I misunderstood your point.

Here's a code snippet to try to highlight what I was referring to:

--8<---------------cut here---------------start------------->8---
$ pushd $(mktemp -d)
/tmp/tmp.iapFd5iBCr ~/
$ mkdir repo-1
$ cd repo-1/
$ git init
$ echo 'a' > file
$ git add file
$ git commit -m commit-a
[master (commit racine) c4112e4] commit-a
 1 file changed, 1 insertion(+)
 create mode 100644 file
$ git init --bare ../remote/
D=C3=A9p=C3=B4t Git vide initialis=C3=A9 dans /tmp/tmp.iapFd5iBCr/remote/
$ git remote add origin gcrypt::../remote
$ git push origin
gcrypt: Repository not found: ../remote
gcrypt: Setting up new repository
gcrypt: Remote ID is :id:KjDStP/7m18wJagDO0/o
=C3=89num=C3=A9ration des objets: 3, fait.
D=C3=A9compte des objets: 100% (3/3), fait.
Total 3 (delta 0), r=C3=A9utilis=C3=A9s 0 (delta 0), r=C3=A9utilis=C3=A9s d=
u pack 0
gcrypt: Encrypting to: --throw-keyids --default-recipient-self
gcrypt: Requesting manifest signature
To gcrypt::../remote
 * [new branch]      master -> master
$ cd ..
$ git clone gcrypt::remote repo-2
Clonage dans 'repo-2'...
gcrypt: Decrypting manifest
gpg: selecting card failed: No such device
gpg: Signature faite le mer. 02 sept. 2020 03:43:12 -03
gpg:                avec la clef RSA 5BDAE9B8B2F6C6BCBB0D6CE581F90EC3CD3560=
60
gpg: Bonne signature de =C2=AB=C2=A0EuAndreh <eu@euandre.org>=C2=A0=C2=BB [=
ultime]
gcrypt: Remote ID is :id:KjDStP/7m18wJagDO0/o
R=C3=A9ception d'objets: 100% (3/3), fait.
$ cd repo-2/
$ echo 'b' > file
$ git add file
$ git commit -m commit-b
[master a5f5f0b] commit-b
 1 file changed, 1 insertion(+), 1 deletion(-)
$ git remote set-url origin gcrypt::../remote
$ git push origin
gcrypt: Decrypting manifest
gpg: selecting card failed: No such device
gpg: Signature faite le mer. 02 sept. 2020 03:43:12 -03
gpg:                avec la clef RSA 5BDAE9B8B2F6C6BCBB0D6CE581F90EC3CD3560=
60
gpg: Bonne signature de =C2=AB=C2=A0EuAndreh <eu@euandre.org>=C2=A0=C2=BB [=
ultime]
=C3=89num=C3=A9ration des objets: 3, fait.
D=C3=A9compte des objets: 100% (3/3), fait.
Total 3 (delta 0), r=C3=A9utilis=C3=A9s 0 (delta 0), r=C3=A9utilis=C3=A9s d=
u pack 0
gcrypt: Encrypting to: --throw-keyids --default-recipient-self
gcrypt: Requesting manifest signature
To gcrypt::../remote
   c4112e4..a5f5f0b  master -> master
$ cd ../repo-1/
$ echo 'c' > file
$ git add file
$ git commit -m commit-c
[master 3d71137] commit-c
 1 file changed, 1 insertion(+), 1 deletion(-)
--8<---------------cut here---------------end--------------->8---

At this point, repo-1 has changes that conflict with repo-2, and remote
doesn't know about it, but we can resolve it locally:

--8<---------------cut here---------------start------------->8---
$ git pull origin master
gcrypt: Decrypting manifest
gpg: selecting card failed: No such device
gpg: Signature faite le mer. 02 sept. 2020 03:44:34 -03
gpg:                avec la clef RSA 5BDAE9B8B2F6C6BCBB0D6CE581F90EC3CD3560=
60
gpg: Bonne signature de =C2=AB=C2=A0EuAndreh <eu@euandre.org>=C2=A0=C2=BB [=
ultime]
Depuis gcrypt::../remote
 * branch            master     -> FETCH_HEAD
Fusion automatique de file
CONFLIT (contenu) : Conflit de fusion dans file
La fusion automatique a =C3=A9chou=C3=A9 ; r=C3=A9glez les conflits et vali=
dez le r=C3=A9sultat.
$ cat file
<<<<<<< HEAD
c
=3D=3D=3D=3D=3D=3D=3D
b
>>>>>>> a5f5f0b6c780a7ab3f7e28f373b06fa9a10c443f
--8<---------------cut here---------------end--------------->8---

This is just the general behaviour that I'm aiming towards, but that is
really just a long shot, I still need to reasearch more on it.

At a glance I can't really tell if that is what you mean when citing
SQLCipher.=20

>> If sound and robust, what's the downside of detecting conflicts on
>> individual attributes?
>
> It relies on extensive reification to achieve the property that we
> want: that each semantically conflicting write results in a
> conflicting attribute change.=20
>
> As a contrived example example: notes have both a title and a body.=20
>
> A user has two computers.
>
> On Computer A, the user decides to change their first note to
> something else, updating both the title and body.=20
>
> On Computer B, they do the same thing to a different title and body.=20
>
> With attribute-based conflict resolution it is possible to end up with
> a note that has a combination of A's title and B's body =E2=80=94 a
> combination that never existed on either computer.=20
>
> This gets even worse when you treat deletion specially, when new
> assertions don't conflict but do alter domain semantics, or when some
> of the conflicts affect the way the rest of the data is interpreted
> (e.g., when you have an identity property or a lookup ref).=20
>=20
> The only data modeling solution to this case is to introduce an
> immutable 'note node', and have the _change_ be to have Note One point
> to a different note node =E2=80=94 one attribute change!=20
>=20
> In a JSON-ish system like Pouch you'd swap an entire JS object within
> the same attribute, rather than having two different attributes.=20
>
> Developers =E2=80=94 at least, non-Clojure developers =E2=80=94 rarely th=
ink in these
> terms, and now everywhere else you need to make really careful
> decisions about whether you point to the 'note' or the 'node'. Getting
> it wrong means dangling pointers.=20
>
> Using per-attribute automatic merging without having an expert
> ontologist model your domain is a recipe for weird edge-case conflict
> resolution bugs that result in data soup or data loss.=20

TBH I'm not aiming for automatic conflict resolution.

Why not use the granularity of attributes to detect conflict, but not to
resolve them?

Let me add to the scenario you presented: notes have a title, a body and
an author.

A user has two computers.

On Computer A, the user decides to change their first note to
something else, updating both the title and body.=20

On Computer B, they do the same thing to a different title and body.=20

On both cases, the author stays the same.

When the DB instance on Computer A gets data from Computer B, the query:

SELECT * FROM notes WHERE id =3D 1234;

would return both, and some metadata saying that it has a conflict. Now
the programmer needs to decide how to deal with these: use timestamps,
pick a winning device, sort by revision ID and pick the first, show them
on a UI to the user to resolve, etc.

Either way, the DB knows that the note has the title and body attributes
in conflict, but the author is fine. The aggregate entity itself isn't
in conflict, but some of it's attributes.

The situation that you describe where one would end up with a note that
is a combination of A's title and B's body would only happen if the DB
tried to solve things "automagically".

However, I agree on the costly constant reification being required, and
for long disconnected histories to be merged together this might be
expensive.=20

>> If an Datomic-like entity with 10 attributes has 4 of them change by
>> different database instances, but only 1 change is on the same
>> attribute, would this type of detection offer more granularity instead?
>
> My thinking is mostly: you need some kind of schema mechanism to be
> able to recognize which properties can be reconciled safely, and you
> also cannot examine each tuple independently in many cases. It's
> complicated. Something like Dropbox's datastore conflict resolution
> (if I remember right) is fine much of the time, but wrong some of the
> time.=20

Couldn't you leverage the schema for that? If the said conflicting
attribute has:

- cardinality one and both DBs changed it, we have a conflict;
- cardinality many and both changed the same attribute, we have a
  conflict;
- cardinality may and Computer A changed an entry and Computer B changed
  another one, we don't have a conflict.

and so on.

I wouldn't build it with an exaustive search, but just leverage the
schema to make those decisions.

I don't know Dropbox's datastore, is this[0] the one you're refferring
to?=20

[0]: https://dropbox.tech/developers/how-the-datastore-api-handles-conflict=
s-part-1-basics-of-offline-conflict-handling

> Even source code merging isn't this simple, no? A conceptual operation
> like "add this line underneath this other line" is the thing you're
> trying to represent, but that line has _context_ =E2=80=94 it might have
> moved, the file might have been renamed, or someone else already put a
> line there.=20
>
> You've probably seen this with b0rked merges in software version
> control, where the result of a merge won't compile because of
> duplicate identifiers thanks to hunks moving around=E2=80=A6=20
>
> But trying to apply text-based merging to semantic data only works if
> there are no constraints in your data, or if you have fast-forward
> merges =E2=80=94 merging can violate uniqueness properties in all kinds of
> ways.=20

Yep, that was an oversimplification.

<hr />

Again, thanks for engaging, this discussion feels like a golden nugget.

From me@jonas-schuermann.name Wed Sep  2 12:48:41 2020
Received: from mout-p-201.mailbox.org (mout-p-201.mailbox.org [80.241.56.171])
	by mail-b.sr.ht (Postfix) with ESMTPS id E843EFF14B
	for <~euandreh/public-inbox@lists.sr.ht>; Wed,  2 Sep 2020 12:48:40 +0000 (UTC)
Received: from smtp1.mailbox.org (smtp1.mailbox.org [80.241.60.240])
	(using TLSv1.2 with cipher ECDHE-RSA-CHACHA20-POLY1305 (256/256 bits))
	(No client certificate requested)
	by mout-p-201.mailbox.org (Postfix) with ESMTPS id 4BhNzC5xW2zQlX2;
	Wed,  2 Sep 2020 14:48:39 +0200 (CEST)
X-Virus-Scanned: amavisd-new at heinlein-support.de
Received: from smtp1.mailbox.org ([80.241.60.240])
	by spamfilter03.heinlein-hosting.de (spamfilter03.heinlein-hosting.de [80.241.56.117]) (amavisd-new, port 10030)
	with ESMTP id 3fpZQfKdp0ux; Wed,  2 Sep 2020 14:48:36 +0200 (CEST)
Subject: Re: The database I wish I had
To: EuAndreh <eu@euandre.org>, ~euandreh/public-inbox@lists.sr.ht
References: <e0b3b042-5eda-bcde-59a5-a5b61fecd2e5@jonas-schuermann.name>
 <87y2ltyeqg.fsf@euandre.org>
From: =?UTF-8?Q?Jonas_Sch=c3=bcrmann?= <me@jonas-schuermann.name>
Message-ID: <acd49073-c46e-2bae-da55-de1d8cc42533@jonas-schuermann.name>
Date: Wed, 2 Sep 2020 14:48:35 +0200
MIME-Version: 1.0
In-Reply-To: <87y2ltyeqg.fsf@euandre.org>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
X-MBO-SPAM-Probability: 
X-Rspamd-Score: -0.85 / 15.00 / 15.00
X-Rspamd-Queue-Id: 6F42037F
X-Rspamd-UID: dfaa19

> Hmmm, I haven't heard this term before. Are these [0] the complete 
> categories that you mean?
> 
> [0]: https://en.wikipedia.org/wiki/Complete_category

Yes, exactly. The idea is pretty abstract and while I know a little bit
about category theory I'm by no means an expert on the subject. This
blog post [1] is a very down-to-earth, practical explanation how the
concept of (co)completed categories can be applied to conflict
representation in version control systems.

[1] https://jneem.github.io/merging/

From rnewman@twinql.com Wed Sep  2 16:14:41 2020
Received: from a58-46.smtp-out.us-west-2.amazonses.com (a58-46.smtp-out.us-west-2.amazonses.com [54.240.58.46])
	by mail-b.sr.ht (Postfix) with ESMTPS id AF4DCFF12F
	for <~euandreh/public-inbox@lists.sr.ht>; Wed,  2 Sep 2020 16:14:40 +0000 (UTC)
Authentication-Results: mail-b.sr.ht;
	dkim=pass (1024-bit key) header.d=twinql.com header.i=@twinql.com header.b=grgVRwHH;
	dkim=pass (1024-bit key) header.d=amazonses.com header.i=@amazonses.com header.b=SGue23U5
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple;
	s=qvs2k2sd2ivtrflyu2nswmravuqizjml; d=twinql.com; t=1599063278;
	h=Subject:From:To:Cc:Date:Mime-Version:Content-Type:Content-Transfer-Encoding:In-Reply-To:References:Message-Id;
	bh=UZ3b7SzbsIcKi0Si4ZISwM274bq+ODQFNXqhUN8KRVo=;
	b=grgVRwHHOiZAo7asJgQxtXN5anM8LDtht/oVf7XVjh3IJNFPui4FSOPJK+Eh4Xdf
	MniQ9lndbRWU+3IR3fr3G6NKqYNdfpJb74/j/LvjnnrwZqxQQJFt99pQmwAKADGJAyG
	ZIwzp6kjG9pz3Uk1Bh+kRa2mHBIMmgfenRoqjiAI=
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple;
	s=hsbnp7p3ensaochzwyq5wwmceodymuwv; d=amazonses.com; t=1599063278;
	h=Subject:From:To:Cc:Date:Mime-Version:Content-Type:Content-Transfer-Encoding:In-Reply-To:References:Message-Id:Feedback-ID;
	bh=UZ3b7SzbsIcKi0Si4ZISwM274bq+ODQFNXqhUN8KRVo=;
	b=SGue23U5gqkhyLMKyKwkBGrqzxh9Ds7ua97mgDIB1BcWDmf58EDy2KhBrlkOtn0y
	IP8gxNfgYZDYdJFvPt8GmlkDg8nQNYW29iWNcf8ppMHCCEchNB85KrXAOtyBXYiR2oa
	nKriOwTP4/GeXifOl7+T8eIEkFuZI6DVL5Gas5SA=
Subject: Re: The database I wish I had
From: =?UTF-8?Q?Richard_Newman?= <rnewman@twinql.com>
To: =?UTF-8?Q?EuAndreh?= <eu@euandre.org>
Cc: =?UTF-8?Q?=7Eeuandreh/public-inbox=40lists=2Esr=2Eht?=
  <~euandreh/public-inbox@lists.sr.ht>
Date: Wed, 2 Sep 2020 16:14:38 +0000
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
In-Reply-To: <878sdsy6sc.fsf@euandre.org>
References: <7454DA45-C5A4-4A54-8D02-1F3BD37726E9@twinql.com> 
 <010101744a592b75-1dce9281-f0b8-4226-9d50-fd2c7901fa72-000000@us-west-2.amazonses.com> 
 <87sgc1xu7k.fsf@euandre.org> 
 <9652AAE0-10DF-4D13-ABE8-B30667BE27DD@twinql.com> 
 <010101744b260d62-1bb508ab-647f-4877-bdf9-f988667d56e1-000000@us-west-2.amazonses.com> 
 <878sdsy6sc.fsf@euandre.org> 
 <7027FD8B-1B30-435F-A840-7DC61A722E0B@twinql.com>
X-Priority: 3 (Normal)
X-Mailer: Amazon WorkMail
Thread-Index: AQHWgHcPJ4udfXYrRaOwhoO7EcsDJQAErSAlAAfQ7+gAIgKd3wAzRhy+
Thread-Topic: The database I wish I had
X-Original-Mailer: Apple Mail (2.3445.104.14)
X-Wm-Sent-Timestamp: 1599063277
Message-ID: <010101744f99448f-fa8dd88e-0daf-41f8-b3ca-6ea551c300e4-000000@us-west-2.amazonses.com>
X-SES-Outgoing: 2020.09.02-54.240.58.46
Feedback-ID: 1.us-west-2.An468LAV0jCjQDrDLvlZjeAthld7qrhZr+vow8irkvU=:AmazonSES

> I've used Common Lisp a lot before, and I heard a lot about=0D=0A> Alle=
groGraph, but I've never worked with it. Would you recommed it as a=0D=0A=
> place to take inspirations from=3F=0D=0A=0D=0AHelping to build it was a=
 very formative experience for me. twinql was the name of my first SPARQL=
 query engine, which became the genesis of AllegroGraph's, and it stuck a=
s a domain name!=0D=0A=0D=0ASome of AllegroGraph's performance is the res=
ult of decades of experience (Steve Haflich comes to mind) and having som=
e of the same exceptional developers work on it as worked on the compiler=
, so its internals are excellent inspiration. The UPI (how EAV are repres=
ented) is a good idea executed well. AllegroGraph is also a good example =
of how important tooling is for exploring this kind of semi-structured da=
ta.=0D=0A=0D=0A=0D=0A> These look just like the EAVT and AVET indexes in =
Datomic.=0D=0A=0D=0AYes, exactly! It's a very natural way to model a fixe=
d-width tuple store.=0D=0A=0D=0A=0D=0A> I was considering having all inde=
xes be derived from the raw data=0D=0A> transported between instances, bu=
t it might get computationally=0D=0A> expensive, and making those indexes=
 replicable sounds like a good idea.=0D=0A=0D=0AIt depends where you're r=
eplicating. In Datomic's case each of the query nodes gets a replicated c=
opy of the index chunks it needs (and you can thus control cache warmth b=
y routing queries to different nodes), because those nodes are meant to b=
e fungible, and reads from S3 (or wherever) are cheap, so there's no sens=
e doing the work twice.=0D=0A=0D=0AIf you were to stand up a hot spare, y=
ou would probably just stream the log and have the far end create its own=
 indices. That's particularly true if the other store needs different ind=
ices, or is materializing views.=0D=0A=0D=0ASidenote: it surprises me tha=
t Datomic Cloud stores its log in DynamoDB. It's hard to imagine a good p=
rimary key scheme that wouldn't at some point exhaust write capacity, and=
 DDB makes different availability and durability tradeoffs than S3. I'm c=
urious how that decision was made.=0D=0A=0D=0AObligatory disclaimer: I wo=
rk at AWS, but the above is not a statement on behalf of AWS!=0D=0A=0D=0A=
=0D=0A>> When I sync them, I need the data store to know that the visits =
I made=0D=0A>> to each page _were visits to the same thing_.=0D=0A>=20=0D=
=0A> Isn't that a domain-specifc problem of identity=3F=0D=0A=0D=0AYes, a=
bsolutely. I take the position that the goal of a database is to help me =
efficiently, accurately, and ergonomically represent my domain. If it doe=
sn't, every developer has to solve that problem at the application level,=
 and that limits interoperability, correctness, performance, and expressi=
veness.=0D=0A=0D=0AThe ability for this data layer to be able to help you=
 with problems like syncing also relies on its ability to leverage some i=
nformation about your domain =E2=80=94 compare to when you check in an im=
age file or a tarball to Git, and it can't help you detect or resolve con=
flicts at anything more granular than the containing file, even if that f=
ile has its own internal structure.=0D=0A=0D=0A=0D=0A> I would solve this=
 by allowing those two entries to cooexist without any=0D=0A> conflict, a=
nd do aggregation queries to answers those questions,=0D=0A> something li=
ke (pseudo-SQL):=0D=0A>=20=0D=0A> SELECT DISTINCT(url) FROM urls;=0D=0A>=20=
=0D=0A> ...to show the all the visited URLs, and:=0D=0A>=20=0D=0A> SELECT=
 SUM(1) from URLS GROUP BY url=0D=0A>=20=0D=0A> ...to show the visit coun=
t for each URL.=0D=0A>=20=0D=0A> WDYT=3F=0D=0A=0D=0AThe advantage is inde=
ed that you have removed the ability for data to conflict.=0D=0A=0D=0AThe=
 disadvantage is that you have pushed a domain identity and merging into =
application code, where:=0D=0A=0D=0A* Every query needs to implement this=
 kind of unification in multiple places, which is more expensive and horr=
endously difficult to get right, not to mention very hard for developers =
to read.=0D=0A=0D=0A* Application developers can get it wrong (e.g., dele=
ting from History needs to retract every entry with a given URL), and you=
 make domain operations more complex as a result. Sometimes this adds val=
ue =E2=80=94 e.g., this kind of accumulative storage gives you magic prop=
erties like lifetime undo! =E2=80=94 but I suspect that putting that in t=
he log is better.=0D=0A=0D=0A* You lose some conveniences like lookup ref=
s, because you've removed the concept of a 1:1 lookup altogether.=0D=0A=0D=
=0A* You lose the trivial ability to detect and model _changes_, which I =
find is natural for developers: this approach essentially makes the log p=
art of the datom state. You can still do retract/assert locally, for a pa=
rticular entity, but if you want to e.g., get the latest title for a page=
, you need to reduce over the log with some ordering.=0D=0A=0D=0A=0D=0AI =
agree with your position in principle: the idea of figuring out the curre=
nt state dynamically by rolling up history from different machines is ver=
y event-sourcey and is theoretically pleasant. Every device would simply =
append new non-conflicting _observations_ to the log. I do advocate for t=
his in some circumstances.=0D=0A=0D=0AHowever, in practice we usually hav=
e some stable idea of what we want current state to be (particularly of s=
chema itself!), and some domain concepts around identity and cardinality =
that we want to enforce and use to build concepts like change, conflict, =
and merging, and we need that maintained state to be fast and obvious bec=
ause we use it when making new assertions.=0D=0A=0D=0AIn essence, the pro=
blem I was trying to solve with syncing in Mentat was: how do I allow dev=
elopers to talk about their schema in such a way that they could get sync=
ing for nearly free=3F That a sync engine could, in most cases, automatic=
ally figure out which entities were the same, decide on a stable ordering=
 of operations, and rewrite history to produced the expected merged state=
, such that in the above example all the visits stitch together, we know =
the latest title, and the log has all the historical titles=3F=0D=0A=0D=0A=
Using observation-structured storage to remove conflict is definitely a v=
aluable tool in the toolbox for doing that: the more you can structure yo=
ur data as new observations about stably identified things, the easier ti=
me you have with syncing. I believe, but have not proved, that a system w=
ith only identity properties and cardinality-many properties, perhaps wit=
h some additional restrictions on retraction and uniqueness, will never h=
ave conflicts when merging.=0D=0A=0D=0ABut I also think that the best dev=
eloper ergonomics are actually slightly _more_ expressive than even Datom=
ic or Mentat =E2=80=94 developers told me that they wanted cardinality-N,=
 and to be able to express types or domain restrictions on properties, an=
d nesting, and so on =E2=80=94 and delivering those ergonomics means taki=
ng care of turning the log into a simple, concrete current state for them=
 to query.=0D=0A=0D=0APhrased differently: in the above, your developers =
want some way to write=0D=0A=0D=0A  SELECT url, title FROM pages=0D=0A=0D=
=0Aand=0D=0A=0D=0A  SET title FOR page WITH url 'foo.com' TO 'whatever'=0D=
=0A=0D=0AWhether you do that by figuring out at sync time the current tit=
le for pages with a given URL, or whether you give them some affordance t=
o dynamically roll that up from the log, gets to be the same thing if you=
 think of Mentat's 'datoms' table and schema as a way for developers to d=
escribe how Mentat should roll up current state from the log!=0D=0A=0D=0A=
=0D=0A> I used to share your view on needing a migration scape hatch, but=
 I=0D=0A> don't anymore. On the company I work those kind of database mig=
rations=0D=0A> never really occur. People either create a new attribute, =
copy data from=0D=0A> a database to another, process it using ETL, comput=
e things on the fly=0D=0A> with functions, and many other strategies. We'=
ve already had to comply=0D=0A> with excisions in production, but never o=
n schema, only on data itself.=0D=0A=0D=0AI actually view what you descri=
be as migration: the point at which you're defining a new attribute and s=
toring things twice, or doing a batch job to move things over, is exactly=
 a data migration, if perhaps smeared over space and time!=0D=0A=0D=0AThe=
se still have all the problems that traditional schema migrations have: d=
ealing with data that doesn't conform, and dealing with clients that don'=
t know about the new format.=0D=0A=0D=0A=0D=0A> So I don't think that's a=
n actual issue, and I would insist that the=0D=0A> "Grow your schema" app=
roach actually does work at scale, on the long=0D=0A> run, for multiple s=
cenarios.=20=0D=0A=0D=0AI agree. I also think that one of the ways it doe=
s so is by pushing the work of tracking versions and vocabulary into appl=
ication code, either formally or informally (e.g., checking for the exist=
ence of a property, coordinating the upgrade of client code, etc.) =E2=80=
=94 the clients need to know how to work with the schema as it grows!=0D=0A=
=0D=0AThis was the motivation for me spending time on vocabulary manageme=
nt, which we did end up building:=0D=0A=0D=0Ahttps://github.com/mozilla/m=
entat/wiki/Proposal:-application-schema-coordination-and-versioning=0D=0A=
=0D=0A=0D=0A> At a glance I can't really tell if that is what you mean wh=
en citing=0D=0A> SQLCipher.=20=0D=0A=0D=0AAh, we were talking about two d=
ifferent kinds of encryption!=0D=0A=0D=0AWe were using SQLCipher for loca=
l block-level encryption.=0D=0A=0D=0AAt Mozilla we would additionally use=
 Firefox Account encryption to end-to-end encrypt data as it passed throu=
gh Sync. The two are complementary. Sync acts exactly as you demonstrate =
with git-remote-gcrypt: two local devices can access cleartext and are re=
sponsible for resolving conflicts, and the service itself sees only ciphe=
rtext and does not involve itself in how conflicts are resolved.=0D=0A=0D=
=0AMy point in bringing this up is that many systems have a central entit=
y that can help in resolving conflicts; if the central entity can only ev=
er see ciphertext, it's limited to whatever ordering or collision propert=
ies you enable via the envelope. For that Git system I imagine it tracks =
refs, and the Git object store is opaque.=0D=0A=0D=0A=0D=0A> TBH I'm not =
aiming for automatic conflict resolution.=0D=0A>=20=0D=0A> Why not use th=
e granularity of attributes to detect conflict, but not to=0D=0A> resolve=
 them=3F=0D=0A=0D=0AIn short:=0D=0A=0D=0A* End user software is no longer=
 built (as iSync was) to present data conflicts to end users for resoluti=
on. The goal is conflict resolution.=0D=0A* You cannot leave a store in a=
n inconsistent state until conflicts have been addressed, because you sti=
ll need to handle local writes. You cannot disconnect a device until conf=
licts have been addressed; users expect continuity of service. Whenever y=
ou are able, you have to do _something_.=0D=0A* _Someone_ has to write th=
e code to resolve conflicts. Eventually you're going to have some dev com=
e up with the bright idea of annotating "data objects" with some kind of =
conflict resolution attributes, and then have an automated "conflict reso=
lver"=E2=80=A6 at which point you've Greenspun yourself a sync engine and=
 schema language. This works much better when it's part of the storage an=
d sync later, and can participate in transaction boundaries and annotate =
writes.=0D=0A=0D=0AThis is the doc I wrote to outline the motivations for=
 Mentat at Mozilla:=0D=0A=0D=0Ahttps://mozilla.github.io/firefox-browser-=
architecture/text/0008-sync-and-storage-review-packet.html=0D=0A=0D=0AThe=
 Lay Summary in particular is worth a read.=0D=0A=0D=0AWhen we built Fire=
fox for iOS we managed to put storage and syncing inside the same compone=
nt, so finally we were in a position to track changes during writes, and =
could get it right. Mentat was the next logical step: how do we let other=
 teams use a storage system that will still do a good job of syncing thei=
r data without them having to get elbow deep in how a sync engine works, =
and without them having to put all the syncing gunk =E2=80=94 timestamps/=
versions, UUIDs, etc. =E2=80=94 in their data model=3F=0D=0A=0D=0AWhat do=
es Mentat need to know, schema-wise, to be able to do that kind of declar=
ative conflict resolution=3F We didn't finish answering that question, bu=
t I think the intended outcome is still valid.=0D=0A=0D=0A=0D=0A> Either =
way, the DB knows that the note has the title and body attributes=0D=0A> =
in conflict, but the author is fine. The aggregate entity itself isn't=0D=
=0A> in conflict, but some of it's attributes.=0D=0A=0D=0AThis is what Co=
uch/Pouch do, I think. Well, it actually resolves the conflict and leaves=
 a conflict marker=E2=80=A6 no idea whether anyone actually writes code t=
o check that!=0D=0A=0D=0A=0D=0A> Couldn't you leverage the schema for tha=
t=3F If the said conflicting=0D=0A> attribute has:=0D=0A>=20=0D=0A> - car=
dinality one and both DBs changed it, we have a conflict;=0D=0A> - cardin=
ality many and both changed the same attribute, we have a=0D=0A>  conflic=
t;=0D=0A> - cardinality may and Computer A changed an entry and Computer =
B changed=0D=0A>  another one, we don't have a conflict.=0D=0A>=20=0D=0A>=
 and so on.=0D=0A=0D=0AYes, that's exactly the approach we were taking. T=
he important difference is that you also need to zip together identities =
during this process =E2=80=94 for example, two assertions of a cardinalit=
y-one attribute are not a conflict if the two entities at the other end t=
urn out to be the same. Couch etc. don't have that concept.=0D=0A=0D=0A(I=
n an open-world system like OWL-DL, this cardinality conflict actually be=
comes implication of equality!)=0D=0A=0D=0A> I don't know Dropbox's datas=
tore, is this[0] the one you're refferring=0D=0A> to=3F=20=0D=0A=0D=0AYep=
! It's a really great bit of thinking.=0D=0A=0D=0A

From eu@euandre.org Thu Sep  3 01:00:21 2020
Received: from box.euandre.org (box.euandre.org [46.101.160.115])
	by mail-b.sr.ht (Postfix) with ESMTPS id 71E69FF0FA
	for <~euandreh/public-inbox@lists.sr.ht>; Thu,  3 Sep 2020 01:00:20 +0000 (UTC)
Authentication-Results: mail-b.sr.ht;
	dkim=pass (2048-bit key) header.d=euandre.org header.i=@euandre.org header.b=rMNL9qSC
Received: from authenticated-user (box.euandre.org [46.101.160.115])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	(No client certificate requested)
	by box.euandre.org (Postfix) with ESMTPSA id E3DD6102122;
	Wed,  2 Sep 2020 22:00:16 -0300 (-03)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=euandre.org; s=mail;
	t=1599094818; bh=VQ0pHLJALciNNh1+kpFy1DVLAgL5kn8a36k3GAV6u7E=;
	h=From:To:Cc:Subject:In-Reply-To:References:Date:From;
	b=rMNL9qSC0p+9DH1RAD5wEPqn9/v63VAiwAiMKQuTvccTrH5EnO2qlB7mKU594brWM
	 P+5qCkBJymi/N2hK+BvoROdWT24A6NPyG0rORpg00rmuJsrpyn6HkpbHzOcx3CwYdH
	 WUVzXoZQndMFNa4nN4P/KE4Mh/jA/4TvVO/4QvfrJqpnpfKdPJeIgFEdrCEQL+3ZIP
	 b/DjqLRRSkV3fljHnB2MhsCMe/od5BD+wDEza/x+HvOwTxFHvalbWT1QVj3saKxiTG
	 VbD8xe43tls9agJsSh/z8MErjmNI4Ngd0ygfIu8QbbLNswkpQe3RMru5lYi2o49N55
	 1bnUqfFRkHs/w==
From: EuAndreh <eu@euandre.org>
To: Jonas =?utf-8?Q?Sch=C3=BCrmann?= <me@jonas-schuermann.name>,
 ~euandreh/public-inbox@lists.sr.ht
Cc: 
Subject: Re: The database I wish I had
In-Reply-To: <acd49073-c46e-2bae-da55-de1d8cc42533@jonas-schuermann.name>
References: <e0b3b042-5eda-bcde-59a5-a5b61fecd2e5@jonas-schuermann.name>
 <87y2ltyeqg.fsf@euandre.org>
 <acd49073-c46e-2bae-da55-de1d8cc42533@jonas-schuermann.name>
Date: Wed, 02 Sep 2020 21:34:51 -0300
Message-ID: <87d033g0k4.fsf@euandre.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Jonas Sch=C3=BCrmann <me@jonas-schuermann.name> writes:

> Yes, exactly. The idea is pretty abstract and while I know a little bit
> about category theory I'm by no means an expert on the subject. This
> blog post [1] is a very down-to-earth, practical explanation how the
> concept of (co)completed categories can be applied to conflict
> representation in version control systems.
>
> [1] https://jneem.github.io/merging/

This sounds interesting, thanks for the link.

From eu@euandre.org Thu Sep  3 01:07:15 2020
Received: from box.euandre.org (box.euandre.org [46.101.160.115])
	by mail-b.sr.ht (Postfix) with ESMTPS id 20958FF0FA
	for <~euandreh/public-inbox@lists.sr.ht>; Thu,  3 Sep 2020 01:07:15 +0000 (UTC)
Authentication-Results: mail-b.sr.ht;
	dkim=pass (2048-bit key) header.d=euandre.org header.i=@euandre.org header.b=Y4O+t3yg
Received: from authenticated-user (box.euandre.org [46.101.160.115])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	(No client certificate requested)
	by box.euandre.org (Postfix) with ESMTPSA id B209DFC81E;
	Wed,  2 Sep 2020 22:07:12 -0300 (-03)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=euandre.org; s=mail;
	t=1599095234; bh=Y2H9I+SgM0rzOycDHDHlfDDvDGVB79YoDxvEzTudos4=;
	h=From:To:Cc:Subject:In-Reply-To:References:Date:From;
	b=Y4O+t3yg4rBzXtK6CL570EiO+7TXdlHZ1OAjSDRUjr71MbaIt+HonABTUommz90+U
	 +FkpruCMrLNQIyuveS07REukjQVp/zKHrH41FLFDaRUoBIg1OuxvIp4I+Ug1M+HQAC
	 tS1tPA0oqbqmrI7l2ByFEP58VyYhRNpYAbrpSMJUI4gt/Busfq5TFLQfnuTZSaGbNy
	 9AsJYnD04/B+VRQHcrST55I8sVyaHkxQM0xck0J7iEnJesbTxGU6adg89UL7fTd4Jb
	 LP4PognYY5bvYrsRP1eTQVkm8Iefevp0X+fgNdQsvBkee8WLAykSq4jkMpPaBEyMl3
	 WIq9JcFaLACew==
From: EuAndreh <eu@euandre.org>
To: Jeff Rhyason <jeff@rhyason.org>, ~euandreh/public-inbox@lists.sr.ht
Cc: 
Subject: Re: building MSTs->CRDTs-> Re: The database I wish I had
In-Reply-To: <99C92E06-E0BC-4DB0-BF63-20FD44DC4062@rhyason.org>
References: <99C92E06-E0BC-4DB0-BF63-20FD44DC4062@rhyason.org>
Date: Wed, 02 Sep 2020 22:06:59 -0300
Message-ID: <87v9gveki4.fsf@euandre.org>
MIME-Version: 1.0
Content-Type: text/plain

Jeff Rhyason <jeff@rhyason.org> writes:

> It's amazing to see this discussion. Let's keep it going!
>
> My friend and I started Indelible[1] (now defunct) for a lot of the
> same reasons you want to build a datastore -- there aren't enough
> options for a datastore that's immutable, relational,
> highly-vaailable, and private.. We're still super-passionate about the
> ideas, love Datomic but want something a bit different, decided not to
> commercialize it and start more simply with just the data structure.
> We don't want a huge thing, just a library, to keep coordination down;
> the complexity doesn't spread out of the client.
>
> Where we are at now is we have a great versioned immutable B-Tree-like
> structure that is the foundation for the verioned and diffable
> database we want.  FWIW The Go implementation of the tree is at
> https://github.com/jrhy/mast .  We have plans to do a CRDT based on
> it, possibly add an SQL layer, but frankly, having a
> strongly-consistent, verioned + diffable tree, is pretty good for now.
> We'd love to collaborate!

This looks interesting! I'll take a deeper look later.

I'll be particularly interested in seeing the implementation of
persistent B-Trees that are durable.

From eu@euandre.org Thu Sep  3 23:39:44 2020
Received: from box.euandre.org (box.euandre.org [46.101.160.115])
	by mail-b.sr.ht (Postfix) with ESMTPS id B8167FF1D6
	for <~euandreh/public-inbox@lists.sr.ht>; Thu,  3 Sep 2020 23:39:43 +0000 (UTC)
Authentication-Results: mail-b.sr.ht;
	dkim=pass (2048-bit key) header.d=euandre.org header.i=@euandre.org header.b=oQR0LJk6
Received: from authenticated-user (box.euandre.org [46.101.160.115])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	(No client certificate requested)
	by box.euandre.org (Postfix) with ESMTPSA id 2F2B2FCB45;
	Thu,  3 Sep 2020 20:39:39 -0300 (-03)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=euandre.org; s=mail;
	t=1599176381; bh=0OTcSYr5H5tf2Nj44ZROMSLLvq65wMQATwFWuwPcGcA=;
	h=From:To:Cc:Cc:Subject:In-Reply-To:References:Date:From;
	b=oQR0LJk66tmm7FUnmKKEUg3GVM6KyB0ybBX7bvke8hAkcTr/UOADsGEa64ZOgq47F
	 5+6d43VfXXhxXQie9IypcAHdqe5eYTXL0Yh6ymknjM9GIPHEhGRgzbK+jeTe6EK2KH
	 y2bC4HqyQ0Xn9Ijj+m+OEQCACknw1vsIvVH3qQveSft3mRmIKIibpDL5uxMaGbNRHK
	 V0ts3LFy07Knz4QI5fsPRqBwsKNQV9t+F5eL5ITa6OSC3JY+QclEbcYheBW80V42IT
	 8+hb6HzscpFlxvBtdi1fG7PbnUtWAp/7j6sK8AtVYRCK5OybOAy0VpJhGGbFCZdale
	 s3u7XVTPdzmeQ==
From: EuAndreh <eu@euandre.org>
To: Richard Newman <rnewman@twinql.com>
Cc: "~euandreh/public-inbox@lists.sr.ht" <~euandreh/public-inbox@lists.sr.ht>
Cc: 
Subject: Re: The database I wish I had
In-Reply-To: <010101744f99448f-fa8dd88e-0daf-41f8-b3ca-6ea551c300e4-000000@us-west-2.amazonses.com>
References: <7454DA45-C5A4-4A54-8D02-1F3BD37726E9@twinql.com>
 <010101744a592b75-1dce9281-f0b8-4226-9d50-fd2c7901fa72-000000@us-west-2.amazonses.com>
 <87sgc1xu7k.fsf@euandre.org>
 <9652AAE0-10DF-4D13-ABE8-B30667BE27DD@twinql.com>
 <010101744b260d62-1bb508ab-647f-4877-bdf9-f988667d56e1-000000@us-west-2.amazonses.com>
 <878sdsy6sc.fsf@euandre.org>
 <7027FD8B-1B30-435F-A840-7DC61A722E0B@twinql.com>
 <010101744f99448f-fa8dd88e-0daf-41f8-b3ca-6ea551c300e4-000000@us-west-2.amazonses.com>
Date: Thu, 03 Sep 2020 20:39:23 -0300
Message-ID: <87h7se5t1w.fsf@euandre.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

I started quoting many parts of your email, and things started to get
confusing. Each answer that I gave was referring to each other, and I
almost made a loop.

So I've rewritten everything as a single piece of prose. I added some
snippets of your text to better contextualize what I was talking about,
but I tried to address what you said directly. Here it goes.

# On migrations

I still don't think that I've insisted enough on *strict* schema growth.

The wiki page you pointed to reinforces that: it talks a lot about how
to coordinate attributes that changed their names, change in vocabulary,
shared ownership of changes in schema and even schema versioning!

All of them are the opposite of what I want, and they're "schema
breakage", not "schema growth".

What I mean by that is the same thing that Rich Hickey proposes as a
what growth and breakage means in his talk "Spec-ulation"[0].

All of those forms of schema change are instances of breakage, which
should have no place here. They're all sources of undesired
incompatibility. I'm not sure if Datomic allows you to change the type
of an attribute, and if it does this can't be copied, since we don't
have a transactor to coordinate such change.

If you want to rename an attribute, don't! Create a new one! Maybe you
could retract schema and assert back the same schema, but you shouldn't
overwrite it with an incompatible name, or attribute type. This is a
trap that you shoudln't put yourself in, and maybe even this could be
enforced.

This model of an ever-growing vocabulary is how HTML works. At some
point people realised that using <b> to mark something as bold was not
such a good idea, since it coupled semantics with presentation. So a new
tag was created, <strong>, which only represents the semantic part of
<b>. And <b> still continues to exist, and that's fine.

The same can be said about JavaScript. I've written a fair share of it
in the past 2 years, and I don't think I ever used the 'var' keyword,
only 'let' and 'const'. And yet, every JavaScript runtime still
recognizes it, and even transpiler emit it as their output, and that's
also fine.

Maybe a linter or the browser tooling would warn you against the usage
of those old features, but they continue to work. If an equivalent of a
"vocabulary change" happened, web pages would be brittle by design more
brittle, and old pages would stop working as time passes. That isn't a
principle I would like to adopt.

When more than one database instance can add data, the programmer needs
to think of it as an open world, where new things are accrued, but old
things don't disapper. Changing the vocabulary mid-air only makes this
open world brittle so "migrations" can be performed.

I see no reason for having this type of migration you describe, and
insist on *strict* schema growth, it being a saner, more robust and
completly fine alternative.

[0]: https://www.youtube.com/watch?v=3DoyLBGkS5ICk

# On conflicts and ergonomics

You mentioned ergonomics thrice on your message, and I believe that your
insistence of it is a sign of the coupling of two different layers of
the database, and makes ease of use drive the fundamentals. I also think
that your statement on conflict resolution being a requirement is not a
good idea. Let me expand on that.

I'd argue that conflict resolution is inherit complexity [1] of creating
a decentralized client-side application that can sync data.

That means that the responsability to solve conflicts is not the
database's, but the application's responsability, and making decisions
of conflict resolution on the database limits its application, and I
wouldn't consider it to be general purpose.

When two offline database instances change the same field, there's no
escaping from the bottom reality, which is: there is a conflict, and it
needs to be resolved somewhere. There's no layer of ergonomics that can
change this fact, and picking a strategy for conflict resolution is
undoubtly domain-specific.

You mentioned that I proposed a solution on URL history tracking that
fowards all of the responsability to the programmer, but I was only
giving a simple example, and that same problem could be tackled in many
different ways, and the person creating the application has a decision
to make on it.

Let's expand on the URL history on the browser example from earlier, and
explore how to handle identity.

The application developer now has to decide what they want to consider a
URL being "the same" as another, and that's what I think you meant when
you talked about pushing domain identity to the application.

What does the programmer wants the identity of a URL to be? Is it a
UUID attributed to it when asserting the entitiy on the database? If so,
you would need to do things like the SELECT DISTINCT and SELECT SUM I
proposed, inevitably. The programmer may otherwise try to avoid this by
always starting from an existing database, so the history could be
common whenever a new database instance starts. One would just not
simply 'git init', they would 'git clone' in order to work with shared
identities that are created in a distributed way.

Is the identity of a URL its string? If so, the URL would have to be
treated as an URI, and that string would be effectively global, and
updates would have to be handled with lots of care.

The database *can't* make those decision. What it *can* do is provide
tools that allow the programmer to work with this inherent complexity.
And I see no way around it. No amout of ergonomics can solve this, and
removing the ability to have 1:1 lookups is the nature of decentralized
client-side applications, it's not me trying to forward a problem that
the database should solve to the programmer.=20

A simple comparison is: whenever we talk about code, we use SHAs to
pinpoint which piece of code we mean. What does a SHA say about the
existence of other branches that disagree? Nothing! You can't ever
assume that the commit you're on is the latest version of a file, or
even is the only version of a file. While you were not looking, any
number of new versions could have been created, which means that when
you fetch, there isn't a single README. There are *at least* two of
them: README from master, and README from origin/master. They may be the
same, and they may not be the same, but saying just "the README file" is
an incomplete depiction of it: you either say "my README file" or talk
about both of them, which do coexist.

There's no denial that this model is much more complex than the usual
1:1 lookup we're used to, but trying to hide this in the database
implementation somehow would be a bad idea. The programmer brought this
to themself when they choose to build a decentralized client-side
application.=20

However, with a solid foundation, there are still lots of room for work
on ergonomics, like:=20

1. diffing algorithms for text like you mention would totally
make sense to be in a "contrib/" directory, so that people could pick
different diffing algorithms for different string fields.

2. whenever a 'dev comes up with the bright idea of annotating "data
objects" with some kind of conflict resolution', I could point them to a
"IConflictResolutionStrategy" interface for they to implement. Their
implementation could even end-up under said "contrib/" directory. Even
better if "IConflictResolutionStrategy" could be composed! A programmer
would then pick different strategies for different entities in their
domain, like in a buffet.

3. offer some low-level building blocks that can be applied to different
domains, like tombstones, max(a.last-modified, b.last-modified), etc.

...and so on.

The point is: the database is not in a position to make this type of
decision. Only the programmer is, and they must be aware of it.

You said that:

> the goal of a database is to help me efficiently, accurately, and
> ergonomically represent my domain=20

For me, the crucial part is the accurately comes first, and ergonomics
comes second. No accuracy should be sacrificed in name of ergonomics,
since the inherent complexity is already too big to be dealt with.

The syncing part is not free.

[1]: As defined in http://curtclifton.net/papers/MoseleyMarks06a.pdf

# On history rewriting

You talked about how Mentat was trying to reconcile changes by rewriting
the history in a stable and consistent order.

I'd argue that instead of a linear log where things are rebased, a DAG
log where things are merged is a more flexible approach. Let me show you
where I'm coming from first.

You said that:

> Phrased differently: in the above, your developers want some way to
> write
>
>   SELECT url, title FROM pages
>
> and
>
>   SET title FOR page WITH url 'foo.com' TO 'whatever'

This may still be true, but with a caveat: you'd be asserting facts to
the local "timeline", in a somewhat similar way that the Mentat wiki
describes[2] (I feel funny giving you a link to Mentat's wiki).=20

What I'd do differently is to don't try so hard to make everyone
converge on the same thing always, and represent timelines in a more
flexible way.

Instead of having two types of timelines, =CE=B1 and =CE=B2, there would be=
 a
multitude of timelines, one for each instance of the database. Let's say
we have 3 of them for the purpose of explanation.

Instead of adding an extra 'timeline_id' identifier, I'd add a
':db/txParent' to the transaction entity, with cardinality many, and the
database instance would always know it's latest local parent. This is
effectivelly like a branch, but not a shared one, and transations with
more than one ':db/txParent' represent merges. Those would only happen
when explicitly solving conflicts, and any given database instances
could know about a multitude of timelines.

This way, how can one know the current state of the database?=20

>From the first root datom, the database would rollup every datom taking
the timelines and merges in consideration, and it would be the sum of
data from all timelines. When a conflict exists, the local timeline is
always picked so the database can still serve writes, and the conflict
would be presented when serving reads.=20

The obvious problem of always favouring the local timeline is that each
timeline would show different "primary" answers, but that's sounds
somewhat reasonable to me to enable the database to keep working while
offline, and while the conflict isn't resolved.

This also means that "sync" are two distinct things: exchanging timeline
data, and reconciling that data locally. The only part that's online is
the "exchange", and instances resolve conflict locally. This way, the
database is always available, even when offline, and even when conflict
is present.

But how would queries over history behave? No clue yet =F0=9F=98=AC.

I tried not to sound too hand wavy, and I've only prototyped a small
part of this in the past, so there are probably many more problems with
this approach. I don't claim this is a sound solution yet, but it does
handle some of the problems that Mentat faced in a different way. I'd be
interested to see you poking holes on it.=20

When combined with strict schema growth, new data always comes after
it's schema declaration, so the application doesn't have to coordinate
around distributed schema evolution.

[2]: https://github.com/mozilla/mentat/wiki/Multiple-timeline-support

# On CouchDB/PouchDB's entity conflicts

CouchDB/PouchDB's most fundamental value is the document. This means
that a single document write is atomic (not true for clustered CouchDB),
but you end up rewritting the full document ([3], on time 3:35). If you
represent an entity as a document, the whole entity is in conflict, not
the attribute that changed.

So it doesn't know the difference between two changes on unrelated
attributes and full rewrites of the full document: everything is a
conflict in CouchDB/PouchDB's view.

On another talk ([4] on time 5:30, also by Joan Touzet), she mentions
the desire to add partial updates to CouchDB, but I don't know if it was
pursued.

Checking for conflicts on revisions is considered good practice ([1], on
time 10:21) on the community.

[3]: https://www.youtube.com/watch?v=3DBKQ9kXKoHS8
[4]: https://www.youtube.com/watch?v=3D0gAZHIYBh-g

# On zipping identities together

=F0=9F=91=8D

# Misc

TIL: I didn't know you worked on AllegroGraph.

P.S.: I haven't reviewed this text too much, and it might have gotten a
bit confusing. I'm happy to clarify any points, and I'll consider
writing a more formal version of this braindump later. I chose to send
this like this over taking more time and letting the discussion cool
down.=20

From eu@euandre.org Fri Sep  4 00:00:13 2020
Received: from box.euandre.org (box.euandre.org [46.101.160.115])
	by mail-b.sr.ht (Postfix) with ESMTPS id 7A675FF1B9
	for <~euandreh/public-inbox@lists.sr.ht>; Fri,  4 Sep 2020 00:00:12 +0000 (UTC)
Authentication-Results: mail-b.sr.ht;
	dkim=pass (2048-bit key) header.d=euandre.org header.i=@euandre.org header.b=teyCS1z1
Received: from authenticated-user (box.euandre.org [46.101.160.115])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	(No client certificate requested)
	by box.euandre.org (Postfix) with ESMTPSA id 3507CFCB45;
	Thu,  3 Sep 2020 21:00:09 -0300 (-03)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=euandre.org; s=mail;
	t=1599177611; bh=mYcY4i3Y9/GUAx3/nY2Lu91JMvyDWaWOxTADQnXB+MM=;
	h=From:To:Cc:Cc:Subject:In-Reply-To:References:Date:From;
	b=teyCS1z1iKrOm7S9ivB92sQQ5uq6Kb9TRtkrIvyFYqPWIP9zFVq4sYuuU5NQZwDSi
	 GHHeO9otqioa+dWLVcXYITaxTvgD3uAaiI/Xa9Nzj3swN1DWuzGwJ7Ovqf8vQDBSQI
	 0VROFULtzElG2iNTv3tyWy6C7Hu12whmbfteUtq6aTzonf9pFIP6Wio6wGakmS9px9
	 xwKDfuZgrdN8tUWhjAM+Joz9bwRgRg08+hJOf9Jd9JKShTgZGosTIOYDl8HnfQwR7F
	 fZbYza/n4vBwLdkpDlOpX9wVdVprXmvM3cD+zoPQ/ukRomUmjQ6KFETBOksZ4t7pY1
	 E/KZ/wEEBxRJA==
From: EuAndreh <eu@euandre.org>
To: Richard Newman <rnewman@twinql.com>
Cc: "~euandreh/public-inbox@lists.sr.ht" <~euandreh/public-inbox@lists.sr.ht>
Cc: 
Subject: Re: The database I wish I had
In-Reply-To: <87h7se5t1w.fsf@euandre.org>
References: <7454DA45-C5A4-4A54-8D02-1F3BD37726E9@twinql.com>
 <010101744a592b75-1dce9281-f0b8-4226-9d50-fd2c7901fa72-000000@us-west-2.amazonses.com>
 <87sgc1xu7k.fsf@euandre.org>
 <9652AAE0-10DF-4D13-ABE8-B30667BE27DD@twinql.com>
 <010101744b260d62-1bb508ab-647f-4877-bdf9-f988667d56e1-000000@us-west-2.amazonses.com>
 <878sdsy6sc.fsf@euandre.org>
 <7027FD8B-1B30-435F-A840-7DC61A722E0B@twinql.com>
 <010101744f99448f-fa8dd88e-0daf-41f8-b3ca-6ea551c300e4-000000@us-west-2.amazonses.com>
 <87h7se5t1w.fsf@euandre.org>
Date: Thu, 03 Sep 2020 20:57:28 -0300
Message-ID: <87eeni5s7r.fsf@euandre.org>
MIME-Version: 1.0
Content-Type: text/plain

I better get a POC so I can stop talking in "what ifs".

I was more focused on the storage implementation side of it, but I'm
considering not implementing a storage backend for this and just using
SQLite instead (more detailed in [0]).

Building durable persistent data structures on top of SQLite backed
storage is still a bit distant, but the model I presented could benefit
from a POC.

I'll try starting that instead of giving so much focus on storage, and
share it later.

[0]: https://euandre.org/2020/08/31/the-database-i-wish-i-had.html#fn:posix-sqlite

From eu@euandre.org Fri Sep  4 02:12:34 2020
Received: from box.euandre.org (box.euandre.org [46.101.160.115])
	by mail-b.sr.ht (Postfix) with ESMTPS id DE901FF196
	for <~euandreh/public-inbox@lists.sr.ht>; Fri,  4 Sep 2020 02:12:33 +0000 (UTC)
Authentication-Results: mail-b.sr.ht;
	dkim=pass (2048-bit key) header.d=euandre.org header.i=@euandre.org header.b=aOTLZVo3
Received: from authenticated-user (box.euandre.org [46.101.160.115])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	(No client certificate requested)
	by box.euandre.org (Postfix) with ESMTPSA id 13742FCB45;
	Thu,  3 Sep 2020 23:12:30 -0300 (-03)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=euandre.org; s=mail;
	t=1599185552; bh=0eOLRgKRFwQtp0D1PSgenzUsGttO7assNfrY8hnPqz8=;
	h=From:To:Cc:Cc:Subject:In-Reply-To:References:Date:From;
	b=aOTLZVo3JPMEsqZSTB2G6Y+T0rDJDcBXrDuU5+R0SjJN9lX+c01/ir+CO/+Av22VH
	 IhKMuNJU1nVjzOf4Qs0CnmJO1Agzeuq+pf41gU0qIx0kZVSaUZP8PgXzpG1DUvfQ5T
	 tEKNEHhRVFapOZ9HYeYzbMcUdO4z4haaEkuxerFOwwvIudLPkmCCeJH7xFouMvkMSN
	 vxyKaJK7RdWzBkiXZutwKzS6BEbvbalzdMf7IcLXAlmYsUdV3bMPH0Bvg5HsxvLY1I
	 YQNTiPp8yRiojv69hKqGKByUvhPveXVtbZNt3k8y4cwU5UAaAXnTQrNwSFhy3pxB+9
	 fwaRNRHiXi1mQ==
From: EuAndreh <eu@euandre.org>
To: Richard Newman <rnewman@twinql.com>
Cc: "~euandreh/public-inbox@lists.sr.ht" <~euandreh/public-inbox@lists.sr.ht>
Cc: 
Subject: Re: The database I wish I had
In-Reply-To: <87eeni5s7r.fsf@euandre.org>
References: <7454DA45-C5A4-4A54-8D02-1F3BD37726E9@twinql.com>
 <010101744a592b75-1dce9281-f0b8-4226-9d50-fd2c7901fa72-000000@us-west-2.amazonses.com>
 <87sgc1xu7k.fsf@euandre.org>
 <9652AAE0-10DF-4D13-ABE8-B30667BE27DD@twinql.com>
 <010101744b260d62-1bb508ab-647f-4877-bdf9-f988667d56e1-000000@us-west-2.amazonses.com>
 <878sdsy6sc.fsf@euandre.org>
 <7027FD8B-1B30-435F-A840-7DC61A722E0B@twinql.com>
 <010101744f99448f-fa8dd88e-0daf-41f8-b3ca-6ea551c300e4-000000@us-west-2.amazonses.com>
 <87h7se5t1w.fsf@euandre.org> <87eeni5s7r.fsf@euandre.org>
Date: Thu, 03 Sep 2020 23:02:55 -0300
Message-ID: <87a6y65meo.fsf@euandre.org>
MIME-Version: 1.0
Content-Type: text/plain

Here's what I was referring to when talking about "durable persistent
data structures": https://www.youtube.com/watch?v=Cym4TZwTCNU

From eu@euandre.org Sat Sep  5 09:07:44 2020
Received: from box.euandre.org (box.euandre.org [46.101.160.115])
	by mail-b.sr.ht (Postfix) with ESMTPS id 6A917FF1F5
	for <~euandreh/public-inbox@lists.sr.ht>; Sat,  5 Sep 2020 09:07:43 +0000 (UTC)
Authentication-Results: mail-b.sr.ht;
	dkim=pass (2048-bit key) header.d=euandre.org header.i=@euandre.org header.b=grYKK93t
Received: from authenticated-user (box.euandre.org [46.101.160.115])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	(No client certificate requested)
	by box.euandre.org (Postfix) with ESMTPSA id 92EB0FC83F;
	Sat,  5 Sep 2020 06:07:40 -0300 (-03)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=euandre.org; s=mail;
	t=1599296862; bh=RjoAEI4vfrQ1VCfrySR+wMnwePv+AnUVT1ZbWcm8O1A=;
	h=From:To:Cc:Cc:Subject:In-Reply-To:References:Date:From;
	b=grYKK93td5WpquSSrkfBXBzUeJ25jSXwwjOE/x5c8rfR1DVAT1sDu7NfGVP+80pBW
	 oG7IM1+Kpjz7PL15Bg01iDOXTLXMD/4GeikxqSUUG2I9Pj0NEO4eNzl6hNTfNDBuMA
	 E7cz3qVVonEDHIYGlXvavHAWxWU6MXoSdDvEn0K2zNPU3qmo5PJ0vTKYNQTPkInY1N
	 DRSUHv6DgfTUUYyrXdcw0rIPhQdNvobloBT29Q7LOtvXcBtiMhWlLMG30ifV+ETmhU
	 h6AIw5myD387Nf7GRL6TfmtM1zWotf92jSop38NRnUCOJQ8iBpUrTt74qaSsuhcDbx
	 p4C1GSTXxTj2g==
From: EuAndreh <eu@euandre.org>
To: Richard Newman <rnewman@twinql.com>
Cc: "~euandreh/public-inbox@lists.sr.ht" <~euandreh/public-inbox@lists.sr.ht>
Cc: 
Subject: Re: The database I wish I had
In-Reply-To: <87a6y65meo.fsf@euandre.org>
References: <7454DA45-C5A4-4A54-8D02-1F3BD37726E9@twinql.com>
 <010101744a592b75-1dce9281-f0b8-4226-9d50-fd2c7901fa72-000000@us-west-2.amazonses.com>
 <87sgc1xu7k.fsf@euandre.org>
 <9652AAE0-10DF-4D13-ABE8-B30667BE27DD@twinql.com>
 <010101744b260d62-1bb508ab-647f-4877-bdf9-f988667d56e1-000000@us-west-2.amazonses.com>
 <878sdsy6sc.fsf@euandre.org>
 <7027FD8B-1B30-435F-A840-7DC61A722E0B@twinql.com>
 <010101744f99448f-fa8dd88e-0daf-41f8-b3ca-6ea551c300e4-000000@us-west-2.amazonses.com>
 <87h7se5t1w.fsf@euandre.org> <87eeni5s7r.fsf@euandre.org>
 <87a6y65meo.fsf@euandre.org>
Date: Sat, 05 Sep 2020 06:07:31 -0300
Message-ID: <878sdosib0.fsf@euandre.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

> The obvious problem of always favouring the local timeline is that each
> timeline would show different "primary" answers, but that's sounds
> somewhat reasonable to me to enable the database to keep working while
> offline, and while the conflict isn't resolved.

Also a possible ergonomics layer that can be built on top of some lower lev=
el
primitive:

Suppose we want to add a profile page to an app, but the user will be
able to edit their profile while offline.

Such profile page contains 3 parts: name, bio and links. Here's how it
could look like:

{:profile/name  "Foo Bar"
 :profile/links #{{:link/title    "Personal webpage"
                   :link/position 0
                   :link/address  #uri "https://example.com"}
                  {:link/title    "Mastodon profile"
                   :link/position 1
                   :link/address  #uri "https://example.org"}
                  {:link/title    "Photo album"
                   :link/position 2
                   :link/address  #uri "https://example.net"}}
 :profile/bio   "My lengthy bio (...)"}

For each of those attributes, one could want to handle conflicts in a diffe=
rent
way:

1. For the :profile/name, you could want to be fully conservative, and never
   resolve any conflict whatsoever. If the user changes their name to "Foo =
Bax"
   in one device and to "Foo Baz" in another, you may choose to not solve t=
his
   conflict on the application. Instead you could just pick one of them to =
show
   to the user on their profile page, say "Foo Baz", but add an exclamation=
 mark
   icon =E2=9D=97=EF=B8=8F beside it. When the user touches the icon, the a=
pp asks: "do you want
   to keep 'Foo Baz' or would you like to change it to 'Foo Bax'?". This co=
uld
   be done multiple times for multiple concurrent conflicts, and when user
   confirms their choice, that's a merge that resolves the conflict for all
   instances when propagated.

   The upside is that the app doesn't try to be smart about reconciling the
   :profile/name, showing things like "Foo Baxz" or "Foo BazFooBax". The
   downside is that if the app doesn't handle this explicitly, they'll be
   perpetually inconsistent.

2. For the :profile/links, specifically the :link/position attribute, you m=
ight
   try an intermediate approach. Say you want to handle reordering less
   conservatively. Say the user changes the position of the "Photo album" l=
ink
   to 1 in a device, and to 0 on another device, and that is their actual
   desired final position for this link. You could solve it by doing
   something trivial, like picking the transaction UUID with the lowest
   alphanumerical value. This gives you consistent choices across instances=
, and
   a potential nuisance for the user: if the winning final state is with "P=
hoto
   album" a 0, nothing needs to be done, and all instances agree with this.=
 If
   it isn't, then the user should reorder again, and all instances would st=
ill agree.

   The upside of this is that this conflict resolution strategy allows you =
to
   never stay conflicted for this particular attribute, and you could have =
1:1
   lookups back. The downside is that this assumes that the transactions th=
at
   are reordering links contain only :link/position attribute asserts.

3. For the :profile/bio, you could be extremely optimistic, and try to comb=
ine
   all edits. This could be done by modeling it using commutative edits, si=
milar
   to how in [0], and let everything always converge. Maybe even adopt the
   binary enconding of edits proposed in [1], and encode all of the edit hi=
story
   in the database itself. They talk a lot about the metadata overhead of t=
he
   edit history, and recording every edit as they come in would also add a
   datom-related overhead, too. It might be more advantegeous to accumulate=
 the
   edit history, and when the app closes or a relevent event occurs, all ed=
its
   are combined into a single binary, and that would be a single datom entr=
y.
   This would remove the datom-related overhead while still persisting the =
full
   edit history.

   The upside of this is that the text could always converge, you would get
   1:1 lookups back and you wouldn't have to deal with diffs on a huge stri=
ng,
   neither would the user. The downside is that you'll need a layer on top =
of the
   transactions to actually derive the text, and querying would get trickie=
r.
   The database would give you the ordered list of all binary enconding thi=
ngs,
   and a function would turn that into text.

   Another downside is that this is very much an open research problem, and
   probably has a lot of maturity to be gained. It would be very powerful if
   this could be built on top of some database primitive, without the datab=
ase
   having to explicitly support it.

All of that to handle the conflicts of *each attribute*, and they're all
the programmer's responsability to choose.

If the database was versatile enough, an ideal world the programmer could w=
rite:

;; schema definition
[{:db/ident :profile/name
  :db/valueType :db.type/string
  :db/cardinality :db.cardinality/one
  :db/conflict-strategy :db.conflict-strategy/none}
 {:db/ident :link/position
  :db/valueType :db.type/long
  :db/cardinality :db.cardinality/one
  :db/conflict-strategy :db.conflict-strategy/simple-winner}
 {:db/ident :profile/bio
  :db/valueType :db.type/string
  :db/cardinality :db.cardinality/one
  :db/conflict-strategy :db.conflict-strategy/automerge}
 ...]

And be done with it.

If those :db/conflict-strategy fields could be open and extensible, the dat=
abase
could learn new algorithms to deal with conflict without changing it's
internals, and people could teach how translate the :profile/bio from
:db.type/string the actual :db.type/bytes that would be actually stored.

Also, I think this is what I was trying to say with "coupling two differente
layers of the database". However, I haven't tested any of this yet.

[0]: https://doi.org/10.1016/j.jpdc.2010.12.006
[1]: https://github.com/automerge/automerge-perf/tree/master/columnar

From rnewman@twinql.com Sat Sep  5 20:04:34 2020
Received: from a58-61.smtp-out.us-west-2.amazonses.com (a58-61.smtp-out.us-west-2.amazonses.com [54.240.58.61])
	by mail-b.sr.ht (Postfix) with ESMTPS id D36DAFF13B
	for <~euandreh/public-inbox@lists.sr.ht>; Sat,  5 Sep 2020 20:04:32 +0000 (UTC)
Authentication-Results: mail-b.sr.ht;
	dkim=pass (1024-bit key) header.d=twinql.com header.i=@twinql.com header.b=qiZnhSkC;
	dkim=pass (1024-bit key) header.d=amazonses.com header.i=@amazonses.com header.b=MQAUAk7Q
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple;
	s=qvs2k2sd2ivtrflyu2nswmravuqizjml; d=twinql.com; t=1599336271;
	h=Subject:From:To:Cc:Date:Mime-Version:Content-Type:Content-Transfer-Encoding:In-Reply-To:References:Message-Id;
	bh=lPgy0nUilFxkCQGCuQxBgZW5Udn+JVGxl6bDms54Sk4=;
	b=qiZnhSkCMRBNxcfwIf1Ecic54ZZmlMpioyl5vuE02tb5tFBnHk+bAFJcpIwJInMS
	EdJiWj0xSnX7x4dBpD6JqwFHvQfx3wvs+tW6yxyAvjIQcFYb4qoC8ufiI33atGIOW/R
	rrhRvIaDsmSyeT53ab+mytd/6WCxBnufMhHh5rrE=
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple;
	s=hsbnp7p3ensaochzwyq5wwmceodymuwv; d=amazonses.com; t=1599336271;
	h=Subject:From:To:Cc:Date:Mime-Version:Content-Type:Content-Transfer-Encoding:In-Reply-To:References:Message-Id:Feedback-ID;
	bh=lPgy0nUilFxkCQGCuQxBgZW5Udn+JVGxl6bDms54Sk4=;
	b=MQAUAk7QsnWFnb7gUs7do0tFKJ/ZEjJ+XsIFxth+m525oY5TIo0jyYTBcFxAeQU3
	ruo+SqNRgJBJVtzqIHoghOuG8UqNq50MkyFLgv+agjPe95SKqQPIgnHeraWBXXccyjN
	C42JTZ/q9uSxe/Kxy2m7iHmbpGF3+O3tP1PMHXLk=
Subject: Re: The database I wish I had
From: =?UTF-8?Q?Richard_Newman?= <rnewman@twinql.com>
To: =?UTF-8?Q?EuAndreh?= <eu@euandre.org>
Cc: =?UTF-8?Q?=7Eeuandreh/public-inbox=40lists=2Esr=2Eht?=
  <~euandreh/public-inbox@lists.sr.ht>
Date: Sat, 5 Sep 2020 20:04:31 +0000
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
In-Reply-To: <87h7se5t1w.fsf@euandre.org>
References: <7454DA45-C5A4-4A54-8D02-1F3BD37726E9@twinql.com> 
 <010101744a592b75-1dce9281-f0b8-4226-9d50-fd2c7901fa72-000000@us-west-2.amazonses.com> 
 <87sgc1xu7k.fsf@euandre.org> 
 <9652AAE0-10DF-4D13-ABE8-B30667BE27DD@twinql.com> 
 <010101744b260d62-1bb508ab-647f-4877-bdf9-f988667d56e1-000000@us-west-2.amazonses.com> 
 <878sdsy6sc.fsf@euandre.org> 
 <7027FD8B-1B30-435F-A840-7DC61A722E0B@twinql.com> 
 <010101744f99448f-fa8dd88e-0daf-41f8-b3ca-6ea551c300e4-000000@us-west-2.amazonses.com> 
 <87h7se5t1w.fsf@euandre.org> 
 <B032A2E0-0679-4FF6-9A80-06450A8F36B1@twinql.com>
X-Priority: 3 (Normal)
X-Mailer: Amazon WorkMail
Thread-Index: AQHWgHcPJ4udfXYrRaOwhoO7EcsDJQAErSAlAAfQ7+gAIgKd3wAzRhy+AHUc9SwA0i0RQQ==
Thread-Topic: The database I wish I had
X-Original-Mailer: Apple Mail (2.3445.104.14)
X-Wm-Sent-Timestamp: 1599336269
Message-ID: <010101745fdecd5f-52ab4d88-26f1-4a6d-879d-3234c1956260-000000@us-west-2.amazonses.com>
X-SES-Outgoing: 2020.09.05-54.240.58.61
Feedback-ID: 1.us-west-2.An468LAV0jCjQDrDLvlZjeAthld7qrhZr+vow8irkvU=:AmazonSES

Ah, thank you for the conversation refactor!=0D=0A=0D=0A> The wiki page y=
ou pointed to reinforces that: it talks a lot about how=0D=0A> to coordin=
ate attributes that changed their names, change in vocabulary,=0D=0A> sha=
red ownership of changes in schema and even schema versioning!=0D=0A>=20=0D=
=0A> All of them are the opposite of what I want, and they're "schema=0D=0A=
> breakage", not "schema growth".=0D=0A=0D=0AMy two observations are thes=
e:=0D=0A=0D=0AOn the one hand, front-end developers tend to think in term=
s of change =E2=80=94 "this thing's title was X and is now Y" =E2=80=94 a=
nd states =E2=80=94 "give me a list of things and their current titles so=
 I can search them or show them in a table".=0D=0A =20=0D=0AThat view of =
change extends to schema/data, because the schema stretches most of the w=
ay towards the front-end itself, sometimes all the way to a "view model" =
in the MVVM sense.=0D=0A =20=0D=0AAnd it involves migration, because no f=
ront-end developer (or back-end developer, for that matter) wants to have=
 to write queries that accommodate all of the different ways a particular=
 concept might have been represented in the last ten years. Doing so is m=
entally taxing, is very difficult to test, and results in wildly variable=
 performance. They want to fix their data once, or define how to roll it =
up into a single state once, and make sure the old representation never g=
ets used again. And these representations do change, often significantly;=
 making sure they don't requires tremendous up-front deep thought (for ex=
ample, my modeling attempt for embedded videos), and results in something=
 that is often normalized to the point of non-viability (already a concer=
n with tuple stores).=0D=0A=0D=0AOn the other hand, storage/syncing devel=
opers learn to love immutable logs, because it makes a ton of sense. Sync=
able storage systems should be based on an immutable timeline of appended=
 observations of the world.=0D=0A=0D=0A=0D=0AThere is obvious conflict be=
tween these two. Resolving that conflict involves some complexity, and I =
felt it better to address that complexity once, inside the storage system=
=2E=0D=0A=0D=0AI didn't feel that it was worthwhile to build a storage sy=
stem that required front-end developers to adjust their mental model so f=
ar from what they're trying to achieve on a daily basis, because they hav=
e a choice in which tools they use.=0D=0A=0D=0AFrom my perspective, the p=
rimary goal has to be to give them a system they're willing to use (a cle=
ar, current-focused schema and the ability to represent change), with the=
 additional goal of a useful log and better formalisms underneath to supp=
ort syncing. Putting it the other way 'round would simply lead to them us=
ing SQLite or JSON files, omitting syncing, building bad ad hoc sync solu=
tions, and otherwise continuing in the way that Mozilla had been doing th=
ings.=0D=0A=0D=0AMentat, like Datomic, is a stateful layer on top of log-=
structured data, with the current schema =E2=80=94 like the current state=
 of the world! =E2=80=94 being materialized from the log.=0D=0A=0D=0A=0D=0A=
> When more than one database instance can add data, the programmer needs=
=0D=0A> to think of it as an open world, where new things are accrued, bu=
t old=0D=0A> things don't disapper. Changing the vocabulary mid-air only =
makes this=0D=0A> open world brittle so "migrations" can be performed.=0D=
=0A=0D=0AI am a fan of open-world systems =E2=80=94 I spent 7+ years work=
ing on semantic web stuff! =E2=80=94 but I'm not aware of any client appl=
ications with any degree of adoption that use the open-world model. Havin=
g multiple components share state doesn't require an open world per se.=0D=
=0A=0D=0ABeing able to evolve the representation of existing data, just l=
ike being able to 'change' the state of the world, is regarded as a *feat=
ure* by developers, and they typically would rather sacrifice interoperab=
ility than take on the burden of dealing with a pile of data that only ge=
ts more crufty over time. In the 'schemaless' world (e.g., DynamoDB), dev=
elopers take a mix of painful approaches (ranging from adding version tag=
s that drive switches in application code, to doing manual migrations bet=
ween tables) to clean up old data when the previous representation cannot=
 stretch far enough.=0D=0A=0D=0AI tried with Mentat to strike a balance h=
ere: to design a system that would allow multiple teams to interact, defi=
ning their vocabularies in an ever-growing way, with specific transformat=
ion steps as needed, and being able to detect and handle conflicting voca=
bulary usage within that closed world. You can absolutely use Mentat as a=
 system with no vocabulary modification ever, complicating your queries t=
o accommodate, but it would not have met the needs of its audience:=0D=0A=
=0D=0AD: "So, I realized that having a page -> visit timestamp property d=
oesn't actually do what we want; we need to be able to mark each visit wi=
th which device visited the page. How do I change the schema and the exis=
ting data=3F"=0D=0AR: "You can't; make a new property. Every time you wan=
t to retrieve the visits for the page, everywhere in your app, your queri=
es need to look at both properties."=0D=0AD: "Can I write some function t=
hat seamlessly transforms the old data to the new format and works with s=
yncing=3F"=0D=0AR: "I wrote a doc about that, but we decided that purity =
was better. You can write code to manually retract the old stuff, but oth=
er syncing devices can still mess you up. Sorry."=0D=0A=0D=0AResolving co=
nflicts and supporting sync was a core part of Mentat's charter. The last=
 thing I wanted was for developers to think the system was hard to use, o=
r to build a second storage layer on top to maintain a snapshot of the cu=
rrent data in an app-centric format.=0D=0A=0D=0A=0D=0A> You mentioned erg=
onomics thrice on your message, and I believe that your=0D=0A> insistence=
 of it is a sign of the coupling of two different layers of=0D=0A> the da=
tabase, and makes ease of use drive the fundamentals.=0D=0A=0D=0AYes, exa=
ctly. The goal of Mentat wasn't to build a pure log-structured store: it =
was to build a store with a developer interface that wasn't too alien to =
an engineer who might store data in JSON or SQLite, but with internals th=
at made sharing data, syncing data, and evolving data easier and safer.=0D=
=0A=0D=0A=0D=0A> I also think=0D=0A> that your statement on conflict reso=
lution being a requirement is not a=0D=0A> good idea. Let me expand on th=
at.=0D=0A>=20=0D=0A> I'd argue that conflict resolution is inherit comple=
xity [1] of creating=0D=0A> a decentralized client-side application that =
can sync data.=0D=0A>=20=0D=0A> That means that the responsability to sol=
ve conflicts is not the=0D=0A> database's, but the application's responsa=
bility, and making decisions=0D=0A> of conflict resolution on the databas=
e limits its application, and I=0D=0A> wouldn't consider it to be general=
 purpose.=0D=0A=0D=0AAh, here's an interesting point.=0D=0A=0D=0AThere ar=
e two positions here: am I vending a database, or am I vending a storage =
solution for a client app=3F=0D=0A=0D=0ABuilding Firefox, Firefox for And=
roid, and Firefox for iOS convinced me that implementing a synchronizatio=
n solution that is not tightly coupled with the storage layer =E2=80=94 t=
hat is, one that is just another consumer of the same storage interface t=
hat the rest of the app uses =E2=80=94 is difficult and error-prone. Sync=
 needs to see writes as they happen, and synchronization has particular i=
mplications for how data is stored.=0D=0A=0D=0AI also found that offloadi=
ng the difficulties in syncing =E2=80=94 conflict resolution, tracking ch=
ange, merging identities, etc. =E2=80=94 to application developers leads =
to worse outcomes.=0D=0A=0D=0AMy feeling is that it should be possible fo=
r the application to educate the storage layer about how to handle most o=
f the conflicts it encounters, and to model its data in a way that makes =
conflicts meaningful and minimal, without sacrificing the kinds of data r=
epresentations with which developers feel comfortable.=0D=0A=0D=0A=0D=0A>=
 When two offline database instances change the same field, there's no=0D=
=0A> escaping from the bottom reality, which is: there is a conflict, and=
 it=0D=0A> needs to be resolved somewhere. There's no layer of ergonomics=
 that can=0D=0A> change this fact, and picking a strategy for conflict re=
solution is=0D=0A> undoubtly domain-specific.=0D=0A=0D=0AThe approach I w=
as taking for syncing in Mentat involved rewriting (or permanently associ=
ating) identifiers based on uniqueness constraints. That is, as a develop=
er I could transact a schema fragment:=0D=0A=0D=0A=0D=0A  {:history/url :=
db/unique :db.unique/value}=0D=0A=0D=0A=0D=0Aand thereafter say things li=
ke=0D=0A=0D=0A=0D=0A  {:history/url "https://foo.com/"=0D=0A   :history/t=
itle "Today's title"}=0D=0A=0D=0A=0D=0AIf the log already contains this, =
with a local ID:=0D=0A=0D=0A=0D=0A  [1234 :history/url "https://foo.com/"=
]=0D=0A=0D=0A=0D=0Athe new transact would add to the log something like:=0D=
=0A=0D=0A=0D=0A  [1234 :history/title "Today's title"]=0D=0A=0D=0A=0D=0AA=
nother device can record similar data:=0D=0A=0D=0A=0D=0A  {:history/url "=
https://foo.com/"=0D=0A   :history/title "A different title"}=0D=0A=0D=0A=
=0D=0Awhich would transact=0D=0A=0D=0A=0D=0A  [4567 :history/title "A dif=
ferent title"]=0D=0A=0D=0A=0D=0AA query on either platform can omit the I=
D, using a lookup ref instead:=0D=0A=0D=0A=0D=0A  [[:history/url "https:/=
/foo.com/"] :history/title =3Ftitle]=0D=0A=0D=0A=0D=0AIn both cases we ge=
t a 1:1 association between a URL and a 'page'. The use of lookup refs an=
d the schema definition allows the developer to use the [attribute, URL] =
pair as an external identifier in this context, while using an efficient =
identifier internally.=0D=0A=0D=0AWhen the two transaction logs are merge=
d, it follows from the schema definition that B(4567) =3D=3D=3D A(1234), =
and that identity can be used in the store to zip the two graphs together=
 in a way that is seamless to the developer: the 1:1 association is prese=
rved. In practice we found that holding on to those numeric IDs, which ca=
n change during syncing, in application code was rarely or never necessar=
y.=0D=0A=0D=0AThis example is what I mean by 'teaching' the database abou=
t domain definitions of identity. In a relational database, the schema is=
 largely used to define structure and constraints. The way it feels to wr=
ite a schema in Datomic/Mentat is much closer to 'explaining' how various=
 kinds of new assertions and sync behaviors will interact with the existi=
ng data =E2=80=94 the schema constraints are so often used to drive looku=
p refs and updates, and of course there's no tabular structure.=0D=0A=0D=0A=
This can get a merge all the way down to "this page's title changed from =
X to Y at time T on one device and from X to Z on another at time Y; whic=
h do you want=3F" which is _also_ something that you can teach the databa=
se to automatically resolve =E2=80=94 annotate the schema to say that a c=
onflict takes the time-latest value.=0D=0A=0D=0A=0D=0A> They may be the=0D=
=0A> same, and they may not be the same, but saying just "the README file=
" is=0D=0A> an incomplete depiction of it: you either say "my README file=
" or talk=0D=0A> about both of them, which do coexist.=0D=0A=0D=0AI actua=
lly used this as an example when explaining Mentat =E2=80=94 timelines ar=
e branches, but the `datoms` table (the data you actually work with) is a=
 Git "working tree". Developers (and end users) need a working tree and a=
 single state of the world.=0D=0A=0D=0A=0D=0A> The point is: the database=
 is not in a position to make this type of=0D=0A> decision. Only the prog=
rammer is, and they must be aware of it.=0D=0A=0D=0AWhich kinds of progra=
mmers are your audience=3F=0D=0A=0D=0AIn my experience, most developers a=
re (a) not aware of the difficulties here, (b) do not have the time to di=
g in to the details, and (c) would rather take reasonable defaults (e.g.,=
 latest wins) or even data duplication or corruption than sacrifice simpl=
icity to get it right. That is: if you presented them with the option of =
a distributed sync system where they had to handle multiple timelines, us=
e pluggable conflict-resolution modules, and write complicated queries to=
 deal with that complexity, they are instead going to use JSON files.=0D=0A=
=0D=0AIt would not be inaccurate to say that Mentat is/was _already_ way =
too much of a shift for most developers; a tuple store with a log is expe=
nsive and confusing in ways that a key-value store is not.=0D=0A=0D=0A=0D=
=0A> You said that:=0D=0A>=20=0D=0A>> the goal of a database is to help m=
e efficiently, accurately, and=0D=0A>> ergonomically represent my domain=20=
=0D=0A>=20=0D=0A> For me, the crucial part is the accurately comes first,=
 and ergonomics=0D=0A> comes second. No accuracy should be sacrificed in =
name of ergonomics,=0D=0A> since the inherent complexity is already too b=
ig to be dealt with.=0D=0A=0D=0ATo broaden your point a little bit, my po=
sition on that has shifted over the years, after working on systems that =
prioritized correctness over adoption or user experience (e.g., Firefox S=
ync's perfect crypto, which caused _so many users_ to lose data because t=
hey weren't prepared to manage long-lived secure keys, and couldn't under=
stand the setup flow=E2=80=A6).=0D=0A=0D=0AA system that is perfect but t=
hat people =E2=80=94 developers or end users =E2=80=94 don't use it is _w=
orse_ than a system that makes compromises and gets some adoption. As Guy=
 Steele put it about Java:=20=0D=0A=0D=0A  "We were not out to win over t=
he Lisp programmers; we were after the C++ programmers. We managed to dra=
g a lot of them about half-way to Lisp."=0D=0A=0D=0AI think a brief tange=
nt into ontology would be interesting. I know your point here is about ac=
curately modeling change, but a similar set of positions exists when mode=
ling information itself.=0D=0A=0D=0AThe 'correct' way to model the world =
is as a time-series sequence of independent observations, with everything=
 else being materialized from it through a carefully designed ontological=
 lens, such that one doesn't have to accidentally make false statements i=
n order to say something true.=0D=0A=0D=0AAt one end is the kind of ontol=
ogy I drew on a whiteboard once, nearly twenty years ago, in which the co=
ncept of a music "album" is broken down into issuances, releases, track o=
rders, renditions, recordings, performers, names, identities, collectives=
, etc. =E2=80=94 it's incredibly complex how one recording of a group of =
people in a room might be recorded multiple times, remastered over the ye=
ars, edited for various releases, and appear on multiple formats in diffe=
rent versions, and how different versions of songs might have different b=
ands or people playing different parts.=0D=0A=0D=0AMusic nerds really car=
e about this, though perhaps not down to the level of detail to which I m=
odeled it.=0D=0A=0D=0AAt the other end is how this kind of metadata is re=
ally represented in the real world.=0D=0A=0D=0ATitle: "Landslide"=0D=0AAr=
tist: "Fleetwood Mac"=0D=0APlay Count: 15=0D=0ATrack Number: 2=0D=0A=0D=0A=
The theoretically pure way to model all this stuff makes it possible to s=
ay "this recording of Landslide is of a performance by a band named Fleet=
wood Mac =E2=80=94 the particular instance of Fleetwood Mac that included=
 Stevie Nicks, not the one that included Bob Welch". Let's not even get i=
nto the fact that there are two albums recorded by Fleetwood Mac titled "=
Fleetwood Mac" =E2=80=94 (title, artist) isn't compound-unique!=0D=0A=0D=0A=
But no user wants to say "Alexa, play Fleetwood Mac" and have her reply "=
do you mean the band 'Fleetwood Mac' that recorded songs released by Warn=
er Bros, or the band 'Fleetwood Mac' that recorded songs released by Blue=
 Horizon"=3F They will exchange a certain amount of precision to cast a w=
ider and more usable net.=0D=0A=0D=0APut bluntly: apps have trouble getti=
ng the darn play counts on two devices to add up to the same number, let =
alone this degree of precision. I would rather make it _possible_ for the=
m to model data at a reasonable degree of accuracy, with no more difficul=
ty than they would have writing a CREATE TABLE statement, and help them g=
et their play counts right when they sync two devices, rather than force =
them to take an extreme position on accuracy and leave them to figure out=
 how to list how many times the user played a song.=0D=0A=0D=0A=0D=0A> Th=
is way, how can one know the current state of the database=3F=20=0D=0A>=20=
=0D=0A> From the first root datom, the database would rollup every datom =
taking=0D=0A> the timelines and merges in consideration, and it would be =
the sum of=0D=0A> data from all timelines. When a conflict exists, the lo=
cal timeline is=0D=0A> always picked so the database can still serve writ=
es, and the conflict=0D=0A> would be presented when serving reads.=20=0D=0A=
>=20=0D=0A> The obvious problem of always favouring the local timeline is=
 that each=0D=0A> timeline would show different "primary" answers, but th=
at's sounds=0D=0A> somewhat reasonable to me to enable the database to ke=
ep working while=0D=0A> offline, and while the conflict isn't resolved.=0D=
=0A=0D=0AMy view is that end users expect all of their devices to present=
 the same canonical view of their data, and manual conflict resolution sh=
ould be rare/unnecessary/modeled as an explicit change or an explicit sta=
te (a write like any other).=0D=0A=0D=0AFurther, developers expect querie=
s to be fast and writes and queries to behave the same when devices are s=
ynced, regardless of which device performs the write or runs the query.=0D=
=0A=0D=0AThat makes it difficult to either diverge between devices for lo=
ng periods, or to expect routine manual involvement in conflicts.=0D=0A=0D=
=0AIt seems to me that the longer these distinct timelines exist, the mor=
e conflicts will breed, as writes reflect different local states, and so =
the difficulty of presenting a coherent view increases over time.=0D=0A=0D=
=0AI expected probably one place to do manual conflict resolution in Fire=
fox: password management. Even there it might be better to turn every con=
flict into an explicit "previous passwords" entry, or to expose the log a=
s a dynamic list of previous passwords =E2=80=94 that data is valuable fo=
r detecting website breaches. That is, it's better to surface the alterna=
tive data as a normal change and reach agreement with other devices, rath=
er than preserving some kind of pending conflict.=0D=0A=0D=0ABy modeling =
rebases or merges as a coherent set of retractions of conflicting data, i=
t's still possible to implement log-oriented features like querying *ever=
y* title a page has ever had, but eagerly reach a shared timeline. ("Show=
 me that news article I remember reading".)=0D=0A=0D=0A=0D=0A> This also =
means that "sync" are two distinct things: exchanging timeline=0D=0A> dat=
a, and reconciling that data locally.=0D=0A=0D=0AAgreed.=0D=0A=0D=0A=0D=0A=
> The only part that's online is=0D=0A> the "exchange", and instances res=
olve conflict locally. This way, the=0D=0A> database is always available,=
 even when offline, and even when conflict=0D=0A> is present.=0D=0A=0D=0A=
The difference IMO is that the goal of conflict resolution is that every =
device converges on the same canonical timeline, and does so as quickly a=
s possible.=0D=0A=0D=0A=0D=0A> But how would queries over history behave=3F=
 No clue yet =F0=9F=98=AC.=0D=0A>=20=0D=0A> I tried not to sound too hand=
 wavy, and I've only prototyped a small=0D=0A> part of this in the past, =
so there are probably many more problems with=0D=0A> this approach. I don=
't claim this is a sound solution yet, but it does=0D=0A> handle some of =
the problems that Mentat faced in a different way. I'd be=0D=0A> interest=
ed to see you poking holes on it.=0D=0A=0D=0AIt's a really difficult prob=
lem! My suspicion is that querying history is either broadly a union ("th=
ese pages all had this title at some point") or very much an expert featu=
re, analogous to browsing your own 'undo' tree. But note that the union a=
pproach can benefit from identifier rewriting to represent derived equali=
ty at a particular point in time, even in the presence of conflicts.=0D=0A=
=0D=0A=0D=0A> When combined with strict schema growth, new data always co=
mes after=0D=0A> it's schema declaration, so the application doesn't have=
 to coordinate=0D=0A> around distributed schema evolution.=0D=0A=0D=0AYes=
, that's broadly the position I took with Mentat: by definition, data on =
each timeline comes after its schema declaration, so we have an ordered m=
echanism for merging and also for migrating data as the schema evolves.=0D=
=0A=0D=0A=0D=0A> TIL: I didn't know you worked on AllegroGraph.=0D=0A=0D=0A=
Seems like every few years I end up back on data modeling and graph stora=
ge :D=0D=0A=0D=0A=0D=0A> If the database was versatile enough, an ideal w=
orld the programmer could write:=0D=0A>=20=0D=0A> ;; schema definition=0D=
=0A> [{:db/ident :profile/name=0D=0A>  :db/valueType :db.type/string=0D=0A=
>  :db/cardinality :db.cardinality/one=0D=0A>  :db/conflict-strategy :db.=
conflict-strategy/none}=0D=0A=0D=0AYes, that's pretty much where we were =
headed with Mentat. My key observations were that (a) you need really goo=
d tree-zipping, and frequent syncs, to minimize conflicts; (b) tree-zippi=
ng and merges practically require that internal identifiers be rewritten =
when identity is determined; (c) it is undesirable to stop the world, and=
 so when a conflict is detected you end up with continuing divergence whi=
ch will be even harder to merge later; (d) because this is a single graph=
, a conflict _must_ be resolved in some way before you can merge two head=
s; (e) lossy merges (squash then rebase or squash then merge) can help.=0D=
=0A=0D=0A

From eu@euandre.org Sat Sep  5 22:51:10 2020
Received: from box.euandre.org (box.euandre.org [46.101.160.115])
	by mail-b.sr.ht (Postfix) with ESMTPS id D9930FF12D
	for <~euandreh/public-inbox@lists.sr.ht>; Sat,  5 Sep 2020 22:51:09 +0000 (UTC)
Authentication-Results: mail-b.sr.ht;
	dkim=pass (2048-bit key) header.d=euandre.org header.i=@euandre.org header.b=w2kZ+s6p
Received: from authenticated-user (box.euandre.org [46.101.160.115])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	(No client certificate requested)
	by box.euandre.org (Postfix) with ESMTPSA id 2130DFC822;
	Sat,  5 Sep 2020 19:51:06 -0300 (-03)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=euandre.org; s=mail;
	t=1599346268; bh=PWBfaRn3jOle7Z3Lc1INXcX50+v5RZ5HNwANJeQm8Js=;
	h=From:To:Cc:Cc:Subject:In-Reply-To:References:Date:From;
	b=w2kZ+s6pUTUZJSwLwi3NmcFMqy3a8mBj7PuNX5oJD9O3R4MZM4Td6AZVsjHQQLRVG
	 IRQHyT9t8jngQXu0mkRtU2F4IZoKEPMQAVrRBOBq+f4+J1OR5zynUcT7UKE+hmWDtr
	 lJqDHIbUEzNoS8yLpWXZqib6kwHFqF8B/TLtbl4scllrpf+Q0eq7ZbDpjYHfotNt7C
	 xZ8bFSDeeSP6Ec93nh82bOpXqAEhaTA/vf7qh7T1qWKdjmFvDV3WldUtxjmMcCDvgS
	 PYZ5Laxlrvoq8li4XF55rArXMOb8gftJMy3QY0rPRmyvpFLKFFB423i+NnIctnu93o
	 ewcl54UnP6Iaw==
From: EuAndreh <eu@euandre.org>
To: Richard Newman <rnewman@twinql.com>
Cc: "~euandreh/public-inbox@lists.sr.ht" <~euandreh/public-inbox@lists.sr.ht>
Cc: 
Subject: Re: The database I wish I had
In-Reply-To: <010101745fdecd5f-52ab4d88-26f1-4a6d-879d-3234c1956260-000000@us-west-2.amazonses.com>
References: <7454DA45-C5A4-4A54-8D02-1F3BD37726E9@twinql.com>
 <010101744a592b75-1dce9281-f0b8-4226-9d50-fd2c7901fa72-000000@us-west-2.amazonses.com>
 <87sgc1xu7k.fsf@euandre.org>
 <9652AAE0-10DF-4D13-ABE8-B30667BE27DD@twinql.com>
 <010101744b260d62-1bb508ab-647f-4877-bdf9-f988667d56e1-000000@us-west-2.amazonses.com>
 <878sdsy6sc.fsf@euandre.org>
 <7027FD8B-1B30-435F-A840-7DC61A722E0B@twinql.com>
 <010101744f99448f-fa8dd88e-0daf-41f8-b3ca-6ea551c300e4-000000@us-west-2.amazonses.com>
 <87h7se5t1w.fsf@euandre.org>
 <B032A2E0-0679-4FF6-9A80-06450A8F36B1@twinql.com>
 <010101745fdecd5f-52ab4d88-26f1-4a6d-879d-3234c1956260-000000@us-west-2.amazonses.com>
Date: Sat, 05 Sep 2020 19:50:56 -0300
Message-ID: <87r1rfsur3.fsf@euandre.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

I see what to mean by being accessible, and it does make sense.

If done properly this might be the reality, where nobody chooses to
handle conflicts manually because they'd rather have a consistent view
by picking one of the possible algorithms that are available for them to
pick for each attribute. I'll experiment with this a bit, and if=20
feasible, the "perpetual inconsistency" might be an internal view, and
something available for those who choose to customize their conflict
resolution strategy. This would be ideal, and perfectly aligned with
what you said on "the application educating the storage layer about how
to handle most of the conflicts it encouters". The unique URL you showed
is a tempting challenge to reach, and the domain identity being
teachable sounds indeed really powerful.

This also means that with those defaults the desired 1:1 lookups would
be restored. I don't think that doing manual resolution for each
attribute is desired, and I wouldn't impose this on the programmer. In
fact, I'd want to use it myself most of the time.=20

That would also mean an intersection between the two conflicting forces
you mentioned, where both sides would get what they wanted: a developer
could use it to quickly prototype something by picking the suggested
defaults and not finding many surprises, while allowing another
developer to fine tune their strategy when desired.

It's a database, with a robust storage solution inside it, but primarily
a database.

If not possible, I tend to err on the side of less ease. Not because I
want to be a purist, but because I want to do this for myself =F0=9F=98=AC.=
 The
title of the article was actually honest on this, and I've actually had
this exact though a few times: "I'm struggling with this database, but
there must be more people like me with the same struggles. I bet someone
has already solved, or will solve soon enough". I'm the target audience,
and if I do something that solves my problem, than I would consider it a
success :). It may not be widespread but I'd feel better knowing that
I've built something that works over something that works most of the
time. Again, I don't want to be a purist, I just want to create a
database that I'd want for myself, and whoever sees themselves on the
same position could benefit. If ease can be added on top of it with some
choices made for you, great! From your comment, I think you'd agree with
that last part.=20=20=20=20=20

I'm not trying to convert anyone on this, though. I'd be happy to share
the code and learning, and welcome those who are interested and
attracted to this, but broad adoption and world domination are not
goals.

The applications that I tried to write using the existing solutions were
just good enough to get me started, but I always felt abandoned when I
wanted to evolve it somehow, or add some specific functionality without
losing data, or some similar trade-off. Most of those times I'd rather
deal with that complexity than choose an easier path.

Today whenever I see a client-side database that syncs, I go to the
documentation and search for "conflict". I'll either get a page with
some in-depth explanation, or there will be a list of choices they made
for me with no way out of it, like last write win, and permanent
deletion tombstones. Sure those are valuable, but I'd rather be able to
choose them myself. I already know by now that I will need some tweaked
conflict resolution someday, and they won't be able to provide it for
me without an ad-hoc approach.=20

If someone would value something more familiar and more
fragile/opinionated over something very different and more sound, then
maybe they're correct to pick SQLite or some JSON files, and having data
loss in some edge cases would be just fine in the problem they're
solving. I'd encourage them to stick to SQLite, even, its a great piece
of software.=20

Also, there's something more profound on this, I believe:=20

Having a solid foundation could potentially help people reason about the
problem with good tools, instead of looking like a solid foundation that
will crack if you step on the wrong place. Instead of helping, it could
be hurting the application, and the superficial ease that many solutions
offer sometimes feel more like a trap for me to figure out myself later.
To me it's like someone saying: "have this relational fully ACID
performant database!", and only later I realize that it isn't actually
ACID by default, and if all change all the knobs to make it ACID, it
becomes unusuably slow (or maybe even those knobs aren't there at all).=20

I'd argue that such approach may look weirder and more foreign to any
developer, but it would actually be "easier". This reminds me of
"Simple Made Easy" [0]

[0]: https://www.infoq.com/presentations/Simple-Made-Easy/

Back on schema growth: one doesn't have to make a perfect plan of the
data models so no mental tax gets to them later.

The solution you mentioned sounds just fine: write a function that
transforms the old data and feed it to the new attribute, and retract
everything else. From that point on, you've effectively migrated the
data, and you'll only need to consider the historical schema when doing
historical queries. Nothing broke on the process, and people might just
forget about how it used to be in the past. No mental tax, unless you
want to dig history.

The existence of the new attribute serves the same purpose that the
version tag does.

You still have an immutable log, and you've evolved your schema to
better represent the domain. I don't see how those conflict here. The
biggest objection would probably be the "wasted storage" of the old
data.=20

But changing schema types and versioning them is something else. It
might even be possible to implement, doing rollups of the schema just
like what's done with the rest of the data like you mentioned, but I
just don't like this idea overall, I think it has brittleness written
all over it. Maybe I'm just being grumpy.=20=20=20

About your album ontology: I like it! Coincidence or not, I did 5 years
of music major before dropping out of college, so I'd consider myself a
"music nerd"! But I agree that one needs the database to help them model
the domain in a useful way, and I wouldn't force everybody to have a
particular view of it. Ultimately the database should be a useful tool
to solve a problem.=20

From andi@notmuch.email Tue Oct  6 07:53:02 2020
Received: from mx.h4ck.space (mx.h4ck.space [159.69.146.50])
	by mail-b.sr.ht (Postfix) with ESMTPS id 57F5AFF0FF
	for <~euandreh/public-inbox@lists.sr.ht>; Tue,  6 Oct 2020 07:53:02 +0000 (UTC)
Authentication-Results: mail-b.sr.ht;
	dkim=pass (1024-bit key) header.d=notmuch.email header.i=@notmuch.email header.b=iFvGMOSq
Date: Tue, 6 Oct 2020 09:52:58 +0200
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=notmuch.email;
	s=mail; t=1601970781;
	bh=RfiC8xpAHKtbuatmn10k3tqQKuSGYPCep+XBVi1qDe0=;
	h=Date:From:To:Subject;
	b=iFvGMOSqx3JpTHGUue5rMzrKPzZiiAFpjJBgA/1P8lgnk/7gBNl51mU5M7m42XXS2
	 cGLwhRQAF3y/x9IoyNGAAUy/hRg+u1VGeRPoV+wFWlitZNgH6dWMuos8XNxLv7Z+dr
	 X1MbaF5gSIQaRehQyg7oyDhd9TO2WcKj6jRCia60=
From: andi@notmuch.email
To: ~euandreh/public-inbox@lists.sr.ht
Subject: crate2nix
Message-ID: <20201006075258.52tz5u5a36fpbfwy@wrt>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline

Hi,

Just saw your crate2nix code and it looks very nice and clean!

Just a heads-up, since you might not aware of all the alternatives out
there, there is a version from Eelco that basically does the same as
yours but also handles git dependencies. I did blog about that briefly
last month [1].

I am curious if you were aware of that and/or naersk [2] as both of them
also come with the promise of no generated files and no *2nix commands
to execute.


Regards,

Andi



[1]: https://andreas.rammhold.de/posts/nix-rust/#import-cargohttpsgithubcomedolstraimport-cargo
[2]: https://github.com/nmattia/naersk

From eu@euandre.org Tue Oct  6 09:09:49 2020
Received: from box.euandre.org (box.euandre.org [46.101.160.115])
	by mail-b.sr.ht (Postfix) with ESMTPS id 9ED40FF107
	for <~euandreh/public-inbox@lists.sr.ht>; Tue,  6 Oct 2020 09:09:48 +0000 (UTC)
Authentication-Results: mail-b.sr.ht;
	dkim=pass (2048-bit key) header.d=euandre.org header.i=@euandre.org header.b=sXxvO0HC
Received: from authenticated-user (box.euandre.org [46.101.160.115])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	(No client certificate requested)
	by box.euandre.org (Postfix) with ESMTPSA id DF0A2FC991;
	Tue,  6 Oct 2020 06:09:45 -0300 (-03)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=euandre.org; s=mail;
	t=1601975387; bh=R37p/WxSUhw6s3+NsQXWU1P6/NnfwRrO7khrmNnifPs=;
	h=From:To:Cc:Subject:In-Reply-To:References:Date:From;
	b=sXxvO0HCnYXM8ovt5uOk5bZ/3ZTzOR9HVWXgURy/MicF6fwxTJJfOMCC4+b/ZanpX
	 hcQc+ZSsiOgevD6oR8MZE630bHDpXNs/FskZFM0gRAwG68dfJ/lpj2ogo5GQ9fEbKy
	 oH0MD7Yd2suVGevV9oULcadlt5g8Y75Yuazzwx0HOoF12A4BQWtRrBsBV/Qp7yNoZm
	 xjNIf9MwbD2MsEWlF/K9JeDEdsHazcpQh2RiVd1/G497B9F4yEbE5HmkOP5aGP5jUv
	 yudSn0cBCWQ7S/FXeHXUyhZeebiQnFWkpJeIfUSEhGSOoim1GE/gMi1U7rA1qsWaoy
	 jVVWomlh//r5w==
From: EuAndreh <eu@euandre.org>
To: andi@notmuch.email, ~euandreh/public-inbox@lists.sr.ht
Cc: 
Subject: Re: crate2nix
In-Reply-To: <20201006075258.52tz5u5a36fpbfwy@wrt>
References: <20201006075258.52tz5u5a36fpbfwy@wrt>
Date: Tue, 06 Oct 2020 06:09:27 -0300
Message-ID: <87r1qbafg8.fsf@euandre.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

andi@notmuch.email writes:

> Hi,

Hi there!

> Just saw your crate2nix code and it looks very nice and clean!

=E2=98=BA=EF=B8=8F

> Just a heads-up, since you might not aware of all the alternatives out
> there, there is a version from Eelco that basically does the same as
> yours but also handles git dependencies. I did blog about that briefly
> last month [1].

I wasn't aware of it. At a first glance, it looks remarkably similar!
Maybe I'm on to something =F0=9F=98=AC. I'll try it later, thanks for the l=
inks.

> I am curious if you were aware of that and/or naersk [2] as both of them
> also come with the promise of no generated files and no *2nix commands
> to execute.

I did try naersk. It was the best one I found, in fact. It does use
Cargo under the hood to drive things, but it doesn't expose the
environment it builds. With it you can parameterize the Cargo commands,
but not run it by yourself.

In fact, there's a snippet that I copied directly from naersk[0]. I
didn't mention it on the README, only on the commit log.

[0]: https://git.euandreh.xyz/cargo2nix/commit/?id=3Dc75e7e2d3ee7f7e074ea9f=
8e5b959d6395338acc

From m@kluv.in Tue Dec 22 16:29:11 2020
Authentication-Results: mail-b.sr.ht; dkim=none 
Received: from se15-1.privateemail.com (se15-1.privateemail.com [198.54.127.72])
	by mail-b.sr.ht (Postfix) with ESMTPS id B579CFF0D7
	for <~euandreh/public-inbox@lists.sr.ht>; Tue, 22 Dec 2020 16:29:11 +0000 (UTC)
Received: from new-01.privateemail.com ([68.65.122.22])
	by se15.registrar-servers.com with esmtpsa (TLSv1.2:AES128-GCM-SHA256:128)
	(Exim 4.92)
	(envelope-from <m@kluv.in>)
	id 1krkX5-0004vZ-Jo
	for ~euandreh/public-inbox@lists.sr.ht; Tue, 22 Dec 2020 08:29:10 -0800
Received: from MTA-08-1.privateemail.com (unknown [10.20.147.18])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by NEW-01.privateemail.com (Postfix) with ESMTPS id B5E4C60041
	for <~euandreh/public-inbox@lists.sr.ht>; Tue, 22 Dec 2020 16:29:05 +0000 (UTC)
Received: from MTA-08.privateemail.com (localhost [127.0.0.1])
	by MTA-08.privateemail.com (Postfix) with ESMTP id 91A866006F
	for <~euandreh/public-inbox@lists.sr.ht>; Tue, 22 Dec 2020 11:29:05 -0500 (EST)
Received: from DESKTOP1997JCK (unknown [10.20.151.208])
	by MTA-08.privateemail.com (Postfix) with ESMTPA id F339F6004A
	for <~euandreh/public-inbox@lists.sr.ht>; Tue, 22 Dec 2020 16:29:04 +0000 (UTC)
From: <m@kluv.in>
To: <~euandreh/public-inbox@lists.sr.ht>
Subject: Progress on 'The database I wish I had'
Date: Tue, 22 Dec 2020 17:29:03 +0100
Message-ID: <000001d6d87f$90db0f90$b2912eb0$@kluv.in>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit
X-Mailer: Microsoft Outlook 16.0
Thread-Index: AdbYfzkizJpIDb+oQYeLozSx36pHnw==
Content-Language: no
X-Virus-Scanned: ClamAV using ClamSMTP
X-Originating-IP: 68.65.122.22
X-SpamExperts-Domain: o1.privateemail.com
X-SpamExperts-Username: out-01
Authentication-Results: registrar-servers.com; auth=pass (plain) smtp.auth=out-01@o1.privateemail.com
X-SpamExperts-Outgoing-Class: unsure
X-SpamExperts-Outgoing-Evidence: Combined (0.57)
X-Recommended-Action: accept
X-Filter-ID: Pt3MvcO5N4iKaDQ5O6lkdGlMVN6RH8bjRMzItlySaT8/DiuTODf6wrobfALkT2HqPUtbdvnXkggZ
 3YnVId/Y5jcf0yeVQAvfjHznO7+bT5wWykU6VVuoVZYG6cNLn3c4E9P7Wdo8qPSPEXiwPI9MYyaH
 o/9wB5YLuaEkARV33qkqPXaGse7C78HuChWU138c4pWx7WxWFZpsjT/483ewgsz4HwYGTNBY7kFu
 mOeXVIneVHayp+oaTkTmqro7//B+gpZK9MmGPsPG1IFhoAOccOwmVIoqJOmt2f1p/FeBBYf0It0W
 sNcsjnTm+4g485SS8OTf6I2qJgqFlok7aJfKCMNBaM6YjzsuHtjxewJtlZX1S4orbC27wIbqb1J8
 wyDQ43DG7JDHGkUFlLvaUAJ9EsZg/9mPb0OnPglkYW72F/Odw+XGlIW1bb6iLQaqIs5BLfTttFI5
 MCNL/izpcNORuAUvossjam0/HVDFzCeLVAjI+ht+2XwDC3Hj+WjRz7duYp7tG8ix8mThYIoihNfI
 Pg3O+76Hxe6qkGjOqN7KXlq8vamDuW8qjcPFOC9xtigG+MEYMo0NDeyXVO3tNSgVgckOnz0mswVX
 ZkpFVe5ZgGA0MLjqgL337Aba0izaNK+kAlchHs+Xxee6P8LI+rvlGOeqKR5Od1/Dbcw8eAyTq1JN
 7i5WVjtE146jFUwt8Q10AQGCef4uNE610hEkYogGmsvnLjiRqfPZZiHIniXn+ZdiHE9X9GlwwRvN
 2KIObvd055RG8/vjrz5/4Qa1aRJkvZZIZN1+KRx7WOWIKK5eEM2Qqv1jJWtJXZNrnBE/zutvXj2E
 w+ZVKNlbNM488ODolw/r6z7OkCVgacEoYdDR+UFRXxKF5tPxTxfD0dMN+t5ZjqKOEsXEdN1Fhcn9
 wF7BNjheMPxghFsjIwINNqkbXC/wv1jB3a9oGPa4BHVKDIPK97/T4LRRVYxF+VXiiOfHJN40eTXl
 WiUAYdLmsJdAoPKCpWwKtkkGG+bEnfOEkWTNI3SjTCvjMfNBc9ze9o81pXKSQ+GI7QB7PH97h6/L
 6Wa2uSQw0nxsyGH4ASfmmhbNFFwa8AyQYqjO7qYtiXb+9Q==
X-Report-Abuse-To: spam@se16.registrar-servers.com

Hi EuAndreh.

I am curious if you found any existing solution for the proposed database? I
have been looking at irmin (which was mentioned in all forums) a lot lately,
largely as substitute to libgit2, but recently also as a database.

It looks like the conversation have tapered off, or moved elsewhere? In any
case it was interesting stumbling across your post, seeing as I have been
having very similar ideas.

Regards,
Martin



From eu@euandre.org Tue Dec 22 21:37:15 2020
Authentication-Results: mail-b.sr.ht; dkim=fail header.d=euandre.org header.i=@euandre.org
Received: from box.euandre.org (box.euandre.org [46.101.160.115])
	by mail-b.sr.ht (Postfix) with ESMTPS id 40985FF0DB
	for <~euandreh/public-inbox@lists.sr.ht>; Tue, 22 Dec 2020 21:37:15 +0000 (UTC)
Received: from authenticated-user (box.euandre.org [46.101.160.115])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	(No client certificate requested)
	by box.euandre.org (Postfix) with ESMTPSA id 4B3B9FC4BF;
	Tue, 22 Dec 2020 18:37:11 -0300 (-03)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=euandre.org; s=mail;
	t=1608673033; bh=9xQtiA8BJBLUgJEdjCq1SNS1YmrPaYDMrBvQ2oYoGpY=;
	h=From:To:Cc:Subject:In-Reply-To:References:Date:From;
	b=P++6DaNXwhc5G/perUnlDlWN9zHzoyzmASEDPcZ0nqbNUpKJy9sV5OmC7etQTyPMO
	 bAAwkUw3Wl/tn9CaLxzYJo+IClWYQEVJDmAMwlcCIJa3Ol+fSQsceBxK1a5za5EA8W
	 SzBrXwvpbYG9ISVwBUY+fNkqwAoRP2lQ8Sy3/pvrA0Kq894eXg08tbvlTJh3H6rh3Z
	 mla6VfP5qftYM6oWWhGSs6EJkGtmMDA+1UkAn+ASnSROejuxcPm1Cp2qmT2VoEPapP
	 kmHV3bjCM9v4nN/1bPy1rBvhwhx1O9UfNWvUYGWnhb24Tu0zK68FA20ZXlQc1bRnGA
	 yBrW5x0bzEZxg==
From: EuAndreh <eu@euandre.org>
To: m@kluv.in, ~euandreh/public-inbox@lists.sr.ht
Cc: 
Subject: Re: Progress on 'The database I wish I had'
In-Reply-To: <000001d6d87f$90db0f90$b2912eb0$@kluv.in>
References: <000001d6d87f$90db0f90$b2912eb0$@kluv.in>
Date: Tue, 22 Dec 2020 18:36:56 -0300
Message-ID: <87h7oda5fr.fsf@euandre.org>
MIME-Version: 1.0
Content-Type: text/plain

<m@kluv.in> writes:

> Hi EuAndreh.

Hi there!

> I am curious if you found any existing solution for the proposed database? I
> have been looking at irmin (which was mentioned in all forums) a lot lately,
> largely as substitute to libgit2, but recently also as a database.

I have written it down as a place to steal ideas from, which I plan to
do in the future as I mature more on the field, but irmin isn't a
solution, being written in OCaml.

> It looks like the conversation have tapered off, or moved elsewhere? In any
> case it was interesting stumbling across your post, seeing as I have been
> having very similar ideas.

I'm working on it, slowly but surely.

The previous "comment" link used to point to my email address first, and
the this public-inbox second. Because of that I had many conversations
done in private messages, where I imagine that people just got the first
link. I responded those in private, but many had similar enough
comments, and I wrote an article on that [0]. FYI: I recently added a
"articles by category" Atom feed, if you prefer a filtered list [1] of
articles.

On the ParsecC project mentioned on the article:
I'm already much more comfortable with creating a C API, but I got to a
point that I needed a proper error handling solution. I stumbled upon an
existing implementation of a parser combinator in C [2].

The point I'm at right now is to read and understand that source code.
In fact, it is 1 of 4 open tabs on my browser right now, before writing
this message.

If I find its error handling good enough, I'll replicate it in ParsecC,
and then in libedn.

I've also started reading a good book on the subject [3]. I can't speak
yet to all of it, but I like what I've seen so far.

I happy to discuss more about it, or answer questions you may have. And
also, I'm happy to accept contributions!

[0]: https://euandre.org/2020/11/12/durable-persistent-trees-and-parser-combinators-building-a-database.html
[1]: https://euandre.org/articles-by-category.html#mediator
[2]: https://github.com/wbhart/Cesium3/tree/combinators
[3]: https://www.immutablearchitecture.com/