Highway - The Beginning

I haven't head as much time as I have hoped to develop Highway last year. So during the beginning of this year I'm focusing on developing a prototype product based on Highway and developing Highway itself in the process.

Motivation behind Highway

As Airport applications will be developed it will be beneficial to store some of the data of that application centrally. For example a scheduling app might contain your own, family or group schedule as well as schedules for public events. Highway is intended to provide an implementation of the central storage solution behind the same exact API as Airport will be using locally. With it you would be using a single query definition to retrieve the data from both the local AIR database as well as from a central server. Likewise a single update statement would be used to update either the local AIR database or a central storage solution or both.

The Technical Challenge

The trick with Highway is that past a certain scale serving centrally located data from a relational database becomes inefficient and just too expensive. There are other technologies out there that the job of scaling a web app much cheaper. Specifically full text search engines have been developed to efficiently retrieve data based on a free text query and document database have been developed to very efficiently retrieve data based on an Id or some other normalized querying means.

The core insight that Highway utilizes (and attempts to prove in the process) is that it should be possible to declaratively map relational fields to fields or documents in non-relational storage engines (or not fully relational storage engines) and have the system automatically forward qualifying queries to the right engines and automatically update non-relational engines as the relational data gets updated. And to do so in a performant and distributed way.

The approach

For my initial implementation I've chosen Vespa and ScyllaDb as the initial non-relational platforms to implement Highway with. The idea for Highway is to be flexible enough to incorporate just about any kind of storage system into the mix. It will do so by taking a technology specific set of decorators and utilities and converting them into an internal, implementation agnostic notation, which it will then interpret at runtime and perform the required query and data maintenance operations.

Highway will also attempt to off-load as much computation as possible to build-time, thereby arriving a very efficient set of instructions that in needs to perform at run time do do the necessary work.

Also at build-time Highway will provide a report of which queries are being served from which storage engines and which save/delete calls update them. It will also warn if there are queries still being served from the relational db (if they haven't specifically been marked as such) and also warn if there are non-sql defined entities/documents that are not being maintained but are being read from.

Finally Highway will provide hooks into the system which will allow developers to add additional processing at any stage of the process. I've done some preliminary research and have chosen @node-ts/bus as the distributed application platform for that reason.

Please stay tuned for specific examples as I start to implement them... :)