janderland/fql: Foundation DB Query Language

FQL provides a query language and an alternative client API for Foundation DB.
Some things this project aims to achieve are:

With the Foundation DB client library (>= v6.2.0) and Go (>= v1.20) installed,
you can simply run go build in the root of this repo. This will create an
fql binary in the root of the repo.

Building, linting, and testing can all be performed in a Docker environment.
This allows any host to perform these operations with only Docker as a
dependency. The build.sh script can be used to perform these
operations. This is the same script used by the CI/CD workflow of this repo.

To build, lint, & test the current state of the codebase, run ./build.sh --verify. To learn more about the build script, run ./build.sh --help.

FQL is available as a Docker image for executing queries. The first argument
passed to the container is the contents of the cluster file. The remaining
arguments are passed to the FQL binary.

# 'my_cluster:baoeA32@172.20.3.33:4500' is used as the contents
# for the cluster file. '-log' and '/my/dir()=42' are passed
# as args to the FQL binary.
docker run docker.io/janderland/fql 'my_cluster:baoeA32@172.20.3.33:4500' -log '/my/dir()=42'

Within the cluster file contents (first argument), any instances of a
hostname wrapped in curly braces (e.g. ‘{my_hostname}’) are replaced by the
equivalent IP address. FDB doesn’t support connecting to a cluster via
hostnames, so this functional provides a workaround. This can simplify
connecting to a Docker instance of FDB.

docker network create my_net
docker run --network my_net --name fdb -d foundationdb/foundationdb

# The substring '{fdb}' in the first argument will be replaced with
# the IP address of the FDB container started above before the cluster
# file is written to disk.
docker run --network my_net docker.io/janderland/fql 'docker:docker@{fdb}:4500' -log '/my/dir()=42'

Here is the syntax definition for the query language. Currently,
FQL is focused on reading & writing key-values created using the directory and
tuple layers. Reading or writing keys of arbitrary byte strings is not
supported.

FQL queries are a textual representation of a specific key-value or a schema
describing the structure of many key-values. These queries have the ability to
write a key-value, read one or more key-values, and list directories.

This section will explain the components and structure of an FQL query. The
semantic meaning of these queries will be explained below in the Kinds of
Queries section.

FQL utilizes textual representations of the element types supported by the
tuple layer. These types are known as primitives. Besides as tuple elements,
primitives can also be used as the value portion of a key-value.

When primitives are used as tuple elements, they are encoded using the tuple
layer. When they are used as the value portion of a key-value, they are
encoded by FQL as outlined below.

Ideally, the encoding of these primitives would align with common community
practices to maximize usefulness. Let me know if you believe it doesn’t.

Even though a big int encoding is supported by the tuple layer, FQL does
not currently support using big ints.

A directory is specified as a sequence of strings, each prefixed by a forward
slash:

The strings of the directory do not need quotes if they only contain
alphanumericals, underscores, dashes, or periods. To use other symbols, the
strings must be quoted:

The quote character may be backslash escaped:

A tuple is specified as a sequence of elements, separated by commas, wrapped in
a pair of curly braces. The elements may be a tuple or any of the primitive
types.

("one", 2, 0x03, ( "subtuple" ), 5825d3f8-de5b-40c6-ac32-47ea8b98f7b4)

The last element of a tuple may be the ... token.

Any combination of spaces, tabs, and newlines is allowed after the opening
brace and commas.

A key-value is specified as a directory, tuple, equal symbol, and value appended
together:

/my/dir("this", 0)=0xabcf03

The value following the equal symbol may be any of the primitives or a tuple:

/my/dir(22.3, -8)=("another", "tuple")

The value can also be the clear token.

/some/where("home", "town", 88.3)=clear

A variable may be used in place of a directory element, tuple element, or value.

/my/dir/("first", , "third")=

If the variable is a tuple element or value, it may contain a list of primitive
types separated by pipes, except for the nil type. The variable may also
contain the any type which is equivalent to specifying every type. Specifying
no types is also equivalent to specifying the any type.

This section showcases the various kinds of FQL queries, their semantic
meaning, and the equivalent FDB API calls implemented in Go.

Set queries write a single key-value. The query must not contain the clear
or ... tokens, nor a variable.

/my/dir("hello", "world")=42

db.Transact(func(tr fdb.Transaction) (interface{}, error) {
  dir, err := directory.CreateOrOpen(tr, []string{"my", "dir"}, nil)
  if err != nil {
    return nil, err
  }

  val := make([]byte, 8)
  binary.LittleEndian.PutUint64(val, 42)
  tr.Set(dir.Pack(tuple.Tuple{"hello", "world"}), val)
  return nil, nil
})

Clear queries delete a single key-value. The query must contain the clear
token as it’s value and must not contain the ... token or variables.

/my/dir("hello", "world")=clear

db.Transact(func(tr fdb.Transaction) (interface{}, error) {
  dir, err := directory.Open(tr, []string{"my", "dir"}, nil)
  if err != nil {
    if errors.Is(err, directory.ErrDirNotExists) {
      return nil, nil
    }
    return nil, err
  }

  tr.Clear(dir.Pack(tuple.Tuple{"hello", "world"}))
  return nil, nil
})

Read-single queries read a single key-value. These queries must not have the
... token or a variable in their key. The value must be a variable.
Deserialization of the value is attempted for each type in the order specified
by the variable. The first successful deserialization is used as the output. If
the value cannot be deserialized as any of the types specified then the
key-value is not returned or an error is returned, depending on configuration.

/my/dir(99.8, 7dfb10d1-2493-4fb5-928e-889fdc6a7136)=

db.Transact(func(tr fdb.Transaction) (interface{}, error) {
  dir, err := directory.Open(tr, []string{"my", "dir"}, nil)
  if err != nil {
    if errors.Is(err, directory.ErrDirNotExists) {
      return nil, nil
    }
    return nil, err
  }

  val := tr.MustGet(dir.Pack(tuple.Tuple{99.8,
    tuple.UUID{0x7d, 0xfb, 0x10, 0xd1, 0x24, 0x93, 0x4f, 0xb5, 0x92, 0x8e, 0x88, 0x9f, 0xdc, 0x6a, 0x71, 0x36}))
  
     
  if len(val) == 8 {
      return binary.LittleEndian.Uint64(val), nil
  }
  return string(val), nil
})

As a shorthand, these query may be specified without the = token or value.
This implies an empty variable as the value. In the code block below, the
three queries are equivalent.

/my/dir(99.8, 7dfb10d1-2493-4fb5-928e-889fdc6a7136)
/my/dir(99.8, 7dfb10d1-2493-4fb5-928e-889fdc6a7136)=
/my/dir(99.8, 7dfb10d1-2493-4fb5-928e-889fdc6a7136)=

Read-many queries read a range of values based on a key prefix. These
queries have a ... token or a variable in their key. If a key-value is
encountered which does not match the schema defined by the query then the
key-value is not returned or an error is returned, depending on configuration.
These queries are implemented using FDB’s range-read mechanism with
additional filtering performed on the client. Care must be taken with these
queries as they may result in large amounts of data being sent to the
client and most of the data being filtered out.

/people(3392, , )=(, ...)

db.ReadTransact(func(tr fdb.ReadTransaction) (interface{}, error) {
  dir, err := directory.Open(tr, []string{"people"}, nil)
  if err != nil {
    if errors.Is(err, directory.ErrDirNotExists) {
      return nil, nil
    }
    return nil, err
  }

  rng, err := fdb.PrefixRange(dir.Pack(tuple.Tuple{3392}))
  if err != nil {
    return nil, err
  }

  var results []fdb.KeyValue
  iter := tr.GetRange(rng, fdb.RangeOptions{}).Iterator()
  for iter.Advance() {
    kv := iter.MustGet()

    tup, err := dir.Unpack(kv.Key)
    if err != nil {
      return nil, err
    }

    if len(tup) != 3 {
      return nil, fmt.Errorf("invalid kv: %v", kv)
    }

    switch tup[0].(type) {
    default:
      return nil, fmt.Errorf("invalid kv: %v", kv)
    case string | int64:
    }

    val, err := tuple.Unpack(kv.Value)
    if err != nil {
      return nil, fmt.Errorf("invalid kv: %v", kv)
    }
    if len(val) == 0 {
      return nil, fmt.Errorf("invalid kv: %v", kv)
    }
    if _, isInt := val[0].(uint64); !isInt {
      return nil, fmt.Errorf("invalid kv: %v", kv)
    }

    results = append(results, kv)
  }
  return results, nil
})

If only a directory is provided as a query, then the directory layer is queried.
Empty variables may be included as placeholders for any directory name.

db.ReadTransact(func(tr fdb.ReadTransaction) (interface{}, error) {
  root, err := directory.Open(tr, []string{"root"}, nil)
  if err != nil {
    if errors.Is(err, directory.ErrDirNotExists) {
      return nil, nil
    }
    return nil, err
  }

  oneDeep, err := root.List(tr, nil)
  if err != nil {
    return nil, err
  }

  var results [][]string
  for _, dir1 := range oneDeep {
    twoDeep, err := root.List(tr, []string{dir1, "items"})
    if err != nil {
      return nil, err
    }

    for _, dir2 := range twoDeep {
      results = append(results, []string{"root", dir1, dir2})
    }
  }
  return results, nil
})

Keep your files stored safely and securely with the SanDisk 2TB Extreme Portable SSD. With over 69,505 ratings and an impressive 4.6 out of 5 stars, this product has been purchased over 8K+ times in the past month. At only $129.99, this Amazon’s Choice product is a must-have for secure file storage.

Help keep private content private with the included password protection featuring 256-bit AES hardware encryption. Order now for just $129.99 on Amazon!