Module Private.Make

Parameters

module K : Index_intf.Key
module _ : Platform.S
module C : Cache.S

Signature

include Index_intf.S with type key = K.t with type value = V.t
type t

The type for indexes.

type key = K.t

The type for keys.

type value = V.t

The type for values.

type cache

The type for caches of index instances.

val empty_cache : unit -> cache

Construct a new empty cache of index instances.

val v : ?flush_callback:(unit -> unit) -> ?cache:cache -> ?fresh:bool -> ?readonly:bool -> ?throttle:[ `Overcommit_memory | `Block_writes ] -> ?lru_size:int -> log_size:int -> string -> t

The constructor for indexes.

  • parameter flush_callback

    A function to be called before any new bindings are persisted to disk (including both automatic flushing and explicit calls to flush or close).

This can be used to ensure certain pre-conditions are met before bindings are persisted to disk. (For instance, if the index bindings are pointers into another data-structure d, it may be necessary to flush d first to avoid creating dangling pointers.)

  • parameter cache

    used for instance sharing.

  • parameter fresh

    whether an existing index should be overwritten.

  • parameter read_only

    whether read-only mode is enabled for this index.

  • parameter throttle

    the strategy to use when the cache are full and and async in already in progress. Block_writes (the default) blocks any new writes until the merge is completed. Overcommit_memory does not block but continues to fill the (already full) cache.

  • parameter log_size

    the maximum number of bindings in the `log` IO.

  • parameter lru_size

    the maximum number of recently-read index bindings kept in memory. Defaults to 30_000.

val clear : t -> unit

clear t clears t so that there are no more bindings in it.

val find : t -> key -> value

find t k is the binding of k in t.

val mem : t -> key -> bool

mem t k is true iff k is bound in t.

val replace : ?overcommit:bool -> t -> key -> value -> unit

replace t k v binds k to v in t, replacing any existing binding of k.

If overcommit is true, the operation does not triger a merge, even if the caches are full. By default overcommit is false.

val filter : t -> ((key * value) -> bool) -> unit

filter t p removes all the bindings (k, v) that do not satisfy p. This operation is costly and blocking.

val iter : (key -> value -> unit) -> t -> unit

Iterates over the index bindings. Limitations:

  • Order is not specified.
  • In case of recent replacements of existing values (since the last merge), this will hit both the new and old bindings.
  • May not observe recent concurrent updates to the index by other processes.
val flush : ?no_callback:unit -> ?with_fsync:bool -> t -> unit

Flushes all internal buffers of the IO instances.

  • Passing ~no_callback:() disables calling the flush_callback passed to v.
  • If with_fsync is true, this also flushes the OS caches for each IO instance.
val close : ?immediately:unit -> t -> unit

Closes all resources used by t, flushing any internal buffers in the instance.

If immediately is passed, this operation will abort any ongoing background processes. This guarantees not to corrupt the store, but may require additional work to be done on the next startup.

val sync : t -> unit

sync t syncs a read-only index with the files on disk. Raises RW_not_allowed if called by a read-write index.

val is_merging : t -> bool

is_merging t returns true if t is running a merge. Raises RO_not_allowed if called by a read-only index.

val merge : t -> unit

merge t forces a merge for t.

If there is no merge running, this operation is non-blocking, i.e. it returns immediately, with the merge running concurrently.

If a merge is running already, this operation blocks until the previous merge is complete. It then launches a merge (which runs concurrently) and returns.

val try_merge : t -> unit

try_merge is like merge but is a no-op if the number of entries in the write-ahead log is smaller than log_size.

module Checks : sig ... end

Offline fsck-like utility for checking the integrity of Index stores built using this module.

type merge_stages = [
  1. | `After
  2. | `After_clear
  3. | `After_first_entry
  4. | `Before
]

Some operations that trigger a merge can have hooks inserted at the following stages:

  • `Before: immediately before merging (while holding the merge lock);
  • `After_clear: immediately after clearing the log, at the end of a merge;
  • `After_first_entry: immediately after adding the first entry in the merge file, if the data file contains at least one entry;
  • `After: immediately after merging (while holding the merge lock).
type merge_result = [
  1. | `Completed
  2. | `Aborted
]
type 'a async

The type of asynchronous computation.

val replace' : ?hook:[ `Merge of merge_stages ] Hook.t -> ?overcommit:bool -> t -> key -> value -> merge_result async option

replace' t k v is like replace t k v but returns a promise of a merge result if the replace call triggered one.

val close' : hook:[ `Abort_signalled ] Hook.t -> ?immediately:unit -> t -> unit

`Abort_signalled: after the cancellation signal has been sent to any concurrent merge operations, but before blocking on those cancellations having completed.

val clear' : hook:[ `Abort_signalled | `IO_clear ] Hook.t -> t -> unit
val try_merge_aux : ?hook:merge_stages Hook.t -> ?force:bool -> t -> merge_result async

try_merge_aux t tries to merge t. If force is false (the default), a merge is performed only if there is more entries in the write-ahead log than the configured limits. If force is set, the merge is performed no matter what.

val await : 'a async -> ('a, [ `Async_exn of exn ]) Stdlib.result

Wait for an asynchronous computation to finish.

val replace_with_timer : ?sampling_interval:int -> t -> key -> value -> unit

Time replace operations. The reported time is an average on an number of consecutive operations, which can be specified by sampling_interval. If sampling_interval is not set, no operation is timed.

val sync' : ?hook: [ `Before_offset_read | `After_offset_read | `Reload_log | `Reload_log_async ] Hook.t -> t -> unit

Hooks:

  • `Before_offset_read: before reading the generation number and the offset.
  • `After_offset_read: after reading the generation number and offset.
val log : t -> (key * value) list option
val log_async : t -> (key * value) list option