LMDB源码官方介绍

2022-04-15 14:44:50

前期研究了一下LMDB与RocksDB的效率，发现LMDB在查询方面确实要快于RocksDB。于是准备从理论到源码做一个对LMDB做一个系统的分析。

首先拿到代码后发现只有四个关键文件，于是本文对官方的英文解释做一个翻译，整体学习一下官方是如何对LMDB进行介绍的

我们这次要分析的文件为lmdb.h：

/** @file lmdb.h
 *  @brief Lightning memory-mapped database library
 *
 *  @mainpage   Lightning Memory-Mapped Database Manager (LMDB)
 *
 *  @section intro_sec Introduction
 *  LMDB is a Btree-based database management library modeled loosely on the
 *  BerkeleyDB API, but much simplified. The entire database is exposed
 *  in a memory map, and all data fetches return data directly
 *  from the mapped memory, so no malloc's or memcpy's occur during
 *  data fetches. As such, the library is extremely simple because it
 *  requires no page caching layer of its own, and it is extremely high
 *  performance and memory-efficient. It is also fully transactional with
 *  full ACID semantics, and when the memory map is read-only, the
 *  database integrity cannot be corrupted by stray pointer writes from
 *  application code.
 *
 *  The library is fully thread-aware and supports concurrent read/write
 *  access from multiple processes and threads. Data pages use a copy-on-
 *  write strategy so no active data pages are ever overwritten, which
 *  also provides resistance to corruption and eliminates the need of any
 *  special recovery procedures after a system crash. Writes are fully
 *  serialized; only one write transaction may be active at a time, which
 *  guarantees that writers can never deadlock. The database structure is
 *  multi-versioned so readers run with no locks; writers cannot block
 *  readers, and readers don't block writers.
 *
 *  Unlike other well-known database mechanisms which use either write-ahead
 *  transaction logs or append-only data writes, LMDB requires no maintenance
 *  during operation. Both write-ahead loggers and append-only databases
 *  require periodic checkpointing and/or compaction of their log or database
 *  files otherwise they grow without bound. LMDB tracks free pages within
 *  the database and re-uses them for new write operations, so the database
 *  size does not grow without bound in normal use.
 *
 *  The memory map can be used as a read-only or read-write map. It is
 *  read-only by default as this provides total immunity to corruption.
 *  Using read-write mode offers much higher write performance, but adds
 *  the possibility for stray application writes thru pointers to silently
 *  corrupt the database. Of course if your application code is known to
 *  be bug-free (...) then this is not an issue.
 *
 *  If this is your first time using a transactional embedded key/value
 *  store, you may find the \ref starting page to be helpful.
 *
 *  @section caveats_sec Caveats
 *  Troubleshooting the lock file, plus semaphores on BSD systems:
 *
 *  - A broken lockfile can cause sync issues.
 *    Stale reader transactions left behind by an aborted program
 *    cause further writes to grow the database quickly, and
 *    stale locks can block further operation.
 *
 *    Fix: Check for stale readers periodically, using the
 *    #mdb_reader_check function or the \ref mdb_stat_1 "mdb_stat" tool.
 *    Stale writers will be cleared automatically on some systems:
 *    - Windows - automatic
 *    - Linux, systems using POSIX mutexes with Robust option - automatic
 *    - not on BSD, systems using POSIX semaphores.
 *    Otherwise just make all programs using the database close it;
 *    the lockfile is always reset on first open of the environment.
 *
 *  - On BSD systems or others configured with MDB_USE_POSIX_SEM,
 *    startup can fail due to semaphores owned by another userid.
 *
 *    Fix: Open and close the database as the user which owns the
 *    semaphores (likely last user) or as root, while no other
 *    process is using the database.
 *
 *  Restrictions/caveats (in addition to those listed for some functions):
 *
 *  - Only the database owner should normally use the database on
 *    BSD systems or when otherwise configured with MDB_USE_POSIX_SEM.
 *    Multiple users can cause startup to fail later, as noted above.
 *
 *  - There is normally no pure read-only mode, since readers need write
 *    access to locks and lock file. Exceptions: On read-only filesystems
 *    or with the #MDB_NOLOCK flag described under #mdb_env_open().
 *
 *  - An LMDB configuration will often reserve considerable \b unused
 *    memory address space and maybe file size for future growth.
 *    This does not use actual memory or disk space, but users may need
 *    to understand the difference so they won't be scared off.
 *
 *  - By default, in versions before 0.9.10, unused portions of the data
 *    file might receive garbage data from memory freed by other code.
 *    (This does not happen when using the #MDB_WRITEMAP flag.) As of
 *    0.9.10 the default behavior is to initialize such memory before
 *    writing to the data file. Since there may be a slight performance
 *    cost due to this initialization, applications may disable it using
 *    the #MDB_NOMEMINIT flag. Applications handling sensitive data
 *    which must not be written should not use this flag. This flag is
 *    irrelevant when using #MDB_WRITEMAP.
 *
 *  - A thread can only use one transaction at a time, plus any child
 *    transactions.  Each transaction belongs to one thread.  See below.
 *    The #MDB_NOTLS flag changes this for read-only transactions.
 *
 *  - Use an MDB_env* in the process which opened it, not after fork().
 *
 *  - Do not have open an LMDB database twice in the same process at
 *    the same time.  Not even from a plain open() call - close()ing it
 *    breaks fcntl() advisory locking.  (It is OK to reopen it after
 *    fork() - exec*(), since the lockfile has FD_CLOEXEC set.)
 *
 *  - Avoid long-lived transactions.  Read transactions prevent
 *    reuse of pages freed by newer write transactions, thus the
 *    database can grow quickly.  Write transactions prevent
 *    other write transactions, since writes are serialized.
 *
 *  - Avoid suspending a process with active transactions.  These
 *    would then be "long-lived" as above.  Also read transactions
 *    suspended when writers commit could sometimes see wrong data.
 *
 *  ...when several processes can use a database concurrently:
 *
 *  - Avoid aborting a process with an active transaction.
 *    The transaction becomes "long-lived" as above until a check
 *    for stale readers is performed or the lockfile is reset,
 *    since the process may not remove it from the lockfile.
 *
 *    This does not apply to write transactions if the system clears
 *    stale writers, see above.
 *
 *  - If you do that anyway, do a periodic check for stale readers. Or
 *    close the environment once in a while, so the lockfile can get reset.
 *
 *  - Do not use LMDB databases on remote filesystems, even between
 *    processes on the same host.  This breaks flock() on some OSes,
 *    possibly memory map sync, and certainly sync between programs
 *    on different hosts.
 *
 *  - Opening a database can fail if another process is opening or
 *    closing it at exactly the same time.
 *
 *  @author Howard Chu, Symas Corporation.
 *
 *  @copyright Copyright 2011-2018 Howard Chu, Symas Corp. All rights reserved.
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted only as authorized by the OpenLDAP
 * Public License.
 *
 * A copy of this license is available in the file LICENSE in the
 * top-level directory of the distribution or, alternatively, at
 * <http://www.OpenLDAP.org/license.html>.
 *
 *  @par Derived From:
 * This code is derived from btree.c written by Martin Hedenfalk.
 *
 * Copyright (c) 2009, 2010 Martin Hedenfalk <martin@bzero.se>
 *
 * Permission to use, copy, modify, and distribute this software for any
 * purpose with or without fee is hereby granted, provided that the above
 * copyright notice and this permission notice appear in all copies.
 *
 * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
 * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
 * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
 * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
 * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
 * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
 * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
 */

解释如下：

LMDB是在BerkeleyDB的基础上基于B树与来管理数据的数据库。其所有的数据均通过mmap从内存中读出，所以不需要maclloc以及memcpy操作参与。由于不需要page缓存层的参与，所以该数据库非常简单与高效。同样该DB使用ACID语义，当内存映射为只读时，数据库完整性不会被应用程序代码中的指针写入破坏。

该DB支持多线程多进程并发读写操作。数据页使用“写时复制”策略，所以避免了数据的重写，所以支持很好的错误恢复特性。其写操作是顺序进行的，在一个时间只允许一个写操作进行因此保证了写者不会死锁。数据结构是多版本的所以读者可以在无锁的环境下进行操作；除此之外，写者不会阻塞写操作，反之也是。

不同于其他使用写前日志或追加写方案的的数据库，LMDB在操作时没有进行维护工作。上述两种方案均需要定期进行检查点保存或者log合并。而LMDB会追踪空闲的页并重复利用他们进行后续写，所以数据库的大小并不会增长的太严重。

mmap机制被用于只读或者读写映射，默认情况下呗设置为只读从而防止数据库被破坏。使用读写模式可以提高写性能，但是一些应用如果编写不严格会破坏数据库。

次使用kv存储，我们需要看一下下面的内容：

为了解决文件锁问题，需要对BSD系统增加信号量机制：