You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

848 lines
32 KiB

  1. Bolt [![Coverage Status](https://coveralls.io/repos/boltdb/bolt/badge.svg?branch=master)](https://coveralls.io/r/boltdb/bolt?branch=master) [![GoDoc](https://godoc.org/github.com/boltdb/bolt?status.svg)](https://godoc.org/github.com/boltdb/bolt) ![Version](https://img.shields.io/badge/version-1.0-green.svg)
  2. ====
  3. Bolt is a pure Go key/value store inspired by [Howard Chu's][hyc_symas]
  4. [LMDB project][lmdb]. The goal of the project is to provide a simple,
  5. fast, and reliable database for projects that don't require a full database
  6. server such as Postgres or MySQL.
  7. Since Bolt is meant to be used as such a low-level piece of functionality,
  8. simplicity is key. The API will be small and only focus on getting values
  9. and setting values. That's it.
  10. [hyc_symas]: https://twitter.com/hyc_symas
  11. [lmdb]: http://symas.com/mdb/
  12. ## Project Status
  13. Bolt is stable and the API is fixed. Full unit test coverage and randomized
  14. black box testing are used to ensure database consistency and thread safety.
  15. Bolt is currently in high-load production environments serving databases as
  16. large as 1TB. Many companies such as Shopify and Heroku use Bolt-backed
  17. services every day.
  18. ## Table of Contents
  19. - [Getting Started](#getting-started)
  20. - [Installing](#installing)
  21. - [Opening a database](#opening-a-database)
  22. - [Transactions](#transactions)
  23. - [Read-write transactions](#read-write-transactions)
  24. - [Read-only transactions](#read-only-transactions)
  25. - [Batch read-write transactions](#batch-read-write-transactions)
  26. - [Managing transactions manually](#managing-transactions-manually)
  27. - [Using buckets](#using-buckets)
  28. - [Using key/value pairs](#using-keyvalue-pairs)
  29. - [Autoincrementing integer for the bucket](#autoincrementing-integer-for-the-bucket)
  30. - [Iterating over keys](#iterating-over-keys)
  31. - [Prefix scans](#prefix-scans)
  32. - [Range scans](#range-scans)
  33. - [ForEach()](#foreach)
  34. - [Nested buckets](#nested-buckets)
  35. - [Database backups](#database-backups)
  36. - [Statistics](#statistics)
  37. - [Read-Only Mode](#read-only-mode)
  38. - [Mobile Use (iOS/Android)](#mobile-use-iosandroid)
  39. - [Resources](#resources)
  40. - [Comparison with other databases](#comparison-with-other-databases)
  41. - [Postgres, MySQL, & other relational databases](#postgres-mysql--other-relational-databases)
  42. - [LevelDB, RocksDB](#leveldb-rocksdb)
  43. - [LMDB](#lmdb)
  44. - [Caveats & Limitations](#caveats--limitations)
  45. - [Reading the Source](#reading-the-source)
  46. - [Other Projects Using Bolt](#other-projects-using-bolt)
  47. ## Getting Started
  48. ### Installing
  49. To start using Bolt, install Go and run `go get`:
  50. ```sh
  51. $ go get github.com/boltdb/bolt/...
  52. ```
  53. This will retrieve the library and install the `bolt` command line utility into
  54. your `$GOBIN` path.
  55. ### Opening a database
  56. The top-level object in Bolt is a `DB`. It is represented as a single file on
  57. your disk and represents a consistent snapshot of your data.
  58. To open your database, simply use the `bolt.Open()` function:
  59. ```go
  60. package main
  61. import (
  62. "log"
  63. "github.com/boltdb/bolt"
  64. )
  65. func main() {
  66. // Open the my.db data file in your current directory.
  67. // It will be created if it doesn't exist.
  68. db, err := bolt.Open("my.db", 0600, nil)
  69. if err != nil {
  70. log.Fatal(err)
  71. }
  72. defer db.Close()
  73. ...
  74. }
  75. ```
  76. Please note that Bolt obtains a file lock on the data file so multiple processes
  77. cannot open the same database at the same time. Opening an already open Bolt
  78. database will cause it to hang until the other process closes it. To prevent
  79. an indefinite wait you can pass a timeout option to the `Open()` function:
  80. ```go
  81. db, err := bolt.Open("my.db", 0600, &bolt.Options{Timeout: 1 * time.Second})
  82. ```
  83. ### Transactions
  84. Bolt allows only one read-write transaction at a time but allows as many
  85. read-only transactions as you want at a time. Each transaction has a consistent
  86. view of the data as it existed when the transaction started.
  87. Individual transactions and all objects created from them (e.g. buckets, keys)
  88. are not thread safe. To work with data in multiple goroutines you must start
  89. a transaction for each one or use locking to ensure only one goroutine accesses
  90. a transaction at a time. Creating transaction from the `DB` is thread safe.
  91. Read-only transactions and read-write transactions should not depend on one
  92. another and generally shouldn't be opened simultaneously in the same goroutine.
  93. This can cause a deadlock as the read-write transaction needs to periodically
  94. re-map the data file but it cannot do so while a read-only transaction is open.
  95. #### Read-write transactions
  96. To start a read-write transaction, you can use the `DB.Update()` function:
  97. ```go
  98. err := db.Update(func(tx *bolt.Tx) error {
  99. ...
  100. return nil
  101. })
  102. ```
  103. Inside the closure, you have a consistent view of the database. You commit the
  104. transaction by returning `nil` at the end. You can also rollback the transaction
  105. at any point by returning an error. All database operations are allowed inside
  106. a read-write transaction.
  107. Always check the return error as it will report any disk failures that can cause
  108. your transaction to not complete. If you return an error within your closure
  109. it will be passed through.
  110. #### Read-only transactions
  111. To start a read-only transaction, you can use the `DB.View()` function:
  112. ```go
  113. err := db.View(func(tx *bolt.Tx) error {
  114. ...
  115. return nil
  116. })
  117. ```
  118. You also get a consistent view of the database within this closure, however,
  119. no mutating operations are allowed within a read-only transaction. You can only
  120. retrieve buckets, retrieve values, and copy the database within a read-only
  121. transaction.
  122. #### Batch read-write transactions
  123. Each `DB.Update()` waits for disk to commit the writes. This overhead
  124. can be minimized by combining multiple updates with the `DB.Batch()`
  125. function:
  126. ```go
  127. err := db.Batch(func(tx *bolt.Tx) error {
  128. ...
  129. return nil
  130. })
  131. ```
  132. Concurrent Batch calls are opportunistically combined into larger
  133. transactions. Batch is only useful when there are multiple goroutines
  134. calling it.
  135. The trade-off is that `Batch` can call the given
  136. function multiple times, if parts of the transaction fail. The
  137. function must be idempotent and side effects must take effect only
  138. after a successful return from `DB.Batch()`.
  139. For example: don't display messages from inside the function, instead
  140. set variables in the enclosing scope:
  141. ```go
  142. var id uint64
  143. err := db.Batch(func(tx *bolt.Tx) error {
  144. // Find last key in bucket, decode as bigendian uint64, increment
  145. // by one, encode back to []byte, and add new key.
  146. ...
  147. id = newValue
  148. return nil
  149. })
  150. if err != nil {
  151. return ...
  152. }
  153. fmt.Println("Allocated ID %d", id)
  154. ```
  155. #### Managing transactions manually
  156. The `DB.View()` and `DB.Update()` functions are wrappers around the `DB.Begin()`
  157. function. These helper functions will start the transaction, execute a function,
  158. and then safely close your transaction if an error is returned. This is the
  159. recommended way to use Bolt transactions.
  160. However, sometimes you may want to manually start and end your transactions.
  161. You can use the `Tx.Begin()` function directly but **please** be sure to close
  162. the transaction.
  163. ```go
  164. // Start a writable transaction.
  165. tx, err := db.Begin(true)
  166. if err != nil {
  167. return err
  168. }
  169. defer tx.Rollback()
  170. // Use the transaction...
  171. _, err := tx.CreateBucket([]byte("MyBucket"))
  172. if err != nil {
  173. return err
  174. }
  175. // Commit the transaction and check for error.
  176. if err := tx.Commit(); err != nil {
  177. return err
  178. }
  179. ```
  180. The first argument to `DB.Begin()` is a boolean stating if the transaction
  181. should be writable.
  182. ### Using buckets
  183. Buckets are collections of key/value pairs within the database. All keys in a
  184. bucket must be unique. You can create a bucket using the `DB.CreateBucket()`
  185. function:
  186. ```go
  187. db.Update(func(tx *bolt.Tx) error {
  188. b, err := tx.CreateBucket([]byte("MyBucket"))
  189. if err != nil {
  190. return fmt.Errorf("create bucket: %s", err)
  191. }
  192. return nil
  193. })
  194. ```
  195. You can also create a bucket only if it doesn't exist by using the
  196. `Tx.CreateBucketIfNotExists()` function. It's a common pattern to call this
  197. function for all your top-level buckets after you open your database so you can
  198. guarantee that they exist for future transactions.
  199. To delete a bucket, simply call the `Tx.DeleteBucket()` function.
  200. ### Using key/value pairs
  201. To save a key/value pair to a bucket, use the `Bucket.Put()` function:
  202. ```go
  203. db.Update(func(tx *bolt.Tx) error {
  204. b := tx.Bucket([]byte("MyBucket"))
  205. err := b.Put([]byte("answer"), []byte("42"))
  206. return err
  207. })
  208. ```
  209. This will set the value of the `"answer"` key to `"42"` in the `MyBucket`
  210. bucket. To retrieve this value, we can use the `Bucket.Get()` function:
  211. ```go
  212. db.View(func(tx *bolt.Tx) error {
  213. b := tx.Bucket([]byte("MyBucket"))
  214. v := b.Get([]byte("answer"))
  215. fmt.Printf("The answer is: %s\n", v)
  216. return nil
  217. })
  218. ```
  219. The `Get()` function does not return an error because its operation is
  220. guaranteed to work (unless there is some kind of system failure). If the key
  221. exists then it will return its byte slice value. If it doesn't exist then it
  222. will return `nil`. It's important to note that you can have a zero-length value
  223. set to a key which is different than the key not existing.
  224. Use the `Bucket.Delete()` function to delete a key from the bucket.
  225. Please note that values returned from `Get()` are only valid while the
  226. transaction is open. If you need to use a value outside of the transaction
  227. then you must use `copy()` to copy it to another byte slice.
  228. ### Autoincrementing integer for the bucket
  229. By using the `NextSequence()` function, you can let Bolt determine a sequence
  230. which can be used as the unique identifier for your key/value pairs. See the
  231. example below.
  232. ```go
  233. // CreateUser saves u to the store. The new user ID is set on u once the data is persisted.
  234. func (s *Store) CreateUser(u *User) error {
  235. return s.db.Update(func(tx *bolt.Tx) error {
  236. // Retrieve the users bucket.
  237. // This should be created when the DB is first opened.
  238. b := tx.Bucket([]byte("users"))
  239. // Generate ID for the user.
  240. // This returns an error only if the Tx is closed or not writeable.
  241. // That can't happen in an Update() call so I ignore the error check.
  242. id, _ = b.NextSequence()
  243. u.ID = int(id)
  244. // Marshal user data into bytes.
  245. buf, err := json.Marshal(u)
  246. if err != nil {
  247. return err
  248. }
  249. // Persist bytes to users bucket.
  250. return b.Put(itob(u.ID), buf)
  251. })
  252. }
  253. // itob returns an 8-byte big endian representation of v.
  254. func itob(v int) []byte {
  255. b := make([]byte, 8)
  256. binary.BigEndian.PutUint64(b, uint64(v))
  257. return b
  258. }
  259. type User struct {
  260. ID int
  261. ...
  262. }
  263. ```
  264. ### Iterating over keys
  265. Bolt stores its keys in byte-sorted order within a bucket. This makes sequential
  266. iteration over these keys extremely fast. To iterate over keys we'll use a
  267. `Cursor`:
  268. ```go
  269. db.View(func(tx *bolt.Tx) error {
  270. // Assume bucket exists and has keys
  271. b := tx.Bucket([]byte("MyBucket"))
  272. c := b.Cursor()
  273. for k, v := c.First(); k != nil; k, v = c.Next() {
  274. fmt.Printf("key=%s, value=%s\n", k, v)
  275. }
  276. return nil
  277. })
  278. ```
  279. The cursor allows you to move to a specific point in the list of keys and move
  280. forward or backward through the keys one at a time.
  281. The following functions are available on the cursor:
  282. ```
  283. First() Move to the first key.
  284. Last() Move to the last key.
  285. Seek() Move to a specific key.
  286. Next() Move to the next key.
  287. Prev() Move to the previous key.
  288. ```
  289. Each of those functions has a return signature of `(key []byte, value []byte)`.
  290. When you have iterated to the end of the cursor then `Next()` will return a
  291. `nil` key. You must seek to a position using `First()`, `Last()`, or `Seek()`
  292. before calling `Next()` or `Prev()`. If you do not seek to a position then
  293. these functions will return a `nil` key.
  294. During iteration, if the key is non-`nil` but the value is `nil`, that means
  295. the key refers to a bucket rather than a value. Use `Bucket.Bucket()` to
  296. access the sub-bucket.
  297. #### Prefix scans
  298. To iterate over a key prefix, you can combine `Seek()` and `bytes.HasPrefix()`:
  299. ```go
  300. db.View(func(tx *bolt.Tx) error {
  301. // Assume bucket exists and has keys
  302. c := tx.Bucket([]byte("MyBucket")).Cursor()
  303. prefix := []byte("1234")
  304. for k, v := c.Seek(prefix); bytes.HasPrefix(k, prefix); k, v = c.Next() {
  305. fmt.Printf("key=%s, value=%s\n", k, v)
  306. }
  307. return nil
  308. })
  309. ```
  310. #### Range scans
  311. Another common use case is scanning over a range such as a time range. If you
  312. use a sortable time encoding such as RFC3339 then you can query a specific
  313. date range like this:
  314. ```go
  315. db.View(func(tx *bolt.Tx) error {
  316. // Assume our events bucket exists and has RFC3339 encoded time keys.
  317. c := tx.Bucket([]byte("Events")).Cursor()
  318. // Our time range spans the 90's decade.
  319. min := []byte("1990-01-01T00:00:00Z")
  320. max := []byte("2000-01-01T00:00:00Z")
  321. // Iterate over the 90's.
  322. for k, v := c.Seek(min); k != nil && bytes.Compare(k, max) <= 0; k, v = c.Next() {
  323. fmt.Printf("%s: %s\n", k, v)
  324. }
  325. return nil
  326. })
  327. ```
  328. Note that, while RFC3339 is sortable, the Golang implementation of RFC3339Nano does not use a fixed number of digits after the decimal point and is therefore not sortable.
  329. #### ForEach()
  330. You can also use the function `ForEach()` if you know you'll be iterating over
  331. all the keys in a bucket:
  332. ```go
  333. db.View(func(tx *bolt.Tx) error {
  334. // Assume bucket exists and has keys
  335. b := tx.Bucket([]byte("MyBucket"))
  336. b.ForEach(func(k, v []byte) error {
  337. fmt.Printf("key=%s, value=%s\n", k, v)
  338. return nil
  339. })
  340. return nil
  341. })
  342. ```
  343. ### Nested buckets
  344. You can also store a bucket in a key to create nested buckets. The API is the
  345. same as the bucket management API on the `DB` object:
  346. ```go
  347. func (*Bucket) CreateBucket(key []byte) (*Bucket, error)
  348. func (*Bucket) CreateBucketIfNotExists(key []byte) (*Bucket, error)
  349. func (*Bucket) DeleteBucket(key []byte) error
  350. ```
  351. ### Database backups
  352. Bolt is a single file so it's easy to backup. You can use the `Tx.WriteTo()`
  353. function to write a consistent view of the database to a writer. If you call
  354. this from a read-only transaction, it will perform a hot backup and not block
  355. your other database reads and writes.
  356. By default, it will use a regular file handle which will utilize the operating
  357. system's page cache. See the [`Tx`](https://godoc.org/github.com/boltdb/bolt#Tx)
  358. documentation for information about optimizing for larger-than-RAM datasets.
  359. One common use case is to backup over HTTP so you can use tools like `cURL` to
  360. do database backups:
  361. ```go
  362. func BackupHandleFunc(w http.ResponseWriter, req *http.Request) {
  363. err := db.View(func(tx *bolt.Tx) error {
  364. w.Header().Set("Content-Type", "application/octet-stream")
  365. w.Header().Set("Content-Disposition", `attachment; filename="my.db"`)
  366. w.Header().Set("Content-Length", strconv.Itoa(int(tx.Size())))
  367. _, err := tx.WriteTo(w)
  368. return err
  369. })
  370. if err != nil {
  371. http.Error(w, err.Error(), http.StatusInternalServerError)
  372. }
  373. }
  374. ```
  375. Then you can backup using this command:
  376. ```sh
  377. $ curl http://localhost/backup > my.db
  378. ```
  379. Or you can open your browser to `http://localhost/backup` and it will download
  380. automatically.
  381. If you want to backup to another file you can use the `Tx.CopyFile()` helper
  382. function.
  383. ### Statistics
  384. The database keeps a running count of many of the internal operations it
  385. performs so you can better understand what's going on. By grabbing a snapshot
  386. of these stats at two points in time we can see what operations were performed
  387. in that time range.
  388. For example, we could start a goroutine to log stats every 10 seconds:
  389. ```go
  390. go func() {
  391. // Grab the initial stats.
  392. prev := db.Stats()
  393. for {
  394. // Wait for 10s.
  395. time.Sleep(10 * time.Second)
  396. // Grab the current stats and diff them.
  397. stats := db.Stats()
  398. diff := stats.Sub(&prev)
  399. // Encode stats to JSON and print to STDERR.
  400. json.NewEncoder(os.Stderr).Encode(diff)
  401. // Save stats for the next loop.
  402. prev = stats
  403. }
  404. }()
  405. ```
  406. It's also useful to pipe these stats to a service such as statsd for monitoring
  407. or to provide an HTTP endpoint that will perform a fixed-length sample.
  408. ### Read-Only Mode
  409. Sometimes it is useful to create a shared, read-only Bolt database. To this,
  410. set the `Options.ReadOnly` flag when opening your database. Read-only mode
  411. uses a shared lock to allow multiple processes to read from the database but
  412. it will block any processes from opening the database in read-write mode.
  413. ```go
  414. db, err := bolt.Open("my.db", 0666, &bolt.Options{ReadOnly: true})
  415. if err != nil {
  416. log.Fatal(err)
  417. }
  418. ```
  419. ### Mobile Use (iOS/Android)
  420. Bolt is able to run on mobile devices by leveraging the binding feature of the
  421. [gomobile](https://github.com/golang/mobile) tool. Create a struct that will
  422. contain your database logic and a reference to a `*bolt.DB` with a initializing
  423. contstructor that takes in a filepath where the database file will be stored.
  424. Neither Android nor iOS require extra permissions or cleanup from using this method.
  425. ```go
  426. func NewBoltDB(filepath string) *BoltDB {
  427. db, err := bolt.Open(filepath+"/demo.db", 0600, nil)
  428. if err != nil {
  429. log.Fatal(err)
  430. }
  431. return &BoltDB{db}
  432. }
  433. type BoltDB struct {
  434. db *bolt.DB
  435. ...
  436. }
  437. func (b *BoltDB) Path() string {
  438. return b.db.Path()
  439. }
  440. func (b *BoltDB) Close() {
  441. b.db.Close()
  442. }
  443. ```
  444. Database logic should be defined as methods on this wrapper struct.
  445. To initialize this struct from the native language (both platforms now sync
  446. their local storage to the cloud. These snippets disable that functionality for the
  447. database file):
  448. #### Android
  449. ```java
  450. String path;
  451. if (android.os.Build.VERSION.SDK_INT >=android.os.Build.VERSION_CODES.LOLLIPOP){
  452. path = getNoBackupFilesDir().getAbsolutePath();
  453. } else{
  454. path = getFilesDir().getAbsolutePath();
  455. }
  456. Boltmobiledemo.BoltDB boltDB = Boltmobiledemo.NewBoltDB(path)
  457. ```
  458. #### iOS
  459. ```objc
  460. - (void)demo {
  461. NSString* path = [NSSearchPathForDirectoriesInDomains(NSLibraryDirectory,
  462. NSUserDomainMask,
  463. YES) objectAtIndex:0];
  464. GoBoltmobiledemoBoltDB * demo = GoBoltmobiledemoNewBoltDB(path);
  465. [self addSkipBackupAttributeToItemAtPath:demo.path];
  466. //Some DB Logic would go here
  467. [demo close];
  468. }
  469. - (BOOL)addSkipBackupAttributeToItemAtPath:(NSString *) filePathString
  470. {
  471. NSURL* URL= [NSURL fileURLWithPath: filePathString];
  472. assert([[NSFileManager defaultManager] fileExistsAtPath: [URL path]]);
  473. NSError *error = nil;
  474. BOOL success = [URL setResourceValue: [NSNumber numberWithBool: YES]
  475. forKey: NSURLIsExcludedFromBackupKey error: &error];
  476. if(!success){
  477. NSLog(@"Error excluding %@ from backup %@", [URL lastPathComponent], error);
  478. }
  479. return success;
  480. }
  481. ```
  482. ## Resources
  483. For more information on getting started with Bolt, check out the following articles:
  484. * [Intro to BoltDB: Painless Performant Persistence](http://npf.io/2014/07/intro-to-boltdb-painless-performant-persistence/) by [Nate Finch](https://github.com/natefinch).
  485. * [Bolt -- an embedded key/value database for Go](https://www.progville.com/go/bolt-embedded-db-golang/) by Progville
  486. ## Comparison with other databases
  487. ### Postgres, MySQL, & other relational databases
  488. Relational databases structure data into rows and are only accessible through
  489. the use of SQL. This approach provides flexibility in how you store and query
  490. your data but also incurs overhead in parsing and planning SQL statements. Bolt
  491. accesses all data by a byte slice key. This makes Bolt fast to read and write
  492. data by key but provides no built-in support for joining values together.
  493. Most relational databases (with the exception of SQLite) are standalone servers
  494. that run separately from your application. This gives your systems
  495. flexibility to connect multiple application servers to a single database
  496. server but also adds overhead in serializing and transporting data over the
  497. network. Bolt runs as a library included in your application so all data access
  498. has to go through your application's process. This brings data closer to your
  499. application but limits multi-process access to the data.
  500. ### LevelDB, RocksDB
  501. LevelDB and its derivatives (RocksDB, HyperLevelDB) are similar to Bolt in that
  502. they are libraries bundled into the application, however, their underlying
  503. structure is a log-structured merge-tree (LSM tree). An LSM tree optimizes
  504. random writes by using a write ahead log and multi-tiered, sorted files called
  505. SSTables. Bolt uses a B+tree internally and only a single file. Both approaches
  506. have trade-offs.
  507. If you require a high random write throughput (>10,000 w/sec) or you need to use
  508. spinning disks then LevelDB could be a good choice. If your application is
  509. read-heavy or does a lot of range scans then Bolt could be a good choice.
  510. One other important consideration is that LevelDB does not have transactions.
  511. It supports batch writing of key/values pairs and it supports read snapshots
  512. but it will not give you the ability to do a compare-and-swap operation safely.
  513. Bolt supports fully serializable ACID transactions.
  514. ### LMDB
  515. Bolt was originally a port of LMDB so it is architecturally similar. Both use
  516. a B+tree, have ACID semantics with fully serializable transactions, and support
  517. lock-free MVCC using a single writer and multiple readers.
  518. The two projects have somewhat diverged. LMDB heavily focuses on raw performance
  519. while Bolt has focused on simplicity and ease of use. For example, LMDB allows
  520. several unsafe actions such as direct writes for the sake of performance. Bolt
  521. opts to disallow actions which can leave the database in a corrupted state. The
  522. only exception to this in Bolt is `DB.NoSync`.
  523. There are also a few differences in API. LMDB requires a maximum mmap size when
  524. opening an `mdb_env` whereas Bolt will handle incremental mmap resizing
  525. automatically. LMDB overloads the getter and setter functions with multiple
  526. flags whereas Bolt splits these specialized cases into their own functions.
  527. ## Caveats & Limitations
  528. It's important to pick the right tool for the job and Bolt is no exception.
  529. Here are a few things to note when evaluating and using Bolt:
  530. * Bolt is good for read intensive workloads. Sequential write performance is
  531. also fast but random writes can be slow. You can use `DB.Batch()` or add a
  532. write-ahead log to help mitigate this issue.
  533. * Bolt uses a B+tree internally so there can be a lot of random page access.
  534. SSDs provide a significant performance boost over spinning disks.
  535. * Try to avoid long running read transactions. Bolt uses copy-on-write so
  536. old pages cannot be reclaimed while an old transaction is using them.
  537. * Byte slices returned from Bolt are only valid during a transaction. Once the
  538. transaction has been committed or rolled back then the memory they point to
  539. can be reused by a new page or can be unmapped from virtual memory and you'll
  540. see an `unexpected fault address` panic when accessing it.
  541. * Be careful when using `Bucket.FillPercent`. Setting a high fill percent for
  542. buckets that have random inserts will cause your database to have very poor
  543. page utilization.
  544. * Use larger buckets in general. Smaller buckets causes poor page utilization
  545. once they become larger than the page size (typically 4KB).
  546. * Bulk loading a lot of random writes into a new bucket can be slow as the
  547. page will not split until the transaction is committed. Randomly inserting
  548. more than 100,000 key/value pairs into a single new bucket in a single
  549. transaction is not advised.
  550. * Bolt uses a memory-mapped file so the underlying operating system handles the
  551. caching of the data. Typically, the OS will cache as much of the file as it
  552. can in memory and will release memory as needed to other processes. This means
  553. that Bolt can show very high memory usage when working with large databases.
  554. However, this is expected and the OS will release memory as needed. Bolt can
  555. handle databases much larger than the available physical RAM, provided its
  556. memory-map fits in the process virtual address space. It may be problematic
  557. on 32-bits systems.
  558. * The data structures in the Bolt database are memory mapped so the data file
  559. will be endian specific. This means that you cannot copy a Bolt file from a
  560. little endian machine to a big endian machine and have it work. For most
  561. users this is not a concern since most modern CPUs are little endian.
  562. * Because of the way pages are laid out on disk, Bolt cannot truncate data files
  563. and return free pages back to the disk. Instead, Bolt maintains a free list
  564. of unused pages within its data file. These free pages can be reused by later
  565. transactions. This works well for many use cases as databases generally tend
  566. to grow. However, it's important to note that deleting large chunks of data
  567. will not allow you to reclaim that space on disk.
  568. For more information on page allocation, [see this comment][page-allocation].
  569. [page-allocation]: https://github.com/boltdb/bolt/issues/308#issuecomment-74811638
  570. ## Reading the Source
  571. Bolt is a relatively small code base (<3KLOC) for an embedded, serializable,
  572. transactional key/value database so it can be a good starting point for people
  573. interested in how databases work.
  574. The best places to start are the main entry points into Bolt:
  575. - `Open()` - Initializes the reference to the database. It's responsible for
  576. creating the database if it doesn't exist, obtaining an exclusive lock on the
  577. file, reading the meta pages, & memory-mapping the file.
  578. - `DB.Begin()` - Starts a read-only or read-write transaction depending on the
  579. value of the `writable` argument. This requires briefly obtaining the "meta"
  580. lock to keep track of open transactions. Only one read-write transaction can
  581. exist at a time so the "rwlock" is acquired during the life of a read-write
  582. transaction.
  583. - `Bucket.Put()` - Writes a key/value pair into a bucket. After validating the
  584. arguments, a cursor is used to traverse the B+tree to the page and position
  585. where they key & value will be written. Once the position is found, the bucket
  586. materializes the underlying page and the page's parent pages into memory as
  587. "nodes". These nodes are where mutations occur during read-write transactions.
  588. These changes get flushed to disk during commit.
  589. - `Bucket.Get()` - Retrieves a key/value pair from a bucket. This uses a cursor
  590. to move to the page & position of a key/value pair. During a read-only
  591. transaction, the key and value data is returned as a direct reference to the
  592. underlying mmap file so there's no allocation overhead. For read-write
  593. transactions, this data may reference the mmap file or one of the in-memory
  594. node values.
  595. - `Cursor` - This object is simply for traversing the B+tree of on-disk pages
  596. or in-memory nodes. It can seek to a specific key, move to the first or last
  597. value, or it can move forward or backward. The cursor handles the movement up
  598. and down the B+tree transparently to the end user.
  599. - `Tx.Commit()` - Converts the in-memory dirty nodes and the list of free pages
  600. into pages to be written to disk. Writing to disk then occurs in two phases.
  601. First, the dirty pages are written to disk and an `fsync()` occurs. Second, a
  602. new meta page with an incremented transaction ID is written and another
  603. `fsync()` occurs. This two phase write ensures that partially written data
  604. pages are ignored in the event of a crash since the meta page pointing to them
  605. is never written. Partially written meta pages are invalidated because they
  606. are written with a checksum.
  607. If you have additional notes that could be helpful for others, please submit
  608. them via pull request.
  609. ## Other Projects Using Bolt
  610. Below is a list of public, open source projects that use Bolt:
  611. * [Operation Go: A Routine Mission](http://gocode.io) - An online programming game for Golang using Bolt for user accounts and a leaderboard.
  612. * [Bazil](https://bazil.org/) - A file system that lets your data reside where it is most convenient for it to reside.
  613. * [DVID](https://github.com/janelia-flyem/dvid) - Added Bolt as optional storage engine and testing it against Basho-tuned leveldb.
  614. * [Skybox Analytics](https://github.com/skybox/skybox) - A standalone funnel analysis tool for web analytics.
  615. * [Scuttlebutt](https://github.com/benbjohnson/scuttlebutt) - Uses Bolt to store and process all Twitter mentions of GitHub projects.
  616. * [Wiki](https://github.com/peterhellberg/wiki) - A tiny wiki using Goji, BoltDB and Blackfriday.
  617. * [ChainStore](https://github.com/pressly/chainstore) - Simple key-value interface to a variety of storage engines organized as a chain of operations.
  618. * [MetricBase](https://github.com/msiebuhr/MetricBase) - Single-binary version of Graphite.
  619. * [Gitchain](https://github.com/gitchain/gitchain) - Decentralized, peer-to-peer Git repositories aka "Git meets Bitcoin".
  620. * [event-shuttle](https://github.com/sclasen/event-shuttle) - A Unix system service to collect and reliably deliver messages to Kafka.
  621. * [ipxed](https://github.com/kelseyhightower/ipxed) - Web interface and api for ipxed.
  622. * [BoltStore](https://github.com/yosssi/boltstore) - Session store using Bolt.
  623. * [photosite/session](https://godoc.org/bitbucket.org/kardianos/photosite/session) - Sessions for a photo viewing site.
  624. * [LedisDB](https://github.com/siddontang/ledisdb) - A high performance NoSQL, using Bolt as optional storage.
  625. * [ipLocator](https://github.com/AndreasBriese/ipLocator) - A fast ip-geo-location-server using bolt with bloom filters.
  626. * [cayley](https://github.com/google/cayley) - Cayley is an open-source graph database using Bolt as optional backend.
  627. * [bleve](http://www.blevesearch.com/) - A pure Go search engine similar to ElasticSearch that uses Bolt as the default storage backend.
  628. * [tentacool](https://github.com/optiflows/tentacool) - REST api server to manage system stuff (IP, DNS, Gateway...) on a linux server.
  629. * [SkyDB](https://github.com/skydb/sky) - Behavioral analytics database.
  630. * [Seaweed File System](https://github.com/chrislusf/seaweedfs) - Highly scalable distributed key~file system with O(1) disk read.
  631. * [InfluxDB](https://influxdata.com) - Scalable datastore for metrics, events, and real-time analytics.
  632. * [Freehold](http://tshannon.bitbucket.org/freehold/) - An open, secure, and lightweight platform for your files and data.
  633. * [Prometheus Annotation Server](https://github.com/oliver006/prom_annotation_server) - Annotation server for PromDash & Prometheus service monitoring system.
  634. * [Consul](https://github.com/hashicorp/consul) - Consul is service discovery and configuration made easy. Distributed, highly available, and datacenter-aware.
  635. * [Kala](https://github.com/ajvb/kala) - Kala is a modern job scheduler optimized to run on a single node. It is persistent, JSON over HTTP API, ISO 8601 duration notation, and dependent jobs.
  636. * [drive](https://github.com/odeke-em/drive) - drive is an unofficial Google Drive command line client for \*NIX operating systems.
  637. * [stow](https://github.com/djherbis/stow) - a persistence manager for objects
  638. backed by boltdb.
  639. * [buckets](https://github.com/joyrexus/buckets) - a bolt wrapper streamlining
  640. simple tx and key scans.
  641. * [mbuckets](https://github.com/abhigupta912/mbuckets) - A Bolt wrapper that allows easy operations on multi level (nested) buckets.
  642. * [Request Baskets](https://github.com/darklynx/request-baskets) - A web service to collect arbitrary HTTP requests and inspect them via REST API or simple web UI, similar to [RequestBin](http://requestb.in/) service
  643. * [Go Report Card](https://goreportcard.com/) - Go code quality report cards as a (free and open source) service.
  644. * [Boltdb Boilerplate](https://github.com/bobintornado/boltdb-boilerplate) - Boilerplate wrapper around bolt aiming to make simple calls one-liners.
  645. * [lru](https://github.com/crowdriff/lru) - Easy to use Bolt-backed Least-Recently-Used (LRU) read-through cache with chainable remote stores.
  646. * [Storm](https://github.com/asdine/storm) - A simple ORM around BoltDB.
  647. * [GoWebApp](https://github.com/josephspurrier/gowebapp) - A basic MVC web application in Go using BoltDB.
  648. If you are using Bolt in a project please send a pull request to add it to the list.