Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 55 additions & 9 deletions docs/server/kb/linux-setting-memlock.mdx
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: "Linux: Setting memlock when using encrypted database"
sidebar_label: "Linux: Setting `memlock` when using encrypted database"
description: "Set the memlock ulimit on Linux to allow RavenDB's Voron storage engine to lock memory pages and prevent swapping for optimal performance."
description: "Configure the memlock ulimit on Linux so that RavenDB can lock memory pages required by encrypted databases."
sidebar_position: 4
---

Expand All @@ -11,20 +11,66 @@ import TabItem from '@theme/TabItem';
import CodeBlock from '@theme/CodeBlock';
import LanguageSwitcher from "@site/src/components/LanguageSwitcher";
import LanguageContent from "@site/src/components/LanguageContent";
import Panel from "@site/src/components/Panel";
import ContentFrame from "@site/src/components/ContentFrame";

# Linux: Setting memlock when using encrypted database
Encrypted database uses extensively sodium library which requires high values of locked memory limits.
`memlock` refers to memory that will not be paged out, and its limit can be viewed using `ulimit -l`.
The modification of `memlock` limit settings can be achieved by in running session with `prlimit`:

Example, for 1MB limit:
<Admonition type="note" title="">

* When [encryption at rest](../security/encryption/encryption-at-rest.mdx) is enabled, RavenDB uses the libsodium library to lock certain memory regions in order to prevent plaintext data from being swapped to disk.
Linux imposes a per-process limit on how much memory the process can lock (`memlock`).
If this limit is too low, RavenDB cannot open the encrypted database and throws an `InsufficientMemoryException`.

* The exception message includes the exact `prlimit` command needed to raise the limit, with the required size in bytes already filled in.

* In this article:
* [Raising the limit for a running process](linux-setting-memlock.mdx#raising-the-limit-for-a-running-process)
* [Setting the limit persistently](linux-setting-memlock.mdx#setting-the-limit-persistently)

</Admonition>

<Panel heading="Raising the limit for a running process">

Run the command shown in the exception message, replacing `<pid>` with the RavenDB process ID
and `<size-in-bytes>` with the value from the error:

```bash
sudo prlimit --pid <pid> --memlock=<size-in-bytes>
```
prlimit -p pid --memlock 1MB:1MB

<br />

This takes effect immediately but does not persist across restarts.

</Panel>

<Panel heading="Setting the limit persistently">

Add the following to `/etc/security/limits.conf`:

```text
* soft memlock unlimited
* hard memlock unlimited
```

Persistent settings can be achieved by adding to `/etc/security/limits.conf` the following:
<br />

Log out and back in (or restart the service) for the change to take effect.
Verify the new limit with:

```bash
ulimit -l
```
* soft memlock 1000
* hard memlock 1000

<Admonition type="note" title="">
If RavenDB runs as a `systemd` service, set the memory lock limit in the service unit file instead of `limits.conf`:

```ini
[Service]
LimitMEMLOCK=infinity
```

</Admonition>

</Panel>
4 changes: 2 additions & 2 deletions docs/server/security/common-errors-and-faq.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -513,7 +513,7 @@ In **Linux**, it is the admin's responsibility to configure higher limits manual
</CodeBlock>
</TabItem>

To figure out what the new limit should be, look at the exception thrown by RavenDB, which includes this size.

To figure out what the new limit should be, look at the exception thrown by RavenDB, which includes this size.
For persistent and service-level configuration, see [Linux: Setting memlock when using encrypted database](../kb/linux-setting-memlock.mdx).


7 changes: 4 additions & 3 deletions docs/server/security/encryption/encryption-at-rest.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -35,14 +35,14 @@ Once a request is made, RavenDB will start a transaction (either read or write)
request, and when the transaction is finished, modified pages are encrypted and written back to the datafile. The data is
[locked](../../../server/security/encryption/encryption-at-rest.mdx#locking-memory) during the transaction by default, and it is zeroed when the transaction completes.

<Admonition type="danger" title="Important things to be aware of:" id="important-things-to-be-aware-of" href="#important-things-to-be-aware-of">
<Admonition type="warning" title="Important things to be aware of:" id="important-things-to-be-aware-of" href="#important-things-to-be-aware-of">
1. RavenDB makes sure that **no data is written to disk as plain text**. It will always be encrypted.
2. Indexed fields (the actual terms and values being indexed) will reside in memory as plain text.
3. Data of the current transaction will reside in memory as plain text and only for the duration of the transaction. When the transaction ends, the used memory is safely zeroed.
4. Loading documents from the database (using the Studio, the Client API, REST API) means that they will be decrypted to plain text on the server and then sent to the client (securely) by HTTPS. Once the data is received on the client side it is no longer encrypted. RavenDB does not provide encryption on the client side.
</Admonition>

<Admonition type="danger" title="Indexing transaction size" id="indexing-transaction-size" href="#indexing-transaction-size">
<Admonition type="warning" title="Indexing transaction size" id="indexing-transaction-size" href="#indexing-transaction-size">
Indexing is most efficient when it is performed in the largest transactions possible. However, using encryption can be very memory intensive, and if memory
runs out before the transaction completes, the entire transaction will fail. To avoid this, you can limit the size of indexing batches in encrypted
databases using [Indexing.Encrypted.TransactionSizeLimitInMb](../../../server/configuration/indexing-configuration.mdx#indexingencryptedtransactionsizelimitinmb).
Expand Down Expand Up @@ -77,7 +77,8 @@ In **Linux**, it is the admin's responsibility to configure higher limits manual
</CodeBlock>
</TabItem>

To figure out what the new limit should be, look at the exception thrown by RavenDB, which includes this size.
To figure out what the new limit should be, look at the exception thrown by RavenDB, which includes this size.
For persistent and service-level configuration, see [Linux: Setting memlock when using encrypted database](../../kb/linux-setting-memlock.mdx).

## What about Encryption in Transit?

Expand Down
62 changes: 54 additions & 8 deletions versioned_docs/version-6.2/server/kb/linux-setting-memlock.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -10,20 +10,66 @@ import TabItem from '@theme/TabItem';
import CodeBlock from '@theme/CodeBlock';
import LanguageSwitcher from "@site/src/components/LanguageSwitcher";
import LanguageContent from "@site/src/components/LanguageContent";
import Panel from "@site/src/components/Panel";
import ContentFrame from "@site/src/components/ContentFrame";

# Linux: Setting memlock when using encrypted database
Encrypted database uses extensively sodium library which requires high values of locked memory limits.
`memlock` refers to memory that will not be paged out, and it's limit can be viewed usign `ulimit -l`.
The modification of `memlock` limit settings can be achieved by in running session with `prlimit`:

Example, for 1MB limit:
<Admonition type="note" title="">

* When [encryption at rest](../security/encryption/encryption-at-rest.mdx) is enabled, RavenDB uses the libsodium library to lock certain memory regions in order to prevent plaintext data from being swapped to disk.
Linux imposes a per-process limit on how much memory the process can lock (`memlock`).
If this limit is too low, RavenDB cannot open the encrypted database and throws an `InsufficientMemoryException`.

* The exception message includes the exact `prlimit` command needed to raise the limit, with the required size in bytes already filled in.

* In this article:
* [Raising the limit for a running process](linux-setting-memlock.mdx#raising-the-limit-for-a-running-process)
* [Setting the limit persistently](linux-setting-memlock.mdx#setting-the-limit-persistently)

</Admonition>

<Panel heading="Raising the limit for a running process">

Run the command shown in the exception message, replacing `<pid>` with the RavenDB process ID
and `<size-in-bytes>` with the value from the error:

```bash
sudo prlimit --pid <pid> --memlock=<size-in-bytes>
```
prlimit -p pid --memlock 1MB:1MB

<br />

This takes effect immediately but does not persist across restarts.

</Panel>

<Panel heading="Setting the limit persistently">

Add the following to `/etc/security/limits.conf`:

```text
* soft memlock unlimited
* hard memlock unlimited
```

Persistant settings can be achieved by adding to `/etc/security/limits.conf` the following:
<br />

Log out and back in (or restart the service) for the change to take effect.
Verify the new limit with:

```bash
ulimit -l
```
* soft memlock 1000
* hard memlock 1000

<Admonition type="note" title="">
If RavenDB runs as a `systemd` service, set the memory lock limit in the service unit file instead of `limits.conf`:

```ini
[Service]
LimitMEMLOCK=infinity
```

</Admonition>

</Panel>
Original file line number Diff line number Diff line change
Expand Up @@ -512,7 +512,7 @@ In **Linux**, it is the admin's responsibility to configure higher limits manual
</CodeBlock>
</TabItem>

To figure out what the new limit should be, look at the exception thrown by RavenDB, which includes this size.

To figure out what the new limit should be, look at the exception thrown by RavenDB, which includes this size.
For persistent and service-level configuration, see [Linux: Setting memlock when using encrypted database](../kb/linux-setting-memlock.mdx).


Original file line number Diff line number Diff line change
Expand Up @@ -34,14 +34,14 @@ Once a request is made, RavenDB will start a transaction (either read or write)
request, and when the transaction is finished, modified pages are encrypted and written back to the datafile. The data is
[locked](../../../server/security/encryption/encryption-at-rest.mdx#locking-memory) during the transaction by default, and it is zeroed when the transaction completes.

<Admonition type="danger" title="Important things to be aware of:" id="important-things-to-be-aware-of" href="#important-things-to-be-aware-of">
<Admonition type="warning" title="Important things to be aware of:" id="important-things-to-be-aware-of" href="#important-things-to-be-aware-of">
1. RavenDB makes sure that **no data is written to disk as plain text**. It will always be encrypted.
2. Indexed fields (the actual terms and values being indexed) will reside in memory as plain text.
3. Data of the current transaction will reside in memory as plain text and only for the duration of the transaction. When the transaction ends, the used memory is safely zeroed.
4. Loading documents from the database (using the Studio, the Client API, REST API) means that they will be decrypted to plain text on the server and then sent to the client (securely) by HTTPS. Once the data is received on the client side it is no longer encrypted. RavenDB does not provide encryption on the client side.
</Admonition>

<Admonition type="danger" title="Indexing transaction size" id="indexing-transaction-size" href="#indexing-transaction-size">
<Admonition type="warning" title="Indexing transaction size" id="indexing-transaction-size" href="#indexing-transaction-size">
Indexing is most efficient when it is performed in the largest transactions possible. However, using encryption can be very memory intensive, and if memory
runs out before the transaction completes, the entire transaction will fail. To avoid this, you can limit the size of indexing batches in encrypted
databases using [Indexing.Encrypted.TransactionSizeLimitInMb](../../../server/configuration/indexing-configuration.mdx#indexingencryptedtransactionsizelimitinmb).
Expand Down Expand Up @@ -76,7 +76,8 @@ In **Linux**, it is the admin's responsibility to configure higher limits manual
</CodeBlock>
</TabItem>

To figure out what the new limit should be, look at the exception thrown by RavenDB, which includes this size.
To figure out what the new limit should be, look at the exception thrown by RavenDB, which includes this size.
For persistent and service-level configuration, see [Linux: Setting memlock when using encrypted database](../../kb/linux-setting-memlock.mdx).

## What about Encryption in Transit?

Expand Down
62 changes: 54 additions & 8 deletions versioned_docs/version-7.0/server/kb/linux-setting-memlock.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -10,20 +10,66 @@ import TabItem from '@theme/TabItem';
import CodeBlock from '@theme/CodeBlock';
import LanguageSwitcher from "@site/src/components/LanguageSwitcher";
import LanguageContent from "@site/src/components/LanguageContent";
import Panel from "@site/src/components/Panel";
import ContentFrame from "@site/src/components/ContentFrame";

# Linux: Setting memlock when using encrypted database
Encrypted database uses extensively sodium library which requires high values of locked memory limits.
`memlock` refers to memory that will not be paged out, and it's limit can be viewed usign `ulimit -l`.
The modification of `memlock` limit settings can be achieved by in running session with `prlimit`:

Example, for 1MB limit:
<Admonition type="note" title="">

* When [encryption at rest](../security/encryption/encryption-at-rest.mdx) is enabled, RavenDB uses the libsodium library to lock certain memory regions in order to prevent plaintext data from being swapped to disk.
Linux imposes a per-process limit on how much memory the process can lock (`memlock`).
If this limit is too low, RavenDB cannot open the encrypted database and throws an `InsufficientMemoryException`.

* The exception message includes the exact `prlimit` command needed to raise the limit, with the required size in bytes already filled in.

* In this article:
* [Raising the limit for a running process](linux-setting-memlock.mdx#raising-the-limit-for-a-running-process)
* [Setting the limit persistently](linux-setting-memlock.mdx#setting-the-limit-persistently)

</Admonition>

<Panel heading="Raising the limit for a running process">

Run the command shown in the exception message, replacing `<pid>` with the RavenDB process ID
and `<size-in-bytes>` with the value from the error:

```bash
sudo prlimit --pid <pid> --memlock=<size-in-bytes>
```
prlimit -p pid --memlock 1MB:1MB

<br />

This takes effect immediately but does not persist across restarts.

</Panel>

<Panel heading="Setting the limit persistently">

Add the following to `/etc/security/limits.conf`:

```text
* soft memlock unlimited
* hard memlock unlimited
```

Persistant settings can be achieved by adding to `/etc/security/limits.conf` the following:
<br />

Log out and back in (or restart the service) for the change to take effect.
Verify the new limit with:

```bash
ulimit -l
```
* soft memlock 1000
* hard memlock 1000

<Admonition type="note" title="">
If RavenDB runs as a `systemd` service, set the memory lock limit in the service unit file instead of `limits.conf`:

```ini
[Service]
LimitMEMLOCK=infinity
```

</Admonition>

</Panel>
Original file line number Diff line number Diff line change
Expand Up @@ -512,7 +512,7 @@ In **Linux**, it is the admin's responsibility to configure higher limits manual
</CodeBlock>
</TabItem>

To figure out what the new limit should be, look at the exception thrown by RavenDB, which includes this size.

To figure out what the new limit should be, look at the exception thrown by RavenDB, which includes this size.
For persistent and service-level configuration, see [Linux: Setting memlock when using encrypted database](../kb/linux-setting-memlock.mdx).


Original file line number Diff line number Diff line change
Expand Up @@ -34,14 +34,14 @@ Once a request is made, RavenDB will start a transaction (either read or write)
request, and when the transaction is finished, modified pages are encrypted and written back to the datafile. The data is
[locked](../../../server/security/encryption/encryption-at-rest.mdx#locking-memory) during the transaction by default, and it is zeroed when the transaction completes.

<Admonition type="danger" title="Important things to be aware of:" id="important-things-to-be-aware-of" href="#important-things-to-be-aware-of">
<Admonition type="warning" title="Important things to be aware of:" id="important-things-to-be-aware-of" href="#important-things-to-be-aware-of">
1. RavenDB makes sure that **no data is written to disk as plain text**. It will always be encrypted.
2. Indexed fields (the actual terms and values being indexed) will reside in memory as plain text.
3. Data of the current transaction will reside in memory as plain text and only for the duration of the transaction. When the transaction ends, the used memory is safely zeroed.
4. Loading documents from the database (using the Studio, the Client API, REST API) means that they will be decrypted to plain text on the server and then sent to the client (securely) by HTTPS. Once the data is received on the client side it is no longer encrypted. RavenDB does not provide encryption on the client side.
</Admonition>

<Admonition type="danger" title="Indexing transaction size" id="indexing-transaction-size" href="#indexing-transaction-size">
<Admonition type="warning" title="Indexing transaction size" id="indexing-transaction-size" href="#indexing-transaction-size">
Indexing is most efficient when it is performed in the largest transactions possible. However, using encryption can be very memory intensive, and if memory
runs out before the transaction completes, the entire transaction will fail. To avoid this, you can limit the size of indexing batches in encrypted
databases using [Indexing.Encrypted.TransactionSizeLimitInMb](../../../server/configuration/indexing-configuration.mdx#indexingencryptedtransactionsizelimitinmb).
Expand Down Expand Up @@ -76,7 +76,8 @@ In **Linux**, it is the admin's responsibility to configure higher limits manual
</CodeBlock>
</TabItem>

To figure out what the new limit should be, look at the exception thrown by RavenDB, which includes this size.
To figure out what the new limit should be, look at the exception thrown by RavenDB, which includes this size.
For persistent and service-level configuration, see [Linux: Setting memlock when using encrypted database](../../kb/linux-setting-memlock.mdx).

## What about Encryption in Transit?

Expand Down
Loading
Loading