{"id":301,"date":"2022-09-28T19:52:28","date_gmt":"2022-09-29T03:52:28","guid":{"rendered":"https:\/\/stasosphere.com\/entrepreneur-being\/?p=301"},"modified":"2025-07-22T21:12:07","modified_gmt":"2025-07-23T05:12:07","slug":"mmap-memory-leak-investigation","status":"publish","type":"post","link":"https:\/\/stasosphere.com\/entrepreneur-being\/301-mmap-memory-leak-investigation\/","title":{"rendered":"A Deep Investigation into MMAP Not Leaking Memory"},"content":{"rendered":"\n

This write up is going to demonstrate that while it looks that a mmap<\/code>‘ed file IO looks like it’s leaking memory it actually is not.<\/p>\n

HuggingFace’s datasets<\/a> project uses MMAP to make datasets available to multiple processes in an efficient way. This is very important since typically a machine learning training program will use a Dataloader which may use multiple workers, or alternatively the same dataset is simply accessed by multiple processes.<\/p>\n

An issue was posted<\/a> that suggested that a datasets<\/code>-based program leaks memory with each iteration. This triggered an extensive research into understanding that MMAP doesn’t leak memory and bringing a lot of deepeer understanding of the different components used under the hood of datasets<\/code>.<\/p>\n

If you’d like to gain a deeper understanding into why and how please read on.

<\/p>\n

Emulating a computer with just 1GB of memory<\/h2>\n

Since we don’t want to crash our computer while debugging memory issues we are going to emulate a computer with just 1GB of memory and no swap memory. Unless such computer has a protection from programs using more memory than the computer has most of the time such computers start thrashing<\/a> and eventually crash.<\/p>\n

To accomplish that we are going to start a cgroups-controlled shell which will kill any program started from that shell and which consumes more than 1GB of memory (and give it no swap memory either):<\/p>\n

$ systemd-run --user --scope -p<\/span> MemoryHigh=1<\/span>G -p<\/span> MemoryMax=1<\/span>G -p<\/span> MemorySwapMax=0<\/span>G --setenv=\"MEMLIMIT=1GB\"<\/span> bash\n<\/pre>\n
I’m setting MEMLIMIT=1GB<\/code> env variable so that at any moment I can check if I’m in the right shell by printing:<\/p>\n
$ echo<\/span> $MEMLIMIT<\/span>\n1GB<\/pre>\n
Let’s validate that this shell allows a program to allocate under 1GB of RSS RAM, but will kill it if it tries to allocate more than that:<\/p>\n
# 7<\/span> * 128<\/span>M chars\n$ <\/span>python -c \"import sys, os, psutil; a='a'*7*2**27; print(f'{psutil.Process(os.getpid()).memory_info().rss >> 20}MB');\"<\/span>\n908<\/span>MB\n\n# 8<\/span> * 128<\/span>M chars\n$ <\/span>python -c \"import sys, os, psutil; a='a'*8*2**27; print(f'{psutil.Process(os.getpid()).memory_info().rss >> 20}MB');\"<\/span>\nKilled<\/span>\n<\/pre>\n
So we can see that < ~1GB works, but beyond an allocation that asks for more than 1GB of resident memory gets killed.<\/p>\n
In the rest of this write up let’s use shell A, which is unlimited (or rather limited to an actual available memory on your computer) and shell B, where a program started from it can only allocate 1GB of resident memory.<\/p>\n
Sidenote: Linux memory management and reporting is super-complicated and one could probably easily write a whole book about it. Resident Set Size (RSS) is typically the easiest to use to measure the approximate actual memory usage by the program. It doesn’t tell you the whole truth, but most of the time it’s good enough to detect memory leaks. Therefore in this write up this is the metric we are going to use.<\/p>\n
Simple IO debug program<\/h2>\n
Now let’s write a simple debug program that will create a file with a few very large lines, and then it’ll read them sequentially using a normal IO, but if we set --mmap<\/code> it’ll switch to memory mmaped API via the mmap<\/code> module.<\/p>\n
Additionally, if --accumulate<\/code> flag is passed the program will accumulate the lines it reads into a single string.<\/p>\n
$ cat python mmap-no-leak-debug.py\nimport<\/span> gc\nimport<\/span> mmap\nimport<\/span> os\nimport<\/span> psutil\nimport<\/span> sys\n\nPATH = \".\/tmp.txt\"<\/span>\n# create a large data file with a few long lines<\/span>\nif<\/span> not<\/span> os.path.exists(PATH):\n    with<\/span> open(PATH, \"w\"<\/span>) as<\/span> fh:\n        s = 'a'<\/span>* 2<\/span>**27<\/span> + \"\\n\"<\/span> # 128MB<\/span>\n        # write ~2GB file<\/span>\n        for<\/span> i in<\/span> range(16<\/span>):\n            fh.write(s)\n\nproc = psutil.Process(os.getpid())\ndef<\/span> mem_read<\/span>()<\/span>:<\/span>\n    gc.collect()\n    return<\/span> proc.memory_info().rss \/ 2<\/span>**20<\/span>\n\nprint(f\"{'idx':>4} {'RSS':>10}   {'\u0394 RSS':>12}   {'\u0394 accumulated':>10}\"<\/span>)\n\ncontent = ''<\/span>\nmem_after = mem_before_acc = mem_after_acc = mem_before = proc.memory_info().rss \/ 2<\/span>**20<\/span>\nprint(f\"{0:4d} {mem_after:10.2f}MB {mem_after - 0:10.2f}MB {0:10.2f}MB\"<\/span>)\n\nmmap_mode = True<\/span> if<\/span> \"--mmap\"<\/span> in<\/span> sys.argv else<\/span> False<\/span>\n\nwith<\/span> open(PATH, \"r\"<\/span>) as<\/span> fh:\n\n    if<\/span> mmap_mode:\n        mm = mmap.mmap(fh.fileno(), 0<\/span>, access=mmap.ACCESS_READ)\n\n    idx = 0<\/span>\n    while<\/span> True<\/span>:\n        idx += 1<\/span>\n        mem_before = mem_read()\n        line = mm.readline() if<\/span> mmap_mode else<\/span> fh.readline()\n        if<\/span> not<\/span> line:\n            break<\/span>\n\n        if<\/span> \"--accumulate\"<\/span> in<\/span> sys.argv:\n            mem_before_acc = mem_read()\n            content += str(line)\n            mem_after_acc = mem_read()\n\n        mem_after = mem_read()\n\n        print(f\"{idx:4d} {mem_after:10.2f}MB {mem_after - mem_before:10.2f}MB {mem_after_acc - mem_before_acc:10.2f}MB\"<\/span>)<\/pre>\n
The four output columns are:<\/p>\n
 idx<\/span>        RSS          \u0394 RSS   \u0394 accumulated<\/pre>\n
    \n
  1. the line number (starting from 1)<\/li>\n
  2. the total RSS reported at the end of each iteration<\/li>\n
  3. the RSS delta of each iteration<\/li>\n
  4. the accumulated buffer delta<\/li>\n<\/ol>\n
    And as you can see we force Python’s garbage collection via gc.collect()<\/code> before taking RSS (Resident Set Size) measurements. This is a very crucial step when debugging memory usages and leaks in particular and especially if you delete some objects and want to make sure that memory is actually freed as Python’s garbage collection mechanism is not immediate.<\/p>\n
    Normal IO diagnostics<\/h2>\n
    First, let’s run normal IO without accumulating any strings and simply discarding those.<\/p>\n
    shell A $ python mmap-no-leak-debug.py\n idx        RSS          \u0394 RSS   \u0394 accumulated\n   0<\/span>      12.37<\/span>MB      12.37<\/span>MB       0.00<\/span>MB\n   1<\/span>     269.66<\/span>MB     257.29<\/span>MB       0.00<\/span>MB\n   2<\/span>     269.68<\/span>MB       0.02<\/span>MB       0.00<\/span>MB\n   3<\/span>     269.68<\/span>MB       0.00<\/span>MB       0.00<\/span>MB\n   4<\/span>     269.69<\/span>MB       0.01<\/span>MB       0.00<\/span>MB\n   5<\/span>     269.69<\/span>MB       0.00<\/span>MB       0.00<\/span>MB\n   6<\/span>     269.70<\/span>MB       0.01<\/span>MB       0.00<\/span>MB\n   7<\/span>     269.70<\/span>MB       0.00<\/span>MB       0.00<\/span>MB\n   8<\/span>     269.70<\/span>MB       0.01<\/span>MB       0.00<\/span>MB\n   9<\/span>     269.70<\/span>MB       0.00<\/span>MB       0.00<\/span>MB\n  10<\/span>     269.71<\/span>MB       0.01<\/span>MB       0.00<\/span>MB\n  11<\/span>     269.71<\/span>MB       0.00<\/span>MB       0.00<\/span>MB\n  12<\/span>     269.71<\/span>MB       0.00<\/span>MB       0.00<\/span>MB\n  13<\/span>     269.71<\/span>MB       0.00<\/span>MB       0.00<\/span>MB\n  14<\/span>     269.71<\/span>MB       0.00<\/span>MB       0.00<\/span>MB\n  15<\/span>     269.71<\/span>MB       0.00<\/span>MB       0.00<\/span>MB\n  16<\/span>     145.96<\/span>MB    -123.75<\/span>MB       0.00<\/span>MB<\/pre>\n
    We read in a loop a 128MB line and discard it.<\/p>\n
    We can see the memory is very low and steady, with some fluctuations when Python decided to release some memory. The program allocates more than 128MB due to a new line character in the string – this is a peculiar Python behavior.<\/p>\n
    The bottom line is that the program doesn’t appear to be leaking any memory.<\/p>\n
    MMAP’ed IO diagnostics<\/h2>\n
    Now let’s do the exact same operation but this time using mmap<\/code>‘s IO:<\/p>\n
    shell A $ python mmap-no-leak-debug.py --mmap\nidx        RSS          \u0394 RSS   \u0394 accumulated\n   0<\/span>      12.39<\/span>MB      12.39<\/span>MB       0.00<\/span>MB\n   1<\/span>     268.25<\/span>MB     255.87<\/span>MB       0.00<\/span>MB\n   2<\/span>     396.47<\/span>MB     128.22<\/span>MB       0.00<\/span>MB\n   3<\/span>     524.47<\/span>MB     128.00<\/span>MB       0.00<\/span>MB\n   4<\/span>     652.47<\/span>MB     128.00<\/span>MB       0.00<\/span>MB\n   5<\/span>     780.47<\/span>MB     128.00<\/span>MB       0.00<\/span>MB\n   6<\/span>     908.47<\/span>MB     128.00<\/span>MB       0.00<\/span>MB\n   7<\/span>    1036.47<\/span>MB     128.00<\/span>MB       0.00<\/span>MB\n   8<\/span>    1164.47<\/span>MB     128.00<\/span>MB       0.00<\/span>MB\n   9<\/span>    1292.47<\/span>MB     128.00<\/span>MB       0.00<\/span>MB\n  10<\/span>    1420.47<\/span>MB     128.00<\/span>MB       0.00<\/span>MB\n  11<\/span>    1548.47<\/span>MB     128.00<\/span>MB       0.00<\/span>MB\n  12<\/span>    1676.47<\/span>MB     128.00<\/span>MB       0.00<\/span>MB\n  13<\/span>    1804.47<\/span>MB     128.00<\/span>MB       0.00<\/span>MB\n  14<\/span>    1932.47<\/span>MB     128.00<\/span>MB       0.00<\/span>MB\n  15<\/span>    2060.47<\/span>MB     128.00<\/span>MB       0.00<\/span>MB\n  16<\/span>    2188.47<\/span>MB     128.00<\/span>MB       0.00<\/span>MB<\/pre>\n
    Whoah! It looks like there is a major leak here. On each iteration the program keeps on growing by 128MB despite us discarding the read data. What’s going on?<\/p>\n
    The theoretical explanation is simple – MMAP was designed to make IO faster and shared by multiple processes – so if there is a lot of available RAM, the MMAP API will use as much of it as it can and in order to speed things up it won’t normally release it back to the OS. For example, if you have two programs reading the same sections from the same MMAP’ed file only the first program will incur the delay of copying the data from disc to RAM. The other program will read it directly from RAM. Since MMAP doesn’t know which sections will be accessed next it simply keeps everything it read in the memory if there is enough of it.<\/p>\n
    But you’d say this is very bad and that’s a terrible design. But wait, it only keeps it in memory if nobody else wants the memory, and it immediately releases that unused memory back to the operating system as soon as such demand arises.<\/p>\n
    Proof that there is no leak<\/h2>\n
    To show that the memory does get released as soon as it’s needed let’s re-run this same program in shell B, where only 1GB of memory is allowed to be allocated.<\/p>\n
    shell B $ systemd-run --user --scope -p MemoryHigh=1<\/span>G -p MemoryMax=1<\/span>G -p MemorySwapMax=0<\/span>G --setenv=\"MEMLIMIT=1GB\"<\/span> bash\nshell B $ python mmap-no-leak-debug.py --mmap\n idx        RSS          \u0394 RSS   \u0394 accumulated\n   0<\/span>      12.48<\/span>MB      12.48<\/span>MB       0.00<\/span>MB\n   1<\/span>     268.51<\/span>MB     256.03<\/span>MB       0.00<\/span>MB\n   2<\/span>     396.73<\/span>MB     128.22<\/span>MB       0.00<\/span>MB\n   3<\/span>     524.73<\/span>MB     128.00<\/span>MB       0.00<\/span>MB\n   4<\/span>     652.73<\/span>MB     128.00<\/span>MB       0.00<\/span>MB\n   5<\/span>     780.73<\/span>MB     128.00<\/span>MB       0.00<\/span>MB\n   6<\/span>     908.73<\/span>MB     128.00<\/span>MB       0.00<\/span>MB\n   7<\/span>    1036.73<\/span>MB     128.00<\/span>MB       0.00<\/span>MB\n   8<\/span>    1164.73<\/span>MB     128.00<\/span>MB       0.00<\/span>MB\n   9<\/span>    1292.73<\/span>MB     128.00<\/span>MB       0.00<\/span>MB\n  10<\/span>    1420.73<\/span>MB     128.00<\/span>MB       0.00<\/span>MB\n  11<\/span>    1548.73<\/span>MB     128.00<\/span>MB       0.00<\/span>MB\n  12<\/span>    1676.73<\/span>MB     128.00<\/span>MB       0.00<\/span>MB\n  13<\/span>    1804.73<\/span>MB     128.00<\/span>MB       0.00<\/span>MB\n  14<\/span>    1932.73<\/span>MB     128.00<\/span>MB       0.00<\/span>MB\n  15<\/span>    2060.73<\/span>MB     128.00<\/span>MB       0.00<\/span>MB\n  16<\/span>    2188.69<\/span>MB     127.95<\/span>MB       0.00<\/span>MB<\/pre>\n
    A surprise, it appears that the program managed to allocate >2GB of memory when we double checked that it should have been killed as soon as it reached 1GB RSS since we limited the shell to allow only <1GB memory allocation!<\/p>\n
    We will understand better shortly what’s going on, but it’s clear that cgroups that controls the memory usage is aware that while it accounts that MMAP’ed memory to the RSS counter of the program it’s aware that the program itself isn’t using most of this memory!<\/p>\n
    Interim observation: we can’t rely on RSS memory stats to diagnose memory leaks when MMAP is used.<\/p>\n
    Let’s create memory pressure<\/h2>\n
    This is where our --accumulate<\/code> flag comes in. It’s going to help us to see that RSS is “misreporting” the actual memory used by the program.<\/p>\n
    First we run it with normal IO:<\/p>\n
    shell A $ python mmap-no-leak-debug.py --accumulate\n idx        RSS          \u0394 RSS   \u0394 accumulated\n   0<\/span>      12.30<\/span>MB      12.30<\/span>MB       0.00<\/span>MB\n   1<\/span>     269.60<\/span>MB     257.29<\/span>MB       0.00<\/span>MB\n   2<\/span>     525.49<\/span>MB     255.89<\/span>MB     127.93<\/span>MB\n   3<\/span>     653.49<\/span>MB     128.00<\/span>MB     127.87<\/span>MB\n   4<\/span>     781.50<\/span>MB     128.01<\/span>MB     127.87<\/span>MB\n   5<\/span>     909.50<\/span>MB     128.00<\/span>MB     127.87<\/span>MB\n   6<\/span>    1037.51<\/span>MB     128.01<\/span>MB     127.87<\/span>MB\n   7<\/span>    1165.51<\/span>MB     128.00<\/span>MB     127.87<\/span>MB\n   8<\/span>    1293.52<\/span>MB     128.01<\/span>MB     127.87<\/span>MB\n   9<\/span>    1421.52<\/span>MB     128.00<\/span>MB     127.87<\/span>MB\n  10<\/span>    1549.53<\/span>MB     128.01<\/span>MB     127.87<\/span>MB\n  11<\/span>    1677.53<\/span>MB     128.00<\/span>MB     127.87<\/span>MB\n  12<\/span>    1805.53<\/span>MB     128.00<\/span>MB     127.87<\/span>MB\n  13<\/span>    1933.53<\/span>MB     128.00<\/span>MB     127.87<\/span>MB\n  14<\/span>    2061.53<\/span>MB     128.00<\/span>MB     127.87<\/span>MB\n  15<\/span>    2189.53<\/span>MB     128.00<\/span>MB     127.87<\/span>MB\n  16<\/span>    2193.78<\/span>MB       4.25<\/span>MB     127.87<\/span>MB<\/pre>\n
    where RSS reports correctly 128*16 ~= 2048<\/code>MB and then some for the other bits of the program, but the ballpark matches.<\/p>\n
    Now let’s activate MMAP and re-run:<\/p>\n
    shell A $ python mmap-no-leak-debug.py --mmap --accumulate\n idx        RSS          \u0394 RSS   \u0394 accumulated\n   0<\/span>      12.37<\/span>MB      12.37<\/span>MB       0.00<\/span>MB\n   1<\/span>     396.39<\/span>MB     384.02<\/span>MB     128.13<\/span>MB\n   2<\/span>     652.48<\/span>MB     256.09<\/span>MB     128.00<\/span>MB\n   3<\/span>     908.48<\/span>MB     256.00<\/span>MB     128.00<\/span>MB\n   4<\/span>    1164.48<\/span>MB     256.00<\/span>MB     128.00<\/span>MB\n   5<\/span>    1420.48<\/span>MB     256.00<\/span>MB     128.00<\/span>MB\n   6<\/span>    1676.48<\/span>MB     256.00<\/span>MB     128.00<\/span>MB\n   7<\/span>    1932.48<\/span>MB     256.00<\/span>MB     128.00<\/span>MB\n   8<\/span>    2188.48<\/span>MB     256.00<\/span>MB     128.00<\/span>MB\n   9<\/span>    2444.48<\/span>MB     256.00<\/span>MB     128.00<\/span>MB\n  10<\/span>    2700.48<\/span>MB     256.00<\/span>MB     128.00<\/span>MB\n  11<\/span>    2956.48<\/span>MB     256.00<\/span>MB     128.00<\/span>MB\n  12<\/span>    3212.48<\/span>MB     256.00<\/span>MB     128.00<\/span>MB\n  13<\/span>    3468.48<\/span>MB     256.00<\/span>MB     128.00<\/span>MB\n  14<\/span>    3724.48<\/span>MB     256.00<\/span>MB     128.00<\/span>MB\n  15<\/span>    3980.48<\/span>MB     256.00<\/span>MB     128.00<\/span>MB\n  16<\/span>    4236.46<\/span>MB     255.98<\/span>MB     128.00<\/span>MB<\/pre>\n
    Here we can see that RSS reports 2x memory than it actually uses.<\/p>\n
    And now let’s create pressure using our 1GB-limited shell B and use normal IO with accumulation:<\/p>\n
    shell B $ systemd-run --user --scope -p MemoryHigh=1<\/span>G -p MemoryMax=1<\/span>G -p MemorySwapMax=0<\/span>G --setenv=\"MEMLIMIT=1GB\"<\/span> bash\nshell B $ python mmap-no-leak-debug.py --accumulate\n idx        RSS          \u0394 RSS   \u0394 accumulated\n   0<\/span>      12.38<\/span>MB      12.38<\/span>MB       0.00<\/span>MB\n   1<\/span>     269.41<\/span>MB     257.04<\/span>MB       0.00<\/span>MB\n   2<\/span>     525.55<\/span>MB     256.14<\/span>MB     127.93<\/span>MB\n   3<\/span>     653.55<\/span>MB     128.00<\/span>MB     127.87<\/span>MB\n   4<\/span>     781.56<\/span>MB     128.01<\/span>MB     127.87<\/span>MB\n   5<\/span>     909.56<\/span>MB     128.00<\/span>MB     127.87<\/span>MB\nKilled<\/pre>\n
    As you can easily see the program gets killed once it reaches 1GB of RSS. It managed to perform 5 iterations, thus on iteration 6 it tries to accumulate 6*128=768<\/code> plus the current readline<\/code> read of 128MB, plus the memory used by the rest of the program, it crosses 1GB and gets killed before finishing iteration 6.<\/p>\n
    Also it might be useful to compare with the same run with shell A. You can see that RSS of the shell B run is quite different from shell A. The reported RSS doesn’t grow as fast.<\/p>\n
    Now let’s run the MMAPed version:<\/p>\n
    shell B $ systemd-run --user --scope -p MemoryHigh=1<\/span>G -p MemoryMax=1<\/span>G -p MemorySwapMax=0<\/span>G --setenv=\"MEMLIMIT=1GB\"<\/span> bash\nshell B $ python mmap-no-leak-debug.py --mmap --accumulate\n idx        RSS          \u0394 RSS   \u0394 accumulated\n   0<\/span>      12.51<\/span>MB      12.51<\/span>MB       0.00<\/span>MB\n   1<\/span>     396.52<\/span>MB     384.00<\/span>MB     128.13<\/span>MB\n   2<\/span>     652.60<\/span>MB     256.08<\/span>MB     128.00<\/span>MB\n   3<\/span>     908.60<\/span>MB     256.00<\/span>MB     128.00<\/span>MB\n   4<\/span>    1164.60<\/span>MB     256.00<\/span>MB     128.00<\/span>MB\n   5<\/span>    1420.60<\/span>MB     256.00<\/span>MB     128.00<\/span>MB\nKilled\n<\/pre>\n
    You can see it gets killed in the exactly same iteration as when it was run without MMAP.<\/p>\n
    You can see that while the RSS numbers are bigger than that of the normal IO run, the program gets killed in the exact same iteration. which tells us the actual memory usage with normal IO and mmap’ed IO is either very similar or very likely exactly the same.<\/p>\n
    What about PyArrow?<\/h2>\n
    Originally this whole research started from this Issue<\/a> in the datasets<\/code><\/a> repo. It looked like a dataset loaded via pyarrow<\/code> leaked on every iteration.<\/p>\n
    Quentin Lhoest<\/a> reduced it to a simple pyarrow<\/code> program<\/a><\/p>\n
    $ cat mmap-no-leak-debug<\/span>-pyarrow.py\nimport<\/span> psutil\nimport<\/span> os<\/span>\nimport<\/span> gc\nimport<\/span> pyarrow as pa\n\nARROW_PATH = \"tmp.arrow\"<\/span>\n\nif<\/span> not<\/span> os<\/span>.path.exists(ARROW_PATH):\n    arr = pa.array([b\"a\"<\/span> * (200<\/span> * 1024<\/span>)] * 1000<\/span>)  # ~200<\/span>MB\n    table<\/span> = pa.table<\/span>({\"a\"<\/span>: arr})\n\n    with open(ARROW_PATH, \"wb\"<\/span>) as f<\/span>:\n        writer = pa.RecordBatchStreamWriter(f, schema=table<\/span>.schema)\n        writer.write_table(table<\/span>)\n        writer.close()\n\ndef memory_mapped_arrow_table_from_file(filename<\/span>: str) -> pa.Table<\/span>:\n    memory_mapped_stream = pa.memory_map(filename)\n    opened_stream = pa.ipc.open_stream(memory_mapped_stream)\n    pa_table = opened_stream.read_all()\n    return<\/span> pa_table\n\n\ntable<\/span> = memory_mapped_arrow_table_from_file(ARROW_PATH)\narr = table<\/span>[0<\/span>]\n\nprint<\/span>(f\"{'idx':>8} {'RSS':>10} {'\u0394 RSS':>15}\"<\/span>)\n\nmem_before = psutil.Process(os<\/span>.getpid()).memory_info().rss \/ (1024<\/span> * 1024<\/span>)\nfor<\/span> idx, x in<\/span> enumerate(arr):\n    if<\/span> idx % 100<\/span> == 0<\/span>:\n        gc.collect()\n        mem_after = psutil.Process(os<\/span>.getpid()).memory_info().rss \/ (1024<\/span> * 1024<\/span>)\n        print<\/span>(f\"{idx:4d}  {mem_after:12.4f}MB {mem_after - mem_before:12.4f}MB\"<\/span>)\n<\/pre>\n
    which when run produced the familiar leak-like pattern:<\/p>\n
    $ python mmap-no-leak-debug-pyarrow.py\n     idx        RSS           \u0394 RSS\n   0<\/span>       51.3164<\/span>MB       2.5430<\/span>MB\n 100<\/span>       69.9805<\/span>MB      21.2070<\/span>MB\n 200<\/span>       90.6055<\/span>MB      41.8320<\/span>MB\n 300<\/span>      107.1055<\/span>MB      58.3320<\/span>MB\n 400<\/span>      127.7305<\/span>MB      78.9570<\/span>MB\n 500<\/span>      148.3555<\/span>MB      99.5820<\/span>MB\n 600<\/span>      164.8555<\/span>MB     116.0820<\/span>MB\n 700<\/span>      185.4805<\/span>MB     136.7070<\/span>MB\n 800<\/span>      206.1055<\/span>MB     157.3320<\/span>MB\n 900<\/span>      226.7305<\/span>MB     177.9570<\/span>MB\n<\/pre>\n
    But if we run it from a shell that is only allowed 100MB of allocated memory:<\/p>\n
    $ systemd-run --user --scope -p MemoryHigh=0.1<\/span>G -p MemoryMax=0.1<\/span>G -p MemorySwapMax=0<\/span>G --setenv=\"MEMLIMIT=0.1GB\"<\/span> bash\n$ python mmap-no-leak-debug-pyarrow.py\n     idx        RSS           \u0394 RSS\n   0<\/span>       51.2852<\/span>MB       2.4609<\/span>MB\n 100<\/span>       70.4102<\/span>MB      21.5859<\/span>MB\n 200<\/span>       86.9102<\/span>MB      38.0859<\/span>MB\n 300<\/span>      107.5352<\/span>MB      58.7109<\/span>MB\n 400<\/span>      128.1602<\/span>MB      79.3359<\/span>MB\n 500<\/span>      148.7852<\/span>MB      99.9609<\/span>MB\n 600<\/span>      165.2852<\/span>MB     116.4609<\/span>MB\n 700<\/span>      185.9102<\/span>MB     137.0859<\/span>MB\n 800<\/span>      206.5352<\/span>MB     157.7109<\/span>MB\n 900<\/span>      227.1602<\/span>MB     178.3359<\/span>MB<\/pre>\n
    So it reports it allocated ~200MB of RSS, yet it runs just fine without getting killed.<\/p>\n
    There is no leak here.<\/p>\n
    What about HuggingFace datasets?<\/h2>\n
    In another Issue<\/a> a very similar datasets-iterator-is-leaking report was submitted.<\/p>\n
    So let’s use a similar datasets<\/code> reproduction example here but we will use a larger dataset.<\/p>\n
    $ cat mmap-no-leak-debug-datasets.py\nfrom<\/span> datasets import<\/span> load_dataset\nimport<\/span> gc\nimport<\/span> os\nimport<\/span> psutil\nimport<\/span> sys\n\nkeep_in_memory = True<\/span> if<\/span> \"in-mem\"<\/span> in<\/span> sys.argv else<\/span> False<\/span>\n\nproc = psutil.Process(os.getpid())\ndef<\/span> mem_read<\/span>()<\/span>:<\/span>\n    gc.collect()\n    return<\/span> proc.memory_info().rss \/ 2<\/span>**20<\/span>\n\ndataset = load_dataset(\"wmt19\"<\/span>, 'cs-en'<\/span>, keep_in_memory=keep_in_memory, streaming=keep_in_memory)['train'<\/span>]\nprint(f\"Dataset len={len(dataset)}\"<\/span>)\n\nprint(f\"{'idx':>8} {'RSS':>10} {'\u0394 RSS':>15}\"<\/span>)\nstep = 1<\/span>_000_000\nmem_start = 0<\/span>\nfor<\/span> idx, i in<\/span> enumerate(range(0<\/span>, len(dataset), step)):\n    if<\/span> idx == 4<\/span>: # skip the first few iterations while things get set up<\/span>\n        mem_start = mem_read()\n    mem_before = mem_read()\n    x = dataset[i:i+step]\n    mem_after = mem_read()\n    print(f\"{idx:8d} {mem_after:12.4f}MB {mem_after - mem_before:12.4f}MB \"<\/span>)\nmem_end = mem_read()\n\nprint(f\"Total diff: {mem_end - mem_start:12.4f}MB \"<\/span>)<\/pre>\n
    Let’s run it in a normal shell first:<\/p>\n
    $ python mmap-no-leak-debug-datasets.py\nDataset len=7270695\n     idx        RSS           \u0394 RSS\n       0 <\/span>    775.7773MB     609.9805MB\n       1 <\/span>    849.6016MB      73.8242MB\n       2 <\/span>    876.1445MB      26.5430MB\n       3 <\/span>    941.3477MB      65.2031MB\n       4 <\/span>    984.9570MB      43.6094MB\n       5 <\/span>   1053.6445MB      68.6875MB\n       6 <\/span>   1164.2852MB     110.6406MB\n       7 <\/span>   1252.5312MB      88.2461MB\n       8 <\/span>   1368.6523MB     116.1211MB\n       9 <\/span>   1445.7266MB      77.0742MB\n      10 <\/span>   1564.5195MB     118.7930MB\n      11 <\/span>   1678.7500MB     114.2305MB\n      12 <\/span>   1729.9844MB      51.2344MB\n      13 <\/span>   1866.1953MB     136.2109MB\nTotal diff:    1700.3984MB<\/pre>\n
    You can see the mid-column of total RSS memory keeps on growing in MBs. The last column is by how much it has grown during a single iteration of the script (0.5M items).<\/p>\n
    And now let’s run in a 1GB limited shell:<\/p>\n
    $ systemd-run --user --scope -p MemoryHigh=1G -p MemoryMax=1G -p MemorySwapMax=0G --setenv=\"MEMLIMIT=1GB\" bash\n$ python mmap-no-leak-debug-datasets.py\nDataset len=7270695\n     idx        RSS           \u0394 RSS\n       0 <\/span>    775.8516MB     610.1797MB\n       1 <\/span>    849.5820MB      73.7305MB\n       2 <\/span>    876.1328MB      26.5508MB\n       3 <\/span>    941.3281MB      65.1953MB\n       4 <\/span>    984.9375MB      43.6094MB\n       5 <\/span>   1053.6328MB      68.6953MB\n       6 <\/span>   1164.0273MB     110.3945MB\n       7 <\/span>   1252.5273MB      88.5000MB\n       8 <\/span>   1368.3906MB     115.8633MB\n       9 <\/span>   1445.7188MB      77.3281MB\n      10 <\/span>   1564.2656MB     118.5469MB\n      11 <\/span>   1678.7383MB     114.4727MB\n      12 <\/span>   1729.7227MB      50.9844MB\n      13 <\/span>   1866.1875MB     136.4648MB\nTotal diff:    1700.5156MB<\/pre>\n
    No problem at all.<\/p>\n
    So we now know there is no leak there and it’s just the OS includes in RSS memory that will be released as soon as it’s needed.<\/p>\n
    How to debug real memory leaks while using MMAP<\/h2>\n
    So how does one debug an actual memory that might be elsewhere in the code while using MMAP.<\/p>\n
    Well, you have to disable MMAP for the duration of your debug session and then re-enabled it back when you want high performance.<\/p>\n
    As you have seen at the beginning of this article switching from mmap<\/code> to normal IO is very simple to do.<\/p>\n
    In the case of datasets<\/code> you’d turn MMAP functionality off with keep_in_memory=True<\/code> as in:<\/p>\n
    dataset<\/span> = load_dataset(\"wmt19\"<\/span>, 'cs-en'<\/span>, keep_in_memory=True<\/span>, streaming=False<\/span>)['train'<\/span>]\n<\/code><\/pre>\n
    This loads the dataset in RAM, and now you should be able to debug your potential leak.<\/p>\n
    Let’s test after modifying our last program:<\/p>\n
    - dataset = load_dataset(\"wmt19\"<\/span>, 'cs<\/span>-en', keep_in_memory=False<\/span>, streaming=False<\/span>)['train<\/span>']\n+ dataset = load_dataset(\"wmt19\"<\/span>, 'cs<\/span>-en', keep_in_memory=True<\/span>, streaming=False<\/span>)['train<\/span>']<\/pre>\n
    Now in the normal unlimited shell we run:<\/p>\n
    $ python mmap-no-leak-debug-datasets.py --in-mem\nDataset len=7270695\n     idx        RSS           \u0394 RSS\n       0 <\/span>   1849.5391MB     469.5781MB\n       1 <\/span>   1833.0391MB     -16.5000MB\n       2 <\/span>   1803.4609MB     -29.5781MB\n       3 <\/span>   1811.5312MB       8.0703MB\n       4 <\/span>   1803.9531MB      -7.5781MB\n       5 <\/span>   1811.7734MB       7.8203MB\n       6 <\/span>   1836.0391MB      24.2656MB\n       7 <\/span>   1839.5938MB       3.5547MB\n       8 <\/span>   1855.9688MB      16.3750MB\n       9 <\/span>   1850.5430MB      -5.4258MB\n      10 <\/span>   1865.3398MB      14.7969MB\n      11 <\/span>   1876.2461MB      10.9062MB\n      12 <\/span>   1853.0469MB     -23.1992MB\n      13 <\/span>   1881.4453MB      28.3984MB\nTotal diff:     501.4844MB<\/pre>\n
    The RSS memory is more stable but fluctuates because the records are different, and the dataset can be huge to load into memory.<\/p>\n
    Using synthetic MMAP-disabled dataset to debug memory leaks<\/h2>\n
    Therefore the easiest approach is to create a synthetic dataset of desired length with all records being the same. That way the data is no longer a factor in the memory usage patterns as it’s always the same.<\/p>\n
    $ cat<\/span> ds<\/span>-synthetic-no<\/span>-mmap.py<\/span>\nfrom datasets import load_from_disk, Dataset\nimport gc\nimport sys\nimport os\nimport psutil\n\nproc = psutil.Process(os.getpid<\/span>())\ndef mem_read():\n    gc.collect()\n    return<\/span> proc.memory_info().rss \/ 2<\/span>**20<\/span>\n\nDS_PATH = \"synthetic-ds\"<\/span>\nif<\/span> not os.path.exists<\/span>(DS_PATH):\n    records = 1<\/span>_000_000\n    print<\/span>(\"Creating a synthetic dataset\"<\/span>)\n    row = dict(foo=[dict(a<\/span>='a'<\/span>*500<\/span>, b<\/span>='b'<\/span>*1000<\/span>)])\n    ds<\/span> = Dataset.from_dict({k<\/span>: [v] * records for<\/span> k<\/span>, v in row.items<\/span>()})\n    ds<\/span>.save_to_disk(DS_PATH)\n    print<\/span>(\"Done. Please restart the program\"<\/span>)\n    sys.exit<\/span>()\n\ndataset = load_from_disk(DS_PATH, keep_in_memory=True)\nprint<\/span>(f<\/span>\"Dataset len={len(dataset)}\"<\/span>)\n\nprint<\/span>(f<\/span>\"{'idx':>8} {'RSS':>10} {'\u0394 RSS':>15}\"<\/span>)\nmem_start = 0<\/span>\nstep = 50<\/span>_000\nwarmup_iterations = 4<\/span>\nfor<\/span> idx, i in enumerate(range<\/span>(0<\/span>, len<\/span>(dataset), step)):\n    if<\/span> idx == warmup_iterations:<\/span> # skip the first<\/span> few iterations while<\/span> things get<\/span> set<\/span> up<\/span>\n        mem_start = mem_read()\n    mem_before = mem_read()\n    _ = dataset[i:i+step]\n    mem_after = mem_read()\n    print<\/span>(f<\/span>\"{i:8d} {mem_after:12.4f}MB {mem_after - mem_before:12.4f}MB\"<\/span>)\nmem_end = mem_read()\n\nprint<\/span>(f<\/span>\"Total diff: {mem_end - mem_start:12.4f}MB (after {warmup_iterations} warmup iterations)\"<\/span>)\n<\/pre>\n
    We run this program once to create the dataset, and then the second time to profile its memory usage:<\/p>\n
    $ python ds-synthetic-no-mmap.py<\/span>\nCreating a<\/span> synthetic dataset\nDone. Please restart the program<\/pre>\n
    $ python ds-synthetic-no-mmap.py\nDataset len=1000000\n     idx        RSS           \u0394 RSS\n       0 <\/span>   1649.6055MB      95.1992MB\n   50000 <\/span>   1728.4961MB      78.8906MB\n  100000 <\/span>   1728.7109MB       0.2148MB\n  150000 <\/span>   1729.2539MB       0.5430MB\n  200000 <\/span>   1729.0039MB      -0.2500MB\n  250000 <\/span>   1729.5039MB       0.5000MB\n  300000 <\/span>   1729.2539MB      -0.2500MB\n  350000 <\/span>   1729.7539MB       0.5000MB\n  400000 <\/span>   1729.5039MB      -0.2500MB\n  450000 <\/span>   1730.0039MB       0.5000MB\n  500000 <\/span>   1729.7539MB      -0.2500MB\n  550000 <\/span>   1730.2539MB       0.5000MB\n  600000 <\/span>   1730.0039MB      -0.2500MB\n  650000 <\/span>   1730.5039MB       0.5000MB\n  700000 <\/span>   1730.2539MB      -0.2500MB\n  750000 <\/span>   1730.7539MB       0.5000MB\n  800000 <\/span>   1730.5039MB      -0.2500MB\n  850000 <\/span>   1731.0039MB       0.5000MB\n  900000 <\/span>   1730.7539MB      -0.2500MB\n  950000 <\/span>   1731.2539MB       0.5000MB\nTotal diff:       2.0000MB (after 4 <\/span>warmup iterations)\n<\/pre>\n
    This is much better. There are still tiny fluctuations due to Python and you can see in the code I skipped the first few iterations in the code while things are being set up.<\/p>\n
    But otherwise now you can easily debug the rest of your code for any memory leaks since datasets<\/code> are in non-MMAP mode and the records size doesn’t fluctuate.<\/p>\n
    Of course, do not forget to flip load_from_disk(..., keep_in_memory=True)<\/code> to False<\/code> when the debugging process is over so that you get back the performance speed up provided by MMAP.<\/p>\n
    I wrote these notes mainly for myself to ensure I have a good understanding of this complex use-case. And I hope you have gained some understanding from it as well.<\/p>\n","protected":false},"excerpt":{"rendered":"
    A step-by-step demonstration of mmap not leaking memory even though it appears to be leaking memory like there is no tomorrow. <\/p>\n","protected":false},"author":2,"featured_media":303,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[28,48],"tags":[51,31,49,52,50],"class_list":["post-301","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-machine-learning","category-software-engineering","tag-datasets","tag-machine-learning","tag-mmap","tag-pyarrow","tag-python"],"yoast_head":"\nA Deep Investigation into MMAP Not Leaking Memory - Entrepreneur Being<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/stasosphere.com\/entrepreneur-being\/301-mmap-memory-leak-investigation\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"A Deep Investigation into MMAP Not Leaking Memory - Entrepreneur Being\" \/>\n<meta property=\"og:description\" content=\"A step-by-step demonstration of mmap not leaking memory even though it appears to be leaking memory like there is no tomorrow.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/stasosphere.com\/entrepreneur-being\/301-mmap-memory-leak-investigation\/\" \/>\n<meta property=\"og:site_name\" content=\"Entrepreneur Being\" \/>\n<meta property=\"article:published_time\" content=\"2022-09-29T03:52:28+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-07-23T05:12:07+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/stasosphere.com\/entrepreneur-being\/wp-content\/uploads\/2022\/09\/A-Deep-Investigation-into-MMAP-Not-Leaking-Memory.png\" \/>\n\t<meta property=\"og:image:width\" content=\"940\" \/>\n\t<meta property=\"og:image:height\" content=\"788\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"stas\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"stas\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"8 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/stasosphere.com\/entrepreneur-being\/301-mmap-memory-leak-investigation\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/stasosphere.com\/entrepreneur-being\/301-mmap-memory-leak-investigation\/\"},\"author\":{\"name\":\"stas\",\"@id\":\"https:\/\/stasosphere.com\/entrepreneur-being\/#\/schema\/person\/554642dec8ca3206478ceaefac2b48ac\"},\"headline\":\"A Deep Investigation into MMAP Not Leaking Memory\",\"datePublished\":\"2022-09-29T03:52:28+00:00\",\"dateModified\":\"2025-07-23T05:12:07+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/stasosphere.com\/entrepreneur-being\/301-mmap-memory-leak-investigation\/\"},\"wordCount\":1814,\"commentCount\":2,\"image\":{\"@id\":\"https:\/\/stasosphere.com\/entrepreneur-being\/301-mmap-memory-leak-investigation\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/stasosphere.com\/entrepreneur-being\/wp-content\/uploads\/2022\/09\/A-Deep-Investigation-into-MMAP-Not-Leaking-Memory.png\",\"keywords\":[\"datasets\",\"machine learning\",\"mmap\",\"pyarrow\",\"python\"],\"articleSection\":[\"Machine Learning\",\"Software Engineering\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/stasosphere.com\/entrepreneur-being\/301-mmap-memory-leak-investigation\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/stasosphere.com\/entrepreneur-being\/301-mmap-memory-leak-investigation\/\",\"url\":\"https:\/\/stasosphere.com\/entrepreneur-being\/301-mmap-memory-leak-investigation\/\",\"name\":\"A Deep Investigation into MMAP Not Leaking Memory - Entrepreneur Being\",\"isPartOf\":{\"@id\":\"https:\/\/stasosphere.com\/entrepreneur-being\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/stasosphere.com\/entrepreneur-being\/301-mmap-memory-leak-investigation\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/stasosphere.com\/entrepreneur-being\/301-mmap-memory-leak-investigation\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/stasosphere.com\/entrepreneur-being\/wp-content\/uploads\/2022\/09\/A-Deep-Investigation-into-MMAP-Not-Leaking-Memory.png\",\"datePublished\":\"2022-09-29T03:52:28+00:00\",\"dateModified\":\"2025-07-23T05:12:07+00:00\",\"author\":{\"@id\":\"https:\/\/stasosphere.com\/entrepreneur-being\/#\/schema\/person\/554642dec8ca3206478ceaefac2b48ac\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/stasosphere.com\/entrepreneur-being\/301-mmap-memory-leak-investigation\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/stasosphere.com\/entrepreneur-being\/301-mmap-memory-leak-investigation\/#primaryimage\",\"url\":\"https:\/\/stasosphere.com\/entrepreneur-being\/wp-content\/uploads\/2022\/09\/A-Deep-Investigation-into-MMAP-Not-Leaking-Memory.png\",\"contentUrl\":\"https:\/\/stasosphere.com\/entrepreneur-being\/wp-content\/uploads\/2022\/09\/A-Deep-Investigation-into-MMAP-Not-Leaking-Memory.png\",\"width\":940,\"height\":788,\"caption\":\"A Deep Investigation into MMAP Not Leaking Memory\"},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/stasosphere.com\/entrepreneur-being\/#website\",\"url\":\"https:\/\/stasosphere.com\/entrepreneur-being\/\",\"name\":\"Entrepreneur Being\",\"description\":\"What can be done without a safety net\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/stasosphere.com\/entrepreneur-being\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/stasosphere.com\/entrepreneur-being\/#\/schema\/person\/554642dec8ca3206478ceaefac2b48ac\",\"name\":\"stas\",\"sameAs\":[\"https:\/\/stasosphere.com\/\"],\"url\":\"https:\/\/stasosphere.com\/entrepreneur-being\/author\/stas\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"A Deep Investigation into MMAP Not Leaking Memory - Entrepreneur Being","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/stasosphere.com\/entrepreneur-being\/301-mmap-memory-leak-investigation\/","og_locale":"en_US","og_type":"article","og_title":"A Deep Investigation into MMAP Not Leaking Memory - Entrepreneur Being","og_description":"A step-by-step demonstration of mmap not leaking memory even though it appears to be leaking memory like there is no tomorrow.","og_url":"https:\/\/stasosphere.com\/entrepreneur-being\/301-mmap-memory-leak-investigation\/","og_site_name":"Entrepreneur Being","article_published_time":"2022-09-29T03:52:28+00:00","article_modified_time":"2025-07-23T05:12:07+00:00","og_image":[{"width":940,"height":788,"url":"https:\/\/stasosphere.com\/entrepreneur-being\/wp-content\/uploads\/2022\/09\/A-Deep-Investigation-into-MMAP-Not-Leaking-Memory.png","type":"image\/png"}],"author":"stas","twitter_misc":{"Written by":"stas","Est. reading time":"8 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/stasosphere.com\/entrepreneur-being\/301-mmap-memory-leak-investigation\/#article","isPartOf":{"@id":"https:\/\/stasosphere.com\/entrepreneur-being\/301-mmap-memory-leak-investigation\/"},"author":{"name":"stas","@id":"https:\/\/stasosphere.com\/entrepreneur-being\/#\/schema\/person\/554642dec8ca3206478ceaefac2b48ac"},"headline":"A Deep Investigation into MMAP Not Leaking Memory","datePublished":"2022-09-29T03:52:28+00:00","dateModified":"2025-07-23T05:12:07+00:00","mainEntityOfPage":{"@id":"https:\/\/stasosphere.com\/entrepreneur-being\/301-mmap-memory-leak-investigation\/"},"wordCount":1814,"commentCount":2,"image":{"@id":"https:\/\/stasosphere.com\/entrepreneur-being\/301-mmap-memory-leak-investigation\/#primaryimage"},"thumbnailUrl":"https:\/\/stasosphere.com\/entrepreneur-being\/wp-content\/uploads\/2022\/09\/A-Deep-Investigation-into-MMAP-Not-Leaking-Memory.png","keywords":["datasets","machine learning","mmap","pyarrow","python"],"articleSection":["Machine Learning","Software Engineering"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/stasosphere.com\/entrepreneur-being\/301-mmap-memory-leak-investigation\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/stasosphere.com\/entrepreneur-being\/301-mmap-memory-leak-investigation\/","url":"https:\/\/stasosphere.com\/entrepreneur-being\/301-mmap-memory-leak-investigation\/","name":"A Deep Investigation into MMAP Not Leaking Memory - Entrepreneur Being","isPartOf":{"@id":"https:\/\/stasosphere.com\/entrepreneur-being\/#website"},"primaryImageOfPage":{"@id":"https:\/\/stasosphere.com\/entrepreneur-being\/301-mmap-memory-leak-investigation\/#primaryimage"},"image":{"@id":"https:\/\/stasosphere.com\/entrepreneur-being\/301-mmap-memory-leak-investigation\/#primaryimage"},"thumbnailUrl":"https:\/\/stasosphere.com\/entrepreneur-being\/wp-content\/uploads\/2022\/09\/A-Deep-Investigation-into-MMAP-Not-Leaking-Memory.png","datePublished":"2022-09-29T03:52:28+00:00","dateModified":"2025-07-23T05:12:07+00:00","author":{"@id":"https:\/\/stasosphere.com\/entrepreneur-being\/#\/schema\/person\/554642dec8ca3206478ceaefac2b48ac"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/stasosphere.com\/entrepreneur-being\/301-mmap-memory-leak-investigation\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/stasosphere.com\/entrepreneur-being\/301-mmap-memory-leak-investigation\/#primaryimage","url":"https:\/\/stasosphere.com\/entrepreneur-being\/wp-content\/uploads\/2022\/09\/A-Deep-Investigation-into-MMAP-Not-Leaking-Memory.png","contentUrl":"https:\/\/stasosphere.com\/entrepreneur-being\/wp-content\/uploads\/2022\/09\/A-Deep-Investigation-into-MMAP-Not-Leaking-Memory.png","width":940,"height":788,"caption":"A Deep Investigation into MMAP Not Leaking Memory"},{"@type":"WebSite","@id":"https:\/\/stasosphere.com\/entrepreneur-being\/#website","url":"https:\/\/stasosphere.com\/entrepreneur-being\/","name":"Entrepreneur Being","description":"What can be done without a safety net","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/stasosphere.com\/entrepreneur-being\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/stasosphere.com\/entrepreneur-being\/#\/schema\/person\/554642dec8ca3206478ceaefac2b48ac","name":"stas","sameAs":["https:\/\/stasosphere.com\/"],"url":"https:\/\/stasosphere.com\/entrepreneur-being\/author\/stas\/"}]}},"_links":{"self":[{"href":"https:\/\/stasosphere.com\/entrepreneur-being\/wp-json\/wp\/v2\/posts\/301","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/stasosphere.com\/entrepreneur-being\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/stasosphere.com\/entrepreneur-being\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/stasosphere.com\/entrepreneur-being\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/stasosphere.com\/entrepreneur-being\/wp-json\/wp\/v2\/comments?post=301"}],"version-history":[{"count":12,"href":"https:\/\/stasosphere.com\/entrepreneur-being\/wp-json\/wp\/v2\/posts\/301\/revisions"}],"predecessor-version":[{"id":348,"href":"https:\/\/stasosphere.com\/entrepreneur-being\/wp-json\/wp\/v2\/posts\/301\/revisions\/348"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/stasosphere.com\/entrepreneur-being\/wp-json\/wp\/v2\/media\/303"}],"wp:attachment":[{"href":"https:\/\/stasosphere.com\/entrepreneur-being\/wp-json\/wp\/v2\/media?parent=301"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/stasosphere.com\/entrepreneur-being\/wp-json\/wp\/v2\/categories?post=301"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/stasosphere.com\/entrepreneur-being\/wp-json\/wp\/v2\/tags?post=301"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}