{"id":7782,"date":"2023-10-04T10:31:31","date_gmt":"2023-10-04T18:31:31","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/?p=7782"},"modified":"2025-04-24T17:13:48","modified_gmt":"2025-04-24T17:13:48","slug":"understanding-memory-mapping-in-numpy-for-deep-learning-pt-1","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/understanding-memory-mapping-in-numpy-for-deep-learning-pt-1\/","title":{"rendered":"Understanding Memory Mapping in Numpy for Deep Learning: Pt 1"},"content":{"rendered":"\n\n\n\n\n
\"\"\/
Photo by Ian Talmacs<\/a> on Unsplash<\/a><\/figcaption><\/figure>\n\n\n\n

In the previous article<\/a>, we discussed the possibility of having memory constraints when trying to get into deep learning. I mentioned Numpy\u2019s memory mapping; in this article, we go deeper into the topic and offer an excellent guide to get you started.<\/p>\n\n\n\n

There might be a question about why Numpy memory mapping is necessary and who the target demographic is for this particular technique. Now, we can dive into the nitty-gritty details.<\/p>\n\n\n\n

Memory-Mapping<\/h1>\n\n\n\n

Numpy describes its memory mapping feature as accessing small segments of a file without loading the entire file into memory.<\/p>\n\n\n\n

This feature highlights the existence of a memory shortage problem for some people. When working with deep learning, arrays might become larger than what could fit in memory. This spells out a problem as it will prevent the computer from running as intended.<\/p>\n\n\n\n

Memory mapping will favor someone with access to ample storage compared to memory size (which is often the case). The first step is to save the array as a Numpy binary. Then, from that point onwards, we will load it via memory mapping and perform any operations we deem fit.<\/p>\n\n\n\n

Saving Binaries<\/h1>\n\n\n\n

A binary can easily be described as a file format used to store information. Storing arrays in binaries is convenient as it means you don\u2019t have to waste precious time trying to remake the array whenever you want to return to a problem. A binary allows for easy retrieval of this information whenever needed.<\/p>\n\n\n\n

Numpy offers \u201csave()\u201d and \u201csavez()\u201d when you want to save binaries. This functionality is often easy to work with as it has few parameters. You must describe the binary\u2019s path and the array to be stored in binary format. If you have no other needs, then you\u2019re good to go.<\/p>\n\n\n\n

For example, if you have an array that stores image data such as training data(X_train and y_train) and you want to store it, you would do the following:<\/p>\n\n\n\n

#Splitting training data for image into X and y<\/span>\nX_train = []\ny_train = []\n\nfor<\/span> features, labels in<\/span> training_data:\n    X_train.append(features)\n    y_train.append(labels)\n\n#Saving into a binary<\/span>\nfrom<\/span> numpy import<\/span> save\n\nsave(\"X_train.npy\"<\/span>, X_train)\nsave(\"y_train.npy\"<\/span>, y_train)<\/span><\/pre>\n\n\n\n
The above code snippet will save one binary each for the X_train array and the y_train array, respectively.<\/p>\n\n\n\n
Once the binary file is saved, there are multiple ways to load the file into memory to perform operations. The standard method involves just loading them as they are into memory. The second way requires memory mapping.<\/p>\n\n\n\n
Loading Binaries<\/h1>\n\n\n\n
Numpy offers a few tools to load binaries as desired. The first method is by using \u201cload()\u201d while the second is using \u201cmemmap().\u201d There are advantages and disadvantages to using each one.<\/p>\n\n\n\n
The first method allows someone to use memory mapping or load directly, as efficiently and fluidly as possible. The only difference is an argument away.<\/p>\n\n\n\n
I made two binary files to perform this example and to show how load() works. The first binary is a 1.4 GB file we will memory map, and the second is a 32 KB file that will be entirely loaded into memory.<\/p>\n\n\n\n
\"\"\/
Screenshot by Author<\/figcaption><\/figure>\n\n\n\n
\"\"\/
Screenshot by Author<\/figcaption><\/figure>\n\n\n\n
from<\/span> numpy import<\/span> load\n\n#Load binary as y_train<\/span>\ny_train = load(\"y_train.npy)<\/span><\/span><\/pre>\n\n\n\n
Running the above code will call the entirety of y_train into memory, and we can check this using the \u201csys\u201d library. We expect it to occupy a memory amount similar to the storage size.<\/p>\n\n\n\n
import<\/span> sys\n\n#checking the size of the file in memory<\/span>\nprint<\/span>(f'y_train memory occupied: {sys.getsizeof(y_train)}<\/span> bytes'<\/span>)<\/span><\/pre>\n\n\n\n
\"\"\/
Screenshot by Author<\/figcaption><\/figure>\n\n\n\n
It has occupied 32 KB, as we expected it to. We have demonstrated what loading into memory does and expect it to behave similarly if a different, larger file is loaded. This is where our memory mapping comes into the picture, as we can load a smaller amount of content into memory.<\/p>\n\n\n\n
The \u201cX_train\u201d file is 1.4 GB, but only a tiny fraction of the memory will be occupied when we alter the code we wrote.<\/p>\n\n\n\n
from<\/span> numpy import<\/span> load\nimport<\/span> sys\n\n#Loads the X_train.npy <\/span>\nX_train = load(\"X_train.npy\"<\/span>, mmap_mode='r'<\/span>)<\/span><\/pre>\n\n\n\n
Here, we include the \u201cmmap_mode\u201d argument and choose \u201cr\u201d as our preferred mode. It refers to read-only and is one of four modes that are offered. All the modes are as defined below in the Numpy documentation:<\/p>\n\n\n\n
\"\"\/
Courtesy of Numpy<\/a><\/figcaption><\/figure>\n\n\n\n
Now, we can perform a similar test to the previous one and check how much memory X_train will use after using the memory-mapped mode.<\/p>\n\n\n\n
print<\/span>(f'X_train memory occupied: {sys.getsizeof(X_train)}<\/span> bytes'<\/span>)<\/span><\/pre>\n\n\n\n
\"\"\/
Screenshot by Author<\/figcaption><\/figure>\n\n\n\n
Impressively enough, our much larger file now occupies significantly less memory when memory-mapped. It only occupies 184 bytes but remains in a format that could be used for tasks such as training models or tasks like predicting labels without much additional overhead.<\/p>\n\n\n\n
#model fitting will work just as usual with memory-mapped variables<\/span>\n\nmodel.fit(X_train, y_train)\n\n#predicting will also similarly be the same<\/span>\ny_predict = model.predict(X_test)<\/span><\/pre>\n\n\n\n
Disclaimer<\/h1>\n\n\n\n
This method is not a magic bullet for all your problems, as there are a few things to remember when using memory mapping.<\/p>\n\n\n\n
One consideration is whether you want to operate on the loaded binary. If you desire to work on the memory-mapped file, it will become an array and occupy the memory you set out to save. It is, therefore, advisable to split the data into slices to iterate on any given operation.<\/p>\n\n\n\n
Another consideration is the modes offered when using \u201cmmap_mode()\u201d. If the binary needs changes, then the appropriate mode may not be the default mode that only supports read-only.<\/p>\n\n\n\n
Wrap Up\u2026<\/h1>\n\n\n\n
Memory mapping is a great way to save memory when performing operations that may occupy a lot of space. In this article, we learned how to use Numpy\u2019s \u201cload()\u201d to load things entirely into the memory and how to perform memory mapping.<\/p>\n\n\n\n
Numpy also offers \u201cmemmap()\u201d for this particular job, but it has different dynamics that we will explore in the second part of this article.<\/p>\n","protected":false},"excerpt":{"rendered":"
In the previous article, we discussed the possibility of having memory constraints when trying to get into deep learning. I mentioned Numpy\u2019s memory mapping; in this article, we go deeper into the topic and offer an excellent guide to get you started. There might be a question about why Numpy memory mapping is necessary and […]<\/p>\n","protected":false},"author":79,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"customer_name":"","customer_description":"","customer_industry":"","customer_technologies":"","customer_logo":"","footnotes":""},"categories":[6,7],"tags":[],"coauthors":[176],"class_list":["post-7782","post","type-post","status-publish","format-standard","hentry","category-machine-learning","category-tutorials"],"yoast_head":"\nUnderstanding Memory Mapping in Numpy for Deep Learning: Pt 1 - Comet<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.comet.com\/site\/blog\/understanding-memory-mapping-in-numpy-for-deep-learning-pt-1\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Understanding Memory Mapping in Numpy for Deep Learning: Pt 1\" \/>\n<meta property=\"og:description\" content=\"In the previous article, we discussed the possibility of having memory constraints when trying to get into deep learning. I mentioned Numpy\u2019s memory mapping; in this article, we go deeper into the topic and offer an excellent guide to get you started. There might be a question about why Numpy memory mapping is necessary and […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.comet.com\/site\/blog\/understanding-memory-mapping-in-numpy-for-deep-learning-pt-1\/\" \/>\n<meta property=\"og:site_name\" content=\"Comet\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/cometdotml\" \/>\n<meta property=\"article:published_time\" content=\"2023-10-04T18:31:31+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-04-24T17:13:48+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/0*v1ps6vVnMykXbtMS\" \/>\n<meta name=\"author\" content=\"Mwanikii Njagi\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@Cometml\" \/>\n<meta name=\"twitter:site\" content=\"@Cometml\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Mwanikii Njagi\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 minutes\" \/>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Understanding Memory Mapping in Numpy for Deep Learning: Pt 1 - Comet","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.comet.com\/site\/blog\/understanding-memory-mapping-in-numpy-for-deep-learning-pt-1\/","og_locale":"en_US","og_type":"article","og_title":"Understanding Memory Mapping in Numpy for Deep Learning: Pt 1","og_description":"In the previous article, we discussed the possibility of having memory constraints when trying to get into deep learning. I mentioned Numpy\u2019s memory mapping; in this article, we go deeper into the topic and offer an excellent guide to get you started. There might be a question about why Numpy memory mapping is necessary and […]","og_url":"https:\/\/www.comet.com\/site\/blog\/understanding-memory-mapping-in-numpy-for-deep-learning-pt-1\/","og_site_name":"Comet","article_publisher":"https:\/\/www.facebook.com\/cometdotml","article_published_time":"2023-10-04T18:31:31+00:00","article_modified_time":"2025-04-24T17:13:48+00:00","og_image":[{"url":"https:\/\/miro.medium.com\/v2\/resize:fit:700\/0*v1ps6vVnMykXbtMS","type":"","width":"","height":""}],"author":"Mwanikii Njagi","twitter_card":"summary_large_image","twitter_creator":"@Cometml","twitter_site":"@Cometml","twitter_misc":{"Written by":"Mwanikii Njagi","Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.comet.com\/site\/blog\/understanding-memory-mapping-in-numpy-for-deep-learning-pt-1\/#article","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/blog\/understanding-memory-mapping-in-numpy-for-deep-learning-pt-1\/"},"author":{"name":"Mwanikii Njagi","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/c7043b3e6b992af7b3220aa1f27d2162"},"headline":"Understanding Memory Mapping in Numpy for Deep Learning: Pt 1","datePublished":"2023-10-04T18:31:31+00:00","dateModified":"2025-04-24T17:13:48+00:00","mainEntityOfPage":{"@id":"https:\/\/www.comet.com\/site\/blog\/understanding-memory-mapping-in-numpy-for-deep-learning-pt-1\/"},"wordCount":900,"publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"image":{"@id":"https:\/\/www.comet.com\/site\/blog\/understanding-memory-mapping-in-numpy-for-deep-learning-pt-1\/#primaryimage"},"thumbnailUrl":"https:\/\/miro.medium.com\/v2\/resize:fit:700\/0*v1ps6vVnMykXbtMS","articleSection":["Machine Learning","Tutorials"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.comet.com\/site\/blog\/understanding-memory-mapping-in-numpy-for-deep-learning-pt-1\/","url":"https:\/\/www.comet.com\/site\/blog\/understanding-memory-mapping-in-numpy-for-deep-learning-pt-1\/","name":"Understanding Memory Mapping in Numpy for Deep Learning: Pt 1 - Comet","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.comet.com\/site\/blog\/understanding-memory-mapping-in-numpy-for-deep-learning-pt-1\/#primaryimage"},"image":{"@id":"https:\/\/www.comet.com\/site\/blog\/understanding-memory-mapping-in-numpy-for-deep-learning-pt-1\/#primaryimage"},"thumbnailUrl":"https:\/\/miro.medium.com\/v2\/resize:fit:700\/0*v1ps6vVnMykXbtMS","datePublished":"2023-10-04T18:31:31+00:00","dateModified":"2025-04-24T17:13:48+00:00","breadcrumb":{"@id":"https:\/\/www.comet.com\/site\/blog\/understanding-memory-mapping-in-numpy-for-deep-learning-pt-1\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.comet.com\/site\/blog\/understanding-memory-mapping-in-numpy-for-deep-learning-pt-1\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/blog\/understanding-memory-mapping-in-numpy-for-deep-learning-pt-1\/#primaryimage","url":"https:\/\/miro.medium.com\/v2\/resize:fit:700\/0*v1ps6vVnMykXbtMS","contentUrl":"https:\/\/miro.medium.com\/v2\/resize:fit:700\/0*v1ps6vVnMykXbtMS"},{"@type":"BreadcrumbList","@id":"https:\/\/www.comet.com\/site\/blog\/understanding-memory-mapping-in-numpy-for-deep-learning-pt-1\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.comet.com\/site\/"},{"@type":"ListItem","position":2,"name":"Understanding Memory Mapping in Numpy for Deep Learning: Pt 1"}]},{"@type":"WebSite","@id":"https:\/\/www.comet.com\/site\/#website","url":"https:\/\/www.comet.com\/site\/","name":"Comet","description":"Build Better Models Faster","publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.comet.com\/site\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.comet.com\/site\/#organization","name":"Comet ML, Inc.","alternateName":"Comet","url":"https:\/\/www.comet.com\/site\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","width":310,"height":310,"caption":"Comet ML, Inc."},"image":{"@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/cometdotml","https:\/\/x.com\/Cometml","https:\/\/www.youtube.com\/channel\/UCmN63HKvfXSCS-UwVwmK8Hw"]},{"@type":"Person","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/c7043b3e6b992af7b3220aa1f27d2162","name":"Mwanikii Njagi","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/image\/1a3c516cf04aca9418dfb2213081f4df","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/08\/cropped-1_2jy9gyk0G_yaniWm8gJFVA-1-96x96.webp","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/08\/cropped-1_2jy9gyk0G_yaniWm8gJFVA-1-96x96.webp","caption":"Mwanikii Njagi"},"url":"https:\/\/www.comet.com\/site\/blog\/author\/freddynjagigmail-com\/"}]}},"_links":{"self":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/7782","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/users\/79"}],"replies":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/comments?post=7782"}],"version-history":[{"count":1,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/7782\/revisions"}],"predecessor-version":[{"id":15526,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/7782\/revisions\/15526"}],"wp:attachment":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/media?parent=7782"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/categories?post=7782"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/tags?post=7782"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/coauthors?post=7782"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}