{"id":7784,"date":"2023-10-04T10:33:24","date_gmt":"2023-10-04T18:33:24","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/?p=7784"},"modified":"2025-04-24T17:06:10","modified_gmt":"2025-04-24T17:06:10","slug":"understanding-memory-mapping-in-numpy-for-deep-learning-pt-2","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/understanding-memory-mapping-in-numpy-for-deep-learning-pt-2\/","title":{"rendered":"Understanding Memory Mapping in Numpy for Deep Learning: Pt 2"},"content":{"rendered":"\n\n\n\n\n
\n
\n
\n
\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>
Photo by Heliberto Arias<\/a> on Unsplash<\/a><\/figcaption><\/figure>\n

In the previous article<\/a>, we demonstrated the benefits and ease with which someone could integrate memory mapping into their workflow. A simple overview is that memory mapping allows an individual to occupy a small amount of memory rather than loading an entire array or object.<\/p>\n

Numpy offers multiple solutions for any given problem, and we explored memory mapping using the \u201cmmap_mode\u201d argument presented in Numpy\u2019s \u201cload().\u201d The difference is that now we will look at Numpy\u2019s other solution.<\/p>\n

When there may be a need for further capacity to manipulate values and leverage more tools when doing memory mapping, the go-to solution would be Numpy\u2019s \u201cmemmap().\u201d<\/p>\n

Let\u2019s get to it.<\/p>\n

A Brief Intro<\/h1>\n

One of the first realizations you encounter with memmap is that it will not be as straightforward as you may have assumed. The documentation provides an intricate web of warnings on how to use it, and you quickly learn why.<\/p>\n

For instance, a simple documentation scan mentions that you \u201cmust specify shape\u201d as the array will be provided as a 1-dimensional array by default. One way we can do this is by taking the binary with information of an \u201cX_train\u201d array in the previous article.<\/p>\n

import<\/span> numpy as<\/span> np\n\nX_train = np.load(\"X_train.npy\"<\/span>, mmap_mode='r'<\/span>)\n\nprint<\/span>(X_train.shape)<\/span><\/pre>\n
\n
\"\"<\/figure>
<\/picture><\/div>\n
Array shape screenshot by the author<\/figcaption>\n<\/figure>\n
Attempting the same thing using memmap will yield a dramatically different result that aligns with the warning aired in the documentation.<\/p>\n
X_train = np.memmap(\"X_train.npy\"<\/span>, mode = 'r'<\/span>)<\/span><\/pre>\n
\n
\"\"<\/figure>
<\/picture><\/div>\n
Array shape screenshot by the author<\/figcaption>\n<\/figure>\n
The value above is truly a 1-dimensional array with many elements, as indicated by a lack of values trailing the comma.<\/p>\n
But can we fix this and load the values appropriately? What points are vital and need consideration when using memmap? We can now answer these questions by paying attention to the tiny details a person could easily miss.<\/p>\n
Binary Formats and Arguments<\/h2>\n
In the example above, I attempted to use a binary in NumPy format. Even after trying the appropriate method of loading the binary, I still could not get the proper elements out of it.<\/p>\n
By visual inspection, we see many floats when we use the \u201cload\u201d method, but that doesn\u2019t happen when using \u201cmemmap.\u201d<\/p>\n
\n
\"\"<\/figure>
<\/picture><\/div>\n
Array section using \u201cLoad\u201d by author<\/figcaption>\n<\/figure>\n
\n
\"\"<\/figure>
<\/picture><\/div>\n
Array section using \u201cmemmap\u201d<\/figcaption>\n<\/figure>\n
As seen above, one gives an array of floats while the other offers an array of integers. We will still get different values even if we try assigning a float type in memmap using the data type argument(dtype=).<\/p>\n
One of the reasons we are running into these issues is because of the binary file format we use. We are using Numpy binaries and expecting that they will work perfectly, yet \u201cmemmap()\u201d requires us to use different formats, such as the Raw Binary Format (RBF).<\/p>\n
We can convert the Numpy array we loaded earlier into a Raw Binary Format and attempt memory mapping the array.<\/p>\n
import<\/span> numpy as<\/span> np\n\n#remember we loaded X_train using \"load()\" previously <\/span>\n\nX_train.tofile(\"X_train_raw.rbf\"<\/span>)<\/span><\/pre>\n
\n
\"\"<\/figure>
<\/picture><\/div>\n
Screenshot of the above file by author<\/figcaption>\n<\/figure>\n
The binary file created above is smaller than the original Numpy binary by a few hundred megabytes. Will it mean a smaller memory occupation? We will investigate later.<\/p>\n
Now, we can attempt memory mapping with the lessons we acquired:<\/p>\n
    \n
  1. Pay attention to shape.<\/li>\n
  2. Specify data type.<\/li>\n
  3. Mode \u2014 Whether it is read-only(r), read and write(r+) etc.<\/li>\n<\/ol>\n
    Memory Mapping<\/h1>\n
    We are going to name the variable that will store the memory-mapped array \u201cX_train_mapped.\u201d<\/p>\n
    #Using memmap to memory map the binary<\/span>\n\nX_train_mapped = np.memmap(\"X_train_raw.rbf\"<\/span>,\n                            shape=(4002<\/span>, 120<\/span>, 120<\/span>, 3<\/span>),\n                            mode = 'r'<\/span>,\n                            dtype = 'float64'<\/span>)\n\n#printing the first element of the stored array<\/span>\nprint<\/span>(X_train_mapped[0<\/span>])<\/span><\/pre>\n
    Now that all arguments are theoretically in the appropriate form, we can run the code and see if the first element is what we desire.<\/p>\n
    \n
    \"\"<\/figure>
    <\/picture><\/div>\n
    Screenshot by author<\/figcaption>\n<\/figure>\n
    Now, it is identical to the information we had initially stored. The data looks precisely as it was meant to because we have used the correct binary format, the appropriate shape, and, most importantly, the correct data type.<\/p>\n
    Since it works, we can now explore the question of memory.<\/p>\n
    Memory Usage<\/h1>\n
    Is there any inherent advantage when using this particular method? In the previous article, we highlighted that a 1.4 GB file only occupied 184 bytes after being memory-mapped.<\/p>\n
    In this article, we created and stored a raw binary from that same file. It occupies 805.4 MB. Now, we can check how much space it will take on the memory using the \u201csys\u201d library.<\/p>\n
    import<\/span> sys\n\n#getting the amount of memory occupied by X_train_mapped<\/span>\nprint<\/span>(f'X_train_mapped memory occupied: {sys.getsizeof(X_train_mapped)}<\/span> bytes'<\/span>)<\/span><\/pre>\n
    \n
    \"\"<\/figure>
    <\/picture><\/div>\n
    Screenshot by author<\/figcaption>\n<\/figure>\n
    Well, we can see no change in the amount of memory occupied. It is still 184 bytes, which is still impressive, to say the least.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n
    \n
    \n
    \n
    I believe that we now have a comprehensive understanding of both \u201cload()\u201d and \u201cmemmap()\u201d for the tasks we need to use them for. A simple overview of what we did would be as follows for those who want to use memmap():<\/p>\n
      \n
    1. Store arrays in Raw Binary Format.<\/li>\n
    2. Ensure you know the array\u2019s shape to write in the \u201cshape\u201d argument.<\/li>\n
    3. Specify the mode you desire to load the array in.<\/li>\n
    4. Make sure that the datatype argument has the correct datatype.<\/li>\n<\/ol>\n
      Conclusion<\/h1>\n
      This is the end of the two-part series on memory mapping, where I have given a comprehensive beginner tutorial on the best method for a deep learning dive. These methods are not limited to deep learning as we have acquired an understanding of binaries and how they could be used to store information.<\/p>\n
      Adios! Til we meet again in the next tutorial. Code away.<\/p>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"
      Photo by Heliberto Arias on Unsplash In the previous article, we demonstrated the benefits and ease with which someone could integrate memory mapping into their workflow. A simple overview is that memory mapping allows an individual to occupy a small amount of memory rather than loading an entire array or object. Numpy offers multiple solutions […]<\/p>\n","protected":false},"author":79,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"customer_name":"","customer_description":"","customer_industry":"","customer_technologies":"","customer_logo":"","footnotes":""},"categories":[6,7],"tags":[],"coauthors":[176],"class_list":["post-7784","post","type-post","status-publish","format-standard","hentry","category-machine-learning","category-tutorials"],"yoast_head":"\nUnderstanding Memory Mapping in Numpy for Deep Learning: Pt 2 - Comet<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.comet.com\/site\/blog\/understanding-memory-mapping-in-numpy-for-deep-learning-pt-2\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Understanding Memory Mapping in Numpy for Deep Learning: Pt 2\" \/>\n<meta property=\"og:description\" content=\"Photo by Heliberto Arias on Unsplash In the previous article, we demonstrated the benefits and ease with which someone could integrate memory mapping into their workflow. A simple overview is that memory mapping allows an individual to occupy a small amount of memory rather than loading an entire array or object. Numpy offers multiple solutions […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.comet.com\/site\/blog\/understanding-memory-mapping-in-numpy-for-deep-learning-pt-2\/\" \/>\n<meta property=\"og:site_name\" content=\"Comet\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/cometdotml\" \/>\n<meta property=\"article:published_time\" content=\"2023-10-04T18:33:24+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-04-24T17:06:10+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/0*gcMbHjSPWJ98QaVm\" \/>\n<meta name=\"author\" content=\"Mwanikii Njagi\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@Cometml\" \/>\n<meta name=\"twitter:site\" content=\"@Cometml\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Mwanikii Njagi\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 minutes\" \/>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Understanding Memory Mapping in Numpy for Deep Learning: Pt 2 - Comet","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.comet.com\/site\/blog\/understanding-memory-mapping-in-numpy-for-deep-learning-pt-2\/","og_locale":"en_US","og_type":"article","og_title":"Understanding Memory Mapping in Numpy for Deep Learning: Pt 2","og_description":"Photo by Heliberto Arias on Unsplash In the previous article, we demonstrated the benefits and ease with which someone could integrate memory mapping into their workflow. A simple overview is that memory mapping allows an individual to occupy a small amount of memory rather than loading an entire array or object. Numpy offers multiple solutions […]","og_url":"https:\/\/www.comet.com\/site\/blog\/understanding-memory-mapping-in-numpy-for-deep-learning-pt-2\/","og_site_name":"Comet","article_publisher":"https:\/\/www.facebook.com\/cometdotml","article_published_time":"2023-10-04T18:33:24+00:00","article_modified_time":"2025-04-24T17:06:10+00:00","og_image":[{"url":"https:\/\/miro.medium.com\/v2\/resize:fit:700\/0*gcMbHjSPWJ98QaVm","type":"","width":"","height":""}],"author":"Mwanikii Njagi","twitter_card":"summary_large_image","twitter_creator":"@Cometml","twitter_site":"@Cometml","twitter_misc":{"Written by":"Mwanikii Njagi","Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.comet.com\/site\/blog\/understanding-memory-mapping-in-numpy-for-deep-learning-pt-2\/#article","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/blog\/understanding-memory-mapping-in-numpy-for-deep-learning-pt-2\/"},"author":{"name":"Mwanikii Njagi","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/c7043b3e6b992af7b3220aa1f27d2162"},"headline":"Understanding Memory Mapping in Numpy for Deep Learning: Pt 2","datePublished":"2023-10-04T18:33:24+00:00","dateModified":"2025-04-24T17:06:10+00:00","mainEntityOfPage":{"@id":"https:\/\/www.comet.com\/site\/blog\/understanding-memory-mapping-in-numpy-for-deep-learning-pt-2\/"},"wordCount":877,"publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"image":{"@id":"https:\/\/www.comet.com\/site\/blog\/understanding-memory-mapping-in-numpy-for-deep-learning-pt-2\/#primaryimage"},"thumbnailUrl":"https:\/\/miro.medium.com\/v2\/resize:fit:700\/0*gcMbHjSPWJ98QaVm","articleSection":["Machine Learning","Tutorials"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.comet.com\/site\/blog\/understanding-memory-mapping-in-numpy-for-deep-learning-pt-2\/","url":"https:\/\/www.comet.com\/site\/blog\/understanding-memory-mapping-in-numpy-for-deep-learning-pt-2\/","name":"Understanding Memory Mapping in Numpy for Deep Learning: Pt 2 - Comet","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.comet.com\/site\/blog\/understanding-memory-mapping-in-numpy-for-deep-learning-pt-2\/#primaryimage"},"image":{"@id":"https:\/\/www.comet.com\/site\/blog\/understanding-memory-mapping-in-numpy-for-deep-learning-pt-2\/#primaryimage"},"thumbnailUrl":"https:\/\/miro.medium.com\/v2\/resize:fit:700\/0*gcMbHjSPWJ98QaVm","datePublished":"2023-10-04T18:33:24+00:00","dateModified":"2025-04-24T17:06:10+00:00","breadcrumb":{"@id":"https:\/\/www.comet.com\/site\/blog\/understanding-memory-mapping-in-numpy-for-deep-learning-pt-2\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.comet.com\/site\/blog\/understanding-memory-mapping-in-numpy-for-deep-learning-pt-2\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/blog\/understanding-memory-mapping-in-numpy-for-deep-learning-pt-2\/#primaryimage","url":"https:\/\/miro.medium.com\/v2\/resize:fit:700\/0*gcMbHjSPWJ98QaVm","contentUrl":"https:\/\/miro.medium.com\/v2\/resize:fit:700\/0*gcMbHjSPWJ98QaVm"},{"@type":"BreadcrumbList","@id":"https:\/\/www.comet.com\/site\/blog\/understanding-memory-mapping-in-numpy-for-deep-learning-pt-2\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.comet.com\/site\/"},{"@type":"ListItem","position":2,"name":"Understanding Memory Mapping in Numpy for Deep Learning: Pt 2"}]},{"@type":"WebSite","@id":"https:\/\/www.comet.com\/site\/#website","url":"https:\/\/www.comet.com\/site\/","name":"Comet","description":"Build Better Models Faster","publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.comet.com\/site\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.comet.com\/site\/#organization","name":"Comet ML, Inc.","alternateName":"Comet","url":"https:\/\/www.comet.com\/site\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","width":310,"height":310,"caption":"Comet ML, Inc."},"image":{"@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/cometdotml","https:\/\/x.com\/Cometml","https:\/\/www.youtube.com\/channel\/UCmN63HKvfXSCS-UwVwmK8Hw"]},{"@type":"Person","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/c7043b3e6b992af7b3220aa1f27d2162","name":"Mwanikii Njagi","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/image\/1a3c516cf04aca9418dfb2213081f4df","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/08\/cropped-1_2jy9gyk0G_yaniWm8gJFVA-1-96x96.webp","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/08\/cropped-1_2jy9gyk0G_yaniWm8gJFVA-1-96x96.webp","caption":"Mwanikii Njagi"},"url":"https:\/\/www.comet.com\/site\/blog\/author\/freddynjagigmail-com\/"}]}},"_links":{"self":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/7784","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/users\/79"}],"replies":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/comments?post=7784"}],"version-history":[{"count":1,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/7784\/revisions"}],"predecessor-version":[{"id":15525,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/7784\/revisions\/15525"}],"wp:attachment":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/media?parent=7784"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/categories?post=7784"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/tags?post=7784"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/coauthors?post=7784"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}