Python Memory

TODO

  • Compute fragmentation of pymalloc
  • Compute fragmentation of the glibc heap
  • Tool to visualize the fragmentation?

Memory Fragmentation

Python Memory Allocators

  • PyMem_RawMalloc(), PyMem_RawRealloc(), PyMem_RawFree(): new in Python 3.4
  • PyMem_Malloc(), PyMem_Realloc(), PyMem_Free()
  • PyObject_Malloc(), PyObject_Realloc(), PyObject_Free()

Current allocators:

  • PyMem_RawMalloc(): system malloc()
  • PyMem_Malloc(): system malloc()
  • PyObject_Malloc(): pymalloc

http://www.python.org/dev/peps/pep-0445/

Windows

Links:

pymalloc

  • Used by PyObject_Malloc()
  • Threshold of 512 bytes
  • Arenas of 256 KB

Arena allocator:

  • Windows: VirtualAlloc(NULL, size, MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE);
  • UNIX: mmap(NULL, size, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);
  • Other: malloc(size);

Linux

/proc/self/statm:

size       total program size
           (same as VmSize in /proc/[pid]/status)
resident   resident set size
           (same as VmRSS in /proc/[pid]/status)
share      shared pages (from shared mappings)
text       text (code)
data       data + stack

http://linux-mm.org/Low_On_Memory

/proc, https://www.kernel.org/doc/Documentation/filesystems/proc.txt:

/proc/buddyinfo
/proc/pagetypeifo
/proc/slabinfo => slabtop program
/proc/meminfo
/proc/vmallocinfo