You should not necessarily be concerned if your ZXTM server appears to have very little free physical memory. By default the Linux kernel allocates any, otherwise unused, physical memory for kernel buffers and caches. Physical memory is allocated as required, but kernel structures (buffers, caches, et al.) are kernel only: they are neither userspace accessible, nor given over to user space. An accurate view of the memory available can be seen by using the 'free' command and paying attention to the buffer/cache memory figures.
To see how much memory is being used by the ZXTM process(es), you can use the GNU utility, 'top'. Pressing SHIFT+M will arrange the list in order of memory usage.