VEOS Partial Process Swapping Reference Date: Dec-2020 This document describes the information regarding the VEOS version 2.7.4 or later. 1 Introduction Partial process swapping (PPS) is the feature which saves and restores the VE memory of stopped VE process. It releases VE memory by saving a part of the memory of one or more temporarily stopped VE processes to VH side memory or a file(swap out). Another program can use the released VE memory. After the new program finished, it restores the memory saved in the VH memory or file to the VE memory (swap in), and the execution of the original program is resumed. 1.1 Term definition This chapter defines the terms used in this document. 1) swap area: VH side memory or a file to which a VE process memory is saved. 2) swap out: Saves a part of the memory of VE processes to a swap area, and release VE memory. 3) swap in: Restores the memory saved in a swap area to VE memory, and the execution of program can be resumed. 4) status: Indicates the execution state of the VE process. The status is displayed with the same symbol as the "ps" command. 5) sub-status: Indicates the state of the VE process. It is a PPS original definition to indicate the progress of swap out/swap in processing. The definition of each sub status is as follows. a swap out is not performed, the same as conventional stop state o swap out is processing s swap out is finished i swap in is processing 6) PPS mode: Modes are defined according to the swap area memory. memory mode: swap area is VH memory file mode: swap area is a file. mix mode: swap area is VH memory and file (Use VH memory firstly) 2 Preset This chapter describes the settings required to use the Partial Process Swapping function. 2.1 About HugePages of VH When VEOS starts, VEOS allocates HugePages as follows: memory mode: Memory size is specified size in a configuration file. file mode : Memory size is 64MB. mix mode : Memory size is specified size in a configuration file + 64MB. If the value of the parameter vm.nr_hugepages is already set in the file "/etc/sysctl.conf", update the value. Otherwise, add the line vm.nr_hugepages to the file to specify the value. ------ # vi /etc/sysctl.conf vm.nr_hugepages = ------ Apply the setting with the sysctl command. ------ # sysctl --system ------ The size of one HugePages is 2MB. If you want to allocate 40GB buffer, you should specify "20480" to above " the number of VH's HugePages". Please see more details in "SX-Aurora TSUBASA Installation Guide". 2.2 About veswap.options 1) The path of veswap.options - To set a common value for all VE nodes /etc/opt/nec/ve/veos/ve-os-launcher.d/veswap.options - To set each VE node individually /etc/opt/nec/ve/veos/ve-os-launcher.d//veswap.options is VE node number 2) Contents of veswap.options You can set the maximum size of swappable VE memory per VE node and the absolute path of swap file. For file mode, VH's host name and VE node number are added to the file path. If you do not specify the owner of the swap file, the owner will be "root". If you do not specify the group of the swap file, the group will be "root". If you specify the owner of the swap file, you need the write permission of the specified user in the directory of the file path. Please modify "veswap.options" as follows. [memory mode] ve-os-launcher@*=--ve-swap-mem-max= [file mode] ve-os-launcher@*=--ve-swap-file-path= ve-os-launcher@*=--ve-swap-file-max= ve-os-launcher@*=--ve-swap-file-user= ve-os-launcher@*=--ve-swap-file-group= [mix mode] ve-os-launcher@*=--ve-swap-mem-max= ve-os-launcher@*=--ve-swap-file-path= ve-os-launcher@*=--ve-swap-file-max= ve-os-launcher@*=--ve-swap-file-user= ve-os-launcher@*=--ve-swap-file-group= 3) Load the configuration Restart VEOS as follows in order to load the configuration. ------ $ systemctl restart ve-os-launcher@ ------ is VE node number 3 veswap command This chapter describes the PPS command "/opt/nec/ve/bin/veswap". 3.1 Synopsis /opt/nec/ve/bin/veswap [-o|-i|-m] pid ... [-H] /opt/nec/ve/bin/veswap -s [-H] /opt/nec/ve/bin/veswap -h /opt/nec/ve/bin/veswap -v 3.2 Description 1) No options Displays the status(S) and sub-status(SUB-S) of the specified VE process. Display example: ------ $ /opt/nec/ve/bin/veswap 1001 1002 1003 1004 1005 1006 1007 1008 1009 S SUB-S PID R - 1001 T o 1002 T s 1003 T i 1004 T a 1005 Z o 1006 Z s 1007 Z i 1008 Z a 1009 ------ 2) Details of options -o Swaps out the VE memory of specified VE process to swap area. The VE memory of the VE process that is in one of the following states is able to swap out. - status is "T" and sub-status is "a" - status is "Z" and sub-status is "a" -i Swaps in the VE memory of specified VE process from swap area. The VE memory of the VE process that is in one of the following states is able to swap in. - status is "T" and sub-status is "s" - status is "Z" and sub-status is "s" - status is "T" and sub-status is "o" - status is "Z" and sub-status is "o" -m Displays the VE memory size information of the specified VE processes. Displays the size of swappable VE memory (SWAPPABLE) and the size of VE memory swapped out (SWAPPED). The sizes are displayed in KB. -n Displays the sizes of non-swappable VE memory (NON-SWAPPABLE). The size is displayed in KB. -s Displays the swapped out VE memory sizes in VE nodes. The sizes are displayed in KB. -H Not display the header. -h, --help Displays the help and exit. -V, --version Displays version information and exit. 3) Details of parameters pid Specifies the ID of VE process. 4 Example of using This chapter shows an example of swapping out the VE memory of the process (VE process A) of VE program A and executing the VE program B. "Pid-A" in the example is the process ID of the process(VE process A) of VE program A. ----- 1) Stop the VE process A with "SIGSTOP". $ kill -s SIGSTOP pid-A 2) Make sure VE process A is in one of the following states - status(S) is "T" and sub-status(SUB-S) is "a" - status(S) is "Z" and sub-status(SUB-S) is "a" $ /opt/nec/ve/bin/veswap pid-A S SUB-S PID T a pid-A 3) Swap out the VE memory after the state of the VE process A becomes one of the following - status(S) is "T" and sub-status(SUB-S) is "a" - status(S) is "Z" and sub-status(SUB-S) is "a" $ /opt/nec/ve/bin/veswap -o pid-A 4) Make sure VE process A is in one of the following states - status(S) is "T" and sub-status(SUB-S) is "s" - status(S) is "Z" and sub-status(SUB-S) is "s" $ /opt/nec/ve/bin/veswap pid-A S SUB-S PID T s pid-A 5) After the state of the VE process A becomes one of the following, please check the free VE memory size. - status(S) is "T" and sub-status(SUB-S) is "s" - status(S) is "Z" and sub-status(SUB-S) is "s" Supplement:If there is not enough free VE memory for executing the VE program, please release more memories. $ /opt/nec/ve/bin/free 6) Execute the VE program B with the same executing method as executing a normal VE program. 7) Wait for the completion of the VE program B. 8) Swap in the saved VE memory of the VE process A. $ /opt/nec/ve/bin/veswap -i pid-A 9) Make sure VE process A is in one of the following states - status(S) is "T" and sub-status(SUB-S) is "a" - status(S) is "Z" and sub-status(SUB-S) is "a" When this status(S) and sub-status(SUB-S) is reached, the swapped out VE memory is swapped in, and the VE process A has returned to the state before swapping out the VE memory. $ /opt/nec/ve/bin/veswap pid-A S SUB-S PID T a pid-A 10) Resume the VE process A with "SIGCONT". $ kill -s SIGCONT pid-A ----- 5 Precautions 1) In the swap in, if the processing cannot be continued due to lack of VE memory, the swap in processing will wait until the VE memory is released. As a result, the swap in process may take more time than expected. You can free more VE memory, for example by swapping out another VE process. When the VE memory is released, the swap in process will resume.