VEOS Partial Process Swapping Reference Date: Jun-2023 This document describes the information regarding the VEOS version 3.1.0 or later. 1 Introduction Partial process swapping (PPS) is the feature which saves and restores the VE memory of stopped VE process. It releases VE memory by saving a part of the memory of one or more temporarily stopped VE processes to VH side memory or a file(swap out). Another program can use the released VE memory. After the new program finished, it restores the memory saved in the VH memory or file to the VE memory (swap in), and the execution of the original program is resumed. 1.1 Term definition This chapter defines the terms used in this document. 1) swap area: VH side memory or a file to which a VE process memory is saved. 2) swap out: Saves a part of the memory of VE processes to a swap area, and release VE memory. 3) swap in: Restores the memory saved in a swap area to VE memory, and the execution of program can be resumed. 4) status: Indicates the execution state of the VE process. The status is displayed with the same symbol as the "ps" command. 5) sub-status: Indicates the state of the VE process. It is a PPS original definition to indicate the progress of swap out/swap in processing. The definition of each sub status is as follows. a swap out is not performed, the same as conventional stop state o swap out is processing s swap out is finished i swap in is processing 6) PPS mode: Modes are defined according to the swap area memory. memory mode: swap area is VH memory file mode: swap area is a file. mix mode: swap area is VH memory and file 2 Preset This chapter describes the settings required to use the Partial Process Swapping function. 2.1 The settings of veswap.options 1) The path of veswap.options - To set a common value for all VE nodes /etc/opt/nec/ve/veos/ve-os-launcher.d/veswap.options - To set each VE node individually /etc/opt/nec/ve/veos/ve-os-launcher.d//veswap.options is VE node number We recommend that you set it up in one of the ways. 2) Contents of veswap.options You can set the maximum size of swappable VE memory per VE node and the absolute path of swap file. For file mode, VH's host name and VE node number are added to the file path. If you do not specify the owner of the swap file, the owner will be "root". If you do not specify the group of the swap file, the group will be "root". If you specify the owner of the swap file, you need the write permission of the specified user in the directory of the file path. Please modify "veswap.options" as follows. [memory mode] ve-os-launcher@*=--ve-swap-mem-max= [file mode] ve-os-launcher@*=--ve-swap-file-path= ve-os-launcher@*=--ve-swap-file-max= ve-os-launcher@*=--ve-swap-file-user= ve-os-launcher@*=--ve-swap-file-group= [mix mode] ve-os-launcher@*=--ve-swap-mem-max= ve-os-launcher@*=--ve-swap-file-path= ve-os-launcher@*=--ve-swap-file-max= ve-os-launcher@*=--ve-swap-file-user= ve-os-launcher@*=--ve-swap-file-group= Note: If both of the common setting /etc/opt/nec/ve/veos/ve-os-launcher.d/veswap.options and individual setting /etc/opt/nec/ve/veos/ve-os-launcher.d//veswap.options are set, the individual setting takes precedence. - If different options are specified, both take effect. For example, ------ In /etc/opt/nec/ve/veos/ve-os-launcher.d/veswap.options, ve-os-launcher@*=--ve-swap-mem-max= In /etc/opt/nec/ve/veos/ve-os-launcher.d/N/veswap.options, ve-os-launcher@*=--ve-swap-file-path= ve-os-launcher@*=--ve-swap-file-max= ve-os-launcher@*=--ve-swap-file-user= ve-os-launcher@*=--ve-swap-file-group= ------ All of the above settings are enabled, and PPS mode is MIX mode. - If the same options are specified, individual setting takes effect. For example, ------ In /etc/opt/nec/ve/veos/ve-os-launcher.d/veswap.options, ve-os-launcher@*=--ve-swap-mem-max= In /etc/opt/nec/ve/veos/ve-os-launcher.d/N/veswap.options, ve-os-launcher@*=--ve-swap-mem-max= ------ The individual setting takes effect, the value is size-2(MB unit). 2.2 The settings of HugePages of VH When VEOS starts, VEOS allocates HugePages as follows: memory mode: Memory size is specified size in a configuration file. file mode : Memory size is 64MB. mix mode : Memory size is specified size in a configuration file + 64MB. After "2.1 The settings of veswap.options", the required HugePages whill be set automatically with the following command. ------ $ sudo /opt/nec/ve/sbin/ve-set-hugepages ------ Please see more details in "SX-Aurora TSUBASA Installation Guide". 2.3 Reflection of Settings Restart VEOS with the following command for reflecting "2.1 The settings of veswap.options" ------ $ sudo systemctl restart 've-os-launcher@*' ------ or ------ $ systemctl restart ve-os-launcher@ ------ is VE node number 2.4 Confirm settings After restarting VEOS, you can confirm the settings of veswap.options with the following command. ------ $ ps -C veos -o flag,pid,cmd | grep '^4' ------ For example, ------ $ ps -C veos -o flag,pid,cmd | grep '^4' 4 810086 /opt/nec/ve/veos/sbin/veos -s /var/opt/nec/ve/veos/veos0.sock -d /dev /veslot0 --ve-swap-mem-max=10240 --ve-swap-file-path=/opt/nec/ve/veos/swap-file --ve-swap-file-max=64 -i /var/opt/nec/ve/veos/ived.sock 4 812439 /opt/nec/ve/veos/sbin/veos -s /var/opt/nec/ve/veos/veos2.sock -d /dev /veslot2 --ve-swap-mem-max=10240 --ve-swap-file-path=/opt/nec/ve/veos/swap-file --ve-swap-file-max=64 -i /var/opt/nec/ve/veos/ived.sock 4 812769 /opt/nec/ve/veos/sbin/veos -s /var/opt/nec/ve/veos/veos3.sock -d /dev /veslot3 --ve-swap-mem-max=10240 --ve-swap-file-path=/opt/nec/ve/veos/swap-file --ve-swap-file-max=64 -i /var/opt/nec/ve/veos/ived.sock 4 814029 /opt/nec/ve/veos/sbin/veos -s /var/opt/nec/ve/veos/veos1.sock -d /dev /veslot1 --ve-swap-mem-max=10240 --ve-swap-file-path=/opt/nec/ve/veos/swap-file --ve-swap-file-max=64 -i /var/opt/nec/ve/veos/ived.sock ------ If options are duplicated, individual settings (later one) take effect. For example, ------ 4 810086 /opt/nec/ve/veos/sbin/veos -s /var/opt/nec/ve/veos/veos0.sock -d /dev /veslot0 --ve-swap-mem-max=10240 --ve-swap-file-path=/opt/nec/ve/veos/swap-file --ve-swap-file-max=64 --ve-swap-mem-max=20480 -i /var/opt/nec/ve/veos/ived.sock ------ The individual setting (later one) "--ve-swap-mem-max" takes effect, the value is 20480MB. 3 veswap command This chapter describes the PPS command "/opt/nec/ve/bin/veswap". 3.1 Synopsis /opt/nec/ve/bin/veswap [-o|-i|-m|-n] pid ... [-H] /opt/nec/ve/bin/veswap [-f] size [-o] pid ... [-H] /opt/nec/ve/bin/veswap -s [-H] /opt/nec/ve/bin/veswap -h /opt/nec/ve/bin/veswap -v 3.2 Description 1) No options Displays the status(S) and sub-status(SUB-S) of the specified VE process. Display example: ------ $ /opt/nec/ve/bin/veswap 1001 1002 1003 1004 1005 1006 1007 1008 1009 S SUB-S PID R - 1001 T o 1002 T s 1003 T i 1004 T a 1005 Z o 1006 Z s 1007 Z i 1008 Z a 1009 ------ 2) Details of options -o Swaps out the VE memory of specified VE process to swap area. The VE memory of the VE process that is in one of the following states is able to swap out. - status is "T" and sub-status is "a" - status is "Z" and sub-status is "a" -f Specifies the VE memory size required when executing an interrupt VE process. The unit is KB. VEOS stops swapping out when the size of free VE memory becomes more than or equal to the specified size. The actual VE free memory size may be larger than the specified value due to the VEOS memory management mechanism. -i Swaps in the VE memory of specified VE process from swap area. The VE memory of the VE process that is in one of the following states is able to swap in. - status is "T" and sub-status is "s" - status is "Z" and sub-status is "s" - status is "T" and sub-status is "o" - status is "Z" and sub-status is "o" -m Displays the VE memory size information of the specified VE processes. Displays the size of swappable VE memory (SWAPPABLE) and the size of VE memory swapped out (SWAPPED). The sizes are displayed in KB. -n Displays the sizes of non-swappable VE memory (NON-SWAPPABLE). The size is displayed in KB. -s Displays the swapped out VE memory sizes in VE nodes. The sizes are displayed in KB. -H Not display the header. -h, --help Displays the help and exit. -V, --version Displays version information and exit. 3) Details of parameters pid Specifies the ID of VE process. 4 Example of using This chapter shows an example of swapping out the VE memory of the process (VE process A) of VE program A and executing the VE program B. "Pid-A" in the example is the process ID of the process(VE process A) of VE program A. "size-B" in the example is the VE memory size required to execute VE program B. ----- 1) Stop the VE process A with "SIGSTOP". $ kill -s SIGSTOP pid-A 2) Make sure VE process A is in one of the following states - status(S) is "T" and sub-status(SUB-S) is "a" - status(S) is "Z" and sub-status(SUB-S) is "a" $ /opt/nec/ve/bin/veswap pid-A S SUB-S PID T a pid-A 3) Swap out the VE memory after the state of the VE process A becomes one of the following - status(S) is "T" and sub-status(SUB-S) is "a" - status(S) is "Z" and sub-status(SUB-S) is "a" $ /opt/nec/ve/bin/veswap -o pid-A or $ /opt/nec/ve/bin/veswap -f size-B -o pid-A If you use the "-f" option, VEOS stops swapping out when the size of free VE memory becomes more than or equal to the specified size. In other words, swap out performs until the free memory size displayed by the "ve-free" command reaches the value(size-B(KB)) specified by "-f" option. The actual VE free memory size may be larger than size-B(KB) due to the VEOS memory management. 4) Make sure VE process A is in one of the following states - status(S) is "T" and sub-status(SUB-S) is "s" - status(S) is "Z" and sub-status(SUB-S) is "s" $ /opt/nec/ve/bin/veswap pid-A S SUB-S PID T s pid-A 5) After the state of the VE process A becomes one of the following, please check the free VE memory size. - status(S) is "T" and sub-status(SUB-S) is "s" - status(S) is "Z" and sub-status(SUB-S) is "s" Supplement:If there is not enough free VE memory for executing the VE program, please release more memories. $ /opt/nec/ve/bin/ve-free 6) Execute the VE program B with the same executing method as executing a normal VE program. 7) Wait for the completion of the VE program B. 8) Swap in the saved VE memory of the VE process A. $ /opt/nec/ve/bin/veswap -i pid-A 9) Make sure VE process A is in one of the following states - status(S) is "T" and sub-status(SUB-S) is "a" - status(S) is "Z" and sub-status(SUB-S) is "a" When this status(S) and sub-status(SUB-S) is reached, the swapped out VE memory is swapped in, and the VE process A has returned to the state before swapping out the VE memory. $ /opt/nec/ve/bin/veswap pid-A S SUB-S PID T a pid-A 10) Resume the VE process A with "SIGCONT". $ kill -s SIGCONT pid-A ----- 5 Precautions 1) In the swap in, if the processing cannot be continued due to lack of VE memory, the swap in processing will wait until the VE memory is released. As a result, the swap in process may take more time than expected. You can free more VE memory, for example by swapping out another VE process. When the VE memory is released, the swap in process will resume.