3.3 This section describes changes between the 3.2.2 and 3.3 versions of VEOS. Summary : Reduced the size of VEOS core files. Type : enhancement Detail : Reduced the size of VEOS core files generated upon abnormal termination. Summary : Improved the performance of running binaries compiled for VE10/VE20 on VE30. Type : enhancement Detail : Performace is improved for a program that heavily uses vector floating point fused multiply-add when binaries compiled for VE10/VE20 are running on VE30. Summary : Fixed an issue where VEOS crashes when a multi-threaded MPI program is forcibly terminated or ends abnormally Type : bugfix Detail : VEOS may crash when a multi-threaded MPI program is run and is forcibly terminated with a qdel command or ends abnormally due to an exception. -In NEC MPI-2.23.0/3.2.0 or later, the issue occurs when all of the following conditions are met: -The environment variable NMPI_SWAP_ON_HOLD is specified or "swap_on_hold 1" is specified in VH's /etc/opt/nec/ve/mpi/necmpi.conf -"SWAPPABLE_VESHM" is in the /opt/nec/ve/veos/share/feature_list of all VHs used for the MPI execution -"ENABLE_REDUCING_DMA_VEMEMORY" is in the /opt/nec/nqsv/share/feature_list of all VHs used for the MPI execution -The MPI runs with NQSV's HYDRA process manager -In the queue settings, "NEC MPI Process Manager" is set to "hydra" or "-T necmpi_hydra" is specified in the job submission script. -In NEC MPI-2.20.0~2.22.0/3.1.0, the issue occurs when the following conditions are met: -The environment variable NMPI_SWAP_ON_HOLD is specified or swap_on_hold 1 is specified in VH's /etc/opt/nec/ve/mpi/necmpi.conf Summary : Fix of issue where floating point division by zero operation on "long double" data type doesn't raise floating point exception. Type : bugfix Detail : Even if the environment variable 'VE_FPE_ENABLE' is set to "DIV", a division-by-zero exception doesn't occur during execution when the divisor is zero during division of "long double" type variable (i.e. quadruple precision real number type or quadruple precision complex number type). Summary : Fix of issue when a test program abort in malloc with assertion message Type : bugfix Detail : When the statically-linked executable of VE30 is executed with "VE_LD_LIBRARY_PATH" environment variable, then it may abort in malloc with assertion message. Summary : Fix of issue when "veda-smi" shows incorrect sensor values for VE30 Type : bugfix Detail : The "veda-smi" command shows incorrect current, voltage, and temperature when executed on a machine with VE30. Summary : Fix of "ve-vmstat" incorrect CPU time Type : bugfix Details : The "ve-vmstat" command is showing incorrect CPU time percentage when run with option "delay". Summary : Fix of issue when VE commands doesn't show any error with invalid VE_NODE_NUMBER Type : bugfix Details : The VE commands like ve-sa1, ve-free when run with incorrect VE_NODE_NUMBER such as "-1", "xyz", "abc123" as input, it fails to show any error. 3.2.2 This section describes changes between the 3.2.1 and 3.2.2 versions of VEOS. Summary : Fix the issue where VEOS might crash, when a VE program exits. Type : bugfix Detail : VEOS might crash if a VE program exits while VEOS is executing hourly overflow handling of performance monitor counter (PMC) registers. 3.2.1 This section describes changes between the 3.1.3 and 3.2.1 versions of VEOS. Summary : Adding one more definition into cmake toolchain file Type : enhancement Detail : CMAKE_Fortran_COMPILER_ID is added in cmake toolchain files for VE specific definition. Detail steps are also given in user guide "VE_Cmake_Toolchain_Guide" to update the cmake toolchain file for a specific project. Summary : Fix the issue where floating-point division operations may result in incorrect values Type : bugfix Detail : The VE program when linked with the options "-march=ve3 -static -lm", floating-point division operations may result in incorrect values such as 0.0 or unexpected values due to the issue of libm. 3.1.3 This section describes changes between the 3.1.1 and 3.1.3 versions of VEOS. Summary : Fix the issue where VEOS may cause a segmentation fault, when execution of a program that calls the mmap() system call on a file. Type : bugfix Detail : During the execution of a program that calls the mmap() system call on a file, VEOS may experience a segmentation fault and crash. The issue occurs if all of the following conditions are met: - Another process calls munmap() on the same file while a process is calling mmap() on the file. - The mmap() system call is called with the MAP_SHARED or MAP_PRIVATE flag. - When the MAP_PRIVATE flag is specified, PROT_READ is specified while PROT_WRITE is not. Summary : Fix the issue where VEOS might crash, after swap in Type : bugfix Detail : When the following program is swapped out by the Partial process swapping (PPS) function and then swapped in, VEOS might occasionally crash. -VE program using DMA transfer -MPI program (VEOS 2.14.1 or later and MPI 2.23.0 or later) 3.1.1 This section describes changes between the 3.0.2 and 3.1.1 versions of VEOS. Summary : Improved swap-out performance for "partial process swapping" function Type : enhancement Detail : The time required for swapping out has been reduced by supporting the following two functions. - Limit the amount of VE memory to be swapped out when the VE memory size required for an interrupt VE process is specified in NQSV. This change applies to NQSV from R1.14 onwards. - Swap out to files on DIMMs and NVMe in parallel. Summery : Supported to execute the binary for previous generation Vector Engine (VE10) on the new generation Vector Engine (VE30). Type : enhancement Detail : Supported to execute the binary for previous generation Vector Engine (VE10) on the new generation Vector Engine (VE30). Please see VEOS document "How to Execute VE program" for detail. Summary : Updated process accounting data in Glibc's acct.h Type : enhancement Detail : Some process accounting fields in ve_acct_v15 and ve_acct_v16 structures of acct.h header file are renamed. Below are the changes in ve_acct_v15 and ve_acct_v16 structures: - 'ac_vlcme' is changed to 'ac_llvml' - 'ac_vlcme2' is changed to 'ac_llvme' - 'ac_l1lmc' is changed to 'ac_l1imc' - 'ac_l1lmcc' is changed to 'ac_l1imcc' Below are the changes in ve_acct_v16 structure: - 'ac_llvml' is changed to 'ac_llvsml' Summary : Fix the issue where VEOS uses twice as many huge pages as expected when running swap out Type : bugfix Detail : When Partial process swapping(PPS) is enabled, VEOS uses twice as many huge pages as expected when running swap out. As a result, the following events may occur. - Program run as urgent jobs is failed - The operation become slow because frequent swapping of VH memory Summary : Fix the issue where VEOS may cause a segmentation fault, when Partial Process Swapping (PPS) is enabled. Type : bugfix Detai l : VEOS may causes a segmentation fault, when interrupted VE processes are forcibly terminated by NQSV, because the VE memory required for execution of an urgent request is large and not available after interrupted VE processes are swapped out using PPS. Summary : Fix the issue where 'CMAKE_LIBRARY_PATH' was setting with wrong path Type : bugfix Detail : In cmake toolchain file, CMAKE_LIBRARY_PATH was always set to /opt/nec/ve/lib even if the content of /etc/opt/nec/ve/default.conf was "ve3". Now it will set as "/opt/nec/ve3/lib" if the content of /etc/opt/nec/ve/default.conf is "ve3". Summary : Fixed that VEOS possibly gets stuck by Partial Process swapping(PPS) swap-out Type : bugfix Detail : VEOS possibly gets stuck when Partial Process swapping(PPS) swap-out is executed. Summary : Fixed an issue where abnormal performance monitor counter (PMC) data is recorded in process accounting after signal handling Type : bugfix Detail : There is possibility that too large PMC values are recorded in process accounting data if a VE process continues after an execution of a signal handler. For example, ac_ex and ac_vx in a process accounting file may become greater than 0x0010000000000000. 3.0.2 This section describes changes between the 2.14.1 and 3.0.2 versions of VEOS. Summary : Supported the new generation of Vector Engine (VE30) Type : enhancement Detail : To support the new generation of Vector Engine (VE30), updated these items. - System software - VEOS - VE driver - etc. - Tools and frameworks for programming - glibc v2.31 - autotools, cmake - etc. Summary : Added "ve-" prefix to the names of VE commands Type : enhancement Detail : VEOS provides commands with "ve-" prefix. For example, the name of "ps" command is changed to "ve-ps". If you need commands without "ve-" prefix, please install a suitable package for usage of the machine from the following compatibility packages: ve-command-name-compat-devel ve-command-name-compat-runtime ve-command-name-compat-front-end Summary : Supported Apptainer for VE application execution Type : enhancement Detail : Users can execute VE applications in containers using Apptainer. An Apptainer recipe file to build a image is available on github. https://github.com/veos-sxarr-NEC/singularity Summary : Support for using VE on virtual machines Track# : 2326 Type : enhancement Details : Supported using VE on virtual machines The management software/virtualization program used with VE is libvirt/qemu-kvm. The number of VEs that can be used in a VM guest is basically one. For details, please check "How to use VE in a virtual machine". Summary : Enhancement to support Auxiliary Vector "AT_HWCAP" to determine VE hardware Type : enhancement Detail : Support of an Auxiliary Vector "AT_HWCAP" to determine VE hardware for glibc and profiler. This information is need by glibc and profiler to determine where VE process is running. Summary : Enhancement to support different version of glibc-ve ld.so by default. Type : enhancement Detail : The executables of different VE generation must decide which glibc version ld.so it should load. By default, if VE_LINKER_VERSION environment variable value is not set it should loads the executable of VE30 by glibc 2.31 ld.so. Similarly, the executable of VE10 invokes glibc 2.21 ld.so by default. Please see VEOS document "How to Execute VE Program" for more information. Summary : Added a new wrapper script for the ve-ldd and ve-ldconfig Type : enhancement Detail : ve-ldconfig: The wrapper script of ldconfig is responsible to execute the different version of ldconfig provided by different version glibc-ve. ve-ldd: The wrapper script of ldd is responsible to execute the different version of ldd binary provided by glibc-ve as per the VE_LINKER_VERSION environment variable value and VE30 ldd existence. Summary : Enhancement about accelerated I/O Type : enhancement Detail : Updated accelerated I/O as below; - Enabling accelerated I/O by default. - Reducing the usage of HugePages and VE memory for accelerated I/O. Accelerated I/O is implemented in libraries in libsysve-ve3 and libsysve-ve1. The latest version of them require veos-3.0.2 or later due to changes in the implementation of the accelerated I/O. If libsysve-ve3-3.0.2, libsysve-ve1-3.0.2 or later versions of them are installed with the veos-2.14.1 or earlier, the accelerated I/O will be disabled. Summary : Support VEO API to get VE architecture number Type : enhancement Detail : Support VEO API to get the VE architecture number of the specified node. Please see VEOS document "The Tutorial and API Reference of Alternative VE Offloading" for detail. Summary : Support asynchronous execution of VH function when using VEO Type : enhancement Detail : When VH functions are executed using VEO API, it is executed asynchronously. Please see VEOS document "The Tutorial and API Reference of Alternative VE Offloading" for detail. Summary : Reduction in the number of HugePages required to use VEO Type : enhancement Detail : The number of HugePages required to use VEO has been reduced from 32 to 16 pages. The number of HugePages required to use VEO depends on the environment variable VE_OMP_NUM_THREADS. Please see VEOS document "The Tutorial and API Reference of Alternative VE Offloading" for detail. Summary : Added support of a new environment variable VE_CORE_LIMIT to specify a set of cores for VE process scheduling. Type : enhancement Detail : Using the environment variable VE_CORE_LIMIT user can specify VE cores while executing a VE program. e.g. VE_CORE_LIMIT=0-1,3,5-7 (available VE core is 0,1,3,5,6 or 7) Summary : Enhanced process accounting feature Type : enhancement Detail : Improved process accounting feature as below; - Added ac_corebitmap attribute - Updated the output of ac_max_nthread field. It will provide the 'maximum number of threads whose state are RUNNING or WAIT at the same time in the process's lifecycle'. In prior versions of veos, this field shows the 'number of threads executed in the process's lifecycle'. - Added performance monitor counter attributes for VE30 To provide the feature, data structure version of process accounting is changed. Data structure for VE10 is version 15, and the structure for VE30 is version 16. NQSV, ve-dump-acct and ve-lastcomm displays the attributes. When user updates VEOS v2.x to v3.0.2, existing data file is converted to version 15 during psacct-ve installation. VEOS 3.0.2 and later require NQSV R1.13-143 or later due to changes in data structure of process accounting. If VEOS 3.0.2 is installed with the previous version of NQSV, NQSV will fail to get process accounts. Summary : Improved the overflow handling of performance monitor counter (PMC) registers. Type : enhancement Detail : Improved the overflow handling of performance monitor counters which are used for PROGINF, etc. Now, veos can handle PMC values overflows multiple times and accumulate the overflowed values into final PMC values. Previously, veos can handle PMC overflow only one time. Summary : Improve performance of memory allocation and deallocation by a VE task Type : enhancement Detail : Updated the handling for memory allocation and deallocation requests from a VE task which improves the latency of mmap(), munmap(), malloc() and free() in a VE program. Summary : VE Driver and V2P Driver Build from SRPM Type : enhancement Details : VE driver, V2P driver now support building from SRPM. Users can now build VE,V2P from SRPM as needed. Summary : Updated the automatic HugePages configuration service Type : enhancement Detail : Updated the automatic HugePages configuration service (ve-set-hugepages.service) as below; - Require the minimum HugePages based on VE architecture - Limit the HugePages size to the half of the main memory in virtual machine environment Please see "SX-Aurora TSUBASA Installation Guide" for detail. Summary : 'mpinfort' was giving error while compiling OSS package using VE cmake toolchain file. Type : enhancement Detail : CMAKE_LINKER '/opt/nec/ve/bin/nld' was replaced with "/usr/bin/true" in cmake toolchain file to avoid the addition of 'libnfort' and other unnecessary libraries. Summary : Fix of segmentation fault in dlopen(). Type : bugfix Detail : When user program is continuously loading libraries using dlopen(), at certain point of time address will reach the limit of prescribed Area for library loading under 64GB and further library loading will fail. This results in segmentation fault in dlopen(). Summary : Fixed an issue where abnormal performance monitor counter (PMC) data is recorded in process accounting Type : bugfix Detail : There is possibility that too large PMC values are recorded in process accounting data if VE processes terminates abnormally and VEOS generates core files in parallel. To avoid the issue, two VEOS processes run per VE node. Summary : Fixed VE 'gdb' command can not catch C++ exceptions Type : bugfix Detail : VE 'gdb' command cannot catch C++ exceptions. Summary : Fixed an issue of failure to allocate buffer for data transfer between VH and VE when using VEO Type : bugfix Detail : When using VEO API to transfer data between VH and VE, the VEO process may time out waiting for buffer release from both VH and VE to allocate buffers. In this case, the data transfer will fail and the error message "alloc_payload timed out!" will be displayed. Summary : Fix of VEDA_FTRACE handle to work correctly. Type : bugfix Detail : When user specified VEDA_FTRACE, ftrace output file was not generated. VEDA_FTRACE handle is corrected to generate ftrace output file. Summary : Fix for VEDA device code to compile with NEC C compiler Type : bugfix Detail : User were failed to compiled VEDA device code with NEC C Compiler. Summary : Fixed an issue where EXSRAR TIMEOUT was reported Type : bugfix Detail : EXSRAR TIMEOUT may be reported because VEOS and ve_exec are not synchronized when VEOS stops and resumes VE core. 2.14.1 This section describes changes between the 2.13 and 2.14.1 versions of VEOS. Summary : Updated VE commands to perform for nodes which NQSV job scheduler allocates. Type : enhancement Detail : Some VE commands (like ps and free) are updated to perform for nodes which NQSV job scheduler allocates. Please refer to VEOS document 'Difference Points for Commands' for detail'. Summary : Support to swap out of shared memory between VE to VE. Type : enhancement Detail : When MPI program is swapped out, increase the amount of swap-out memory and reduce the case which the low priority job is killed. Please see "NEC MPI User's Guide" for the setting method in MPI and for supported models. Summary : Support VE_SIGPROCMASK environment variable to set signal mask on start of VE program Type : enhancement Detail : User can set signal mask when the VE program is started by specifying the signal number in the environment variable VE_SIGPROCMASK. Please see VEOS document "How To Execute VE program" for detail. Summary : Added new features in VEDA Type : enhancement Details : Following enhancements are done in VEDA - - Enabled support for vedaMemsetD2D* set of APIs after Bugfix. Summary : Fix of issue where VE process does not inherit the signal mask from x86 process after execve(). Type : bugfix Detail : The signal mask is not preserved when a VE program is executed after execve() from a VH program. Hence any signal which is blocked in the VH process, will not be blocked in the VE process. Summary : Fix of issue where ve_set_next_thread_worker() always returns an error Type : bugfix Detail : ve_set_next_thread_worker() always returns an error even if it is working properly. Summary : Fix of issue where VEOS may sometimes terminate abnormally when a VE process becomes zombie. Type : bugfix Detail : In a multithreaded process if the main thread terminates and becomes zombie while VEOS is serving a VH-VE SHM request for it, VEOS may terminate abnormally. Summary : Fix of issue where poll() system call fails if 'nfds' is more than 7 Type : bugfix Detail : poll() system call fails with errno as EINVAL if the number of poll file descriptors, i.e. 2nd argument 'nfds', is greater than 7. Summary : Fix of issue where pselect(), ppoll() and epoll_pwait() system calls behave incorrectly if signal mask is passed as NULL. Type : bugfix Detail : If NULL is provided as signal mask in the pselect(), ppoll() and epoll_pwait() system calls as their last argument 'sigmask', then all signals for the VE task is unmasked during the system call execution. Summary : Fix of issue where incorrect Performance Monitor Counter (PMC) values are captured for child process/thread in process accounting. Type : bugfix Detail : Incorrect PMC values will be captured in process accounting for the child process/thread. - PMC values of child process will be larger than actual when parent process invokes fork/vfork after some procedures executed. - PMC values of process itself will be larger than actual when the process invokes pthread_create() after some procedures executed. - PMC values of process itself will be larger than actual when the process uses OpenMP "-mno-create-threads-at-startup" option or automatic parallelization with the option. This will be visible when process accounting is displayed by VE lastcomm, dump-acct command or NQSV. Affected NQSV values are below. VE FPEC, INSTRCT, VEC-INST, VEC-ELMT, VEC-EXE, VE L1 CACHE MISS TIME, VE CPU PORT CONFLICT, VEC ARITH EXEC TIME, VEC LOAD EXEC TIME, VEC-INST LOAD, VE POWER THROTTLING, VE THERMAL THROTTLING, VE MFLOPS, VE MFLOPS(REAL), V. Op. Ratio, FMAE Summary : Fix of issue where VE 'taskset' command doesn't set the required cpu affinity mask to a VE program correctly. Type : bugfix Detail : The cpu affinity mask given to VE 'taskset' command using '-c' option is not correctly applied to the VE program. As a result, a VE program which uses OpenMP may freeze. Summary : Fix of issue where a VE process cannot be swapped out by "Partial Process Swapping" if the VE process receives SIGSTOP during creation of a child thread Type : bugfix Detail : If a VE process receives SIGSTOP during creation of a child thread, internal state of the VE process may get illegal. In this case, 'veswap' command returns failure if swap-out for the VE process is performed. Summary : Fix of issue where "Non Swappable Memory Size Used" is not displayed if a VE program is running in the container Type : bugfix Detail : "Non Swappable Memory Size Used" is displayed if a VE program is running with environment variable VE_PROGINF=YES on a host. However, it is not displayed if a VE program is running with environment variable VE_PROGINF=YES in a container(eg.docker) 2.13 This section describes changes between the 2.11.1 and 2.13 versions of VEOS. Summary : Added new features in VEDA Type : enhancement Details : Following enhancements are done in VEDA - Added support of below new VEDA APIs - Below APIs are removed - Added support of below new VERA APIs - Disabled support for vedaMemsetD2D* set of APIs. - Replaced "bool checkResult" with "int64_t* result" in vedaLaunchKernelEx() to receive return value of kernel. - Replaced "const void *value" to "const VEDAhmemptr value" in vedaArgsSetHMEM Program linking libveda.so needs to be re-compiled when veoffload-veda is updated to v2.13 or later Summary : Added more definitions into cmake toolchain file Type : enhancement Detail : cmake toolchain files are updated. CMAKE_C_COMPILER_ID, CMAKE_CXX_COMPILER_ID and CMAKE_STRIP are added in toolchain files for VE specific definition. Summary : Support optional arguments of VH Call Fortran Type : enhancement Detail : Support API to allocate arguments of VH library function which has optional arguments. Please refer to "The Tutorial and API Reference of VH Call Fortran API" Summary : Supported ftrace output when multiple VE processes are created from VEO Type : enhancement Detail : When a program linked with the ftrace library is executed by creating multiple processes via VEO, performance information files with different filenames are output for each VE process. Summary : Supported low-level VEO API to hook memory allocation and deallocations. Type : enhancement Detail : Low-level VEO API is added to hook memory allocation and deallocation. These API are low-level API and are intended to be called by upper layer software. These API are not intended to be called by a user program. Please see VEOS document "The Tutorial and API Reference of Alternative VE Offloading" for detail. Summary : Improved behavior when VEO API wait for the result of offloaded processing Type : enhancement Detail : If aveorun which is helper program terminates while waiting for the result of the offloaded processing using VEO, the VEO API seems to be hanged. This behavior has been improved and the API now returns a failure result. Summary : Fix of segmentation fault issue of VEDAdeviceptr Type : bugfix Details : Passing VEDAdeviceptr to std::ostream may cause a segmentation fault. Summary : Fix of issue where some VEO APIs return a success value if an VE exception occurs Type : bugfix Detail : If any of the signals SIGABRT, SIGFPE, SIGILL, SIGSEGV, or SIGBUS is received during the processing on the VE side when calling the following VEO APIs, the API will return a value indicating success even though it failed. - veo_load_library() - veo_get_sym() - veo_alloc_mem() Summary : Fix of issue where fortran source was unable to be compiled using VE libtool Type : bugfix Detail : When a fortran program is compiled using libtool then it does not recognise '-rpath' option because '-Wl' was not set before this option in VE libtool. Summary : Fix of issue of VE libtool where '-Wl' and '-fPIC' options were missing for nfort compiler Type : bugfix Detail : When a fortran program is compiled using libtool then it does not recognise '-soname' option because '-Wl' was not set before this option in VE libtool. Also during compilation, '-fPIC' was also missing to create the dynamic library using nfort. 2.11.1 This section describes changes between the 2.10.1 and 2.11.1 versions of VEOS. Summary : Supported Mellanox OFED version 5 Type : enhancement Detail : VEOS supports data transfer using Mellanox OFED version 5. Please note Mellanox OFED version 5 can be used only with RHEL 8.4 / 8.5. Please see "SX-Aurora TSUBASA Installation Guide" for detail. The memory map of VE processes is changed to support Mellanox OFED version 5. The available VE virtual addresses have been limited to 0x6000 0000 0000 --0x6fff ffff ffff. Please see "High level design" for detail. "ve_exec" process uses about 32 TB and some hundreds mega bytes virtual memory, while previously used 1TB and some hundreds mega bytes virtual memory. If you limit the size of virtual memory using "-l vememsz_prc" option, "-l vmemsz_job" option or "--vmemsz-lhost" option of NQSV, please update the size of virtual memory to allow "ve_exec" process to use 32TB and some hundreds mega bytes virtual memory. Please note you need not change the configuration if you use the default values of these options, because the default values are UNLIMITED. Please see the description of qsub, qrsh, qlogin, qalter in "NQSV User's Guide Reference" for these options. If you limit the size of virtual memory using /etc/secure/limits.conf please update the size of virtual memory to allow "ve_exec" process to use 32TB and some hundreds mega bytes virtual memory. Summary : VEDA support Type : enhancement Detail : A new feature, VEDA (VE Driver API), is supported. VEDA makes it easy to port CUDA programs to VE and provides; - CUDA Driver and Runtime API-like API for SX-Aurora TSUBASA programming - cmake modules to build programs for VE and x86_64 Please refer to VEOS document "The Tutorial and API Reference of VEDA" for detail. Summary : Supported new features to show VE product name Type : enhancement Detail : VE 'lscpu' command is updated to show "Model name" as VE product name. New libsysve API 've_get_ve_product_name' shows VE product name of VE node on which VE program is running. Summary : Changed HugePages set by ve-set-hugepages.service Type : enhancement Detail : The number of HugePages set by ve-set-hugepages.service is changed. Please refer to "SX-Aurora TSUBASA Installation Guide" regarding required HugePages. Summary : Supports VEO VESHM, VEO VH-VE SHM and VEO VEDMA API as low-level API. Type : enhancement Detail : Using these API, it is now possible to transfer data between VEs on the same VH. On the other hand, data transfer between VEs on the different VH is not supported. These API are low-level API and are intended to be called by upper layer software. These API are not intended to be called by a user program. Please see VEOS document "The Tutorial and API Reference of Alternative VE Offloading" for detail. Summary : Supported new VH Call for Fortran APIs Type : enhancement Detail : User can pass the following data types to function or subroutine invoked on VH side by using libvhcall-fortran APIs. See VEOS document "The Tutorial and API Reference of VH Call Fortran API". - INTEGER(kind=8) - An array of INTEGER(kind=8) - An array of CHARACTER Summary : Improve log messages of VEOS Type : enhancement Detail : VEOS makes a record of a received signal and its sender into the log file when VEOS terminates. Summary : Providing "debuginfo" packages of libvhcall-fortran for RHEL8 Type : enhancement Details : VE 'gdb' command can debug VE binaries provided by libvhcall-fortran package. Summary : Fix of issue where veos may hang while delivering signal to a VE process. Type : bugfix Detail : While sending a signal to a VE process, VEOS may hang and become non-responsive. Summary : Fix of issue where the signal mask provided in pselect(), ppoll() and epoll_pwait() system calls doesn't take effect. Type : bugfix Detail : The signal mask provided in the pselect(), ppoll() and epoll_pwait() system calls, as their last argument 'sigmask', doesn't take effect if a signal is delivered during preprocessing of the system calls. Summary : Fix of issue where error messages related to loading are insufficient Type : bugfix Detail : If the binary file or the shared object cannot be opened for any reason at the time of loading then that object name is not displayed to a log file and also to stderr. Summary : Fix of issue where large size stack memory expansion fails in VE program. Type : bugfix Detail : When a large amount of stack memory expansion is performed in VE program, it fails to expand stack due to VH side insufficient memory and the program terminates with 'Unable to grow stack' error. Summary : Fixed of issue where memory leak occurs when VESHM attach fails Type : bugfix Detail : VEOS does not free a small area of memory on VH side, when a VE process fails to attach a VESHM area. It fails when the specified VESHM area to be attached does not exist. Please note NEC MPI uses VESHM functionality. Therefore, a memory leak may occur when MPI communication fails. Summary : Fix of the AVEO issue where deadlock occur when transferring arguments larger than 31MB of VE function. Type : bugfix Detail : If an argument which is larger than 31MB is transferred by veo_call_async() or veo_call_sync() in a multi-threaded VEO program, a deadlock may occur depending on the timing. Summary : Fix of issue that the return value and error number may be incorrect when write request using Accelerated I/O fails. Type : bugfix Detail : When write request more than 8MB using Accelerated I/O fails, the return value is 0 and the error number is not set. Summary : Fix of some issues with VE "gdb" Type : bugfix Details : Fixed the following issues with VE "gdb" - gdb terminates abnormally with SIGSEGV when "map" or "sleep" command without arguments are executed. - gdb prints an invalid value when printing "long double" type with the "print" command. - gdb returns an invalid value when "return" command is executed for float-type function. - When a function with "long double" type argument is called with the "call" command, the value is not passed to the function correctly. - After executing "set backtrace exclude-start-thread", "backtrace" command prints "start_thread" and "__clone" functions. - "backtrace" command may print address 0. Summary : Fix of issue where "ve_sysstat-collect@N" fails to start Type : bugfix Details : "ve_sysstat-collect@N" fails to start every ten minutes, if the corresponding VE card have been removed or the VE node number assignment have been changed, after the "sysstat-ve" package installation. The issue occurs on RHEL8. The issue does not occurs on RHEL7. N is VE node number in the above description. 2.10.1 This section describes changes between the 2.9.1 and 2.10.1 versions of VEOS. Summary : Supported FTRACE feature of aveorun. Type : enhancement Detail : In addition to the performance information of the user program, it is now possible to get the performance information of aveorun which is helper program. To get the performance information, set the environment variable VEORUN_BIN to /opt/nec/ve/libexec/aveorun-ftrace and run the program that uses VEO. Then ftrace.out file is generated by ftrace runtime libraries. Summary : Support of traceback with AVEO. Type : enhancement Detail : It is now possible to get the traceback information on VE program. To get the traceback information, compile the VE program with "-traceback", set the environment variable VE_TRACEBACK to VERBOSE, and run the VEO program. Please see VEOS document "The Tutorial and API Reference of Alternative VE Offloading" for detail. Summary : Supported the new API "mallinfo2" which is an extension of existing API "mallinfo". Type : enhancement Detail : When malloc information is retrieved using "mallinfo", an overflow may occur because its information is stored into integer variables. "mallinfo2" stores retrieved malloc information into size_t variables to avoid this overflow. Summary : Removed the limitation of maximum number of environment variables passed to the VE program Type : enhancement Detail : The limitation of maximum number of environment variables is removed. User can pass environment variables to VE program as much as a VH program. Summary : The performance improvement of VE AIO Type : enhancement Detail : The performance of VE AIO is improved by processing I/O request in parallel. When the IO size is 16MB or more, The performance of ve_aio_read() is improved up to 2.5 times faster, and the performance of ve_aio_write() is improved up to 2 times faster to the conventional performance. Please note that VE AIO gets non atomic by default by this improvement. User can use VE AIO as atomic I/O as before. In this case, the performance of VE AIO is not improved. Please see VEOS document "The Tutorial and API Reference of libsysve" for more detail. Summary : Providing "debuginfo" packages of VE binary for RHEL8 Type : enhancement Details : VE 'gdb' command can debug VE binaries provided by the following packages. - glibc-ve - libsysve - veoffload-aveorun Summary : Improved swap out of data transfer area by MPI Type : enhancement Details : The area where data was transferred by MPI could not be swapped out by "Partial Process Swapping". It is possible to swap out some of them. Please see "NEC MPI User's Guide" for the setting method in MPI and for supported models. Summary : Fix of issue where VE process accounting file might get corrupted Type : bugfix Detail : If VE process accounting is switched from 'OFF' to 'ON' state during VE program execution, then there is a possibility that VE process accounting file may contain an invalid record. This results in corruption of process accounting file. Summary : Fix of issue where VE memory may be leaked Type : bugfix Detail : If a VE process is killed abnormally during creation of new task using clone() system call, the VE memory required by new task may be leaked. Summary : Fix of issue where a small VH memory is leaked in pseudo process Type : bugfix Detail : When shmat() system call fails with EINVAL due to invalid shmid value, a small VH memory is leaked in pseudo process. Summary : Fix of issue where VE 'ps' command may cause segmentation fault in VEOS Type : bugfix Detail : If VE process is getting terminated and VE 'ps' command is executed at the same time then VEOS may terminate with segmentation fault. Summary : Fix of thread safety issue for AVEO. Type : bugfix Detail : If the following APIs are executed in a multi-threaded program, it may cause unexpected behavior. For example, it may destroy or reference unexpected contexts or processes and cause abnormal termination. - veo_context_open() - veo_context_close() - veo_free_hmem() - veo_get_context() - veo_hmemcpy() - veo_num_contexts() - veo_proc_destroy() Summary : Fix of issue where VE memory is leaked in AVEO. Type : bugfix Detail : If a VE thread created by veo_context_open() is closed by veo_context_close(), a memory leak will occur. However, if the VEO process is terminated or VE process is destroyed by veo_proc_destroy(), no memory leak will occur. Summary : Fix of issue where veo_cotnext_sync() returned before commands finished. Type : bugfix Detail : If veo_conctext_sync() is executed in a multithreaded VEO program, it may return without waiting for all commands to be processed. 2.9.1 This section describes changes between the 2.8.3 and 2.9.1 versions of VEOS. Summary : Added support of cmake Type : enhancement Detail : Users can use cmake to build a new program or port an existing program by specifying a toolchain file with option '-DCMAKE_TOOLCHAIN_FILE'. cmake-ve package provides two different toolchain files to build MPI and non-MPI program. Please see VEOS document "VE CMake Toolchain User Guide" for detail. Summary : Enhancement of autotools for VE Type : enhancement Detail : VE autotools are updated to build a new program or port an existing program for VE. Please see VEOS document "VE autotools Users Guide" for detail. Summary : Supported automatic HugePages configuration service Type : enhancement Detail : The service, ve-set-hugepages.service, calculates the required number of HugePages and the overcommit HugePages. Then it sets them to vm.nr_hugepages and vm.nr_overcommit_hugepages during the VH bootstrap. For more detail, please refer to "SX-Aurora TSUBASA Installation Guide" and the VEOS document, "How to execute VE program". Summary : Support a new command 'veosctl' to display and update time-slice and timer-interval of VEOS scheduler Type : enhancement Detail : Using 'veosctl' command, all users can display the current time-slice and timer-interval of VEOS scheduler. Privileged user can update the VEOS scheduler's time-slice and timer-interval dynamically using 'veosctl' command without VEOS restart. Please see VEOS document "How to Execute VE Program" for more information. Summary : The performance improvement of Accelerated I/O Type : enhancement Detail : When the IO size is 4MB or more, the performance of Accelerated I/O is improved up to 1.5 times faster to the conventional performance. Please note that read/write family system calls will not be atomic when the IO size is more than 4MB. Please see VEOS document "How To Execute VE program" for detail. Summary : Supported an environment variable and an API which indicate Accelerated I/O usage. Type : enhancement Detail : Users can set the environment variable VE_ACC_IO_VERBOSE=1 to display whether accelerated IO is enabled or disabled to standard error when a VE process exits. A new API 've_is_acc_io_enabled()' which returns the status, enabled or disabled, is also available. Please see VEOS document "The tutorial and API reference of libsysve" for detail. Summary : Updated the behaviour to abort the VE task when stack limit is or VE memory is exhausted. Type : enhancement Detail : VEOS is updated to abort the VE task with the message "Unable to grow stack" when stack limit of the VE task is or VE memory is exhausted. Summary : Fix of issue where veos may terminate abnormally with SIGSEGV when executing a VE process with VE 'time' command Type : bugfix Detail : If VE process executed using VE 'time' command creates a child VE process, veos may sometimes terminate abnormally with SIGSEGV. Summary : Fix of issue where a System V shared memory segment which is not marked for destruction may be removed Type : bugfix Detail : For a System V shared memory segment, if S_IXOTH is set to its permission mode flags while creating it using shmget() system call, the segment is removed after its reference count decreases to zero even if it is not marked for destruction. Summary : Fix of issue where shmctl() system call may fail to mark a System V shared memory segment for destruction and a small VH memory leak Type : bugfix Detail : If a shared memory segment's permission is set as zero while creating it by shmget() system call, IPC_RMID command of shmctl() system call fails to mark it for destruction and errno is set to EACCESS. Due to this a small VH memory is also leaked in VEOS. Summary : Fix of issue where IPC_SET command of shmctl() system call fails to change permission mode of a System V shared memory segment to 0 Type : bugfix Detail : When trying to change permission of a System V shared memory segment to 0 using shmctl() system call's IPC_SET command, it failed to change permissions of the shared memory segment although shmctl() returns success. Summary : Fix of issue where VE 'time -v' command shows incorrect values for "Page size" and "Maximum resident set size" fields in the output Type : bugfix Detail : When a VE process is executed using VE 'time -v' command, "Page size" and "Maximum resident set size" fields are incorrectly shown as 0 for the VE process. Summary : Fix of issue where tracee process does not terminate when VE 'strace' command killed with SIGKILL. Type : bugfix Detail : When a VE process is executed using ported 'strace' command and if strace is killed with SIGKILL then tracee process does not proceed ahead for further execution and does not terminate. 2.8.3 This section describes changes between the 2.8.2 and 2.8.3 versions of VEOS. Summary : Fix of issue where a ve_exec remains in zombie state in VH. Type : bugfix Detail : In a multi-threaded process, if the process is terminated while a pthread_create() request was in progress at that time, then there is a possibility that VE process is not deleted from VEOS and a ve_exec may remain in zombie state in VH. 2.8.2 This section describes changes between the 2.8.1 and 2.8.2 versions of VEOS. Summary : Support for environment variable that makes VE process debugging easier for VEO. Type : enhancement Detail : When users want to debug VE process, set environment variable VEO_DEBUG first. Then users execute VEO program and a VE debugger gdb starts whenever a VE process starts. Please see VEOS document "The Tutorial and API Reference of Alternative VE Offloading" for detail. Summary : Supported two new VEO APIs. Type : enhancement Detail : The first API is veo_version_string() which returns VEO version. The second API is veo_proc_identifier() which returns VE process identifier. VEO can identify up to 16 VE processes. VEO internally assigns a number from 0 to 15 to the VE process and treats it as VE process identifier. If there are unused the numbers, the smallest number will be assigned when a VE process is created. Summary : Improved the handling for memory allocation and deallocation related system call e.g mmap(2) and munmap(2) system call in VEOS to improve the performance of VE task. Type : enhancement Detail : Performance of VE task may degrade significantly when a VE task performs memory allocation and deallocation related system call e.g mmap(2). This is because after handling of system call, VEOS may not able execute some of the threads of VE task instantly, rather execution of these threads are deferred to next scheduling cycle. Summary : Updated libvhve header files for C++ programs Type : enhancement Detail : The header files, vhve.h is updated so that C++ user programs include them. Summary : Added a new script for the job scheduler confirms status of VEs Type : enhancement Detail : The job scheduler confirms that VEs status are ONLINE before start job. And it confirms that there were no failure of VEs during job running too. Summary : Fix the issue where core allocation among the VE task is biased in the partitioning (NUMA) mode Type : bugfix Detail : When the VE tasks running on VE NUMA mode are traced then the core allocation among the traced or non-traced VE tasks may become biased i.e a VE task executed without --cpunodebind option may be allocated a high load NUMA node. Summary : Fix of issue where memory is leaked at VH side Type : bugfix Details : Small VH memory may be leaked in VEOS when a VE process creation failed due to not enough memory at VH. Summary : Fix of issue where vector registers of VE process are not set by ve_ptrace(). Type : bugfix Detail : When ve_ptrace() sets the vector registers of VE process by PTRACE_SETREGSET request, desired values are not set. Hence, user can not set desired values to the vector registers by using gdb which uses ve_ptrace() internally. Summary : Fix of issue ve_drv service will not starts after update of ve_drv-kmod package fails. Type : bugfix Detail : If you update ve_drv-kmod package without '/opt/nec/ve/sbin/terminate-all-ve-service' command, update fails and then ve_drv module will not loaded automatically on system reboot. Summary : Fix of issue where VEO API calls increase VH memory usage. Type : bugfix Detail : The following VEO APIs cause a memory leak at VH side. If users call these APIs repeatedly, VH memory usage continues to increase. - veo_call_async() - veo_call_sync() - veo_call_async_vh() - veo_read_mem() - veo_write_mem() - veo_async_read_mem() - veo_async_write_mem() 2.8.1 This section describes changes between the 2.7.6 and 2.8.1 versions of VEOS. Summary : gdb supports displaying vector registers as integers. Type : enhancement Detail : gdb has displayed vector registers (v0-v63) as floating point numbers, but now it displays integers in addition to floating point numbers. Summary : Improved handling of the arguments transferred by veo_call_async(). Type : enhancement Detail : veo_call_async() internally saves the arguments. Therefore, user can rewrite the arguments immediately after the veo_call_async() call and make the next call. Summary : Fix of issue where new VE process or thread creation may fail due to VE process data duplication in veos. Type : bugfix Detail : Creation of new task using fork(), vfork() or clone() may fail in veos if data of another task having the same pid already exists in veos in zombie state. With default kernel 'pid_max' value ('/proc/sys/kernel/pid_max'), occurrence of this issue is quite rare. However, with low 'pid_max' value this issue is observed frequently. Please see the next topic related to this issue. Summary : Fix of issue where data of a VE process created with VE 'time' command is not cleaned up from veos if the 'time' process itself is killed. Type : bugfix Detail : A VE process created using VE 'time' command is not removed from veos if the 'time' process which created it is terminated before the VE process exits. In this case the resources allocated to the VE process is also not released. Summary : Fixed the issue where the VH process would wait forever when VEO program causes SIGBUS. Type : bugfix Detail : If the VE process generated by VEO causes SIGBUS while the VH process is waiting for the processing result of the VE process, the VE process will be terminated. However, the VH process will wait forever for the result because the VH process is not notified of the termination of the VE process. Summary : Fix of issue where VE program cause a segmentation fault Type : bugfix Detail : There is an issue where a VE program causes a segmentation fault when you execute it at first time after you install RHEL8 or CentOS8 and SX-Aurora TSUBASA software into SX-Aurora TSUBASA. This issue also occurs when the number of huge pages is not sufficient. 2.7.6 This section describes changes between the 2.7.5 and 2.7.6 versions of VEOS. Summary : Support for MPI programs which call VEO API Type : enhancement Detail : VH side MPI program can use VEO functionality by using VEO API (AVEO API). Please see VEOS document "The Tutorial and API Reference of Alternative VE Offloading" and MPI document "NEC MPI User's Guide" for detail. Summary : Supported heterogeneous memory(HMEM) API of AVEO. Type : enhancement Detail : Users can use HMEM API to allocate VE/VH buffers. The memory allocated by HMEM API is used as heterogeneous memory. HMEM API for allocating memory returns the address with an identifier used by the VEO library. This allows users to transfer data between VE and VH process simply by specifying the address to HMEM API without explicitly specifying the process. Please see VEOS document "The Tutorial and API Reference of Alternative VE Offloading" for details and notes. Summary : Disabled generation of core file if VE program is executed when VE hardware state is UNAVAILABLE Type : enhancement Detail : Disabled core dump generation of 've_exec' process on VH if VE hardware state is UNAVAILABLE. Summary : Enhancement of VH Call to make symbols available for subsequently loaded libraries. Type : enhancement Detail : vhcall_install() loads a library to a VH process by dlopen() with RTLD_GLOBAL option. Symbols are made available for reference resolution in subsequently loaded libraries. Summary : Fix of issue that VEOS start failed Type : bugfix Detail : VE driver initialization may fail at starting.As a result,VEOS could not start. Summary : Fix of issue where VEOS may stop responding while process's coredumping is in progress. Type : bugfix Detail : In a multi-threaded VE process, if SIGKILL is received by a thread while coredumping was already in progress on behalf a different thread, then coredumping process doesn't terminate immediately. During this time VEOS stops responding while coredumping is in progress and again becomes responsive after completion of coredump. Summary : Fix of segmentation fault in pidstat command. Type : bugfix Detail : If pidstat command is used in multithreaded environment with interval option, then it might lead to a situation leading to segmentation fault due to insufficient memory allocation. 2.7.5 This section describes changes between the 2.7.4 and 2.7.5 versions of VEOS. Summary : Supported environment variables of AVEO to optimize data transfer Type : enhancement Detail : Users can set the below environment variables to optimize data transfer between VH and VE. VEO_RECVFRAG, VEO_RECVCUT VEO_SENDFRAG, VEO_SENDCUT Please see VEOS document "The Tutorial and API Reference of Alternative VE Offloading" for detail. Summary : Fix of issue where multithreaded VH side programs that invoke VEO APIs lead to unpredictable results. Type : bugfix Detail : VEO APIs are not thread-safe. If two or more threads invoke VEO APIs in parallel, the behavior is unpredictable. Summary : Fix of issue where VE MPI process remains in zombie state. Type : bugfix Detail : If one of threads which is created by OpenMP or automatic parallelization of NEC compiler causes an exception (e.g. segmentation fault) during another thread invokes MPI function, ve_exec may remains in zombie state after its termination. Summary : Fix of issue where VE process remains in zombie state if uid of process invoking 've_get_regvals()' is different from the uid of requested process. Type : bugfix Detail : If uid of a process invoking 've_get_regvals()' api, included in libveosinfo, is different from the uid of the target process then the target process remains in zombie state after its termination. Summary : Fix of issue where the result of a request that raises an exception frequently becomes success value. Type : bugfix Detail : If the user specifies an incorrect address for veo_write_mem() or veo_async_write_mem() and the function raises an exception, the result of the request could be a success value. Subsequent VEO API calls may fail even if the two functions return success value. 2.7.4 This section describes changes between the 2.7.2 and 2.7.4 versions of VEOS. Summary : Supported Singularity for VE application execution Type : enhancement Detail : Users can execute VE applications in containers using Singularity. A Singularity recipe file to build a image is available on github. https://github.com/veos-sxarr-NEC/singularity Summary : Supported AVEO with PROGINF. Type : enhancement Detail : AVEO analysis information is available in PROGINF. To get the information, specify YES or DETAIL in the environment variable VE_PROGINF when executing the AVEO program. Summary : Supported features to set owner and group of swap file of "Partial Process Swapping" Type : enhancement Detail : When swapping out VE memory to a file, enable to set owner and group of swap file by "veswap.options". Please see VEOS document "VEOS Partial Process Swapping Reference" for more information on "veswap.options". Summary : Balancing the load of interrupts from VEs Type : enhancement Detail : "vesysinit" package installs the scripts which balances the load of interrupts from VEs for stable execution of software on VH. It also installs the irqbalance policy script to avoid irqbalance from changing the destination of interrupts from VEs. Summary : Updated the VE driver to improve interrupt performance. Type : enhancement Detail : VE driver is updated to improve interrupt performance. Summary : Updated the behaviour to disable the core dump of VH when stack limit is exhausted. Type : enhancement Detail : VEOS is updated to disable the core dump of VH when VE process is aborted on exhausting its stack limit. Summary : Fix of issue where the VEOS may terminate abnormally when MPI process exits. Type : bugfix Detail : If MPI process transfers data on System V shared memory and exits without detaching the shared memory, VEOS may terminate abnormally. Summary : Fix of issue where the status of the VEs may not become ONLINE after installation of "ve-devel" package group. Type : bugfix Detail : The status of the VEs may not become ONLINE after installation of "ve-devel" package group in the RHEL8.1 or CentOS8.1 environment. Summary : Fixed an issue where VH process aborts when it calls veo_get_context() with an invalid index. Type : bugfix Detail : If the user passes an invalid argument to the second argument of veo_get_context(), the non-existent element of the vector type variable is accessed. As a result, an out_of_range exception is thrown, then the VH process aborts because the AVEO library does not catch the exception. Summary : Fix of issue where VE process state is temporarily updated to WAIT state after STOP state. Type : bugfix Detail : If a VE binary executed with execve() system call or using VE commands (like time, taskset, prlimit and strace) is stopped by SIGSTOP signal immediately after starting its execution, sometimes the process's state is temporarily changed to WAIT state after STOP state. The state will be STOP state again after the temporary WAIT state. Summary : Fix of issue where pthread_mutex_init() does not fail with ENOTSUP if mutex attributes is 'PTHREAD_PRIO_INHERIT'. Type : bugfix Detail : When 'PTHREAD_PRIO_INHERIT' attribute is set using pthread_mutexattr_setprotocol() then at the time of initialization using pthread_mutex_init(), glibc does not check if 'PTHREAD_PRIO_INHERIT' is supported in VEOS or not and returns success instead of ENOTSUP. Summary : Fix the issue where clock_gettime() returns incorrect value for clock id CLOCK_THREAD_CPUTIME_ID. Type : bugfix Detail : clock_gettime() returns incorrect value for thread-specific CPU-time clock. 2.7.2 This section describes changes between the 2.7 and 2.7.2 versions of VEOS. Summary : Supported features to get the size of memory which cannot be swapped out by "Partial Process Swapping" Type : enhancement Detail : 'veswap -n' shows the size of non-swappable VE memory which a VE process has used. And 've_get_nonswappable' of libsysve gets the maximum size of non-swappable VE memory which caller VE process has used. Please see VEOS document "VEOS Partial Process Swapping Reference" or "The Tutorial and API Reference of libsysve" for detail. Summary : Improvement in "load average" implementation. Type : enhancement Detail : Load average implementation was updated to display load average as sum of active tasks on all the cores in a VE node Summary : Disabled generation of core files for VH binary 've_exec' if VE program terminated with hardware exception. Type : enhancement Detail : Two core files are generated such as 'core..ve' (for VE binary './ve_a.out') and 'core.' (for VH program 've_exec') when a VE program causes any hardware exception. The VH side core file in this case is unnecessary and hence are disable now. Summary : Fixed the issue where VEOS may hang or terminate abnormally after fork()/clone() failed due to VE process's PID duplication in VEOS Type : bugfix Detail : VEOS may stop responding or terminate abnormally after it rejects creation of new task because of PID duplication due to improper cleanup of task in VEOS. Summary : Fix the issue where VE process is getting hanged on polling for all the file descriptors of a set. Type : bugfix Detail : VE process gets hanged in poll(2), when it is polling for all the file descriptors of a set of to become ready and set contains multiple file descriptors. Summary : Fix of issue where program fails to store the signal mask of a context after swapcontext() is called. Type : bugfix Detail : If a signal is masked before storing the context using getcontext() API then after the call of swapcontext() , the program loses the saved signal mask context. Summary : Fix of issue where program gives 'segmentation fault' when setcontext() or swapcontext() is called. Type : bugfix Detail : If a signal is pending in the sigpending list and the corresponding signal handler is invoked at the time of restoring/switching of context with setcontext()/swapcontext() APIs , then the program crashes with 'segmentation fault'. Summary : Fix the issue where VE process is getting blocked on fcntl(2). Type : bugfix Detail : VE process gets blocked in fcntl(2) with F_SETLKW command, when conflicting lock is held on the file and the call is interrupted by signal. Summary : Fix the issue where VE 'sa1' command creates dummy file of size 0 and VE 'sa2' command failed to process that dummy file. Type : bugfix Detail : On RHEL7 environment, VE 'sa1' command creates dummy/erroneous "saDD_*" file of size 0 when value of macro 'HISTORY' is configured with value greater than 28 in '/etc/sysconfig/ve_sysstat' file and current date file does not exist in '/var/opt/nec/ve/log/sa/' directory. In such case, VE 'sa2' command failed while processing this dummy "saDD_*" file to create "sarDD_*" file. Summary : Fix the issue where VE_NODE_NUMBER environment variable does not come to effect correctly for VE 'sa1' and VE 'sa2' commands. Type : bugfix Detail : On RHEL7 environment, if VE_NODE_NUMBER environment variable is set and value of macro 'HISTORY' is configured with value greater than 28 in '/etc/sysconfig/ve_sysstat' file. In this case, - VE 'sa2' command creates "sarDD_node" files for all node's current date "saDD_Node" files exist at path "/var/opt/nec/ve/log/sa/" instead of "sarDD_node" file for only specified VE node. - VE 'sa1' command creates symbolic links for all node's current date "saDD_Node" files exist at path "/var/opt/nec/ve/log/sa/" instead of "saDD_Node" file's link for only specified VE node. 2.7 This section describes changes between the 2.6.2 and 2.7 versions of VEOS. Summary : Supported Alternative VE Offload Type : enhancement Detail : The VEO implementation has changed to Alternative VE Offload(AVEO). It is a faster and much lower latency replacement to the previous VEO implementation which brings multi-VE support, simultaneous debugging of VE and VH side, API extensions. NEC supports the previous VEO implementation until the end of Mar. 2021. Please see VEOS document "The Tutorial and API Reference of Alternative VE Offloading" for VEO migration and AVEO installation. Summary : Supported new VH Call for Fortran APIs to pass an INTEGER array Type : enhancement Detail : User can pass an INTEGER array to function or subroutine invoked on VH side by using libvhcall-fortran APIs. See VEOS document "The Tutorial and API Reference of VH Call Fortran API". Summary : Fix the issue where PPID of VEO process is wrong value in process accounting Type : bugfix Detail : When a user creates a VE process using VEO API, PPID recorded in process account is -1. 2.6.2 This section describes changes between the 2.5 and 2.6.2 versions of VEOS. Summary : Enhancements of Partial Process Swapping Type : enhancement Detail : The following functions are supported. - Swap out VE memory to a file. - Swap out when swap in does not proceed due to no enough memory. - Swap out zombie processes. Summary : Enhancements of environment variable VE_ACC_IO Type : enhancement Detail : Users can change the value of VE_ACC_IO to change huge pages usage of accelerated IO. When this environment variable is set to 1, accelerated I/O is enabled, with huge pages whose size is default value. Default value is 64MB. When this environment variable is set to 2, accelerated I/O is enabled, with huge pages whose size is 8MB * the number of cores of VE. Summary : Improved load rebalancing feature Type : enhancement Detail : Load rebalancing feature is improved to take into account the status of VE tasks. VEOS will relocate a VE task from a VE core which has more than one VE tasks, which are running or sleeping, to the VE core which meets following conditions. - There is no VE task which is running. - There is no VE task which is sleeping. Summary : Improved VE task creation algorithm Type : enhancement Detail : VE task creation algorithm is improved to take into account the status of existing VE tasks. VEOS creates a new VE task for NUMA node which has the fewest VE tasks running or sleeping. It also assigns its VE task to the VE core which has the fewest VE tasks which are running or sleeping. Summary : Supporting times() function/system call Type : enhancement Detail : times(2) system call is newly supported. Summary : Improved VE process system call latency. Type : enhancement Detail : The algorithm for searching a VE task in VEOS is improved which in turn improves the overall latency of system calls. Summary : Added information to VE process accounting information Type : enhancement Detail : VE Process accounting information is updated. The additional information is showed by --ve-info option of dump-acct command. lastcomm command is supported. The additional information is also showed by --ve-info option of lastcomm command. Summary : For converting VE process accounting file, new command 'convert-acct' is supported. Type : enhancement Detail : The new tool convert-acct is supported. It is the tool needed by the process accounting improvement. When reading the VE process accounting file which has multi version VE process accounting data with dump-acct or lastcomm command, this tool converts it to a readable form. Summary : Added suspend/resume functionality to VE process accounting based on available space in root filesystem Type : enhancement Detail : When VE process accounting is enabled, it will be suspended if available space in root file system is less than or equal to 2% and again it will be resumed automatically if the available space grows to 4% or more. Summary : Fix of issue where data updated may be invisible, when VE processes are resumed after Swap-in. Type : bugfix Detail : Data updated by a VE thread is invisible from other VE threads or VE processes due to cache coherency problem between NUMA nodes, when VE processes are resumed after Swap-in. This leads to a hung of MPI program or data corruption. This issue is reproduced only in the following cases. - When VE processes, which uses VE memory on other NUMA node that are different from the one running on, are resumed after swap-in. - When VE proccesses, which execute MPI program on A100-1 series, are resumed after swap-in. Summary : Fix the issue where RLIMIT_CPU resource limit doesn't come to effect correctly in a multi-threaded program Type : bugfix Detail : When a multi-threaded program is executed with RLIMIT_CPU resource limit (inherited from ulimit or set using setrlimit(2) or prlimit command or VE_LIMIT_OPT), it doesn’t get SIGXCPU signal after the CPU execution time for the thread group reaches soft limit. Summary : Fix the issue where VE process accounting data of few processes may not be captured if accounting is enabled twice in succession Type : bugfix Detail : When VE process accounting is already enabled and user again enables it, accounting data of one or more processes terminating at the same time may not be captured. This can occur during log rotation also. Summary : Fix the issue where VE 'sar -w' command adds an extra newline. Type : bugfix Detail : On RHEL8 environment, VE 'sar -w' command adds an extra newline after each entry in output statistics. Summary : Fix the issue where 'restart_ve_psacct' doesn't start 'psacct-ve' service at veos restart. Type : bugfix Detail : On RHEL8 environment, 'restart_ve_psacct' script is not able to start 'psacct-ve' service at veos restart, if the service was enabled at veos restart. Summary : Fix the issue where process accounting continue to be written in renamed old file after log rotation. Type : bugfix Detail : When log rotation for VE psacct was executing by cron job and 'psacct-ve' service was active then VEOS was writing the process accounting information to the renamed old file after log rotation instead of writing in the newly created file. Summary : Fix the issue where 'vmstat' and 'mpstat' commands get abort when VE node 0 is offline. Type : bugfix Detail : On RHEL8 environment, both 'vmstat' and 'mpstat' command is aborted when VE node 0 is either offline or does not exist. And 'vmstat' command is aborted on RHEL7 environment also when VE node 0 is offline or doesn't exist. Summary : Improve log messages of VEOS about VE memory mapping. Type : enhancement Detail : Change log level of VEOS messages about VE memory mapping to applicable level Summary : Improved performance of VE memory mapping daemon Type : enhancement Detail : Improve a response of VE memory mapping daemon(vemmd) for a request from IB HCA driver. That leads to improve MPI performance. 2.5 This section describes changes between the 2.4.3 and 2.5 versions of VEOS. Summary : RHEL8.1/x86_64 support Type : enhancement Detail : RHEL8.1/x86_64 is newly supported. Please see README for Supported Platforms and Operating Systems. Summary : Unnecessary archive files and libtool archives for VH programs are removed Type : enhancement Detail : Archive files (.a) and libtool archives (.la) for VH programs are removed from development packages. These files are unnecessary because VH programs are dynamically linked with libraries. Archive files and libtool archives for VE programs are provided as before. Summary : Changed the limit of VEOS core file size to 1GB. A Type : enhancement Detail : Changed the limit of VEOS core file size from no-limit to 1GB. Summary : Improved log messages of VEO. Type : enhancement Detail : VEO records the failure cause of veo_load_library() in a log file of VEO. Summary : Fix the issue where VE thread is getting abort after activation of the new context. Type : bugfix Detail : VE thread is getting abort, after activation of the new context by using swapcontext() if the new context is earlier modified with API makecontext(). Summary : Fix of issue where VE program performance may degrade if processes which are swapped out exist. Type : bugfix Detail : When processes which are swapped out by "Partial Process Swapping" exist at the time of creation of a process or a thread, two running threads may be assigned the same core due to inadequate process creation algorithm. In this case, VE program performance degrades. User can check which core the thread is executed on by pidstat command. Summary : Fix of issue where load rebalancing feature does not work when some processes are swapped out. Type : bugfix Detail : When threads more than the number of cores are running on VE, the performance of VE program is decreased remarkably. At this time, even if the number of running threads is decreased less than the number of cores by swap-out of "Partial Process Swapping", load rebalancing feature does not work and performance of VE program is not improved. Summary : Fix of issue where clock_gettime() fails if errno is already set Type : bugfix Detail : When a program which is compiled with '-no-proginf' option invokes the first clock_gettime() with non-zero errno, the clock_gettime() will fail. Summary : Fix the issue where VEO may fail to load a shared library. Type : bugfix Detail : veo_load_library() fails to load a shared library if never referenced symbol cannot be resolved. Summary : Fix of issue where memory is leaked at VH side Type : bugfix Details : Small memory may be leaked at VH side when a VE MPI process transfer data between processes on other VE nodes. 2.4.3 This section describes changes between the 2.4.1 and 2.4.3 versions of VEOS. Summary : Fixed an abort issue of VEOS during system call handling Type : bugfix Detail : VEOS may detects unintended timeout and aborts, when it tries to start a VE core after it handles a system call request from VE program. When VEOS aborts, running VE processes also terminate abnormally. 2.4.1 This section describes changes between the 2.4 and 2.4.1 versions of VEOS. Summary : Supported ftrace feature through VEO. Type : enhancement Detail : When following conditions are established, ftrace.out file is generated by ftrace runtime libraries. - A user link and compile a VE program with "-ftrace" option. - VEO program calls veo_proc_destroy(). Please see VEOS document "The tutorial and API reference of veoffload" for detail. Summary : Supported new VEO APIs which can specify stack size of VEO context. Type : enhancement Detail : VEO supports new APIs which can specify stack size of VEO context as attribute. User can create a VEO context which has a specified sized stack by using the APIs. Please see VEOS document "The tutorial and API reference of veoffload" for detail. Summary : Changed runtime library search path of veorun. Type : enhancement Detail : veorun is an executable for VE, and VEO uses it to execute a VE function. The runtime library search path of veorun is changed to /opt/nec/ve/lib so that VE shared library which user loads using veo_load_library() does not depend on runtime libraries of specific version of 'ncc' and 'nfort' compiler. Summary : Improved performance of clock_gettime() system call. Type : enhancement Detail : Improved performance of clock_gettime() with below clock id system call. When a process or thread makes the request for system time using clock_gettime() with below clock id, then first request will be served using VH syscall. On subsequent requests, time will be calculated at VE end instead of invoking VH syscall for same thread-group. CLOCK_REALTIME CLOCK_REALTIME_COARSE CLOCK_MONOTONIC_COARSE CLOCK_MONOTONIC_RAW CLOCK_BOOTTIME CLOCK_THREAD_CPUTIME_ID Summary : Updated VEOS scheduler to instantly schedule the runnable VE task on VE core Type : enhancement Detail : Runnable VE task was not scheduled on the VE core until next scheduler timer interval of VEOS scheduler. With this enhancement VEOS scheduler will schedule the runnable VE task on VE core when no other task is executing on VE core. Summary : Updated the behaviour to abort the VE task when stack limit is exhausted. Type : enhancement VEOS is updated to abort the VE task with the message "Unable to grow stack" when stack limit of the VE task is exhausted. Summary : Fix the issue where VE task stack can grow beyond the stack limit. Type : bugfix Detail : VEOS allocates the stack area beyond the stack limit of VE task if it requires the stack area which cannot be accommodated within the stack limit. Summary : Fix the issue of failure of VEO process creation when a user executes a VEO program through the job scheduler. Type : bugfix Detail : When following conditions are established, VEO process creation may fail. - A user executes a VEO program through the job scheduler. - Above VEO program calls veo_proc_create_static(). Summary : Fixed the abort issue during VEOS shutdown. Type : bugfix Detail : VEOS aborts when VEOS tries to access data of the invalid/deleted task during shutdown. 2.4 This section describes changes between the 2.3 and 2.4 versions of VEOS. Summary : Supported VH Call API for VE Fortran program Type : enhancement VH Call is a functionality which invokes a shared library function compiled for VH from VE process. User can invoke a function or subroutine in VH Fortran library or module by using the VH Call API. Please see VEOS document "The tutorial and API reference of libvhcall-fortran". 2.3 This section describes changes between the 2.2.2 and 2.3 versions of VEOS. Summary : Supported Docker for VE application execution Type : enhancement Detail : Users can execute VE applications in containers using Docker. A Dockerfile to build a image is available on github. https://github.com/veos-sxarr-NEC/docker_container Summary : Changed RSS calculation algorithm of shared memory Type : enhancement When VE memory is shared between VE processes, VEOS divides the size of the shared memory between sharing processes. The divided values is counted to RSS(resident set size) of each process. Summary : Added new environment variable VE_LIMIT_OPT to set resource limit and support RSS limit Type : enhancement The environment variable VE_LIMIT_OPT can be used to set resource limits of VE processes. Please see VEOS document "How to execute VE program" for detail. Memory allocation fails if a process exceeds RSS limit. Summary : Support of Partial Process Swapping Type : enhancement Detail : It provides a function that swapped out a part of the memory of a stopped VE process into HugePages reserved on the VH side and frees the VE memory. The freed VE memory is able to use from another program. It also provides a function that restores the memory swapped out to the VH side to the VE memory and returns the VE process to the state before the memory was swapped out. Please see VEOS document "VEOS Partial Process Swapping Reference" for usage. Summary : Functional enhancements of GDB Type : enhancement Detail : There are following functional enhancements. - The version of GDB is updated 7.12.1 from 7.11.1. - Some commands are added. Please see VEOS document "Difference Points for GDB". Summary : Supported an environment variable, VE_ACC_IO Type : enhancement Detail : Users can change the IO-offloading-method of a process to Accelerated I/O by setting VE_ACC_IO=1. Users can also disable Accelerated I/O when VE_ACC_IO is undefined or set to zero. Summary : Added support to show VE process and its statistics created by VEO API using commands. Type : enhancement Detail : The command 'ps', 'top', 'pidstat', 'prtstat' and 'pmap' are modified to display VE processes which are executed using VEO API called by a child thread. Summary : Supported new data transfer APIs for VH Call Type : enhancement Detail : A VH thread spawned by a function which is executed by VH call can send/receive data to/from a memory of a VE thread which has invoked the function on VH side. Summary : Fix the issue where VE GDB shows the main thread in execution instead of child thread during core file debugging. Type : bugfix Detail : During core file debugging in the VE GDB, the main thread information is displayed even when the child thread has caused process termination. It is because the VEOS always marks the parent thread information as the current thread in execution. Summary : Fix the issue where child threads state is not changing to STOP('T') when main thread is in Zombie('Z') state and SIGSTOP signal is delivered to main thread. Type : bugfix Detail : Child threads state in a thread group are not updated to STOP('T') state when main thread is in Zombie('Z') state and SIGSTOP signal is delivered to the main thread. Summary : Fix the issue that all signals were not delivered to a program using VEO Type : bugfix Detail : No signals were delivered to a program using VEO because all signals were masked after veo_proc_create(). Summary : Fix the issue that SIGCONT did not resume execution of a program executed by VEO. Type : bugfix Detail : VEOS is required to be notified of SIGCONT to resume execution from SIGSTOP. However, SIGCONT did not resume VE part in a program using VEO since libveo did not notify VEOS of SIGCONT. 2.2.2 This section describes changes between the 2.2 and 2.2.2 versions of VEOS. Summary : Load Balancing of a VEO OpenMP program Type : enhancement Detail : VEO sets a thread affinity of a context thread automatically to avoid conflicts with an OpenMP thread when threads more than the number of VE cores are created. If you use a statically-linked veorun binary, please re-create the veorun by mk_veorun_static. Summary : Supported VH function call on a VEO context thread Type : enhancement Detail : VEO now supports a new API function veo_call_async_vh() to execute a VH function on a VEO context thread. Summary : Fix the issue where fork() system call in a multi-threaded VE process hangs. Type : bugfix Detail : When a multi-threaded VE process invokes fork() system call and simultaneously other threads are performing I/O using read() system call where starting address or size of the VE buffer is not 8 byte aligned, then fork() system call hangs. Summary : Fix the issue that a VE function call with arguments narrower than 64 bit by VEO leads to undefined behavior. Type : bugfix Detail : The upper bits of integer arguments narrower than 64 bit is undefined because VEO passes integer arguments without signed or zero extension. A VE function call with such integer arguments leads to undefined behavior due to the undefined upper bits. 2.2 This section describes changes between the 2.1.3 and 2.2 versions of VEOS. Summary : Improved instruction cache performance Type : enhancement Detail : Memory layout of VE processes is modified in order to improve instruction cache performance of VE core. It leads to better performance of VE program. New memory layout needs more than or equal to 1024 bytes gap between text section and data section which are defined in an executable or a shared library. If the size of the gap is less than 1024 bytes, loading fails with an error message, "Failed to validate the gap between text/data". In this case, please re-link the executable or the shared library. 've_validate_binary' command can check the gap size. Please see the next topic for more information. Summary : New command 've_validate_binary' Type : enhancement Detail : New command /opt/nec/ve/bin/ve_validate_binary is added. This command checks the size of the gap between text section and data section of an executable or a shared library. If the gap is less than 1024 byte, it shows a message, "DATA/TEXT gap is less: ". Summary : Updated the value of si_code set during the invocation of signal handler routine corresponding to some of the HW exceptions. Type : enhancement Detail : si_code present in siginfo_t structure passed as an argument during the invocation of signal handler routine has been updated. Below is the updated list of si_code set corresponding to HW exceptions : HW exceptions si_code Missing page exception 'SEGV_MAPERR' Memory access exception 'BUS_ADRALN' Host missing page exception 'SEGV_MAPERR' Host missing space exception 'SEGV_MAPERR' I/O access exception 'BUS_ADRERR' Illegal instruction format exception 'ILL_ILLOPC' Summary : The enhancement of VEO APIs not to specify VE node number explicitly Type : enhancement Users can specify -1 as the VE node number to veo_proc_create() or veo_proc_create_static(). If -1 is specified, a VE process is created on the default VE node (VE node '0'). If -1 is specified and environment variable VE_NODE_NUMBER is set, a VE process is created on the VE node specified by VE_NODE_NUMBER. Summary : The enhancement of VEO APIs for the job scheduler Type : enhancement Detail : If a user executes a program which uses VEO APIs through the job scheduler, the VE node number specified by the program is treated as a logical number. It is translated into physical VE node number assigned by the job scheduler. For example, if the VE node number specified by the program is 0, and the job scheduler assigns VE node #4, #5 and #6, the VE process will be created on VE node #4. Summary : Functional enhancements of GDB Type : enhancement Detail : There are following functional enhancements. - GDB can attach a VE process without the path of an executable binary. - GDB supports "target" and "target core" command. - GDB can use input/output redirection. Summary : Checking the assignment of VE nodes by the job scheduler Type : enhancement Detail : VEOS checks the assignment of VE nodes by the job scheduler. If the job scheduler does not assign any VE node, execution of the VE program fails. If a user directly executes VE program without the job scheduler, it will be executed without checking the assignment. Summary : Fix the issue where a new request is put in request queue after exception occurs. Type : bugfix Detail : When the function called using VEO API made an exception occur, the context can't continue to execute functions any more. After exception occurs, a new request issued is put in request queue. If user wait the result of the request, VEO API seems to be hanged because the new request which is accepted by the context isn't executed. Summary : Fix the issue where VE memory for exited orphan process is not cleaned up if the parent VE process was executed by VE 'time' command. Type : bugfix Detail : VE memory for exited child process is not cleaned up if the parent VE process, executed by VE 'time' command, exits before the child process. Summary : Fix the issue where API related to locale returns error when referring to the native environment. Type : bugfix Detail : The VE process using API related to locale doesn't recognize locale correctly and returns error ENOENT when referring to the native environment which is specified through LANG environment variable. (eg. LANG=en_GB.UTF-8, LANG=en_US.UTF-8 Summary : Fix the issue that GDB can not show functions of glibc by "backtrace" command Type : bugfix Detail : GDB can not show functions of glibc library correctly by "backtrace" command. Summary : Fix the issue that GDB gets an incorrect vector register. Type : bugfix Detail : GDB shows an invalid value when GDB prints vector register, It is because GDB does not deal with the value of the vector register correctly. Summary : Fix the issue where usage of macro __FE_DENORM leads to unexpected result. Type : bugfix Detail : Use of macro __FE_DENORM is causing unexpected result because VE does not support denormal number. So, removing the support of __FE_DENORM macro. 2.1.3 This section describes changes between the 2.1 and 2.1.3 versions of VEOS. Summary : Supported performance improvement of gettimeofday() when 'timezone' argument is specified Type : enhancement Detail : The API gettimeofday() invokes the system call on very first call and invokes the function call which returns timezone in VE side, on subsequent callings when 'timezone' argument is specified as an argument. That improves performance of gettimeofday(). Summary : Supported to call VEO API from a child thread on VH side. Type : enhancement Detail : VEO supports to call VEO API from a child thread on VH side and create a VE process. When a VE process is created from a child thread on VH, VE DMA or VH-VE SHM is not supported. Summary : Supported new VEO APIs which can set 16 bit and 8 bit signed and unsigned integer arguments. Type : enhancement Detail : VEO supports new VEO APIs which can set 16 bit integer/unsigned integer, 8 bit integer/unsigned integer. Because user can set veo_args to the integer types by using the VEO APIs, user can pass the four integer types to VE process. Summary : Supported to use VE program written in Fortran(VEO). Type : enhancement Detail : VEO supports a library implemented in Fortran except a quadruple precision real number or a variable length character string as an argument or a return value. Summary : Supported linking static libraries without exporting symbols (VEO) Type : enhancement Detail : Users can link static libraries without exporting symbols on creating a static veorun by the new '--link-nosym' option of 'mk_veorun_static'. This enhancement reduces the memory consumption, and speeds up the starting and the symbol search of a program dependent on large libraries such as BLAS. Summary : Improved VE process loading performance. Type : enhancement Detail : VE process loading functionality is optimized for faster loading. Also updated VE process scheduling functionality to execute process immediately if the assigned core is idle. Summary : Updated PPID value in process accounting file to display PPID of a new VE process as the 'PID of process' which created it. Type : enhancement Detail : The dump-acct command displays PPID as 'PID of process' which created it instead of '1' for any new VE process which is executed with 've_exec' or with any commands for VE (like taskset, prlimit, time and strace for VE). Summary : Fix the issue in system() API where the memory of the VE process is not freed until the VH process created by system() is exited. Type : bugfix Detail : When VE process invokes system() API to execute a VH process in background e.g. 'system("/bin/sleep 180 &")', the memory resources of the VE process is not freed even after its exit, untill the VH process created with system() exits. Summary : Fix the issue in system() API when an input string represented by 'command' argument may get corrupted. Type : bugfix Detail : When VE program invokes 'system(command)' then extra characters may get appended in the string literal represented by 'command' which leads to unexpected behaviour of given command. Summary : Fix the issue in gettimeofday() when this API returns incorrect 'timeval' structure values if it is invoked after 1 hour of it's previous call. Type : bugfix Detail : When VE program invokes gettimeofday() and subsequently invokes gettimeofday() again after the time frame of 1 hour is elapsed, the results in the 'timeval' structure returns garbage. Summary : Fix the issue in delivering SIGALARM when timer expire happens during disabling the timer through setitimer() system call. Type : bugfix Detail : When VE process invokes setitimer() system call to disable the registered timer and at the same time timer also expires. In this case SIGALARM signal delivers to the VE process after returning from setitimer() system call. This changes SIGALARM delivering so that it is delivered during setitimer() system call. Summary : Fix the issue of changing data which is read by VE AIO Type : bugfix Detail : When multiple ve_aio_read() is called at a time under one of following condition, some read data may be changed unexpectedly. - Starting address of buffer on VE is not multiple of 8 - Size of buffer on VE is not multiple of 8 Summary : Fix of the issue where VEO programs with libveosinfo may terminate abnormally Type : bugfix Detail : VEO programs linked with 'libveosinfo', a library to get VE and VEOS information, may terminate abnormally due to a symbol conflict between 'libveosinfo' and 'libvepseudo', a library composing a pseudo process, used by VEO. To avoid such a conflict, symbols of internal functions and variables of 'libvepseudo' are now hidden. Summary : Fix of issue where VEO user can not run veorun statically linked when user directories mounted with NOSUID. Type : bugfix Detail : VEO user can not execute a program with statically linked veorun if the underlying filesystem of the executable is mounted with "nosuid" mount option. Summary : Fix the issue of stalling VEO request. Type : bugfix Detail : When the function called using VEO API made an exception occur, the context can't continue to execute functions any more. But, when race condition occurred in VEO library, calling the function using asynchronous VEO API, the context which made an exception occur may accept a new request. In that case, when you wait for a result using the request ID returned by the request, VEO API seems to be hanged because the new request which is accepted by the context isn't executed. Summary : Fix the issue where permission of VE process stack mapping is set to 'PROT_WRITE'. Type : bugfix Detail : VE 'pmap' command shows the stack mapping permission of VE process as 'PROT_WRITE' instead of 'PROT_WRITE|PROT_READ'. Summary : Fix the issue where process tracing using VE 'strace' command failed if execve() system call is called from traced process. Type : bugfix Detail : If execve() system call or any exec() family of functions are called from traced VE process then 'strace' command failed with error as "Bad address". Summary : Fix the issue of compilation error while including header file 'strings.h' in the user program when compiling through 'nc++' compiler with compilation option '-U__GNUC_MINOR__'. Type : bugfix Detail : When the header file strings.h is included in the user program and the program is compiled through nc++ compiler with compilation option -U__GNUC_MINOR__ ,the compiler prompts error. Summary : Fix the issue of warning or error occurring when program using VH Call is compiled. Type : bugfix When program using vh_call_args_alloc() is compiled as C program, warning is occurred. In case of specifying -std=c11 option or compiling as C++ program, error is occurred. 2.1 This section describes changes between the 2.0.3 and 2.1 versions of VEOS. Summary : Supported NUMA mode Type : enhancement Detail : The partitioning mode of VE card (HW) is available, so VEOS supports VEOS NUMA mode to control the partitioning mode. In the partitioning mode, user can use VE card like NUMA. Please see VEOS document "How to execute VE program" for usage. Summary : Supported VE Offloading framework (VEO). Type : enhancement Detail : VEO provides accelerator-style programming on VE. VEO controls the execution of a VE process from VH main program. VEO enables to develop a program using both VEs and VH computing resources. Please see VEOS document "The tutorial and API reference of veoffload". Summary : Improvement VH Call API for passing arguments Type : enhancement Detail : VH Call is a functionality which invokes a shared library function compiled for VH from VE process. This improvement allows user to invoke a function passing any types and any number of arguments. Using the enhanced API, invoking a Fortran library function is available. Please see VEOS document "The tutorial and API reference of libsysve". Summary : Supported "ldd" command of glibc for VE Type : enhancement Detail : ldd prints the shared libraries required by each dynamically linked binary for VE architecture. Summary : Supported Accelerated I/O with Infiniband environment Type : enhancement Detail : Accelerated I/O with Infiniband environment is supported. Please see VEOS document "How to execute VE program" for usage. Summary : Modified the functionality of multiple commands to use VE node '0' when VE_NODE_NUMBER environment variable is not set. Type : enhancement Detail : VE node '0' is used by default when below listed commands are executed without specifying any node using VE_NODE_NUMBER environment variable. - tload, top (irrespective of options) - iostat, mpstat, pidstat, sadc, sadf, sar, vmstat, free (when executed with 'delay/interval/count’ options) - accton (when executed with 'filename') - sa (when executed with '--other-usracct-file/--other-savacct-file/ --other-acct-file' options) - taskset, prlimit, time, strace (to launch a new process) Summary : Improved VE and VH data transfer performance. Type : enhancement Detail : VE driver function which is a part of the data transfer between VE and VH is improved. By this enhancement, the performance of data transfer, like read()/write() system call, is improved. Summary : Improvement of the log file name of VEOS to include VE node number Type : enhancement Detail : The log file name of VEOS is changed from veos.log.X to veosN.log.X (where N is VE node number and X is sequential number). The directory where log files are stored is not changed from /var/opt/nec/ve/veos. Summary : Removed the packages relating to musl-libc Type : enhancement Detail : The packages relating to musl-libc are removed because NEC ended the support of musl-libc. The removed packages are below: musl-libc-ve musl-libc-ve-devel libsysve-musl libsysve-musl-devel veos-musl-header If they are installed on your system, please uninstall them. Summary : Improved execve() system call to always set VE_NODE_NUMBER environment variable for VE processes. Type : enhancement Detail : Updated execve() system call in order to export VE_NODE_NUMBER environment variable to the new process even if envp, the third argument, is passed as NULL in execve() syscall. The VE node number in which the process is executing can be determined using VE_NODE_NUMBER environment variable. Summary : Updated libsysve header files for C++ programs Type : enhancement Detail : The header files, libvhcall.h, veaio.h, vedma.h and vhshm.h are updated so that C++ user programs includes them. Summary : Changed I/O buffer allocation in glibc Type : enhancement Detail : File I/O API (fread()/fwrite()/etc.) get I/O buffer from heap instead of anonymous page. Summary : Updated the algorithm to calculate start time, elapse time and user time for VE process. Type : enhancement Detail : Improved the algorithm to calculate elapse time and user time in units same as the VH. Also, start time of VE process is now preserved across execve(). Summary : Updated the functionality of commands for VE to remove difference of PPID value between the commmands output and getppid() system call. Type : enhancement Detail : Commands for VE were showing PPID as 1 for any new VE process executed with 've_exec'. The commands 'top', 'ps' and 'prtstat' are updated to display PPID of VE process as the 'PID of shell' which created it. And the behavior is same for getppid() system call. Summary : Fix the issue where data in file backed shared memory is not written back to the file. Type : bugfix Detail : Data of file backed memory which is mapped with 'MAP_SHARED' flag and 'PROT_WRITE' permission is not carried through to the underlying file on munmap() if the same area is mapped again with 'MAP_PRIVATE' flag and 'PROT_READ' prior to the munmap() syscall or prior to process exit. Summary : Fix the issue where msync() syscall may fail for file backed memory Type : bugfix Detail : msync() syscall fails on file backed memory, if the file is mapped by mmap() with the first argument set to the valid address and one of below conditions is true. - The start address of the file backed memory is not 512MB aligned, the file backed memory is beyond 512MB boundary and the page size is 2MB. - The start address of the file backed memory is not 16GB aligned, the file backed memory is beyond 16GB boundary and the page size is 64MB. If the file is mapped by mmap() with the first argument set to NULL, the issue does not occur. Summary : Fix the issue where mmap() request without MAP_FIXED flag and at a user specified address may fail. Type : bugfix Detail : Memory mapping using mmap() syscall without MAP_FIXED flag fails if the user specified address is not 512MB aligned for 2MB page size, or not 16GB aligned for 64MB page size. Summary : Fix the issue where VE core dump file is created with wrong file name when privileged user tries to create the core file at path "/". Type : bugfix Detail : When a privileged user tries to generate VE core dump file at path "/" to capture core dump of VE process, VEOS creates the VE core file with wrong name. Summary : Fix of issue where program gives segmentation fault when setjmp() is invoked in shared library. Type : bugfix Detail : If setjmp() is called from shared library, then one of the VE register got updated unexpectedly due to which program gives segmentation fault in further processing. Summary : Fix the issue where "/opt/nec/ve/bin/pidstat -l" command gives segmentation fault. Type : bugfix Detail : If VE "/opt/nec/ve/bin/pidstat -l" command made an attempt to display process statistics for a zombie process then the command terminates with segmentation fault. Summary : Fix the issue where VE 'time' command fails if the executed VE program invoked VH binary with execve() system call. Type : bugfix Detail : If a VE process created with VE 'time' command executes a VH binary using execve() system call, 'time' command fails with "No such process" error. Summary : Fix the issue where accounting file shows the presence of a 'Dummy_task' which is not created by user. Type : bugfix Detail : If VE process accounting is enabled and VE 'time' command failed to execute a new process, a 'Dummy_task' entry is captured in accounting file although a task in that name was never created. Summary : Fix the issue where accounting file shows "Dummy_task" entry when VE 'time' command executes VH binary. Type : bugfix Detail : If VE 'time' command tries to executes a VH binary then it executes successfully but VE accounting file captures a "Dummy_task" entry. Hence the similar commands 'taskset', 'prlimit' and 'time' has been restricted to execute only VE binaries and VH binaries will not be executed directly using these commands. 2.0.3 This section describes changes between the 2.0.1 and 2.0.3 versions of VEOS. Summary : Improvement of memory consumption in glibc for APIs which use internal buffer. Type : enhancement Detail : APIs which use internal buffer, like printf(), dprintf() etc, consume only 2MB page irrespective of VE binary's default pagesize. In comparison with previous version, the memory consumption for these APIs is reduced by 62MB in case the default pagesize is 64MB. Summary : Improvement of memcpy() performance Type : enhancement Detail : To improve the performance areas where vector memcpy() is used internally for copying/transfering of smaller size data. Example of some of the APIs where performance will be enhanced are fread() , fwrite() , strcpy() , stpcpy() , strncat() , strcat() etc. Summary : Improvement of memory consumed by system() Type : enhancement Detail : To overcome the limitation of consuming VE memory during execution of VH program through system(), request will be served in VH. User can now execute a VH program through system() at VE even with limited memory. Summary : Support of openat() version limitation in glibc Type : enhancement Detail : Updated the handling in library to set the permissions to a hard link file created using 'linkat()' when its target(oldpath) is a symbolic link file descriptor returned while opening a file using 'openat()' with flags combination 'O_TMPFILE | O_RDWR' and the required mode. Summary : Fix of the issue where VEOS performance degrades when an attempt is made to read VE process registers. Type : bugfix Detail : When an attempt is made to read VE process registers using ve_get_regvals() which is included in libveosinfo then VEOS proformance degrades significantly. Summary : Fix the issue where sched_setaffinity() system call or taskset may causes segmentation fault in multithreaded VE environment. Type : bugfix Detail : In multithreaded VE environment when VE task is migrated from one VE core to another VE core either by using sched_setaffinity() system call or by taskset command, memory/address translation related context for VE task are not getting restored properly to the newly migrated VE core, this causes segmentation fault(SIGSEGV) for VE task. 2.0.1 Summary : Support of glibc for VE Type : enhancement Detail : glibc for VE is available. Please see VEOS document "How to execute VE program" and "Difference Point for Glibc-libc" for additional information. Summary : Support of Accelerated I/O Type : enhancement Detail : "Accelerated I/O" improves I/O performance by efficient data transfer between VE and VH.This feature is implemented in 'libveaccio'. Accelerated I/O is enabled when this library is loaded at runtime. Please see VEOS document "How to execute VE program" for usage. Summary : Support of NON-ATOMIC-IO on VE architecture Type : enhancement Detail : All read/write family and send/recv family system call handlers (except send/recv using struct msghdr and mmsghdr) have been enhanced to support I/O operations in non atomic mode. In non atomic mode I/O if the requested I/O size is greater than 64MB, the I/O operation will be broken into multiple fix I/O each of size 64MB. This non atomic I/O feature will reduce the memory footprint at VH side. Summary : Fix of issue where ve_exec may terminate abnormally with SIGSEGV Type : bugfix Detail : When VE process is running in multithreading environment and a VE thread invokes exit_group(), exit_group() procedure unmaps a memory region, used for tracing status of VE task in ve_exec process. If at the same time another thread tries to access the mapped region, segmentation fault will occur. Summary : Fix of the issue where VE process hangs during ve_get_regvals() Type : bugfix Detail : VE task is never scheduled on VE core due to some wrong VE task handling at VEOS while serving ve_get_regvals(). As a result VE process seems to be hanged. Summary : Fix of the issue where redundant entries are added to external variable "environ" when execve() syscall is invoked. Type : bugfix Detail : When VE program invokes execve() recursively to load VE binary, redundant entries of some environment variables like LOG4C_RCPATH, VE_EXEC_PATH etc. gets added to global extern variable "environ". This may lead to failure of execve() syscall when the total number of environment variable becomes large. Summary : Fix the issue where VE 'sadc' command do not exit. Type : bugfix Detail : VE 'sadc' command hangs due to incorrect HW data file parsing. Therefore, 'sadc' processes executed through cron job at intervals run indefinitely. 1.3.2 Summary : Performance improvement in write() family syscall. Type : enhancement Detail : Updated the system call handling for reading the data from VE memory to improve the performance of write() family syscall like write(), writev(), pwrite() and pwritev(). Summary : Improved VE and VH data transfer performance. Type : enhancement Detail : The management function of VH memory for data transfer which VE driver offers is improved. By the improvement which is for VH memory management list, the performance of data transfer between VE to VH, like read()/write() system call, is improved. Summary : Improved log messages of VE AIO. Type : enhancement Detail : Improved log messages. When a process is aborted by a VE AIO error, an error message is printed to standard error (stderr). Summary : Fix of the issue where VEOS aborted while scheduling vforked VE process. Type : bugfix Details : The VE task struct of vforked VE process is not properly populated at VEOS. Due to this during scheduling of vforked child VEOS aborted. Summary : Fix of the issue where vforked VE process remains in zombie state. Type : bugfix Details : VE process remains in zombie state as deadlock occurred in VEOS when vforked child VE process while exiting tries to wake up already exiting parent VE process. Summary : Fix the issue where VE process got hanged when it made an attempt to execute signal handler on alternate stack in cascaded manner. Type : bugfix Details : Due to improper reading of core register VEOS wrongly assumed process is not executing on alternate stack. This lead to undefined behaviour. Summary : Fix the issue where VE process got SIGSEGV while trying to reduce heap below heap start address. Type : bugfix Details : Due to wrong handling in brk() syscall TLS area mapped by libraries is unmapped, which results in VE process termination with SIGSEGV. Summary : Fix the issue where 'freeram' field requested by sysinfo() system call is not updated properly. Type : bugfix Details : VE process gets wrong value of freeram (amount of free memory) because sysinfo() system call returns value which is not updated properly. Summary : Fix for the issue where VE process cleanup did not occur even if it has invoked pthread_exit(). Type : bugfix Detail : VE process cleanup was not done properly when SIGCONT signal was delivered to exiting task. So, the resources corresponding to VE task was not released. Summary : Fix the issue where VE strace command failed to execute VE process correctly when VE process stack is grown. Type : bugfix Detail : VE strace command can not trace grow() system call which performs VE process's stack growing, so VE strace gives undefined behavior. 1.3 Summary : Improved performance of clock_gettime() and gettimeofday() system calls Type : enhancement Detail : Improved performance of clock_gettime() with 'CLOCK_MONOTONIC' clock id and gettimeofday() system calls. When a process or thread makes the request for system time using gettimeofday() or monotonic time using clock_gettime(), then first request will be served using VH syscall. On subsequent requests, time will be calculated at VE end instead of invoking VH syscall for same thread-group. Summary : Supported function to do Asynchronous I/O for VE(VE AIO) Type : enhancement Detail : Asynchronous I/O operation for VE is supported. 'libsysve' library includes VE AIO APIs. VE programs can do their own tasks while data transfer between VE and VH, and actual I/O are done asynchronously. Summary : Added print messages to stderr during failure in loading to help user analyse the cause of failure. Type : enhancement Detail : Additional print messages have been added to stderr in case of failure during loading. This will provide helpful information to user regarding failure in loading an executable. Summary : Fix of issue where VE process may terminate abnormally with SIGBUS Type : bugfix Detail : VE process may terminate abnormally with SIGBUS when user puts a futex variable on bss area of a shared library. Summary : Fix of issue where VE process may terminate abnormally with SIGFPE Type : bugfix Detail : VE process may terminate abnormally with SIGFPE when user invokes mprotect() system call with NULL address as an argument. Summary : Fix of issue where a VE Thread may hang indefinitely. Type : bugfix Detail : Thread would hang indefinitely in a scenario wherein the thread (created using pthread_create) tries to join an already exited parent process (parent process is a related process created using fork). Summary : Fix of issue where write() family system calls are sometimes unsuccessful. Type : bugfix Detail : write() family system calls sometimes fail if fork()/vfork()/system() is invoked in parallel by other thread. The system calls mean write(), writev(), pwrite() and pwritev(). Summary : Fix for the issue where signal mask and pending set of a thread are not preserved across execve(). Type : bugfix Detail : Signal mask of thread is not preserved across execve() i.e. if a thread has blocked a signal and does execve() then for the newly loaded process signal will not be masked. Also the signal pending for VE process at veos are not preserved across execve(). Summary: Fix of issue where VE process returns wrong information of being executing over alternate stack, even though it is not executing on alternate stack. Type : bugfix Detail : The stack information maintained at VEOS is not latest after VE process executing the signal handler over alternate stack does longjump() to return to normal stack. Due to this sigaltstack() syscall returned wrong information of still executing over alternate stack. Summary : Fix of issue where swprintf() returns wrong value for '%lc' format specifier Type : bugfix Detail : swprintf() returns -1 instead of number of bytes written when used with format specifier '%lc'(wide characters). Summary : Fix of issue where user process consumes 64MB memory instead of 2MB for dynamically loaded data used by musl-libc Type : bugfix Detail : When user uses APIs like localtime() with binary compiled with 64MB alignment, then memory consumed internally by library for zoneinfo data present in path "/usr/share/zoneinfo/*" is 64MB although actual requirement is less than 2MB. Summary : Fix of issue where timer_create() system call returned incorrect values instead of returning -1 for Process/Thread CPU clock id. Type : bugfix Detail : When timer_create() system call is invoked with clock_id obtained from clock_getcpuclockid()/pthread_getcpuclockid(), it doesn't fail with errno as EINVAL. (timer_create() for VE doesn't support Process/Thread CPU clock id.) Summary : Fix of issue where clock_nanosleep() system call may sleep for an indefinite time instead of returning -1 for Process/Thread CPU clock id. Type : bugfix Detail : When clock_nanosleep() system call is invoked with clock_id obtained from clock_getcpuclockid()/pthread_getcpuclockid(), it doesn't fail with errno as EINVAL. (clock_nanosleep() for VE doesn't support Process/Thread CPU clock id.) Summary : Fix of issue where log rotation of accounting file fails if process accounting for VE has never been enabled. Type : bugfix Detail : When SELinux is enabled, the logrotate invoked by 'cron.daily' failed to rotate VE accounting file (Only if accounting is never enabled) because of wrong context of directory "/var/opt/nec/ve/account" which should be "acct_data_t" instead of "var_t". 1.2.2 Summary : Increased maximum number of arguments and environment variables allowed during loading and execve system call. Type : enhancement Detail : The maximum number of arguments and environment variables allowed during loading and execve() system call have been increased from 256 to 512. Summary : New API of libveosinfo to get user registers Type : enhancement Detail : New API named ve_get_regvals() is added to libveosinfo. This API can be used to get user registers of a VE process from a VH process. 'veosinfo-devel' package is required to build your program with libveosinfo. Summary : Fix of issue where VEOS terminated abnormally when thread creation request failed at VEOS. Type : bugfix Detail : If thread creation request from main process created by ve_exec fails at VEOS, it may result in abnormal VEOS termination while cleaning up the partially created VE task as certain uninitialized fields were being accessed. Summary : Fix of issue where VE process got killed while executing signal handler. Type : bugfix Detail : VE process terminates abnormally while executing signal handler as VEOS sets unaligned memory address for DMA while creating stack frame of signal handler. Summary : Fix of issue where /dev/veslotN is removed on update Type : bugfix Detail : A link /dev/veslotN was removed unexpectedly when a "change" action was triggered. The "change" action is triggered by 'udevadm trigger' when some RPM packages are updated. Summary : Fix of issue whare updating 'veos' package gets stuck Type : workaround Detail : Updating 'veos' package got stuck under certain conditions because the old version of 'veos' package restarts 'systemd-binfmt' and it never finishes due to the problem relating to autofs. Summary : Fix of issue where top/mpstat/pidstat commands show wrong user and idle percentage. Type : bugfix Detail : 'top' command shows wrong user and idle percentage in %Cpu(s) field of default view. Also 'mpstat' and 'pidstat' commands display wrong %user and %idle, if executed without interval/count. Summary : Fix of issue where 'prtstat' command shows wrong values in some output fields. Type : bugfix Detail : 'prtstat' command shows incorrect data in pgrp, tty_nr, session, tpgid, num_threads, exit_signal fields. Summary : Fix of issue where clock_gettime() system call set incorrect error number. Type : bugfix Detail : clock_gettime() system call failed with errno set to EFAULT instead of EINVAL when an invalid clock_id is passed. Also, when clock_id corresponds to INIT process's clock, then failure is returned with errno set to EINVAL, as VEOS does not have an executing INIT process. 1.2.1 Summary : Load rebalancing feature added to improve the performance of VEOS. Type : enhancement Detail : With load rebalancing feature support, VEOS will relocate a VE task from a VE core which has two or more VE task to the VE core which has no task. This will improve the performance of multi-processes and multi-threads programs including MPI. Summary : Improved I/O performance Type : enhancement Detail : The I/O performance for simultaneous access from many processes is improved by DMA transfer improvement. Summary : Fix of issue where VE user cannot run their programs when user directories mounted with NOSUID Type : bugfix Detail : VEOS cannot execute a program or use a program with execve() if the underlying filesystem of the executable is mounted with "nosuid" mount option. Summary : Fix of issue where clock_gettime() system call filled wrong timespec values in buffer provided. Type : bugfix Detail : clock_gettime() system call will always fill the execution time of the calling process even when the "clock_id" returned by clock_getcpuclockid() corresponded to some other process. Summary : Fix of issue where parent resumes execution before its vforked child has exited. Type : bugfix Detail : VE program will get undesired result of wait() syscall when parent resumes execution before its vforked child has exited. Summary : Fix of issue where VE program having multiple threads terminates abnormally while invoking malloc() and free() in parallel Type : bugfix Detail : VE program having multiple threads which invokes malloc() and free() in parallel terminates abnormally due to protection issue of memory management data maintained by musl-libc. Summary : Fix of issue where mmap() system call get wrong error codes Type : bugfix Detail : VE program will get wrong error codes in certain failure cases of mmap() system call i.e. if invalid size and invalid fd is passed to mmap() then errno EBADF is to be set but before the fix errno EINVAL was set and if MAP_FIXED flag is passed with NULL address to mmap() then errno EPERM is to be set but before the fix ENOMEM was set. Summary : Fix of issue where getresgid() system call get wrong error code Type : bugfix Detail : getresgid() return EFAULT unexpectedly because one of the argument from expected argument list was missing while processing syscall on VEOS. 1.2 Summary : Improved DMA transfer for NEC system software Type : enhancement Detail : The performance of DMA transfer for NEC system software which is linked to VE programs, such as ScaTeFS library and so on, is improved. It is a dedicated function for NEC system software; therefore, NEC doesn't support any problem when you use this function directly without NEC system software. Summary : Response improvement of VEOS about memory release request from a VE process. Type : enhancement Detail : The response of releasing memory request is improved because VEOS clears released physical memory asynchronously. When VEOS doesn't have enough cleared physical memory to allocate, VE process is kept waiting for completion of clearing memory. Summary : Improved data-transfer performance and reduced memory consumption Type : enhancement Detail : The performance of large size data transfer done via system call (read(), write() and so on) is improved. Summary : License change of libvepseudo to LGPL Type : enhancement Detail : The license of libvepseudo which is one of components of Pseudo process is changed from GPL to LGPL. Summary : Fix of issue where VE process terminates abruptly Type : bugfix Detail : VE program execution may terminate abruptly while doing vfork() when the caller has already mapped a file on memory of size zero, which restarts veos unexpectedly and terminates of all the VE processes executing on that VE node. Summary : Fix of issue where VEOS virtual memory usage keeps on increasing. Type : bugfix Detail : When VE process creates coredump then virtual memory usage of VEOS increases gradually. However, virtual memory usage is not restored to original amount even after coredump creation was over. Summary : VE process did not receive SIGSEGV signal even though the hardware exception "host missing page exception" occurred on VE. Type : bugfix Detail : VE process will not be terminated with expected signal(SIGSEGV) and can continue its execution. Summary : Fix for issue where VE child process creations using fork() fails. Type : bugfix Detail : Once a VE process invokes sched_setaffinity with invalid CPU_SET every subsequent invocation of fork() system call fails. Summary : Fix of issue where psacct-ve service failed when VH boots. Type : bugfix Detail : VE 'psacct-ve' service depends upon running VEOS but, on VH reboot, this service gets restarted before VEOS restarts (if service is enabled) leading to the failure in restart of 'psacct-ve' service. Summary : Fix the issue where count of VE blocked processes was wrong for vmstat and sar command. Type : bugfix Detail : VE specific "vmstat" and "sar" command were displaying some count of blocked process but VEOS doesn't maintain any such state which can provide count of 'blocked' processes (process waiting specifically for IO). Therefore, blocked process count should be 0. 1.1.1 Summary : Fix of issue where VEOS aborted during execve() processing if the task is killed by any signal Type : bugfix Detail : During execve() processing, if the task is killed by any signal, VEOS fails to update a task struct flag which lead to removing of already removed task from core. It resulted in veos abort. Summary : Fix of issue where VEOS aborted when SIGKILL to thread group leader was sent during execve() Type : bugfix Detail : While VEOS is handling execve() request from a thread and serving SIGKILL signal for the thread group leader at the same time, VEOS aborts. Summary : Fix of issue where VEOS consumes a huge VH memory Type : bugfix Detail : VEOS consumes a huge VH memory when VEOS creates a core dump file of a VE process whose memory segment size is huge. Summary : Fix of issue where VEOS may show undefined behavior due to loss of synchronisation while updating its data structures. Type : bugfix Detail : VEOS can show undefined behaviour due to loss of synchronisation. This can occur while, setting of VE process's CPU affinity is ongoing in parallel with some other request ongoing for same VE process at VEOS. Summary : Fix of issue where DMA descriptor table was wrongly updated for VE process Type : bugfix Detail : VE program which is doing DMA transfer sometimes will get wrong content of DMA descriptor table as migration was not done properly by VEOS. Summary : Fix of issue where VE program may return fork() failure even if child process is created successfully Type : bugfix Detail : Due to a race scenario in fork() handling, even if child process is created successfully, parent may return fork() failure as EAGAIN Summary : Fix of issue where a VE process may hang indefinitely. Type : bugfix Detail : A VE process may hang indefinitely as it is not being scheduled on core. Summary : Fix of issue where VE program terminates abnormally Type : bugfix Detail : VE program terminates abnormally while executing signal handlers due to memory corruption done by VEOS while creating stack frame of signal handler. Summary : Fix of issue where memory is leaked at VH side Type : bugfix Details : Memory is leaked at VH side when a VE process creates a child process and the child process terminates. When the parent VE process terminates, leaked memory is freed. Summary : Fix of issue where VE process execution becomes slow Type : bugfix Details : After executing multiple VE process, VEOS skips scheduling decision in some cases due to inconsistent state of its core's data structure. Due to this performance of VE process degrades. Summary : Fix of issue where GDB backtrace command Type : bugfix Detail : The "backtrace" command of GDB does not display the frame which is interrupted by signal. Summary : Improved privileged DMA transfer performance Type : enhancement Detail : The performance of privileged DMA transfer for VEOS is improved. It improves performance of system call(read()/write()) and etc. Summary : Supported function to improve data-transfer rate of libraries such as MPI etc. Type : enhancement Detail : To improve data-transfer rate of the MPI library or other system libraries, VEOS provides a function which allows VE program to access to System V shared memory on VH. These libraries create System V shared memory using "huge pages". This function is used only by appropriate MPI library or other system libraries. And NEC doesn't support any problem when you use this function directly without these libraries. Summary : Improved context switch performance Type : enhancement Detail : The performance of a context switch between VE processes/threads is improved. Summary : Improved VE process core dump Type : enhancement Detail : Improved core dumping performance of VE process. Earlier VE core dumping was taking longer time in certain scenarios when large I/O is in progress on VH host. During that duration all the threads of the thread group remain in uninterruptible sleep. This duration is reduced now. Summary : Support of the system activity data collection and store by sar Type : enhancement Detail : The system activity data collection and store by sar is available. Updated VE sysstat service from 've_sysstat' to 've_sysstat@'. (where N is VE node number) Summary : Display "VEOS RESTART" instead of "LINUX RESTART" in message of 'sar' command Type : enhancement Detail : Whenever VEOS is restarted then output messages from 'sar' command will show "VEOS RESTART" instead of "LINUX RESTART". Summary : Porting of 'ipcs and ipcrm' Type : enhancement Detail : Added support of 'ipcs' and 'ipcrm' commands in 'util-linux-ve' package. 1.0.3 Summary : RHEL7.4/x86_64 support Type : enhancement Detail : RHEL7.4/x86_64 is newly supported. Please see README for Supported Platforms and Operating Systems. Summary : Distribute binfmt configuration file Type : enhancement Detail : A configuration file for VE binary format (ve.conf) is stored to /usr/lib/binfmt.d/. A user can execute a VE program without specifying ve_exec command. If multiple VE nodes exist, the VE node which executes VE program is specified by environment variable (VE_NODE_NUMBER). 1.0.2 Summary : Fix of issue where VE process is killed when a signal handler is invoked Type : bugfix Detail : A VE process may be killed on executing a signal handler due to insufficient stack space required for signal handler execution and frame setup Summary : Fix of issue where VE process is hanged Type : bugfix Detail : VE program execution may stall while executing signal handler. This can happen in rare race scenario when VE process consume all of its stack and then signal is generated for the process. Summary : Fix of issue where VE process terminates abruptly Type : bugfix Detail : VE program execution may terminates abruptly when parallel request of shmdt() and shmctl() system call arrives at veos for the same shared memory segment, which restarts veos unexpectedly and termination of all the VE processes executing on that VE node. Summary : Fix of issue where pthread_create() fails if large TLS size is used with user defined thread stack Type : bugfix Detail : With large TLS size and user defined stack area, pthread_create() fails to set up stack for thread and returns EAGAIN. Summary : Fix the issue where "sar -m" with interval producing error Type : bugfix Detail : The "sar" command is showing error while trying to get sensor information. Summary : Signal delivery to VE task is not delayed until the execution of kill(2) series system call. Type : enhancement Detail : When VE task sends signal to itself through kill(2) series system call like tkill(2),tgkill(2) etc. The signals are immediately delivered to VE task. Summary : Improve log messages of VEOS Type : enhancement Detail : Change log level of VEOS messages to applicable level. Summary : Change package file name Type : enhancement Detail : TSUBASA standard package name rule are applied to all of VEOS package files. 1.0.1 - initial release Trademarks Linux is a registered trademark of Linus Torvalds in the United States and other countries. InfiniBand is a trademark or service mark of InfiniBand Trade Association. Mellanox is trademark or registered trademark of Mellanox Technologies in Israel and other countries. All other product, brand, or trade names used in this publication are the trademarks or registered trademarks of their respective trademark owners.