Skip to main content

Search

Welcome to Tibero GTS!

Increased CPU Usage of the systemd-udevd Process When Using TAS in a Linux Environment

Document Type | Troubleshooting

Category | Administration

Applicable Product Version | 7FS02PS

Document Number | TADTS074

 

This article describes the cause and solution for continuous CPU usage by the systemd-udevd process after installing TAS in a Linux environment.

 

Issue

On only one of the servers configured with TAS, rather than on all servers, continuous CPU usage by the /usr/lib/systemd/systemd-udevd process occurs at 3-second intervals.  

# top -p $(pgrep -d, systemd-udevd) -d 0.5

top - 18:40:54 up 132 days,  7:05,  2 users,  load average: 0.89, 1.89, 2.04
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.0 us,  2.2 sy,  0.0 ni, 91.0 id,  5.2 wa,  0.2 hi,  0.2 si,  0.0 st
MiB Mem :  24062.5 total,   8846.5 free,  10483.1 used,   4732.9 buff/cache
MiB Swap:  32764.0 total,  28402.4 free,   4361.6 used.   9410.7 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 112580 root      20   0  118780  11520   5336 S   2.0   0.0   0:04.40 systemd-udevd

When monitoring with udevadm, repeated change events occur.

# udevadm monitor --kernel --udev --property

UDEV  [11431041.072503] change   /devices/.../block/sdc (block)
ACTION=change
DEVPATH=/devices/.../block/sdc
DEVNAME=sdc
DEVTYPE=disk
MAJOR=8
MINOR=32
SEQNUM=3140620
SUBSYSTEM=block
--Repeated events occur

 

Cause

At one of the TAS instances, the DMON thread opens and closes all DISKs matching AS_DISKSTRING every 3 seconds to check their status.
The DISK open/close events caused by the DMON thread are detected by UDEV's inotify, causing the UDEV settings created when configuring the TAS DISKs to be applied repeatedly with the same content. This process results in unnecessary CPU usage.   
--TAS tip 
AS_DISKSTRING="/dev/tasdisks/tas-redo-disk-*,/dev/tasdisks/tas-data-disk-*,/dev/tasdisks/tas-arch-disk-*"

# cat /etc/udev/rules.d/60-tas-disk.rules
KERNEL=="sd?", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id -g -u -d %N", RESULT=="36000c291fdc9305ae0861208f4d56c8e", SYMLINK+="tasdisks/tas-redo-disk-1", OWNER="tibero", GROUP="dba", MODE="0660"
KERNEL=="sd?", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id -g -u -d %N", RESULT=="36000c29faed404a64023aa2c053948f7", SYMLINK+="tasdisks/tas-redo-disk-2", OWNER="tibero", GROUP="dba", MODE="0660"
KERNEL=="sd?", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id -g -u -d %N", RESULT=="36000c29bf5e3c2f05f4df7eaef5eccb6", SYMLINK+="tasdisks/tas-data-disk-1", OWNER="tibero", GROUP="dba", MODE="0660"
KERNEL=="sd?", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id -g -u -d %N", RESULT=="36000c291aa08cea3f427ce13b9ced0d3", SYMLINK+="tasdisks/tas-data-disk-2", OWNER="tibero", GROUP="dba", MODE="0660"
KERNEL=="sd?", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id -g -u -d %N", RESULT=="36000c29e4695b6bd1efc13eed3fbd9b7", SYMLINK+="tasdisks/tas-arch-disk-1", OWNER="tibero", GROUP="dba", MODE="0660"
KERNEL=="sd?", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id -g -u -d %N", RESULT=="36000c29774499ea5a03ee60625ee250d", SYMLINK+="tasdisks/tas-arch-disk-2", OWNER="tibero", GROUP="dba", MODE="0660"
Note
DMON (Disk Monitor): Checks the status of the DISKs managed by TAS at regular intervals and performs failure and recovery operations.

 

Solutions

Add the OPTIONS:="nowatch" statement to the UDEV RULES settings for configuring TAS DISKs to disable UDEV's inotify watch for DISKs with no changes.

# cat /etc/udev/rules.d/60-tas-disk.rules
KERNEL=="sd?", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id -g -u -d %N", RESULT=="36000c291fdc9305ae0861208f4d56c8e", SYMLINK+="tasdisks/tas-redo-disk-1", OWNER="tibero", GROUP="dba", MODE="0660", OPTIONS:="nowatch"
KERNEL=="sd?", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id -g -u -d %N", RESULT=="36000c29faed404a64023aa2c053948f7", SYMLINK+="tasdisks/tas-redo-disk-2", OWNER="tibero", GROUP="dba", MODE="0660", OPTIONS:="nowatch"
KERNEL=="sd?", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id -g -u -d %N", RESULT=="36000c29bf5e3c2f05f4df7eaef5eccb6", SYMLINK+="tasdisks/tas-data-disk-1", OWNER="tibero", GROUP="dba", MODE="0660", OPTIONS:="nowatch"
KERNEL=="sd?", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id -g -u -d %N", RESULT=="36000c291aa08cea3f427ce13b9ced0d3", SYMLINK+="tasdisks/tas-data-disk-2", OWNER="tibero", GROUP="dba", MODE="0660", OPTIONS:="nowatch"
KERNEL=="sd?", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id -g -u -d %N", RESULT=="36000c29e4695b6bd1efc13eed3fbd9b7", SYMLINK+="tasdisks/tas-arch-disk-1", OWNER="tibero", GROUP="dba", MODE="0660", OPTIONS:="nowatch"
KERNEL=="sd?", SUBSYSTEM=="block", PROGRAM=="/lib/udev/scsi_id -g -u -d %N", RESULT=="36000c29774499ea5a03ee60625ee250d", SYMLINK+="tasdisks/tas-arch-disk-2", OWNER="tibero", GROUP="dba", MODE="0660", OPTIONS:="nowatch"

Verify that the CPU usage of the systemd-udevd process has decreased and that repeated change events no longer occur when monitoring with udevadm.

# top -p $(pgrep -d, systemd-udevd) -d 0.5

top - 20:11:35 up 132 days,  8:36,  2 users,  load average: 1.66, 1.47, 1.41
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.2 us,  0.5 sy,  0.0 ni, 91.5 id,  7.5 wa,  0.2 hi,  0.0 si,  0.0 st
MiB Mem :  24062.5 total,   9255.0 free,  10546.2 used,   4261.3 buff/cache
MiB Swap:  32764.0 total,  28358.2 free,  4405.8 used.   9390.6 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 112580 root      20   0  118780  11660   5476 S   0.0   0.0   0:10.17 systemd-udevd


# udevadm monitor --kernel --udev --property

monitor will print the received events for:
UDEV - the event which udev sends out after rule processing
KERNEL - the kernel uevent
--No repeated events