What type of malware specializes in avoiding detection by accessing lower layers of the operating system or by using undocumented functions to make alterations?

Rootkit

Explanation:

A rootkit is a type of malware that can give authorization to the computer without knowing the user. The rootkit is providing administrator access. The Rootkit can control the overall system without knowing the user it means it has the capability to access or change the configuration of the system.

A rootkit is avoiding detection to the user by accessing the lower layers of the operating system because it is difficult to detect this malware.