Managing Disk Space

This section serves as a guide to managing disk space to ensure that your disks or partitions do not fill up. It is applicable to all use cases, including when using your own image or the CA hardened image.
apip42
This section serves as a guide to managing disk space to ensure that your disks or partitions do not fill up. It is applicable to all use cases, including when using your own image or the CA hardened image.
Common Causes of Full Partitions
Both Docker and rsyslog store their data under the 
/var
 partition (or directory). The partition may fill up under these circumstances:
  • Any of the following partitions are too small: 
    /var, /var/log, /var/lib/docker
    .
  • Old Docker containers and images that are no longer in use are not cleaned out regularly.
  • A system experiences heavy load, leading to a large volume of logs being collected.
  • Logging is configured with a rotation policy that allows logs to remain on disk for too long.
We recommend regular management of your disk space using these methods:
2
2
Partition Size Check
Recommended Minimum Sizes
We recommend allotting the following minimum sizes for partitions on low-volume systems. If a system is expected to experience heavy loads, consider allotting twice the disk space or more.
Partition Name
Disk Space
Used For
/
10 GB
unpacked copy of offline installer to /opt
/tmp
16 GB
temporary storage of packed (gzipped) copy of offline installer
/var/log
10 GB
logs from rsyslog and journald
/var/log/audit
10 GB
logs from SELinux
/var
20 GB
docker containers, images, and volumes in /var/lib/docker
Checking Your Partition
To see how much space is left on each of your hard disk partitions, use the 
df
 command:
df -h
An example output of a system using CA hardened image partitioning layout is shown next:
Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg00-lv_root 40G 1.3G 37G 4% / devtmpfs 16G 0 16G 0% /dev tmpfs 16G 0 16G 0% /dev/shm tmpfs 16G 33M 16G 1% /run tmpfs 16G 0 16G 0% /sys/fs/cgroup /dev/xvda1 976M 55M 855M 7% /boot /dev/mapper/vg00-lv_home 9.8G 45M 9.2G 1% /home /dev/mapper/vg00-lv_tmp 16G 45M 15G 1% /tmp /dev/mapper/vg00-lv_var 141G 120M 134G 1% /var /dev/mapper/vg00-lv_vartmp 9.8G 44M 9.2G 1% /var/tmp /dev/mapper/vg00-lv_log 9.8G 47M 9.2G 1% /var/log /dev/mapper/vg00-lv_audit 9.8G 46M 9.2G 1% /var/log/audit tmpfs 3.2G 0 3.2G 0% /run/user/1000
Although this layout is sufficient for most users, the following limitations can be observed:
  • Collecting more than 10 GB of system logs in less than 28 days on a system with default logging configurations would result in a full 
    /var/log
     partition.
  • Downloading multiple copies of the offline installer and extracting them in the 
    /home
     or 
    /tmp
     partition would result in filling those partitions.
  • Performing many upgrades of the Portal application without pruning old Docker containers and images could result in a full 
    /var
     partition.
Docker Pruning
Docker stores data in the 
/var/lib/docker
 directory, which will be stored in the 
/var
 partition if using the CA hardened image partition layout.
When a container exits, Docker does not automatically remove the container. When performing a CA API Portal upgrade and new images are downloaded, Docker does not automatically remove the old images.
To manually remove old Docker containers and images, run:
docker system prune -a
Warning!
Ensure that the CA API Portal is running when using this command. Because pruning removes all unused containers and images, any stopped containers are removed, and any images that are not associated with a running container are also removed.
Info:
For more information, see Docker System Prune documentation.
Log Management
Manual Log Management
The default configuration on a modern (RHEL 7.X and CentOS 7.x) Linux system is to write logs to the 
/var/log/messages
 file. Each week, this file is recreated and the old one is renamed with the current date appended to the name.
By default, 4 archived log files are kept on disk. The following example shows how the configuration looks on a lightly used system:
[[email protected] /var/log]$ ls -alh messages* -rw-------. 1 root root 793K Apr 19 12:01 messages -rw-------. 1 root root 954K Mar 26 13:40 messages-20180326 -rw-------. 1 root root 912K Apr 3 11:25 messages-20180403 -rw-------. 1 root root 787K Apr 9 10:09 messages-20180409 -rw-------. 1 root root 1.1M Apr 15 03:15 messages-20180415
If a system is heavily used and collects a large volume of logs, you can adjust the log policy to keep fewer logs, to rotate logs more often, or both.
To manually adjust your logging policy:
  1. Access log policy settings from the 
    /etc/logrotate.conf
     file. Default settings are shown next:
    # see "man logrotate" for details # rotate log files weekly weekly # keep 4 weeks worth of backlogs rotate 4
  2. Edit the file to your desired values. For example, if you want to adjust the policy to rotate files 
    daily
     and only keep 
    10 days
     of logs, enter the following values:
    # see "man logrotate" for details # rotate log files daily daily # keep 10 days worth of backlogs rotate 10
  3. Restart rsyslog service to load the latest settings:
    sudo systemctl restart rsyslog
If the amount of logging volume on a system suddenly increases, the disks can still fill up if the retention period is too long and the volume is too high. This requires manual intervention to tune the logging parameters further. For a more automated approach, see the next option.
Automated Log Management
For a more automated approach that removes old logs based on lack of disk space, the system can be configured to use journald in persistent logging mode.
To implement automated log management via journald:
  1. Enable persistent logging by configuring the following options in 
    /etc/systemd/journald.conf
    :
    Storage=persistent RateLimitInterval=1s RateLimitBurst=10000 SystemMaxUse=8G SystemKeepFree=2G SystemMaxFileSize=1G ForwardToSyslog=no
    This configuration ensures that:
    • journald stores the journal files on disk in the 
      /var/log/journal
       directory instead of in memory at the default location of /run/log/journal, and will not forward entries to syslog.
    • It allows every service to send it 10,000-log entries per second and rotates the files when they reach 1 GB. 
    • It uses up to 8 GB of disk space to store these files and ensures that 2 GB is left available for files that are not controlled by journald (such as syslog).
    The preceding configuration assumes a 10-GB /var/log partition. If the /var/log partition is larger, adjust the
    SystemMaxUse
    and
    SystemKeepFree
    parameters to add up to the total size of the /var/log partition.
  2. To prevent rsyslog from capturing and duplicating the data that journald is already writing to disk, ensure that the following entries are
    commented out
    in 
    /etc/rsyslog.conf
    :
    #ModLoad imjournal # provides access to the systemd journal #$IMJournalStateFile imjournal.state
  3. Restart journald and rsyslog to load the latest settings:
    sudo systemctl restart systemd-journald sudo systemctl restart rsyslog