docker容器根目錄為只讀的解決辦法

luoyibu 發(fā)布于2019-06-28 15:05 / 4019人閱讀

摘要：?jiǎn)栴}描述今天在啟動(dòng)容器的時(shí)候發(fā)現(xiàn)一段時(shí)間后宿主機(jī)上所有的容器的根目錄全部變成了只讀，并且宿主機(jī)日志報(bào)磁盤相關(guān)的錯(cuò)容器內(nèi)結(jié)果如下宿主機(jī)報(bào)錯(cuò)如下

問題描述

今天在啟動(dòng)docker容器的時(shí)候發(fā)現(xiàn)一段時(shí)間后宿主機(jī)上所有的容器的根目錄全部變成了只讀，并且宿主機(jī)message日志報(bào)磁盤相關(guān)的錯(cuò)

容器內(nèi)mount結(jié)果如下

[root@zk-1 ~]# mount
/dev/mapper/docker-253:0-4298664622-7830c39693a73c13e80cf2a22a46558b22adcc5adf94ca1c893f44ece878601e on / type ext4 (ro,relatime,stripe=16,data=ordered)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,nosuid,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
/dev/mapper/centos-root on /nfsc type xfs (rw,relatime,attr2,inode64,noquota)
tmpfs on /run/secrets type tmpfs (rw,nosuid,nodev,noexec,relatime)
/dev/mapper/centos-root on /etc/resolv.conf type xfs (rw,relatime,attr2,inode64,noquota)
/dev/mapper/centos-root on /etc/hostname type xfs (rw,relatime,attr2,inode64,noquota)
/dev/mapper/centos-root on /etc/hosts type xfs (rw,relatime,attr2,inode64,noquota)

宿主機(jī)報(bào)錯(cuò)如下

Jan 13 15:06:01 docker2 systemd: Starting Session 6 of user root.
Jan 13 15:06:01 docker2 systemd: Started Session 6 of user root.
Jan 13 15:06:31 docker2 systemd: Starting Session 7 of user root.
Jan 13 15:06:31 docker2 systemd-logind: New session 7 of user root.
Jan 13 15:06:31 docker2 systemd: Started Session 7 of user root.
Jan 13 15:07:01 docker2 systemd: Starting Session 8 of user root.
Jan 13 15:07:01 docker2 systemd: Started Session 8 of user root.
Jan 13 15:07:38 docker2 kernel: device-mapper: thin: 253:3: reached low water mark for data device: sending event.
Jan 13 15:07:44 docker2 kernel: device-mapper: thin: 253:3: switching pool to out-of-data-space (queue IO) mode
Jan 13 15:08:01 docker2 systemd: Starting Session 9 of user root.
Jan 13 15:08:01 docker2 systemd: Started Session 9 of user root.
Jan 13 15:08:44 docker2 kernel: device-mapper: thin: 253:3: switching pool to out-of-data-space (error IO) mode
Jan 13 15:08:44 docker2 kernel: EXT4-fs warning (device dm-5): ext4_end_bio:332: I/O error -28 writing to inode 1320256 (offset 13077839872 size 8388608 starting block 9269408)
Jan 13 15:08:44 docker2 kernel: Buffer I/O error on device dm-5, logical block 9269408
Jan 13 15:08:44 docker2 kernel: Aborting journal on device dm-4-8.
Jan 13 15:08:44 docker2 kernel: Buffer I/O error on device dm-5, logical block 9269409
Jan 13 15:08:44 docker2 kernel: Buffer I/O error on device dm-5, logical block 9269410
Jan 13 15:08:44 docker2 kernel: EXT4-fs error (device dm-4): ext4_journal_check_start:56: Detected aborted journal
Jan 13 15:08:44 docker2 kernel: EXT4-fs (dm-4): Remounting filesystem read-only
Jan 13 15:08:44 docker2 kernel: Buffer I/O error on device dm-5, logical block 9269411
Jan 13 15:08:44 docker2 kernel: Buffer I/O error on device dm-5, logical block 9269412
Jan 13 15:08:44 docker2 kernel: Buffer I/O error on device dm-5, logical block 9269413
Jan 13 15:08:44 docker2 kernel: Buffer I/O error on device dm-5, logical block 9269414
Jan 13 15:08:44 docker2 kernel: Buffer I/O error on device dm-5, logical block 9269415
Jan 13 15:08:44 docker2 kernel: Buffer I/O error on device dm-5, logical block 9269416
Jan 13 15:08:44 docker2 kernel: Buffer I/O error on device dm-5, logical block 9269417
Jan 13 15:08:44 docker2 kernel: EXT4-fs warning (device dm-5): ext4_end_bio:332: I/O error -28 writing to inode 1320256 (offset 13077839872 size 8388608 starting block 9269424)
Jan 13 15:08:44 docker2 kernel: EXT4-fs warning (device dm-5): ext4_end_bio:332: I/O error -28 writing to inode 1320256 (offset 13077839872 size 8388608 starting block 9269440)
Jan 13 15:08:44 docker2 kernel: EXT4-fs warning (device dm-5): ext4_end_bio:332: I/O error -28 writing to inode 1320256 (offset 13077839872 size 8388608 starting block 9269456)
Jan 13 15:08:44 docker2 kernel: EXT4-fs warning (device dm-5): ext4_end_bio:332: I/O error -28 writing to inode 1320256 (offset 13077839872 size 8388608 starting block 9269472)
Jan 13 15:08:44 docker2 kernel: EXT4-fs warning (device dm-5): ext4_end_bio:332: I/O error -28 writing to inode 1320256 (offset 13077839872 size 8388608 starting block 9269488)
Jan 13 15:08:44 docker2 kernel: EXT4-fs warning (device dm-5): ext4_end_bio:332: I/O error -28 writing to inode 1320256 (offset 13077839872 size 8388608 starting block 9269504)
Jan 13 15:08:44 docker2 kernel: EXT4-fs warning (device dm-5): ext4_end_bio:332: I/O error -28 writing to inode 1320256 (offset 13077839872 size 8388608 starting block 9269520)
Jan 13 15:08:44 docker2 kernel: EXT4-fs warning (device dm-5): ext4_end_bio:332: I/O error -28 writing to inode 1320256 (offset 13077839872 size 8388608 starting block 9269536)
Jan 13 15:08:44 docker2 kernel: EXT4-fs warning (device dm-5): ext4_end_bio:332: I/O error -28 writing to inode 1320256 (offset 13077839872 size 8388608 starting block 9269552)
Jan 13 15:08:44 docker2 kernel: Aborting journal on device dm-5-8.
Jan 13 15:08:44 docker2 kernel: EXT4-fs error (device dm-5) in ext4_da_write_end:2782: IO failure
Jan 13 15:08:44 docker2 kernel: EXT4-fs error (device dm-5): ext4_journal_check_start:56: Detected aborted journal
Jan 13 15:08:44 docker2 kernel: EXT4-fs (dm-5): Remounting filesystem read-only
Jan 13 15:08:44 docker2 kernel: EXT4-fs error (device dm-5) in ext4_do_update_inode:4504: Journal has aborted
Jan 13 15:08:44 docker2 kernel: EXT4-fs error (device dm-5): mpage_map_and_submit_extent:2229: comm kworker/u98:3: Failed to mark inode 1320256 dirty
Jan 13 15:08:44 docker2 kernel: EXT4-fs error (device dm-5) in ext4_writepages:2520: IO failure

第一反映是查看磁盤空間使用情況

[root@zk-1 ~]# df -Th
Filesystem           Type   Size  Used Avail Use% Mounted on
/dev/mapper/docker-253:0-4298664622-7830c39693a73c13e80cf2a22a46558b22adcc5adf94ca1c893f44ece878601e
                     ext4    99G   49G   46G  52% /
tmpfs                tmpfs  126G     0  126G   0% /dev
shm                  tmpfs   64M     0   64M   0% /dev/shm
tmpfs                tmpfs  126G     0  126G   0% /sys/fs/cgroup
/dev/mapper/centos-root
                     xfs    1.7T  113G  1.6T   7% /nfsc
tmpfs                tmpfs  126G     0  126G   0% /run/secrets
/dev/mapper/centos-root
                     xfs    1.7T  113G  1.6T   7% /etc/resolv.conf
/dev/mapper/centos-root
                     xfs    1.7T  113G  1.6T   7% /etc/hostname
/dev/mapper/centos-root
                     xfs    1.7T  113G  1.6T   7% /etc/hosts

根目錄下還剩余46G未使用；感覺甚是怪異，于是網(wǎng)上搜索很多資料終于有了相關(guān)解釋
參考網(wǎng)絡(luò)上的一片文章：http://jpetazzo.github.io/201...

具體原因

docker服務(wù)啟動(dòng)的時(shí)候默認(rèn)會(huì)創(chuàng)建一個(gè)107.4G的data文件，而后啟動(dòng)的容器的所有更改內(nèi)容全部存儲(chǔ)至這個(gè)data文件中；也就是說當(dāng)容器內(nèi)產(chǎn)生的相關(guān)data數(shù)據(jù)超過107.4G后容器就再也沒有多余的空間可用，從而導(dǎo)致所有容器的根目錄變?yōu)橹蛔x！

宿主機(jī)的docker info信息如下

[root@docker2 ~]# docker info
Containers: 169
Images: 1672
Storage Driver: devicemapper
 Pool Name: docker-253:0-4298664622-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 107.4 GB
 Data Space Total: 107.4 GB
 Data Space Available: 0 B
 Metadata Space Used: 137.4 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.01 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.107-RHEL7 (2015-10-14)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.2.1-1.el7.elrepo.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 40
Total Memory: 251.9 GiB
Name: docker2.stg.1qianbao.com
ID: JMZF:IQ6H:RDBK:XNSN:W3IO:ZAQH:RRFB:XRIT:4I72:KOKD:R34K:FD5L
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

由于我的容器比較多（169個(gè)jboss應(yīng)用），故直接導(dǎo)致整個(gè)環(huán)境不可用。

解決方案

停止docker服務(wù)

service docker stop

刪除/var/lib/docker下面的所有文件（刪除后你的鏡像和容器都沒有了，建議將有用的鏡像先備份或者上傳至倉(cāng)儲(chǔ)里面）

rm -rf /var/lib/docker/*

使用更大的文件或磁盤或邏輯卷創(chuàng)建/var/lib/docker/devicemapper/devicemapper/data文件

使用文件：dd if=/dev/zero of=/var/lib/docker/devicemapper/devicemapper/data bs=1G count=0 seek=1000這樣將會(huì)創(chuàng)建一個(gè)虛擬的1000G大小的data文件，如果不加seek參數(shù)count直接為1000的話則是創(chuàng)建了一個(gè)結(jié)結(jié)實(shí)實(shí)的1000G的文件

使用磁盤：ln -s /dev/sdb /var/lib/docker/devicemapper/devicemapper/data

使用邏輯卷：ln -s /dev/mapper/centos-dockerdata /var/lib/docker/devicemapper/devicemapper/data

我用的是第一種使用文件的方法創(chuàng)建了一個(gè)1.6T的虛擬文件

mkdir -p /var/lib/docker/devicemapper/devicemapper/
dd if=/dev/zero  of=/var/lib/docker/devicemapper/devicemapper/data bs=1G count=0 seek=1600

創(chuàng)建完成后啟動(dòng)docker服務(wù)

service docker start

這時(shí)再看下docker info中的data池

[root@docker2 ~]# docker info
Containers: 169
Images: 1701
Storage Driver: devicemapper
 Pool Name: docker-253:0-2355438-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 90.9 GB
 Data Space Total: 1.611 TB
 Data Space Available: 1.52 TB
 Metadata Space Used: 147.5 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.107-RHEL7 (2015-10-14)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.2.1-1.el7.elrepo.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 40
Total Memory: 251.9 GiB
Name: docker2.stg.1qianbao.com
ID: JMZF:IQ6H:RDBK:XNSN:W3IO:ZAQH:RRFB:XRIT:4I72:KOKD:R34K:FD5L
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

此后，你的data文件有多大就決定了你的宿主機(jī)上所有容器可用的空間的大??！

另一方面：也可以通過docker啟動(dòng)參數(shù)的--storage-opt選項(xiàng)來限制每個(gè)容器初始化的磁盤大小，如-storage-opt dm.basesize=80G 這樣每個(gè)容器啟動(dòng)后起根目錄的總空間就是80G

[root@zk-1 ~]# df -Th
Filesystem           Type   Size  Used Avail Use% Mounted on
/dev/mapper/docker-253:0-27661746-8b7f953fb4759982ad82235c27e39dfe7190b55180d63cbcf3aa2fdc6569d43a
                     ext4    79G  785M   74G   2% /
tmpfs                tmpfs  126G     0  126G   0% /dev
shm                  tmpfs   64M     0   64M   0% /dev/shm
tmpfs                tmpfs  126G     0  126G   0% /sys/fs/cgroup
tmpfs                tmpfs  126G     0  126G   0% /run/secrets
/dev/mapper/centos-root
                     xfs    1.5T  338G  1.2T  23% /wls/wls81/zookeeper.out
/dev/mapper/centos-root
                     xfs    1.5T  338G  1.2T  23% /etc/resolv.conf
/dev/mapper/centos-root
                     xfs    1.5T  338G  1.2T  23% /etc/hostname
/dev/mapper/centos-root
                     xfs    1.5T  338G  1.2T  23% /etc/hosts

云服務(wù)器 GPU云服務(wù)器的解決辦法微軟的解決辦法 use的解決辦法換ip的解決辦法

文章版權(quán)歸作者所有，未經(jīng)允許請(qǐng)勿轉(zhuǎn)載,若此文章存在違規(guī)行為，您可以聯(lián)系管理員刪除。

轉(zhuǎn)載請(qǐng)注明本文地址：http://systransis.cn/yun/26519.html

發(fā)表評(píng)論

登陸后可評(píng)論

0條評(píng)論

luoyibu

男|高級(jí)講師

我要關(guān)注我要私信

TA的文章

ESP8266-NodeMCU項(xiàng)目（三）：ESP8266-NodeMCU+Blinker+紅外模塊（

閱讀 2725·2021-10-08 10:04
python-tkinter（7）實(shí)現(xiàn)各種個(gè)樣的撩妹鼠標(biāo)拖尾

閱讀 2764·2021-09-06 15:02
最全面的水平垂直居中方案與flexbox布局

閱讀 883·2019-08-30 13:50
前端開發(fā)篇——瀏覽器默認(rèn)樣式及css初始化

閱讀 1586·2019-08-30 13:21
CSS 屬性選擇器的深入挖掘

閱讀 2615·2019-08-30 11:15
nginx + 一個(gè)端口部署多個(gè)單頁(yè)應(yīng)用（history模式）

閱讀 2156·2019-08-29 17:19
react-router v4.x 源碼拾遺2

閱讀 1620·2019-08-26 13:55
『多圖警告』手撕排序算法 - 前端進(jìn)階必備

閱讀 1286·2019-08-26 10:15

成人国产在线小视频_日韩寡妇人妻调教在线播放_色成人www永久在线观看_2018国产精品久久_亚洲欧美高清在线30p_亚洲少妇综合一区_黄色在线播放国产_亚洲另类技巧小说校园_国产主播xx日韩_a级毛片在线免费

資訊專欄INFORMATION COLUMN

上云采購(gòu)季！| 2核2G4M爆款云服務(wù)器低至59元/年，更有多臺(tái)、長(zhǎng)期優(yōu)惠，快來選購(gòu)！

docker容器根目錄為只讀的解決辦法

相關(guān)文章

**【轉(zhuǎn)】Docker那些事（一）**

**并存共生or相愛相殺？容器、虛擬機(jī)與Docker概念全解析**

**使用docker部署PostgreSQL數(shù)據(jù)庫(kù)**

**使用docker部署PostgreSQL數(shù)據(jù)庫(kù)**

Docker 入門與實(shí)踐

發(fā)表評(píng)論

0條評(píng)論

luoyibu

男|高級(jí)講師

TA的文章

ESP8266-NodeMCU項(xiàng)目（三）：ESP8266-NodeMCU+Blinker+紅外模塊（

python-tkinter（7）實(shí)現(xiàn)各種個(gè)樣的撩妹鼠標(biāo)拖尾

最全面的水平垂直居中方案與flexbox布局

前端開發(fā)篇——瀏覽器默認(rèn)樣式及css初始化

CSS 屬性選擇器的深入挖掘

nginx + 一個(gè)端口部署多個(gè)單頁(yè)應(yīng)用（history模式）

react-router v4.x 源碼拾遺2

『多圖警告』手撕排序算法 - 前端進(jìn)階必備

最新活動(dòng)

資訊專欄INFORMATION COLUMN

上云采購(gòu)季！| 2核2G4M爆款云服務(wù)器低至59元/年，更有多臺(tái)、長(zhǎng)期優(yōu)惠，快來選購(gòu)！

docker容器根目錄為只讀的解決辦法

相關(guān)文章

發(fā)表評(píng)論

0條評(píng)論

男|高級(jí)講師

TA的文章

最新活動(dòng)

上云采購(gòu)季！| 2核2G4M爆款云服務(wù)器低至59元/年，更有多臺(tái)、長(zhǎng)期優(yōu)惠，快來選購(gòu)！