Linux硬盘盘符分配原则
在Linux系统中,若存在多块硬盘,内核分配盘符的顺序是/dev/sda、/dev/sdb、/dev/sdc … …。在系统启动过程中,内核会按照扫描到硬盘的顺序分配盘符。
内核中分配盘符函数,见链接http://ilinuxkernel.com/?p=794
Linux内核通过IDR(integer ID)机制来分配盘符,即找到一个空闲的整数。内核针对SCSI盘符,从0开始分配整数。整数0对应的盘符为/dev/sda,关系如下:
若index=0,则分配给此块SCSI硬盘的盘符为sda;
若index=1,则分配给此块SCSI硬盘的盘符为sdb;
… …
若index=25,则分配给此块SCSI硬盘的盘符为sdz;
如在有12硬盘的服务器中,每个物理槽位均插一块硬盘,且没有做RAID,硬盘物理槽位关系如下:
0 | 3 | 6 | 9 |
1 | 4 | 7 | 10 |
2 | 5 | 8 | 11 |
那么Linux内核启动后,IDR机制分配的盘符的整数为0、1、2、3、4、5、6、7、8、9、10、11,对应的硬盘盘符如下:
系统运行过程中,若拔掉第5块盘,如下图所示:
则此时系统中,/dev/sde盘符消失,内核针对SCSI盘符的IDR整数4就空闲出来。当再次将该硬盘插入时,得到的盘符仍然是/dev/sde。
非热插拔硬盘盘符分配示例
如下图,第5个硬盘物理槽位没有硬盘。系统重启后,盘符对应关系如下。系统中没有盘符/dev/sdl,缺少不是最后一块硬盘,而是第5块硬盘。
下图是第5块和第9块物理槽位硬盘不在位时,OS盘符对应关系。
热插拔硬盘盘符分配示例
下图是第1、5、9块硬盘不在位时,OS盘符对应关系,此时没有热插拔硬盘。
系统运行过程中,先热插拔一块到第1个物理槽位,此时得到的盘符为/dev/sdj,然后插入一块硬盘到第9个物理槽位,此时分配的盘符为/dev/sdk。
若插入2块硬盘后,机器重启,则盘符对应关系如下。从这里我们可以看出,硬盘盘符和物理槽位没有必然直接对应关系。
硬盘盘符为什么会漂移
服务器只有12块物理硬盘,但在系统运行过程中或更换硬盘时,会出现/dev/sdm、/dev/sdn等类似盘符。
下面示例,系统运行过程中,我们把第5块硬盘拔出再插入,此时得到的硬盘盘符可能为/dev/sdm。
当我们拔出硬盘后,内核会调用sd_remove()函数卸载硬盘,正常情况下会清除该硬盘盘符占用的所有资源,包括SCSI盘符对应的IDR整数。但我们拔出硬盘时,若系统有进程正在访问该硬盘,则内核不会删除对应IDR的整数,该整数就会被占用,再次插入硬盘时,就分配新的IDR整数给盘符,导致盘符漂移。
如上面第5块硬盘,内核分配的IDR值为4,但硬盘拔出再插入后,应为IDR值为4没被释放,内核找到空闲的IDR就会12,此时盘符就变为/dev/sdm。盘符变为/dev/sdn等时,以此类推。
看了你的文章,解开了我的迷惑。我现在使用sas9200-8e,也遇到了系统盘符发生变化的问题。我的系统盘为单独一个硬盘没有做raid,能否修改mpt2sas驱动的扫描过程解决此问题。还有没有别的办法?
可以修改mpt2sas驱动解决盘符问题。
你好!很高兴你能回复我的提问。
我们的问题是这样的:我们主板上板载的SAS卡为sas2008控制芯片。我们又在主板pcie插槽上扩展出了一个SAS卡,型号为LSISAS9200-8E,其控制芯片也是sas2008,所以两者的驱动一样都为mpt2sas。所以在系统启动的时候出现了问题,启动时会先扫描LSISAS9200-8E,导致系统从LSISAS9200-8E所连接的硬盘引导,而不是从板载的sas卡连接的本地硬盘引导,操作系统无法启动。
请问下,怎样能解决掉这个问题。(mpt2sas同mptsas的扫描拓扑结构不一样。)
可以调整mpt2sas驱动中,对多个控制器扫描的顺序。你可以先分析代码和修改,若有问题,我可以再找时间帮你搞定。
你好!很高兴收到你的回复。
我通过抓打印分析,当加载mpt2sas驱动,注册到pci总线后,pci会遍历驱动所在的总线上的所有设备,然后将这些设备与当前驱动进行匹配,将设备与驱动联系起来。我的问题是pci遍历总线上设备的时候,两个设备的遍历顺序前后不同。mpt2sas驱动我已经看,能否加我MSN ch.zhouyh@gmail.com 再向你请教讨论。
可以直接发邮件交流,我很少打开聊天工具。chenyiquan AT huawei ! com
这个应该通过BIOS启动引导顺序就可以解决吧。
是的。这个问题应该是通过改变BIOS扫描PCI-E设备顺序来解决。同一个驱动扫描硬盘顺序,应该是通过内核找到的PCI设备先后顺序。
请问有具体的操作步骤么?
你好,我有一套双机,主机上使用了parted分区,sdc分为sdc1,sdc2,而备机上面仍然只看见sdc,无法看到sdc1,但是在备机上用 parted print也确实能看见两个分区,就是盘符不出现
执行一下#partprobe命令试一下。
博主,你好!
现在我环境中用了两块ssd做了软raid,希望能够拔一块盘后插入一块新盘,能够做到数据恢复!
遇到的问题是磁盘拔插会导致盘符乱序,比如拔出的是sda,插入的是sdo了,请教博主有无办法保证盘符不变更?或者说在磁盘拔掉的时候释放所有该盘符的占用。
当两块ssd做的软raid不是系统盘时,可以通过停止阵列,再拔插,保证盘符不会变更,我猜测是软raid里面会存放盘符信息,导致磁盘拔了仍然占用盘符
可以在SAS控制器驱动里,实现盘古绑定,根据SSD盘物理槽位ID固定分配盘符。
这个需要厂家的固件配合吧?LSI的mptsas卡固件对些设计不对,2个卡后,8个线是:1234 8765
我在帝联时,实现的就是在mptsas根据硬件上报的槽位对应的id num做的,不过,只在华为上验证了,mptsas,mpt2sas/mpt3sas没机器,并且不用绑定厂商(2015年做的),从你这里可学到不少东西啊,兄弟,有机会想请你吃个饭
华为服务器
能留言吗
你不在华为了?
离开快3年了。
你好,chen,我是两个SAS转卡扩展带16个硬盘,怎么能将盘符与port固定
驱动中控制硬件扫描顺序,然后对应Port固定即可。
对驱动不是很了解,能不能再详细说一下,谢谢谢谢
你好,向您请教,
只挂载了一个硬盘,系统运行中,向硬盘里写数据,运行几天之后会发生盘符变化,从sda1变成sdb1.中间没有插拔过。
应该是有闪断。你检查内核日志信息,应该有相关记录。
我查了下系统,没有找到和硬盘等相关的日志记录呢。
闪断一般和什么有关系,电源供电不足?硬盘本身有问题?
是系统没有安装syslog日志,我安装上syslog日志,现在可以看到syslog和messages,要等再次复现才能看到有没有对应的日志信息。
拿到了这些日志,能帮忙看看吗?给出点建议?谢谢!
Mar 31 01:50:17 diveo kernel: [297193.873432] alloc_contig_range: [10480, 10498) PFNs busy
Mar 31 01:56:02 diveo kernel: [297539.123490] ata1: SATA link down (SStatus 0 SControl 300)
Mar 31 01:56:07 diveo kernel: [297544.638513] ata1: SATA link down (SStatus 0 SControl 300)
Mar 31 01:56:13 diveo kernel: [297549.977455] ata1: SATA link down (SStatus 0 SControl 300)
Mar 31 01:56:13 diveo kernel: [297549.977468] ata1.00: disabled
Mar 31 01:56:13 diveo kernel: [297549.977504] sd 0:0:0:0: rejecting I/O to offline device
Mar 31 01:56:13 diveo kernel: [297549.977533] blk_update_request: I/O error, dev sda, sector 1880362536 op 0x1:(WRITE) flags 0x0 phys_seg 2 prio class 0
Mar 31 01:56:13 diveo kernel: [297549.977545] EXT4-fs warning (device sda1): ext4_end_bio:345: I/O error 10 writing to inode 34078779 starting block 235045318)
Mar 31 01:56:13 diveo kernel: [297549.977554] Buffer I/O error on device sda1, logical block 235045061
Mar 31 01:56:13 diveo kernel: [297549.977574] EXT4-fs warning (device sda1): ext4_end_bio:345: I/O error 10 writing to inode 34078779 starting block 235045319)
Mar 31 01:56:13 diveo kernel: [297549.977601] blk_update_request: I/O error, dev sda, sector 1122826416 op 0x1:(WRITE) flags 0x0 phys_seg 37 prio class 0
Mar 31 01:56:13 diveo kernel: [297549.977611] EXT4-fs warning (device sda1): ext4_end_bio:345: I/O error 10 writing to inode 34754548 starting block 140353339)
Mar 31 01:56:13 diveo kernel: [297549.977630] blk_update_request: I/O error, dev sda, sector 1951562152 op 0x1:(WRITE) flags 0x800 phys_seg 16 prio class 0
Mar 31 01:56:13 diveo kernel: [297549.977700] ata1.00: detaching (SCSI 0:0:0:0)
Mar 31 01:56:13 diveo kernel: [297549.977720] Aborting journal on device sda1-8.
Mar 31 01:56:13 diveo kernel: [297549.977752] EXT4-fs error (device sda1) in ext4_reserve_inode_write:5783: Journal has aborted
Mar 31 01:56:13 diveo kernel: [297549.977771] blk_update_request: I/O error, dev sda, sector 1950615552 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
Mar 31 01:56:13 diveo kernel: [297549.977782] blk_update_request: I/O error, dev sda, sector 1950615552 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
Mar 31 01:56:13 diveo kernel: [297549.977790] Buffer I/O error on dev sda1, logical block 243826688, lost sync page write
Mar 31 01:56:13 diveo kernel: [297549.977804] blk_update_request: I/O error, dev sda, sector 2048 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
Mar 31 01:56:13 diveo kernel: [297549.977821] EXT4-fs warning (device sda1): ext4_convert_unwritten_extents:4769: inode #34078779: block 1222: len 1: ext4_ext_map_blocks returned -30
Mar 31 01:56:13 diveo kernel: [297549.977832] blk_update_request: I/O error, dev sda, sector 2048 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
Mar 31 01:56:13 diveo kernel: [297549.977840] EXT4-fs error (device sda1): ext4_check_bdev_write_error:215: comm kworker/u16:0: Error while async write back metadata
Mar 31 01:56:13 diveo kernel: [297549.977850] Buffer I/O error on dev sda1, logical block 0, lost sync page write
Mar 31 01:56:13 diveo kernel: [297549.977864] EXT4-fs (sda1): I/O error while writing superblock
Mar 31 01:56:13 diveo kernel: [297549.977868] EXT4-fs error (device sda1): mpage_map_and_submit_extent:2523: inode #34754549: comm kworker/u16:2: mark_inode_dirty error
Mar 31 01:56:13 diveo kernel: [297549.977871] JBD2: Error -5 detected when updating journal superblock for sda1-8.
Mar 31 01:56:13 diveo kernel: [297549.977884] blk_update_request: I/O error, dev sda, sector 2048 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
Mar 31 01:56:13 diveo kernel: [297549.977888] blk_update_request: I/O error, dev sda, sector 2048 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
Mar 31 01:56:13 diveo kernel: [297549.977891] Buffer I/O error on dev sda1, logical block 0, lost sync page write
Mar 31 01:56:13 diveo kernel: [297549.977901] EXT4-fs (sda1): I/O error while writing superblock
Mar 31 01:56:13 diveo kernel: [297549.977905] EXT4-fs error (device sda1): mpage_map_and_submit_extent:2525: comm kworker/u16:2: Failed to mark inode 34754549 dirty
Mar 31 01:56:13 diveo kernel: [297549.977912] blk_update_request: I/O error, dev sda, sector 2048 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
Mar 31 01:56:13 diveo kernel: [297549.977916] Buffer I/O error on dev sda1, logical block 0, lost sync page write
Mar 31 01:56:13 diveo kernel: [297549.977921] EXT4-fs (sda1): I/O error while writing superblock
Mar 31 01:56:13 diveo kernel: [297549.977924] EXT4-fs error (device sda1) in ext4_writepages:2827: Journal has aborted
Mar 31 01:56:13 diveo kernel: [297549.977931] Buffer I/O error on dev sda1, logical block 0, lost sync page write
Mar 31 01:56:13 diveo kernel: [297549.977936] EXT4-fs (sda1): I/O error while writing superblock
Mar 31 01:56:13 diveo kernel: [297549.977963] EXT4-fs warning (device sda1): ext4_end_bio:345: I/O error 10 writing to inode 34754549 starting block 140353420)
Mar 31 01:56:13 diveo kernel: [297549.977997] Buffer I/O error on dev sda1, logical block 0, lost sync page write
Mar 31 01:56:13 diveo kernel: [297549.978033] EXT4-fs (sda1): previous I/O error to superblock detected
Mar 31 01:56:13 diveo kernel: [297549.978042] Buffer I/O error on dev sda1, logical block 0, lost sync page write
Mar 31 01:56:13 diveo kernel: [297549.978047] EXT4-fs error (device sda1): ext4_journal_check_start:83: Detected aborted journal
Mar 31 01:56:13 diveo kernel: [297549.978052] EXT4-fs (sda1): I/O error while writing superblock
Mar 31 01:56:13 diveo kernel: [297549.978062] EXT4-fs error (device sda1) in ext4_reserve_inode_write:5783: Journal has aborted
Mar 31 01:56:13 diveo kernel: [297549.978069] EXT4-fs (sda1): Remounting filesystem read-only
Mar 31 01:56:13 diveo kernel: [297549.978074] EXT4-fs (sda1): ext4_writepages: jbd2_start: 1024 pages, ino 34216876; err -30
Mar 31 01:56:13 diveo kernel: [297549.978102] Buffer I/O error on dev sda1, logical block 0, lost sync page write
Mar 31 01:56:13 diveo kernel: [297549.978125] EXT4-fs (sda1): I/O error while writing superblock
Mar 31 01:56:13 diveo kernel: [297549.978142] EXT4-fs error (device sda1): ext4_convert_unwritten_extents:4774: inode #34078779: comm kworker/u16:0: mark_inode_dirty error
Mar 31 01:56:13 diveo kernel: [297549.978176] Buffer I/O error on dev sda1, logical block 0, lost sync page write
Mar 31 01:56:13 diveo kernel: [297549.978198] EXT4-fs (sda1): I/O error while writing superblock
Mar 31 01:56:13 diveo kernel: [297549.978231] Buffer I/O error on device sda1, logical block 235045062
Mar 31 01:56:13 diveo kernel: [297549.978258] Buffer I/O error on device sda1, logical block 140353046
Mar 31 01:56:13 diveo kernel: [297549.978272] Buffer I/O error on device sda1, logical block 140353047
Mar 31 01:56:13 diveo kernel: [297549.978284] Buffer I/O error on device sda1, logical block 140353048
Mar 31 01:56:13 diveo kernel: [297549.978295] Buffer I/O error on device sda1, logical block 140353049
Mar 31 01:56:13 diveo kernel: [297549.978307] Buffer I/O error on device sda1, logical block 140353050
Mar 31 01:56:13 diveo kernel: [297549.978319] Buffer I/O error on device sda1, logical block 140353051
Mar 31 01:56:13 diveo kernel: [297549.978331] Buffer I/O error on device sda1, logical block 140353052
Mar 31 01:56:13 diveo kernel: [297549.978342] Buffer I/O error on device sda1, logical block 140353053
Mar 31 01:56:13 diveo kernel: [297549.978704] Buffer I/O error on dev sda1, logical block 69, lost async page write
Mar 31 01:56:14 diveo kernel: [297551.565951] JBD2: Error while async write back metadata bh 69.
Mar 31 01:56:14 diveo kernel: [297551.566993] JBD2: Error while async write back metadata bh 136839642.
Mar 31 01:56:14 diveo kernel: [297551.567117] JBD2: Error while async write back metadata bh 138936338.
Mar 31 01:56:14 diveo kernel: [297551.567165] JBD2: Error while async write back metadata bh 138936632.
Mar 31 01:56:14 diveo kernel: [297551.567982] JBD2: Error while async write back metadata bh 138944725.
Mar 31 01:56:14 diveo kernel: [297551.567999] JBD2: Error while async write back metadata bh 138944727.
Mar 31 01:56:14 diveo kernel: [297551.572775] JBD2: Error while async write back metadata bh 143654928.
Mar 31 01:56:14 diveo kernel: [297551.572798] JBD2: Error while async write back metadata bh 143654944.
Mar 31 01:56:15 diveo kernel: [297551.572865] JBD2: Error while async write back metadata bh 143663148.
Mar 31 01:56:15 diveo kernel: [297551.762939] sd 0:0:0:0: [sda] Synchronizing SCSI cache
Mar 31 01:56:15 diveo kernel: [297551.763037] sd 0:0:0:0: [sda] Synchronize Cache(10) failed: Result: hostbyte=0x04 driverbyte=0x00
Mar 31 01:56:15 diveo kernel: [297551.763043] sd 0:0:0:0: [sda] Stopping disk
Mar 31 01:56:15 diveo kernel: [297551.763062] sd 0:0:0:0: [sda] Start/Stop Unit failed: Result: hostbyte=0x04 driverbyte=0x00
Mar 31 01:56:19 diveo kernel: [297555.667669] EXT4-fs error (device sda1): __ext4_find_entry:1609: inode #34078723: comm java: reading directory lblock 0
Mar 31 01:56:19 diveo kernel: [297555.671601] EXT4-fs warning (device sda1): dx_probe:768: inode #34078725: lblock 0: comm java: error -5 reading directory block
Mar 31 01:56:22 diveo kernel: [297559.167298] EXT4-fs error (device sda1): __ext4_find_entry:1609: inode #34078723: comm java: reading directory lblock 0
掉盘了。可能是盘本身故障了,可能是SATA链路不稳定。
你好,请教一个问题,文章开头说的按照扫描到硬盘的顺序分配盘符,这个扫描顺序是怎么扫描的呢
扫描顺序,是硬件初始化时发现设备的顺序。和SAS/RAID控制器,或者BIOS初始化NVMe等设备的顺序。
测试是否能评论
可以评论。