记录一些常用的服务器崩坏处理

记录一些常用的服务器崩坏处理

应用级的崩溃

gitlab起不来，redis无法up

rm /var/opt/gitlab/redis/dump.rdb

mongodb起不来，报错WiredTiger权限被拒绝

删除旧锁
rm -rf /var/lib/mongodb/journal
rm -rf /var/lib/mongodb/mongod.lock
rm -rf /var/lib/mongodb/WiredTiger.lock
修复数据库并生成新锁
mongod -f /etc/mongod.conf –repair
修改权限，注意是mongodb不是mongod
chown -R mongodb:mongodb *

gitlab起不来，postgresql无法up

2024-11-05_00:30:50.91275 FATAL:  could not create lock file "postmaster.pid": Read-only file system

检查文件系统，是不是盘符已经不是rw而是ro了，然后请按照系统级崩溃修复

2024-11-05_02:01:58.66054 LOG:  starting PostgreSQL 14.11 on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
2024-11-05_02:01:58.66154 LOG:  listening on Unix socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"
2024-11-05_02:01:58.69397 LOG:  database system was interrupted; last known up at 2024-11-01 07:56:44 GMT
2024-11-05_02:01:58.87096 LOG:  unexpected pageaddr 1/5557E000 in log segment 000000010000000100000061, offset 5758976
2024-11-05_02:01:58.87101 LOG:  invalid primary checkpoint record
2024-11-05_02:01:58.87103 PANIC:  could not locate a valid checkpoint record
2024-11-05_02:01:58.87359 LOG:  startup process (PID 3633) was terminated by signal 6: Aborted
2024-11-05_02:01:58.87364 LOG:  aborting startup due to startup process failure
2024-11-05_02:01:58.88879 LOG:  database system is shut down

su gitlab-psql
/opt/gitlab/embedded/postgresql/14/bin/pg_resetwal /var/opt/gitlab/postgresql/data

数据库级的崩溃

gitlab中的repo丢了【有备份】

请注意，一定要保证备份可用，且版本号一致
gitlab-rake gitlab:backup:restore BACKUP=1546916920_2019_01_08_10.5.1

mongo中的表丢了【有备份】

恢复mongo吧，mongorestore -u root -p

mysql起不来了【无备份】

配置/etc/mysql/mysql.conf.d/mysqld.cnf
[mysqld]
innodb_force_recovery =1
1、2、3可以逐个尝试,然后导出，然后初始化，然后导入

系统级的崩溃

CentOS系统进入紧急模式，文件系统损坏

xfs_repair -L /dev/mapper/centos-root

Debian系统进入紧急模式，文件系统损坏

fsck /dev/sdb2

ubuntu网络ip找不到了

修改netplan下的配置文件，netplan apply好几次，还是不行
最后我终于想起来我在ESXi的管理界面里改过网络适配器绑定的网络。
虚拟机关机，改好网络适配器对应的网络，重新连接，一切正常

虚拟机级的崩溃

重做日志失败，虚拟机无法启动

这个问题我放弃了，把机器返回上一个快照节点了，
网上说的一些方案都不好使，以后有机会还是换一个好用的虚拟化平台吧