System fails to start up correctly, with 2 dnsmasq/ujail processes
Username: KPapad
Origin: https://bugs.openwrt.org/index.php?do=details&task_id=1559
I have noticed frequent problems with OpenWrt trunk: about 30% of the time it doesn’t start up correctly (but it seems to boot OK and dmesg messages seem fine), ps shows two dnsmasq/ujail processes (note: I’ve been using ujail for the past 2+ years, without serious problems other than
FS#878
).
When OpenWrt gets stuck with 2 dnsmasq/ujail processes, invoking logread produces the error: “Failed to find log object: Not found” multiple times. If I kill the ujail/dnsmasq processes OpenWrt will just reboot (!)
I’ve tested this on two different OpenWrt VMs and on two different systems running VBox 5.1 and 5.2 under Ubuntu.
Here is what a ps looks like when the LEDE VM has booted normally:
root@10.0.3.141's password:
BusyBox v1.28.3 () built-in shell (ash)
| |.-----.-----.-----.| | | |.----.| |_ | - || _ | -__| || | | || _|| _| |_______|| __|_____|__|__||________||__| |____| |__| W I R E L E S S F R E E D O M
9c0ddafd
OpenWrt SNAPSHOT, r6965-root@OpenWrt:# ps
PID USER VSZ STAT COMMAND
1 root 13384 S /sbin/procd
2 root 0 SW [kthreadd]
3 root 0 IW [kworker/0:0]
4 root 0 IW< [kworker/0:0H]
5 root 0 IW [kworker/u2:0]
6 root 0 IW< [mm_percpu_wq]
7 root 0 SW [ksoftirqd/0]
8 root 0 IW [rcu_sched]
9 root 0 IW [rcu_bh]
10 root 0 SW [migration/0]
11 root 0 SW [cpuhp/0]
12 root 0 IW< [netns]
13 root 0 IW [kworker/u2:1]
17 root 0 IW [kworker/0:1]
95 root 0 SW [oom_reaper]
96 root 0 IW [kworker/u2:2]
232 root 0 IW< [writeback]
234 root 0 IW< [crypto]
235 root 0 IW< [kintegrityd]
237 root 0 IW< [kblockd]
304 root 0 IW< [ata_sff]
325 root 0 IW< [watchdogd]
379 root 0 IW [kworker/u2:3]
439 root 0 SW [kswapd0]
525 root 0 IW< [pencrypt]
527 root 0 IW< [pdecrypt]
554 root 0 IW< [acpi_thermal_pm]
623 root 0 SW [scsi_eh_0]
624 root 0 IW< [scsi_tmf_0]
627 root 0 SW [scsi_eh_1]
638 root 0 IW< [scsi_tmf_1]
641 root 0 SW [scsi_eh_2]
642 root 0 IW< [scsi_tmf_2]
645 root 0 IW< [ipv6_addrconf]
646 root 0 IW [kworker/u2:4]
724 root 0 IW< [kworker/0:1H]
732 root 0 IW< [ext4-rsv-conver]
818 root 0 IW< [ext4-rsv-conver]
973 root 9012 S /sbin/ubusd
978 root 2744 S /sbin/askfirst /usr/libexec/login.sh
979 root 3112 S /bin/ash --login
1273 root 0 IW< [cfg80211]
1512 root 7064 S /sbin/logd -S 64
1743 root 15564 S /sbin/netifd
1800 root 11260 S /usr/sbin/odhcpd
1860 root 2944 S /usr/sbin/dropbear -F -P /var/run/dropbear.1.pid -p 22 -K 300 -T 3
2149 root 3108 S udhcpc -p /var/run/udhcpc-eth1.pid -s /lib/netifd/dhcp.script -f -t 0 -i eth1 -x hostname:OpenWrt -C -O 121
2162 root 3108 S udhcpc -p /var/run/udhcpc-eth2.pid -s /lib/netifd/dhcp.script -f -t 0 -i eth2 -x hostname:OpenWrt -C -O 121
2169 root 3108 S udhcpc -p /var/run/udhcpc-vlan40.pid -s /lib/netifd/dhcp.script -f -t 0 -i vlan40 -x hostname:OpenWrt -C -O 121
2172 root 3108 S udhcpc -p /var/run/udhcpc-vlan50.pid -s /lib/netifd/dhcp.script -f -t 0 -i vlan50 -x hostname:OpenWrt -C -O 121
2687 root 10032 S {dnsmasq} /sbin/ujail -n dnsmasq -u -l -r /dev/null -r /dev/urandom -r /etc/TZ -r /etc/dnsmasq.conf -r /etc/ethers -r /etc/group -r /etc/hosts -r /etc/passwd -r /sbin/hotplug-call
2700 dnsmasq 13612 S /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pid
2705 root 0 IW [kworker/0:2]
2947 root 7092 S /usr/sbin/softflowd -d -i eth0 -m 8192 -n 10.0.3.100:2055 -p /var/run/softflowd.pid -c /var/run/softflowd.ctl -v 5 -T full -s 100 -p /var/run/softflowd.pid
2990 root 7908 S /usr/lib/ipsec/starter --daemon charon --nofork
3067 root 78176 S /usr/lib/ipsec/charon
3121 root 3112 S< /usr/sbin/ntpd -n -N -S /usr/sbin/ntpd-hotplug -p 1.gr.pool.ntp.org -p 0.europe.pool.ntp.org -p 2.europe.pool.ntp.org
3921 root 3256 S /bin/sh /usr/sbin/mwan3track wan eth1 online 10.0.3.141 8.8.4.4 208.67.222.222
4500 root 3256 S /bin/sh /usr/sbin/mwan3track wan2 eth2 online 10.0.3.70 8.8.8.8 208.67.220.220
4912 root 3060 S /usr/sbin/dropbear -F -P /var/run/dropbear.1.pid -p 22 -K 300 -T 3
4917 root 3112 S sleep 5
4922 root 3112 S sleep 5
4923 root 3112 S -ash
4931 root 3108 R ps
root@OpenWrt:# uname -a
Linux OpenWrt 4.14.41 #0 SMP Mon May 21 16:07:48 2018 x86_64 GNU/Linux
root@OpenWrt:# cat /etc/openwrt_*
DISTRIB_ID='OpenWrt'
DISTRIB_RELEASE='SNAPSHOT'
DISTRIB_REVISION='r6965-9c0ddafd'
DISTRIB_TARGET='x86/64'
DISTRIB_ARCH='x86_64'
DISTRIB_DESCRIPTION='OpenWrt SNAPSHOT r6965-9c0ddafd'
DISTRIB_TAINTS=''
r6965-9c0ddafd
root@OpenWrt:#
And here is what ps on the same system (same VM, same /etc/config etc) looks like when it fails to work correctly:
PID USER VSZ STAT COMMAND 1 root 13384 S /sbin/procd 2 root 0 SW [kthreadd] 3 root 0 IW [kworker/0:0] 4 root 0 IW< [kworker/0:0H] 5 root 0 IW [kworker/u2:0] 6 root 0 IW< [mm_percpu_wq] 7 root 0 SW [ksoftirqd/0] 8 root 0 IW [rcu_sched] 9 root 0 IW [rcu_bh] 10 root 0 SW [migration/0] 11 root 0 SW [cpuhp/0] 12 root 0 IW< [netns] 13 root 0 IW [kworker/u2:1] 29 root 0 IW [kworker/0:1] 75 root 0 SW [oom_reaper] 76 root 0 IW [kworker/u2:2] 232 root 0 IW< [writeback] 233 root 0 IW< [crypto] 235 root 0 IW< [kintegrityd] 236 root 0 IW< [kblockd] 238 root 0 IW< [ata_sff] 325 root 0 IW< [watchdogd] 438 root 0 SW [kswapd0] 524 root 0 IW< [pencrypt] 526 root 0 IW< [pdecrypt] 545 root 0 IW< [acpi_thermal_pm] 622 root 0 SW [scsi_eh_0] 623 root 0 IW< [scsi_tmf_0] 626 root 0 SW [scsi_eh_1] 637 root 0 IW< [scsi_tmf_1] 640 root 0 SW [scsi_eh_2] 641 root 0 IW< [scsi_tmf_2] 644 root 0 IW< [ipv6_addrconf] 645 root 0 IW [kworker/u2:3] 646 root 0 IW [kworker/u2:4] 728 root 0 IW< [kworker/0:1H] 732 root 0 IW< [ext4-rsv-conver] 818 root 0 IW< [ext4-rsv-conver] 973 root 9016 S /sbin/ubusd 978 root 2744 S /sbin/askfirst /usr/libexec/login.sh 979 root 3112 S /bin/ash --login 1269 root 0 IW< [cfg80211] 1801 root 11328 S /usr/sbin/odhcpd 2697 root 10032 S {dnsmasq} /sbin/ujail -n dnsmasq -u -l -r /dev/null 2705 root 10032 R {dnsmasq} /sbin/ujail -n dnsmasq -u -l -r /dev/null 2759 root 3108 R ps