Sysadmin Empty logs (messages, secure etc)

Few days ago my vps got frozen at night. CPU usage went really high. Logrotate worked about the same time when cpu% rose. There's now empty log files. Like

Code:

0 Jul 15 03:18 messages

Saw this yesterday and last time stamp is also from yesterday. So I'm not sure if this is because the freeze. I also tryed to clean some leftovers from appoptics on 14th of July. Like yum clean all and rm - f the directory it suggests.

Journalctl is broken. Fixed it. Journalctl --verify passed. But after reboot.

Code:

# journalctl --verify
3c7938: Entry timestamp out of synchronization
File corruption detected at /var/log/journal/78a00fc288b042228ee5e603
f80c004d/system.journal:3c7938 (of 8388608 bytes, 47%).
FAIL: /var/log/journal/78a00fc288b042228ee5e603f80c004d/system.journa
l (Bad message)

Can this be related to empty logs?

What to try next? How to fix?

---

 

Command was ok.
rm -rf /var/cache/yum (forgot option r from my post )

Forgot to mention. Yesterday when I couldn't fix this, I renamed all size 0 log files to logfilename.back like messages.back to see if logrotate can create new file. It didn't. Today I renamed those back to normal.

Will continue with that repairing guide. Hopefully the answer is there.

Thank You!

 

Actually that was pretty much same like I have tryed.
I couldn't fix journal so I deleted it. Deleting needs that systemd-journald and systemd-journald.socket are stopped. If I start them manually before reboot command, journal will be corrupted after reboot. I rebooted without starting those and journal is ok after reboot. Not going to reboot again now... It collects data nicely. Logfiles are still empty.

 

At least I found a workaround.

rsyslog not logging

There's a bug, but it should be fixed...
Bug 1088021 – Changing a VM host's time disables rsyslog file logging

More about similar problem.
Logging not working on Centos 7

Just don't know, how long the workaround works...

 

Update

This is how I can make journalctl working.
If you want to save parts of old journal check how journalctl --vacuum-time=7d works.

Code:

cd /var/log/journal/78a00fc288b042228ee5e603f80c004d/  <-- locate your own correct dir
systemctl stop systemd-journald
systemctl stop systemd-journald.socket
rm system*
rm /var/lib/rsyslog/imjournal.state
systemctl start systemd-journald
journalctl
journalctl --verify

After journalctl was working, to make /var/log/messages working needed:

Code:

logrotate -f /etc/logrotate.conf
rm /var/lib/rsyslog/imjournal.state
systemctl restart rsyslog

(This is propably not the best way, but at least it didn't kill the server)