Welcome to Centmin Mod Community
Register Now

Sysadmin Empty logs (messages, secure etc)

Discussion in 'System Administration' started by Meirami, Jul 16, 2018.

  1. Meirami

    Meirami Member

    93
    8
    8
    Dec 21, 2017
    Ratings:
    +29
    Local Time:
    1:56 AM
    Few days ago my vps got frozen at night. CPU usage went really high. Logrotate worked about the same time when cpu% rose. There's now empty log files. Like
    Code:
    0 Jul 15 03:18 messages
    Saw this yesterday and last time stamp is also from yesterday. So I'm not sure if this is because the freeze. I also tryed to clean some leftovers from appoptics on 14th of July. Like yum clean all and rm - f the directory it suggests.

    Journalctl is broken. Fixed it. Journalctl --verify passed. But after reboot.
    Code:
    # journalctl --verify
    3c7938: Entry timestamp out of synchronization
    File corruption detected at /var/log/journal/78a00fc288b042228ee5e603
    f80c004d/system.journal:3c7938 (of 8388608 bytes, 47%).
    FAIL: /var/log/journal/78a00fc288b042228ee5e603f80c004d/system.journa
    l (Bad message)
    Can this be related to empty logs?

    What to try next? How to fix?
     
  2. eva2000

    eva2000 Administrator Staff Member

    35,522
    7,833
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,074
    Local Time:
    8:56 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    rm command can be dangerous if you ran it on wrong directory - review the output for your command history via command
    Code (Text):
    history
    

    filter grep on rm command
    Code (Text):
    history | grep rm
    

    see if you mistakenly removed directories you shouldn't have

    maybe try this guide though your mileage may vary Repairing CENTOS 7 Journal Corruption
     
  3. Meirami

    Meirami Member

    93
    8
    8
    Dec 21, 2017
    Ratings:
    +29
    Local Time:
    1:56 AM
    Command was ok.
    rm -rf /var/cache/yum (forgot option r from my post :) )

    Forgot to mention. Yesterday when I couldn't fix this, I renamed all size 0 log files to logfilename.back like messages.back to see if logrotate can create new file. It didn't. Today I renamed those back to normal.

    Will continue with that repairing guide. Hopefully the answer is there.

    Thank You!
     
  4. Meirami

    Meirami Member

    93
    8
    8
    Dec 21, 2017
    Ratings:
    +29
    Local Time:
    1:56 AM
    Actually that was pretty much same like I have tryed.
    I couldn't fix journal so I deleted it. Deleting needs that systemd-journald and systemd-journald.socket are stopped. If I start them manually before reboot command, journal will be corrupted after reboot. I rebooted without starting those and journal is ok after reboot. Not going to reboot again now... It collects data nicely. Logfiles are still empty.
     
    • Informative Informative x 1
  5. eva2000

    eva2000 Administrator Staff Member

    35,522
    7,833
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,074
    Local Time:
    8:56 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
  6. Meirami

    Meirami Member

    93
    8
    8
    Dec 21, 2017
    Ratings:
    +29
    Local Time:
    1:56 AM
    At least I found a workaround.

    rsyslog not logging

    There's a bug, but it should be fixed...
    Bug 1088021 – Changing a VM host's time disables rsyslog file logging

    More about similar problem.
    Logging not working on Centos 7

    Just don't know, how long the workaround works...
     
    • Informative Informative x 1
  7. Meirami

    Meirami Member

    93
    8
    8
    Dec 21, 2017
    Ratings:
    +29
    Local Time:
    1:56 AM
    Update

    This is how I can make journalctl working.
    If you want to save parts of old journal check how journalctl --vacuum-time=7d works.
    Code:
    cd /var/log/journal/78a00fc288b042228ee5e603f80c004d/  <-- locate your own correct dir
    systemctl stop systemd-journald
    systemctl stop systemd-journald.socket
    rm system*
    rm /var/lib/rsyslog/imjournal.state
    systemctl start systemd-journald
    journalctl
    journalctl --verify
    

    After journalctl was working, to make /var/log/messages working needed:
    Code:
    logrotate -f /etc/logrotate.conf
    rm /var/lib/rsyslog/imjournal.state
    systemctl restart rsyslog
    
    (This is propably not the best way, but at least it didn't kill the server)
     
    Last edited: Aug 8, 2018
    • Informative Informative x 1
  8. eva2000

    eva2000 Administrator Staff Member

    35,522
    7,833
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,074
    Local Time:
    8:56 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Might want to fix the cd path
     
..