Confirm disk is broken when it passes all diagnostics

Posted by Halfgaar on Server Fault See other posts from Server Fault or by Halfgaar
Published on 2013-07-01T19:27:34Z Indexed on 2013/07/01 23:07 UTC
Read the original article Hit count: 449

Filed under:
|

I have a system with a potentially broken disk, but the disk passes all manner of diagnostics. I have been unable to confirm that the disk is broken. What are my options?

I could just replace the disk, but because this situation is very similar to another more severe situation I have (long story), I'd like to actually make a proper diagnosis as opposed to randomly binning hardware.

The issue and history is this:

  1. I had a Debian Linux PC (500 MHz P3) acting as router, nagios and munin.
  2. It crashed every couple of weeks. No logs or dmesg could be obtained (because it's an old Compaq that only boots when you configure it as keyboardless, making connecting a keyboard later, once it's booted, impossible).
  3. At the time, I just replaced the computer with another Compaq (P4 2.4 GHz) because I thought the hardware was faulty. However, it still crashed every couple of weeks.
  4. the difference is that on this computer, I can still SSH into it. It gives all kinds of errors on hda.

I'd like to confirm that the disk is broken, but nothing I do confirms this:

  1. SMART error logs shows no errors. Normally when a disk starts acting up, SMART my pass, but it still records a read-error in the error log.
  2. SMART self-test (smartctl -t long /dev/sda) completes without errors.
  3. re-allocated sector count (a tell-tale parameter) has been 31 all its life, even when the disk was still in use in my desktop PC years ago, and it still is. The figure never changed.
  4. dd if=/dev/sda of=/dev/null bs=4096 passes with flying colors.

What else can I do to assess the health of the drive?

Again, this is not about making this router fully functional again, this is a disk forensic question, because it just so happens that I have another server that potentially has the same problem, and knowing the answer to this will possibly help me greatly.

For the record, below are logs and such.

This is the smartctl -a output:

smartctl 5.40 2010-07-12 r3124 [i686-pc-linux-gnu] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda 7200.7 and 7200.7 Plus family
Device Model:     ST3120026A
Serial Number:    5JT1CLQM
Firmware Version: 3.06
User Capacity:    120,034,123,776 bytes
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   6
ATA Standard is:  ATA/ATAPI-6 T13 1410D revision 2
Local Time is:    Mon Jul  1 21:18:33 2013 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                    was completed without error.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      (  24) The self-test routine was aborted by
                    the host.
Total time to complete Offline 
data collection:         ( 430) seconds.
Offline data collection
capabilities:            (0x5b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    No Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                    No General Purpose Logging support.
Short self-test routine 
recommended polling time:    (   1) minutes.
Extended self-test routine
recommended polling time:    (  85) minutes.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   050   046   006    Pre-fail  Always       -       47766662
  3 Spin_Up_Time            0x0003   097   096   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       10
  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       31
  7 Seek_Error_Rate         0x000f   084   060   030    Pre-fail  Always       -       820305
  9 Power_On_Hours          0x0032   048   048   000    Old_age   Always       -       46373
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       605
194 Temperature_Celsius     0x0022   036   065   000    Old_age   Always       -       36
195 Hardware_ECC_Recovered  0x001a   050   046   000    Old_age   Always       -       47766662
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   196   000    Old_age   Always       -       6
200 Multi_Zone_Error_Rate   0x0000   100   253   000    Old_age   Offline      -       0
202 Data_Address_Mark_Errs  0x0032   100   253   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Aborted by host               80%     46361         -
# 2  Extended offline    Completed without error       00%     46358         -
# 3  Short offline       Completed without error       00%     12046         -
# 4  Extended offline    Completed without error       00%     10472         -
# 5  Short offline       Completed without error       00%     10471         -
# 6  Short offline       Completed without error       00%     10471         -
# 7  Short offline       Completed without error       00%      6770         -
# 8  Extended offline    Aborted by host               90%      5958         -
# 9  Extended offline    Aborted by host               90%      5951         -
#10  Short offline       Completed without error       00%      5024         -
#11  Extended offline    Aborted by host               80%      5024         -
#12  Short offline       Completed without error       00%      3697         -
#13  Short offline       Completed without error       00%       237         -
#14  Short offline       Completed without error       00%       145         -
#15  Short offline       Completed without error       00%        69         -
#16  Extended offline    Completed without error       00%        68         -
#17  Short offline       Completed without error       00%        66         -
#18  Short offline       Completed without error       00%        49         -
#19  Short offline       Completed without error       00%        29         -
#20  Short offline       Completed without error       00%        29         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

And this is the dmesg error when it has crashed (which repeats for a bunch of different sectors):

[1755091.211136] sd 0:0:0:0: [sda] Unhandled error code
[1755091.211144] sd 0:0:0:0: [sda] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
[1755091.211151] sd 0:0:0:0: [sda] CDB: Read(10): 28 00 08 fe ad 38 00 00 08 00
[1755091.211166] end_request: I/O error, dev sda, sector 150908216

© Server Fault or respective owner

Related posts about disk

Related posts about smart