SQL Query Stuck in Statistics State

Question

I've been troubleshooting this complex query today, thinking it was a performance issue. It appears that the query is getting stuck into the 'Statistics' state. It is currently in that state for 1300 seconds.

I've checked the indexes for the tables involved -- I've optimized the table -- what could be causing this hang?

SELECT
    Import_Values.id,
    Import_Values.part_id,
    Import_Values.qty,
    Import_Values.note,
    Parts.partterminologyname,
    GROUP_CONCAT(BaseVehicle.YearID, ' ', Make.MakeName, ' ', Model.modelname, ' ', SubModel.SubModelName SEPARATOR ', '),
    GROUP_CONCAT(DISTINCT(EngineDesignation.EngineDesignationName) SEPARATOR ', '),
    GROUP_CONCAT(DISTINCT(EngineVIN.EngineVINName) SEPARATOR ', '),
    GROUP_CONCAT(DISTINCT(EngineBase.Liter) SEPARATOR ', '),
    GROUP_CONCAT(DISTINCT(EngineBase.CC) SEPARATOR ', '),
    GROUP_CONCAT(DISTINCT(EngineBase.CID) SEPARATOR ', '),
    GROUP_CONCAT(DISTINCT(EngineBase.Cylinders) SEPARATOR ', '),
    GROUP_CONCAT(DISTINCT(EngineBase.BlockType) SEPARATOR ', '),
    GROUP_CONCAT(DISTINCT(EngineBase.EngBoreIn) SEPARATOR ', '),
    GROUP_CONCAT(DISTINCT(EngineBase.EngBoreMetric) SEPARATOR ', '),
    GROUP_CONCAT(DISTINCT(EngineBase.EngStrokeIn) SEPARATOR ', '),
    GROUP_CONCAT(DISTINCT(EngineBase.EngStrokeMetric) SEPARATOR ', '),
    GROUP_CONCAT(DISTINCT(FuelDeliveryType.FuelDeliveryTypeName) SEPARATOR ', '),
    GROUP_CONCAT(DISTINCT(FuelDeliverySubType.FuelDeliverySubTypeName) SEPARATOR ', '),
    GROUP_CONCAT(DISTINCT(FuelSystemControlType.FuelSystemControlTypeName) SEPARATOR ', '),
    GROUP_CONCAT(DISTINCT(FuelSystemDesign.FuelSystemDesignName) SEPARATOR ', '),
    GROUP_CONCAT(DISTINCT(Aspiration.AspirationName) SEPARATOR ', '),
    GROUP_CONCAT(DISTINCT(CylinderHeadType.CylinderHeadTypeName) SEPARATOR ', '),
    GROUP_CONCAT(DISTINCT(FuelType.FuelTypeName) SEPARATOR ', '),
    GROUP_CONCAT(DISTINCT(IgnitionSystemType.IgnitionSystemTypeName) SEPARATOR ', '),
    GROUP_CONCAT(DISTINCT(Mfr.MfrName) SEPARATOR ', '),
    GROUP_CONCAT(DISTINCT(EngineVersion.EngineVersion) SEPARATOR ', '),
    GROUP_CONCAT(DISTINCT(Valves.ValvesPerEngine) SEPARATOR ', '),
    GROUP_CONCAT(DISTINCT(BedLength.BedLength) SEPARATOR ', '),
    GROUP_CONCAT(DISTINCT(BedLength.BedLengthMetric) SEPARATOR ', ')
    FROM 
    Import_Values
    INNER JOIN BaseVehicle 
        ON Import_Values.base_vehicle_id=BaseVehicle.BaseVehicleID
    INNER JOIN Parts 
        ON Import_Values.part_type_id=Parts.PartTerminologyID
    INNER JOIN Make
        ON BaseVehicle.MakeID=Make.MakeID
    INNER JOIN Model
        ON BaseVehicle.ModelID=Model.ModelID
    INNER JOIN Vehicle
        ON Import_Values.base_vehicle_id=Vehicle.BaseVehicleID
    INNER JOIN SubModel
        ON Vehicle.SubModelID=SubModel.SubModelID
    INNER JOIN VehicleToEngineConfig
        ON Vehicle.VehicleID=VehicleToEngineConfig.VehicleID
    INNER JOIN EngineConfig
        ON VehicleToEngineConfig.EngineConfigID=EngineConfig.EngineConfigID
    INNER JOIN EngineDesignation
        ON EngineConfig.EngineDesignationID=EngineDesignation.EngineDesignationID
    INNER JOIN EngineVIN
        ON EngineConfig.EngineVINID=EngineVIN.EngineVINID
    INNER JOIN EngineBase
        ON EngineConfig.EngineBaseID=EngineBase.EngineBaseID
    INNER JOIN FuelDeliveryConfig
        ON EngineConfig.FuelDeliveryConfigID=FuelDeliveryConfig.FuelDeliveryConfigID
    INNER JOIN FuelDeliveryType
        ON FuelDeliveryConfig.FuelDeliveryTypeID=FuelDeliveryType.FuelDeliveryTypeID
    INNER JOIN FuelDeliverySubType
        ON FuelDeliveryConfig.FuelDeliverySubTypeID=FuelDeliverySubType.FuelDeliverySubTypeID
    INNER JOIN FuelSystemControlType
        ON FuelDeliveryConfig.FuelSystemControlTypeID=FuelSystemControlType.FuelSystemControlTypeID
    INNER JOIN FuelSystemDesign
        ON FuelDeliveryConfig.FuelSystemDesignID=FuelSystemDesign.FuelSystemDesignID
    INNER JOIN Aspiration
        ON EngineConfig.AspirationID=Aspiration.AspirationID
    INNER JOIN CylinderHeadType
        ON EngineConfig.CylinderHeadTypeID=CylinderHeadType.CylinderHeadTypeID
    INNER JOIN FuelType
        ON EngineConfig.FuelTypeID=FuelType.FuelTypeID
    INNER JOIN IgnitionSystemType
        ON EngineConfig.IgnitionSystemTypeID=IgnitionSystemType.IgnitionSystemTypeID
    INNER JOIN Mfr
        ON EngineConfig.EngineMfrID=Mfr.MfrID
    INNER JOIN EngineVersion
        ON EngineConfig.EngineVersionID=EngineVersion.EngineVersionID
    INNER JOIN Valves
        ON EngineConfig.ValvesID=Valves.Valvesid
    INNER JOIN VehicleToBedConfig
        ON Vehicle.VehicleID=VehicleToBedConfig.VehicleID
    INNER JOIN BedConfig
        ON VehicleToBedConfig.BedConfigID=BedConfig.BedConfigID
    INNER JOIN BedLength
        ON BedConfig.BedLengthID=BedLength.BedLengthID
    GROUP BY part_id

Answer 1

I ran into the same problem recently: MySQL started to choke (stuck in state 'statistics') on queries with a lot of tables joined together. I found a good blog post explaining why this happens and how to solve it.

Basically at least in MySQL 5.5, the default value for the configuration parameter optimizer_search_depth is 62 which will cause the query optimizer to take exponentially more time as the number of tables in the query increases. After a certain point it will start to take days or even longer to finish the query.

However, if you set the optimizer_search_depth to 0 in your my.cnf, MySQL automatically chooses suitable depth, but limits it to 7 which is not yet too complex.

Apparently this issue has been fixed in MySQL 5.6 but I haven't tested it myself yet.

Answer 2

I had the same issue on a Centos server 4 cores and 180GB ram. A simple query was taking forever to run and stuck on statistics state just like what you have explained. I ran OPTIMIZE TABLE on the tables it was querying and like 5 minutes later problem was solved. However, I never found out if it really solved the issue or not.

Just a suggestion.

Answer 3

I spent the day yesterday dealing with a similarly troublesome query...

Another option to quickly eliminate query optimizer hangs is to add STRAIGHT_JOIN just after SELECT.

For example:

SELECT STRAIGHT_JOIN
table1.column_a,
table1.column_b,
table2.column_a,
table2.column_b
FROM table1
JOIN table2 USING(column_a)

This is not an ideal solution, since it is entirely bypassing query optimization for joins, but if you're stuck solving a mission-critical problem in a legacy application like I was yesterday, it might help. I do not recommend slapping this on every query, but it's an effective solution in some cases.

Answer 4

In my case it was a huge database - many databases, many tables. After startup Mariadb was working slowly, all the queries not executing fast enough but stuck on "statistics". I did 3 things:

• increased file limit to 4000000 in etc/systemd/system/multi-user.target.wants/mariadb.service
• increased open_files_limit and table_definition_cache to about 2500000
• changed table_open_cache and table_definition_cache to about 3000
• installed "libjemalloc" for Mariadb as memory usage was still growing.

That solved the problem. Of course you don't have to set these numbers so high, that depends on your number of databases, tables & files.