[sql] How to do a SUM() inside a case statement in SQL server

You could use a Common Table Expression to create the SUM first, join it to the table, and then use the WHEN to to get the value from the CTE or the original table as necessary.

WITH PercentageOfTotal (Id, Percentage) 
AS 
(
    SELECT Id, (cnt / SUM(AreaId)) FROM dbo.MyTable GROUP BY Id
)
SELECT 
    CASE
        WHEN o.TotalType = 'Average' THEN r.avgscore
        WHEN o.TotalType = 'PercentOfTot' THEN pt.Percentage
        ELSE o.cnt
    END AS [displayscore]
FROM PercentageOfTotal pt
    JOIN dbo.MyTable t ON pt.Id = t.Id