The best hotel hands down in Victoria is the Fairmont Empress (ask for the Gold rooms). Service and attention is amazing!
Second I would say would be the Brentwood Bay Lodge and Spa.
Third would be the Magnolia Inn and 4th would be the Grande Pacific.
All nice in thier own way.
Bengal room lounge at the Empress has the best martini's!
Hope this helps.