Thomas Southworth, Middleton Conservative Association