To the north of Jami Masjid are the residential buildings of Abul Fazal and his elder brother Faidi. These two buildings have similar plan and dimensions and are built of red sandstone. The building on the south-eastern side is said be the house of Abul Fazal. This is a rectangular single-storied building. It has a tibara verandah with three openings in front and two pairs of carved and stylized columns. The fluting capitals of the columns with beautiful brackets supporting the lintels of roof look pretty attractive. The two small chambers on either side of verandah direct to the terrace of the building.