The reason we use a rectangular format is because our printing industry is built around four-sided imaging. This is no accident as it is extremely wasteful to print or manufacture paper that is not four sided. Imagine trying to cut radius shapes instead of straight sides, this would take much of the uniformity away from the industry and would be detrimental to pricing.
Circular sensors would also be wasteful. The sensor shape is intended to be complimentary to our printing systems. A round sensor would have to contain the additional photo cells required but rarely used to reproduce a round image. The camera's image processor would have to manage this data too.
