I'm using a single sprite sheet image as the main texture for my breakout game. The image is this:

My code is a little confusing, since I'm creating two elements from the same Texture using a Point, to represent the element size and its position on the sheet, a Vector, to represent its position on the viewport and a Rectangle that represents the element itself.

Texture2D sheet;

Point paddleSize = new Point(112, 24);
Point paddleSheetPosition = new Point(0, 240);
Vector2 paddleViewportPosition;
Rectangle paddleRectangle;

Point ballSize = new Point(24, 24);
Point ballSheetPosition = new Point(160, 240);
Vector2 ballViewportPosition;
Rectangle ballRectangle;
Vector2 ballVelocity;

My initialization is a little confusing as well, but it works as expected:

paddleViewportPosition = new Vector2((GraphicsDevice.Viewport.Bounds.Width - paddleSize.X) / 2, GraphicsDevice.Viewport.Bounds.Height - (paddleSize.Y * 2));
            paddleRectangle = new Rectangle(paddleSheetPosition.X, paddleSheetPosition.Y, paddleSize.X, paddleSize.Y);

Random random = new Random();
ballViewportPosition = new Vector2(random.Next(GraphicsDevice.Viewport.Bounds.Width), random.Next(GraphicsDevice.Viewport.Bounds.Top, GraphicsDevice.Viewport.Bounds.Height / 2));
ballRectangle = new Rectangle(ballSheetPosition.X, ballSheetPosition.Y, ballSize.X, ballSize.Y);
ballVelocity = new Vector2(3f, 3f);

The problem is I can't detect the collision properly, using this code:

if(ballRectangle.Intersects(paddleRectangle))
{
    ballVelocity.Y = -ballVelocity.Y;
}

What am I doing wrong?