let the sides of the larger rectangle measure a and b with a being the shorter side while the lengths of the smaller rectangle measures c and d
let r be the diagonal of the smaller one = sqrt(c2 + d2)
r must be less than the diagonal of the larger one to be able to be rotated inside it ie r < sqrt(a2 + b2)
to touch 3 sides, r must be GREATER than the length of the SHORTEST side of the larger rectangle, if it is smaller, it can be rotated up to 90deg but only two corners will touch two sides at most until it is fully rotated to 90deg when two sides including 3 corners can be touching two sides of the larger rectangle.
the angle of the smaller rectangles diagonal is alpha and tan(alpha) = c/d
in a coordinate setup, place the larger rectangle as [0,a] and [0,b] rotate the smaller rectangle about a chosen point, and impose that three of the rotated vertices land on the edges x=0, x=a, y=0, or y=b.
the small rectangle will be rotated by angle theta such that 3 of its corners touch the side of the larger one
horizontal projection of smaller triangle (H) = c cos theta + d sin theta
vertical projection of smaller triangle (V) = c sin theta + d cos theta
where theta is positive and < 90deg
if one wished to solve for all 4 corners touching the sides, H = a and V = b, but for 3 corners we will just use H = a hence, a = c cos theta + d sin theta
rearrange using trigonometric auxiliary angle identity:
solve for theta