"In practice problem 6, no.18 a and b, why can’t we use the formula A* = Q-1AP? By using this formula, it gives the same answer as your solution. Please clarify"

This method does in fact work not just for square matrices as long as the dimensions correspond.

In this case the dimensions correspond with the dimensions of the transformation so when you do the matrix multiplication everything works out.

What if you swapped the roles of the bases? You would find that the matrix multiplication is undefined because the dimensions don't match. In this case you would have no choice but to transform the basis vectors and do it that way.

I derived the matrix method which is guaranteed to work for square matrices. It can be used in some other cases too and you may feel free to use it if the dimensions match. If they do not, then resort to the transforming of basis vectors method. Both are achieving the same thing.