How to know which reviewers have declined my invitation?

Go to the Reviews hub from the dashboard:

Then select the List of Reviewers and click on the All Statuses filter.

Filter the reviewers according to their status and select Declined Invitation to see the list of reviewers who have declined your invitation to be part of the review process. If Declined Invitation is not shown on the status list, then none of your reviewers has declined. 

• Ready means that a reviewer has accepted the invitation to review and has chosen their topics. 
• Accepted Invitation means they have accepted your invitation but have yet to complete some fields such as topics. 
• Pending means that the reviewer did not give any reaction so far for your invitation