They just violated HVCC:
Home Valuation Code of Conduct
I. Appraiser Independence Safeguards
B. No employee, director, officer, or agent of the lender, or any other third party acting as joint venture partner, independent contractor, appraisal company, appraisal management company, or partner on behalf of the lender, shall influence or attempt to influence the development, reporting, result, or review of an appraisal through coercion, extortion, collusion, compensation, inducement, intimidation, bribery, or in any other manner including but not limited to:
(8) allowing the removal of an appraiser from a list of qualified appraisers, or the addition of an appraiser to an exclusionary list of disapproved appraisers, used by any entity, without prompt written notice to such appraiser, which notice shall include written evidence of the appraiser’s illegal conduct, a violation of the Uniform Standards of Professional Appraisal Practice (USPAP) or state licensing standards, substandard performance, improper or unprofessional behavior or other substantive reason for removal (except that this prohibition will not preclude the management of appraiser lists for bona fide administrative reasons based on written, management-approved policies);
Not wanting to pay your fees is not a substantive reason for removal.
Lets put the hurt on their working relationship. Here is the Freddie Mac HVCC complaint form: