In this work, acceleration feedback is utilized in a hierarchical control structure for robust trajectory control of a quadrotor helicopter subject to external disturbances where reference attitude angles are determined through a nonlinear optimization algorithm. Furthermore, an acceleration-based disturbance observer (AbDOB) is designed to estimate disturbances acting on the positional dynamics of the quadrotor. For the attitude control, nested position, velocity, and inner acceleration feedback loops consisting of PID and PI type controllers are developed to provide high stiffness against external disturbances. Reliable angular acceleration is estimated through a cascaded filter structure. Simulation results show that the proposed controllers provide robust trajectory tracking performance when the aerial vehicle is subject to wind gusts generated by the Dryden wind model along with the uncertainties and measurement noise. Results also demonstrate that the reference attitude angles calculated through nonlinear optimization are smooth and within the desired bounds.