A mathematical formula designed to predict population fluctuations in a community could be called a

I think its Ecological model

Obviously not an experiment, becasue nothing affected in real world. Obviously not a system, because a system is a real world thing that functions, like a traffic system. A formula is not an observation, rather an observation is what one uses to deduce a formula. So clearly it is a model.