Systems theory in archaeology is the application of systems theory and systems thinking in archaeology. It originated with the work of Ludwig von Bertalanffy in the 1950s, and is introduced in archaeology in the 1960s with the work of Sally R. Binford & Lewis Binford's "New Perspectives in Archaeology" and Kent V. Flannery's "Archaeological Systems Theory and Early Mesoamerica".
Bertalanffy attempted to construct a general systems theory that would explain the interactions of different variables in a variety of systems, no matter what those variables actually represented. A system was defined as a group of interacting parts and the relative influence of these parts followed rules which, once formulated could be used to describe the system no matter what the actual components were.
Binford stated the problem in New Perspectives in Archaeology, identifying the Low Range Theory, the Middle range theory, and the Upper Range Theory.
At the time Binford thought the Middle Range Theory may be as far as archaeologists could ever go, but in the mid-1970s some believed that Systems Theory offered the definitive Upper Range Theory.
Archaeologist Kent Flannery did some very important and pioneering work in this field in his paper Archaeological Systems Theory and Early Mesoamerica. Systems theory allowed archaeologists to treat the archaeological record in a completely new way. No longer did it matter what you were looking at, because you were breaking it down to its elemental system components. Culture may be subjective, but as long as you treat it mathematically the same way as you treat a retreating glacier then unless you attack the model of Systems Theory in general then your results were undeniably objective. In other words, the problem of cultural bias no longer had any meaning, unless it was a problem with Systems Theory itself. Culture was now just another natural system that could be explained in mathematical terms.
Unfortunately archaeologists found it was rarely possible to use Systems Theory in a rigorously mathematical way. While it provided a wonderful framework for describing interactions in terms of types of feedback within the system, it was rarely possible to put the quantitative values that Systems Theory requires for full use, as Flannery himself admits. The result was that in the long run Systems Theory was less useful in explaining change as it was in describing it. 
Systems Theory also eventually went on to show that predictions that a high amount of cultural regularities would be found were certainly overly optimistic during the early stages of Processual archaeology. Ironically enough this is exactly the opposite of what Processual archaeologists were hoping it would be able to do with Systems Theory. However it was not completely a disappointment and Systems Theory is still used to describe how variables inside a cultural system can interact.
Systems Theory, at least, was important in the rise of Processual archaeology and was a call against Culture-Historical methods of past generations. It held argument that one could contemplate the past impartially and sidestep pitfalls through rigour.