Called cyclically to process tokens from tokensIn to tokensOut. The implementation should read the tokens in tokensIn, and append (possibly modified) tokens to tokensOut. The processed tokens need to be removed from tokensIn.
All tokens have been processed and no new tokens will be appended. Should be set if tokensIn is empty and has EndOfData set, and there are no pending tokens due to the state of the modifier function block.