Below are some relevant references in the field. You can suggest more in the Mailing list.
Books
An Introduction to Statistical Learning with Applications in R, by Gareth James, Daniela Witten, Trevor Hastie and Robert Tibshirani. Available for free at http://www-bcf.usc.edu/~gareth/ISL/ , but you should buy it!
The Elements of Statistical Learning: Data Mining, Inference, and Prediction, by Trevor Hastie, Robert Tibshirani and Jerome Friedman. Available for free at http://statweb.stanford.edu/~tibs/ElemStatLearn/ , but you also should buy it!
Network Intrusion Detection and Prevention - Concepts and Techniques, by A. A. Ghorbani, Wei Lu and Mahbod Tavallaee.
Data Mining Tools for Malware Detection, by Mehedy Masud, Latifur Khan and Bhavani Thuraisingham.
Papers and other references
Sebastián García, Alejandro Zunino and Marcelo Campo Detecting Botnet Traffic from a Single Host In: Handbook of Research on Emerging Developments in Data Privacy (2014)
David Zhao, Issa Traore, Bassam Sayed, Wei Lu, Sherif Saad, Ali Ghorbani, Dan Garant, Botnet detection based on traffic behavior analysis and flow intervals, Computers & Security (2013)
Prathibha, .P.G and Dileesh, .E.D Design of a Hybrid Intrusion Detection System using Snort and Hadoop, International Journal of Computer Applications (2013)
Wei Lu, Goaletsa Rammidi and Ali A. Ghorbani, Clustering botnet communication traffic based on n-gram feature selection, Computer Communications (2011)
Tarem Ahmed, Mark Coates and Anukool Lakhina, Multivariate Online Anomaly Detection Using Kernel Recursive Least Squares, IEEE Communications Society (2007)
Tarem Ahmed, Boris Oreshkin and Mark Coates, Machine Learning Approaches to Network Anomaly Detection, In: Proceeding SYSML07 USENIX (2007)
R.W. Thommes and M.J. Coates Modeling Virus Propagation in Peer-to-Peer Networks, In: Information, Communications and Signal Processing (2005)
Rui M. Castro, Mark J. Coates and Robert D. Nowak, Likelihood Based Hierarchical Clustering, IEEE Transactions on Signal Processing (2004)