Cyberbullying d etection o n m ulti-m odal d ata u sing p re-t rained d eep l earning a rchitectures

Introduction: The present article is the product of the research “Cyberbullying Detection on Multi-Modal Data Using Pre-Trained Deep Learning Architectures.”, developed at Pondicherry Engineering College in the year 2020. Problem: Identification of cyberbullying activities on multi-modal data of social media Objective: To propose a model that can identify cyberbullying activity for text and image data. Methodology: This paper has extracted the features of using two pre-trained architectures for text data and image data, in order to identify cyberbullying activities on multi-modal data, concatenated text features and image features, before supplying them as inputs to the classifier. Results: An analysis has been performed on the proposed approach implemented on multi-modal data with Recall, and F1-Score as measures. Grad-cam visualization is presented for images to show highlighting regions. Conclusion: The results indicate that the proposed approach is efficient when compared with the baseline methods. Originality: The proposed approach is effective and conceptualized to improve cyberbullying detection on multi-modal data. Limitations: There is a need to develop a model which can identify bullying graphical images and videos


[1] INTRODUCTION
Social media networks such as Facebook, Twitter, Instagram etc… assembled a large audience at one place irrespective of boundaries all over the world. According to the statistics reported by Statista * , a web-based report, the current social media population stands at 4.14 billion, which is more than half of the total world population. On one side, there is popularity and increase in the number of users on the social media platforms, on the other side, it also attracts illicit, criminal, unlawful activities executed by illegitimate users, such as online hate speech, online trolling, cyberbullying etc. One of the most harmful activities that affect the teenagers and youth of social media is cyberbullying. Teenage girls are more affected victims than boys (34.5% boys and girls 38.5%) [1] . Cyberbullying happens when the digital platforms are used as a medium to bully someone through shaming, degrading and demeaning which can lead to mental breakdowns. It creates severe psychological disorders [2] and sometimes leaves the victims with to suicidal tendencies. Since 2010, the instances of cyberbullying have increased rapidly as more children become targets of bullying. Cyberbullying is a more heinous crime than traditional bullying as it happens anytime and anywhere. One of the biggest challenges faced when dealing with cyberbullying is its With the rapid increase of digital technologies over the last two decades, netizens share their opinions through different formats such as texting, images, videos, and emojis. Due to the existence of abundant forms of expression of data, cyberbullying becomes a challenging task. Cyberbullying text messages might involve short texts, misspelt texts, texts with embedded symbols. Cyber-bullying through images might comprise complex facial expressions, animals, or some embarrassing images [13]. Other modes of cyberbullying are encompassed using a combination of text and image, image and video, text and emoji, etc. Most of the research works pursued in the literature place an emphasis on cyberbullying detection with text data only. Only a limited number of studies are in existence on cyberbullying detection using image data and multi-modal data. These scenarios motivated researchers to mitigate cyberbullying activities on social media for multi-modal data.
The contribution to this research work is twofold.
1. To propose a neural network-based method to handle multi-modal data for cyberbullying detection.

2.
To perform an extensive analysis on multi-modal dataset.
The rest of the paper is organized as follows. Section 2 describes related works in cyberbullying detection. Section 3 details the methodology adopted for the proposed work. Section 4 deals with datasets and the experimental results. Section 5 concludes the work with directions on future work.

[2] RELATED WORKS
In this section, the earlier works related to cyberbullying detection with text data and image data are discussed.
Reynolds et al [3] used language-based methods to identify cyberbullying messages. The authors crawled the data from Formspring.me website. They manually labeled the tweets with the help of Amazon's Mechanical Turk service, based on the bad words in each record. They used 'NUM' and 'NORM' as two features to train the model.  Akshi kumar and Nitin Sachedeva [15] studied the significance of soft computing techniques for cyberbullying detection. Zhang X et.al [16] proposed novel convolutional neural network based on pronunciation (PCNN). Threshold movement, cost function adjustment and yielding hybrid solutions were utilized to handle class imbalance problems. Cyberbullying datasets from Twitter and Formspring.me social networking sites comprised the corpus. Tripati K et al [17] proposed an ALBERT-based fine-tuning model for cyberbullying detection; the model achieved better performance than GRU and BERT. Sayanta Paul and Sriparna Saha [18] presented an innovative method of BERT for cyberbullying identification. The proposed method was able to classify bullying tweets on three real world datasets. K kumari et al [19,20] proposed a single layer convolutional neural network for cyberbullying identification on multi-modal data. The

[3] METHODOLOGY
The proposed architecture is presented and discussed in detail in the preceding section. The proposed technique to detect cyberbullying in text and image data fundamentally consists of four components as illustrated in Fig. 1 viz., a) Input data, b) Pre-processing, c) Proposed approach and d) Classification.

Problem statement
Let T and I represent text and images of a given tweet or comment. Let X T € R d1 and X I € R d2 denote the features of text and images respectively, the goal is to design a model that can predict the label of a given comment using features X I and X T . This is formulated as a minimization loss function as provided belowin Equation-1. min f(Ө) = f (y | X I , X T ; Ө) (1) where Ɵ represents the model parameters and y symbolizes the combined label.  The dataset employed in [19] is utilized as input data, which is an amalgamation of data from different social media platforms; namely Facebook, Instagram, and Twitter.
Finally, 2100 samples were collected for training the model. Each record comprises two fields namely text comment and image. The combination of text and image yields six possible cases for bullying and non-bullying records. Table-1    You are not fat you are just chubby.

Case 2 Comment text: Non-Bullying
I am taller than you.

Non-Bullying
Source: own work

c) Proposed Approach
Most of the research works conducted for cyberbullying detection principally focuses on text-based features. Though text-based data serves as a primary source of information to classify the comments as bullying and non-bullying, a more refined and richer representation would improve the detection system to conduct the classification more precisely. Hence, a robust detection system may require the presence of several modalities to resolve ambiguities, if any, that may arise by inferring the multiple-modal information fusion. A multimodal detection system normally performs better than any one of its individual components. With this hypothesis, we propose a novel approach for cyberbullying detection by leveraging the information from two modalities; i.e., the text and images. The input data comprises both the text and images for any given tweet or comment and is provided with a binary label, either as bullying or non-bullying. In the proposed approach, the features are first extracted from the text using RoBERTa, a neural network architecture, and the Xception model to extract the features from images. The contribution of the proposed approach is that it leverages the information from both the text and images to enhance the performance of the system. Also, we demonstrate the neural networks attention on images that help the model to decide bullying vs non-bullying.

Xception Model to Extract the Image Embeddings
The Xception model [21] stands for extreme version of Inceptions and is fundamentally Data. The LightGBM classifier follows leaf-wise tree growth which is advantageous for the classification task. Fig.2 shows the leaf-wise growth in LightGBM classifier.
1000 estimators (Sequential Decision Trees) with maximum depth of 5 were set as input parameters for LightGBM classifier. Each node indicates a decision to classify the tasks.  Table 4. Algorithm of the proposed classification scheme

Input:
Raw-text and Image of Tweets or comments.

Step 5
Test: Given test text (Z T ) and Image (Z I ) , Y Prediction ß Model Ɵ (Z T, Z I )

[4] RESULTS
The following section presents the experimental methodology, metrics employed for evaluation and the respective outcomes. The proposed technique is implemented in Python using packages such as Numpy, Pandas, matplotlib, Scikit-Learn, LightGBM, and Tensorflow in the Linux operating system. The methods were run on an Intel i7 8th Gen 12core CPU processor and Nvidia Max-Q 1070 32GB RAM.

Evaluation metrics
To evaluate the proposed approach, the evaluation metrics such as precision, recall and F1-score are considered.
The precision is defined as the ratio of correct predictions of bullying to total number of predictions of bullying and is calculated as given in Eq. (2).

Experiment methodology
We conducted experiments with no sampling, over-sampling and under-sampling modes using five-fold cross validation. The proposed approach achieved a weighted average F1-score of 80% as compared to existing approaches. The proposed method is able to achieve recall and F1-scores of 92% and 86% for bullying class respectively. Fig. 3 shows recall of bullying class which is able to classify bullying tweets 13% more efficiently than existing methods and Fig. 4 shows F1-scores for bullying class which is 6% more efficient compared to the existing approaches. Fig. 5 shows the weighted F1-score. Table 4 shows experimental results of Kumari et.al, [20]. The authors initially proposed a CNN architecture to identify bullying on multi-modal data. Later, they proposed a genetic algorithm to find the best features from text and image data and be able to classify the multi-modal bullying data efficiently.  We apply Gradient-weighted Class Activation Mapping (Grad-CAM) [23] to highlighting important regions which causes bullying in images.

CONCLUSION AND FUTURE DIRECTIONS
The occurrence of cyberbullying increases in proportion with technological growth and high-speed digital devices in general and sophisticated online social media platforms in specific. The existing approaches to detect cyberbullying rely on data at the text level. In order to improve the performance of the cyberbullying detectors, a more e-ISSN 2357-6014 / Vol. 17, no. 3 / september-december 2021 / Bogotá D.C., Colombia Universidad Cooperativa de Colombia refined approach was presented in this work by considering combinatorial data that include text as well as image data. The proposed approach was effective in identifying cyberbullying using combinatorial data and achieved 92% recall, and an 82% F1-score for bullying class. The weighted F1-score of proposed model achieves 80%.
Though the work proposed addresses the detection of cyberbullying, it considered the text-level features that are in the English language and requires additional mechanisms to apply the detection process in other languages. As part of the future work, the detection of cyberbullying is aimed to be attempted in Indian regional languages.