Traffic and mobility aware delay modeling for software-defined networks (SDN)

ISTANBUL TECHNICAL UNIVERSITYF GRADUATE SCHOOL OF SCIENCE ENGINEERING AND TECHNOLOGY

TRAFFIC AND MOBILITY AWARE DELAY MODELING FOR SOFTWARE-DEFINED NETWORKS (SDN)

Ph.D. THESIS Müge ÖZÇEV˙IK

Department of Computer Engineering Computer Engineering Programme

ISTANBUL TECHNICAL UNIVERSITYF GRADUATE SCHOOL OF SCIENCE ENGINEERING AND TECHNOLOGY

TRAFFIC AND MOBILITY AWARE DELAY MODELING FOR SOFTWARE-DEFINED NETWORKS (SDN)

Ph.D. THESIS Müge ÖZÇEV˙IK

(504152512)

Department of Computer Engineering Computer Engineering Programme

˙ISTANBUL TEKN˙IK ÜN˙IVERS˙ITES˙I F FEN B˙IL˙IMLER˙I ENST˙ITÜSÜ

YAZILIM TANIMLI A ˘GLAR ˙IÇ˙IN

TRAF˙IK VE HAREKET DUYARLI GEC˙IKME MODEL˙I

DOKTORA TEZ˙I Müge ÖZÇEV˙IK

(504152512)

Bilgisayar Mühendisli˘gi Anabilim Dalı Bilgisayar Mühendisli˘gi Programı

Müge ÖZÇEV˙IK, a Ph.D. student of ITU Graduate School of Science Engineering and Technology 504152512 successfully defended the thesis entitled “TRAFFIC AND MOBILITY AWARE DELAY MODELING FOR SOFTWARE-DEFINED NETWORKS (SDN)”, which she prepared after fulfilling the requirements specified in the associated legislations, before the jury whose signatures are below.

Thesis Advisor : Assoc. Prof. Dr. Berk CANBERK ... Istanbul Technical University

Jury Members : Assoc. Prof. Dr. Feza BUZLUCA ... Istanbul Technical University

Prof. Dr. Öznur ÖZKASAP ... Koç University

Prof. Dr. Fatih ALAGÖZ ... Bo˘gaziçi University

Assist. Prof. Dr. Gökhan SEÇ˙INT˙I ... Istanbul Technical University

Date of Submission : 27 March 2019 Date of Defense : 3 May 2019

To my huge family, "Learn from yesterday, live for today, hope for tomorrow. The important thing is not to stop questioning." Albert Einstein.

FOREWORD

Firstly; I would like to thank my advisor Assoc. Prof. Dr. Berk CANBERK. It is an honor to be one of the PhD graduates of Broadband Communication Research Group (BCRG). I’m extremely grateful to be a member of this family during my BSc, MSc, and PhD degrees in Computer Engineering Department of Istanbul Technical University. I believe that the thing laid behind the success was keeping our communication reliable without any packet loss. He is always taking attention to each task without considering its significance level; and therefore, I have tried to be worthy of his labor during our 10 years passing together. Thanks to his never-ending struggle in my academic writing skill; now, I can see this improvement in myself. I believe that we will reach more academic success in BCRG.

Secondly; I would like to thank my thesis committee Assoc. Prof. Dr. Feza BUZLUCA and Prof. Dr. Öznur ÖZKASAP. Their suggestions and comments in each progresses made the thesis more clear and helped to map my research with real-life scenarios. I would like to thank them for being always available and kindly answering all my questions. It was an honor to have an opportunity for research discussion with them. I will also be gladded to enhance my research by considering their experiences. Thirdly; another thank also goes out to BCRG, and especially our first PhD graduate Dr. Gökhan SEÇ˙INT˙I. They were my supporters during my assistantship. It was quite well that Prof. CANBERK has merged us in joint studies. It has made us a member of an endless brotherhood. Thanks for this family. Moreover; I have special thanks to my colleagues: Dr. Ahmet ARI ¸S, Kübra ADALI, Tu˘gçe B˙ILEN, Elham DEHGHAN B˙IYAR. We’ve shared our feelings in each step of our academic progress. Thanks for their sincerity.

Fourthly, I would also like to acknowledge and thank the ASELSAN for their support through the Graduate Scholarship for Turkish Academicians. This thesis was also supported by the Research Fund of the Istanbul Technical University with a project number: 40800.

Lastly and foremost; I would like to thank my huge family. Especially; my sincere gratitude is reserved for my couple Yusuf ÖZÇEV˙IK for his all support on my academic career. Despite our missed moments from homeland during this process, I believe that this thesis is worthy of this. Then, I would like to thank my all parents who had supported me for this thesis. I have always tried to make them proud whole my life. Thanks for all.

May 2019 Müge ÖZÇEV˙IK

TABLE OF CONTENTS Page FOREWORD... ix TABLE OF CONTENTS... xi ABBREVIATIONS ... xv SYMBOLS ...xvii

LIST OF TABLES ... xix

LIST OF FIGURES ... xxi

SUMMARY ...xxiii

ÖZET ...xxvii

1. INTRODUCTION ... 1

1.1 Purpose of the Thesis... 5

1.1.1 Traffic management ... 6

1.1.1.1 Traffic heterogeneity management ... 6

1.1.1.2 Traffic intensity management ... 11

1.1.2 Mobility management... 13

1.1.2.1 Handover management for eMBB services ... 13

1.1.2.2 Handover management for URLLC services... 16

1.2 Contribution of the Thesis ... 17

1.2.1 Traffic management ... 18

1.2.1.1 Traffic heterogeneity management ... 18

1.2.1.2 Traffic intensity management ... 20

1.2.2 Mobility management... 21

1.2.2.1 Handover management for eMBB services ... 21

1.2.2.2 Handover management for URLLC services... 22

1.3 Literature Review ... 23

1.3.1 Traffic management in UDN ... 23

1.3.1.1 Edge centric approaches ... 24

1.3.1.2 Traffic type aware approaches ... 25

1.3.1.3 Network-centric approaches ... 25

1.3.2 Mobility management in UDN ... 26

1.3.2.1 Edge centric approaches ... 26

1.3.2.2 Network-centric approaches ... 27

1.3.3 Software-Defined Networks (SDN)... 28

1.3.3.1 OpenFlow protocol ... 30

1.3.3.2 SDN controllers ... 32

1.3.3.3 Traffic and mobility management in SDN... 35

2. TRAFFIC HETEROGENEITY MANAGEMENT ... 37

2.1 Network Architecture ... 37

2.2 Proposed e2eDelay (Di(t)) Model ... 38

2.2.1 Data plane effect on e2eDelay... 39

2.2.1.1 Network virtualization sub-plane ... 40

2.2.1.2 Controller sub-plane ... 40

2.2.2 Control plane effect on e2eDelay ... 48

2.3 Performance Evaluation ... 51

2.3.1 Data plane effect on e2eDelay (Di(t)) ... 52

2.3.1.1 Heterogeneity of traffic flows (Hj(t)) effect ... 53

2.3.1.2 Intensity of eNBs (ρj(t)) effect ... 53

2.3.2 Control plane effect on e2eDelay (Di(t))... 54

2.3.2.1 Complexity analysis... 54

2.3.2.2 Experimental analysis ... 55

2.3.3 E2eDelay (Di(t)) comparison both data and control planes ... 57

2.4 Conclusion ... 58

3. TRAFFIC INTENSITY MANAGEMENT... 59

3.1 Network Architecture ... 59

3.2 The System Architecture of Proposed SDPN... 61

3.2.1 E2eDelay (D) calculation ... 62

3.2.2 E2eDelay optimization ... 65

3.2.2.1 Level 1: Platooning... 66

3.2.2.2 Level 2: Load balancing in core ... 67

3.3 Performance Evaluation ... 71

3.3.1 Level 1 ... 72

3.3.2 Level 2 ... 76

3.3.3 e2eDelay (D) ... 77

3.4 Conclusion ... 77

4. HANDOVER MANAGEMENT FOR EMBB SERVICES ... 79

4.1 Network Architecture of SDUN ... 79

4.2 Proposed SDUN Framework ... 80

4.2.1 e2eDelay (D(t)) analytical model & handover procedure ... 80

4.2.1.1 e2eDelay (D(t)) ... 81

4.2.1.2 Xn-handover procedure in UDN [1] [2] ... 82

4.2.1.3 Handover procedure in SDUN... 82

4.2.2 SDUN controller framework ... 83

4.2.2.1 Network virtualization ... 84

4.2.2.2 Edge delay (We(t))... 85

4.2.2.3 Core delay (Wc(t)) ... 88

4.2.3 The cost analysis of the SDUN controller, O(k4log2(k)): ... 90

4.2.3.1 Network virtualization, O(k4)... 90

4.2.3.2 Parallel edge delay optimization, O(klog(k)) ... 90

4.2.3.3 Parallel shortest delay path, O(k4log2(k)) ... 91

4.2.4 Implementation of flow forwarding processing in SDUN... 92

4.3.1 Core delay... 96

4.3.2 Edge delay ... 98

4.3.3 SDUN controller delay ... 100

4.3.4 Overall e2eDelay (D(t))... 102

4.4 Conclusion ... 102

5. HANDOVER MANAGEMENT FOR URLLC SERVICES ... 105

5.1 Network Architecture and VNF Definitions... 105

5.2 Serivce Chains in Proposed SDUN/NFV ... 106

5.2.1 Service chains during handover... 106

5.2.2 The length of service chains and e2eDelay ... 107

5.3 Queuing Theoretical Model of e2eDelay ... 109

5.3.1 Data plane ... 110

5.3.2 Control plane ... 112

5.4 Performance Evaluation ... 112

5.5 Conclusion ... 116

6. CONCLUSIONS AND FUTURE WORK... 117

REFERENCES... 121

APPENDICES ... 131

APPENDIX A.1 ... 133

1.1 Total waiting time in M/M/c/K... 133

APPENDIX A.2 ... 135

2.1 Total Waiting Time in M/G/1/K ... 135

ABBREVIATIONS

3GPP : The 3rd Generation Partnership Project 4G : The Forth Generation Mobile Networks 5G : The Fifth Generation Mobile Networks AR : Augmented Reality

BS : Base Station

CAPEX : Capital Expenditure

CRUD : Create, Read, Update, Delete DRB : Data Radio Bearer

e2eDelay : End to End Delay

EM : End Marker

eMBB : Enhanced Mobile Broadband

eNB : ENodeB

EPC : Evolved Packet Core ETH_TYPE : Ethernet Type

ETSI : European Telecommunications Standards Institute EUTRAN : Evolved Terrestrial Radio Access Network

GPRS : General packet radio service

GTP-U : GPRS tunneling protocol - user plane HTTP : HyperText Transfer Protocol

IETF : Internet Engineering Task Force IMT : International Mobile Communications IN_PORT : Ingress Port

IP : Internet Protocol IP_PROTO : IP Protocol Number

IPV4_DST : IP Version 4 Destination Address IPV4_SRC : IP Version 4 Source Address

ISO : International Organization of for Standardization LTE : Long Term Evolution

MANO : Management and Organization

METIS : Mobile and wireless communications Enablers for 2020 MILP : Mixed Integer Linear Programming

MIMO : Multiple Input Multiple Output MPLS : Multiprotocol Label Switching MvNF : Macrocell virtual Network Function NFV : Network Function Virtualization NP : Non Deterministic Polynomial Time

OF : OpenFlow

OPEX : Operating Expenses

OXM : OpenFlow Extensible Match PGW : Packet Data Network Gateway QoE : Quality of Experience

REST : Representational State Transfer RRC : Radio Resource Control

RSSI : Received Signal Strength Indication RSU : Road Side Unit

SDN : Software Defined Network

SDPN : Software Defined Platoon Network SDUN : Software Defined Ultra Dense Network SeNB : Source eNodeB

SGW : Serving Gateway

SOAP : Simple Object Access Protocol SON : Self Organize Network

SvNF : Smallcel virtual Network Function TCP : Tranmission Control Protocol TeNB : Target eNodeB

TLS : Transport Layer Security TLV : Type Length Value TX : Transmited Packets UDN : Ultra Dense Network UDP : User Datagram Protocol

UE : User Equipment

URLLC : Ultra Reliable Low Latency Communication vEPC : Virtual Evolved Packet Core

vNF : Virtual Network Function VR : Virtual Reality

V2I : Vehicle to Infrastructure V2V : Vehicle to Vehicle

SYMBOLS

i : The identifier of traffic flow generated by UE/ traffic flow j : The identifier of eNB/ OpenFlow (OF) switch/ Cell λ : The traffic arrival rate of OF switch

µ : The service rate of OF switch ρ (t) : The rraffic intensity of OF switch ψ : The traffic heterogeneity threshold δ : The traffic intensity threshold γ : Topology Utilization

γ1,2,3 : The speed up factor of SDUN controller

ρ_m,st : Traffic load threshold of macrocells, smallcells α : Time period of SDUN Controller

C : Capacity of edges in Graph

D : e2eDelay of UE

Dc : e2eDelay in Control Plane

Dd : e2eDelay in Data Plane

DSDUN : e2eDelay in proposed SDUN

DUDN : e2eDelay in conventional UDN

E : Edges in Graph

G : Graph

Hj(t) : Heterogeneity rate of traffic flows in eNBj

k : k-ary fat tree topology

Lq : The number of packets in a queue

m, M : The number of UE in topology

n, N : The number of OpenFlow switch in topology P : The set of Vehicle Platoons

Pc : The probability of being in physical coverage of RSU

Ph : The handover probability of UE

Pj : Serving probability over OF Switch

Pmm’,ss’ : Transition probability between intra tiers

Psm,ms : Transition probability between inter tiers

TM(t) : Mobility interruption time

TCPj(t) : The number of TCP based traffic flows in OF switch

UDPj(t) : The number of UDP based traffic flows in OF switch

UEs,d : Source or Destination UE

W(t) : Waiting Time of traffic flow in OF switch WB(t) : Waiting Time of Background traffic

WF(t) : Waiting Time of Foreground traffic

Wc(t) : Waiting Time in Core network

We(t) : Waiting Time in Edge network

Ws(t) : Processing time of Controller

WTi(t) : Transmission time of UEitraffic flow

LIST OF TABLES

Page Table 1.1 : eMBB and URLLC service comparison [3] [4] [5] [6] [7] [8]. ... 3 Table 1.2 : e2eDelay(sec) and loss details... 10 Table 1.3 : MBB/eMBB comparison in terms of mobility. ... 14 Table 1.4 : Managing an OF table by default SDN controller [9]... 31 Table 1.5 : SDN controller comparison [10]. ... 32 Table 2.1 : Simulation parameters... 51 Table 3.1 : Mobility parameters for pedestrain and vehicles in dense urban

(eMBB-UMx) [11] topology (macrocells with outdoor smallcells). .. 71 Table 3.2 : Markov parameters, and the details of URLLC and eMBB flows. ... 72 Table 3.3 : Total number of URLLC and eMBB when topology utilization γ

is increased. ... 72 Table 4.1 : Simulation parameters [11] [12] [1]... 95 Table 4.2 : Total number of UEd(m). ... 95

Table 5.1 : VNF definitions. ... 106 Table 5.2 : Dense urban (eMBB-UMx) [11] vehicular topology (macrocells

LIST OF FIGURES

Page Figure 1.1 : Mobile data traffic and identifying it in terms of 5G new services [3]. 1 Figure 1.2 : Identifying mobile data traffic in case of UDP and TCP. ... 3 Figure 1.3 : System architecture of the thesis. ... 5 Figure 1.4 : Loss ratio (%) & e2eDelay per packet (sec) of a TCP flow under

increased different background traffic in ultra-dense 5G network... 10 Figure 1.5 : Comparison of vehicle platooning approaches... 11 Figure 1.6 : Problematic situation of e2eDelay in 5G handover. ... 15 Figure 1.7 : An example network architecture of platoon vehicles. ... 16 Figure 1.8 : SDN/NFV architecture for the vehicular networks. ... 17 Figure 1.9 : Progress of main contributions. ... 19 Figure 1.10 : Comparison of 5G technologies. ... 29 Figure 1.11 : Packet flow through the processing pipeline [9]... 31 Figure 1.12 : Simplified flowchart detailing packet flow through an OpenFlow

switch [9]. ... 33 Figure 2.1 : SDUN controlled 5G network architecture. ... 38 Figure 2.2 : Proposed system framework of each SDUN controller process in

ultra-dense 5G network... 40 Figure 2.3 : Queuing model of OF switchjaccording to different traffic types. ... 44

Figure 2.4 : Data plane effect on e2eDelay (D_i(t)) according to heterogeneity of traffic flows (Hj(t))... 53

Figure 2.5 : Data plane effect on e2eDelay (Di(t)) according to load intensity

of eNBs (ρj(t))... 53

Figure 2.6 : Complexity analysis of control plane algorithms. ... 55 Figure 2.7 : Control plane effect on e2eDelay (Di(t)) according to γ4and γ6... 56

Figure 2.8 : Control plane effect on e2eDelay (D_i(t)) according to γ2... 56

Figure 2.9 : e2eDelay (Di(t)) comparison with conventional LTE and

proposed SDUN-based system. ... 57 Figure 3.1 : Network architecture and user types in proposed platoon networks. 59 Figure 3.2 : Proposed system architecture of SDPN... 61 Figure 3.3 : Analytical model of SDPN data plane... 63 Figure 3.4 : ρst and ρmt analysis for each transition between layers. ... 68

Figure 3.5 : Simulation environment of SDPN. ... 71 Figure 3.6 : Number of platoon, independent vehicle, and Pmm’ when γ =

0.005, 50% URLLC, and 50% eMBB traffic in the topology... 73 Figure 3.7 : Total URLLC request (platoons and independent vehicles) and

Pmm’according to topology utilization γ. ... 74

Figure 3.8 : DLevel2 (msec) for different ρ when 10% URLLC, and 90%

Figure 3.9 : e2eDelay, D (sec) for different topology utilization γ during a simulation period. ... 78 Figure 4.1 : Network architecture of SDUN. ... 80 Figure 4.2 : Analytical model of e2eDelay in both UDN and SDUN... 81 Figure 4.3 : SDUN controller framework. ... 83 Figure 4.4 : An example topology where delay matrix is generated as W1 in

equation 4.6... 85 Figure 4.5 : Structure of proposed Parallel Edge Delay Optimization Algorithm. 87 Figure 4.6 : Flow forwarding process in proposed SDUN... 92 Figure 4.7 : Page load time (msec) in a 4-ary fat-tree topology... 93 Figure 4.8 : Performance evaluation environment. ... 94 Figure 4.9 : The ONOS Controller and Mininet Environment. ... 96 Figure 4.10 : The QUIC Web Server and a Mobile Client (Google Chrome web

browser). ... 97 Figure 4.11 : The Dynamically Embedded OpenFlow Rules. ... 97 Figure 4.12 : Core (Wc(t)), Edge (We(t)), and e2eDelay (D(t)) for k=4-ary

fat-tree topology with delivery ratio(%) analysis. ... 99 Figure 4.13 : Cost analysis of SDUN controller. ... 100 Figure 4.14 : Scalability level of SDUN controller when n=721... 101 Figure 5.1 : Proposed architecture of vehicular network. ... 107 Figure 5.2 : Comparison of service chains before and after the proposed

SDUN/NFV architecture... 108 Figure 5.3 : Comparison of service chains and an OpenFlow table of a RSU... 109 Figure 5.4 : Dynamic markov model of RSU. ... 110 Figure 5.5 : Proposed SDUN/NFV mobility management simulation results. ... 114 Figure 5.6 : Number of assigned vehicles in an example RSU... 115

TRAFFIC AND MOBILITY AWARE DELAY MODELING FOR SOFTWARE-DEFINED NETWORKS (SDN)

SUMMARY

According to the Cisco Visual Networking Index (VNI) report, the data traffic generated from mobile devices is expected to reach 49.0 exabytes per month in 2021. To handle this increase, there is a need to focus on mobile user requirements for specific 5th Generation (5G) services. These applications are newly called as an Ultra-Reliable Low Latency Communication (URLLC) and an Enhanced Mobile Broadband (eMBB). URLLC services can be exemplified with a specific application for fully automated driving in a platoon. It decreases fuel consumption in different road dynamics, enhances traffic efficiency, and ensures safety by controlling the space between vehicles. Thanks to the Vehicle to Infrastructure (V2I) communication over 5G links, the leader vehicle takes the dynamic rules from the remote control center. It forwards maneuvers data to the followers in a platoon via the Vehicle to Vehicle (V2V) communication over IEEE 802.11p links. This service requires reliable data transfer and strict latency in downlink traffic flow during mobility. On the other hands; eMBB services are the next generation of entertainment experiences such as Augmented Reality (AR) and Virtual Reality (VR) in 360-degree video streaming for games, education, training and also tomorrows’ services such as autonomous driving. It offers 1080p, 2K, 4K and also 8K video quality. Although entertainments do not need strict latency, the eMBB service for autonomous driving requires very-low latency due to directly having a vital role and it does not desire any interruption on the 5G connection during mobility.

According to International Mobile Communications (IMT-2020), eMBB and URLLC services need 5G reduced latency requirement; i.e. keeping End-to-End Delay (e2eDelay) in a few milliseconds. It is calculated as Data and Control planes for downlink traffic from the remote host to mobile end-user. Data plane is modeled by queuing and processing delays in Core and Edge networks; whereas Control plane has a processing time of centralized controller due to signaling during mobility procedures. In this thesis, we study a new e2eDelay model while considering the traffic and mobility challenges of URLLC and eMBB services. Therefore; this leads us to investigate the following question: How to reach 5G reduced-latency for eMBB and URLLC services under huge traffic heterogeneity and intensity in mobile Ultra-Dense Networks (UDN)?

In this thesis; we propose a traffic and mobility aware Software-Defined Ultra-Dense Network (SDUN) that offers 5G reduced e2eDelay for eMBB and URLLC services. The aforementioned research question can be only handled by the SDUN framework via network-centric monitoring and reconfigurable data plane without any physical touch. SDUN offers an approach by considering traffic heterogeneity and huge traffic intensity in UDN. Here, it proposes joint consideration of edge and core networks.

These are studied in following sub-modules separately: Traffic Heterogeneity, Traffic Intensity, Handover Management for eMBB and Handover Management for URLLC services. The details are summarized in the following paragraphs; respectively.

In Traffic Heterogeneity Management, we believe that foreground TCP (URLLC) traffic flow is ’squeezed’ by UDP (eMBB) background. The reason for it is increasing waiting time in a queue of each forwarding device and extra transmission delay for each timeout in TCP congestion control mechanism. Here, traffic Heterogeneity is defined by the rate between the number of UDP and TCP traffic flows. According to the 3rd Generation Partnership Project (3GPP) Release 14, conventional Long Term Evolution - Self Organize Networks (LTE-SON) does not consider Heterogeneity rate of traffic flows while balancing the load between neighbor eNodeBs (eNBs). In order to reduce e2eDelay of foreground TCP traffic flow, an optimal path should be selected by considering both load Intensity and traffic Heterogeneity level of eNBs. To do this, we propose a SDUN-based softwarization approach brought by 5G networks with three-fold contributions: virtualization of topology graph (G), e2eDelay optimization which is run in terms of both load Intensity (ρj) and Heterogeneity rate

(Hj), and a novel Queuing Theory based OpenFlow (OF) switch model. However;

due to being a bottleneck in centralized SDUN-Controller, we propose to accelerate the processing rate with novel three heuristics including shortest path and e2eDelay optimization algorithms running in a parallel manner. More specifically, this process is combined into a novel closed-form expression of e2eDelay in two main parts: Data and Control plane effects. As a result, the proposed SDUN-based e2eDelay model serves foreground TCP traffic flow 74% and 98% less e2eDelay than LTE-SON and conventional LTE.

In Traffic Intensity Management, we believe that autonomous driving in a platoon network requires reliable data transfer and strict latency on downlink traffic. These two requirements have been addressed in 5G under the URLLC specification. In this thesis, we focus on this 5G service, and we design a novel Software-Defined Platoon Network (SDPN) by optimizing the e2eDelay. Our SDPN defines the e2eDelay optimization as a Mixed Integer Linear Problem (MILP) which considers both the V2V and V2I links in a platoon. Here, due to the NP-hard characteristic of our MILP optimization and to reduce the computational complexity of the optimization at the same time, we divide the problem space into two-sub levels: Platooning and Load Balancing. In Level 1, we find the optimum set cover of vehicles by building platoons using a novel Centralized Set Cover algorithm. In Level 2; by considering the transactions between the macrocell and small cell tiers, we balance the traffic load via a novel Load Balancing algorithm. To further accelerate our SDPN controller, we define an analytical methodology to dynamically manipulate load thresholds (ρm,s) for macrocell and small cell tiers in

Level 2. According to performance evaluation, our SDPN serves nearly 70% more traffic load in a topology by keeping e2eDelay under 3.5 milliseconds with a 45% improvement over the conventional approach.

In Handover Management for eMBB services, we believe that handover execution has still been damaging 5G latency requirement due to having three states in virtual Evolved Packet Core (vEPC). Here, the desired e2eDelay should be less than 4 milliseconds without any mobility interruption on an eMBB service of vEPC. To handle this requirement; we need to focus on the Markov model of e2eDelay. It can be measured by the concatenation of Edge and Core delays in the downlink eMBB

service from a remote source to a mobile user. Here, Edge delay is directly affected by the Core network via a decreased packet delivery ratio to Edge under huge-traffic intensity background. Therefore, Target eNodeB decision by considering only Edge network can be misleading. To overcome this, we jointly consider Edge and Core delays which are differently affected by each handover states: (1) preparation, (2) execution, and (3) completion. The joint consideration of Edge and Core can be only handled with a novel cost-effective SDUN framework by dynamically removing state (2). It triggers handover via network-centric monitoring; and then, it predetermines optimal TeNB with a proposed optimization formula and shortest Core path according to traffic intensities of OpenFlow switches. Here; SDUN controller is cost-efficient by the proposed parallel runnable algorithms: Parallel Edge Delay Optimization and Parallel Shortest Delay Path. In the performance evaluation; SDUN is firstly emulated for a specific eMBB traffic, i.e. QUIC based HTTP/3 Video content traffic with 1080p resolution, and secondly simulated in system-level on Matlab. It meets 5G requirements as follows: SDUN decreases Core delay 7.16 milliseconds per a UE under huge-traffic intensity and keeps Edge delay under 5G requirement with 20% more delivery ratio than the conventional one. Moreover, the cost of SDUN controller is analyzed as O(k4log2(k)) and the cost efficiency is observed as the 50% increased scalability level with the acceptable 8% extra virtual memory usage.

In Handover Management for URLLC services; we believe that the handover is executed with network service chain by vEPC thanks to Network Function Virtualization (NFV) to handle end to end mobility and to reduce the OPEX/CAPEX costs of vehicular networks. However; as the number of handover requests for the leader in platoon has increased in an ultra-dense topology with a high number of roadside units (RSUs), the SDUN/NFV controller has become a bottleneck with an increased response time to a vehicle. In this thesis, we investigate the following research question: How to reach few milliseconds e2eDelay for URLLC services in 5G vehicular networks? Therefore, we propose a new SDUN/NFV based handover management without disrupting the centralized manner of SDUN controller. To do this, we define a logical and physical coverage area of RSUs with two newly redesigned VNFs named as Small cell and Macrocell virtual Network Functions (SvNF, MvNF). The former one is for serving vehicle on RSU as a small cell via physical coverage; whereas the latter one is for forwarding data packets over the same RSU that is a macrocell via logical coverage without any handover request. E2eDelay is monitored by the proposed queuing theoretic formula in a novel Handover Triggering Algorithm which checks the length of service chain and determines the optimal time to run handover network function. According to performance results, the proposed SDUN/NFV architecture offers 12 milliseconds reduced e2eDelay, by keeping it under 5G requirements (a few milliseconds) with a service chain up to 7 lengths and keeping the centralized manner of SDN.

YAZILIM TANIMLI A ˘GLAR ˙IÇ˙IN

TRAF˙IK VE HAREKET DUYARLI GEC˙IKME MODEL˙I

ÖZET

Cisco tarafından yayınlanan teknik rapora göre (Visual Networking Index, VNI), mobil cihazlardan üretilen veri trafi˘ginin 2021 yılında 49.0 exabayt seviyesine ula¸sması beklenmektedir. Bu a¸sırı trafik yo˘gunlu˘gu altında, mobil kullanıcıları memnun edebilmek için bu kullanıcıların 5. Nesil (5th Generation, 5G) hizmetlerdeki taleplerine odaklanılmalıdır. Belirtilen bu 5G hizmetleri, Ultra-Güvenilir Dü¸sük Gecikme ˙Ileti¸simi (Ultra-Reliable Low Latency Communication, URLLC) ve Geni¸sletilmi¸s Mobil Bant (enhanced Mobile Broadband, eMBB) olarak adlandırılır. URLLC hizmetleri araç a˘glarında spesifik bir uygulama olan araç takımı (vehicle platoon) ile örneklendirilebilir. Araçların takım halinde ilerlemesi; farklı yol dinamiklerinde bile yakıt ve trafik verimlili˘gini arttırır ve takım içerisindeki araç arası mesafeleri kontrol ederek sürü¸s güvenli˘gini sa˘glar. Araç ile a˘g altyapısı (Vehicle to Infrasructure, V2I) arasındaki 5G ba˘glantısı sayesinde, takım içerisindeki lider araç kontrol merkezinden trafik akı¸s kontrolü için dinamik kuralları alır ve takım içerisindeki kendisini takip eden araçlara manevra için gerekli verileri IEEE 802.11p ba˘glantısı üzerinden araçtan araca (Vehicle to Vehicle, V2V) ile iletir. Bu servis, güvenilir veri ileti¸simi ve katı gecikme (strict latency) ¸sartı gerektirir. Di˘ger bir yandan; eMBB hizmetleri, e˘gitim ve oyunlarda kullanılmak üzere 360 derecelik video akı¸sı sa˘glayan artırılmı¸s ve sanal gerçeklik (Augemented Reality, AR - Virtual Reality, VR) deneyimleri ve yarının otonom arabaları için gerekli veri iletimi olarak tanımlanır. Bu servis; 1080p, 2K, 4K, ve 8K kalitesinde video içeri˘gi sunabilmektedir. E˘glence amaçlı kullanım alanlarında veri iletimindeki gecikme çok önemli olmasa da, otonom araçlarda eMBB servisi hayati rol üstlendi˘gi için çok dü¸sük gecikme ile hizmet vermelidir ve aracın hareketlili˘gi boyunca 5G ba˘glantısında hiçbir kesinti olmamalıdır. Uluslarası mobil ileti¸sim (International Mobile Communications, IMT-2020) raporuna göre, eMBB ve URLLC servislerinin uçtan uca gecikmesinin (end-to-end Delay, e2eDelay) azaltılarak en fazla birkaç milisaniye (a few miliseconds) olacak ¸sekilde tutulması gerekmektedir. Burada, uzak sunucudan hareketli kullanıcıya a¸sa˘gı yönde (downlink) akıtılan trafi˘gin uçtan uca gecikmesi Veri (Data) ve Kontrol (Control) katmanları dikkate alınarak ölçülür. Veri katmanı, Çekirdek (Core) ve Kenar (Edge) a˘glarındaki her bir cihazda gerçekle¸sen kuyrukta bekleme ve i¸sleme süresi olarak modellenmektedir. Kontrol katmanı ise, merkezi kontrolörde hareketlilik prosedürlerinin ko¸sulması için beklenilen i¸slem süresini içerir. Bu tezde; trafik yo˘gunlu˘gu ve hareketlilikten do˘gan problemler dü¸sünülerek, URLLC ve eMBB servisleri için yeni bir uçtan uca gecikme modeli üzerine çalı¸sılmaktadır. Bu nedenle; bu durum kendimize a¸sa˘gıdaki soruyu sormamıza neden olmaktadır: A¸sırı-yo˘gun a˘glarda (Ultra-Dense Networks, UDN) artan trafik heterojenli˘gi ve yo˘gunlu˘gu altında eMBB ve URLLC hizmetleri için uçtan uca gecikme nasıl en aza indirilir?

Bu tezde; eMBB ve URLLC hizmetleri için 5G a˘glarında en aza indirilmi¸s uçtan uca gecikmeyi sunan trafik heterojenli˘gine ve hareketlili˘gine duyarlı bir Yazılım Tanımlı Ultra Yo˘gun A˘g (Software-Defined Ultra-Dense Networks, SDUN) yapısı önerilmektedir. Yukarıda bahsedilen ara¸stırma sorusu, yalnızca önerilen bu SDUN yapısı ile ele alınabilir. Bu yapı topolojiye herhangi bir fiziksel temas gerektirmeden merkezi bir izleme ile dinamik bir ¸sekilde veri katmanını kontrol eder. SDUN, UDN topolojisinde trafik heterojenli˘gini ve a¸sırı trafik yo˘gunlu˘gunu dikkate alan bir yakla¸sım sunmak için kenar ve çekirdek a˘glarını ortak bir ¸sekilde de˘gerlendirir. Bu konu tezde, ¸su alt modüllerde detaylı bir ¸sekilde incelenmektedir: Trafik Heterojenli˘gi Yönetimi (Traffic Heterogeneity Management), Trafik Yo˘gunlu˘gu Yönetimi (Traffic Intensity Management), eMBB servisleri için Devredim Yönetimi (Handover Management for eMBB Services) ve URLLC servisleri için Devredim Yönetimi (Handover Management for URLLC services). Bu alt modüllerin ayrıntılarından a¸sa˘gıdaki paragraflarda sırasıyla bahsedilmektedir.

Trafik heterojenli˘gi yönetimi modülünde; TCP sıkı¸sıklık kontrol mekanizmasındaki kaybolan her paketin zaman a¸sımı için eklenen ekstra hatta çıkma süresi ve bundan dolayı di˘ger paketlerin artan kuyrukta bekleme süreleri nedeniyle, UDP (eMBB) trafi˘ginin TCP (URLLC) trafi˘gini sıkı¸stırdı˘gına inanıyoruz. Bu durum; UDP üzerinde ta¸sınan trafi˘gin bant geni¸sli˘gini di˘ger trafik tipiyle dengeli olmayan bir ¸sekilde TCP’nin kullanması beklenen kısımları da kullanmasından kaynaklanmaktadır. Bu nedenle; a˘gdaki her cihaz içerisinde hizmet verilen UDP ve TCP üzerinde ta¸sınan trafik sayılarının oranı olarak tanımladı˘gımız trafik heterojenli˘gi parametresi önemli hale gelmektedir. 3. Nesil Ortaklık Projesinin (3rd Generation Partnership Project, 3GPP) yayımladı˘gı sürüm 14 teknik özelliklerine göre, geleneksel kendi kendini organize edebilen a˘glarda (Long Term Evolution-Self Organize Networks, LTE-SON) kom¸su baz istasyonları arasındaki trafik yo˘gunlu˘gu dengelenirken heterojenlik oranını dikkate alınmaz. ˙Incelenen TCP trafi˘ginin uçtan uca gecikmesini en aza indirebilmek için, her baz istasyonunun trafik yo˘gunlu˘gu ve trafik heterojenli˘gi parametreleri dikkate alınarak en uygun trafik yolu belirlenmelidir. Bunun için; 5G a˘glarında SDUN tabanlı yazılımla¸stırma yapısı öneriyoruz ve bunu üç temel katkı çerçevesinde sunuyoruz. Bunlar; topoloji grafının sanalla¸stırılması, trafik yo˘gunlu˘gu (ρj) ve trafik heterojenli˘gi

(Hj) parametrelerine göre uçtan uca gecikme (e2eDelay) optimizasyonu, ve yeni

kuyruk teorisi tabanlı OpenFlow a˘g anahtar modeli olarak tanımlanmaktadır. Öte yandan; merkezi SDUN kontrolör yapısının darbo˘gaz sorununu hizmet verme süresini hızlandıracak ¸sekilde çözmek için üç yeni öneri sunmaktayız. Bu öneriler ile uçtan uca gecikme optimizasyon algoritmasını ve en kısa yol algoritmasını parallel ko¸sabilecek ¸sekilde tasarlamı¸s oluyoruz. Bu süreçte uçtan uca gecikme parametresi, Kontrol (Control Plane) ve Veri katmanı (Data Plane) etkisi dü¸sünülerek bir matematiksel kapalı form çerçevesinde yeniden düzenlenmi¸stir. Sonuç olarak; önerilen SDUN tabanlı e2eDelay modeli, LTE-SON ve geleneksel LTE mimarilerine göre %74 ve %98 oranında daha az uçtan uca gecikme ile incelenen TCP akı¸sına hizmet verebilmektedir. Trafik yo˘gunlu˘gu yönetimi modülünde ise; araç takımı a˘glarında (platoon networks) otonom araçların güvenilir veri iletimi ve çok dü¸sük gecikme ihtiyacı oldu˘guna inanıyoruz. Bunlar, URLLC 5G spesifikasyonu altında temel iki ihtiyaç olarak belirtilmektedir. Tezin bu modülünde, URLLC 5G servisi üzerine odaklanılarak uçtan uca gecikmeyi en aza indirmek için Yazılım-Tabanlı Araç Takım A˘gı (Software-Defined Platoon Network, SDPN) tasarlanmı¸stır. Bu SDPN yapısı, araç takım a˘glarında V2V ve V2I ba˘glantılarını göz önüne alarak Karı¸sık Tamsayılı

Do˘grusal Problem (Mixed-Integer Linear Problem, MILP) ile uçtan uca gecikme optimizasyonu tanımlamaktadır. Ancak önerilen bu problemin çözümü polinomsal zamanda çözülemez ve NP (Non-deterministic Polynomial) karma¸sıklıktadır. Bu optimizasyonun en az sürede çözülebilmesi için, problem temel iki alt-seviyeye ayrılmı¸stır: Araç Takımı Olu¸sturma (Platooning) ve Yük Dengeleme (Load Balancing). Seviye 1’de (Level 1), araç takımı olu¸sturmak için araç grafın en uygun küme örtüsünü (set cover) bulan yeni Merkezi Küme Örtüsü algoritması (Centralized Set Cover algorithm) önerilmektedir. Seviye 2’de (Level 2); makro (macrocells) ve küçük hücre (smallcells) katmanları arasındaki trafik yo˘gunlu˘gu geçi¸si, uçtan uca gecikme göz önünde bulundurularak optimize edilmekte ve bunun için yeni Yük Dengeleme algoritması (Load Balancing algorithm) önerilmektedir. SDPN kontrolörünü daha da hızlandırmak için, makro ve küçük hücrelerin yük e¸siklerini (load thresholds, ρm,s) dinamik olarak belirleyen bir analitik metodoloji

tanımlanmı¸stır. Bu sayede, etkin maliyetli bir SDPN yapısı elde edilmi¸stir. Performans de˘gerlendirmesine göre ise, önerilen SDPN yapısı uçtan uca gecikmeyi 3.5 milisaniye altında tutarak geleneksel yapıya göre %45 iyile¸stirme sa˘glamaktadır ve topolojide %70 daha çok trafik yüküne ba¸sarılı bir ¸sekilde hizmet verebilmektedir.

EMBB servisleri için devredim (handover) yönetimi modülünde ise; günümüz devredim prosedürü Sanal Geli¸stirilmi¸s Paket Çekirde˘ginde (virtual Evolvel Paket Core, vEPC) hala üç durumlu olarak gerçeklendi˘gi için, 5G gecikme ihtiyacını istenilen ölçüde kar¸sılayamadı˘gına inanıyoruz. Burada, istenilen uçtan uca gecikme 4 milisaniyenin altında olmalıdır ve eMBB servisinin hareketlili˘gi sırasında herhangi bir ba˘glantı kesintisi ya¸sanmamalıdır. Bu ihtiyaç do˘grultusunda, uçtan uca gecikmenin Markov modeline odaklanılmı¸stır. Bu gecikme, a¸sa˘gı yönde akıtılan eMBB trafi˘ginin kenar (Edge) ve çekirdek (Core) a˘gındaki gecikmeleri dikkate alınarak hesaplanabilir. Burada a¸sırı yo˘gun trafik altında, kenar a˘gına teslim edilebilen paket sayısının azalması nedeniyle kenar gecikmesi çekirdek a˘gından kötü yönde etkilenmektedir. Bu nedenle Hedef Baz ˙Istasyonunu (target eNodeB, TeNB) belirlerken sadece kenar a˘gını dikkate almak uçtan uca gecikme açısından yanlı¸s tercihe neden olabilir. Bu problemi a¸sabilmek için, kenar ve çekirdek gecikmeleri ortak olarak ele alınması önerilmektedir. Bu gecikmeler devredim prosedürünün her a¸samasından farklı ¸sekilde etkilenmektedir. Bu a¸samalar (1) hazırlanma (preparation), (2) gerçekleme (execution), ve (3) sonlandırma (conclusion) olarak adlandırılır. Kenar ve çekirdek a˘gını ortak ¸sekilde en aza indirebilecek ve devredim a¸saması (2)’yi tamamiyle prosedüründen kaldırabilecek tek yapının merkezi bir bakı¸s açısıyla devredim yönetimi sunan yeni bir SDUN yapısı oldu˘guna inanıyoruz. SDUN yapısı; a˘gı merkezi olarak izleme ile devredimin en uygun zamanda tetiklenmesini; ve sonrasında, önerilen optimizasyon formülüne göre en uygun TeNB’nin belirlenmesini ve buna uygun en kısa çekirdek yolunun OpenFlow a˘g anahtarlarının yo˘gunlu˘guna göre çizilmesini sa˘glar. Bu SDUN yapısı, önerilen Paralel Kenar Gecikme Optimizasyonu (Parallel Edge Delay Optimization) ve Paralel En Kısa Gecikme Yolu (Parallel Shortest Delay Path) algoritmaların paralel ko¸sturulabilen yapısı sayesinde uygun maliyetli hale getirilmi¸stir. Performans de˘gerlendirmesinde ise; ilk olarak SDUN QUIC tabanlı HTTP/3 1080p çözünürlü˘gündeki video trafi˘gi akıtan spesifik eMBB uygulaması ile gerçeklenmi¸stir. ˙Ikinci a¸samada ise MATLAB ortamında daha çok OpenFlow a˘g anahtarına sahip topolojiler için sistem seviyesinde simüle edilmi¸stir. Sonuçlara bakıldı˘gında önerilen SDUN yapısı 5G gereksinimlerini ¸su ¸sekilde kar¸sılamaktadır: Kullanıcı ba¸sına önerilen yapı çekirdek a˘gındaki gecikmeyi yakla¸sık 7.16 milisaniye

azaltmı¸stır. Bununla beraber kenar gecikmesini sıfır seviyesi (zero-latency level) altında tutarak geleneksel UDN yapısının devredim yönetimine göre %20 daha fazla paket teslim oranı (delivery ratio) sa˘glamaktadır. Di˘ger bir yandan; SDUN yapısının maliyeti O(k4log2(k)) olarak analiz edilmi¸s ve maliyet verimlili˘ginin kabul edilebilir ekstra %8 sanal hafıza kullanımı yanında %50 arttı˘gı gözlemlenmi¸stir.

URLLC servisleri için devredim yönetimi modülünde ise; uçtan uca hareketlili˘gi ele alabilmek ve araç a˘glarının operasyonel ve sermaye harcamalarını (OPEX/CAPEX) en aza indirgeyebilmek için a˘g servis zincirleri A˘g Fonksiyon Sanalla¸stırması (Network Function Virtualization, NFV) ve vEPC ile gerçeklenmektedir. Ancak; ultra-yo˘gun a˘glarında fazla sayıdaki yol kenarı ünitesi (Road Side Unit, RSU) nedeniyle takım halinde seyir eden araçların liderinin artan devredim iste˘gi SDUN kontrolörünü dar bo˘gaz haline getirmektedir ve kontrolörün lidere verece˘gi cevap süresi artmaktadır. Bu noktada ¸su sorunun cevabını ara¸stırmaktayız: URLLC servisleri için 5G gereksinimi olan uçtan uca gecikme nasıl en aza indirgenebilir? Bu nedenle; merkezi yakla¸sımını bozmadan yeni bir SDUN/NFV kontrolörü tabanlı devredim yönetimi önerilmektedir. Bunun için Küçük Hücre Sanal A˘g Fonksiyonu (Smallcell virtual Network Function, SvNF) ve Makro Hücre Sanal A˘g Fonksiyonu (Macrocell virtual Network Function, MvNF) olmak üzere, yol kenarı ünitesi olarak hizmet veren baz istasyonlarının iki yeni rol tanımı yapılmı¸stır. Bunlardan ilki yol kenarı ünitesinin fiziksel kapsama alanı içerisindeki araca küçük hücre olarak hizmet vermesi, di˘ger ise yol kenarı ünitesinin mantıksal kapsama alanı içerisindeki araca gelen veriyi kom¸su baz istasyonuna yönlendirerek merkezi SDUN kontrolörüne herhangi bir devredim iste˘gi gerektirmeden hizmet vermesidir. Kuyruk teorisiyle modellenen uçtan uca gecikme, önerilen devredim tetikleme algoritması ile SDUN kontrolöründe periyodik olarak izlenmektedir. Bu algoritma, MvNF ve SvNF içeren a˘g hizmet zincir uzunlu˘gunun 5G gereksinimini a¸stı˘gı noktada Devredim A˘g Fonksiyonunu (Handover Network Function) tetikler ve ilgili aracın ba˘glı oldu˘gu yol kenarı ünitesinden kom¸su olana do˘gru devredimini gerçekle¸stirir. Performans de˘gerlendirmesine göre; önerilen SDUN/NFV mimarisi daha az devredim iste˘gi olu¸sturarak ve ölçeklenebilirli˘gini arttırarak SDUN merkezi yapısının korunmasını sa˘glamı¸stır. Önerilen algoritma en fazla 7 a˘g fonksiyonu içeren servis zincirine izin vererek uçtan uca gecikmeyi yakla¸sık 12 milisaniye azaltmı¸stır.

1. INTRODUCTION

(a) Global mobile data traffic in 5G networks.

(b) Identifying mobile data traffic.

Figure 1.1 : Mobile data traffic and identifying it in terms of 5G new services [3]. In the 5th generation (5G) networks, the global mobile data traffic has tremendously increased as shown in Figure 1.1(a). According to the Cisco Visual Networking Index (VNI) report [3], the data traffic generated from mobile devices is expected to reach 49.0 exabytes per month in 2021. This expectation indicates that mobile traffic will grow at a sevenfold increase over data traffic generated in 2016, which results in 47% Compound Annual Growth Rate (CAGR) between 2016-2021. On the other hand; according to the International Organization of Standardization (ISO) 37120 report, the connection number of the mobile user is the main indicator of the quality in smart city

life [13]. To handle it, we need to focus on mobile user demands as in Figure 1.1(b). Here, the traffics generated from new mobile applications are identified such as mobile file sharing, mobile audio, mobile Web/Data/VOIP, and mobile video [3]. These applications are newly called in 5G as Ultra-Reliable Low Latency Communication (URLLC) and an Enhanced Mobile Broadband (eMBB) services [14].

URLLC services can be exemplified with specific applications for fully automated driving in vehicular networks [15]. 5G has METIS and 5GPPP projects related to vehicular applications such as emergency rescue, geographical recommendation, traffic information, traffic flow control, and navigation, etc. More specifically; the vehicle platooning is a new approach that improves fuel efficiency for different road dynamics, enhances traffic efficiency, and ensures safety by controlling the space between vehicles [4]. Thanks to the Vehicle to Infrastructure (V2I) communication over 5G links, the leader of the platoon takes the dynamic rules from the remote control center in network operations cloud (NOC) in downlink manner. It forwards maneuvers data to the followers in a platoon via the Vehicle to Vehicle (V2V) communication over IEEE 802.11p links [16] [17] [18]. Here, this service requires reliable data communication and strict latency.

EMBB services are the next generation of entertainment experience such as Augmented Reality (AR) and Virtual Reality (VR) in 360-degree video streaming for games, education, training and also tomorrows’ services such as autonomous driving. It offers 1080p, 2K, 4K and also 8K video quality. Although entertainments do not need strict latency, the eMBB service for autonomous driving requires very-low latency due to directly having a vital role and it does not desire any interruption on the 5G connection during mobility [19] [20] [21] [22] [23].

While considering the heterogeneity in 5G deployement, we generate Figure 1.2 for eMBB and URLLC services. Here; eMBB and URLLC traffics are carried on User Datagram Protocol (UDP) and Tranmission Control Protocol (TCP); respectively and the details are given in Table 1.1. Nowadays, mobile video is carried by Quick UDP Internet Connections (QUIC) protocol which is defined as multiplexed stream transport over UDP. It is a solution for the challenges of serving video content over TCP based Hyper-Text Transfer Protocol (HTTP) [24] [25] [26] [27]. It removes connection setup and acknowledgment mechanism that results in less delay to end-user. According to

Years

2016 2017 2018 2019 2020 2021

Exabytes per Month

0 10 20 30 40 50 60

UDP : Mobile Video, Audio (68%,83%)

TCP : Mobile File Sharing, Web/Data/VoIP (32%,16%)

47 % CAGR 2016-2021

Figure 1.2 : Identifying mobile data traffic in case of UDP and TCP. Table 1.1 : eMBB and URLLC service comparison [3] [4] [5] [6] [7] [8].

Properties

eMBB

URLLC

Expected Percentage of traffic

83%

16%

Transport layer protocol

UDP

TCP

e2eDelay Requirements

a few milliseconds

1 millisecond

Example Applications

Video Streaming with

Vehicle Platooning

QUIC based HTTP/3

these studies on video content traffic, UDP based traffic occupies 68% of total traffic in 2016 and it is expected to reach 83% in 2021; whereas 16% of total mobile data traffic is generated by TCP as seen in Figure 1.2 [8]. On the light of this traffic heterogeneity, we focus on the 5G requirement of eMBB and URLLC services. According to International Mobile Communications (IMT-2020), they require 5G reduced latency; i.e. keeping end-to-end Delay (e2eDelay) in a few milliseconds [6] [7].

While considering the traffic intensity in 5G, we focus on eMBB characteristics. EMBB flow contains strongly regular and periodic data streams. It has continuous and isochronous communication where data should be delivered within certain time constraints [28] [29]. This service has started to use QUIC based HTTP/3 to reach this delay requirements [5]. This protocol decreases connection setup; and therefore, it reduces the round trip time of a stream. However, it uses more duplicate packets in downlink way to remove the acknowledgment mechanism of previous HTTP versions.

Because of this characteristic; eMBB services cause extra huge traffic intensity in the network than before.

In order to meet the 5G requirement (a few milliseconds e2eDelay) of URLLC and eMBB, small-cells are deployed in 5G networks with efficiently distributed radio resources [30]. This is called an Ultra-Dense Network (UDN). There are many small-cells with small coverage areas and they are very close to end-users where the propagation delay of the edge network is too low. However; while considering the core network of UDN, the signaling overhead has tremendously increased because of multi deployed small cells. This increases traffic intensity and the frequency of handover request in core network [31]. UDN capabilities are limited with a traditional core network that cannot meet this signaling overhead. With an increased 5G wireless deployment, the conventional procedures in backhaul signaling cannot handle 5G reduced latency requirement. As a result, these have led us to investigate the following question: How to reach 5G reduced-e2eDelay for eMBB and URLLC services under huge traffic heterogeneity and intensity in UDN?

In this thesis; we propose a traffic and mobility aware Software-Defined Ultra-Dense Network (SDUN) that offers 5G reduced e2eDelay for eMBB and URLLC services. It reduces Operating Expenses (OPEX) and Capital Expenditures (CAPEX) of 5G deployments by decoupling data and control planes, and by defining Virtual Network Functions (vNFs). Here; we focus on the Markov model of e2eDelay, which can be measured by the concatenation of downlink flow from a remote source to a mobile user. It has components in Edge and Core parts of Data and Control planes in SDUN. However; ultra-dense topology with the high number of base stations increases the number of request to the centralized controller. This causes a bottleneck, which increases the response time to OpenFlow switches in physical topology. To overcome this challenge while keeping a centralized manner of SDUN; we propose parallel runnable algorithms, which require less space and temporal complexities. Throughout this chapter, we’re going to give the details about purpose of the thesis in section 1.1, contribution of the thesis in section 1.2 and the literature review in section 1.3, respectively.

1.1 Purpose of the Thesis

Figure 1.3 : System architecture of the thesis.

In this thesis, we propose three main components for SDUN controller. These are named as Traffic and Mobility management modules, which consider e2eDelay model of the eMBB and URLLC traffics. In Traffic Management, the traffic heterogeneity between eMBB and URLLC services in terms of UDP/TCP ratio, and the change on traffic intensities during macrocells and small cells transitions during mobility path are studied. In Mobility Management, a novel handover procedure is defined for eMBB services, and novel vNFs are defined in order to decrease the handover request of URLLC services to SDUN controller. By considering each module, a novel e2eDelay Markov model is built. The Markov model is one of the most suitable methods to extract analytical equation. Therefore; it is applicable in these sub-modules. The general formula of e2eDelay is proposed as follows:

e2eDelay(D) = Data Plane(Dd) z }| { W_e |{z} Edge + Wc |{z} Core + Control Plane(Dc) z }| { W_s |{z} Controller (1.1)

It has components both for Data and Control planes. The delay in Data plane (Dd)

is calculated as the sum of waiting and processing times of OF switches; whereas the delay in Control plane (Dc) includes the processing time of SDUN controller (Ws).

Moreover; data plane is also separated as Edge and Core networks. Therefore; Dd is

also divided as the Edge Delay (We) and Core Delay (Wc) that are observed delays in

wireless Edge and wired Core, respectively. The details of each module are given in the following subsections:

1.1.1 Traffic management

In this part, we directly studied on UDP based video content traffic; i.e. eMBB service for a mobile user and TCP based URLLC service of platoon networks. In the first part, the negative effect of traffic heterogeneity on URLLC service is detailed; whereas, in the second part, the effect of traffic intensity caused by eMBB service on URLLC traffic of platoon networks is investigated.

1.1.1.1 Traffic heterogeneity management

As seen in Figure 1.2, UDP based eMBB services have much more contents than other types and it is expected to build 83% percentage of global mobile data traffic. This leads most of the studies in the literature to focus on mobile video applications than the others. Under these circumstances, the performance of TCP based URLLC services is directly affected by UDP based eMBB ones. Under ultra-density, we believe that foreground TCP traffic flow is ’squeezed’ by UDP background due to increasing the waiting time in a queue and causing extra transmission delay for each timeout in TCP congestion control mechanism1.

TCP is a reliable packet delivery rather than timely delivery. Therefore, this fact is challenging for TCP based time sensitive URLLC applications [27]. To have less delay on both TCP and UDP flows, there is the main extension of the Internet Engineering Task Force (IETF) on standards for UDP and TCP traffic. For the former one, these are Real-Time Protocol (RTP) and Real-Time Streaming Protocol (RTSP). For the latter one, these are DiffServ, Explicit Congestion Notification (ECN) and improved ECN [27]. However; under ultra-density, it cannot serve traffic flows after certain scalability level because of being only flow and physical based solution. Therefore; we focus on a network-centric solution.

As an addition to this, mobility management in UDN is also a significant challenge due to increased handover frequency between ultra-dense deployed small-cells. This will increase signaling overhead in backhaul [31]. At that point; in order to meet UDN capabilities, wired backhaul should have high-speed and low-delay [32] [33].

1_{Throughout the thesis, the traffic types would be considered as follows:}

Therefore, we study e2eDelay as the performance metric of a flow, which contains both wireless edge and wired backhaul of UDN.

The e2eDelay of a flow generated by end-user is evaluated with propagation delay in links, transmission time in sending packets to a transmission medium, queuing delay that is waiting time in forwarding devices and processing delay that is time passed during the forwarding decisions. The e2eDelay of a TCP flow is measured by considering both the server and client sides. In the server side, it is measured as the time passed from the first packet sent until the acknowledgment of the last packet is received. In the client side, it includes the time passed from the first packet received until ACK of the last packet is sent. As mentioned traffic heterogeneity and mobility challenges in UDN, the e2eDelay of TCP flow is extremely increased whenever TCP timeout occurrence under ultra-dense UDP background.

The current studies do not consider traffic types and intensities of small-cells in traffic and mobility management. Moreover, these approaches cannot serve end-user after a certain scalability level because of not having a global view of the network. UDN networks have requirements such as being dynamic, orchestral, cost-effective, and adaptable [10] network. All of these can be met if the network is programmable and data plane has dummy devices that support open protocol between control and data planes; i.e. OpenFlow protocol. Moreover, thanks to the isolation of data plane complexity to control plane, mobility has become seamless with less control signaling. To the best of our knowledge, all of these requirements can be found only in SDUN softwarization approach brought with 5G networks.

E2eDelay Challenge in 5G:

TCP is a genuine connection-oriented protocol that offers a reliable ordered data transmission. It has embedded congestion and flow control mechanisms. However, these characteristics of TCP flows bring significant challenges for ultra-dense 5G networks. More specifically; when a packet is lost, it can be detected by either 3 duplicated acknowledgment packets (ACKs) or timeout mechanisms. In this case, the communication is interrupted and the receiver side waits for the same TCP packet to be resent. Therefore, the other packets have to wait for resending of the previous packet,

in a queue of the receiver side. This brings extra transmission time on e2eDelay for a packet while waiting in the receiver side.

Under huge traffic intensity, the heterogeneity rate significantly affects the waiting time in transmission of each TCP packets. This is visualized into three sub-graphs of Figure 1.4 and the details are given in Table 1.2. In this figure, there are TCP and UDP traffics generated with the same parameters when packet size is 100 bytes, the packet rate is 500 packet/sec, flow duration is 20 seconds. The traffic flows are generated by same base stations which are called as eNodeBs (eNBs) in LTE. Here, e2eDelay measurements are done by considering only the edge side of the topology. Moreover, maximum buffer size is 1024 bytes2and maximum queue length is 1000 packets3. In the graphs, there is the comparison of two cases named as the Heterogeneous case (1 TCP flow and x UDP flows) and the Homogeneous case (1 TCP flow and x TCP flows) where x is the number of Background traffic flows.

In the first graph of Figure 1.4, the Loss ratio of a TCP flow in the y-axis is observed according to increased background heterogeneous traffic in an ultra-dense 5G network in the x-axis. While the density in the 5G network increases, the Loss ratio of TCP flow becomes higher in both cases because of the congested network. However, in Heterogeneous case, the Loss ratio becomes extremely higher than Homogeneous case under ultra-density. The main reason for this behavior is that UDP has not congestion control scheme as in TCP. Because of not having a congestion control mechanism in UDP, after a packet loss, the sender still continues sending packets and fills bandwidth [34]. For that reason, the congestion occurs in the end. This brings more loss packets for a TCP flow. Namely, UDP flows squeeze a TCP flow more than TCP flows do, therefore more loss packets occur due to this congestion in ultra-dense 5G networks as seen in Figure 1.4.

In the second detailed part of Figure 1.4, the effect of loss packets of a TCP flow on e2eDelay is observed in the y-axes according to simulation time shown in the x-axes. Due to changing congestion window size dynamically during congestion period, the e2eDelay fluctuates in the TCP flow. As mentioned in the first graph of Figure 1.4, there are more loss packets of a TCP flow, in the heterogeneous case than

2_{default parameter in node.py of Mininet}©2018 3_{default parameter in link.py of Mininet}©2018

the homogeneous case. Due to many loss packets and forcing other packets to wait in a buffer of the receiver side, the e2eDelay of a TCP flow reaches higher levels in the heterogeneous case than in homogeneous case as seen in the second detailed part of Figure 1.4. Therefore; due to not having a congestion control mechanism, the UDP flows ’squeeze’ TCP flows and negatively affects them in terms of e2eDelay by bringing about increased waiting time in transmission.

In such scenarios mentioned above, conventional LTE architecture tries to reach less e2eDelay via threshold-based load balancing approach between only neighbor eNBs. However, this enhancement on e2eDelay is weak because of not considering heterogeneity in traffic flows (Hj). According to 3GPP Release 14 [35], the

conventional LTE architecture has the self-organizing capability in order to balance traffic load to enhance e2eDelay of end-user. Nevertheless; in threshold-based load balancing, heterogeneity rate (Hj) of traffic flows in eNBs are not considered. Despite

load is balanced between neighbor eNBs, the e2eDelay of end-users can be higher than the situation before suitable handovers are done. If the target eNB in which end-user handoffs has higher UDP flows than source one has, the e2eDelay of end-user becomes worse despite expecting better e2eDelay. Therefore, in order to decrease e2eDelay, new forwarding path should be selected considering both load intensity and heterogeneity of traffic flows in eNBs.

As a result, traffic type heterogeneity (Hj) in ultra-dense 5G networks has a significant

effect on e2eDelay of a TCP flow due to increasing queue Waiting Time (Wj) and

bringing extra Transmission Time (WT) whenever timeout occurrence. In order

to overcome this challenge, intelligent network management should be deployed that takes new forwarding decisions by considering both load intensity (ρj) and

0 50 100 150 200 Simulation Time (t) 0 0.05 0.1 0.15 e2eDelay (sec)

Homogeneous: 1 TCP and 64 TCP flows

0 50 100 150 200 Simulation Time(t) 0 0.05 0.1 0.15 e2eDelay (sec)

Heterogeneous: 1 TCP and 64 UDP flows

14 8 16 32 64 128 256

Number of Background traffic flows

0 25 50 75 100 Loss ratio (%)

Homogeneous: 1 TCP and x TCP flows Heterogeneous: 1 TCP and x UDP flows

Figure 1.4 : Loss ratio (%) & e2eDelay per packet (sec) of a TCP flow under increased different background traffic in ultra-dense 5G network. Table 1.2 : e2eDelay(sec) and loss details.

#Background Flows 1 2 4 8 16 32 64 128 256 Sent Packets 10000 10000 10000 10000 10000 10000 10000 10000 10000 Homogeneous e2eDelay 0,00278 0,002757 0,002728 0,002736 0,002734 0,002854 0,046041 0,130745 0,136596 case Loss 361 349 351 317 348 310 1833 5143 5428 Heterogeneous e2eDelay 0,002746 0,00276 0,00275 0,002724 0,00279 0,002856 0,094695 0,153451 0,609788 case Loss 359 341 318 321 313 287 4854 6572 7214

1.1.1.2 Traffic intensity management

According to the European Commission; the carbon emission caused by fuel consumption in transportation, has been aimed to decrease 60% level by 2050 [36]. This has led us to investigate a new Intelligent Transport Systems (ITS) with higher fuel efficiency in highways with an approach: Platooning. In a platoon, there is one leader and there are also followers just behind it. Here; thanks to the V2I link, the leader takes the dynamic rules for traffic flow control and accident data from the remote control center in virtual Evolved Packet Core (vEPC). By V2V link, the leader forwards these rules to the following vehicles [16] [17] [18]. Therefore; it decreases fuel consumption in different road dynamics, enhances traffic efficiency, and ensures safety by controlling the space between vehicles [4]. Such a fully automated driving in a platoon requires reliable data transfer and strict latency in downlink traffic flow during mobility. Here; this flow is called URLLC in 5G services [15].

before after

Approach AdHoc Centralized

Communication Technology DSRC, IEEE 802.11p 4G+5G, OF with SDN Traffic Type Reliable Data Transfer URLLC

Traffic Load in Core Moderate High

Figure 1.5 : Comparison of vehicle platooning approaches.

According to IMT-2020, URLLC services require radio-latency under 1 msec and e2eDelay in a few msecs [6] [7]. E2eDelay is measured by the concatenation of V2V and V2I communications in the downlink URLLC service from a remote source to a vehicle. Therefore; by considering both edge and core in the platoon networks, we study e2eDelay in two parts as V2V and V2I.

In V2V part; Figure 1.5 exemplifies two proposed approaches for platooning. Before; there has been an AdHoc approach that vehicles communicate with each other by Dedicated Short-Range Communication (DSRC) over IEEE 802.11p. Here; a vehicle platoon has built locally according to the vehicle-centric decision. However; there have been such challenges as frequency reuse and interference between vehicles during the V2V communication and platoon building. In a platoon network, these challenges cause many packet retransmissions and this negatively affects e2eDelay of URLLC service. Therefore, the platoon should be built as long as possible by decreasing the number of independent vehicles exemplified as V3 in the figure. However; the size of the platoon cannot increase after a certain level because of using AdHoc approach. To overcome these problems, the centralized orchestration of platooning and control of the vehicles are required. Thanks to the global view of the centralized controller, an optimal platoon can be built that offers the following advantages. The fuel efficiency is increased for the whole vehicular network. The frequencies per vehicle can be assigned to them beforehand from a centralized pool. Therefore, the data transfer would become reliable without any packet retransmission in a platoon [37] [38]. As a result; we need a centralized approach to build optimum platoon that reaches these 5G requirements for URLLC services.

In V2I part; the centralized approach is only seen in 3% of the recent studies that try to solve such platooning challenges in the literature. The main reason for it is the lack of V2I technology investments in 5G [39]. Therefore; to keep the advantages of the centralized approach, we investigate also the V2I part of URLLC services in the platoon networks. In V2I part; the load in vEPC has increased because there is huge traffic intensity on core network [3] and the whole signaling of the platoon is now routed over V2I as mentioned in Figure 1.5. The traffic load directly affects e2eDelay of URLLC service due to increasing the queuing and processing delay in the infrastructure formed by small cells and macrocell tiers. Therefore; this leads us to dynamic load balancing between them. In the literature, there are many studies that try to enhance e2eDelay by considering traffic intensity in a network as given in section 1.3. However; because of having local and cell-centric decisions [40], they cannot serve URLLC traffic after certain load levels. To reduce e2eDelay, this part also needs

centralized orchestration that dynamically balances traffic intensity between small cell and macrocell tiers for V2I in the core network.

1.1.2 Mobility management

In this part, we study the effect of handover procedure on eMBB and URLLC services. In the first part, the Xn-handover procedure of 5G is performed by considering states and communication process between eNBs. It is evaluated for eMBB services. In the second part, the Xn-handover procedure is considered as a black box and we focus on the number of handover requests to the centralized controller for URLLC services in platoon networks.

1.1.2.1 Handover management for eMBB services

Video has been evolved from HD to FullHD with the higher requirements on delay and service quality. In Table 1.3, the comparison of MBB and eMBB in terms of mobility requirements are given. EMBB is the next generation of entertainment experiences such as AR and VR in 360-degree video streaming for games, education, training and also tomorrows’ services such as autonomous driving. It offers 1080p, 2K, 4K and also 8K video quality; whereas MBB can only handle resolution up to 720p.

To provide "always on" transmission and seamless mobility, we focus on endless connection to 5G new service: eMBB traffic. According to IMT-2020, it needs reduced-latency requirement in parallel to high data rate; i.e. keeping e2eDelay in a few milliseconds [6]. Although entertainments do not need strict e2eDelay, there is next generation eMBB application that requires very-low latency due to directly having a vital role [41] [42] [43]. For example; in Feb. 2018, Ericsson and Verizon have a "blackout" VR test in a vehicle at historic Indianapolis Motor Speedway. This test executed by a driver having a set of VR glasses that take 360 degrees 4K video captured from a camera on the car. Here; eMBB service for autonomous driving does not desire any interruption on the 5G connection during mobility. Therefore; as given in the Table 1.3, the mobility interruption time should be under 4 msecs for a safe driving whereas it could be reached up to 50 msecs in conventional MBB.

An eMBB traffic should execute Xn-handover to keep 5G communication alive during cell changes in UDN. However; a high number of 5G wireless deployments in UDN

Table 1.3 : MBB/eMBB comparison in terms of mobility.

Next Generation Core 4G, EPC 4G+5G, vEPC

MBB eMBB

Mobility Interruption Time 50 msecs 4 msecs (Desired) Supported Video Resolution <720p 1080p,2K, 4K ,8K

HD FullHD

Runnable Applications Video Streaming

Augmented Reality, Virtual Reality, Autonomous Driving

increases handover frequency and this directly damage e2eDelay requirement [44]. We believe that Xn-handover execution still damages e2eDelay of eMBB services due to the following reasons:

• Huge traffic intensity caused by eMBB, • Core effect on Edge network for eMBB flow, • Edge centric handover triggering,

• The three-state conventional Xn-handover procedure.

EMBB flow contains strongly regular and periodic data streams. It has continuous and isochronous communication where data should be delivered within certain time constraints [28] [29]. Previously; low latency video streaming carried by HTTP protocol, whereas eMBB services have started to use QUIC protocol based on HTTP/3 to satisfy delay requirements [5]. This protocol decreases connection setup; and therefore, reduces the round trip time of the stream. However, it uses more duplicate packets in downlink way to remove the acknowledgment mechanism of previous HTTP versions. Due to this characteristic; eMBB services cause huge traffic intensity in the core than before. Here, the edge is directly affected by the core. Under huge traffic intensity in downlink eMBB traffic from the core to edge; the number of packets reaching to edge is decreased, which results in less Edge Delay (De). At that

point, edge-centric handover can result in a misleading target base station decision. Therefore, to minimize e2eDelay, joint consideration of edge and core networks according to traffic intensity is needed. Moreover; under extra traffic intensity in the core, the mobility interruption cannot be handled by three-state Xn-handover

Figure 1.6 : Problematic situation of e2eDelay in 5G handover.

procedure that does not jointly consider edge and core. This challenge is detailed in following sub-section.

Three-State Handover Challenge

As in Figure 1.6, to keep communication alive along a mobility path, a UE executes Xn-handover procedure in three states [45] [35] [2]: Handover preparation, execution, and completion, which are labeled as (1), (2), and (3) respectively. During Xn-handover; downlink traffic from remote source (UEs) to destination (UEd),

traverses such links: Serving Gateway (SGW), Core switches, Source eNodeB (SeNB), and Target eNodeB (TeNB). Here; e2eDelay is measured by considering this path in both Edge and Core networks. It has the following components: I. Queuing and processing delay per device in Edge and Core, II. Mobility interruption time during handover execution in Core. Here, each handover states differently affects e2eDelay in terms of Edge and Core Delays.

In Edge; TeNB is determined by Edge network according to measurement report which includes received higher-signal strength by UE. However, it can be the worst one after path switching procedure in the state (3) because it may offer extra waiting time in a queue caused by having huge traffic intensity for eMBB services [46]. In Core; downlink traffic routed over SeNB, is buffered in TeNB queue to not lose them after handover execution. This buffer is released after taking downlink data delivery status messages from SeNB and TeNB. During the path switch procedure, UE starts to take downlink data after radio resource control (RRC) connection is completed in TeNB.

However, this buffering brings extra waiting time called as mobility interruption time [11]. To overcome these challenges, the state (2) should be removed by performing control signaling between TeNB and UE before state (3). As a result; there is no need for buffering between SeNB and TeNB, which extremely damages e2eDelay.

On the light of aforementioned challenges, we need a framework for eMBB services that jointly considers traffic intensity in edge and core network for a seamless handover procedure by removing state (2).

1.1.2.2 Handover management for URLLC services

Figure 1.7 : An example network architecture of platoon vehicles.

URLLC services can be exemplified with specific applications in vehicular networks [15]. 5G has METIS and 5GPPP projects related to the vehicular applications such as emergency rescue, geographical recommendation, traffic information, traffic flow control and navigation etc. As shown in Figure 1.7, the vehicle platooning is one of 5G services in vehicular networks. The platoon consists of a platoon leader and the followers behind it. There may be also another vehicle, which is not in the platoon, called as an independent vehicle. Thanks to the V2I over 5G links between Road Side Unit (RSU) and vehicle, the platoon leader takes traffic flow control and accident data from the remote control center, where the network operations are executed in vEPC. Thanks to the V2V communication over IEEE 802.11p links, it forwards acceleration and maneuvers data to the platoon followers [16]. Therefore; it improves fuel efficiency for different road dynamics, enhances traffic efficiency, and ensures safety by controling the space between vehicles in platoon [4].

URLLC of 5G scenarios require reliable data transmission for a remote vehicle control, autonomous driving; and here, the radio latency should be less than 1 millisecond.