Welcome to kubernetes the microservices world for application deployment
Tested On: Ubuntu 16.04
Prerequisite:-
1. Any Hypervisor like (kvm,virtaulbox,hyperv)
Note: I am using Virtualbox (you can install virtualbox as per your own method)
2. Installing minikube # The single node kubernetes cluster manager
Type this command in bash shell:
root@machine:~# curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.8.0/minikube-linux-amd64
root@machine:~# mv minikube /usr/bin/
root@machine:~# chmod +x /usr/bin/minikube
3. Installing kubectl # kubernetes client
root@machine:~# curl -Lo kubectl http://storage.googleapis.com/kubernetes-release/release/v1.3.0/bin/linux/amd64/kubectl
root@machine:~# mv kubectl /usr/bin/
root@machine:~# chmod +x /usr/bin/kubectl
4. Starting minikube # that will download virtualmachine and start it
root@machine:~# minikube start
Checking vm status :
root@machine:~# minikube status
Running
Checking kubernetes client :
root@machine:~# kubectl get nodes
NAME STATUS AGE
minikubevm Ready 15m
Few more commands for minikube :-
1. accessing vm with ssh
root@machine:~# minikube ssh
## .
## ## ## ==
## ## ## ## ## ===
/"""""""""""""""""\___/ ===
~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ / ===- ~~~
\______ o __/
\ \ __/
\____\_______/
_ _ ____ _ _
| |__ ___ ___ | |_|___ \ __| | ___ ___| | _____ _ __
| '_ \ / _ \ / _ \| __| __) / _` |/ _ \ / __| |/ / _ \ '__|
| |_) | (_) | (_) | |_ / __/ (_| | (_) | (__| < __/ |
|_.__/ \___/ \___/ \__|_____\__,_|\___/ \___|_|\_\___|_|
Boot2Docker version 1.11.1, build master : 9b97e11 - Fri Jul 15 19:18:04 UTC 2016
Docker version 1.11.1, build 5604cbe
docker@minikubeVM:~$
Time for creating a sample application using YAML :
1. Sample yaml file
root@machine:/opt/apps/yamls# cat sample.yml ---
apiVersion: v1
kind: Pod
metadata:
name: adhocnetworks
labels:
app: webtest
spec:
containers:
- name: web1
image: nginx
ports:
- containerPort: 80
imagePullPolicy: IfNotPresent
2. checking pods
root@machine:/opt/apps/yamls# kubectl get pods
root@machine:/opt/apps/yamls#
3. Creating pods
root@machine:/opt/apps/yamls# kubectl create -f sample.yml pod "adhocnetworks" created
root@machine:/opt/apps/yamls# kubectl get podsNAME READY STATUS RESTARTS AGE
adhocnetworks 1/1 Running 0 6s
4. Describing pods
root@machine:/opt/apps/yamls# kubectl describe pods adhocnetworksName: adhocnetworks
Namespace: default
Node: minikubevm/10.0.2.15
Start Time: Sun, 23 Sep 2018 17:35:50 +0530
Labels: app=webtest
Status: Running
IP: 172.17.0.3
Controllers: <none>
5. Deleting pods
root@machine:/opt/apps/yamls# kubectl delete pods adhocnetworks
pod "adhocnetworks" deleted
root@machine:/opt/apps/yamls# kubectl get pods
Important: In pod type YAML file if we want to scale up pods then we have to make a new copy of same YAML file with different pod name , that is not correct practicse at all.
Still we are creating new name pod file.
File 1 : root@machine:/opt/apps/yamls# cat nginx_podtype.yml
---
apiVersion: v1
kind: Pod
metadata:
name: adhocpod2 labels: # this is for using in service section for end point creation
app: web
spec:
containers:
- name: front-end
image: nginx
ports:
- containerPort: 80
imagePullPolicy: IfNotPresent
File 2: root@machine:/opt/apps/yamls# cat nginx_podtype1.yml
---
apiVersion: v1
kind: Pod
metadata:
name: adhocpod1
labels:
app: web
spec:
containers:
- name: front-end
image: nginx
ports:
- containerPort: 80
imagePullPolicy: IfNotPresent
Now creating both the pods with kubectl :
root@machine:/opt/apps/yamls# kubectl create -f nginx_podtype.yml
root@machine:/opt/apps/yamls# kubectl create -f nginx_podtype1.yml
Checking pods :
root@machine:/opt/apps/yamls# kubectl get pods
NAME READY STATUS RESTARTS AGE
adhocpod1 1/1 Running 0 16m
adhocpod2 1/1 Running 0 16m
Note: This is not a final solution for scalling
POD summary :-
Because we are running two pods where nginx server is running now there are two big problems :
1. How to make this accessible via URL for External world
2. How to Load balance them
Here Kubernetes provides the Service resource type :-
we can slove above two problems by using service resource type
How kubernetes apply Load balancing ..??
main idea behind this is to use label
we can do this is in two steps :-
i) Apply lables to all the pods where we want to apply service
ii) Apply a "selector" to our service so that defined which labeled pods to target
Label:-
1. Are the entity that provide simple method for organizing kubernetes resources
2. These are key-value pair and can be applied with any resources
Creating service file :-
that will create a service for load balancing :-
root@machine:/opt/apps/yamls# cat nginx_lb.yml apiVersion: v1
kind: Service # service 1
metadata:
name: adhocpodlb # load of balancer name
spec:
type: LoadBalancer
ports:
- port: 81 # client target
protocol: TCP #
targetPort: 80 # container port number
selector:
app: web # label from pod
Running service Module :
root@machine:/opt/apps/yamls# kubectl create -f nginx_lb.yml
Checking service :-
root@machine:/opt/apps/yamls# kubectl get serviceNAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
adhocpodlb 10.0.0.228 <pending> 81/TCP 35m
kubernetes 10.0.0.1 <none> 443/TCP 20h
Important; Due to minikube external ip will not be apprear but if we use any public cloud like GCP , AZURE or AWS .
Minikube :- provide us service which will provide us external IP that will forward request to any of the pod (it doesn't) matter
root@machine:/opt/apps/yamls# minikube service adhocpodlb
It will give you external to access you app
Now here here we are going to define Deployments :
Important: both pod and deployment can create pods but for production deployment is much better and we have number of options to explore like replication of pods
Sample deployment:
root@machine:/opt/apps/yamls# cat nginx_deployment.yml ---
apiVersion: extensions/v1beta1 # here we are not using v1 like pods
kind: Deployment
metadata:
name: adhoc-sitedep
spec:
replicas: 2
template:
metadata:
labels:
app: webappsdep # for calling in service file
spec:
containers:
- name: front-end
image: nginx
ports:
- containerPort: 80
imagePullPolicy: IfNotPresent
Another deployment file with zero downtime:
root@machine:/opt/apps/yamls# cat nginx_deployment.yml
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: adhoc-sitedep
spec:
replicas: 2
minReadySeconds: 15
strategy:
type: RollingUpdate # ensure zero downtime deployments when moving from current version to next version
rollingUpdate:
maxUnavailable: 1 # only 1 pod is allowed to terminate during version update
maxSurge: 1 # when update to new version only 1 pad can be added
template:
metadata:
labels:
app: webappsdep
spec:
containers:
- name: front-end
image: nginx
ports:
- containerPort: 80
imagePullPolicy: IfNotPresent
Note: Rest all the commands are same
kubectl create -f nginx_deployment.yml
kubectl get pods
kubectl get deployments
kubectl describe deployments adhoc-sitedep
kubectl get deployments
kubectl delete deployments adhoc-sitedep
Enjoy the basic of kubernets with minikube and kubectl
Creation of service that is very much similar to individual :
THis is URL provide and Load balancing:
root@machine:/opt/apps/yamls# cat nginx_lb.yml apiVersion: v1
kind: Service # service 1
metadata:
name: adhocpodlb # load of balancer name
spec:
type: LoadBalancer
ports:
- port: 81 # client target
protocol: TCP #
targetPort: 80 # container port number
selector:
app: webdeponelabel # label from pod
# To Open web browser for portal based deployment
root@machine:/opt/apps/yamls# minikube service adhocpodlb
Opening kubernetes service default/adhocpodlb in default browser...
Running Firefox as root in a regular user's session is not supported. ($XAUTHORITY is /home/google/.Xauthority which is owned by google.)
[13689:13689:0926/011850.663370:ERROR:zygote_host_impl_linux.cc(89)] Running as root without --no-sandbox is not supported. See https://crbug.com/638180.
Running Firefox as root in a regular user's session is not supported. ($XAUTHORITY is /home/google/.Xauthority which is owned by google.)
/usr/bin/xdg-open: 778: /usr/bin/xdg-open: iceweasel: not found
/usr/bin/xdg-open: 778: /usr/bin/xdg-open: seamonkey: not found
/usr/bin/xdg-open: 778: /usr/bin/xdg-open: mozilla: not found
/usr/bin/xdg-open: 778: /usr/bin/xdg-open: epiphany: not found
/usr/bin/xdg-open: 778: /usr/bin/xdg-open: konqueror: not found
/usr/bin/xdg-open: 778: /usr/bin/xdg-open: chromium-browser: not found
[13717:13717:0926/011850.728671:ERROR:zygote_host_impl_linux.cc(89)] Running as root without --no-sandbox is not supported. See https://crbug.com/638180.
/usr/bin/xdg-open: 778: /usr/bin/xdg-open: www-browser: not found
/usr/bin/xdg-open: 778: /usr/bin/xdg-open: links2: not found
/usr/bin/xdg-open: 778: /usr/bin/xdg-open: elinks: not found
/usr/bin/xdg-open: 778: /usr/bin/xdg-open: links: not found
/usr/bin/xdg-open: 778: /usr/bin/xdg-open: lynx: not found
/usr/bin/xdg-open: 778: /usr/bin/xdg-open: w3m: not found
xdg-open: no method available for opening 'http://192.168.99.100:32576'
Important: sometimes developers have made some changes in their code and created a new docker image .
That means we can assume that this is new version of the same application so we can deployment will zero down time
Note: Here i have updated new nginx image
root@machine:/opt/apps/yamls# cat nginx_deployment.yml ---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: adhoc-sitedep
spec:
replicas: 2
minReadySeconds: 15
strategy:
type: RollingUpdate # ensure zero downtime deployments when moving from current version to next version
rollingUpdate:
maxUnavailable: 1 # only 1 pod is allowed to terminate during version update
maxSurge: 1 # when update to new version only 1 pad can be added
template:
metadata:
labels:
app: webappsdep
spec:
containers:
- name: front-end
image: nginx:v1 ports:
- containerPort: 80
imagePullPolicy: IfNotPresent
Deploying new version of application :
root@machine:/opt/apps/yamls# kubectl apply -f nginx_deployment.yml --record
deployment "adhoc-sitedep" configured
Checking status :
root@machine:/opt/apps/yamls# kubectl rollout status deployment adhoc-sitedep
deployment adhoc-sitedep successfully rolled out
root@machine:/opt/apps/yamls#
Time for rolling back to previous version :
there are number of reason where we want to restore to its previous version:
root@machine:/opt/apps/yamls# kubectl rollout history deployment adhoc-sitedep
deployments "adhoc-sitedep":
REVISION CHANGE-CAUSE
1 <none>
2 kubectl apply -f nginx_deployment.yml --record
root@machine:/opt/apps/yamls# kubectl rollout undo deployment adhoc-sitedep --to-revision=1
deployment "adhoc-sitedep" rolled back
root@machine:/opt/apps/yamls# kubectl rollout history deployment adhoc-sitedep
deployments "adhoc-sitedep":
REVISION CHANGE-CAUSE
2 kubectl apply -f nginx_deployment.yml --record
3 <none>
Tested On: Ubuntu 16.04
Prerequisite:-
1. Any Hypervisor like (kvm,virtaulbox,hyperv)
Note: I am using Virtualbox (you can install virtualbox as per your own method)
2. Installing minikube # The single node kubernetes cluster manager
Type this command in bash shell:
root@machine:~# curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.8.0/minikube-linux-amd64
root@machine:~# mv minikube /usr/bin/
root@machine:~# chmod +x /usr/bin/minikube
3. Installing kubectl # kubernetes client
root@machine:~# curl -Lo kubectl http://storage.googleapis.com/kubernetes-release/release/v1.3.0/bin/linux/amd64/kubectl
root@machine:~# mv kubectl /usr/bin/
root@machine:~# chmod +x /usr/bin/kubectl
4. Starting minikube # that will download virtualmachine and start it
root@machine:~# minikube start
Checking vm status :
root@machine:~# minikube status
Running
Checking kubernetes client :
root@machine:~# kubectl get nodes
NAME STATUS AGE
minikubevm Ready 15m
Few more commands for minikube :-
1. accessing vm with ssh
root@machine:~# minikube ssh
## .
## ## ## ==
## ## ## ## ## ===
/"""""""""""""""""\___/ ===
~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ / ===- ~~~
\______ o __/
\ \ __/
\____\_______/
_ _ ____ _ _
| |__ ___ ___ | |_|___ \ __| | ___ ___| | _____ _ __
| '_ \ / _ \ / _ \| __| __) / _` |/ _ \ / __| |/ / _ \ '__|
| |_) | (_) | (_) | |_ / __/ (_| | (_) | (__| < __/ |
|_.__/ \___/ \___/ \__|_____\__,_|\___/ \___|_|\_\___|_|
Boot2Docker version 1.11.1, build master : 9b97e11 - Fri Jul 15 19:18:04 UTC 2016
Docker version 1.11.1, build 5604cbe
docker@minikubeVM:~$
Time for creating a sample application using YAML :
1. Sample yaml file
root@machine:/opt/apps/yamls# cat sample.yml ---
apiVersion: v1
kind: Pod
metadata:
name: adhocnetworks
labels:
app: webtest
spec:
containers:
- name: web1
image: nginx
ports:
- containerPort: 80
imagePullPolicy: IfNotPresent
2. checking pods
root@machine:/opt/apps/yamls# kubectl get pods
root@machine:/opt/apps/yamls#
3. Creating pods
root@machine:/opt/apps/yamls# kubectl create -f sample.yml pod "adhocnetworks" created
root@machine:/opt/apps/yamls# kubectl get podsNAME READY STATUS RESTARTS AGE
adhocnetworks 1/1 Running 0 6s
4. Describing pods
root@machine:/opt/apps/yamls# kubectl describe pods adhocnetworksName: adhocnetworks
Namespace: default
Node: minikubevm/10.0.2.15
Start Time: Sun, 23 Sep 2018 17:35:50 +0530
Labels: app=webtest
Status: Running
IP: 172.17.0.3
Controllers: <none>
5. Deleting pods
root@machine:/opt/apps/yamls# kubectl delete pods adhocnetworks
pod "adhocnetworks" deleted
root@machine:/opt/apps/yamls# kubectl get pods
Important: In pod type YAML file if we want to scale up pods then we have to make a new copy of same YAML file with different pod name , that is not correct practicse at all.
Still we are creating new name pod file.
File 1 : root@machine:/opt/apps/yamls# cat nginx_podtype.yml
---
apiVersion: v1
kind: Pod
metadata:
name: adhocpod2 labels: # this is for using in service section for end point creation
app: web
spec:
containers:
- name: front-end
image: nginx
ports:
- containerPort: 80
imagePullPolicy: IfNotPresent
File 2: root@machine:/opt/apps/yamls# cat nginx_podtype1.yml
---
apiVersion: v1
kind: Pod
metadata:
name: adhocpod1
labels:
app: web
spec:
containers:
- name: front-end
image: nginx
ports:
- containerPort: 80
imagePullPolicy: IfNotPresent
Now creating both the pods with kubectl :
root@machine:/opt/apps/yamls# kubectl create -f nginx_podtype.yml
root@machine:/opt/apps/yamls# kubectl create -f nginx_podtype1.yml
Checking pods :
root@machine:/opt/apps/yamls# kubectl get pods
NAME READY STATUS RESTARTS AGE
adhocpod1 1/1 Running 0 16m
adhocpod2 1/1 Running 0 16m
Note: This is not a final solution for scalling
POD summary :-
Because we are running two pods where nginx server is running now there are two big problems :
1. How to make this accessible via URL for External world
2. How to Load balance them
Here Kubernetes provides the Service resource type :-
we can slove above two problems by using service resource type
How kubernetes apply Load balancing ..??
main idea behind this is to use label
we can do this is in two steps :-
i) Apply lables to all the pods where we want to apply service
ii) Apply a "selector" to our service so that defined which labeled pods to target
Label:-
1. Are the entity that provide simple method for organizing kubernetes resources
2. These are key-value pair and can be applied with any resources
Creating service file :-
that will create a service for load balancing :-
root@machine:/opt/apps/yamls# cat nginx_lb.yml apiVersion: v1
kind: Service # service 1
metadata:
name: adhocpodlb # load of balancer name
spec:
type: LoadBalancer
ports:
- port: 81 # client target
protocol: TCP #
targetPort: 80 # container port number
selector:
app: web # label from pod
Running service Module :
root@machine:/opt/apps/yamls# kubectl create -f nginx_lb.yml
Checking service :-
root@machine:/opt/apps/yamls# kubectl get serviceNAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
adhocpodlb 10.0.0.228 <pending> 81/TCP 35m
kubernetes 10.0.0.1 <none> 443/TCP 20h
Important; Due to minikube external ip will not be apprear but if we use any public cloud like GCP , AZURE or AWS .
Minikube :- provide us service which will provide us external IP that will forward request to any of the pod (it doesn't) matter
root@machine:/opt/apps/yamls# minikube service adhocpodlb
It will give you external to access you app
Now here here we are going to define Deployments :
Important: both pod and deployment can create pods but for production deployment is much better and we have number of options to explore like replication of pods
Sample deployment:
root@machine:/opt/apps/yamls# cat nginx_deployment.yml ---
apiVersion: extensions/v1beta1 # here we are not using v1 like pods
kind: Deployment
metadata:
name: adhoc-sitedep
spec:
replicas: 2
template:
metadata:
labels:
app: webappsdep # for calling in service file
spec:
containers:
- name: front-end
image: nginx
ports:
- containerPort: 80
imagePullPolicy: IfNotPresent
Another deployment file with zero downtime:
root@machine:/opt/apps/yamls# cat nginx_deployment.yml
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: adhoc-sitedep
spec:
replicas: 2
minReadySeconds: 15
strategy:
type: RollingUpdate # ensure zero downtime deployments when moving from current version to next version
rollingUpdate:
maxUnavailable: 1 # only 1 pod is allowed to terminate during version update
maxSurge: 1 # when update to new version only 1 pad can be added
template:
metadata:
labels:
app: webappsdep
spec:
containers:
- name: front-end
image: nginx
ports:
- containerPort: 80
imagePullPolicy: IfNotPresent
Note: Rest all the commands are same
kubectl create -f nginx_deployment.yml
kubectl get pods
kubectl get deployments
kubectl describe deployments adhoc-sitedep
kubectl get deployments
kubectl delete deployments adhoc-sitedep
Enjoy the basic of kubernets with minikube and kubectl
Creation of service that is very much similar to individual :
THis is URL provide and Load balancing:
root@machine:/opt/apps/yamls# cat nginx_lb.yml apiVersion: v1
kind: Service # service 1
metadata:
name: adhocpodlb # load of balancer name
spec:
type: LoadBalancer
ports:
- port: 81 # client target
protocol: TCP #
targetPort: 80 # container port number
selector:
app: webdeponelabel # label from pod
# To Open web browser for portal based deployment
root@machine:/opt/apps/yamls# minikube service adhocpodlb
Opening kubernetes service default/adhocpodlb in default browser...
Running Firefox as root in a regular user's session is not supported. ($XAUTHORITY is /home/google/.Xauthority which is owned by google.)
[13689:13689:0926/011850.663370:ERROR:zygote_host_impl_linux.cc(89)] Running as root without --no-sandbox is not supported. See https://crbug.com/638180.
Running Firefox as root in a regular user's session is not supported. ($XAUTHORITY is /home/google/.Xauthority which is owned by google.)
/usr/bin/xdg-open: 778: /usr/bin/xdg-open: iceweasel: not found
/usr/bin/xdg-open: 778: /usr/bin/xdg-open: seamonkey: not found
/usr/bin/xdg-open: 778: /usr/bin/xdg-open: mozilla: not found
/usr/bin/xdg-open: 778: /usr/bin/xdg-open: epiphany: not found
/usr/bin/xdg-open: 778: /usr/bin/xdg-open: konqueror: not found
/usr/bin/xdg-open: 778: /usr/bin/xdg-open: chromium-browser: not found
[13717:13717:0926/011850.728671:ERROR:zygote_host_impl_linux.cc(89)] Running as root without --no-sandbox is not supported. See https://crbug.com/638180.
/usr/bin/xdg-open: 778: /usr/bin/xdg-open: www-browser: not found
/usr/bin/xdg-open: 778: /usr/bin/xdg-open: links2: not found
/usr/bin/xdg-open: 778: /usr/bin/xdg-open: elinks: not found
/usr/bin/xdg-open: 778: /usr/bin/xdg-open: links: not found
/usr/bin/xdg-open: 778: /usr/bin/xdg-open: lynx: not found
/usr/bin/xdg-open: 778: /usr/bin/xdg-open: w3m: not found
xdg-open: no method available for opening 'http://192.168.99.100:32576'
Important: sometimes developers have made some changes in their code and created a new docker image .
That means we can assume that this is new version of the same application so we can deployment will zero down time
Note: Here i have updated new nginx image
root@machine:/opt/apps/yamls# cat nginx_deployment.yml ---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: adhoc-sitedep
spec:
replicas: 2
minReadySeconds: 15
strategy:
type: RollingUpdate # ensure zero downtime deployments when moving from current version to next version
rollingUpdate:
maxUnavailable: 1 # only 1 pod is allowed to terminate during version update
maxSurge: 1 # when update to new version only 1 pad can be added
template:
metadata:
labels:
app: webappsdep
spec:
containers:
- name: front-end
image: nginx:v1 ports:
- containerPort: 80
imagePullPolicy: IfNotPresent
Deploying new version of application :
root@machine:/opt/apps/yamls# kubectl apply -f nginx_deployment.yml --record
deployment "adhoc-sitedep" configured
Checking status :
root@machine:/opt/apps/yamls# kubectl rollout status deployment adhoc-sitedep
deployment adhoc-sitedep successfully rolled out
root@machine:/opt/apps/yamls#
Time for rolling back to previous version :
there are number of reason where we want to restore to its previous version:
root@machine:/opt/apps/yamls# kubectl rollout history deployment adhoc-sitedep
deployments "adhoc-sitedep":
REVISION CHANGE-CAUSE
1 <none>
2 kubectl apply -f nginx_deployment.yml --record
root@machine:/opt/apps/yamls# kubectl rollout undo deployment adhoc-sitedep --to-revision=1
deployment "adhoc-sitedep" rolled back
root@machine:/opt/apps/yamls# kubectl rollout history deployment adhoc-sitedep
deployments "adhoc-sitedep":
REVISION CHANGE-CAUSE
2 kubectl apply -f nginx_deployment.yml --record
3 <none>
Nice post.very interesting and informative
ReplyDeletedevops course in Marathahalli
best devops training in Marathahalli
Devops certification training in Marathahalli
devops training in Marathahalli
devops training institute in marathahalli
Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging.
ReplyDeleteDevOps Training in Electronic City
ReplyDeleteThank you for your great post!
Docker Training in Hyderabad
Docker and Kubernetes Online Training
Docker Training
Docker Online Training
Kubernetes Online Training
Kubernetes Training in Hyderabad
Nice post. Keep updating
ReplyDeleteDocker Training
Docker Online Training
Kubernetes Online Training
Docker and Kubernetes Training
Good Post. I like your blog. Thanks for Sharing.
ReplyDeleteDevops Online Training
Devops Training
Thank You for this wonderful and much required information in this post.
ReplyDeleteAgile DevOps Services in USA
Bollywood News in Hindi - Check out the latest Bollywood news, new Hindi movie reviews, box office collection updates
ReplyDeletevenom 2 Let There Be Carnage Full Movie Download & Review
83 Full Movie Download & Review
This information really helped me a lot. It was very informative.
ReplyDeleteDevops Services
cabuFniara Kim Chandler https://wakelet.com/wake/3OPtmrEeDxAvEcJ87hZT4
ReplyDeletedielowloudons
specmiconho-1986 James Waheed Norton Security
ReplyDeleteWebsite
VMware Workstation
derolana
VrumaeZconha Cyndi Peacock This is there
ReplyDeleteClick here
kizrisoper
betmatik
ReplyDeletekralbet
betpark
tipobet
slot siteleri
kibris bahis siteleri
poker siteleri
bonus veren siteler
mobil ödeme bahis
EKR1