The end of the year is always full of conferences, isn’t it? 🙂 This time I attended DevOpsDays 2019 organized in Warsaw, Poland. Considering number of attendees (around 200-300) you can say that “DevOps” isn’t a hype word anymore. I will do my best to keep this post as short as possible and give you the most valuable feedback from these 2 days 🙂 As always feel free to skim the content and pick out the most useful parts for you.
Let’s get started !
Day 1
Kubernetes Operators – the next frontier in application automation
by Ádám Sándor – Cloud Native Consultant at Container Solutions
So, what are these operators?
Operator is a ready to use package or program written by you, which can be used to automate repeating tasks done manually in Kubernetes.
(You can also check out Kubernetes docs on the subject)
What tasks can operators automate?
– deploying an application on demand
– taking and restoring backups of that application’s state
– simulating failure in all or parts of your cluster to test its resilience
https://kubernetes.io/docs/concepts/extend-kubernetes/operator/
How operators work?
Basically an operator acts as a controller with a reconcilation loop which compares a state defined in .yaml files with the desired state.
Check how the default k8s controller works
The operator builds up on a scheme of default k8s controller, but it works on data/resources specific to your application.
Btw you can find ready to use operators here: https://operatorhub.io/
What are the pros of using operators one might ask?
For me 2 major ones are that:
-developers can manage their applications through yaml files and code (written in Java, Python, go…)
– less need to contact the operation team
-operation people can automate repetitive tasks
Best practices for securing CI/CD pipeline
by Victoria Almazova – Microsoft
As Victoria stated, a very well known singer defined security rules in a good way 🙂
Every breath you take
Sting
Every move you make
Every bond you break
Every step you take
I’ll be watching you
Here is the list of some bullet points from the presentation:
- think about security in design upfront and prevent errors before they occur
- a big pain nowadays is that we still don’t automate security
- if you need to wait two weeks for your pentests to happen how can you call your pipe ‘continuous’?
- DO NOT KEEP CREDENTIALS IN PLAIN TEXT, this was something that Victoria still sees in projects – instead, use credential vaults like https://azure.microsoft.com/pl-pl/services/key-vault/
- be super careful when setting up privileges for users who have access to your pipeline. People with lot of rights there have basically full control of your application.
- get familiar with OWASP security list https://www.owasp.org/index.php/Category:OWASP_Top_Ten_Project
When software development is considered we need to implement and automate security at each stage:
- Development
- adding security plugins for sql injectionadding IDE plugins to see code smells
- Commit
- Static code analysisSecurity unit testsDependency management
- Acceptance
- Security scanningCloud configurationSecurity acceptance testing
- Production
- Security smoke testsConfiguration checksPenetration testing
- Operation
- Continuous monitoringThreat intelligencePenetration testingBlameless postmortems
During the presentation Viktoria also mentioned a few interesting plugins and pages. Check out the list below:
WebGoat – hacking simulator divided into parts (code, authentication, configuration) GitHub page of project .
AKS security tools – can be helpful when trying to check where your app can do better in case of security.
Whitesource bolt – tool which checks for vulnerabilities in your code base, such as:
-licenses of libraries which have changed and you are no longer allowed to use freely ?
-security vulnerabilities discovered in github issues of your libraries
https://bolt.whitesourcesoftware.com/
DAY 2
Kubernetes at Datadog Scale
by Ara Pulido – Technical Evangelist at Datadog
Kubernetes is probably the most successful project since linux kernel.
Ara presented the chart from google trends set up using the keyword ‘kubernetes’, here you can see the last 5 years (kubernetes line is blue):
You might be wondering what repeating drops are? Just for reference 🙂 Red line is a chart for Mariah Carey – All I want for Christmas is you.
After a short introduction and description of the size of kubernetes in Datadog, Ara moved to the main part of presentation – networking in kubernetes. That part of the presentation was realllly dense and it’s a good candiate for a separate blog post but the main parts were:
- service networking – basic method of networking in kubernetes. Pods have their (ephemeral) IP addresses but overall traffic to pods is sent via ingress to services which have constant address (names)
Overlay based (flannel , weavenet)
Flannel creates layer 3 overlay network which spans across all nodes in a cluster. When pods are created (blue rectangles below) they are getting specific addresses at which they can communicate (within the same host). If pods between different hosts need to reach each other, they will use flannel route table (created by flanel daemon).
BGP based (calico)
Calico creates layer 3 network which is using BGP routing protocol to route packets. Thanks to BGP implementation packets are not encapsulated when sent, what gives us performance increase and makes troubleshooting easier (packets are easier to track and are not manipulated).
Data Integrity
by Patryk Hes – Site Reliability Engineer at Google
A very interesting talk about backup reliability. Data restore process is getting more and more demanding because applications gather petabytes or exabytes of data nowadays.
Patryk’s presentation was meant to consider questions like:
- Do I have a backup plan if your system completely fails?
- If so, did you make such a simulation and test whether it will work?
Later in presentation we analyzed 3 aspects of data security:
- durability – making sure that data exists
- integrity – backup data is exactly the same as it was written
- availability – will data be available when we need it?
Integrity before availability
Out of these 3 integrity seems to have the most crucial role nowadays. What’s the point of having data which is corrupted and can inject errors into your system?
For instance:
if the availability of your data is 99,99%, that’s 53 minutes of downtime a year
if the integrity of your data is 99,99% for 2GB, that’s 200 KB of corrupted data
that 200 KB could be anything, a very important document could lose a few important pages or executable could be useless.
So, can you be happy when your data is replicated?
No, you can’t.
What you need is a recovery plan and backup is only part of that.
Patryk gave us a few crucial questions when (re)designing/checking the recovery plan:
- did you recover the right data?
- where will backup be stored and how will you find it?
- how do you recover the data? Is there any ‘magic/ninja’ person involved in the process? What would happen if you could not reach that person?
- how long will it take?
- how much extra storage and power will you need for that?
- are you sure that recovery will not break other parts of the service?
- are you taking backups from different instances (decreasing the chance of error multiplication)
- can you make a selective backup of your system (only part of your data)?
And last but not least, you need to practice, practice, practice your restore. You can be sure that the first time you try it it wont work.
So, the more you do it (without harm to the system) the more confident you will be when day finally comes.
Summary:
DevOpsDays is young and still pretty small conference when compared with .NET Developer Days or other big conferences around the world. Technical level of the event is really good and organization was smooth. The only drawback for me was the amount of presentations. The first day had 5 lectures starting at 9:15 am to 1:50 pm and the second day started at 10:00 am and finished at 12:45 pm. After the lectures during both days Ignite Talks and Open Space talks were organized but amount of knowledge/inspiration taken from these varied a lot. During whole conference you would pick just 9 lectures (45 min long each [there were 2 paths to choose]). I think there could be more ‘meat’ for 2 day long event. But don’t be discouraged, DevOpsDays is one of few conferences in Poland about DevOps and is definitely worth visiting. Just check the program to be sure that you will get proper ‘meat’/time ratio 🙂