RELIEVE THE TALKS 2019 IN VIDEOS AND SLIDES

This talk is about our journey from Nginx & Docker Swarm to Traefik & Nomad. With the increase of load & traffic on our container-environment over the time, we experienced some issues that were unnoticed when built the environment. Due to the way we dynamically configured Nginx with Consul-Template we started to experience a lot of dropped keepalive connections and connection resets. Also the traffic wasn’t distributed evenly throughout our container-infrastructure which led to single instances receiving most of the load. This is why made the decision to change to a reverse-proxy that can get dynamically configured. We were aware of the shortcommings of Docker Swarm (standalone) and sought for a tool that would allow us to distribute containers more evenly without totally reconstructing everything and provide us with self healing capabilities. Performing these changes under the hood, transparent for our developers, was one of our key objectives.

DevOps has been around for a decade. In this time it grew from a vague buzzword to a key-subject discussed in and used by several companies. DevOps is more than a group therapy for frustrated IT engineers with new fancy tools! It has impact on most of our workflows – and more! A cultural refactoring complementing bigger projects with even bigger technical or organisational problems aka challenges. What are the show-stoppers when introducing DevOps methods? What are the technical symptoms? Which problems aren’t actually technical? How to approach these issues? This talk won’t give you an introduction to DevOps. It rather highlights where the usage of DevOps methods creates more problems than it solves. We identify potential causes and discuss possible approaches.

This talk is for the developers who want to learn best practices in using Terraform at companies and projects of various size (from small to very large), get pros&cons on code structuring, compositions, tools. Also, attendees will be able to learn Terraform (and Terragrunt) tricks and gotchas.

So, you’ve migrated your application to Reactive Microservices to get the last ounce of performance from your servers. Still want more? Perhaps you forgot about the logs: logs can be one of the few roadblocks on the road to ultimate performance. At Exoscale, we faced the same challenges as everyone: the application produces logs, and they need to be stored in our log storage – Elasticsearch, with the minimum of fuss and the fastest way possible. In this talk, I’ll show you some insider tips and tricks taken from our experience put you on the track toward fast(er) log management. Without revealing too much, it involves async loggers, JSON, Kafka and Logstash.

In this presentation we will explore best practices and how to plan a hybrid strategy. Disaster recovery according to the size of the event. We will discuss the different concepts: Availability, Backup, Recovery, RTO and RPO. How to replicate your storage, networking, databases and compute instances in the cloud, and the different type of recoveries available in a hybrid architecture. We will go from a Pilot light model to a Multi-Site Active-Active and explain cost and time to recover using these options.

In this session, we’ll walk through the five steps to transformational change that I’ve found to be important. These are really applicable to any continuously improving organization or any large amount of change in a system.

• Establish the vision
• Create shared experiences
• Educate, educate, educate
• Find evangelists
• Get feedback

I’ll elaborate on each item with methods I’ve used in real transformations at multiple companies. I’ll also describe how these all tie into the devops culture, which is really the transformation that’s occurring within the company. What will the audience get? The audience will walk away with an understanding of how to bring and manage a large change to a company. They’ll have some examples to discuss and ideas to implement. They’ll also have some additional resources for learning more.

Mgmt is a real-time automation tool that is fast and safe. One goal of the tool is to allow users to model and manage infrastructure that was previously very difficult or impossible to do so previously.
To showcase this future, we’ll show some exciting real-time demos that include scheduling, distributed state machines, and remote execution. As we get closer to a 0.1 release that we’ll recommend as “production ready”, we’ll look at the last remaining features that we’re aiming to land by then.

Having grown 6666x in the last 3 years; the data generated has grown exponentially. As a Data Engineer at GoJEK, we faced the issue with having our complete team managing infrastructure requests. This led us to create an internal portal for other teams to self-provision their data. This talk is divided into two parts. In the first part, I will cover details about how we have scaled our data engineering infrastructure to manage the scale of more than 40 million messages per day. I will explain the data consumption, aggregation, monitoring and cold storage. This will also cover details about how we scaled our infrastructure to achieve the scale that we are at today. In the second part, I will cover how we created our internal portal for infrastructure orchestration. The infrastructure backed by kubernetes enables teams to self-provision data infrastructure without any supervision.

Ceph claims to be a perfect unified file system for almost every purpose. In theory this can be true but in reality there are of course limitations. Following our way from Firefly (2014) to Luminous (2018) I will show our use cases, the shortcomings and the pitfalls we fell into! This will show Ceph in use for typical cloud and container stacks, simple media archives for applications and the devilish high available shared file systems.

The rise of IoT and smart infrastructure has led to the generation of massive amounts of complex data. In this session, we will talk about time-series data, the challenges of working with time series data, ingestion of this data using data from NYC cabs and running real time queries to gather insights. By the end of the session, we will have an understanding of what time-series data is, how to build streaming data pipelines for massive time series data using Flink, Kafka and CrateDB, and visualising all this data with the help of a dashboard.

We will dive into KubeVirt and see how we could create and manage VMs in Kubernetes In this session we will talk about what is KubeVirt and how it works on a kubernetes platform. KubeVirt allows users to create and manage virtual machines within a Kubernetes Cluster.
This session will be covering the following topics:

• KubeVirt Installation
• Basic KubeVirt objects and components
• How to deploy and manage virtual machines
• KubeVirt Storage
• KubeVirt Networking

Benefits :
Kubernetes is a well established container platform, but migrating applications/services to containers is not always easy. KubeVirt allows in such situations to migrate virtual machine based workloads to the same platform where the containers are already running, thus helping converge IT Infrastructure into one single platform, Kubernetes.

Kubernetes offers a lot of basic resources to accommodate most application requirements but sometimes this is not enough. Kubernetes 1.7 introduced a new CustomResourceDefinition (CRD) resource to create your very own kubernetes resources. Although CRDs themselves are easy to use, they provide no way to define the actual logic behind the custom resource. For this the Metacontroller and Kubeless projects can be used, making the whole process a breeze.

Kubernetes Operators are the next phase of the journey towards automating complex applications in containers. Many Operators that exist today handle initial deployment, but they don’t provide automation for tasks like binary upgrades, configuration updates, and failure recovery. Implementing a production-grade controller for a complex workload typically requires thousands of lines of code and many months of development. As a result, the quality of operators that are available today varies. In DC/OS, the DC/OS commons SDK enables anyone to build service automation for DC/OS using just a declarative spec in most cases. The Kudo project can now leverage this set of automation expertise to enable automated creation of operators for Kubernetes. In this talk I’ll introduce the Kudo project, talk through the conceptual similarities between frameworks and operators, and demo the creation of a Kubernetes operator using Kudo.

Modern infrastructures come distributed. Laws require you to log more information. Because of those two aspects, using single sign on is almost mandatory. Let’s focus on Keycloak and what it can bring to your entreprise, from the faster login time to the centralized users and permissions management. We will also cover the automation and monitoring of the solution.

With DVCSs, branch creation became very easy, but it comes at a certain cost. Long living branches break the flow of the software delivery process, impacting stability and throughput. The session explores why teams are using feature branches, what problems are introduced by using them and what techniques exist to avoid them altogether. It explores exactly what’s evil about feature branches, which is not necessarily the problems they introduce – but rather, the real reasons why teams are using them. After the session, you’ll understand a different branching strategy and how it relates to CI/CD.

This presentation highlights several different methods for backups inside of Ceph (clusters). We often receive requests for both local and remote backups. So we would like to introduce backup methods using external tools as well as some using Ceph’s own ‘rbd export’ or ‘rbd-mirror’ approaches. Learn about the pros and cons of each approach, be warned of possible pitfalls both of native use or OpenStack-based approaches.

Integrating microservices and taming distributed systems is hard. In this talk I will present three challenges we observed in real-life projects and discuss how to avoid them using Open Source orchestration. Communication is complex. With everything being distributed failures are normal, so you need sophisticated failure handling strategies (e.g. stateful retry). A synchronicity requires you to handle timeouts. This is not only about milliseconds, systems get much more resilient when you can wait for minutes, hours or even longer. Distributed transactions cannot simply be delegated to protocols like XA. So, you need to solve the requirement to retain consistency in case of failures. I will not only use slides but demonstrate concrete source code examples available on GitHub.

In our On Premise hosting environment we still run a lot of applications on traditional stacks without using containers. In order to run them in a secured way we created a mature patch automation. Thanks to ansible, rundeck, icinga and a bunch of other opensource tools we are able to update and reboot most of our systems without our customers noticing. We do that throughout the day on a regular base using rundeck, or even on short notice if another “heartbleed” occurs.

We are all drowning in work and automation seems to be our only salvation. But sometimes our best intentions lead to horribly broken systems that need manual intervention or are outright unmaintainable. How does that happen and what can we do to prevent that? In this talk I want to show some mental models and frameworks to create good automation systems that support our teams and reduce the need for manual actions.

There are multiple reasons one might want to run VMs inside containers. One scenario could be running VMs as part of a CI pipeline which executes all build steps inside containers. This can be useful, for example, for emulating bare-metal deployments. In this talk, I will demonstrate how to create a virtual machine inside a Docker container using KVM as the hypervisor driver, explain how the container needs to be configured in terms of privileges and discuss some of the challenges involved.

Mgmt is a real-time automation tool that is fast and safe. One goal of the tool is to allow users to model and manage infrastructure that was previously very difficult or impossible to do so previously.
This presentation will briefly introduce the tool and spend most of the time presenting and demoing some of the advanced and/or lesser-known features in the project. We’re certain the audience will have experienced many infrastructure tool problems before, and we hope to convince you via many live demos that our novel approaches are efficient and elegant.We’ll also dive into some of the internals of the project for new contributors who want to peer into the core code and become an advanced user of the project. Finally we’ll talk about some of the future designs we’re planning and make it easy for new users to get involved and help shape the project. 

As your organization grows, so does your workload—and the IT resources required to manage it. There is no “one-size-fits-all” system management solution, but a centralized, open source tool such as Foreman can help you manage your company’s IT assets by provisioning, maintaining, and updating hosts throughout the complete lifecycle. Foreman is an open source project that helps system administrators manage servers throughout their lifecycle, from provisioning and configuration to orchestration and monitoring. Using Puppet, Ansible and Foreman’s smart proxy architecture, you can easily automate repetitive tasks, quickly deploy applications, and proactively manage change, both on-premise with VMs and bare-metal or in the cloud. The talk will explain the key features like provisioning, monitoring, and configuration management in brief and how easy it becomes to have all these features under the same hood to manage your datacenters.

DevOps describes the culture of communication and collaboration between IT Development and IT Operations departments. Usually this covers system setup, application deployment and all the surrounding tools like CI, metrics and alerting. A core element has always been IT automation using configuration management and CI/CD. But how do we do DevOps in a containerized world? Do we still need configuration management? Is there still collaboration possible? Or iare containers the DevOps killer? In my talk I draw a picture on responsibilities, does and don’ts when it comes to developers and operations working within a container runtime platform. I want to show where we still have DevOps and where we pass responsibilities to a single department only and what kind of security borders we have at hand to work separately but with trust.

Carta helps companies manage and secure their cap table and equity plans. Highly sensitive data. And in a post-GDPR world, data engineers play a critical role in protecting data and limiting access at each step in a data pipeline. In this session, Troy will walk through the steps that Carta’s data team has taken to secure the data pipeline using open source tools. You will leave with a checklist of things to consider when building a data lake, data warehouse, or deploying a data orchestration system. Some of the technologies covered include Apache Airflow, dbt, Docker, S3, Redshift, and Looker. Become a better steward of your customer’s data.

The Internet is filled with outdated information on how you in practice run MySQL and MariaDB Server securely in 2019. Naturally, this is not the fault of the Internet but the fact that the security models with modern MySQL 8 and MariaDB Server 10.3/10.4 have changed, giving you newer options.
More interestingly is also the fact that you tend to run in Wide Area Network (WAN) environments, and replication can happen across cloud-based environments. This is where we will also within this talk setup quickly a fully running 3-node Galera Cluster with MariaDB Server 10.4, in a secure fashion, showing you why you do not have to “run with SELinux turned off” or other poor practices (e.g. turning off the firewall).
Topics covered but not limited to: TLS/SSL for connections, replication, external authentication, encryption, and regular running and configuration of your MySQL and MariaDB Servers (both of which take on many different options).