blog

Communicating Remotely

(Thanks to Nikki Heald and Dave Murphy for their contributions)

For about 7 years, I worked from home participating in a very large Open Source project  (Ubuntu Linux). Since then, I have continued setting up and advising companies on how to work remotely.

While you might be struggling with setting up on short notice your working environment at home, soon you will find your groove, and then the next big thing will hit you: Your usual ways of accessing information and the tools you use to communicate have suddenly, drastically changed.

When you were working in an office, you used Instant Messaging (IM) to communicate with people in other offices every now and then, but you largely relied on face to face conversations or office meetings. Now EVERYTHING seems to happen in IM.

Instant Messaging (IM) is a great tool for remote working but also has massive pitfalls. This is not just your problem but something remote organisations have been struggling with for years. Here are some key best practices:

Keep communication in the open. Set up team chats and invite people from other teams to ask questions in public channels, avoid as much as possible starting conversations on direct and private messages.

It is extremely easy to misinterpret and get frustrated using instant messaging. Keeping conversations in the open makes people think twice about the language they use. It also increases the possibility of someone from the group chat replying to your urgent message faster.

Be kind and patient.  Written communications are always easier for non-native speakers, but it can also make us overestimate the language skills of our colleagues and read too much into what they write. It might be that something you feel is aggressive or even offensive is just a side effect of this. At this point, take a breath, be polite and ask to have a video call.

Move detailed conversations to private messages and then to video call.  The recommended communication flow is to start in an open forum (group chat) , then if the conversation gets into details and it is clogging the channel, ask for it to be moved to a thread or private direct message, finally if still taking too long, set up a video call to address the issue at hand.

IM is asynchronous, really!. Instant messaging gives us the false sense that we have the full attention of our colleague, and that they are going to reply straight away. This is nothing further from the truth! Work or personal life can be interrupting and stressing out the person you are talking to, leading to long pauses or terse replies. Do not get frustrated, relax, they will get back to you as soon as they can.

IM is disruptive. Constant direct IM interruptions can lead to low productivity, this is another reason to use group chats. One good tip for teams is to designate a person (maybe rotate it daily) responsible for answering questions from other teams in your group chat. This means only one of you is getting distracted, but also gives a better responsiveness to the person asking the question.

Document decision outside IM. Instant messaging is ephemeral, even if it is in written form. If you take a decision in a group or private chat, follow it up with an email, a gdoc or whatever else your team uses to share documentation.

Don’t try to read everything. Very quickly IM proliferates and becomes an avalanche of information. Don’t have fear of missing out, you shouldn’t expect to read everyone’s messages AND don’t expect everyone to read your messages. If there is something that you really need people to read, use an email or arrange a meeting.

IM is not that social. Although it is a great tool for GIFs and Memes, it is not going to replace the social interaction that you are used to in the office. Instead, try leaving a video call, such as hangouts, open with the rest of your team members while you work on your tasks, just for chit-chat.

If you have any other questions or you are struggling with managing your team communications while working remotely, don’t hesitate to reach out to me!

 

Developing Yourself at Ebury

One of the key things that motivate me every day to work at Ebury is the company’s passion to invest in its people. This comes directly from Juan and Salva (our founders) and it is lived by all of us in management positions.

But what does it mean for our Tech teams? Originally, when we were 10-20 people in Tech, this happened organically, but over time and with growth, it is important that career development also scales up and it is consistent across teams. 

During the last year and a half, I have been working with the team to define a framework that allows people to plan their development and have meaningful conversations with their leads.

At Ebury Tech, your career can take several paths, but it all starts with mapping what are 6 competencies that we really care about.

chart

6 Competencies and 4 Levels

We have summarised our cultural values into 6 competencies, each of them with 4 levels. Leve 1,2 tent to focus on you, as an individual, while the others focus on your impact in your team(3) and the wider tech organisation(4):

  • Domain Mastery: How good are you in your domain and do others agree? 
  • Team Work: Do you help to achieve others’ objectives? 
  • Continuously Improves: Do you improve and help others improve?
  • Problem Solving: Can you resolve real-life issues with a simple solution?
  • Business Impact: Deliver value, not just code
  • Leadership: Do you set an example to others and take responsibility?

Not everyone  follows the same path

Although it might sound like something that you get from a fortune cookie, it really applies to personal development. At Ebury, we have define 3 loose paths to guide your development in Technology:

People Leadership – This is the path I chose, while still technical, my passion is to help others develop and work with team to achieve high performance

Technical Leadership (Architecture) – Another available path is to focus on your ability to design systems that deliver customer value and to work with others across the organisation to hone the best solutions for our clients.

Technical Leadership (Hands on) – Finally, there is also a path to lead by example, to bring change and best practices from within a squad. 

Screenshot 2020-01-23 at 14.38.13

Putting it all together

Each step in a development path is defined by a set of behavioral and observable expectations. These are a level within each of the 6 competencies.

Note, that I have said observable. This is important because, in order to move from one step to another, your team lead will put you forward a case with examples to a panel of peers that will support your change based on observable facts.  Not everyone will move to the next step at the first try, but you will receive tangible feedback on how to achieve it next time. 

Screenshot 2020-01-23 at 14.51.35

Digg deep, let me know what you think

There is no secret sauce to what we do to invest in our people, hence I am sharing with you and I look forward to your feedback. Hit me up in Linkedin

Ebury: The Fin and the Tech

We often talk about Ebury being a disruptive Fintech company. Today I wanted to go into more detail about what makes Ebury a Financial and Technology company.

FINtech

Ebury’s core business is foreign currency exchange (Forex), and we focus on the enterprise segment of the market. To better illustrate what we do, let’s take as an example a made-up European toy company, which we will call TOYSA.

TOYSA’s biggest season is the Christmas holiday period between December and January. TOYSA sells toys to many countries in Europe including the UK. Their UK sales channel received large orders for their new drone, the TOYSA D500. Their UK office would like to know what should be the retail price in British Pounds (GBP). The TOYSA D500 is produced in a factory in Shenzen, China and a single unit to make costs 790 Chinese Yuan. Let’s say that “today” that is equivalent to 100 Euros. TOYSA would like a 20% margin on their toys, so they retail at 120 Euros in mainland Europe, that “today” translates to 110 GBP.

In order to have the toys ready for sale in December, they need to be ordered in June so they can be manufactured in August, and TOYSA is going to be running a big marketing TV advertising campaign in October.

But TOYSA’s CFO can’t sleep at night… she is worried. What if the Yuan grows strong against the Euro from June to August? Their 100 Euros cost calculation could be completely wrong!! Conversely, with Brexit looming, the British Pound could devalue against the Euro, a retail price of 110GBP could not even cover the cost of manufacturing…

This is where Ebury can help TOYSA. We offer both Spot (right now) and Forward (any time up to two years) contracts that remove the risk of currency fluctuations by providing a fixed exchange rate for a future date. TOYSA’s CFO can use these rates to calculate her profit and retail prices removing risk from the equation.

But… hold on! She is still worrying. She needs to pay for the shipment of drones with money that will come from future D500 sales, which she doesn’t yet have! Yup… Ebury can also help with that. It is called Trade Finance.

finTECH

Ok, now you understand what we do but how do we do it with over 140 currencies? With quite a bit of Django/Python.

Our back-end services, which all run on Amazon EC2 (some of them as part of ECS clusters), take care of financial tasks such as getting quotes for currency exchanges, instructing payments, receiving funds, Anti-Money Laundry checks, reconciliation of accounts, and so on. All new services developed in the last few years run in AWS Elastic Container Service (ECS) and we are right now, in the process of containerising older services.

Customers contract Ebury’s products in 3 different ways:

  • They contact their Ebury account manager. Our Front-office team is pretty large, and to ensure consistency, we automate our sales and operations processes in Salesforce. Some of our back-end systems also have “admin” level consoles for financial reports and complex operations, these are built as Django applications.
  • They use Ebury Online (EBO). online.ebury.com is a Javascript application served by a Django app, that provides a two-factor authentication login into a front-end for customers to manage their transactions and book new currency trades. It connects to our back-end services internal API.
  • They use their own ERP (such as Netsuite). Netsuite is just one of the ways customers can access our REST API, which is a Flask service that provides a programmatic interface to the same internal back-end API used by EBO.

Join Us!

If all of this sounds interesting and you are looking for a new opportunity, we are hiring!! Whether it’s a remote position you are after or you would like to join us in our Central Malaga office, get in touch! We could have the role for you.

Story Estimation

We were having a chat internally at Bitnami with respect: should we try to standardise T-Shirt sizes across the team and also if we ought to have a mapping between user story points and implementation hours.

I felt that this is a discussion that keeps repeating, in conversations internally and externally, and I thought it would be worth it to put some effort into summarising my  thoughts about it.

Story Points vs Task Hours

Normally you will find Scrum coaches recommending the use of T-Shirt or Points for Story sizing, and hours for estimating Tasks. Stories are the unit of product/user value that it is delivered (tasks are just work).

Story sizing (T-Shirt or Points) is not an estimation of time, rather a way to express complexity. Yes, there is a correlation between complexity, effort and hours.  However, Complexity tends to be less variable across team members than hours.

A Story is complex due to the problems to be solved while implementing it, but the time it takes to solve the same problem might vary depending on the skills and experience of the person solving it.

OK, but how do we established consistency in estimating complexity? Human beings are pretty good at comparing similar problems. Problem A is similar to Problem B. So, I normally start by getting a team to agree the sizes of stories/problems that they have completed/solved together in the past. Then use that as reference point going forward.  This common understanding can only be shared by people in the same squad, since it is required a shared experience.

Story sizing can be and it is used to measure velocity, and also burn down progress. However, IMHO they are most useful to identify misalignment (Foo thinks is XXL but Bar thinks is L, how come? maybe Bar has some context that is missing Foo? or the other way around?). This conversation between team members about complexity is where the real value of Story sizing resides.

I think we can all agree that we can not predict the future.  Hence estimating in hours is always highly inaccurate. Inaccuracy grows as you extend the period of time you are estimating over (i.e. something that can take months or it is months away).

So why is it OK to estimate tasks in hours? Because tasks are small break down of stories that are assigned to an individual to implement.  Basically, the estimation is done over the shortest time span possible and taking the skills and experience of that person into account.

Tracking and managing cost

A different (and valuable) problem to solve is tracking the cost of implementing a feature/product/release for financial reasons. Most of us work on either For Profit or Cost Driven (non-for-profit) organisations, so having a record and an indication of where money (and hence hours) are going can be critical for our organisations.

While estimating time is highly inaccurate,  tracking actual time spent on tasks doesn’t have to be. Clearly there is a (human) cost associated on the granularity of this tracking. Which is the right level of tracking will be highly dependent on your business and accounting model.

Enhancing Helm Charts with Operators

I was interested to see if I could blend a Helm Chart (packing and deployment) and an Operator (day-2 operations) in a single solution, so I developed a proof of concept for ChartMuseum.

I have been wanting to play with the new Operator SDK from RedHat for a while. I like the concept of having an api for the life-cycle management. IMHO, operators are a development pattern that matches declarative Kubernetes API objects to lifecycle events like backup, restore, and other type of maintenance tasks.

In general, most operator implementations that I have played with also take care of the deployment of application. In some cases, like the etcd operator, they manage the life-cycle of pods directly rather than using standard objects like deployments or replicasets .

I have been doing some Helm Chart development and I really like the flexibility that Helm gives you to parametrise deployments.  It seemed to me that I could still use a Helm Chart for packing and deployment, and enhance it with an operator to be included in the releases to manage the application management beyond deployment.

As a proof of concept, I decided to try to extend the existing upstream chartmuseum chart with an operator that took care of adding and packaging charts from a github repo.

The basic operations that I set up to automate were:

  • Pulling a new git repository and helm packaging its contents
  • Regularly pulling updates and repackage a repo
  • Removing a git repository and its contents

I needed to be able to ask the chartmuseum to perform these activities. Using the adapter/sidecar pattern, I developed a new container to expose these as http endpoints and package dependencies (git and helm) to be bolted to chartmuseum’s container within the same pod.

So a Custom Resource for this operator will look like this:

apiVersion: "cm.bitnami.com/v1alpha1"
kind: "Chartmuseum"
metadata:
  name: "myrepo"
spec:
  git: "https://github.com/foo/mycharts"
  updateEveryMinutes: 15
  dependencies:
  - name: bitnami
    url: "https://charts.bitnami.com/bitnami"

The git repository has to be publicly available. The helm charts might be pulling dependencies from other repos, so I also added the ability to define these.

updateEveryMinutes is an integer value that indicates how often, in minutes, the git repo should be updated. Instead of having this functionality baked into the operator, it creates Cronjob objects to trigger the update. This objects are gettable and visible to the user.

On creation of a new custom resource of type Chartmuseum, the operator will:

  • Add dependencies as required –  POST /repo/dependency
  • Then add a new repository, which will trigger a git clone, and a packaging of all folders in the repo that contain a “Chart.yaml” file –  POST /repo/new
  • A new Cronjob will be created with the same name and namespace than the Custom Resource that will updateEveryMinutes hit GET /repo/name/update

On deletion of the custom resource, the operator will:

  • Delete the repo and its artifacts  – DELETE /repo/name
  • Delete the Cronjob object

Incorporating this into an existing Helm Chart was relatively simple. I created a new value flag called operator.enabled, which if set to true will add additional manifests to the release by:

  • modifying the main deployment to add the sidecar container,
  • modifying the main service to expose a new port for sync operations,
  • adding CRD for Chartmuseum and a deployment for the Operator,
  • adding RBAC support for the operator to read CRD and create Cronjobs

In summary, I found the operator-sdk really simple to use and work with. It really takes away the fiddly parts of creating an operator and lets you focus on the logic. I also found that the operator and the helm chart worked nicely together and provided a good transparency and control to the user.

If you want to try it, just clone this repo: https://github.com/vtuson/chartmuseum-chart and run:

helm install . -n mychartmuseum

Scaling WordPress In Kubernetes

Cloud Native applications have been designed to be run in microservices architecture where individual components can be scaled separately, data is persisted and sync across replicas and node failures can be easily survived. This can be more tricky with traditional web applications that have not been designed this way, such as WordPress.

Bitnami produces production-ready Docker images and Kubernetes Helm Charts that you can deploy into your Kubernetes Cluster. This blog will make reference to concepts that have been previously covered by our Kubernetes Get Started Guide.

WordPress Is A Stateful App

Although WordPress uses a database backend to persist and maintain user created content, administration changes are persisted to the local drive. These includes the plugins installed, site configuration and CSS/Media used by the web front-end. Multiple replicas of the same WordPress site would expect to have access to a shared drive. Basically, WordPress likes to keep its state close by.

Our Kubernetes Helm Chart for WordPress uses a Deployment to manage and scale the WordPress pods. In order to share this admin-created content across pods, the deployment mounts a volume provisioned by a Persistent Volume Claim(PVC).

When scaling a web app it is important to understand the different types of Access Modes available for persistent storage in Kubernetes: ReadWriteOnce (RWO), ReadOnlyMany (ROX) and ReadWriteMany (RWX).

If you want to administer your WordPress site separately, you could expose a read-only configuration volume to your pods. In this case, ROX would be you best choice since it can be mounted as read-only by many pods across multiple nodes.

If you want to continue using the WordPress admin interface in all its glory, all your pods will need read-write access to a common volume. It is also likely that you would like your pods run in different nodes (for better site availability and scalability). Since a RWO volume can only be mounted in one node at the time, you would really need to use RWX.

Great, Where Do I Get RWX Volume?!

Unfortunately RWX is not a very commonly supported access mode by the current list of volume plugins (see official documentation). So what can you do if you don’t have access to one in your cluster?

Since, WordPress write access does not require a highly performant solution, we are going to share a RWO volume across multiple pods by remapping it as a RWX NFS volume.

Hold on, this is going to be complicated, right? Nope, it is going to be a single command.

There Is A New Chart In Town

A few days back, a new Helm Chart landed on the official Kubernetes repository – Introducing the NFS Server Provisioner.

What this chart does is to deploy all the bits you need to enable dynamically serving NFS persistent volumes (PV) from any other volume type. Most Kubernetes deployment will have a Storage Class that provides RWO volumes. We are going to map a single RWO persistent volume from this existing Storage Class and share it as RWX Storage Class called ‘nfs’.

The following diagram show a summary of the kubernetes objects involved in this process:diagram

Please note that this solution has low fault tolerance, as an outage of the node that has the RWO mounted will affect the whole deployment availability (should not lose data).

As promised, deploying the NFS-Server provisioner is a simple helm command:

$ helm install stable/nfs-server-provisioner --set persistence.enabled=true,persistence.size=10Gi

Deploying WordPress To Work With NFS

Once you have deployed the Helm release, you can check that a new Storage Class called ‘nfs’ is now available in your cluster:

$ kubectl get storageclass

show storageclass results

Also the RXO volume claim has been created. In this case, it is using the standard storage class in minikube (hostpath) and it is bound to the NFS-server statefulset.show rxo pvc

Next we can deploy a WordPress release, using the NFS Storage Class:

$ helm install  --set persistence.accessMode=ReadWriteMany,persistence.storageClass=nfs stable/wordpress  -n scalable

We can see how we now have 3 PVCs in our cluster:

  • a RWO for the MariaDB backend database
  • a RWO for the NFS Server
  • A RWX for the WordPress pods

Inspecting the new PV, we can see that it is served over NFS:nfs pv

Are You Ready To Scale Horizontally?

Almost, the WordPress admin console will require a user session to be served always by the same pod. If you are using an ingress resource (also configurable by the chart) with a Nginx ingress controller, you can define session affinity via annotations. Just add these annotations to your values.yaml file, under ingress.hosts[0].annotations.

Now you just need to scale your WordPress deployment:

$ kubectl scale deploy scalable-wordpress --replicas=3

Happy Highly Available Blogging!