This is intended to be a multi-part series because I have a lot to write about here. This one is about my background in leadership and where my interest in moving into an EM role came from.

As an Engineering Manager I've been asked a lot by colleagues who also want to take a similar path - how did you do it? What does an EM even do? How can I do something like that? Why did you go down that path?

I thought I'd write some thoughts on my journey into the role of an Engineering Manager at work, what the role typically entails, how you might become one and some general thoughts on leadership that you may find interesting.

That one time I lived in Mallorca...

When I worked at trivago as a Señor Software Engineer in Mallorca - which helped fund a replacement Ducati after writing off the one I'd crashed back in England* (*it wasn't my fault!!) -  after six months~ish I was asked by the department lead to take on "one of the most important projects for the company" (brought in several hundred million euros per year, exciting). I would be placed in a Technical Lead role and given the ability to build a team and the software as well. I jumped at the chance - it sounded really interesting. Given that I had no experience in any sort of lead role, I knew this was going to be a real challenge for me. I had no idea what I was in for...

Two years later I was almost burned out. Burned out as in, having lost all enthusiasm for coding, creativity, working or anything in general (which is stress), but also more physically ill than I'd ever been before in my life! I'd hired a team of developers having done the recruiting, interviewing, onboarding and upskilling myself - let me tell you, the recruiting took absolutely ages on it's own - one of the first more junior guys to join the team was about to be fired by the department lead because of ineffective behaviour over a long period of time that I had no idea how to handle because I'd never experienced that before, and the project had been grossly over-engineered. I remember buying books on how to better communicate with different personality types, attending training courses to try and get better at the communication side of things, and trying everything I could to learn what I needed to improve on.

Another one of the first people to join the team barely spoke a word of English and didn't have any experience with distributed systems, so there was a huge communication barrier and learning curve for them. I remember them getting a little frustrated and almost giving up sometimes, just nodding and saying "yes" when I tried to explain something because the language barrier was difficult and playing charades and joking can only get you so far when you don't speak the same language. But by the end of two years together, and it did take a lot of effort from both of us, not only had their English vastly improved so they could communicate with everyone better but they had also become a key member of the team with very valuable contributions. My whole journey there I believe aged me about 5 years.

By the end I had two teams: one team of developers and the other devops. I'd learned an absolute tonne of different technologies and their positives and drawbacks, from distributed systems, docker swarm, real fully automated CI/CD - I implemented upsource and automated our production deployments when code reviews were approved and closed by the code owner - to communication with colleagues and stakeholders and agile processes. Each new member of the team had their own specialisation that they brought in with them and I am immensly grateful to them for having taught me so much in such a short period of time. But the 'startup culture', extreme pressure, zero support and lack of leadership above me had taken it's toll on my health - I was in hospital for random things more times when I was in Mallorca over a sixt month period than I have been over here in Bavaria in two years - even more than ever in my life actually - and it was simply because of this stress.

So it was kinda good, but kinda bad too...

My time at trivago was bittersweet. I met some awesome people there. But looking back it was also stressful to the extreme and definitely affected my health both physically and mentally. You know all those people on twitter that like to say "mental health is important!!" and go on about it forever - they're absolutely not bloody wrong!! At the end when the company was going further downhill and a while before they closed their offices in Mallorca, I left and joined Sixt in Bavaria (which is an absolutely amazing place to live) with the intention of learning Go and working more with distributed systems.

Sixt

The move was pretty simple: Sixt flew me to Bavaria in October, interviewed me, I went to Oktoberfest, flew back to Mallorca and got the job offer from them all within five days. At Sixt I did exactly what I planned on doing - I learned Go (and I'm not just talking about the syntax here; I'm talking best practices, edge cases to be concerned about, the purism from inexperienced developers in the Go community specifically etc.) and then I gave a conference talk at GoDays in Berlin on some of the things I'd learned and some crazy automatic reflection-based dependency injection implementation I'd tried in that time all within two months.

At Sixt I was tasked yet again with leading a team to rewrite another over-engineered piece of software, but this time I focussed on ensuring the team did so with a really decent dose of pragmatism. It was a resounding success! After doing this I said to myself hey, I've done this with Swift before, now I've done it with Go, I could do it with Rust next... and then I realised that actually what I was doing wasn't really a challenge for me any more. The only way up was for me to learn completely unecessary architectures and technologies for 99% of use-cases and then sell them to inexperienced developers as a salesman on twitter! I'm only 32, I don't have to be a sellout just yet...

I'm lucky to be a very fast learner though so picking up a language, learning the frameworks and best practices and then building something production-ready and scalable was just a bit easy for me now. I wanted something else: to have more impact. And so my enthusiasm for work and drive to learn and progress returned. That took a while... burning out is no laughing matter folks.

So why after all the stress at trivago would I want to be in any position of leadership again? I was given some parting words by a colleague of mine there that really had an impact on me:

James, you might not know it, but your team loves you.

I still keep in touch with a lot of these people by the way - before the pandemic we met up for drinks, visited eachother's countries and I went with a bunch of them abroad for a two week bachelor party in Vietnam... but that is an entirely different story that is absolutely not in the scope of this blog post!

But most importantly for me: to know that I had such an impact on the people themselves, their careers and the team that I built and led is still something I think about regularly. Messaging some of them on telegram and telling them that I was considering an Engineering Manager role quickly filled my screen with words of encouragement and praise in response. I want to replicate that success again, this time not in a startup culture, this time with support from those who have done this successfully themselves, and this time with a healthy dose of pragmatism, the knowledge of how to communicate expectations properly to all parties and the maturity, growth and experience behind me needed to be successful.

Figuring out just exactly how to move into another leadership role was to be my next challenge.

Onto what an EM actually does...

So what is an EM? What do they actually do? Here I provide the favourite words of consultants everywhere: "It depends".

Saving this one for Part II. Coming soon...

When I first started working with Go, I wrote a blog post on "bypassing Go's lack of constructors", or initialisers as they're known in Go.

Go doesn't support traditional default constructors, but it does provide constructor-like factory functions useful for initialising types.

This post is effectively a part II where I basically do a 180 on my original idea in the post linked above. But first, why the original post?

Avoiding nil pointers

Go has a lot of strengths as a language, from a simple userland concurrency API to being very quick to set up a microservice communicating over HTTP with only the built-in webserver, which is production ready, or very easily over RPC with other microservices with something like protobuf.

For me, the fact that Go is a strictly typed language is also a strength - you have to be explicit in what you are building and this actually makes your programs easier to read. I'm also starting to see the benefits of only needing small, implicit interfaces defined next to the consumers. This is a big part of SOLID - interface segregation. In fact, my team and I are planning on removing all explicit interfaces from the codebase.

On the flipside, as all languages have, Go has some weaknesses too. One of these is the ability to initialise a struct in an 'invalid state', leading to the potential of accidental nil pointer dereferencing further down the line.

Note how in the above example, the use of Service{} allows the Service to be initialised without dependencies. And there's no valid reason these dependencies should not exist.

This can be mitigated in part through a technique called "making the zero value useful" and involves actually allowing your code to call methods on nil pointers and returning an empty string, or struct, or whatever your return type is. So now if, for some reason, your Service is nil or the repository within it is nil, you just return a default value instead.

So instead of finding out what exactly is wrong with your code at compile-time, a real strength of using a compiled language like Go, you find out not only that something is wrong somewhere in the application, but also that you're not quite sure where or why it is and so debugging throughout the whole stack ensues.

"Useful", it seems, is either in the eye of the beholder or the evangelist; not necessarily in the eye of the software engineer.

Enforcing valid state

The ideal scenario would be to never even have the option of nil pointers for dependencies in the first place, so you never need to write pointless filler code to check if anything that is supposed to exist anyway might be nil. A runtime dependency injector could enforce this, but again this comes with the drawback of losing all those valuable compile-time guarantees. There are no compile-time dependency injectors in Go, only pre-compile-time object graph generators.

Initially in my experiments with Go, I realised that initialisation of structs in a valid state could be enforced at the language level through elaborate use of factory methods, private structs and interfaces, with the factory method returning the interface. Returning an interface is shown in the manual as okay given the case that both the interface and the implementing struct have the exact same methods defined.

With the above, you can no longer write s := Service{} or r := Repository{} because these structs are not exported. The only way to access them is with a call to New(), which returns only the exported interface containing the exact same methods the struct has. An additional point here is that the Service cannot exist without it's non-nillable dependency (we're not allowing a pointer to be passed in here).

This strategy effectively enforced constructor-like semantics one can see in other languages - a single, tightly controlled entrypoint for intialisation that cannot be bypassed (except via the reflect and unsafe packages, which is beyond the scope of this article and likely wouldn't easily sneak past code review anyway).

There were a few drawbacks with this approach. One of them was the introduction of duplicate code because we were creating an explicit interface next to every single struct in the codebase, just to enforce something that should have been in the language by default and may not even happen that regularly. Another drawback was godoc didn't work properly with the methods documented on the interface.

In fact, when using this on a team, it ended up being more trouble than it was worth. Everyone agreed that we had no use for "useful zero values", as the business had no concept for them, and so the only reason we would use "zero values" would be to satisfy the compiler and drawbacks of the language, and would simply consist of 'filler code' - so we opted to leave them out in many cases. This was actually the best outcome for us as we didn't need to care about nils any more unless they were absolutely required and intentional.

What worked best for us

The end result agreed on was: always use the New convention for every single struct. Unfortunately this could not be enforced at the language level, but simple static analysis on CI and a linter could check for initialisation of structs that didn't use a factory method in any file that was not a test, and this ended up providing the best balance for us (tests can use shortcuts when needed).

This approach had four benefits:

• Every single struct had one point of initialisation and validation, so would always be valid once initialised, simply via focussing on New().
• Chances of nil pointers found only at runtime were significantly reduced.
• The "default value" approach would never need to be considered throughout the codebase which removed a lot of the associated cognitive overhead required to work with something that is effectively proprietary in Go.
• New Go developers were able to on-board and work with the codebase faster because they didn't need to understand any unnecessary intricacies.

At the end of the day you should be using a language to bring value and further something. Perfectionism for the sake of it only hinders this progress and being pragmatic when working with programming languages and their drawbacks only gets you where you want to be faster. The extra effort required to try and force the language to do something it couldn't easily handle at the language was an interesting experiment, but that was about it. We solved this with process, agreement and a tiny bit of automation at the CI level to provide us that extra guarantee and that was enough for us.

It has been suggested that I should take a look at Rust next. This language will also have it's drawbacks I'm sure, but perhaps it might align more with the philosophy of 'safety by default' which I'm starting to see real value in, and is something I definitely think should be a goal for many languages to strive towards.

If you're interested in my first blog post where I argue for using the private struct, public interface technique, you can read it here. Otherwise, going forward, using the New convention wherever possible I think is a wiser and more pragmatic decision.

I've been writing Go for around a month now as for me it was a requirement that I get to learn it as a part of my new role at Sixt. One of the more interesting or frustrating aspects of the language, coming from a PHP background, is the ability to initialise objects in an invalid state.

Coming from an OO language, I am used to the ability to define constructors on objects - static methods that are an entrypoint into building an object. Without calling this, and without messing around with reflection, there is no other way to create one of these objects.

Golang doesn't provide constructors but instead advocates for a package-level factory method with the New convention. If only this were then the singular way of initialising an object; unfortuntely this is not always necessarily the case.. and so embarks my internal struggle on 'the right way' of doing things in this programming language.

How can I emulate what I gained from constructors whilst adhering more towards "idiomatic go"?

"Constructors"

Firstly, let's define the goal. I want to provide a very simple object API to only allow third-party consumers to create an object in a valid state. Take a look at the following example that follows Golang's conventions:

type MyStruct struct { }

func NewMyStruct() MyStruct {
    return MyStruct{dep: ADependency{}}
}

type ADependency struct { }

myStruct := NewMyStruct()  // We get back a valid struct

This is great. We have a factory function by convention to create an instance of MyStruct. However, we can also do the following:

myStruct := MyStruct{}

Boom - we now have an instance of MyStruct in an invalid state. Why does this matter?

type MyStruct struct {
	dep *ADependency
}

func (m *MyStruct) DoSomethingWithDep() {
    m.dep.DoSomething()
}

type ADependency struct { 
    num int
}

func (a *ADependency) DoSomething() {
    fmt.Println(a.num)
}

(&MyStruct{}).DoSomethingWithDep()

The above gives us:

panic: runtime error: invalid memory address or nil pointer dereference

This is simply because ADependency was not initialised. It's a nil. This problem exists in part because we are able to initialise this object in an invalid state, and in more complex codebases this can go unnoticed and blow up at runtime further down the line.

What can we do about this to ensure that it doesn't happen accidentally but in a consistent way?

Making things private

The reason we can call MyStruct{} is because the type is exported from the package - meaning it's publically available for third parties (other packages) to use. To fix this, we can make it private:

/* Yep, now it's private because of the lower-cased 'm' */
type myStruct struct {
    num int
}

func NewMyStruct() *myStruct {
    return &myStruct{num: 42}
}

Now, from another package, we cannot call myStruct{}. This struct's initialisation is only available via calling PackageName.NewMyStruct() from other packages. This is great! However, this limitation does not exist from within the same package so you just need to make sure that you enforce (again, not by language design but by your own implementation) that you create this object correctly behind the veil of your package.

But what about type declarations?

So let's presume we have myStruct in Package1, and then a struct within Package2 wants an instance of myStruct passed in via a type declaration on a method signature:

package Package2

type SomeService struct { }

func (*SomeService) DoSomething(obj myStruct) { /* WONT WORK!!*/ }

Oops! Because myStruct is from Package1 and is not an exported type any more, Package2 can't access it. We can't even ask for an interface{} and type assert because the type is not exported; it simply doesn't exist in this context.

Interfaces

One way to maintain a private struct but be able to typehint for it in another package would be via a public interface for MyStruct. You cannot initialise an interface, but we can use the type in other packages:

package Package1

/* The interface is exported, so publically available */
type MyStructsInterface interface {
    SomeInterfaceMethod() int
}

/* Private struct */
type myStruct struct {
    num int
}

func NewMyStruct() myStruct {
    return MyStruct{num: 22}
}

/* With this, myStruct now implicitly implements MyStructsInterface */
func (*myStruct) SomeInterfaceMethod() int { return 42 }

Now in other packages' function declarations we can typehint for an instance of MyStructsInterface and use it's public API, and we're also only able to construct the object in a valid state with the NewMyStruct function. GREAT!

So this just means we need to...

Interface Everything

This means that we now have to fully interface everything that we want to be able to use in another package, if we want to protect from the object being built in a way we do not intend. In some cases this is fine, however interfacing everything is overkill and absolutely unnecessary. What about objects that don't actually have any public methods, just a bunch of properties? We can't interface those with because of Go's implicit interfaces (there's no method to add to an interface to say "our object implements this interface now") and this would be a code smell anyway.

Nil Pointers

Let's say we decide that interfacing everything is a stupid idea - it is not pragmatic nor necessary and we would be doing it because of the lack of a language feature, not because we actually need them, which points to this being a code smell. Instead, let's publically expose that struct so we can typehint for it and see what can go wrong.

type MyStruct struct { 
    child *ChildStruct
}

func (s MyStruct) StructFunc() { 
    /* Here's where things go wrong */
    fmt.Println(s.child.num)
}

type ChildStruct struct {
    num int
}

func DoSomething(obj MyStruct) {
    obj.StructFunc()
}

DoSomething(MyStruct{})

We can pass in a MyStruct initialised only with MyStruct{}. The function it is passed into calls s.child, which will be nil, and when we try and call .num on this nil we will get:

panic: runtime error: invalid memory address or nil pointer dereference

So we would have to guard against this:

type MyStruct struct { 
    child *ChildStruct
}

func (s *MyStruct) StructFunc() { 
    if s.child != nil {
        fmt.Println(s.child.num)
    }
}

Above we declare a function on *MyStruct. What about if s.child isn't a pointer but instead is a value (struct)?

Empty Structs

Consider the following:

type MyStruct struct { 
    /* Note how s.child is not a pointer any more */
    child ChildStruct
}

func (s *MyStruct) StructFunc() {
    if s.child != nil {
        fmt.Println(s.child.num)
    }
}

The != nil check will no longer work. We will get the following panic:

prog.go:22:16: cannot convert nil to type ChildStruct

So instead we have to check for an empty struct:

func (s MyStruct) StructFunc() {
    if s.child == (ChildStruct{}) {
    	return
    }

    fmt.Println(s.child.num)
}

This is just another way through which we have to defend code against nils if we allow the type to be exported.

Where we are now

We have the following options:

• Check for a nil for every pointer we have a type declaration for, in every function in the codebase
• Check for an empty struct for every concrete type we have a type declaration for, in every function in the codebase
• Interface everything so we can only build objects in a valid state, and then there is no easy way to create an invalid MyStruct - we can't use package1.myStruct{} and we can't use new(package1.myStruct) - only package1.NewMyStruct().

But I don't want to interface everything!!

Encapsulation

A language mechanism for restricting direct access to some of the object's components.

I require the ability to restrict any object's initialisation to a single entrypoint which guarantees that it cannot be initialised in an invalid state and rightly so, in whatever programming language I am using that has some object oriented capabilites. But in doing so, in Golang, I require an interface to typehint for it anywhere else because of package visibility and a lack of real constructors.

In PHP for example, we have: __construct() which can be the only way to create an instance of a class, and then via that, and our implementation, we guarantee the object exists in a valid state and we can typehint for that as a dependency elsewhere ensuring we only get that valid object to work on. It doesn't seem like I can do anything this with Golang. Is there anything better?

Implicit Interfaces

Go is special in the way that it's not like Java, PHP or other OO languages with explicit interfaces. For example in PHP, our object would implement an interface only when we explicitly type that it does:

interface MyInterface { }

class MyClass implements MyInterface { } // Explicit

In Go on the other hand, we have implicit interfaces:

type MyInterface interface { 
    MethodMustBeImplemented()
}

type MyStruct struct { }

func (*MyStruct) MethodMustBeImplemented() { } // Implicit

Idiomatic Go suggests that sometimes you should define interfaces not next to or near the class that implements them, as you would do with traditional OO languages, but rather alongside the type that will use it. This is an immensely powerful feature once you get to grips with what it means for your code.

Imagine the following struct again with our New best practice used. Note that we are not exporting the type, so we can avoid the problems mentioned earlier:

package Package1

type myStruct struct { }
func (*myStruct) SomeFunc() int { return 42 }
func (*myStruct) SomeOtherFunc() string { return "42" }

/* Factory */
func NewMyStruct() *myStruct {
    return &myStruct{}
}

Now a struct in Package2 requires the ability to use this struct - but let's think about this for a moment. What exactly does Package2 want? - The answer is not the struct, but the ability to call an API method on the struct.

We could go about declaring a globally available interface, which I've already elaborated wasn't the best way of doing things. We don't want to interface everything.

But let's say that our consumer actually only wants to call one method on this struct. We can define an implicit interface, next to the consumer, only containing the one method!

package Package2

/* Implicit interface time! */
type interfaceToConsume interface {
    SomeFunc() int
}

Immediately on creating this interface in Package2, the struct from Package1 can actually fulfil this requirement and fulfil this interface. Now if we add the following code to Package2, we can take in an instance of MyStruct without explicitly declaring that we need to do so with the struct that is not publically available.

type consumer struct { }

func NewConsumer() *consumer {
    return &consumer{}
}

func (*consumer) DoSomething(obj interfaceToConsume) { 
    /* We can actually pass in a Package1.myStruct here! */
}

How can we pass in a Package1.myStruct when it is private?

package Package3

myStruct := Package1.NewMyStruct()
consumer := Package2.NewConsumer()

consumer.DoSomething(myStruct)

From this we have gained the following:

• Types are private so they cannot be initialised in an invalid state
• We utilise Go's equivalent of constructors: factory functions for initialisation
• Idiomatic Go usage in defining the interface next to the consumer
• The implicit interface defined next to the consumer is also Private, which means we're not going to pollute the global namespace with random interfaces everywhere
• Smaller interfaces for simpler test mocks - we don't need every method, only the ones we are implementing
• Also adhereing more towards the Interface Segregation Principle of SOLID (smaller, more specific interfaces AKA role interfaces)
• You can remove all of the nil and empty struct checks that you repeat throughout the application because you have guaranteed that you can no longer have any of your objects in an invalid state
• In the case that you do want to define a public interface to use everywhere, that's still perfectly fine as well, whether or not it is next to the consumer or traditionally next to the implementation

Understanding this required a bit of a mindset change for me but it helped remove a lot of boilerplate and trust your code more. But there are always tradeoffs. Here however, I think there a less chances of error.

Conclusion

But you said you don't like having to declare an interface for every object! Well, we're not creating one huge interface for each struct and that technicality makes the difference. We are defining specific subsets of functionality that other structs can fulfil next to the consumer!

Usually there are tradeoffs to make in any language. For me, this is just a more fundamental one that I have to consider when designing the code. I would very much like to be able to request a valid type and not have to check that it has been initialised correctly, as there should only be a specific method of initialisation, and I would like to not have to create one huge interface for every object in the application.

It turns out that Go's implicit interfaces can provide the safety needed.

I'm really enjoying learning Go and having my mindset challenged and changed as I learn more. Golang definitely has an interesting way of doing things.

Is this also how you write defensively in Go, or have you found a better way of doing things? Let me know what you think in the comments.

Introduction

It's been a while since I wrote a blog post, (actually, this is my first, I have no excuses!), but I really wanted to touch upon asynchronous programming in Swift. Most of the code you write in Swift must at least take asynchronicity into consideration at some level. When updating the ui after some computation that typically does not belong in the controller layer, even then you have to think about where the correct place for this piece of code should be.

Updating UI on a thread other than the main thread is a common mistake that can result in missed UI updates, visual defects, data corruptions, and crashes.

When beginners start out with iOS, they use callbacks to execute certain functionality when another process has finished. Usually the final part of the chain of functionality which is executed upon operation completion, if required, would be to update the UI. Assuming this were the case, using callbacks we would have to pass an 'Update the UI' callback all the way through the chain of callbacks repeatedly until we get to the end of the chain, finally calling that callback to update the UI, assuming we don't want to pollute our domain / infrastructure logic with a call to updating the UI because we value Separation of Concerns.

Let's take a look at how a piece of code can evolve from simple callbacks, through to utilising async / await, allowing you to write completely blocking code in a background thread with only a few extra lines of boilerplate each time. This is very close to the mayhem that I wrote when I first started out working with Swift.

Callbacks

Our use case: we're building an application that first will make a HTTP request to retrieve some json. Within this json is the URL of an image that we want to download. Finally, once we have downloaded that image, we should present it to the user and display it in the UI. Let's build this with callbacks first.

Step 1 - The initial HTTP Request

The first step would be to download some json using Data(contentsOf:), which just performs a GET request to an endpoint without any headers or additional information. In fact, this is the worst way to perform a download; Data(contentsOf:) should be used for local files only, but for the purposes of this blog post, assume no error checking is absolutely deliberate.

class DefaultViewController: UIViewController
{
    override func viewDidLoad()
    {
        super.viewDidLoad()
        
        let response = try! Data(contentsOf: URL(string: "http://our-data-endpoint.com")!)
        
        let jsonData = try! JSONSerialization.jsonObject(with: response, options: []) as! [String: Any]
        
        let imageUrl = jsonData["image_url"]
    }
}

In this intentionally simplistic example I've removed libraries that we might use, we're using Data(contentsOf:) which you wouldn't do except with anything locally, no error checking is present, guard statements aren't used etc. Force unwrap all the things!!

The first thing we do is synchronously (note: we'll make it async next) download a json string into the response variable, convert that json into an object over which we can iterate in jsonData, and then grab the image_url key from jsonData; so imageUrl now contains the url of the image we want to download.

The problem with this right now is that we're blocking the main thread (and run loop) while the json is downloading. The UI is in the main thread, so if the user is trying to do anything else like dragging or navigating somewhere else at the same time, well it won't work - these actions will be queued to be executed after the download has finished.

What can we do about this? We can use a background thread!

Step 2 - Multi-threading

Grand Central Dispatch, or GCD for short, provides a simple and powerful API through which we can perform 'background' operations in a separate thread.

DispatchQueue.global(qos: .background).async {
    let response = try! Data(contentsOf: URL(string: "http://our-data-endpoint.com")!)
}

The above extra line of code (the first line), assuming we don't count the closing parenthesis at the end, places any calls within those parenthesis onto a background thread. Now we're not blocking the main thread so any UI changes like animations or user-invoked events like button presses can continue to be handled.

The problem is soon that we're going to be violating the principle of single responsibility and this class is quickly going to increase in size. If we place the image download code in here too, then this not only becomes a nightmare to test in the future but harder to refactor too.

Let's write code more towards the 'right way' and split things up a bit.

Step 3 - SRP

Let's place our JsonDownloader into a struct and add an onComplete handler (closure) as a second parameter to be executed when the work has been completed asynchronously.

Why do we have to do this? Well, if we're using .async { } within a synchronous function, we can't just do a return from the async call because the async call returns immediately allowing the runloop to continue handling other things. One cannot simply return a variable from an asynchronous context to a synchronous one, because the asynchronous one returns immediately and the schronous code just continues like nothing happened. So we have to execute the callback function that we can pass in at the end instead.

Note we could still just hardcode the contents of each callback together but then we would end up with callback hell as well, and that's without the cleaner separation being shown here

struct JsonDownloader
{
    public func downloadContentsOfUrl(url: URL, onComplete: @escaping (_ url: URL) -> UIImage) -> Void {
        // This async call will be thrown onto a thread and then this function will return immediately
        DispatchQueue.global(qos: .background).async {
            let json = try! Data(contentsOf: url)
            
            let jsonData = try! JSONSerialization.jsonObject(with: json, options: []) as! [String: Any]

            onComplete(URL(string: jsonData["image_url"] as! String)!)
        }
    }
}

Notice that we've defined the onComplete callback to be a closure that takes a url and must return a UIImage. This is the next step in our process. We need to create this closure in our UIViewController and then pass it into our JsonDownloader to be executed upon completion of the json download at a later time.

Step 4 - Closures, Closures, Closures

First, let's create our closure that takes a String and returns a UIImage. Then we can pass it into our call to JsonDownloader.

class ViewController: UIViewController
{
    override func viewDidLoad()
    {
        super.viewDidLoad()
        
        let imageDownloadCallback: (URL) -> UIImage = { url in
            return UIImage(data: Data(contentsOf: url))!
        }
        
        let jsonDownloader: JsonDownloader = JsonDownloader()
        
        jsonDownloader.downloadContentsOfUrl(url: URL(string: "ourUrl.com")!, onComplete: imageDownloadCallback)
        
        print("This line will likely be printed before the asynchronous call in JsonDownloader finishes")
    }
}

Remember the call to downloadContentsOfUrl(url:) has an asynchronous call in a background thread being executed inside of it, so the print() at the bottom will execute first and then later on our callback will be executed to retrieve the image.

Finally, we need to display this image in the UI. We should absolutely not be updating our UI from anywhere within the business logic of the application, never mind the infrastructure area that makes API calls. It's going to become an unorganised mess in a larger codebase if we do this - UI changes should be made in the view controller, which facilitiates messages between the view and the model.

If we want to try writing our UI updating code in our view controller, but it must be executed at the end of an asynchronous call in some other file in the model layer somewhere else, then what can we do about that? Enter... you guessed it... another callback. This time it must take the instance of UIImage that we have downloaded and display this in the UI.

So now we're passing callbacks within callbacks, and we have to update the JsonDownloader to take both callbacks as well. Take a look at the following awful (but working) code.

struct JsonDownloader
{
    public func downloadContentsOfUrl(
        url: URL,
        onUrlRetrieved: @escaping (_ url: URL, (UIImage) -> Void) -> Void,
        onImageDownloaded: @escaping (_ image: UIImage) -> Void
    ) -> Void {
        // This async call will be thrown onto a thread and then this function will return immediately
        DispatchQueue.global(qos: .background).async {
            let json = try! Data(contentsOf: url)
            
            let jsonData = try! JSONSerialization.jsonObject(with: json, options: []) as! [String: Any]
            
            // Calling our first callback, and passing in the second, violating LoD
            onUrlRetrieved(URL(string: jsonData["image_url"] as! String)!, onImageDownloaded)
        }
    }
}

The first thing to notice here is that downloadContentsOfUrl now takes two callback arguments, one to be executed once the url has been retrieved, and the second one to be executed once the image has been downloaded. This will require passing that uiUpdatingCallback through two functions, just so the second one can utilise it, which violates the Law of Demeter.

class ViewController: UIViewController
{
    override func viewDidLoad()
    {
        super.viewDidLoad()
        
        let uiUpdatingCallback: (UIImage) -> Void = { image in
            // Don't forget, we must do any UI stuff on the main thread
            DispatchQueue.main.async {
                let imageView = UIImageView(image: image)

                imageView.frame = CGRect(x: 0, y: 0, width: 300, height: 300)

                self.view.addSubview(imageView)
            }
        }
        
        let imageDownloadCallback: (URL, (UIImage) -> Void) -> Void = { url, onImageDownloaded in
            let image = UIImage(data: try! Data(contentsOf: url))!

            onImageDownloaded(image)
        }
        
        let jsonDownloader: JsonDownloader = JsonDownloader()

        jsonDownloader.downloadContentsOfUrl(
            url: URL(string: "ourUrl.com")!,
            onUrlRetrieved: imageDownloadCallback,
            onImageDownloaded: uiUpdatingCallback
        )

        print("This line will likely be printed before the asynchronous call in JsonDownloader finishes")
    }
}

Here we are defining our uiUpdatingCallback in the view controller. This will be executed at the end of the process by the imageDownloadCallback which we define next. The imageDownloadCallback will be executed when the JsonDownloader has finished downloading the json data containing the image url asynchronously.

We then pass two callbacks to the downloadContentsOfUrl() function call and this uses the first callback within it, but passes the second callback (the ui updating one) through to be used later once the image has been downloaded asynchronusly.

Can you see how this is starting to become a bit of a nightmare? We just want to be able to define what will happen next in the correct place, synchronously, without having to pass callbacks within callbacks.

Let's rewrite this with async / await.

Async / Await

Let's take a look at the entry point first, our ViewController. There are other things we need to do to make this work involving promises, but for now, this is the top level code.

class ViewController: UIViewController
{
    override func viewDidLoad()
    {
        super.viewDidLoad()
        
        async {
            let jsonDownloader: JsonDownloader = JsonDownloader()
            let imageDownloader: ImageDownloader = ImageDownloader()
        
            let json: [String: AnyObject] = try! await(jsonDownloader.downloadContentsOfUrl("ourUrl.com"))
            
            let image: UIImage = try! await(imageDownloader.downloadImage(json["image_url"]))
            
            DispatchQueue.main.async { 
                let imageView = UIImageView(image: image)
                
                imageView.frame = CGRect(x: 0, y: 0, width: 300, height: 300)
                
                view.addSubview(imageView)
            }
        }
    }
}

We've placed the image downloader callable into a class, but that's not the main change here. Firstly we have async { } which is the syntactically sugared equivalent of dispatching into a background thread. Secondly, we have await(), which will wait for the promise within it to resolve asynchronously. These can be two, separate, asynchronous-using structs and as long as we use promises, they will wait. No callbacks, no passing functions around.

This means that an asynchronous piece of code can return a value, via it's Promise, and the await() call handles the waiting for that value to be returned. We're writing code that looks synchronous!

You may be a little confused about why we don't just do the following directly in the view controller itself:

DispatchQueue.global(qos: .background).async {
    let jsonDownloader: JsonDownloader = JsonDownloader()
    let imageDownloader: ImageDownloader = ImageDownloader()
    
    let json: [String: AnyObject] = jsonDownloader.downloadContentsOfUrl("ourUrl.com")
    
    let image: UIImage = imageDownloader.downloadImage(URL(string: json["image_url"]))
    
    DispatchQueue.main.async {
        // -- SNIP -- Add image to UIImageView and add a subview in UIView
    }
}

Looks great, right? But what if you have the background threading within the JsonDownloader class instead, as many other classes do, both core and third-party, and not within your view controller?

• The call to JsonDownloader will return immediately
• The call to ImageDownloader will try and use a url that will actually be nil because downloadContentsOfUrl has not finished yet
• The application will crash

If you restrict yourself to only using background queues from the view controller, and in many cases you need more than one thread to perform some functionality, you'll be coordinating all this logic from the controller and it's going to become a bit of a nightmare. So much for "fat models and skinny controllers" (or rather "skinny models and skinny controllers" as is the current collective thought process at the moment)!

We need to change our JsonDownloader slightly to work with this though, instead returning Promises from PromiseKit.

struct JsonDownloader
{
    public func downloadContentsOfUrl(_ url: URL) -> Promise<String> {
        return Promise { resolve, reject in
            let jsonData = try! Data(contentsOf: url)

            let jsonData = try! JSONSerialization.jsonObject(with: response, options: []) as! [String: Any]

            return resolve(url: jsonData['image_url'])
        }
    }
}

That's it! Our Promise object now returns a call to resolve which internally is only executed once the download has completed. This enables us to use await() and write more synchronous-looking code without any callbacks.

Conclusion

I've been using both PromiseKit and AwaitKit for several months now and the changes from writing asynchronous code with callbacks to this have been staggering - they've made my development so much easier and more fun!

I hope this post convinces some to take a look at async / await and change from callback hell to something that looks a lot more clean and easy to work with.

Let me know in the comments below if you have callback hell in your current codebase, if you are considering checking this out or if you have any alternatives that work just as well in more complex codebases!