Links 2017-12-31

Note: this is a new experiment I’m running – posting a weekly list of links that I enjoyed. Feedback welcome – also if anyone would like these in their inbox, let me know and I’ll add a subscription option.

Computer latency: 1977-2017 – Dan Luu

A wonderful article detailing how our computing devices have been getting less responsive over the years, and how complexity makes it worse.  There are good reasons why a 34 year-old Apple IIe  is top of Dan’s responsiveness charts, but that doesn’t mean we should be proud of it.

From inboxing to thought showers: how business bullshit took over – The Guardian

A glorious rant against business speak, with a shout-out to one of my favourite TV series, W1A, which skewers this practice wonderfully.

Dozens of Companies Are Using Facebook to Exclude Older Workers From Job Ads – ProPublica

Discrimination of any kind is awful of course, but what’s baffling about this particular case is that companies are hurting themselves by excluding potentially brilliant employees. And this doesn’t just go for older people – the lack of diversity in tech is a directly result of tech companies using stupid hiring practices and then wondering why it’s so hard to get good people.

Decomposing Emoji – objc.io

An illustration of how Swift, Javascript, Ruby and Python all have different answers to the question of “how long is this string”, and how they’re all right. It feels like since Unicode, strings are the new dates – representations of real-world things that most programmers get wrong.

How Facebook’s Political Unit Enables the Dark Art of Digital Propaganda

It’s not shocking that Facebook are making money from rich people looking to change peoples’ minds, but it is shocking how blatant they are about it. I’m expecting Facebook ads paid for by people and organisations outside Ireland to play a big role in Ireland’s abortion referendum next year.

How to scam $200,000 per month and get 67,882 all 5 Star reviews on the app store – reddit

A nice bit of digging on fake reviews on the App Store. My biggest complaint about the App Store – it’s not the 30% cut, or the stringent review criteria, those things I’m happy to put up with. My complaint is that the scammers still win. If the scammers are vanquished and good apps will naturally rise to the top of the sales charts I’d be a much happier app developer. This article shows how much work Apple need to do to fix the App Store and remove the rubbish.

Rust in 2017: what we achieved – rust-lang.org

I’ve not tried out Rust yet – but this is a pretty good example of how to grow a community round your project. I enjoy these “year in review” posts so much, I’m surprised that more people don’t do them (and just to show it’s not just for programmers, I enjoyed Candy Japan’s 2017 review post too)

On the front lines of the GOP’s civil war – Esquire

An interesting account of Republicans in the US trying to take their party back.

Chrome is Not the Standard – Chris Kycho

It still baffles me how Chrome managed to get such market share, especially amongst developers. I’ve only came across a few sites that were Chrome-only – mostly complicated web apps – and I hope that this trend doesn’t continue.

Ask HN: Is it too late to find a mentor after 30 for a software developer? – Hacker News

A Hacker News discussion thread that reminds me of a business idea I have – create a remote part-time version of Recurse Center – pair people up with a remote mentor, charge a subscription fee where most of the money goes to the mentor, to a maximum of x hours a month. Like most of my business ideas, I really just want this service to exist, because I’d definitely pay for it.

The Only McLaren F1 Technician in North America – Road And Track Magazine

I’m mostly bored about cars these days, but I still remember the excitement when the McLaren F1 came out, and this article is a lovely tribute to a truly ground-breaking machine.

To Serve Man, with Software – Jeff Atwood

An eloquent post on the need for software engineers to recognise the responsibility that comes with our increasing power.

Eight opinionated tips for people learning Swift

I was talking to two Objective-C iOS developers recently who told me that they were starting to learn Swift. This got me thinking: what did I wish I knew when I was learning Swift? Here are eight tips that are opinionated: they are not designed to help you learn Swift, but they might help you to work with the language instead of against it.

1. Avoid force unwrapping and force casting

I’ve written about this before – and it annoys me that Xcode sometimes recommends force unwrapping Optionals as a fix-it. Force unwrapping with ! and force casting with as! will bite you in the end – don’t do it. (Possible exception – objects from Storyboards and interface builder files, as their types are defined at runtime).

2. Prefer guard let over if let

You’ll learn that the classical way to unwrap an Optional is to use if let. The problem with if let is that it can cause pyramids of doom of closing braces in your functions. Consider this example:

if let processedString = funcThatReturnsAnOptionalString("parameter") {
    if let secondProcessedString = funcThatReturnsAnOptionalString("another parameter") {
        if !processedString.isEmpty && !secondProcessedString.isEmpty {
            //do stuff
        }
    }
} // yuk

Solution: use guard let introduced in Swift 2:

guard let processedString = funcThatReturnsAnOptionalString("parameter") else {
    //you may want to perform error handling here
    return
}
guard let secondProcessedString = funcThatReturnsAnOptionalString("another parameter") {
    //handle error if necessary
    return
}
if !processedString.isEmpty && !secondProcessedString.isEmpty {
    //do stuff
} //goodbye pyramid of doom

This gives you a fail early pattern and allows you to avoid unnecessary indentation.

3. Unwrap multiple things at once

You’ll run in to another problem once you start using guard let: you’ll find your functions compress horizontally (less indentation) but will expand vertically (more lines of code). Solution: unwrap multiple things at once. Consider the example above compared with:

guard let processedString = funcThatReturnsAnOptionalString("parameter"), let secondProcessedString = funcThatReturnsAnOptionalString("another parameter") else {
    //handle error if necessary
    return
}
if !processedString.isEmpty && !secondProcessedString.isEmpty {
    //do stuff
}

Caveat: I find that if I’m unwrapping more than three things at once, it’s a code smell and I need to refactor. Also: it’s possible to combine guard let with traditional guard statements with conditional clauses: although I’m on the fence whether this is a good idea or impedes readability, I do find that it works as long as I restrict it to conditionals that answer the question “is this thing valid enough for me to work on?”. This would be achieved with the following (make your own mind up if this is taking things too far):

guard let processedString = funcThatReturnsAnOptionalString("parameter"), let secondProcessedString = funcThatReturnsAnOptionalString("another parameter"), !processedString.isEmpty, !secondProcessedString.isEmpty else {
    //handle error if necessary
    return
}
//do stuff

4. Understand that optionals may cause your app to fail differently

To be fair this is less of a problem for Objective-C developers, who are used to the paradigm “sending a message to nil is grand, at least if you’re expecting an object in return, because you’ll just get nil back”. But for developers coming from object-oriented languages such as C++, Java, C#, etc. it might be a bit of a surprise that you get far fewer crashes in Swift apps if you don’t force unwrap, because the language can eliminate a whole class of “null pointer exception” crashes if you use it correctly. This is not the same as eliminating bugs due to unexpected input! Instead of bug reports “the app crashes when I tried to do this”, you’ll start to get reports of “nothing happened when I tried to do this”. Prepare accordingly.

5. Use structs (when you can)

Although Swift supports object-oriented programming with classes, it brings a new more powerful paradigm: protocol oriented programming. This works best with the value types struct and enum and also gets you extra benefits: default initialisers, thread safety, and a way out of the sometimes baroque structures that class inheritance can encourage. But there’s a caveat: note that the two popular object persistence methods used in Swift (Core Data and Realm) use classes. And at the UI layer, UIKit on iOS and AppKit on macOS are all class-based. So your potential for using structs may actually be quite limiting (which can be a bit confusing because most introductions to Swift focus on structs, not classes, which leaves you with a mismatch between learning the language and using it). That said, do try and use structs where you can, especially in the intermediate layers between your view controllers and your model objects.

6. Enum all the things

That’s the title of a talk given by Dave Sims at the Limerick iOS Developer meetup that was so good he famously ended up giving the talk at other meetups in Ireland and the US. Enums are first-class (pardon the pun) value types in Swift that can have functions, properties, and most of the things that structs can have, as well as associated types. They are a tremendously powerful tool when writing swift apps.

7. Avoid AnyObject, and start to think about protocols and generics

Say you have two UITableViews in your app, which are mostly similar – you might be tempted to write one UITableViewController to avoid code duplication. What do you use as your datasource? One way might be to use an array of AnyObjects (AnyObject is a special type which means an instance of any class). Any time you’re relying on run-time type checking is a good prompt to think about protocol orientated programming. At its simplest, create a protocol and make your two types conform to it (they can even be blank). Add any common properties to allow the Swift compiler to check the validity of your code. Generics offer another powerful way of avoiding code duplication and increasing compile time safety. You can even create generic protocols using Protocols with Associated Types.

8. Add defaults to function parameters

Finally a quick one – if a function you’ve written almost solves your current problem, but needs an extra parameter, add it to the existing function with a default value instead of creating a new function. The default value will mean that existing calls to that function won’t break. I’ve often found that making the parameter an Optional type and setting its default value to nil allows an if let in the function body to neatly deal with the extra parameter.

Conclusion

Swift is still a young language, and best practices are by no means settled. To take an example from Erica Sadun’s excellent book Swift Style: An Opinionated Guide to an Opinionated Language, the Swift Standard library team prefer a space either side of a colon in declarations:

class userDetailViewController : UIViewController {

where the Apple developer docs team prefer left-hugging colons almost everywhere:

class userDetailViewController: UIViewController {

(for what it’s worth I prefer the second approach). I don’t even follow all the recommendations listed in this article all of the time in my own code. What’s important is that as you learn the language, you’re aware of working with the language and its opinions, rather than against it.

Starting to break out from the walled garden

The AOL walled garden in the late 90’s. Image from https://invisibleup.neocities.org/articles/4/

When I was at college, I had a night job in a call centre, providing tech support to AOL’s UK customers. It was the late nineties, and although AOL was never as popular in Europe as it was in North America, the company’s vision of a proprietary walled garden giving an ‘internet-like’ experience was still a compelling option for many.

Of course AOL’s influence waned over the years, causing many of us to think that “the internet wants to be free” and that any attempts by a large corporation to supplant the internet was doomed to failure, confirmed by the quick rise and fall of services like Bebo and MySpace.

Now I’m not so sure.

The 2010s have seen the aggressive expansion of social networks such as Twitter and Facebook. When you sign up to these services, you feel like you’re a new customer but of course you’re not, you’re a product. Sophisticated algorithms suck you in to spend more time on these sites, ‘engaging’ more so that you will provide more information to be targeted by ever-more tailored (and profitable) ads.

And the effects have become even more chilling as the algorithms that social networks use spill over into society and politics, encouraging us all to become more polarised, more aggressive and more needy.

There is an alternative, and thankfully there are some very smart people working on it. A return to the open web, where we share for the simple joy of sharing, without profit-maximising algorithms getting in the way.

Host your own content

The basic idea of hosting your own content is not that you pretend that social networks don’t exist, but rather that you take control of your own content, under a domain name that you control, and choose what you share with other services.

I’ve hosted my own content on this site for the past twelve years but for shorter ‘status updates’ I have occasionally used social networks for this purpose.

I’ve created my own ‘microblog’ at https://t.bibby.ie that I will use for short pieces of content, originating from my own site. I will use a new service called micro.blog to share my posts to twitter and (maybe) facebook, but the content will always originate from my own site. If you don’t want to create your own site, micro.blog can do it for you – yes it costs money ($5/month) but if we are to take back control of the web, we need to start thinking like customers, not products. The micro.blog service is still invite-only at the moment, but I have a few invite codes if anyone would like to check it out.

I’m grateful that there are people working on products that can help us liberate ourselves from the walled garden, and I’m hopeful there will always be a place for independence on the web, away from the toxic profit-maximising behaviour of commercial social networks.

Make our street great again

To the project team in charge of redesigning Limerick’s main street:

Please remove through traffic from O’Connell Street.

Expecting people to compete with 1 tonne+ hunks of metal dashing from one side of the city to the other is a recipe for disaster. Let’s admit that “shared space” is the “sorry/not sorry” of urban design when it facilitates traffic throughput.

Let’s be ambitious for our city centre: O’Connell Street has enough room for green spaces, playgrounds, exhibition areas, and much much more.

To attract the best to live, work and learn in our city centre requires a city centre worthy of the best.

I genuinely appreciate the effort that the project team have made to consult the public. I know we share the same hopes and dreams for our city, and I hope you can revise your plans to remove through traffic and centre our great street around our city’s most important asset: our people.

Sent as a submission to the O’Connell St project design team, 29th June 2017.

Don’t use the Force, Luke

(tl;dr: force unwrapping in Swift with ! is bad)

At our last iOS Developer meetup in Limerick, my Worst Case Scenario podcast co-host Dave Sims gave an excellent talk on Optionals in Swift. I’m hoping that Dave will put the content online some day as he really managed to provide a simple yet powerful overview of what Optionals are. He also ended up being invited to give his talk to the CocoaHeads meetup in Indianapolis via Skype – check out Worst Case Scenario Episode 36 for the details about how that came about.

Dave mentioned in his talk that force unwrapping Optionals is generally a bad idea. Xcode doesn’t help the matter though by actually encouraging programmers to force-unwrap optionals:

This is terrible advice for programmers new (or not-so-new…) to Swift. Force unwrapping a nil optional will cause a runtime crash. The whole point of optionals is you make a whole class of bug, where a nil creeps in and causes havoc down the line, impossible. Force-unwrapping throws all of this away with the dubious upside of saving a line or two (and shutting up the compiler). You can argue that the Apple documentation is pretty clear about force unwrapping being a bad idea, but I feel the tools should be encouraging best behaviour. Dave mentioned three common unwrapping techniques:

  1. if let
  2. guard let
  3. The ?? nil coalescing operator to provide a default value

Dave also mentioned Chris Lattner’s recent appearance on Accidental Tech Podcast where Swift’s creator talked about the purpose of the guard statement was to allow early exit (Overcast link that jumps directly to that segment), a style which he is personally in favour of. I also like this style – it reduces the indentation for the main bit of the code. You just have to remember to actually exit the block, either by asserting or returning a default value.

Moving on

For nearly five years I’ve been running my startup Reg Point of Sale. Last week I sent an email round to all my customers to tell them that I’m shutting the business down.

Fortunately, due to the architecture of the system we developed, our customers will still be able to use their systems for the foreseeable future.

It was a tough decision to make. We didn’t take any outside investment, and we don’t owe money to anyone, but it’s still tough to shut down what really was a labour of love.

At some stage I’d like to write up a more detailed retrospective about what went right, and what went wrong. But for now, if you’re interested in more background, I did talk a bit more in detail about this on the latest episode of the Worst Case Scenario podcast that I host with David Sims and Baz Taylor. The relevant section starts about 33’40” in.

Hey Siri, turn the Christmas lights on

My fellow podcast hosts on Worst Case Scenario had both got smart home gear for Christmas. I was feeling a bit left out so I wanted to hack together a low-budget version. Here it is in action:

Now I can use Siri (or the iOS Home app) to turn on my Christmas lights from anywhere!

Ingredients:

  • Raspberry Pi
  • Relay shield from the gear we bought for the hackathon last year (it’s marked FE_SR1y)
  • Christmas lights

The relay shield has a transistor and a diode built in, so I didn’t need to do any fandaglement with a circuit, just hooked up the + to 5v on the Pi, – to Ground, and the switching pin to one of the GPIO pins.

On the Pi, I installed the excellent Homebridge which allowed me to create a HomeKit accessory that Siri can talk to. I used the homebridge-gpio-wpi plugin to talk to the GPIO pins.

It’s obviously super fast on my home wifi network but it’s also surprisingly speedy going through the phone network: iPhone -> phone network -> Apple’s servers -> Apple TV -> Raspberry Pi -> lights takes about 1.3 seconds.

I am toying with the idea of creating a low-budget connected home using 433MHz RF remote controlled sockets which I could control from the Pi with a transmitter board. The other two lads on Worst Case Scenario are also expanding their systems based on the Amazon Echo – subscribe and keep up to date with how we’re getting on!

iOS – checking to see if a word with blank letters is valid using NSSet, NSArray, Core Data and SQLite

Annoying Scrabble words like azo are even more annoying when you use a blank
Annoying Scrabble words like azo are even more annoying when you use a blank

An interesting problem I had recently was to check to see if a string was valid word or not, comparing against a word list with over 170,000 entries.

Checking the string is easy, after loading the words into an NSArray, you can just call containsObject:

NSString *wordToFind = @“the”;
BOOL wordIsValid = [wordArray containsObject:wordToFind];

The code above takes 0.0310 seconds on my iPhone 5s.

It’s even faster with an NSSet:

NSString *wordToFind = @“the”;
BOOL wordIsValid = [wordSet containsObject:wordToFind];

Using an NSSet is over 300x faster than NSArray in this instance: 0.00001s!

Blankety blank

What if you wanted to search to see if the word was valid using blanks, like in Scrabble? NSPredicate makes this very easy:

NSString *wordToFind = @“th?”;
NSPredicate = [NSPredicate predicateWithFormat:@“SELF LIKE %@“,wordToFind];
NSArray *filteredArray = [wordArray filteredArrayUsingPredicate:predicate];

But then I looked at how long it took: 0.44 seconds. Ouch! It’s not acceptable to block the main thread for that long.  There’s a similar method for NSSet:

NSString *wordToFind = @“th?”;
NSPredicate = [NSPredicate predicateWithFormat:@“SELF LIKE %@“,wordToFind];
NSSet *filteredSet = [wordSet filteredSetUsingPredicate:predicate];

which took 0.47 seconds – even worse!

Benchmarking

At this point I decided to benchmark the results by running the test on 500 randomly selected words, where 5% of the letters were turned into blanks. Here’s a sample from my test word list of 500 words:

unclu?tered
succumbs
piggeries
pseu?opregnant
combat?ve

(I have no idea what ‘pseudopregnant’ means…).

Running the test with 500 different words and measuring the time it took to do each test enabled me to record the mean, minimum, maximum and standard deviation of each method. Unfortunately I had to run these tests on the iOS simulator – turns out there’s a subtle bug when repeatedly running an NSPredicate which causes a huge number of NSComparisonPredicate objects to be malloc’ed and not freed, causing the tests above to blow up due to memory pressure on my 5S.

Using the simulator, here’s the results of the test with filteredArrayUsingPredicate: and filteredSetUsingPredicate:

Method Average Min Max Standard Deviation
filteredArrayUsingPredicate:  0.15 0.12 0.34 0.03
filteredSetUsingPredicate:  0.16  0.14  0.54  0.03

Other in-memory methods

I then tried a litany of various methods on NSSet and NSArray to see if I could come up with something faster.

 Using blocks: indexOfObjectPassingTest

    NSPredicate *predicate = [NSPredicate predicateWithFormat:@"SELF LIKE %@",wordToCheck];
    NSUInteger index = [wordArray indexOfObjectPassingTest:^BOOL(id obj, NSUInteger idx, BOOL *stop) {
        return [predicate evaluateWithObject:obj];
    }];
    if(index != NSNotFound)
    {
//can return the found word using index
}

Prefiltering NSArray

This method pre-filters the word list by narrowing it down to only words which have the same first letter (unless we have a blank as the first letter) as the word we’re trying to match.

    NSPredicate *predicate = [NSPredicate predicateWithFormat:@"SELF LIKE %@",wordToCheck];
    
    NSString *firstLetter = [wordToCheck substringToIndex:1];
    NSArray *smallerArray;
    //prefiltering on first letter will only work if first letter is not blank
    if([firstLetter isEqualToString:@"?"])
    {
        smallerArray = wordArray;
    }
    else
    {
        NSPredicate *firstLetterPredicate = [NSPredicate predicateWithFormat:@"SELF BEGINSWITH %@",firstLetter];
        smallerArray = [wordArray filteredArrayUsingPredicate:firstLetterPredicate];
    }
    
    NSArray *filteredArray = [smallerArray filteredArrayUsingPredicate:predicate];
    if([filteredArray count] > 0)
    {
//return matched word here if needed
}

Prefiltering NSSet

    NSPredicate *predicate = [NSPredicate predicateWithFormat:@"SELF LIKE %@",wordToCheck];
    
    NSString *firstLetter = [wordToCheck substringToIndex:1];
    //prefiltering on first letter will only work if first letter is not blank
    NSSet *smallerSet;
    if([firstLetter isEqualToString:@"?"])
    {
        smallerSet = wordSet;
    }
    else
    {
        NSPredicate *firstLetterPredicate = [NSPredicate predicateWithFormat:@"SELF BEGINSWITH %@",firstLetter];
        smallerSet = [wordSet filteredSetUsingPredicate:firstLetterPredicate];
    }
    
    NSSet *filteredSet = [smallerSet filteredSetUsingPredicate:predicate];
    if([filteredSet count] > 0)
    {
//return matched word
}

Using a compound predicate on NSArray

What if we combined the above two methods into a compound predicate – where we check to see if the first letter is the same AND the word is like the word we’re searching for. This might result in a faster match as we can avoid using the expensive LIKE predicate if the first letters are not matching:

    NSPredicate *predicate;
    //if first letter is blank we have to fall back to using normal predicate
    if([firstLetter isEqualToString:@"?"])
    {
        predicate = [NSPredicate predicateWithFormat:@"SELF LIKE %@",wordToCheck];
    }
    else
    {
        predicate = [NSPredicate predicateWithFormat:@"SELF BEGINSWITH %@ AND SELF LIKE %@",firstLetter,wordToCheck];
    }
    
    
    NSArray *filteredArray = [wordArray filteredArrayUsingPredicate:predicate];
    if([filteredArray count] > 0)
    {
//retrieve matched word here
}

Using a compound predicate on NSSet

    NSString *firstLetter = [wordToCheck substringToIndex:1];
    NSPredicate *predicate;
    //if first letter is blank we have to fall back to using normal predicate
    if([firstLetter isEqualToString:@"?"])
    {
        predicate = [NSPredicate predicateWithFormat:@"SELF LIKE %@",wordToCheck];
    }
    else
    {
        predicate = [NSPredicate predicateWithFormat:@"SELF BEGINSWITH %@ AND SELF LIKE %@",firstLetter,wordToCheck];
    }
    
    NSSet *filteredSet = [wordSet filteredSetUsingPredicate:predicate];
    if([filteredSet count] > 0)
    {
//match word here
}

NSArray concurrent enumeration

NSArray also has a method:

- (void)enumerateObjectsWithOptions:(NSEnumerationOptions)opts 
                         usingBlock:(void (^)(ObjectType obj, NSUInteger idx, BOOL *stop))block;

which allows concurrent enumeration when you pass the NSEnumerationConcurrent option:

__block NSString *stringToReturn;
    NSPredicate *predicate = [NSPredicate predicateWithFormat:@"SELF LIKE %@",wordToCheck];
    [wordArray enumerateObjectsWithOptions:NSEnumerationConcurrent
                                usingBlock:^(id obj, NSUInteger idx, BOOL *stop)  {
                                    if([predicate evaluateWithObject:obj])
                                    {
                                        stringToReturn = obj;
                                        *stop = YES;
                                    }
                                }];
    if(stringToReturn)
    {
}

NSSet concurrent enumeration

NSSet offers an almost identical method, except there is no index parameter as NSSets are unordered:

    __block NSString *stringToReturn;
    NSPredicate *predicate = [NSPredicate predicateWithFormat:@"SELF LIKE %@",wordToCheck];
    [wordSet enumerateObjectsWithOptions:NSEnumerationConcurrent
                                usingBlock:^(id obj, BOOL *stop)  {
                                    if([predicate evaluateWithObject:obj])
                                    {
                                        stringToReturn = obj;
                                        *stop = YES;
                                    }
                                }];
    if(stringToReturn)
    {
}

Results

Method Average Min Max Standard Deviation
indexOfObjectPassingTest  0.07 0.0001 0.31 0.04
Prefiltering NSArray  0.07 0.04 0.24 0.03
Prefiltering NSSet 0.08 0.05 0.41 0.03
NSArray compound predicate  0.09 0.06 0.33 0.03
NSSet compound predicate 0.10 0.08 0.26 0.03
NSArray concurrent enumeration  0.10 0.004 0.36 0.06
NSSet concurrent enumeration 0.10 0.01 0.31 0.06

We have managed to do a bit better with some of these methods, but we’re still nowhere near acceptable performance. Maybe Core Data might have some under-the-hood optimisations that might help us:

Core Data

I set up a simple Core Data stack, with one entity “Word” which had one attribute “word” (naming things is hard…). Here’s the method to retrieve a matching word in my data controller:

-(Word *)wordMatchingString:(NSString *)stringToMatch
{
    Word *wordToReturn;
    NSFetchRequest *request = [[NSFetchRequest alloc]init];
    //we want to retrieve all things
    NSEntityDescription *e = [[[persistenceController mom]entitiesByName]objectForKey:@"Word"];
    //set the entity description to the fetch request
    [request setEntity:e];
    NSPredicate *predicate = [NSPredicate predicateWithFormat:@"word LIKE %@",stringToMatch];
    [request setPredicate:predicate];
    //limit to one result
    [request setFetchLimit:1];
    
    NSError *error;
    //execute the fetch request and store it in an array
    NSArray *result = [[persistenceController mainContext] executeFetchRequest:request error:&error];
    //if not successful, throw an exception with the error
    if(!result)
    {
        [NSException raise:@"Fetch failed" format:@"Reason: %@",[error localizedDescription]];
    }
    
    //return the only object in the array
    wordToReturn = [result lastObject];
    return wordToReturn;
}

And here’s the method to match the word:

    Word *foundWord = [[WSDataController sharedController]wordMatchingString:wordToCheck];
    if(foundWord)
    {
}

The results for this were all over the place:

Method Average Min Max Standard Deviation
Core Data  0.10 0.001 0.76 0.06

I had high hopes for Core Data but it’s just not fast or consistent enough for this application.

Descent into SQLite

I was about to give up at this point. I asked Dave Sims if he had any advice and he suggested implementing a Directed Acyclic Word Graph (DAWG) which would be very efficient for finding matches – despite the opportunity to make “yo dawg” jokes I reckoned it might be a bit above my pay grade for some weekend pottering.

I’d read about some iOS developers preferring to deal directly with SQLite for their persistent storage, rather than using Core Data (which uses SQLite as one of its storage options). Relishing the opportunity to type in all caps, I followed a tutorial to get a basic SQLite setup going (the tutorial has a small error in, which XCode will catch with a warning, the solution is to replace the offending line with if (sqlite3_step(compiledStatement) == SQLITE_DONE) { ). Here’s my SQLite query:

    //SQLite uses underscores as the single character wildcard
    NSString *underscoreString = [wordToCheck stringByReplacingOccurrencesOfString:@"?" withString:@"_"];
    NSString *query = [NSString stringWithFormat:@"SELECT word FROM words WHERE word LIKE '%@'",underscoreString];
    NSArray *resultsArray = [[NSArray alloc] initWithArray:[self.dbManager loadDataFromDB:query]];
    if([resultsArray count]>0)
    {
}

This gave the following results:

Method Average Min Max Standard Deviation
SQLite 0.017 0.0014 0.056 0.004

Success!

Using SQLite not only resulted in faster matching, it was also much more reliable than any of the other methods, with a very small standard deviation. Obviously matching a string in an array of 170,000 strings is an edge case, and for most cases any of the other methods would have sufficed.

It’s also worth noting that I was able to run the Core Data and SQLite tests on my own iPhone 5S as they do not suffer the same NSPredicate memory leak as the other methods. Here are the results on the phone:

Method Average Min Max Standard Deviation
Core Data (iPhone 5S) 0.35 0.002 0.78 0.19
SQLite (iPhone 5S) 0.037 0.036 0.042 0.0006

Full results

Method Average Min Max Standard Deviation
filteredArrayUsingPredicate:  0.15 0.12 0.34 0.03
filteredSetUsingPredicate:  0.16  0.14  0.54  0.03
indexOfObjectPassingTest  0.07 0.0001 0.31 0.04
Prefiltering NSArray  0.07 0.04 0.24 0.03
Prefiltering NSSet 0.08 0.05 0.41 0.03
NSArray compound predicate  0.09 0.06 0.33 0.03
NSSet compound predicate 0.10 0.08 0.26 0.03
NSArray concurrent enumeration  0.10 0.004 0.36 0.06
NSSet concurrent enumeration 0.10 0.01 0.31 0.06
Core Data  0.10 0.001 0.76 0.06
SQLite 0.017 0.0014 0.056 0.004
Core Data (iPhone 5S) 0.35 0.002 0.78 0.19
SQLite (iPhone 5S) 0.037 0.036 0.042 0.0006

All hail the new ecosystem wars

pixelYesterday Google announced a new phone. This event wasn’t unusual, Google have announced many phones in its flagship Nexus line in the past – however this time Google proudly announced that they had designed their own phone, rather than commissioning it from a third party manufacturer. Now Apple is not the only player who claims to control ‘both the hardware and the software’. Although I develop for Apple platforms I thought Google’s announcement was intriguing, and if we really are at the start of a new phase in the ecosystem war, it could be really good for pushing the smartphone and related devices forward.

As an aside – We normally discuss events like yesterday’s Google announcement of new phones and other products on our Worst Case Scenario podcast with Baz Taylor and Dave Sims, and Atlantic 302 with Pat Carroll. Yesterday’s announcement intrigued me, and I wanted to jot down some thoughts ahead of recording our next episodes. So if you think any of the reflections below are wide of the mark, I’d encourage you to take a listen to Worst Case Scenario or Atlantic 302 – chances are that either Baz and Dave or Pat will have picked up on any of my spoofing! We’ll be releasing episodes soon that will discuss the Google announcement from a tech and business strategy perspective respectively, I’ll update this article with links when they drop.

The phone isn’t impressive, but the future might be

On first look, the new Google Pixel phone is embarrassingly similar in design to the iPhone 6/6S (granted, I would say the same about the iPhone 7, but c’mon Google, let’s bring a little design innovation here) but if this is a real push by Google to design phones then it makes sense to start out conservatively, and expand over time. If Google are really serious about this, they could bring some much-needed competition to Apple at the high-end of the smartphone market as they develop future models.

The mid-end of the market is there for the taking

The new Google Pixel phone is identically priced to the iPhone in almost all markets. For many people, in many markets, the pricing of the new iPhone and Google Pixel is far too high. We’re also at the stage where, thanks to large leaps in CPU/GPU/storage speed in the last few years, users don’t necessarily need the latest and greatest in technology to have a great smartphone experience. If Google are smart, they will continue to innovate at the high end, but will bring in a newer line of mid-range smartphones over time. Apple half-heartedly addresses this market with the iPhone SE (which I think is a great phone, but it’s Apple’s first device to address the mid-market that isn’t just continuing to sell an older model) but is pretty obviously chasing profit margins, not market share. Obviously there are other manufacturers competing at both the mid and the high end of the smartphone market, but the fragmentation of the market combined with the tendency of carriers and manufacturers to abandon phone models and not give software updates means that there is a real opportunity for Google here.

Apple’s early lead in the connected home market is toast

It frustrates me that Apple came out with a wonderful product for wireless audio a whole twelve years ago with the launch of the AirPort Express. This wonderful device allowed you to wirelessly play music from your Mac (and later, when it was released, your iPhone) to your stereo. It added the ability to play to multiple AirPorts back in 2006, only a year after Sonos had launched their first product (which was expensive and buggy at the start). Since then, Apple have virtually ignored this feature, and certainly failed to build other ‘connected home’ features to it. The AirPort express was last updated in 2012, and doesn’t support modern WiFi standards like wireless ac (introduced in the iPhone 6). Even more inexcusably, Apple never bothered bringing multi-zone AirPlay to iPhones and iPads, despite the fact that modern iPhones have vastly more power than the Macs that supported it back in 2006. Google have launched a new wifi router which uses modern mesh networking techniques (like Eero and Ubiquity), as well as expanding their Chromecast range, which, despite not having native iOS integration, is a better buy these days than the 4 year-old AirPort Express.

How Google might mess this up

All is not lost for Apple however. Google have a bad reputation of throwing things to the wall and seeing what sticks, abandoning products and allowing overlapping products to co-exist (the mess that is Hangouts, Voice and Allo being a prime example). It’s also interesting that these products where released by Google, as opposed to the Nest subdivision of Alphabet that was supposed to focus on consumer hardware. Google’s may not be able to overcome its creepy tendencies and think of its consumer hardware devices as simply another tool to slurp up information about its users. Apple has a stubbornness that can be a strength when it comes to iterating on products and technologies until they are good enough, especially in the hardware space.

The victor in Google v Apple might be us

Ultimately, competition is good. I have never been tempted by any of the flagship Android offerings over the years, including the latest Google Pixel, but that may change in the future. Apple as a company needs good competition to spur them on. In particular I’m hoping that Google’s WiFi and Chromecast products will force Apple to start competing in this market again. An interesting possibility is that with Google becoming an integrated phone manufacturer (despite protestations that the hardware and Android teams are completely separate) is that other Android manufacturers might be tempted to go full in on a third mobile phone operating system, a move that would encourage open standards and which might spur even further innovation.

How I edit podcasts

I do much of the post-production editing for the two podcasts that I’m on: Worst Case Scenario with Baz Taylor and David Sims, and Atlantic 302 with Pat Carroll. Some people have been kind enough to compliment the production quality of the podcasts but I’m still frustrated when I hear how good some of my favourite podcasts sound. I’m still learning about this stuff, and I know I have a long way to go, but I thought I would document my current method. Critiques welcome!

I’m indebted to Jason Snell of SixColors who wrote a very detailed guide on how he edits his podcasts. Much of this guide is based on Jason’s recommendations. Like Jason, I use Logic Pro X for editing, specifically for its Strip Silence function which isn’t available in GarageBand. I add an extra pre-production step of noise reduction in Audacity, as an unacceptable level of hiss was creeping into the recordings without this.

Recording

We sometimes record in person, but most of our episodes are recorded remotely. We sit in our own houses and chat over Skype. We don’t actually record the Skype call, instead we each record our own audio locally and then I mix it together later. We have a pre-recording ritual which consists of three steps:

  1. I count us in to hit Record in GarageBand. This means that the audio tracks will start at roughly the same point.
  2. I ask for five seconds of silence at the start of recording. This provides a convenient place at the start of each track to get a sample of background noise for noise reduction.
  3. I turn up the volume on my headphones, hold the headphones to my mic, and ask each host to talk into their microphone separately. This marks a point where the same audio is appearing both on my audio track (through the headphones) and on the host’s track, identical audio which allows me to sync up the tracks. Of course it’s not truly simultaneous as Skype introduces latency, but it’s good enough.

Sharing

The other hosts on the podcast then send their GarageBand files to me. Baz and Dave are users of the native macOS Mail app, which has a feature to send large files called MailDrop. For Atlantic 302, Pat and I have a shared Dropbox folder that he copies his file over. I mention this because the files involved are large: Atlantic 302 is a 30-minute show and our separate audio files are about 270Mb. Editing podcasts takes up a lot of disk space! (As a side note, I save my working podcast files in a subfolder of Downloads, and exclude the Downloads folder from Time Machine, to prevent backups getting too big).

Export

GarageBand has a default voice recording setting, which adds lots of effects such as reverb. The GarageBand interface is a little confusing, so I created a blank ‘patch’ with no effects whatsoever. I apply this patch in the Library before exporting. I’ve uploaded this blank patch in case you want to use it: it needs to be copied to
~/Music/Audio Music Apps/Patches
– you’ll need to create this folder if it doesn’t exist already.

I export each GarageBand file using the Share > Export Song to Disk menu command, and I save as an uncompressed 16-bit AIFF file. Rather annoyingly, this step creates a stereo AIFF file even though the recorded track from the microphone is mono, I haven’t bothered to figure out how to change this, so I just continue the rest of my workflow in stereo, and then convert to a mono MP3 file at the end.

Noise reduction

Our recordings tend to contain some hiss. If this hiss isn’t removed, it can be amplified at a later stage when applying compression. I use Audacity’s noise reduction function to pretty aggressively reduce noise in the AIFF files before dragging them in to Logic. I open a blank audio file, hit Cmd-Shift-I and select the AIFF for editing. I find the section of the audio at the start where everyone is silent, and select most of it, checking that there isn’t any breathing or noise that isn’t present through the whole recording. Then I select Effect > Noise Reduction and click Get Noise Profile to sample the background noise. I deselect the audio previously selected (important to do this, or the next step will only reduce the noise on the selection) and select Effect > Noise reduction again. You’ll notice there are lots of parameters to adjust, here are the settings that I use, chosen out of trial and error:

noise-reduction-progressNoise reduction (dB): 24; Sensitivity 11.50; Frequency smoothing (bands): 10. The last option, Noise: , should be set to Reduce, not Residue.

This takes a long time, typically nearly two minutes on my 2015 MacBook Pro. I use this time to start exporting the other tracks from GarageBand, and dragging completed files into Logic Pro X. Once the noise reduction has finished, I choose File > Export Audio, and export as another 16-bit AIFF file.

Logic Pro X

Logic Pro X costs €200 in the Irish Mac App Store. I know this is a huge amount – which is why I edited the first few episodes of Worst Case Scenario in Audacity. But Logic has one feature called Strip Silence which saves a huge amount of time and I’m much quicker at editing once I switched.

To start, create a project in Logic. It doesn’t matter what input settings you use, as you will be importing the previously-recorded tracks. Drag your tracks in, you can even drag them in one go, if you do just make sure that you ‘create separate tracks’ in the dialog. Once you have done this, you can delete the original empty track that Logic added for you.

Syncing audio tracks

By now you should have your three audio tracks in Logic. To sync them up, I look for the audio at the start of the file where each host speaks into my mic via my headphones (described earlier). I drag each track to the left and right around this point until the audio is roughly in sync. I find it helpful to zoom in (by pinching the trackpad) to do this, you can sometimes align visually using the waveforms on the track.

Audio effects and EQ

The Inspector sidebar in Logic, showing the Compressor, EQ and Noise Gate applied to Dave's audio (left) with the master Compressor on the right
The Inspector sidebar in Logic, showing the Compressor, EQ and Noise Gate applied to Dave’s audio (left) with the master Compressor on the right

I apply previously saved patches for my co-hosts. You can download these patches: Thomas, Dave, Baz, Pat (you add these to the same folder as the blank patch explained above – Logic and GarageBand use the same plugins in the same folders). These patches contain an EQ (to emphasise and de-emphasise different sound frequencies), a compressor (to make quiet noises louder) and a noise gate (to ignore sound below a certain level). They are different based on certain factors, Baz, Pat and I use the same Pyle dynamic microphone but Baz and I both have fairly quiet voices, so they need boosting. Dave uses a Rode condenser microphone, which provides a lovely loud sound (the Pyle mics are very quiet) but tends to pick up more background noise. For my own patch, I added an extra DeEsser plug-in, as my ‘s’ sound is very hissy. Finally I add a master compressor on the output (it’s listed under Dynamics > Compressor) which effects all tracks, I turn up the compression to 3 and leave everything else at the default.

I try to avoid any clipping on the output meter (any positive number is clipping and will be shown in red), and will tweak the individual track compressor if someone’s audio is clipping, normally using the Make Up knob.

Strip Silence

Strip silence has been applied on Dave and Baz's tracks, but not on mine
Strip silence has been applied on Dave and Baz’s tracks, but not on mine

Now we get to where Logic earns its €200: the Strip Silence feature. This essentially translates a single continuous audio track into ‘blocks’ where someone is actually talking. This makes editing so much easier. The keyboard shortcut for Strip Silence is Ctrl-X (make sure you have the track highlighted), the settings I use are a tweaked version of Marco Arment’s recommendations – 2%/0.6/0.2/0.3 for those of us using dynamic microphones. I change the threshold to 4% for Dave’s microphone: as a condenser it picks up more background noise. As Marco points out, annoyingly Logic doesn’t remember your choices so you have to manually enter these in for every editing session.

Select following

At this point I delete all the chatter before the show starts, and I start editing proper. The one disadvantage of using Strip Silence is that you are left with a lot of empty sections where there are sections of the podcast where nobody is talking. This can be noticeable to the listener (depending on the background noise), so a lot of my editing work involves moving audio ‘back’ so there is no overlap. When you are faced with a period of silence, you want to select the next ‘block’ after the period of silence, and then use the Shift-F to select all blocks of audio following this block. You can then drag the audio back so that it cuts in just after the last person has finished talking.

Editing

As well as removing silences, you’ll often want to cut bits out. In the early days of Worst Case scenario, I was obsessive in cutting every last ‘um’ and ‘ah’, especially with my own voice. I’ve calmed down a bit now I’ve got used to listening my own voice (this took quite a while) and realising that these artefacts appear in everyday speech and the brain is incredibly good at filtering them out.

secondary-editing-toolI do remove any bumps, clicks or any other background noise that I think might distract the listener. Often the noise is in a single block so I can just click and delete. Sometimes you need to divide up a block of audio, the fastest way to do this is to set Logic’s secondary editing tool to be the marquee tool (see graphic). The primary tool should be the pointer tool. The secondary editing tool is invoked by holding down Cmd. You can use the marquee tool to drag over a selection of audio you want to delete, or by clicking once and hitting backspace you can split a block into two pieces.

Intros

On Atlantic 302 we use a spoken word intro and outro that myself and Pat take turns doing and which is recorded at the same time as each episode. For Worst Case Scenario, we use a short noise that Dave created while messing around with an app called cfxr, a little app that creates various sound effects. I turn the input volume down to -13db because otherwise the master compressor will make it too loud.

Export to iTunes

I prefer to export the song to iTunes as an AIFF (File > Share > Song to iTunes). The only metadata I add is the track name. From iTunes I have the export format set as 64 kbps mono (iTunes > Preferences > CD Import Settings > MP3 Encoder > Custom > 128kbps stereo bit rate + mono) so I just convert to MP3 after the AIFF is imported from Logic. Finally I right click the MP3 and Show In Finder to locate the file.