Category: Computers

  • Building the Oscar Contest entry form

    As always, I like to use the Oscar Contest as a way to try to learn something new. This year I decided to build it using AWS Amplify, a set of tools and services that can be used to quickly build and host mobile and web apps across a range of frameworks. I’ve somehow managed to avoid touching React, so I figured I might as well use that too. Here’s the basic architecture I went with:

    Contest architecture

    I figured it might be fun to walk you through the process I used in case you’d like to try something similar. The first step (if you haven’t already) is to install the AWS CLI and Amplify CLI and get them configured with your AWS account credentials. Then I created the basic React project using this command:

    npx create-react-app oscars2021

    That will download a bunch of stuff and set up the basic project files for you.

    You can then switch into that directory and start the app to verify it’s working.

    cd oscars2021
npm start

    Running React app

    Now it’s time to hook your project up to Amplify, using this command:

    amplify init

    A wizard will walk you through setting up various parameters for your app, including your preferred code editor and the type of app you’re building. Here’s what I selected:

    Amplify configuration

    This will initialise your project in the cloud and set up some resources for you.

    Now it’s time to add storage, which in my case meant an Amazon DynamoDB table. This is where I’d be storing each entry as it came in.

    amplify add storage

    This will again kick off a wizard that will walk you through some configuration options. I selected “NoSQL Database” and then set up the columns that I wanted in the table.

    Database config

    I also selected “id” as the partition key, with no sort key, no secondary indexes, and no Lambda trigger.

    More DB config

    The next step is to add the API and Lambda function that will actually record the user’s entry. This is done with the command:

    amplify add api

    Again, a wizard will walk you through configuration. I created a REST API with the path “/entry” and created a new Lambda function using the Serverless ExpressJS template. I also gave the function permission to Create and Read from the storage (aka DynamoDB table) we just set up. I didn’t restrict access to the API, as I want anyone to be able to enter.

    API configuration

    Time to actually update the Lambda function! I went into my preferred code editor (Atom) and opened the project. The function is located in “amplify/backend/function/oscarsfunction/src/app.js”. There are some commented example methods that I deleted and replaced with the code below. This adds the AWS SDK (so I can save to DynamoDB), a method for generating random IDs, and the actual post method to save the entry. (You can download this code from my Github project here.)

    Function code

    The next step is to use Amplify to push the newly created backend storage, API, and function to the cloud! You can do this with the command:

    amplify push

    Amplify will ask you to confirm which resources you’re deploying, and then it will start the process using AWS CloudFormation. This can take a little while.

    Amplify push

    If you make any changes to the function code later, remember to push the changes to Amplify so they are uploaded to AWS! In the meantime, it’s time to install some dependencies I need for the form frontend.

    npm install aws-amplify @aws-amplify/ui-react
npm install bootstrap
npm install react-bootstrap

    Once all that’s done, you can finally edit the form! The actual React app lives in “src/App.js”. I won’t go through everything I did (you can check the code out yourself), but basically I made sure to include Amplify (so the frontend can talk to the backend) as well as React Bootstrap. I also tweaked the CSS and added a couple images. Each time you save a change, the app will recompile and update in your running browser window. I also opened the “public/index.html” file and changed the title and description of the page.

    Form code

    You can also test out the form in the browser to ensure it’s working. When I opened the AWS Console and looked in DynamoDB, I could see entries being saved correctly into the dev environment table. 🎉

    The final step is to deploy the frontend, and Amplify makes this pretty easy too. I created a new repo at Github and then pushed my code to it.

    Github create repo

    Then I went to the AWS Amplify console and clicked on my app. If you click on the “Frontend environments” tab, you’re presented with a range of options for hosting your app.

    Frontend hosting

    I clicked the one for Github and then went through the process of granting access to my Github account. Then I selected the repo I’d just created with the code for my app, and left the branch set to “master.” On the next screen, I left checked the option to “Deploy updates to backend resources with your frontend on every code commit” and created a new “prod” environment as the target. I also had to create a new IAM role for the deployment process. Once you save and deploy, Amplify will grab your code from Github, run the build script and any tests you’ve configured, and deploy the resources into your account. The build for my app takes less than 4 minutes to complete.

    Completed deployment

    The beauty of the CI/CD pipeline is that whenever I modify the code and push it to Github, the whole process will kick off automatically! The Amplify console also gives me the URL to the hosted app, which is where you can enter the contest. When I check DynamoDB now, I can see entries coming through to the prod environment table. When the contest is finished, I can shut down and remove all the app resources by simply running this command:

    amplify delete

    If you’d like to try out Amplify yourself, I can recommend a couple resources. The AWS website has a very simple, step-by-step tutorial to Build a Basic Web Application that you can work through, but it doesn’t include React or the CI/CD part. If you want to copy what I did, check out my colleague Marcia’s YouTube videos  on  building a Contact form with React and automating your CI/CD deployments, which gave me the basics of everything I needed to build the contest entry form. Thanks Marcia!

  • Photo Post

    The Snook bought me a new mouse as an early birthday present. This sucker is 100% going to give me carpal tunnel, and I don’t even care. 🎮 #nes

    The Snook bought me a new mouse as an early birthday present. This sucker is 100% going to give me carpal tunnel, and I don’t even care. 🎮 #nes

  • A new home for w-g

    Well, that was an ordeal.

    A few weeks ago I got an email that my shared web hosting account (that I use for both this site and RoaldDahlFans) was up for renewal, and it reminded me that I didn’t like the host I was using. I was stuck on PHP5 and they were going to charge me more to move to a server with PHP7, and that pissed me off. So I asked Twitter what to do.

    Wordpress hosting tweet

    I got a lot of different answers, most of which were a LOT more expensive than I was spending. I was okay with going up a little bit, but these are both hobby sites rather than professional so anything over, say, $20/month was too much. A few folks threw out the suggestion to use Amazon Lightsail, which was intriguing. After all, it’s always a good idea to eat your own dog food. (I should also mention that these sites are and have always been hosted and paid for by me personally, so they’re not running on my employee AWS account or anything like that.)

    I thought I’d document the process that I followed for the sites, which was based on this blog post and this AWS tutorial. Truthfully, setting up a new WordPress site on Lightsail is super easy and takes less than hour. All the complexity was because I was migrating two very old and very crufty sites from PHP5 to PHP7 and trying to preserve twenty years’ worth of old URLs. So even though I encountered a few hurdles not described in the tutorials, I wouldn’t let that stop you. Read on for the details…

    (more…)

  • Photo Post

    I made dinner! Pressure Cooker Butternut Squash Risotto With Frizzled Sage and Brown Butter from @seriouseats… (I added some chicken!)

    I made dinner! Pressure Cooker Butternut Squash Risotto With Frizzled Sage and Brown Butter from @seriouseats... (I added some chicken!)

  • Photo Post

    I think my @jewelbots battery is dead. 😕 But on the plus side, pleasantly surprised to crack it open and see that it looks replaceable! Great job @sarachippsy 👏

    I think my @jewelbots battery is dead. 😕 But on the plus side, pleasantly surprised to crack it open and see that it looks replaceable! Great job @sarachippsy 👏

  • Knitted Disruption

    Hey knitters! I’m working on a knitting + machine learning project and I need a collection of images of stockinette, garter, seed, and moss stitches. Images like these:

    If you’re willing to spend a few minutes helping, I’d be so grateful! Just email your images to [email address removed as the project is finished!]. You can attach multiple images at once if you like.

    They don’t have to be swatches; they can be closeups from finished articles. They don’t have to be perfectly straight or blocked or anything like that either. I’m looking for a wide variety, to be honest! Stripes and multiple colours are great! Even fairisle. I want to teach the model to disregard colour, so having photos with it is very helpful. Just no lace or cables or crochet… (yet).

    More details

    A few years back I gave a talk at several tech conferences about the overlap between knitting patterns and programming languages. As part of that, I talked a bit about KnitML (an old proposed standard for writing patterns in ways computers can understand) and how it could be used with special software to simulate knitted fabric. A few people asked me afterwards if it could go the other way – from a photo of knitting, can you reverse engineer the pattern?

    It got me thinking. I know that I can “read” knitting. I do it all the time, and I’m sure other knitters do too. So if I can do it, why not a computer? I’m also fortunate in that I work with some very smart Machine Learning experts and have access to run experiments in the Cloud cheaply. So I decided to give it a try.

    I’m starting with image classification. Think facial recognition for knitting swatches. Can I train a model to recognise the difference between stockinette, garter, moss, and seed stitch? The first step is gathering as much training data as I can, hence my request to you all!

    So the more the better. Different wools, different colours, different lighting, different angles. It’s all super useful!

    And I will definitely share the results back afterwards and thank all contributors. 🙂

  • Meetup Wrap – March 18, 2018

    Apologies for the extended radio silence! It’s been a busy couple of weeks and I’ve been travelling all over.

    On March 1st I headed to Perth for our very first YOW! Night of the year. Evan Leybourn was presenting on “business agility” to an eager crowd at Bankwest headquarters.

    If you want to learn more about business agility, you’re in luck – we’re bringing the Business Agility Conference to Australia this year!

    I also got to let everyone know the big news that YOW! West is now YOW! Perth, and it’s moved to September to line up in a tour with Singapore and Hong Kong. Should be fun and allow us to bring even more amazing folks to that part of the world!

    After Perth I headed to INDIA for the first time ever! I was speaking at Agile India 2018 in Bangalore. I didn’t get to see much of the city outside the conference, but the speakers and the community were wonderful. It was also nice to catch up with so many folks from the extended YOW! family like James Stewart, Jeff Patton, and Fred George.

    I was still fine tuning my own presentation, but I did manage to catch some great talks over the three days I was there. One of my favourites was from Dan Makoski, the VP of Design at Walmart. Dan talked about making design more human, humble, and accessible, I especially liked this observation:

    “Only two industries call their customers ‘users.’ Software development… and the illicit drug industry.” – Dan Makoski

    Dan actually had us do some creative exercises during his presentation. In one of them, we used small cans of PlayDoh to imagine a futuristic toothbrush. (Mine was a pill “full of nanobots” that you chew on. 🤖) In another, we had 60 seconds to draw the person next to us. I made a new friend in the process!

    There was a fascinating panel on diversity in leadership facilitated by Teresa Brazen from Cooper in San Francisco. The panelists were Reema Diwan of Taj Resorts, Poornima Girish from Shell, Dr. Sujitha Karnad from Sekai Solutions, Sue Cooper from Cooper, Mary Wharmby from BBVA, and Deepa Bachu from Pensaar.

    My new friend Sohan Maheshwar from AWS gave a talk on building voice-enabled experiences with Alexa. Obviously this is a topic in which I have some interest! There are interesting challenges around feature discoverability when you talk about “zero UI.”

    Jeff Patton‘s session was super popular, as always. He uses a technique where he mixes slides with live sketchnoting, which he projects using a fixed camera above the lectern. It’s a great way to keep an audience engaged with that you’re saying!

    I was fascinated by Nils Kappeyne‘s talk about Shell, one of the largest companies in the world. (Roald Dahl worked for Shell in Africa during the 1930’s!) They’re investing huge amounts of money in IoT, AI/ML, renewable energy, and even working to disrupt themselves in various ways.

    My talk was on Friday, and I started the day with an EPIC STACK – falling flat on my face in front of half the audience! Luckily only my pride was bruised, and I had plenty of time to recover before I spoke. 😂

    The focus on the last day was on DevOps and Continuous Delivery, and it started with a keynote from my friend Gregor Hohpe. Gregor’s talks are always entertaining and filled with useful info! I especially liked his challenge to those who judged their current level of test coverage acceptable.

    My session was called “Building Software That Lasts,” and you can see my slides here. I said that too often in technology, the immediate response when inheriting legacy code is to chuck it out and start over. This is rarely a good idea. I talked about some of the trade-offs and choices you can make to build software that lasts and evolves over time. I had a small but engaged crowd, and I got some good questions after too! I even convinced someone he needs to look into property-based testing.

    I also saw an informative session from Nikhil Barthwal on chaos engineering. One technique he mentioned in particular was running Game Days with your team – events to simulate how you’ll respond in the event of a real service failure.

    My friend James Stewart from the UK spoke about cloud security. James said that security is about gaining trust through demonstrating competence, and every breach you suffer erodes that trust. Customers these days have more and more options, and they will leave (that even goes for government sites).

    The award for my favourite talk title at the conferences goes to Thierry de Pauw for his session “Feature Branching is Evil.” Thierry told us about his experience introducing version control to a team that had never used it before, and he made a lot of good arguments in favour of trunk-based development.

    After Agile India, I headed from Bangalore back to Singapore for a couple more meetups. The first was the excellent Women Who Code Singapore, where I spoke on giving “Better Tech Talks.” Thanks to the excellent folks at Engineers.sg, you can watch it online now! The night also featured a panel of folks covering other aspects of speaking – like coming up with ideas and handling Q&A sessions.

    On the last night of the trip, I attended the Junior Developers Singapore meetup. This is an offshoot of the community that started in Melbourne and has now spread to Perth and beyond!

    There were four excellent talks on the night, but the one that really touched me was by Mai Jianlong. He spoke plainly and bravely about his struggles with depression, and he gave advice to those starting out in tech for how to deal with mental health issues and recognise when you need to get help. It’s a powerful talk, and you can watch it online here:

    It was a wonderful but exhausting two weeks, and I’m definitely happy to be home now!

    Other Stuff

    • The Call for Presentations for YOW! Data and YOW! Lambda Jam has been extended to Friday, so you’ve still got time to get in a submission! Don’t delay any longer though…
    • We have several upcoming YOW! Nights happening in Brisbane, Sydney, Hong Kong, and Perth over the next few weeks – including a very special one we’re doing in conjunction with the AWS Summit!
    • Did you read this Jon Skeet’s post on Stack Overflow Culture? He proposes a covenant for both askers and answerers that basically boils down to: “Don’t be a jerk.” AskMeFi has struggled with this one as well and has some nice guidelines that are worth a look.
  • Weekly Meetup Wrap – February 25, 2018

    Does 2018 seem to be accelerating for anyone else? I went to FIVE tech events last week. 😲

    The first one was GraphDB Sydney – a rare Monday meetup. After a long hiatus, the group reformed late last year and, though it’s still small, it seems to be well supported by Neo4j and Ansarada.

    Josh Yu from Neo4j gave an excellent overview of APOC – which stands for “Awesome Procedures On Cypher.” Cypher is Neo4j’s graph query language, and APOC is a library of more than 300 custom procedures that add cool functionality. Josh ran through a live demo that used clustering to aid in financial fraud detection. I learned a heap from this talk – including that yes, the founder of Neo4j was a big fan of The Matrix. 🙂

    On Tuesday evening I headed over to BlueChilli for the SheStarts Mentor Mixer. This is a program to support startups with women founders, and this is my second year as an advisor (along with some amazing peers). I met with two of the founders to hear about their projects and offer them advice. Interestingly, both women I spoke to asked me whether I thought they should do a coding course. (They were both non-coders.) My advice is that something basic like NodeGirls would be a good idea to give them confidence in working with developers – demystifying the black magic of coding, as it were – as well as empathy and understanding for the skills those folks will bring to the team. I also really enjoyed catching up with some of the alumni from the first SheStarts cohort, like Jessica Christiansen-Franks from Neighbourlytics.

    Wednesday I headed over to Web Analytics Wednesday for what proved to be a very popular session!

    A panel of four experts spoke about all things content engagement: Dominic Laforgia, Head of Data & Insights at Fairfax; Mackenzie Stratford, Digital Analyst and Product Manager at News Life Media; Leon Bombotas, Chief Data Officer & Founder of Newsroom.ly; and Nathan Scully, Senior Analytics Manager at Oneflare. Meetup organiser Simon Rumble fielded questions from the audience on whether dashboards are important, what metrics are useful for different types of businesses, all the different platforms publishers have to support, and how they plan to handle future technologies like “zero UI” interfaces. (Spoiler – nobody really has a plan for that last one.) Great discussion!

    Thursday saw me heading to Melbourne for the first time this year. I love hearing what’s happening in the Victorian tech community! Thursday night I went along to the Melbourne Haskell Users Group to learn about “serverless Haskell” from Alexey Kotlyarov (speaker at Lambda Jam 2017!) and David Overton from Seek.

    The project came out of a Seek Hackathon last year, and it involves wrapping your Haskell code in an executable with Node.js and deploying it to AWS Lambda. They’re already using it in production! Pull requests are welcome. 🙂

    Not going to lie though… The best part of MHUG was going out to dinner with a dozen other folks afterwards for a Malaysian feast!

    Friday night I had an unofficial DevRel meetup with some of the awesome folks in the Melbourne community. It was a floating bar, and there were a lot of beers, french fries, laughs, and late night burgers. (Have I mentioned lately how much I love my job?) ⛵🍻❤

    Saturday was the main event of the week for me: MeasureCamp Melbourne! This “unconference” style event brought together lots of folks from the data engineering, analytics, product, and UX worlds for a full day of talks and discussions.

    After the morning welcome and kickoff, attendees were invited to propose sessions and tack them up on a large schedule for the day. With 5 rooms and 8 slots, there was room for up to 40 different sessions! I decided on the spur of the moment to give a talk on giving “Better Tech Talks – How to do public speaking without sucking.” 🙂

    For the first session, I went to see Scott Sunderland, founder of Tribalism, try to convince us that humans are better than computers. Scott argued that there are things humans are really good at – like filtering out unnecessary information, making intuitive leaps – that computers just can’t do. He also made the analysts in the room happy by predicting that their jobs are the least likely to be automated away in the near future!

    The second talk I went to was from Val Lyashov at Envato talking about how they do analytics. I was impressed that Val presented with just a whiteboard – no slides! He gave a good overview of how they tie their various systems together and answered a lot of questions from the audience. (Oh, and they’re hiring!)

    My last talk before lunch was my new friend Mike Robins from Snowplow. Mike was speaking about a very important topic: General Data Protection Regulation (GDPR). These regulations come into effect on May 25, 2018 for any online business that collects personal data on users in the European Union. It outlines some very specific things you need to do, and the worry is that very few Australian businesses are prepared for it! (My takeaway was: Damn, I’m really happy I’m not in charge of an online business anymore!)

    I spent the first session after lunch frantically preparing for my own talk! Soon it was time to head to the room…

    I essentially went through all the points from my recent LinkedIn article on public speaking. I ended up with about 20 attendees, and they seemed to really enjoy and get a lot out of it! Hopefully some of them will be inspired to volunteer to give talks at their local meetups and events in the future. 🙂

    The session after mine was a tag team presentation from Priscilla Cheung and Moe Kiss from THE ICONIC. I loved this one! They talked about the importance of good UX when presenting your data, and they walked through 10 tips for making it look better.

    I also loved when Moe got opinionated about pie charts. 😂

    In the last block of sessions, I went to see Sarah Crooke from Data Runs Deep talk about how she uses R to combine data from web and mobile analytics. I’ve seen a few talks on R lately, but most of have been from super experienced data engineers. It was nice to see someone who’s learned it recently and who could give advice to those of us less experienced with the language!

    The last talk I saw was actually a repeat for me – it was Johann de Boer  giving his talk on predictive targeted marketing using machine learning. I saw it at Web Analytics Wednesday back in January, but I felt honour bound to support someone from the Sydney community (especially since he’d caught the train to Melbourne the previous night!).

    Many thanks and congratulations to the MeasureCamp organiser for putting on a fantastic event! 👏

    Other Stuff

    I’m heading off to Perth, Bangalore, and Singapore over the next two weeks, so my meetup adventures are going to be on hold for a bit!

    • It’s all happening at YOW! right now. We’ve got SEVEN upcoming YOW! Nights happening across Australia in the coming weeks. Hope to see you at one of them!
    • QUICK. There is a Humble Bundle of functional programming books, but it finishes up early Tuesday morning! I just bought it…
    • Don’t forget that we have two upcoming YOW! events in Sydney with open Calls for Presentations! YOW! Data will be held on May 14-15 and is looking for speakers on data-driven technologies and applications. YOW! Lambda Jam is coming up on May 21-23 and is all about functional programming. Special note: This year we’ve added on an extra day for LJ that will be a full-day workshop aimed at providing an “on-ramp” to FP. (If you can’t make it to Tony’s 3-day course, you should definitely sign up for the LJ one!)
  • Weekly Meetup Wrap – February 11, 2018

    My attempts to scale back my number of meetups and tech events in 2018 is not exactly going to plan. Six weeks in, I’ve been to 14 different events… which, if I extrapolate, puts me at 120 for the year again. Eeek. But yeah, another four this week!

    The first was React Sydney, kicking off 2018 in the new event space at Domain. Organisers Jed and Jess have done a great job building this community over the years. A large crowd turned up to hear from four different speakers.

    The first speaker was Alex Reardon from Atlassian showing off some of the performance enhancements that have been made to react-beautiful-dnd (a library for making beautiful, accessible drag-and-drop lists with React.js). With really big lists (like, say, 500 items), dragging felt sluggish and janky. Alex showed off a few advanced techniques he and his team used to get it feeling snappy and natural. At it’s core, Alex said, optimisation is “about not doing things you don’t need to do.”

    The second talk was from Ajain Vivek from Yahoo7. Ajain’s fascinating talk was about rethinking how the state of your app stored in an object tree inside a single store can be transformed to memory model. He started off by talking about how human memory works, and how that could translate to a storage model for React. He finished with a demo that earned him applause from the audience.

    The third talk was from Lucas Chen, visiting from Brisbane. Lucas walked us through what’s new in React 16 (aka React Fiber). In a nutshell, there’s been a full rewrite but the API remains the same. Ultimately that means not much has changed in terms of your code, but it got faster! He also gave advice on how you can prepare for future changes.

    The final talk of the night was a last minute addition from another Atlassian speaker – Jamie Kyle. (Coincidentally, Jamie is responsible for at least part of my Twenty addiction because he kept tweeting photos at me of screenshots where he’d scored multiple 20’s. 😂)

    Jamie demoed his new tool Unstated.io (“Because it’s a JavaScript library I had to buy a .io domain. That was a really good use of $65…”), a tool for sharing state between React components. Simple, short, and useful!

    My second event for the week was the Sydney Data Science Breakfast meetup. The theme for this meetup was “AI will take our jobs and that’s ok,” and as you might expect it drew quite a crowd!

    The main speaker was Tomer Garzberg of Gronade, reprising his TED talk from this past December. Tomer began by describing factories in China that are entirely dark with no aircon or running water, and where workers labour 24-7. The catch is that all of the workers are robots. This scenario is becoming more and more common, and soon even white collar jobs will begin to be automated away.

    After his entertaining talk, Tomer joined three others for a panel discussion: Michael Allright from The Minerva Collective, Tim Garnsey from Verge Labs, and Peter Xing from KPMG. The audience peppered them with questions about the ethics of machine learning and AI, the economic impacts of replacing humans with robots, and how individuals and companies should cope with the coming disruption.

    My third event for the week was a brand new meetup: Big Data: Engineers and Scientists. Preact Recruitment are hosting these events and they’ve definitely gotten them off to a good start. It’s always nice to attend a meetup in the beautiful event space at Campaign Monitor!

    The first speaker was Simon Aubury from IAG talking about using Kafka to build streaming data pipelines. This was similar to the talk I heard Matt Howlett give a few weeks back, but Simon included lots of architecture examples that really helped everyone understand why Kafka is so useful. He also mentioned KSQL, which is still in developer preview but has a lot of folks very excited about its potential.

    The second talk of the night was a counterpoint to the first. Raul Beristain from Vocus Communications spoke about SQL on Hadoop using Impala, and what are the pros and cons of this approach. Sometimes the data you’re saving isn’t going to be updated later – like support call logs – so you don’t need a system that supports those transactions.

    The third and final talk was by Campaign Monitor’s own Binzi Cao. Binzo spoke at length about Spark SQL and using it to build a rules engine. He showed off some great examples where Spark SQL can make your life easier, like normalising timestamps from disparate data sources.

    My final event of the week was a hands-on Scala workshop hosted by Women Who Code Sydney at Quantium. I figured after my three-day intensive Haskell workshop, I should keep up my functional programming studies, right? 😜

    There were so many attendees we had to split over two different conference rooms! Marina was our facilitator and walked us through some exercises to build a CLI party planner application. Happily, I found that my experience in the Haskell workshop really helped conceptually with some of the things we did. Where I got hung up was on the particular Scala syntax, and my lack of knowledge around functions available in the core library. (That went for everyone, though.) I did manage to solve the last one entirely on my own, which resulted in a happy dance in my chair! 💃

    Of course, it didn’t help that Quantium’s offices are amazing. They opened the blinds in the conference room during lunch, and WHOA. It’s a good thing I don’t work in this office. I’d stare at that view every day.

    Other Stuff