Where is the data?
When I lived in Stockholm, I used to fly from Stockholm to Frankfurt twice a year. I always wondered when I should buy the tickets though. Is it better to book a flight quite early? Or should I wait and buy last-minute tickets?
As a data scientist I have always been envious of Hopper. They collect millions of time-resolved flight prices to predict the most optimal date for flight booking. I also wanted to have this data! But how?
A Raspberry Pi with a docker daemon
Well, I had this Raspberry Pi lying around on the shelf. After an evening (or two?) of Python coding I had a dockerized web scraper, which collected flight prices from Stockholm to Frankfurt – every twelve hours.
After a month I remembered that I had this project running, downloaded the data (oh, that was a lot) and loaded them in R. Unfortunately, it was really hard to see patterns. Flight prices are obviously dependent on loads of parameters. When I was close to have identified some patterns, SAS – one of the airlines serving the connection – started a promotional campaign… Nevertheless, one thing really surprised me: Flight prices are extremely variable over time.
I decided to visualize exactly this.
-30 days until takeoff
So I had flight prices for a bunch of journeys in early 2019. For most of these flights I had the prices directly before takeoff until up to 60 days before departure. After adjusting different time ranges, I began to get the feeling that prices start going up and down from 30 days before takeoff.
I took a pen and paper and tried different ways of how to plot the price trajectories. Once I drew a circle and it reminded me of a stopwatch, which fitted well to what I wanted to convey. Later I realised that the circle also looks like an airplane engine. Perfect!
JavaScript at its best
After adding interactivity, the visualization got so complex that I was relieved to have used Svelte from the beginning on. It allowed me to modularise the code in a straightforward way, and was able to handle the global state.
