A bit more than four years after its inception, two years after its real start, a thousand deployments, about 10 000 commits, more than 1200 unit and integration tests, more than 100 REST API endpoints, Arcsecond.io is approaching v1.0. What a journey, again!
The more I code arcsecond.io, the more I love it, and feel confidence for the years to come. Since the beginning, I knew this endeavour would last for multiple years, and not just be a hobby project. After all, I’ve been developing a macOS app called iObserve for more than 8 years. Thus, I built Arcsecond.io with these long-term objectives: maximum reliability, permanent deployment capability, permanent clean and tested code. And of course: utmost care for the feedback of users and astronomers worldwide. For this, I knew I will have to learn and master whatever it needs, whatever its complexity or technological layer, and develop and master the permanent operations of a pretty nice full-stack software system. This article explains the important choices in the technical stack used in arcsecond.io.
Let me start by showcasing a bit one thing I am pretty proud of today. Although I am very happy to have ported the core of my macOS app iObserve on the web (check it out), one of the coolest feature of arcsecond.io today is Worldwide Observing Activities, that is, the possibility to know in real-time what is being observed in the (largest but not only) observatories in the world, and some scientific satellites!
Even better: you can subscribe to it, and be notified right away if something matches your subscription. Say, you want to be notified when the HST starts an observation, or when an Programme is being performed for a given observer, or when a telescope instrument is being used, or when a region of the sky is being observed, etc. Boum. You’ve got mail! How cool is that?
To achieve this result, it is a long effort, and a rather lengthy list of decisions, optimizations, trials and errors. But the result is what I wanted: even if the user experience is not perfect (some links aren’t always here, aren’t always pointing to complete info etc), its potential is clearly enormous. Now, I have to « fill » it with more connectors.
Ok, let’s dive in. We’ll start with the backend, then move on with the infrastructure, the frontend and finally the code organisation.
The backend server is written with Django 2.2 and Python 3.7 Originally, arcsecond.io started with Django 1.5 or something, and Python 2. The migration to Python 3 was made more than a year ago, and it was clearly a good choice. Almost no third-party libraries broke during the migration. At the beginning, I evaluated the possibility to use Flask instead of Django. But Django comes with a lot of things built-in, in particular its automatic admin portal, which proved to be invaluable over time.
The design choice of the backend was to make a RESTful purely data-centric server. Hence, the rather obvious choice was to use the fantastic Django REST Framework. It is an essential part of arcsecond.io. It is not the purpose of this article to explain REST principles, but simply let me say it is a mindset, and the DRF allows you to deploy this way of doing with lots of ease.
One of the key reason of using Python in the backend was obviously its seamless integration with scientific and astronomical libraries. In particular, it wouldn’t be possible without AstroPy (and its affiliated packages) and NumPy. Once arcsecond.io is making some benefits (hopefully some day!), I will redirect part of it to help fund these projects.
The projects declares 52 direct dependencies (among which django-cors-headers for managing HTTP headers easily, django-cryptography to encrypt some model fields, django-otp to protect the production admin route with 2FA, django-allauth and django-rest-auth to handle the complex process of auth, django-storages to handle multiple storages remotely, psycopg2 to talk to the PostgresQL DB, requests and pyvo to ease the writing of the multiple connectors…).
More importantly, the backend relies oncelery and channels (with the ASGI daphne server), two libraries of crucial importance.
Celery is a distributed task queue. When I was a newbie at the very beginning of this full-stack journey, I asked myself: Why would I need such a thing? Until I realised that a lot, if not most, of the activity of a backend server could be different from serving HTML pages or serialized data. In arcsecond.io the activity is not stable enough to say if the background Celery worker is more important than the Dahne server. However, with permanent parsing of Data Archives, such as ESO’s, Gemini’s and HST’s, as well as Satellite schedules such as Swift’s, and the creation of a permanent flux of observing activities, a great part of the load of the « arcsecond.io » backend is occuring in the background tasks handled by Celery.
The second library is the official Django-supported project to handle more than just HTTP requests, but also WebSockets. And even if we have only one read-only WebSocket route opened in arcsecond.io so far (the live activities here: api.arcsecond.io/activities/live), the migration from a simple WSGI pure-HTTP server (it was gunicorn for a longtime) to channels and daphne, was not a simple one. We had 2 main difficulties. One was that the channels_redis library (providing the backing store for the channels layer of the server) had a bug that was causing the creation of numerous connections in our infrastructure, way beyond what is expected. For a long time we had various errors pointing in various directions, without knowing the true cause of the problem. But once fixed (recently), the result was quite impressive!
The second difficulty was to understand the underlying mechanism of WebSockets, and how one could provide automatically and similarly serialized JSON data. In other words, how to use our carefully crafted REST serializers to format the data in the WebSocket tube the same way it was for HTTP requests. We use the open-source (very early-stage but clean and well written) djangochannelsrestframework library for that.
As for the tests, we use the pytest library, with the standard python mocks when necessary, and these additional and very useful libraries:
freezegun : to easily write tests anywhere you want in time,
factory_boy : to easily create dozens of instances during your tests setup,
django-naomi : to easily see in a browser during development what the emails your backend sent look like,
vcrpy : to very easily replay the « cassettes », that is, replay real network requests.
Arcsecond.io has two servers: one for the backend, and one for the frontend. The two are deployed on the well-known PaaSHeroku. Why Heroku? Well, because a colleague of mine, when I was working back in Switzerland, just mentionned it’s the easiest way of deploying a server. And it is still mostly true today, I am very happy of this decision (even if today I would consider using a serverless stack, amazed by the products of zeit.co for instance).
Some of the many advantages of using Heroku:
Large marketplace of one-click-installation add-ons for bazillions of auxiliary services (storage, logging, monitoring…)
Deploy with a simple git push.
A very easy-to-use CLI allowing you to interact and connect to your servers.
A very easy way to have your hostnames using certificates (using letsencrypt)
So here is the infrastructure of arcsecond.io:
A Daphne server (one Heroku dyno),
A Celery instance for a background worker (one Heroku dyno),
A celery-beat scheduler (one Heroku dyno),
A production PostgresQL DB instance, automatically backed up every hour,
A RedisCloud instance for the backing store of the Daphne server,
A basic Heroku Redis instance for the HTTP cache,
An Express.js server for the frontend (one Heroku dyno),
Multiple Amazon S3 buckets for the storage of the data (price follows data volume),
The domains are bought on the french cloud provider ovh.com (~40$ / year for a .io domain),
I used to pay the Pro level of GitHub to store private repositories, but then it became free,
There are also 2 other free Heroku dynos used for the staging environnements (one for the back, one for the front).
These are mostly first or second-level entries of the services, and as soon as some money flows in, I will move up the level of some of them. Thanks to Heroku it will be as simple as moving a cursor!
There is one key part of the infrastructure that is not mentionned yet: CircleCI. The code of the backend is never pushed in production directly. It always goes through the automatic testing and deployment by CircleCI on a staging environment, then on production.
So far, I count about 750 deploys for the backend, and 260 for the frontend.
One couldn’t finish this part without mentionning the use of the fantastic crash-reporter Sentry. The richness and the ease of use of this tool is really great, and its help to find and correct bugs is invaluable. Even if Django comes with an automatic emailing when an error occurs (all 500 status code ends up in my Inbox), Sentry helps you read the code stack and identify more clearly what happened. And it also provides very easy management of problems, grouping identical ones, alerting when closed ones reappear etc.
This « software system » is in production since more than a year now, and has handled already quite a few visit spikes (hello ProductHunt...). It is ready to support a much larger load without having to change its underlying organisation.
The frontend is a VuejsSingle-Page Application (SPA). The discovery of Vue.js was simply a life-saver. Originally, the frontend of arcsecond.io was written with AngularJS, the first version of the now-called Angular frontend framework (wich is one of the Trilogy, with Vuejs and Reactjs). But the codebase was enormous for the result, and hard to maintain. I tried at least 5 times to migrate to Angular 2, but always failed. And the JSX language of Reactjs appeared to me like a frightning thing and a prohibitive cost to pay (I still find it is a terrible idea for readibility).
Then came Vuejs. Simple. Easy to learn. Natural. No cruft, no crazy syntax (simple v- prefixes like v-if, compared to *ngIf for Angular – really, guys?), a textbook example of how a documentation must be written and displayed (yeah, Python my friend, your documentation is damn difficult to work with). Then came the v3 of the Vuejs Command-Line Interface. Oh man. Even the migration of the project from its original shape to the Vuejs-CLI v3 organisation was a breeze. And thanks God, it abstracts away all Webpack insane complexity (for non-frontend developers, webpack is crazy difficult to learn). To me, Vuejs is what the web development should have always been.
The tests are written with Jest, and vue-test-utils. I have made several attemps with Cypress to make end-to-end tests, but I had trouble installing a stable version on my iMac. Moreover, one could go a long way with Jest, so I am concentrating on unit tests for now.
One last piece that could be categorized as « frontend » in the sense of being a consumer of the APIs and their data is the Arcsecond Command-Line-Interface (CLI), open-source and freely available on GitHub. For a long time I thought it was enough to provide REST APIs of interesting data, I would not commit myself to another Python script / module like every scientist on this planet… But reality is stronger, and I am very happy to have build such tool. It helps integrate api.arcsecond.io more easily into custom workflows.
The organisation of the code
To that day, the backend repository contains 4200 commits, and the frontend repository 4900. Yes, I tend to make a lot of microcommits. The project started for real only two years ago, when I was able to really craft a serious beyond-hello-world frontend app for the long-term. The first commit of the Vuejs frontend was made on Tuesday February 28, 2017. But the first commit of the Python backend was on Tuesday Apr 15, 2014! (Always good to start a project on Tuesdays.)
In the code, I have 295 Vuejs components (.vue files) that allow to build a webapp, which weights 3.1 MB in the end. It is not using Server-Side-Rendering (SSR) so far, and it’s pretty difficult to transform it right now (I have special handlers for dynamic routes for Organisation Portals…). But I intend to increase at least the number of pages using prerendering (see SSR link above). On the backend, even if there is only one « arcsecond » Django monolyth app, the code is split into 24 different Django subapps (which all have their own models, serializers, views, urls, sometimes tasks and/or connectors…).
Yes, the backend app is a monolyth, even if the industry is breezing with microservices. Actually, given the amount of relationships between all the models, the real work of development and its operations appeared to me a lot easier within a monolyth, rater than distinct microservices. I am alone (Eric my friend and partner mostly helps with advices, tests and comments), and arcsecond.io is a kind of product that requires an large amount of development first before starting to be useful. I just can’t spend more time to something else than development.
For the IDEs, I use the pro version of PyCharm since the beginning for the backend (~80$/year). For the frontend I tried Sublime, Atom, VSCode, WebStorm over the years, and finally use PyCharm too, since it has an excellent Vuejs plugin. Finally, I couldn’t commit so fast and so easily, and find the history of the changes so quickly without the excellent Git client Tower (~65$/year – that’s quite a lot just to make commits… but I see Tower more like both a debugging and a management tool at the same time).
The GitHub repositories (repos, in the developers jargon..) are organised as follow:
A single arcsecond-back repo for the backend, sometimes with the help of forks of open-source libraries,
An arcsecond-front repo for the frontend, but using 6 private repos for custom Vuejs libaries,
I should take the time to annouce a bit more often the releases of my various softwares. But software is a mean of expression by itself, in my humble opinion, when practiced everyday. Don’t you read software? … My blog is my code, and this blog is my secondary blog, talking about the first one.
Anyway, please be aware that arcsecond.io is progressing well. It isn’t still ready for a full-blast communication pushed into professional tubes though. But time will come.
In particular, note the fusion of the Objects and Exoplanets page (which are kinda of object, aren’t they?…). It makes the navigation a lot easier. Moreover, Exoplanet Transits have been also added. Pretty nice. Screenshot below, but judge by yourself directly in this example.
Yes, I usually look for icecream colors for the beta banner… Don’t you like this Strawberry-Pistache ?
Moreover APIs are slowly maturing, and some API endpoints are now in pretty good shape. Obviously that includes /observingsites, /objects and /exoplanets. I am working full-speed on /observingruns and /nightlogs right now. But /datasets is also pretty nice already (although not yet complete).
To follow all the progresses see my third blog… the arcsecond.io changelog!
In the meantime, I also released a small update of SwiftAA (v2.0.1) to fix some warnings, making sure it is fully compatible with latest Xcode, and applied a small bugfix about the visibility of a special method (that should be kept internal for that matter). Check it out on GitHub!
I am also busy with new adventures for real professional activities this time. But I tend to usually mutualise the benefits, to increase the pace of releases.
Oh by the way, I am preparing a link between arcsecond.io and iObserve ! It’s time for iObserve’s users to submit new observatories and observing sites not in the app but rather in arcsecond.io. They will further be importable inside the app. It will be great. The benefit is that: it is not manual anymore (which is good for me), but is directly accessible and shared with everybody !
I just reached 50% of documentation for SwiftAA, the best astronomical framework in the language Swift out there! That’s a small milestone, but I’m happy it’s been reached. Now, I must do the same with Unit Tests, which is as important as the documentation.
… but quite a milestone in my master plan (see previous post). After about a year of (discrete periods of intense) work, I’ve decided that SwiftAA, the best collection of astronomical algorithms on Swift, hit the 2.0-alpha stage.
SwiftAA is intended to be the underlying code framework of all scientific computations of the next version of iObserve. With it, I’ll be able to provide tons of details about many objects, and especially about Solar System objects, which are clearly missing in the current app.
It’s an alpha stage, of course. It means a lot of details need to be polished: iOS version polishing, more unit tests, a more consistent handling of numeric types etc. But all of the C++ code is wrapped in Objective-C(++) code, and all that old-style code mimicking the original AA+ one is now « Swifted ». That is, it has been elevated to a lot higher level of expressive formulation.
Complexity remains, since the solar system isn’t quite easy to simplify. Hence, when one has the goal of minimizing the amount of lines of code, to extract the most of it, things aren’t easy to read at first.
But there is a Swift Playground for those interested to learn. I wish I had more time for making this Playground more « ready to use ». But as for now, you need to dive a bit into the thing and the project to actually understand it. But time will come, I’ll prepare a better one.
In my website stats, I noticed that some people keep talking about iObserve, which is great. One post however mentioned the wish to have a Linux version of it. Those interested in what happens here @ onekilopars.ec probably understood that it is also part of the master plan. But current web-based technologies to make cross-platforms apps are difficult to put in place. I’ve tried about 6-7 times. But I don’t give up!
I’ve received about 30 new observatories to be included in the next version of iObserve. That’s really great, as it is the sign of a strong usage of the app (more than 15k downloads so far). They are all in my list of to-dos, but I must say that it is sometimes hard to be motivated to finish this new new version, and I am late. But it will come!
Software is amazing, because it let you create your own world. This is the world as I see it in the future. I learned so much by creating iObserve (over the course of the past years…). And as in science, the main lesson was that I didn’t know much actually…
Hence, since I love challenges, I decided to dive into new questions:
new language for the app iObserve
new architecture for the app
own backend service with specific language
start cloud service with storage and online DB
extract as many macOS components from app into open-source libraries
finally implement dreamed features (night logs, observation simulator…)
I’m sure something is missing…
If someone want to share the bumps ahead on the road, he|she is most welcome.
Clear skies to everyone!
P.S. By the way, arcsecond.io just migrated to Python3…
Dear iObserve users, not many news over here for some time. iObserve 1.5 is still downloaded quite many times every week (reaching 12.5k total downloads!). I can see that some of you send me new observatories. I’ll find some time to create an update of that bunch of new observatories.
Apart from that, I’ve been able to re-compile and re-run iObserve Touch on an iPad with iOS 9.3. And that’s nice, because it wasn’t so easy with the amount of code involved here and there. So maybe I could find some time for a little update as well. It reached almost 6k downloads by itself, wow!
In particular, I am preparing a Swift playground with the best astronomical algorithms in town. The code will be good for developers, for iObserve2, but the Playground could be interesting to teachers! Check it out here the on-going work: https://github.com/onekiloparsec/SwiftAA
Interestingly, this all-code activity is a kind of rest for me. At my startup job, I spend the day interacting with tons of new people (and it’s really great!). So it’s nice to interact with computers a bit. 🙂
Arcsecond.io aims at integrating all sources of astronomical data and information into a unified scheme using modern web techniques.
This is big. Really. If you jump on board, this can be huge!
If you don’t see how big it could be, imagine a world where every resource (the word is key) has a unique, simple, stateless URL. Yes. It means every object, every planet, every lightcurve, telegram, FITS file has a simple, unique URL which returns well-formated fully standard JSON/XML output, consumable by modern web techniques (like AngularJS).
It’s a kind of a super mega SIMBAD or NED services with modern tools and interfaces. And not limited to any type of data. Allowing you to concentrate on stuff that matter: the data. And not its formatting.
Imagine furthermore that your own personal resources: night log, observing runs, reduced data are accessible the same way, through usual individual and group permissions that you personally control.
Imagine even furthermore that community-curated informations is also accessible that way! This is what has been started with observing sites around the world (see below). Imagine now this list constantly updated, and enhanced by informations about domes and telescopes, and further more… instruments and detectors. All accessible freely, always the same way. A kind of scientific data-based wikipedia!
This would allow us to build a bazillion of new services and (web) apps. This is the future of arcsecond.io.
arcsecond.io is intended to full embrace the RESTful principles (that is, the modern way in the web to decouple data from its consumption). I know, there is also the VO. But… Oh well.
Two months already since I made iObserve free, and a month since I started to work in my new startup. And a month without any code! How bizarre! Sorry guys, but as you may have guessed, I made no progress at all on the iObserve front. In fact, during the last month I had only my 7-years old MacBook Pro. Tough to run Xcode and code confortably with it. Now that I received the latest high-end Retina, things have much improved!
Here is the situation at onekiloparsec’s:
iObserve 1.5 is still ongoing. Sky maps coming, along with various bugfixes coming, especially that factor 15 for Right Ascensions when entering coordinates manually. I was planning to put in place an update mechanism outside the MacAppStore, but given the time it takes, I’ll skip it for now. So expect a regular update in the MAS. I can’t make plans for when, but this is definitely on top of the list.
I’ve received a request to support PixInsight XISF in QLFits. That’s very interesting, as I didn’t know about before. I’ll have to check that. Love new stuff.
KPCTabsControl has been updated to play better with AutoLayout projects, because of a developer request. Great spirit among developers around the world; open source can be really cool sometimes.
SwiftAA is stucked in between version 1.0 and 2.0. I really wish I could finish this soon. This is key for the future, but I am stuck on how I should translate pure C++ style / syntax in the most useful / modern Swift style and syntax. Life can be hard sometimes. 😉
Because of SwiftAA is stuck, so is iObserve 2 too… But anyway, this new app depends on the development of arcsecond.io. And I am wondering I could write things in Swift. Probably not on the server side (even if it is possible right now).
But you know what? I’ve been thinking a lot about arcsecond.io lately. I regularly receive emails from the backend about this or this not working well (for instance, duplicate Observatories, yes, I look at you, Crete). I am all aware of it! And I need to do something about it, definitely. Enough said for now. More on that later.
Most welcome to send comments and feedbacks if you can help or discuss. Stay tuned, as always.
Short story: Because I am taking the head of software developments in a newly-created french VR startup, my energy will be focused on that for the coming years. The development at onekilopars.ec will necessarily change to a much slower pace. Nonetheless, I will release iObserve 1.5 and keep supporting everything, including arcsecond.io, as much as time and energy is left. Moreover, I am happy to announce that iObserve is now entirely free. Read on to learn why, for the history of very special app, and the role it has in the life of its developer, with all the details, decisions, downloads, graphics, sales numbers and more!
6 years. What a journey!
Short story: Because I am taking the head of software developments in a newly-created french VR startup, my energy will be focused on that for the coming years. The development at onekilopars.ec will necessarily change to a much slower pace. Nonetheless, I will release iObserve 1.5 and keep supporting everything, including arcsecond.io, but as much as time is left. Moreover, I am happy to announce that iObserve is now entirely free. Read on to learn why, for the history of very special app, and the role it has in the life of its developer, with all the details, decisions, downloads, graphics, sales numbers and more!
December 2009 – July 2010: Take-off
6 years ago, on December 2009, I was in a very peculiar situation. Just two months before, I submitted to the European Research Council (ERC) the most ambitious project I have ever prepared. I was mentally exhausted. I noticed I started to have short-term memory losses… which will continue for more than 6 months. I was still showing up in my astrophysics lab in France, but I just couldn’t keep going doing usual research projects. I was convinced of the revolutionary insight of the project I’ve just submitted, and onto which I bet all my academic future (since ‘usual’ routes for tenure were obscured anyway), and thus my child dream of becoming an astrophysicist too. Of course, I didn’t want to see that ERC reviewers wouldn’t probably take the risk of funding such project (despite what is being advertised). My name was Ego.
On that end of the year, I needed to do different stuff. I just couldn’t simply relax (waiting has always been hard for me…), and I needed some new horizon to look at. On December 28th came that email message from Dr. David Nicholls (to whom I send my warmest regards), from Down Under, kindly informing me that he is a happy user of a small OS X widget of mine, if only I could fix two small bugs…
It was the good old times of OS X 10.5 Leopard with the amazing Exposé and the Dashboard widgets.
Most of us, I am pretty sure, remember that time where we could feel the rising energy and momentum of the Apple platform, combining the best of the two worlds: a true UNIX system onto which one could install most of our astrosoftwares and packages, combined with an elegant OS for our personal digital life.
Back in the times when working at the La Silla Observatory, I started to work on some basic but useful widgets for astronomers. In order of release date: AstroTimes, AstroAirmass, AstroObservability, and finally the one used by Dr Nicholls: the UltimateAstroWidget (the ugliest software name and icon ever, probably), which combines the functionalities of all first 3 widgets. When preparing this post, I managed to retrieve all 4 widgets in my archives! You can download and try all them out (some UI glitches remain, and not all of them work flawlessly, but, fun to try and see what code is inside it – use Ctlr-Click on one widget in the Finder).
That email message was the kick I needed. I really took the widget code and threw it inside an Xcode project, and refactoring literally everything in Objective-C along the way… A little Quicklook plugin, called QLFits, that I started some time before was also helping with the Objective-C transition. I sent to David the version 0.1 of a true OS X app just 5 days after his message! iObserve was born. Look at what it looked like on January 2nd, 2010:
The nice thing is that, while preparing this post, I also found a copy of all old versions of iObserve in my archives… so you can also try yourself! No worries, it has no impact whatsoever on an official version downloaded from the MacAppStore. However, if you had an old version (<1.0), you may need to delete the folder « ~/Library/Application Support/iObserve » (otherwise, the app may crash at launch). Amazing to see that basic Apple/Cocoa APIs are perfectly stable over more than 6 years while so much has changed otherwise.
Within 2 months, I released iObserve 0.2, 0.3, 0.4 and 0.5 (yes these are all download links). David Nicholls sent me awesome feedback and support. His colleagues in offices next to his own also started to use it. I was amazingly happy to have such direct and positive impact. What a contrast with everyday research. iObserve also started to be used in real conditions in Cerró Paranal, home of the Very Large Telescope (thanks to Julien Girard) and Siding Spring (thanks again, David)!
iObserve was a mental life buoy. I started learning as much of the Apple/Xcode/Objective-C stuff as I could. At that time, I considered Xcode 3, the Cocoa IDE, an amazing thing… (Xcode users will sense the irony). I knew I would certainly like a software developer life, given all the software I was already doing since many years (star collisions, basic n-body simulations, image and spectra processing, spectroscopic instrument quicklook, OSX dashboard widgets and plugins, massive stars wind-wind collisions distributed modelling, stellar black-hole and accretion disk emissions along with tons of scripts, wrappers, automators etc). But of course, software was a rescue path (thanks to those who reminded this to me during job interviews…). The clash between doing something I nonetheless like, and having dreamed of something different, remained.
At the end of March 2010, version 0.6 was released, and the ERC panel rejected my project for absurd reasons (no need to go into the details, but there really were artificial). I was kicked out of academia, my child dream was smashed out: I will not become an astrophysicist. 36 years old, 2 young kids, career break, not an easy time. I started to look for a regular job. Beginning of April, version 0.7 was out and by the end of July, version 0.8, and very basic elements of iObserve on an iPad simulator.
This iPad version was truly a good move. On July 27, 2010, I was hired by a small company here in the Grenoble region, for doing iPhone software, thanks to it!
First lesson: a purely personal project, no matter its size, is the best business card you can have with you, for software positions, especially when you have little experience. And I still think the same, you will see why at the bottom.
August 2010 – March 2011: Transition, Transformation and Hitting the Store
Having a ‘regular job’, I started the quite difficult path of transformation to the normal (i.e. industry/business-wise) vocabulary. It was tough. Because I was aware of the privilege of doing something (astrophysical research) aligned with my dreams, up to that time. The awareness of having lost that was unacceptable. One could argue, looking at my publications list, that I didn’t really do what is necessary to get a permanent position (spending time on things like Thorne-Zytkov objects, really)? In my mind, I was not doing enough of it.
A job? That is, getting paid for letting companies use half of your awake time, and the best hours of it? I felt it was tough (of course, there are much tougher things in the world). But it was probably even worse for people around me… who were hesitating between the « how nice and interesting guy who did astronomy » and « what an asshole he is, thinking he knows everything ». Most of them stopped hesitating after some time, quite naturally.
I was behaving like an asshole, I know. But I was simply unable to accept my failure. I considered myself as, finally, the kind of scientist I always aspired to be, taking a step beyond incremental research, and for that reason, being deprived of the means to continue (or in fact, to start for real…), in what constitutes an utmost injustice. The European Research Council review of my project didn’t help me to think otherwise: my project was considered not only good, but rather the exact kind of project ERC wants to support… but no, I can’t have the funds. One of the member of the ERC review panel came to my office in Grenoble and told me exactly this to my face, and suggested to resubmit the next year, which I did, as a loyaulty for myself.
At that time of your life, when you feel you can finally change the world, hearing that, and finally working anonymously in an ugly environment with guys not especially interested in things other than computers (and not Macs…), doing stuff for which I had no particular care was the most shitty feeling I ever felt.
Below an example of the thing I was doing just before leaving academia. I am not even sure it has been published…
Anyway, I not only learned how to behave in a slightly more normal way (that is, starting to get interested in other people), but I also learned how far I was from a real software developer… I had a conceptual understanding and basic knowledge but no real experiences of things like encapsulation, (de)serialization, storage/DB, asynchronous operations, memory pointers and de-reference, thread safety, composition vs inheritance, operator overloading, design patterns, network requests vs sockets etc.
Lesson #2: Given that this technology/software learning takes a lot of time, and is never finished, software positions do not rely so much on diplomas. Which may be hard to hear when you hold a Ph.D and multiple years of experience, with some in very tricky simulations. But this is a mental transition you have to make. What matters then is to participate to the flow. Obviously, only very few people make major contributions (like a new language, a new kernel…). Hence, it is good to accumulate small things you’ve done, usually in open source, for instance in a GitHub account. This is first useful for you, since it gives you a sense of your trajectory.
No true industry-wise experience despite my 8 years of coding, data-processing and simulations, okay. But I knew how to do stuff, and iObserve was the proof of it. It was my own diploma. Because my otherwise skills were totally unimaginable for someone outside research. Could you really consider applying for a job saying you were doing stellar black-holes emission spectra? Ok, I also did public speaking in international conference, wrote books (Ph.D.) and sophisticated technical reports, and being very autonomous at managing projects. But what kind of job is it exactly? That translation of your experience into a more natural language, that fits in the vocabulary of the company you apply to, in front of people who have no fucking idea nor interest for the science, is HARD. Because you just can’t simply explain! (Actually, stop explaining the whole world for every question…) You have to become this. A new yourself. Other candidates also apply for that job, and « being good » is not enough! And what you see is that your life depends on your ability of transforming a life-long passion into a list of skills… My name was Anger.
Resolution #1: Never consider people as a list of skills in your job. Or anywhere else, for that matter. And be nice with people who obviously need to provide explanations for everything. They need some more time to settle down…
Hence, in order to avoid the pain, I immersed myself into iObserve code. I made plans for a universe of features! I started to promise an enormous amount of cool stuff. Did you notice the presence of Observing Runs and Night Logs in the above screenshots (v0.8)? Something I also promise with arcsecond.io… I spent almost a year inside this very long 0.8 serie, preparing Night Logs (see below), which never appeared in the final app.
Another lesson (#3): you may work an enormous amount of time on something, but if existing users actually want something different, you have to change your priorities. That’s a bit different from a pure lean attitude where you look for exactly what the users and/or market want. iObserve development has always been strongly influenced by the feedback from users. But not all of it.
Nevertheless, it was a wonderful canvas to ask myself all the good questions you have to ask when developing a complete product. On February 23, 2011, roughly a year after starting iObserve, I announced on my blog that I was stopping its development… as it was, to focus on a version 1.0 to be submitted to the new MacAppStore. The goal was to stop playing with a toy, and to make a pro app for people to rely on for the coming years. This became a reality on March 23rd, 2011, for a price of 9.99 $US, with a smaller but better subset of features.
That price tag was the result of enormous amount of thinking (as usual, would say people who know me… but I’ve much improved lately). I was about to earn a bit of money with something made with my little hands. But I couldn’t ask too much either, as I wouldn’t have any client (despite very encouraging emails I got from all around the world during the beta stage of development). I thought the price should reflect that it was not « just a small app » (that is, it should be more than 5 $US) but its scope and quality couldn’t put it – yet – in the category of the pro apps (say, 20 $US or more). At that time, I followed closely what the guys at mekentosj.com were doing with their app Papers, dreaming I could have the same success story.
Price went up with time, but you will see below that purchase rate was remarkably stable over the years. The iObserve community grew slowly along with my own coding and modern software experience… and skills.
Other lesson (#4): If someone send you a bug report, verify and admit immediately it is a bug. Emails seemingly aggressive from people upset to find a bug in something they have paid for is perfectly understandable. But you may have a hard time at fixing it if the guy stays upset and you rush to the store for a bugfix release. And admitting rapidly your mistake lets you engage a useful conversation with the reporter, and gives you a chance to promote new features etc. In 6 years, I could remember of only one condescending email to whom I answered appropriately. All the others were pretty much fantastic feedbacks.
April 2011 – December 2013: Crazy Versions, Desert Crossing and False Hopes
iObserve was on the Store. Now what? I spent the two and a half difficult years hitting walls, and slowly mourning my past life. Missing a lot traveling and landscapes, among others. On July 2011, I nonetheless had the great chance to participate to the Apple World Wide Developer Conference (WWDC), the last one with Steve Jobs alive.
But after a year working for Motwin, I realized that I would never make any more progress in there and I had to move (the company itself was not in good shape and management was truly awful). I started to apply in many places, with a strong will at coming back to physics, somehow. I wanted to value a lot more my past experience of physicist. I probably felt that my experience on software wasn’t large enough, that my professional progress in software would be slow, and that I could earn a better position right now by focusing on physics rather than software (I felt I deserved it… which is a very bad attitude). Until then, I never really understood that the rising buzzword of ‘Data Scientist’ was the way businesses and industries swallow former scientists. For me, reducing a scientist to its data analytics… skills, or worse, its purely technical and software ones, was just plain wrong. Hence I never looked for such positions. See how an idealistic vision of the world can be fun? 🙂
Among others, I applied to Alstom, ESRF (they needed a mixed physicist / software guy! they never replied to my emails), CEA (multiple times), Mathworks, Movea, Sofradir (who didn’t want to consider a Swiss guy because of National Defense projects, then I became french, and applied again, but to no avail), Comsol, Cognizant, Wisekey, Xerox etc. All failures. Once, I had 3 interviews within 10 days, and spent a total of 8 hours in interviews. Failed. Most of the companies don’t give a fuck anyway, sometimes never even bothering replying to emails.
Resolution #2: If I need to hire people one day, I will engage actively at setting up a communication channel with every candidate to let her/him know her/his applicant status. I hate waiting. I hate not knowing, with no fixed dates. In that situations, silence from the employer can suck all of your mental energy. And that’s just the opposite of what we should look for in people (not list of skills…) interested by a job.
In the meantime, iObserve was progressing a lot, but I was ridiculously stuck into the 1.0 versioning scheme. Hence the versions 1.0.1, 1.0.2, 1.0.3 to 1.0.13… The reason was that my plans were huge for that app and I considered updates a purely small ones given the beautifully enormous things coming! But version 1.0.5 brought Finding Charts (should have been v1.1), and v1.0.7 brought Star Tracks Plots (with the possibility to import custom ones, should have been v1.2). Not that bad!
Some people play music, makes extreme sports, engage in associations, or prefer drugs, religion or alcohol. All these times, I had iObserve. To go through reality.
Hence, during this second semester of 2013 – unemployed, I had lots of time – , I released iObserve 1.3, with Converters, Exoplanets and Small Bodies (Asteroids + Comets)! A big big update. For 19.99 $US (the price was decided one day before release). Interestingly, this was the time I finally found the right « identity » I wanted for my digital life: onekiloparsec (whose origin is to be found in this article and associated ones). That was it. New website, new Twitter account, new username everywhere. The previous name (Soft Tenebras Lux), was based on a word trick only software engineers knowing Geneva’s motto could possibly understand.
I kept going with updates, while I was preparing iObserve for iPad to also finally reach the Store. It was an enormous amount of work to unify OS X and iOS APIs to be able to use the exact same codebase for both apps. I wouldn’t do it that way anymore today. More experienced, you know… iObserve for iPad was finally released on December 3rd, 2013, with an arm-long list of bugfixes for v1.0.1 on January 2nd, 2014. The whole iObserve project reached officially about 80+ kloc (kilo-lines of code).
In the meantime, from July to December 2013, I was in contact with people at the CEA (Commissariat à l’Energie Atomique, one of the big science / technology institute in France, that has a large campus in Grenoble) to … launch a startup! In a wonderful field I loved: optics and ultra-fast detectors. Yeah, back to Physics again! I thought. No need to go into the details, but it never materialized either. Mostly because managers never wanted to take a risk they considered too large… Does it ring a bell? I told myself: AGAIN?!?!?! Am I that stupid guy who always rushes and bang his head into risky walls?! What is exactly I was doing wrong? Apart from being myself?…
Hence, at the end of December 2013, I accepted a job at Hortis.ch for the Radio Télévision Suisse (RTS, the Swiss national broadcaster) thanks… to iObserve for iPad! During the interview at RTS, I brought only my iPad. They particularly liked the way I abstracted the connections to multiple webservices, and the rigor with which curves were computed and drawn. But I was about to be 40. I was still doing iOS software and not physics. And I had to work 4 days/week away from my family (in Geneva). My name was Capitulation.
January 2014 – December 2015: New Friendship, and New Openings.
So that was it, I was 40 and I was doing iOS software. Why on Earth I wasn’t doing other things, to diversify my portfolio and pretend to other jobs?! Well, I guess today I can answer quite simply: confidence. Or the lack of thereof. And the lack of a personal project outside the Apple platform (which is tightly coupled). Hired as an « iOS expert » at Hortis/RTS, I remember having an attitude to make this clear to everyone… (in a beautiful proof of psychological deny of this lack of confidence, and you need some to understand that being unemployed is an opportunity). Anyway, iObserve proved again to be useful at finding jobs. So why risking to weaken my most important asset?
But one thing was clear: I would not make iOS software all my life. Even with the Apple momentum, once you’ve learned iOS3, 4, 5, 6, and 7, iOS8 comes surprisingly as a smaller event… I needed something bigger, and more complex! One, and only one obvious, way out: startups and entrepreneurship!
Interestingly, the thing I was saying when I left academia was this: the mobile software industry is certainly a good place to make good money and earn a living, and even have a chance to earn enough money to buy me enough time to continue my studies on my own. I just didn’t realize the real journey it would be, both in terms of efforts and also fun, yes, fun, sometimes (and that money wouldn’t be so voluminous, even if it is still much better than academia).
Talking about money, here are the numbers for iObserve. Since the beginning of the OS X app on the store (that is 4 years and 9 months ago), it has been downloaded almost a thousand times (50% from the U.S. 30% from Europe), and not even a hundred times on iOS. This last number is a real disappointment, given the amount of work I’ve put in it. I am pretty proud of the result, but the market is probably not large enough. Given the way it is written, with some very special handling of rotations and split views, it is not easily modernizable to the latest (iOS9) APIs. Hence, I will only continue to support it as much as I can (say, iOS10?), but not more. [update: see the numbers since it became free.]
In total, I earned about 12 k$US (that is, fluctuating between 80 and 200€ every month). For that, I am very grateful to all my customers/users. The fact that iObserve was a paid app made this journey a lot more interesting. And thanks to all, among other small things, I’ve been able to buy some good wine, make nice gifts to my two boys, and offer a really good bike to my wife.
Startup and entrepreneurship? Okay. The reasoning was simple, and backed up with the real numbers above: the market of ‘pro astronomers’, even considering the serious amateurs, is too small to start a business. I had ideas (and already some good code!) of other pro apps to increase the size of my portfolio. But still. These are truly complex apps. I could foresee a big price jump (say up to 200-500 $US), but it would need to come with some serious features that it will anyway takes time to develop. Moreover, no one is making enough money alone with some general public mobile app (even if it pays a lot more on iOS than on Android). I had no money to hire some help, and I didn’t dare to contact investors and banks, considering that the market is small and will remain so anyway. And all these developments at onekilopars.ec were sucking already all the personal time I had left.
So. Let’s look for something else! As a matter of fact, along with iObserve and all the software I was doing, I had a ‘grand public’ idea I was cooking in my head for a long time (ah, mental cooking…). It was about short stories, allowing people to create, branch and update stories of their own or from others. Back in November 2011 already, I presented this idea in a Startup Weekend happening in Grenoble. My idea was not selected; first time I was pitching… But I met a very nice guy whose name is Laurent, of basically the same age, also interested in entrepreneurship. We slowly became very good friends, talking about many things, taking a lot of pleasure at sharing ideas, discoveries, links, articles about technology, entrepreneurship, professional careers, wives and kids etc. We finally decided to launch something based on that idea around stories and fictions.
We learned that we really liked to work and explore together. You know, not the goal is most important, but the journey… Moreover, we had a lot more assets: we know how to deploy a multi-language client/server webapp, we know what difficult questions we need to ask ourselves and what are our true motivations, like every entrepreneurs. Hence we decided to continue, in another subject! We just had to find that subject! And that part was also a lot of fun (Laurent is very good at knowing lots of different technology trends and stuff).
But then came arcsecond.io…
Last summer (2015), all along on-going day work at RTS in Geneva and explorations for startups with Laurent, I was struggling with the name of an idea I had already since quite some time, thanks to PicoLegends. As a matter of fact, RTS colleagues convinced me a while back to write iObserve 2. It would be a totally new, super-pro version of iObserve, with Night Logs and a high price tag. Quite logically (experience talking…) it needed a true dedicated backend server, with a lightweight client consuming its well-formatted data. That is, the idea was to put on a server all that crazy logic of data requests made by every today’s iObserve client apps, which are all different whether it is SIMBAD, NASA ADS, JPL Horizons and its telnet/ASCII interface, or exoplanet.eu…
I wanted to give a name to that server, and I couldn’t find one good. But at the minute I found ‘arcsecond.io‘, everything became clear. I bought the domain name, and started to code at the speed of light. I was very happy. And during my vacations I realized that arcsecond.io was not a simple project, a simple backend for iObserve. It could really be an entire cloud for astronomy, collecting all meaningful data, provide well-formatted APIs for anyone, including me, to consume this data and create new services. Hence the two feet onto which it is based: ‘api.arcsecond.io‘ for the data itself. And ‘www.arcsecond.io‘ for an AngularJS-based web app. My regular job at RTS was taking (by choice) only 4 days a week. Hence I had one day/week, evenings and week-ends left to code arcsecond.io. My name was Software!
But that was a too big Everest for Laurent, my partner. Too much knowledge and experience on my side and not enough on his own. And a not-so-easy business model anyway for arcsecond.io itself! We decided to continue in a dual-coaching mode. I kept doing arcsecond.io and he provides feedback and advises. And on the other hand, he continues to explore other paths for entrepreneurship, while I give him my feedback.
December 2015: Connecting the dots…
You probably know the famous speech made by Steve Jobs when he receive a honorific degree at Standford. If you don’t, here it is. It’s a real good speech.
On December 2015, I was connecting the dots. Finally.
In a conversation at the Science Fest of the … CEA in Grenoble, I talked to a colleague of my wife about all the software I was doing. And he told me that, with some colleagues and other people, he was about to launch a startup about some exciting new Virtual Reality technology. A few days later, while sending him an email with some useful links, I asked him whether they would need an experienced software guy in their startup? I finally meet the co-founders – awesome and diverse people – and they were actually interested in my experience! All of it, somehow. That is, not as a list of skills… I couldn’t thank them enough for this, and the opportunity they gave to me.
When I met them, I talked about all my experience and the products I made, which comprised basically none of those I made during regular jobs. Only my own projects. Of course, regular-job projects contributed a lot to my experience. But that’s small compared to the pride to talk about your own stuff. It makes a key difference in the impression you leave in people’s mind. And my experience was not only iObserve but also arcsecond.io (web frontend + data-based modern APIs backend!), my open-source plugins and SDKs, and even the old image-processing / scientific developments. And also something I forgot to mention: my 2 years of experience with the agile Scrum methodology at RTS in Geneva, in which it is truly implemented throughout the multimedia department. By the way, I am a certified Professional Scrum Developer, since August 2015; you know, diplomas/certifications can still make some good…
Quite naturally, I jumped on board, and joined the startup.
So here is the end of that journey, and I will start a new one. I will now build teams of people and take careof all software developments in a very exciting startup that will integrate an amazing technology stack and deliver a unique and fantastic VR experience! Can’t be more excited. And relieved.
One of my very best friend told me once that he likes (he lives by?) a quote from Churchill:
Success is the ability to go from one failure to another with no loss of enthusiasm.
When I read it, I thought: yeah, yeah, easy to say… But he was right. iObserve was my reserve of enthusiasm. My companion for never giving up.
My name is Cédric, also known as @onekiloparsec on the Internet. And I am happy to make iObserve, my beloved 6-years-old app made during countless night hours, free for everyone from now on. It will remain on the MacAppStore for a while, to give me the time to prepare the transition to a custom distribution channel from my website. If you nonetheless would like to pay something for it, you can donate some Bitcoins to the following address.